Skip to content

fix: Add is_finished and align is_empty semantics in ApifyRequestQueueClient#1008

Open
Mantisus wants to merge 5 commits into
apify:masterfrom
Mantisus:queue-is-finished
Open

fix: Add is_finished and align is_empty semantics in ApifyRequestQueueClient#1008
Mantisus wants to merge 5 commits into
apify:masterfrom
Mantisus:queue-is-finished

Conversation

@Mantisus

@Mantisus Mantisus commented Jun 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Description

Add is_finished and align is_empty semantics in ApifyRequestQueueClient

Closes: #987
Follow-up: apify/crawlee-python#1982

@Mantisus Mantisus self-assigned this Jun 22, 2026
@Mantisus Mantisus requested a review from vdusek June 22, 2026 23:26
vdusek
vdusek requested changes Jun 23, 2026
View reviewed changes

@vdusek vdusek left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One comment

Comment thread src/apify/storage_clients/_apify/_request_queue_shared_client.py Outdated
@Mantisus Mantisus requested a review from vdusek June 23, 2026 11:25
vdusek
vdusek reviewed Jun 25, 2026
View reviewed changes

@vdusek vdusek left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Three more comments

Comment thread src/apify/storage_clients/_apify/_request_queue_shared_client.py
Comment thread src/apify/storage_clients/_apify/_request_queue_single_client.py Outdated
Comment thread tests/integration/test_request_queue.py Outdated
vdusek
vdusek reviewed Jun 25, 2026
View reviewed changes

@vdusek vdusek left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we also use a fix here please? Same reason as explained here apify/crawlee-python#1982 (review)

Claude suggests one of these:

  • fix: prevent RQ softlock by correcting is_empty in ApifyRequestQueueClient
  • fix: add is_finished and align is_empty semantics in ApifyRequestQueueClient

I am ok with both of them

@Mantisus Mantisus changed the title feat: Implement the is_finished method in the ApifyStorageClient fix: Add is_finished and align is_empty semantics in ApifyRequestQueueClient Jun 25, 2026
vdusek
vdusek reviewed Jun 26, 2026
View reviewed changes

@vdusek vdusek left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's get back to this, once Crawlee 1.8.0 is released.

@vdusek vdusek self-requested a review June 26, 2026 13:06
vdusek
vdusek reviewed Jun 30, 2026
View reviewed changes

@vdusek vdusek left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do I understand correctly that this requires a new version of Crawlee?

If yes, then this would be a breaking change on our side, and we'd need to bump the lower version boundary. Is there a way to make this non-breaking so we don't have to bump it?

@Mantisus

Mantisus commented Jun 30, 2026

Copy link
Copy Markdown
Collaborator Author

If yes, then this would be a breaking change on our side, and we'd need to bump the lower version boundary. Is there a way to make this non-breaking so we don't have to bump it?

I added a small workaround. This will allow us to maintain compatibility until we're ready to bump the Crawlee version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vdusek vdusek vdusek requested changes

Requested changes must be addressed to merge this pull request.

Assignees

@Mantisus Mantisus

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Implement the is_finished method in the ApifyStorageClient

3 participants

@Mantisus @vdusek @apify-service-account