Skip to content

test: fix flaky test_large_batch_operations #780

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
vdusek wants to merge 1 commit into from
Closed

test: fix flaky test_large_batch_operations #780

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 10 additions & 6 deletions tests/integration/apify_api/test_request_queue.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -548,15 +548,19 @@ async def test_large_batch_operations(request_queue_apify: RequestQueue) -> None
total_count = await rq.get_total_count()
assert total_count == 500, f'total_count={total_count}'

# Process all in chunks to test performance
# Process all requests using a retry loop to handle eventual consistency —
# fetch_next_request() can temporarily return None even when requests exist.
processed_count = 0
max_consecutive_empty_fetches = 10
consecutive_empty_fetches = 0

while not await rq.is_empty():
while consecutive_empty_fetches < max_consecutive_empty_fetches:
request = await rq.fetch_next_request()

# The RQ is_empty should ensure we don't get None
assert request is not None, f'request={request}'

if request is None:
consecutive_empty_fetches += 1
await asyncio.sleep(0.5)
continue
consecutive_empty_fetches = 0
await rq.mark_request_as_handled(request)
processed_count += 1

Expand Down
Loading