test: add e2e tests for Crawlee crawlers as Apify Actors #784
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
t-tooling
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #784 +/- ##
=======================================
Coverage 81.86% 81.86%
=======================================
Files 46 46
Lines 2696 2696
=======================================
Hits 2207 2207
Misses 489 489
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
4 tasks
Add 6 e2e tests (one per crawler type) verifying that each Crawlee crawler works correctly when deployed as an Actor on the Apify platform. Each test exercises link discovery, data extraction (push_data), and KVS storage against a local 5-page e-commerce test server. Crawlers covered: BasicCrawler, HttpCrawler, BeautifulSoupCrawler, ParselCrawler, PlaywrightCrawler, AdaptivePlaywrightCrawler. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Playwright browser process uses ~244MB at startup, exceeding the 256MB default. Both PlaywrightCrawler and AdaptivePlaywrightCrawler tests timed out due to memory pressure. Add memory_mbytes parameter to make_actor and set it to 1024MB for Playwright tests. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
enqueue_links/add_requests), data extraction (push_data), and KVS storage (Actor.set_value)conftest.py: ASGI test server, Playwright Dockerfile template, product data expectations, and a_verify_crawler_resultshelper that checks run status, dataset contents, and KVS recordsRelates: #785
Test plan
poe e2e-testswithAPIFY_TEST_USER_API_TOKENset to verify all 6 crawler tests pass on the Apify platformpoe lint🤖 Generated with Claude Code