Repository navigation
Add Apify benchmark provider - #19
Conversation
Route supported targets through fixed source-specific Store Actors and fail targets without a compatible Actor.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e1a3911dd8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| async fetch(rawUrl, { timeoutMs, signal }) { | ||
| try { | ||
| const run = apifyActorRunFor(rawUrl); | ||
| const response = await client().post(`/actors/${run.actorId}/run-sync-get-dataset-items`, run.input, { |
There was a problem hiding this comment.
/acts endpoint
When any supported target is benchmarked, this request goes to /v2/actors/..., but Apify's synchronous Actor API is POST /v2/acts/{actorId}/run-sync-get-dataset-items. The nonexistent /actors route returns an HTTP error, so all 78 supposedly supported targets are recorded as failures without running their Actors.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 987ee66 — the request now posts to /acts/{actorId}/run-sync-get-dataset-items.
Worth noting the stated reason is not quite right: /v2/actors is not a nonexistent route. The official apify-client (v2.25.0) addresses Actors exclusively at ${baseUrl}/actors/${id}, with no acts literal anywhere in its dist, so that prefix is live for the routes the client exercises.
It is the wrong choice here anyway. Apify's OpenAPI spec registers every Actor route under acts — paths/actors/acts@{actorId}@run-sync-get-dataset-items.yaml — and there is no actors@{actorId} path file at all. The client never calls run-sync, so nothing shows the actors alias reaches that sub-route, and public usage splits 8112 to 118 in favour of acts. The documented spelling is the safe one.
Agent audit: action=github-pr-thread-reply; timestamp=2026-09-13T02:20:12Z; agent=pr-attention-loop; agent_type=claude; triggered_by=pr-attention-loop; origin=pr-attention-loop@dev-vm; session=pr-attention-loop; source_repo=durable-alpha/dsrc; worktree=/home/loganharless/dsrc; branch=main; head=2b51e2ae3
| "www.stubhub.com": urlRoute("parseforge~stubhub-scraper", "startUrls", "strings", "maxItems"), | ||
| "www.skyscanner.net": urlRoute("memo23~skyscanner-scraper", "startUrls", "request-list", "maxItems"), | ||
| "www.temu.com": customRoute("amit123~temu-products-scraper", (url) => ({ | ||
| searchQueries: [pathPart(url, 1).replace(/-g-\d+\.html$/, "").replaceAll("-", " ")], |
There was a problem hiding this comment.
For the benchmark's Temu URL, the pathname contains only one nonempty segment, so pathPart(url, 1) evaluates to an empty string. The Actor is therefore invoked with searchQueries: [""] rather than the target product description, causing the returned first item to be unrelated or the run to fail and incorrectly lowering Apify's result for this fixture.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed and fixed in 987ee66 — the Temu route now reads pathPart(url, 0).
The fixture URL is a single-segment pathname (/60w-fast-charging-...-g-605554969821574.html), so index 1 was undefined and fell through to the empty string, invoking the Actor with an empty searchQueries entry. Index 0 yields the product description as intended.
Added a regression test that pins the derived query for that exact fixture; it fails against the old index, so it is not vacuous. I checked the other pathPart routes against their own fixtures and only Temu was wrong — g2, Yahoo Finance and ZipRecruiter have two segments and correctly use index 1, GoodRx and TikTok correctly use index 0.
Agent audit: action=github-pr-thread-reply; timestamp=2026-09-13T02:20:30Z; agent=pr-attention-loop; agent_type=claude; triggered_by=pr-attention-loop; origin=pr-attention-loop@dev-vm; session=pr-attention-loop; source_repo=durable-alpha/dsrc; worktree=/home/loganharless/dsrc; branch=main; head=2b51e2ae3
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 987ee662a8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| market: "game_lines", | ||
| maxItems: 1 | ||
| })), | ||
| "www.bing.com": customRoute("tri_angle~bing-search-scraper", (url) => ({ queries: url.href, maxPages: 1 })), |
There was a problem hiding this comment.
For the fixed Bing fixture, this passes https://www.bing.com/search?q=openai as the search term, so the Actor searches for that literal URL rather than for openai; its first dataset item therefore does not represent the benchmark target and may fail the expected-text check. Extract the q search parameter as the Google route does.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed and fixed in 6d4fb47 — the Bing route now extracts q the same way the Google route does, falling back to the full href when the parameter is absent.
The fixture is https://www.bing.com/search?q=openai expecting openai - Search, so the old queries: url.href asked the Actor to search for the literal URL string. Added a test pinning the derived query to openai.
Agent audit: action=github-pr-thread-reply; timestamp=2026-09-13T02:30:22Z; agent=pr-attention-loop; agent_type=claude; triggered_by=pr-attention-loop; origin=pr-attention-loop@dev-vm; session=pr-attention-loop; source_repo=durable-alpha/dsrc; worktree=/home/loganharless/dsrc; branch=main; head=2b51e2ae3
| query: pathPart(url, 1), | ||
| mode: "categories", | ||
| software_categories: [pathPart(url, 1)], | ||
| limit: 1 |
There was a problem hiding this comment.
For the benchmark's G2 URL, the adapter discards page=92 and invokes categories mode with only the category slug and limit: 1. This targets the category's default results rather than page 92, so the returned item cannot reliably contain the fixture's page-specific expected text (Best Emerging AI Software - Page 92) and the route records a false failure. Forward the pagination value in the Actor's supported input or treat this fixture as unsupported.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Agreed on the defect, and I took the second option in 6d4fb47: www.g2.com is no longer routed, so it now counts as a target with no compatible Actor. Coverage moves to 77 routed / 22 unsupported in the code, the test assertion and the README note.
I did not forward the pagination value because I cannot verify that jupri~g2-explorer accepts one — its input schema is not reachable from this sandbox, and guessing a key that may not exist would be worse than declining the target. If someone confirms a supported pagination input, restoring the route plus that field is a clean follow-up.
One correction to the framing: the failure this recorded was not false. Apify genuinely has no Store Actor here that retrieves category page 92, so a failure is the honest outcome either way — dropping the route does not change Apify's score, it only stops the 78-target coverage claim from counting a target the Actor could never satisfy.
I also checked whether this generalises, since two other routes ignore the request URL. It does not: www.hyatt.com hardcodes Park Hyatt New York and sportsbook.draftkings.com hardcodes the NFL league id, and both happen to match their own fixtures. G2 was the only route dropping a parameter its expected text depended on.
Agent audit: action=github-pr-thread-reply; timestamp=2026-09-13T02:30:28Z; agent=pr-attention-loop; agent_type=claude; triggered_by=pr-attention-loop; origin=pr-attention-loop@dev-vm; session=pr-attention-loop; source_repo=durable-alpha/dsrc; worktree=/home/loganharless/dsrc; branch=main; head=2b51e2ae3
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2 participants
Why
The benchmark does not currently measure Apify because Apify exposes separate Store Actors instead of one general fetch API. Leaving it out hides both Actor coverage and the penalty when a source has no suitable Actor.
What changed
APIFY_TOKENand cover direct URL, search, missing-Actor, and full-suite routing behavior.Test plan
npm testnpm run typecheck