_get_or_create_sub_manager is only reached from add_request and add_requests, but every read
path — fetch_next_request, is_empty, is_finished, get_handled_count, get_total_count,
purge, drop — iterates _sub_managers. On a fresh process that dict is empty.
- Requests a previous run left in a
throttled-<domain> queue stay invisible until something adds a
new request for that same domain. Until then fetch_next_request takes its
if not self._sub_managers: return None short-circuit and the crawl reports itself finished with
work still queued.
purge() and drop() skip them for the same reason, so they cannot be cleared either.
The sub-queues live under a stable alias specifically so they outlive the process, which this
defeats. The TS port opens every configured domain up front rather than on insert
(apify/crawlee#3741).
From #1762.
_get_or_create_sub_manageris only reached fromadd_requestandadd_requests, but every readpath —
fetch_next_request,is_empty,is_finished,get_handled_count,get_total_count,purge,drop— iterates_sub_managers. On a fresh process that dict is empty.throttled-<domain>queue stay invisible until something adds anew request for that same domain. Until then
fetch_next_requesttakes itsif not self._sub_managers: return Noneshort-circuit and the crawl reports itself finished withwork still queued.
purge()anddrop()skip them for the same reason, so they cannot be cleared either.The sub-queues live under a stable alias specifically so they outlive the process, which this
defeats. The TS port opens every configured domain up front rather than on insert
(apify/crawlee#3741).
From #1762.