Skip to content

ThrottlingRequestManager domain matching misses punycode, trailing dots and IPv6 literals #2144

Description

@janbuchar

domains is normalized with d.lower() while lookups go through URL(url).host. yarl normalizes
differently, so three configurations never match anything:

  • domains=['xn--hky-ela4t.cz']URL(...).host decodes it back to háčky.cz
  • domains=['example.com'] against a root-dotted http://example.com./… — yarl keeps the dot
  • domains=['[::1]'] — yarl strips the brackets

Each is a silently unthrottled domain. Same symptom as #2126, different cause: there the recording
code never runs, here it runs and looks up a key that cannot exist. The TS port puts both sides
through the URL parser so they land on the same form (apify/crawlee#3741).

From #1762.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions