Skip to content

Conversation

@Kleissner
Copy link

@Kleissner Kleissner commented Oct 27, 2019

Re #71

Kleissner added 2 commits October 27, 2019 12:30
…e response body.

By default it is -1 which means disabled.
@mna
Copy link
Member

mna commented Oct 27, 2019

Hello Peter,

Thanks for the PR, makes sense again, but a couple comments:

  • how would you feel to use 0 as default limit to mean "no limit" instead, so that a zero-value Options struct would work without a limit (a read limit of 0 does not make much sense, you should emit a HEAD request if you're not interested in the body)?
  • as for the other PR, if you could add a unit test with a > 0 read limit that checks the resulting behaviour (e.g. if it stops reading before an <a> anchor, that anchor's URL should not be enqueued in the crawler).

Thanks,
Martin

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants