requestHandler and postNavigationHooks seem redundant

[matjaeck]

Which package is the feature request for? If unsure which one to select, leave blank

@crawlee/browser (BrowserCrawler)

Feature

Providing requestHandler is required but IMO it is redundant with what postNavigationHooks are there for. The semantics are IMO also rather confusing, as the function does not handle network requests but rather navigation.

Replace requestHandler(ctx) with new navigationHandler(ctx, gotoOptions) and fix confusing request-handling semantic when navigation-handling is meant.

Thoughts? What can you do in a user defined requestHandler that you cannot do in user defined post navigation hooks?

Motivation

Fix #1631.
Reduce confusion.

Ideal solution or implementation, and any additional constraints

not sure, needs feedback

Alternative solutions or implementations

not sure, needs feedback

Other context

No response

[B4nan]

Thanks for the feedback, but requestHandler is an integral part of the crawler design and its life cycle, we are not interested in changing this, nor in renaming it (maybe we can reiterate on that in a year or so, but a new major version is out of scope for discussions at this point). If you don't need it, just pass in an empty callback like () => {}.

Providing requestHandler is required but IMO it is redundant with what postNavigationHooks are there for.

All hooks are optional, so requestHandler can be redundant only if you actually use them. Changing this (making some hooks required) would be a huge breaking change and only cause more confusion. Removing the concept of requestHandler would be equally huge breaking change, for no clear reason.

[matjaeck]

Hi, alright, thanks for the response.

All hooks are optional, so requestHandler can be redundant only if you actually use them.

This is not really a counter argument to that nothing stops you from moving all crawling logic from post hooks to requestHandler, i.e. they are used in the same way. This equals redundancy in my book. But I get your point about breaking changes, hence the request for feedback.

Sidenote: I liked the original "handlePageFuntion" thingy a lot more, as it does what the name says. Handling requests is not what requestHandler does, as it handles a crawling context, in most use cases after requests have been loaded and handled. I understand that in your view "request" equals "navigation request" (same with request queue). It just a little bit a preoccupied term :)

Removing the concept of requestHandler would be equally huge breaking change, for no clear reason.

Totally understandable that you don't want to introduce breaking changes in a hurry.
If you like redundant features and don't care about misleading naming, then I agree there's no reason for a change :)

[DavidDuwaer]

running into this too. maybe i haven't found the right documentation, but to me it seems there are two competing APIs inside my PlaywrightCrawler: pre/post navigation hooks and requestHandler.

if it's about backward compatibility, why can't Apify just mark the hooks as deprecated in jsdoc and push new users towards equivalent use in requestHandler in that same jsdoc?

[DavidDuwaer]

No one cares if they stay around for decades while being marked deprecated, but at least new users will see a single coherent target API.

[janbuchar]

The intended usage of the API is that you should use the requestHandler for your crawling/scraping logic - reading the target website, working with storages, handling new links... The postNavigationHooks are meant mostly for adjusting/extending the navigation logic - logging, checking status codes or blocking, etc.

It is true that the terminology around this has some redundancy (i.e., Request can be multiple things), but I believe that *Hooks are understood as "optional functionality meant for customization" for most people.