Named or Alias'ed Request_queue doesn't get used when using storage client

[jaemingo-hhh]

If I have the following code

configuration = service_locator.get_configuration()
configuration.purge_on_start = True
configuration.storage_dir = "/crawlee/storage"


storage_client: FileSystemStorageClient = FileSystemStorageClient()
dataset_client = await storage_client.create_dataset_client(alias="test-ds")
await storage_client.create_rq_client(alias="test-rq")
service_locator.set_storage_client(storage_client)

PlaywrightCrawler(
    storage_client=storage_client,
)

It creates both the alias dataset and the alias request queue in the /crawlee/storage. The test-ds subfolder is populated. However, the request queue is being appended in the default subfolder and not the test-rq alias subfolder.

Upon closer look, the get_request_manager doesn't seem to open up without a name or alias, which makes me think that it is intentional? I would expect that if I use the create_rq_client with an alias or a name, then the crawler will try to grab use it instead of always defaulting to the unnamed default storage.

[jaemingo-hhh]

As a workaround, I added the following stanza and included it in the PlaywrightCrawler

request_queue = await RequestQueue.open(
    alias="test-rq",
    storage_client=service_locator.get_storage_client(),
    configuration=service_locator.get_configuration(),
)


PlaywrightCrawler(
    storage_client=storage_client,
    request_manager=request_queue
)

[Mantisus]

Hi @jaemingo-hhh

Thank you for your interest in the project.

Use classes from crawlee.storages with the storage_client parameter.

storage_client = FileSystemStorageClient()

dataset = await Dataset.open(alias='my_dataset', storage_client=storage_client)
rq = await RequestQueue.open(alias='my_request_queue', storage_client=storage_client)

Methods such as create_dataset_client are not intended to be called directly. StorageClients are essentially backends, although the name may be somewhat misleading. See #1592.

[jaemingo-hhh]

I see. Thank you, that makes more sense.