Python: Missing methods on async_tiff.store classes

[maxrjones]

I'd like to be able to cache a file in a MemoryStore for use with async_tiff, but am blocked by the lack of public get methods on the async_tiff store classes. Would it be possible to expose the get and put methods?

For example:

from urllib.parse import urlparse
import async_tiff as atiff
import asyncio
import obstore

async def open_tiff(*, urlpath: str, store: atiff.store.MemoryStore) -> atiff.TIFF:
    return await atiff.TIFF.open(urlpath, store=store)

url = "https://github.com/mdsumner/rema-ovr/raw/refs/heads/main/rema_mosaic_1km_v2.0_filled_cop30_dem.tif"
parsed = urlparse(url)
urlpath = parsed.path

# Cache in memory using async_tiff
https_store = atiff.store.HTTPStore.from_url(f"{parsed.scheme}://{parsed.netloc}")
memory_store = atiff.store.MemoryStore()
print(dir(memory_store))
print(dir(obstore.store.MemoryStore()))
buffer = https_store.get(urlpath).bytes()
memory_store.put(urlpath, buffer)
asyncio.run(open_tiff(urlpath, memory_store))

['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__']
['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__firstlineno__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__static_attributes__', '__str__', '__subclasshook__', '__weakref__', 'copy', 'copy_async', 'delete', 'delete_async', 'get', 'get_async', 'get_range', 'get_range_async', 'get_ranges', 'get_ranges_async', 'head', 'head_async', 'list', 'list_with_delimiter', 'list_with_delimiter_async', 'put', 'put_async', 'rename', 'rename_async']

    buffer = https_store.get(urlpath).bytes()
             ^^^^^^^^^^^^^^^
AttributeError: 'builtins.HTTPStore' object has no attribute 'get'

[kylebarron]

Would it be possible to expose the get and put methods?

This is the intentional difference between obstore and pyo3-object-store (and its derived projects like async-tiff). obstore is intended to present a Python API for Python users to make requests, while pyo3-object_store is intended so that Python users can construct the store objects for Rust code to be making requests.

If we wanted to expose get and put here, that would require re-implementing all of obstore into async-tiff or drastically increasing the scope of pyo3-object-store. Neither of which I'd like to do.

The best way forward would be to document how to exchange store objects that are based on pyo3-object_store. E.g. you should be able to do

obstore_store = obstore.store.S3Store(...)
args, kwargs = obstore_store. __getnewargs_ex__()
async_tiff_store = async_tiff.store.S3Store(*args, **kwargs)

this is what happens under the hood when you pass an obstore store object into an async tiff API.

[maxrjones]

This approach doesn't work for caching in a memory store:

from urllib.parse import urlparse
import async_tiff as atiff
import obstore

url = "https://github.com/mdsumner/rema-ovr/raw/refs/heads/main/rema_mosaic_1km_v2.0_filled_cop30_dem.tif"
parsed = urlparse(url)
urlpath = parsed.path

parsed = urlparse(url)
urlpath = parsed.path

# Cache in memory using obstore
http_store = obstore.store.HTTPStore.from_url(f"{parsed.scheme}://{parsed.netloc}")
memory_store = obstore.store.MemoryStore()
buffer = http_store.get(urlpath).bytes()
memory_store.put(urlpath, buffer)
# Create an async_tiff store
args, kwargs = memory_store. __getnewargs_ex__()
async_tiff_memory_store = atiff.store.MemoryStore(*args, **kwargs)

AttributeError: 'MemoryStore' object has no attribute '__getnewargs_ex__'

[maxrjones]

An alternative could be to add support for prefetch=True to prefetch the entire file.

[kylebarron]

This approach doesn't work for caching in a memory store:

True, but I'm not sure there's really any alternative there. These are two different Rust libraries each with their own idea of a MemoryStore in a way that can't interact natively.

One other option is to add support for external obspec objects in pyo3-object_store. This would mean that every request would go through python, but that you could use an obstore memorystore as the backing for the requests of async-tiff.

[maxrjones]

One other option is to add support for external obspec objects in pyo3-object_store. This would mean that every request would go through python, but that you could use an obstore memorystore as the backing for the requests of async-tiff.

This seems much nicer from a user perspective, since it's awkward to switch back and forward between obstore stoes and async-tiff stores. Is there a performance downside?

[kylebarron]

Going through Python would mean that every request is limited by the GIL, though they could still be async.

[maxrjones]

An alternative could be to add support for prefetch=True to prefetch the entire file.

What do you think about this alternative? It would also solve my issue and allow getting around #89 without needing to guess the prefetch amount.

[weiji14]

Going through Python would mean that every request is limited by the GIL, though they could still be async.

Not if we use the Python 3.13 free-threaded/nogil builds - https://pyo3.rs/v0.25.1/free-threading.html?highlight=gil_used#supporting-free-threaded-python-with-pyo3. I think we'll eventually want to declare free-threaded support in async-tiff & pyo3-object_store, though it'll require a fair amount of work to set up CI builds, tests and all.

[kylebarron]

Even free-threaded builds will have some amount of overhead of going through Python. You can look at kylebarron/arro3#275 for progress on supporting 3.13t

[kylebarron]

I think this can be closed with the fix in #140, where we have a much smarter implementation than just a prefetch.

I don't think it makes sense for separation of concerns for async-tiff store classes to ever expose the obstore API. Perhaps we could allow obstore's functional API to work on obstore-like inputs, so you could call obstore.get(store) where store was created with async-tiff. If that sounds valuable, you can make an issue in obstore about that.