Add Pyodide support and CI jobs for Zarr#1902
Add Pyodide support and CI jobs for Zarr#1902agriyakhetarpal wants to merge 97 commits intozarr-developers:v3from
Conversation
…rs#1586) Bumps [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) from 1.8.10 to 1.8.11. - [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases) - [Commits](pypa/gh-action-pypi-publish@v1.8.10...v1.8.11) --- updated-dependencies: - dependency-name: pypa/gh-action-pypi-publish dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…pers#1587) Bumps [conda-incubator/setup-miniconda](https://github.com/conda-incubator/setup-miniconda) from 2.3.0 to 3.0.1. - [Release notes](https://github.com/conda-incubator/setup-miniconda/releases) - [Changelog](https://github.com/conda-incubator/setup-miniconda/blob/main/CHANGELOG.md) - [Commits](conda-incubator/setup-miniconda@v2.3.0...v3.0.1) --- updated-dependencies: - dependency-name: conda-incubator/setup-miniconda dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Prevent runtime-overhead in doing this check multiple times
Even if this is only a test, list comprehensions are faster than repeatedly call append(). Also use tuple instead of list when possible. Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
Bumps [numcodecs](https://github.com/zarr-developers/numcodecs) from 0.11.0 to 0.12.1. - [Release notes](https://github.com/zarr-developers/numcodecs/releases) - [Changelog](https://github.com/zarr-developers/numcodecs/blob/main/docs/release.rst) - [Commits](zarr-developers/numcodecs@v0.11.0...v0.12.1) --- updated-dependencies: - dependency-name: numcodecs dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joe Hamman <joe@earthmover.io>
…ers#1558) Co-authored-by: Joe Hamman <joe@earthmover.io>
It's faster and probably more readable. Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> Co-authored-by: Joe Hamman <joe@earthmover.io>
Bumps [pymongo](https://github.com/mongodb/mongo-python-driver) from 4.5.0 to 4.6.1. - [Release notes](https://github.com/mongodb/mongo-python-driver/releases) - [Changelog](https://github.com/mongodb/mongo-python-driver/blob/master/doc/changelog.rst) - [Commits](mongodb/mongo-python-driver@4.5.0...4.6.1) --- updated-dependencies: - dependency-name: pymongo dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joe Hamman <joe@earthmover.io>
Starting with codespell 2.2.2, options can be specified in `pyrpoject.toml` in addition to `setup.cfg`: https://github.com/codespell-project/codespell#using-a-config-file Specifying options in a config file instead of command line options in `.pre-commit-config.yaml` ensures codespell uses the same options when run as pre-commit hook or from the command line in the repository root directory.
* Bump fsspec from 2023.10.0 to 2023.12.1 Bumps [fsspec](https://github.com/fsspec/filesystem_spec) from 2023.10.0 to 2023.12.1. - [Commits](fsspec/filesystem_spec@2023.10.0...2023.12.1) --- updated-dependencies: - dependency-name: fsspec dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Update s3fs as well * Fix s3fs --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Josh Moore <josh@openmicroscopy.org>
* Add type hints to zarr.create * Use protocol for MetaArray * Use protocol for Synchronizer * Fix Path typing * Add release note * Fix dim separator typing * Ignore ... in coverage reporting * Fix chunk typing --------- Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 4.7.1 to 5.0.0. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@v4.7.1...v5.0.0) --- updated-dependencies: - dependency-name: actions/setup-python dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joe Hamman <joe@earthmover.io>
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2 to 3. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@v2...v3) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* chore: update pre-commit hooks updates: - https://github.com/charliermarsh/ruff-pre-commit → https://github.com/astral-sh/ruff-pre-commit - [github.com/astral-sh/ruff-pre-commit: v0.0.224 → v0.1.8](astral-sh/ruff-pre-commit@v0.0.224...v0.1.8) - [github.com/psf/black: 23.10.1 → 23.12.0](psf/black@23.10.1...23.12.0) - [github.com/codespell-project/codespell: v2.2.5 → v2.2.6](codespell-project/codespell@v2.2.5...v2.2.6) - [github.com/pre-commit/pre-commit-hooks: v4.4.0 → v4.5.0](pre-commit/pre-commit-hooks@v4.4.0...v4.5.0) - [github.com/pre-commit/mirrors-mypy: v1.3.0 → v1.7.1](pre-commit/mirrors-mypy@v1.3.0...v1.7.1) * Attempt to fix ruff * Use isinstance --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Josh Moore <josh@openmicroscopy.org>
updates: - [github.com/astral-sh/ruff-pre-commit: v0.1.8 → v0.1.9](astral-sh/ruff-pre-commit@v0.1.8...v0.1.9) - [github.com/psf/black: 23.12.0 → 23.12.1](psf/black@23.12.0...23.12.1) - [github.com/pre-commit/mirrors-mypy: v1.7.1 → v1.8.0](pre-commit/mirrors-mypy@v1.7.1...v1.8.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Bump fsspec from 2023.12.1 to 2023.12.2 Bumps [fsspec](https://github.com/fsspec/filesystem_spec) from 2023.12.1 to 2023.12.2. - [Commits](fsspec/filesystem_spec@2023.12.1...2023.12.2) --- updated-dependencies: - dependency-name: fsspec dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Update requirements_dev_optional.txt --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joe Hamman <joe@earthmover.io>
Bumps [pytest-doctestplus](https://github.com/scientific-python/pytest-doctestplus) from 1.0.0 to 1.1.0. - [Release notes](https://github.com/scientific-python/pytest-doctestplus/releases) - [Changelog](https://github.com/scientific-python/pytest-doctestplus/blob/main/CHANGES.rst) - [Commits](scientific-python/pytest-doctestplus@v1.0.0...v1.1.0) --- updated-dependencies: - dependency-name: pytest-doctestplus dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
updates: - [github.com/astral-sh/ruff-pre-commit: v0.1.9 → v0.1.11](astral-sh/ruff-pre-commit@v0.1.9...v0.1.11) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Drop 3.8 and add 3.12 * Try removing line_profiler * Also bump the minimal numpy to 1.21 * Drop 3.12 again * Revert "Try removing line_profiler" This reverts commit 837854b. * Update release.rst --------- Co-authored-by: Joe Hamman <joe@earthmover.io> Co-authored-by: jakirkham <jakirkham@gmail.com>
updates: - [github.com/astral-sh/ruff-pre-commit: v0.1.11 → v0.1.13](astral-sh/ruff-pre-commit@v0.1.11...v0.1.13) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Bump actions/download-artifact from 3 to 4 Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 3 to 4. - [Release notes](https://github.com/actions/download-artifact/releases) - [Commits](actions/download-artifact@v3...v4) --- updated-dependencies: - dependency-name: actions/download-artifact dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Also bump upload-artifact see https://github.com/actions/download-artifact?tab=readme-ov-file#breaking-changes > Downloading artifacts that were created from action/upload-artifact@v3 and below are not supported. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joe Hamman <joe@earthmover.io> Co-authored-by: Josh Moore <josh@openmicroscopy.org>
zarr-developers#1615) * Update tutorial.rst to include section about accessing Zip Files on S3 Per discussion here, add information about about accessing zip files on s3: zarr-developers#1613 * Update release.rst * Implement d-v-b's suggestions --------- Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> Co-authored-by: Josh Moore <josh@openmicroscopy.org>
* doc(v3): add v3 roadmap and design document * Update v3-roadmap-and-design.md * updates after latest round of reviews * Update v3-roadmap-and-design.md Co-authored-by: Norman Rzepka <code@normanrz.com> * Update v3-roadmap-and-design.md Co-authored-by: Sanket Verma <svsanketverma5@gmail.com> --------- Co-authored-by: Norman Rzepka <code@normanrz.com> Co-authored-by: Sanket Verma <svsanketverma5@gmail.com>
updates: - [github.com/astral-sh/ruff-pre-commit: v0.1.13 → v0.1.14](astral-sh/ruff-pre-commit@v0.1.13...v0.1.14) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This patch removes fasteners and disables zarr.sync which uses process and thread Co-authored-by: Wei Ouyang <oeway007@gmail.com>
…pers#1824) Bumps [conda-incubator/setup-miniconda](https://github.com/conda-incubator/setup-miniconda) from 3.0.3 to 3.0.4. - [Release notes](https://github.com/conda-incubator/setup-miniconda/releases) - [Changelog](https://github.com/conda-incubator/setup-miniconda/blob/main/CHANGELOG.md) - [Commits](conda-incubator/setup-miniconda@v3.0.3...v3.0.4) --- updated-dependencies: - dependency-name: conda-incubator/setup-miniconda dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…#1807) * dep(docs): deprecate experimental v3 support in docs * Apply suggestions from code review Co-authored-by: Josh Moore <josh@openmicroscopy.org> Co-authored-by: Sanket Verma <svsanketverma5@gmail.com> --------- Co-authored-by: Josh Moore <josh@openmicroscopy.org> Co-authored-by: Sanket Verma <svsanketverma5@gmail.com>
Bumps [h5py](https://github.com/h5py/h5py) from 3.10.0 to 3.11.0. - [Release notes](https://github.com/h5py/h5py/releases) - [Changelog](https://github.com/h5py/h5py/blob/master/docs/release_guide.rst) - [Commits](h5py/h5py@3.10.0...3.11.0) --- updated-dependencies: - dependency-name: h5py dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joe Hamman <joe@earthmover.io>
Bumps [redis](https://github.com/redis/redis-py) from 5.0.3 to 5.0.4. - [Release notes](https://github.com/redis/redis-py/releases) - [Changelog](https://github.com/redis/redis-py/blob/master/CHANGES) - [Commits](redis/redis-py@v5.0.3...v5.0.4) --- updated-dependencies: - dependency-name: redis dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
… remove in v3 (zarr-developers#1801) * deprecate(stores): add deprecation warnings to DBMStore, LMDBStore, SQLiteStore, MongoDBStore, RedisStore, and ABSStore * filter warnings in pytest config * more deprecation warnings in docstrings * add release note
* use np.inf instead of PINF/NINF * update release notes
updates: - [github.com/astral-sh/ruff-pre-commit: v0.4.1 → v0.4.3](astral-sh/ruff-pre-commit@v0.4.1...v0.4.3) - [github.com/psf/black: 24.4.0 → 24.4.2](psf/black@24.4.0...24.4.2) - [github.com/pre-commit/mirrors-mypy: v1.9.0 → v1.10.0](pre-commit/mirrors-mypy@v1.9.0...v1.10.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Enable ruff/bugbear rules (B) As suggested by Repo-Review. * Fix ruff/bugbear issue (B007) B007 Loop control variable `key` not used within loop body https://docs.astral.sh/ruff/rules/unused-loop-control-variable/ * Fix ruff/bugbear issue (B015) B015 Pointless comparison. Did you mean to assign a value? Otherwise, prepend `assert` or remove it. https://docs.astral.sh/ruff/rules/useless-comparison/ * Fix ruff/bugbear issues (B028) B028 No explicit `stacklevel` keyword argument found https://docs.astral.sh/ruff/rules/no-explicit-stacklevel/ * Fix ruff/bugbear issues (B904) B904 Within an `except` clause, raise exceptions with `raise ... from err` or `raise ... from None` to distinguish them from errors in exception handling https://docs.astral.sh/ruff/rules/raise-without-from-inside-except/ * Document changes in docs/release.rst * Disable ruff/bugbear rule (B017) B017 `pytest.raises(Exception)` should be considered evil https://docs.astral.sh/ruff/rules/assert-raises-exception/ --------- Co-authored-by: Joe Hamman <joe@earthmover.io>
* doc: cleanup release notes for 2.18.0
updates: - [github.com/astral-sh/ruff-pre-commit: v0.4.3 → v0.4.4](astral-sh/ruff-pre-commit@v0.4.3...v0.4.4) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
There was a problem hiding this comment.
I added a couple of review comments below to ease the code review process for project maintainers and core developers. Besides that, there is another test case that fails and I have had a hard time debugging it so far. I have managed to get down to two failing tests across the entire test suite by skipping tests where applicable using @pytest.mark.skipif() decorators.
Additionally, I suppose that proper updates to the Zarr documentation and a CHANGELOG update would be required to inform people about the limitations of running Zarr in a Pyodide runtime and which features are not supported. I would be happy to make those changes as I am directed to; and edit the docstrings for the relevant sections in the public API, in subsequent commits and review comments as necessary.
zarr/tests/test_core.py
Outdated
| try: | ||
| from numcodecs.tests.common import greetings | ||
| except ModuleNotFoundError: | ||
| greetings = [ | ||
| '¡Hola mundo!', 'Hej Världen!', 'Servus Woid!', 'Hei maailma!', | ||
| 'Xin chào thế giới', 'Njatjeta Botë!', 'Γεια σου κόσμε!', | ||
| 'こんにちは世界', '世界,你好!', 'Helló, világ!', 'Zdravo svete!', | ||
| 'เฮลโลเวิลด์' | ||
| ] |
There was a problem hiding this comment.
I am unaware of the reason why numcodecs.tests.common cannot import this to the namespace – for this while, I have added this list of strings verbatim here. I will investigate the cause for the error.
zarr/tests/test_storage.py
Outdated
| ) | ||
| return store | ||
|
|
||
| @pytest.mark.xfail(reason="Skip under Pyodide for now, don't know how to handle umask") |
There was a problem hiding this comment.
This is one of the tests I would like to receive help on – I am not entirely aware of the differences in the Pyodide (Emscripten MemFS to be specific) and the CPython file systems, and I see that retrieving the 0o777 octal (execute permissions) seems to bring some insights. I have marked this as xfail for now.
pyproject.toml
Outdated
| 'numcodecs[msgpack]>=0.10.0; sys_platform == "emscripten"', # TODO: can be removed | ||
| 'msgpack; sys_platform == "emscripten"', |
There was a problem hiding this comment.
Both of these lines currently do not work for some reason – I am unable to import MsgPack from numcodecs even after I added them, which is why I am explicitly installing msgpack inside the Pyodide virtual environment for now, which seems to make this work. I will take a look at why these options do not work as I continue to work on this PR.
68b6f62 to
80fb6c2
Compare
|
@agriyakhetarpal -- thanks for this PR. I want to point you to #1777 and see if we can focus your efforts toward the |
|
Thank you for the redirection, @jhamman. I shall change the base branch of the PR, fix any conflicts, and rebase as necessary in a moment. |
|
Okay, wayyy too many conflicts! 😄 I think this will take me a while, but should be doable without repeating these changes as a separate PR. |
89aa103 to
bf89533
Compare
|
I cherry-picked and backed up my commits in a different branch and a stash, I'll recover them from there. I think a separate PR would be better, I'll copy what I documented over to the other one. |
|
I have a suggestion which I hope is not unsolicited: please make the Please ignore if this has been evaluated before or if there is a nuance to the discussion that I might have missed as a newcomer who has not contributed to Zarr before. Thanks! |
Important
Please view #1903 instead