Release 0.11.0 - #162
Merged
Merged
Conversation
Added new methods of parallel upload to bedbase and qdrant
Contributor
There was a problem hiding this comment.
Pull request overview
Release 0.11.0 focuses on HPC-friendly workflows (chunking + SLURM orchestration) for both PEP processing and Qdrant reindexing, along with UMAP output format changes and a more robust Refgenie chrom-sizes fallback.
Changes:
- Added HPC/SLURM orchestration commands for
run-pepand Qdrant reindexing (vectorize → upload → status). - Refactored UMAP outputs to write a single parquet (and optional JSON) with updated column set.
- Bumped project version to 0.11.0 and updated changelog; improved Refgenie seqcol download fallback.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/encode_processing/encode_into_parts.py | Adds a utility script to split a CSV into parts (currently hard-coded and fragile). |
| pyproject.toml | Bumps package version to 0.11.0. |
| docs/changelog.md | Adds 0.11.0 changelog entry. |
| bedboss/scripts/make_umap.py | Refactors UMAP export logic (single parquet + optional JSON) and CLI-facing output options. |
| bedboss/refgenome_validator/refgenie_chrom_sizes.py | Adds URL-based fallback for seqcol JSON when Refgenie fetch fails. |
| bedboss/qdrant_index/vectorize.py | Adds connection-free (DB-free) vectorization workers for HPC jobs. |
| bedboss/qdrant_index/upload.py | Adds uploader that reads parquet vectors and upserts to Qdrant + updates DB flags. |
| bedboss/qdrant_index/qdrant_index.py | Defers BedBaseAgent import to reduce import-time side effects. |
| bedboss/qdrant_index/qdrant_hpc.py | Adds SLURM chunking/orchestration + status reporting for Qdrant reindex jobs. |
| bedboss/qdrant_index/qdrant_cli.py | Introduces bedboss qdrant ... Typer subcommands for HPC reindex workflows. |
| bedboss/qdrant_index/init.py | Switches to lazy export of add_to_qdrant via __getattr__. |
| bedboss/const.py | Replaces tiered parquet column constants with a single UMAP_PARQUET_COLUMNS list. |
| bedboss/cli.py | Registers new qdrant subapp and adds run-pep-hpc / status commands; updates UMAP CLI options. |
| bedboss/bedclassifier/bedclassifier.py | Normalizes column indices after dropping NaN columns to prevent indexing errors. |
| bedboss/bedboss_hpc.py | Adds SLURM orchestration implementation for bedboss run-pep chunking/resume. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+20
to
+23
| if __name__ == "__main__": | ||
| divide_csv( | ||
| "/home/bnt4me/virginia/repos/bedboss/scripts/encode_processing/encode_bed_hg38.csv" | ||
| ) |
There was a problem hiding this comment.
This script is currently committed with a hard-coded local absolute path in __main__, which will fail for other users and in CI. Please make the input path a CLI argument (argparse/typer) or remove the __main__ block from the repo version.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes:
TODO:
__version__.pyfile