Skip to content

README.md

Latest commit

 

History

History
157 lines (104 loc) · 4.78 KB

README.md

File metadata and controls

157 lines (104 loc) · 4.78 KB

DIVE Python Packages

There are several important python packages in this application

  • dive_server is a collection of girder plugins for the web server
  • dive_tasks is a collection of girder worker plugins for the celery worker
  • scripts has general command-line utilities
  • dive_utils is shared code between the above packages

Prerequisites

Set up your system as described in the Basic Deployment

Development

In development, the server and client are run in separate processes. In production, the client is built and bundled as static files into the server image.

This python project uses uv for Python dependency management.

# Optional, for intellisense or whatever.  Not required for docker-compose
uv sync

Running in development with docker

# Copy .env.default and make any changes
cp .env.default .env

# Option 1) Build the project from source
docker-compose build
# Option 2) Pull pre-build images
docker-compose pull

# Start the project in default mode (GPU-enabled workers)
docker-compose up -d

# Start the project in CPU-only mode
docker-compose --profile cpu up -d

# The web server has hot reload, so code changes will
# immediately trigger a server restart.

# The Celery workers do not have hot reload.
# To test code changes, a restart is needed
docker-compose up girder_worker_default
# or
docker-compose up girder_worker_pipelines
# or
docker-compose up girder_worker_training

Access the server at http://localhost:8010

Worker profiles and feature availability

Docker Compose now separates workers by profile:

  • Default mode: runs girder_worker_default plus girder_worker_pipelines and girder_worker_training.
  • --profile cpu: runs girder_worker_default only.

If GPU workers are not present, DIVE disables pipeline/training actions in the web interface and rejects related API job launch requests.

To work on the Vue client, see development instructions in ../client.

Unit Testing and Static Checks

Automation is done with Tox with tox-uv.

# run only lint checks
uv run tox -e lint

# run only type checks
uv run tox -e type

# run only unit tests
uv run tox -e testunit

# run only a particular test
uv run tox -e testunit -- -k test_image_sort

# run all three tests above
uv run tox

# automatically format all code to comply to linting checks
uv run tox -e format

# run mkdocs and serve the documentation page
uv run tox -e docs

# creates docs in the /site folder for eventual deployment
uv run tox -e builddocs

Integration Testing

Get an API key from production Girder. DO NOT use a full-scoped token, use a read only token.

# start the server
docker-compose up -d

# set an API key from production girder
export GIRDER_API_KEY=CHANGEME

# run the tests
uv run tox -e testintegration

# run only a particular test (be mindful of dependencies)
uv run tox -e testintegrationkeyword test_pipelines

After integration tests are complete, visually inspect the results to make sure all jobs completed, new datasets open correctly, etc.

Debug utils and command line tools

# Requires a local uv installation
uv sync

# show options
uv run dive --help

# build the standalone executable into ./dist
uv run tox -e buildcli

Metadata properties

This section explains the metadata properties used to record application state in Girder. These properties can be modified through the Girder UI editor.

Dataset

Image chips that compose a video are stored as girder items in a folder. Videos are stored as a single item in its own folder. The parent folder must have the following metadata.

  • annotate (boolean) marks a folder as a valid DIVE dataset
  • type ('video' | 'image-sequence') dataset type
  • fps (number) annotation framerate, not to be confused with video raw framerate
  • ffprobe_info (JSON) output of ffprobe for raw input video
  • confidenceFilters (JSON) map of filter name to float in [0, 1]
  • imageEnhancements (JSON) values for image enhancements (brightness, contrast, saturation, sharpen)
  • customTypeStyline (JSON) map of class name to GeoJS display attributes.
  • foreign_media_id (string) For "cloned" datasets, this is an objectId pointer to the source media

Video Item

  • codec (string) video codec
  • source_video (boolean) whether the video is a raw user upload or a trancoded video

uv Lockfile Refresh

The girder-large-image plugin dependencies are hosted externally and can occasionally shift beneath pinned resolution. If a build fails because of stale lock metadata, clear uv/pip caches and regenerate the lockfile from the server folder:

uv cache clean && pip cache purge && rm -rf ~/.cache/uv ~/.cache/pip && rm -f uv.lock && uv lock