Welcome to the Data Commons Platform (DCP) developer guide. This document serves as the day-to-day workbench manual for engineers developing, testing, and debugging code within the datacommons monorepo.
- Looking for the PR process and contribution checklist? Refer to CONTRIBUTING.md.
- New to DCP and want to deploy a test instance on GCP? Follow the Developer Onboarding Codelab.
- Looking for platform architecture and data flows? Read Platform Architecture.
This repository is structured as a Python monorepo managed by uv, alongside Terraform infrastructure configurations and integration test suites.
datacommons/
├── packages/ # Python monorepo packages managed via uv
│ ├── datacommons-cli/ # User-facing CLI entrypoint (datacommons)
│ ├── datacommons-admin/ # Administrative logic, scaffolding, and cloud clients
│ ├── datacommons-db/ # Database models, Spanner client (DCGraph), migrations
│ ├── datacommons-schema/ # Pydantic models for JSON-LD and MCF parsing
│ └── datacommons-api/ # API service layer components
│
├── infra/
│ └── dcp/ # Terraform modules for provisioning GCP infrastructure
│ ├── main.tf # Root Terraform configuration
│ ├── variables.tf # Root variable definitions
│ ├── terraform.tfvars.template # Configuration template for deployments
│ ├── scripts/ # Test scripts (check_terraform.py)
│ └── modules/ # Submodules (spanner, storage, auth, ingestion, etc.)
│
├── docs/ # Centralized platform documentation
│ ├── README.md # Documentation architecture and standards
│ ├── developer_guide.md # This document (workbench manual)
│ ├── user_guide.md # Master operational manual for DCP Admins
│ ├── codelabs/ # Hands-on interactive tutorials
│ └── architecture/ # Technical deep dives and specifications
│
├── tests/ # Integration tests and automated cloud probers
│ └── integration/ # Hermetic Docker Compose test suite and GCP testbed
│
└── experimental/ # Experimental tools and artifact build scripts
| Package Path | Package Name | Responsibility |
|---|---|---|
packages/datacommons-cli |
datacommons-cli |
Thin distribution wrapper that exposes the datacommons console script and routes subcommands. Published to PyPI. |
packages/datacommons-admin |
datacommons-admin |
Core administrative logic: template downloading for admin init, Spanner migration triggers for admin init-db, and Cloud Workflows API integration for admin ingest start. Published to PyPI. |
packages/datacommons-db |
datacommons-db |
Database layer containing SQLAlchemy models, the DCGraph Spanner client, and versioned migration DDL scripts in migration_scripts/. Published to PyPI. |
packages/datacommons-schema |
datacommons-schema |
Graph schema data models and format converters between MCF and compact JSON-LD (unpublished prototype). |
packages/datacommons-api |
datacommons-api |
Internal API endpoints and service interfaces (unpublished prototype). |
Note
Unpublished Prototype Packages: packages/datacommons-schema and packages/datacommons-api are not published to PyPI. They represent initial architecture explorations that the team pivoted away from, preserved in the workspace for potential future reuse. Only datacommons-cli, datacommons-admin, and datacommons-db are actively built, versioned, and published.
The repository root defines a unified workspace in pyproject.toml:
[tool.uv.workspace]
members = [
"packages/*",
"tools",
]When you run commands using uv, all member packages are installed into a single shared virtual environment located at .venv/ in editable mode. Any changes you make to packages/datacommons-admin/ or packages/datacommons-db/ take effect immediately without requiring reinstallation.
Always scope dependency additions to the specific package:
# Add a runtime dependency to datacommons-admin
uv add --package datacommons-admin <dependency-name>
# Add a development dependency to the root workspace
uv add --dev <dependency-name>To run unreleased CLI code directly from your working tree without installing the package globally, use uv run --package datacommons-cli:
# Run admin init using local code
uv run --package datacommons-cli datacommons admin init
# Run database setup against an existing deployment
uv run --package datacommons-cli datacommons admin init-db
# Trigger data ingestion using local logic
uv run --package datacommons-cli datacommons admin ingest start --imports <dataset_name>- Define the Click command in
packages/datacommons-admin/datacommons_admin/<group>/<group>_cli.py. - Register the command on the group in
packages/datacommons-admin/datacommons_admin/admin_cli.py. - If the command reads Terraform attributes, pass the root state flags (
project_id,instance_name,tf_state_location) intoget_terraform_outputs()inpackages/datacommons-admin/datacommons_admin/core/terraform/state.pyto load the validatedTerraformOutputsdataclass defined inpackages/datacommons-admin/datacommons_admin/core/terraform/models.py. This resolves outputs seamlessly from either local state (terraform output -json) or remote GCS backend state (--project-idand--instance-name, or--tf-state-location). - If adding new required Terraform outputs, export them in
infra/dcp/modules/stack/outputs.tf(if delegated) andinfra/dcp/outputs.tf, add the typed field toTerraformOutputsincore/terraform/models.py, and update test fixtures inpackages/datacommons-admin/tests/conftest.py. Run the core unit and contract test suite to verify synchronization:uv run pytest packages/datacommons-admin/tests/core/
- Entity Models: Graph models (
NodeRecord,EdgeRecord,ObservationRecord) reside inpackages/datacommons-db/datacommons_db/models/. - Schema Migrations: Schema alterations are managed as versioned Python migration scripts in
packages/datacommons-db/datacommons_db/migrations/migration_scripts/.- For instructions on authoring, naming, and testing migrations, consult the Schema Migrations Developer Guide.
- Validation, Unit Testing, and Module Reference: For
check_terraform.pyusage,.tftest.hclunit test authoring, and rootvariables.tf/outputs.tfreferences, consult the Infrastructure Guide. - Stack Architecture: For the dual entrypoint design, variable propagation pipeline, and cross-module IAM wiring, read Terraform Stack Architecture.
When testing changes to infra/dcp/modules/ inside a personal deployment directory created by admin init without pushing commits to a remote Git branch:
- Option A (Direct Local Path): In your deployment's
main.tf, replace the remote Git reference with your local monorepo path:module "stack" { source = "/absolute/path/to/datacommons/infra/dcp/modules/stack"
- Option B (Local Symlink): Create a symlink inside your deployment folder pointing to the local
modulesdirectory:Then pointln -s /absolute/path/to/datacommons/infra/dcp/modules ./modules
main.tfto the local symlink:module "stack" { source = "./modules/stack"
- Re-Initialize and Plan: Run
terraform init -upgrade && terraform planin your deployment directory so Terraform reads your localinfra/dcp/modules/directly.
When testing cross-repository features or validating unreleased changes against active development branches, deploy your instance against head builds (main branch and :latest containers) instead of pinned releases.
Running on latest involves four platform layers:
-
Terraform Infrastructure Modules (
mainbranch): In~/dcp-deployments/<namespace>/main.tf, point the root stack module to themainbranch ofdatacommons(or use a local symlink toinfra/dcp/modules/stack):module "stack" { source = "git::https://github.com/datacommonsorg/datacommons.git//infra/dcp/modules/stack?ref=main" }
-
Platform Containers & Dataflow Flex Template (
dcp_version = "latest"): In~/dcp-deployments/<namespace>/terraform.tfvars, setdcp_versiontolatest:dcp_version = "latest"
Setting
dcp_version = "latest"activates runtime behaviors across container images:- Container Image Resolution: Pins all four Cloud Run services and jobs (
datacommons-services,datacommons-data,datacommons-aggregation-helper,datacommons-ingestion-helper) to the:latesttag in Container Registry (gcr.io/datcom-ci/...:latest) or Artifact Registry. - Cache-Busting Image Pulls (
FORCE_RESTARTandskip_container_restarts): Google Cloud Run resolves image tags to SHA-256 digests at deployment definition update time, not at request time. Ininfra/dcp/modules/stack/main.tf, Cloud Run services and jobs include an environment variableFORCE_RESTART = var.global.skip_container_restarts ? "" : timestamp(). Whenskip_container_restarts = false(the default),FORCE_RESTARTevaluates to the current timestamp on every run, forcing Cloud Run to create a new revision and pull the newest:latestimage digest. If yourterraform.tfvarssetsskip_container_restarts = true(as configured in dcp_developer_onboarding.md to prevent revision churn during shared development),FORCE_RESTARTremains empty across applies. When testing against:latest, setskip_container_restarts = falseso thatterraform applypulls fresh images. - Dataflow Flex Template: Directs the ingestion pipeline to the unpinned stable Beam template (
gs://datcom-templates/templates/flex/ingestion-stable.json).
- Container Image Resolution: Pins all four Cloud Run services and jobs (
-
Admin CLI on Latest: Run the CLI against the latest
mainbranch either on-the-fly viauvxor through a local repository clone:- Option A (
uvxpointing tomain): Execute the latest CLI directly from GitHub without installing packages globally. Becausedatacommons-clipinsdatacommons-adminto a released version, supply--withfor the sibling packages to ensureuvxruns all packages from the target Git ref instead of pulling released versions from PyPI:uvx --refresh \ --from "git+https://github.com/datacommonsorg/datacommons.git@main#subdirectory=packages/datacommons-cli" \ --with "git+https://github.com/datacommonsorg/datacommons.git@main#subdirectory=packages/datacommons-admin" \ --with "git+https://github.com/datacommonsorg/datacommons.git@main#subdirectory=packages/datacommons-db" \ --with "git+https://github.com/datacommonsorg/datacommons.git@main#subdirectory=packages/datacommons-schema" \ datacommons admin <command>
[!NOTE] If scaffolding a new deployment workspace with
admin initwhile running againstmain, pass--tf-git-ref main(or manually set?ref=maininmain.tf). By default,admin initpins the Terraform module source inmain.tfto the released version tag corresponding to the CLI version (for example,v1.x.x). - Option B (Local monorepo checkout):
Because the
uvworkspace links all member packages in editable mode by default, run the CLI directly usinguv:Any local edits made incd /path/fork/of/datacommonsorg/datacommons uv run --package datacommons-cli datacommons admin <command>
packages/datacommons-cliorpackages/datacommons-admintake effect immediately without manual reinstallation.
- Option A (
-
Applying Latest Updates: Pull updated module commits and apply the plan:
cd ~/dcp-deployments/<namespace> terraform init -upgrade terraform plan -out=tfplan terraform apply tfplan
DCP microservices and batch pipelines run in serverless Google Cloud Run containers and Cloud Dataflow Apache Beam workers. The images originate from multiple repositories across the Data Commons ecosystem:
- For high-level container topology, container roles, and end-to-end data flows, refer to Platform Architecture.
- For the automated release candidate tagging, promotion, and publishing pipelines, refer to the Release Guide.
- This section serves as the developer workbench guide for building custom development containers from source and overriding them in your deployment workspace.
Caution
Protected CI/CD Tags Rule: Never build, push, or overwrite tags that are reserved for CI/CD or platform automation, such as :latest, :stable, or version release tags (for example, release tags like v1.x.x or release candidates like 1.x.xrc1). Overwriting these tags corrupts automated integration tests, release candidate staging, and production deployments. Always use a descriptive, user-scoped tag for development builds (for example, <username>-<feature> or <username>-test-$(date +%s)).
| Component Name | Role | Dockerfile & Build Script | Destination Registry (Dev) | terraform.tfvars Override |
|---|---|---|---|---|
datacommons-services |
Envoy, Mixer API, Website serving | website:build/cdc_services/Dockerfilewebsite:scripts/push_cdc_services_image.sh |
gcr.io/<project_id>/datacommons-services:<tag> |
datacommons_services_image |
datacommons-data |
Preprocessor batch job | import:simple/Dockerfileimport:simple/scripts/push_image.sh |
gcr.io/<project_id>/datacommons-data:<tag> |
ingestion_preprocessing_job_image |
datacommons-aggregation-helper |
Postprocessor aggregation job | import:pipeline/workflow/aggregation-helper/Dockerfile |
gcr.io/<project_id>/datacommons-aggregation-helper:<tag> |
ingestion_postprocessing_job_image |
datacommons-ingestion-helper |
Lock coordination & migrations | import:pipeline/workflow/ingestion-helper/Dockerfile |
us-docker.pkg.dev/<project_id>/<repository>/datacommons-ingestion-helper:<tag> |
ingestion_helper_service_image |
ingestion-flex |
Apache Beam Dataflow pipeline | import:pipeline/ingestion/cloudbuild.yaml |
us-docker.pkg.dev/<project_id>/<repository>/dataflow-templates/ingestion:<tag>gs://<storage_artifacts_bucket_name>/templates/flex/ingestion-<tag>.json |
ingestion_dataflow_template_gcs_path |
Build custom container images and push them to Google Container Registry (GCR) or Artifact Registry:
The website repository incorporates mixer as a Git submodule. If your changes involve code inside Mixer, align the submodule before triggering the build:
cd /path/fork/of/datacommonsorg/website
# (Optional) Align submodules:
# Option A: Checkout specific feature branches:
cd mixer && git checkout <mixer_feature_branch> && cd ..
# Option B: Sync submodules with upstream master:
git submodule update --remote --merge mixer
# Build and push custom datacommons-services image to development project:
export PROJECT_ID="datcom-website-dev"
export SERVICES_TAG="<username>-<feature>-$(date +%s)"
./scripts/push_cdc_services_image.sh "$SERVICES_TAG" "$PROJECT_ID"
# Resulting Image URI:
# gcr.io/$PROJECT_ID/datacommons-services:<SERVICES_TAG>Note
scripts/push_cdc_services_image.sh invokes build/ci/cloudbuild.push_cdc_services_image.yaml, which resolves Git commit hashes for submodules and tags the container image. The second argument specifies the destination GCP project ($PROJECT_ID). If omitted, it defaults to the shared datcom-ci project (gcr.io/datcom-ci/datacommons-services:<SERVICES_TAG>). Images in both registries can be deployed to Cloud Run via datacommons_services_image in terraform.tfvars.
cd /path/fork/of/datacommonsorg/import
git checkout <feature_branch>
# Build and push custom datacommons-data image to development project:
export PROJECT_ID="datcom-website-dev"
export PREPROCESSOR_TAG="<username>-<feature>-$(date +%s)"
./simple/scripts/push_image.sh "$PREPROCESSOR_TAG" "$PROJECT_ID"
# Resulting Image URI:
# gcr.io/$PROJECT_ID/datacommons-data:<PREPROCESSOR_TAG>Note
simple/scripts/push_image.sh uploads only the simple/ directory from your working tree (including uncommitted changes) and builds it with simple/cloudbuild.yaml, using gcr.io/datcom-ci/datacommons-data:latest as the layer cache. The second argument specifies the destination GCP project ($PROJECT_ID); if omitted, it defaults to datcom-ci. To build and run the image locally instead, see Local Docker Build & Run.
cd /path/fork/of/datacommonsorg/import
cd pipeline/workflow/aggregation-helper
export PROJECT_ID="datcom-website-dev"
export POSTPROCESSOR_TAG="<username>-<feature>-$(date +%s)"
gcloud builds submit . \
--project="$PROJECT_ID" \
--tag="gcr.io/$PROJECT_ID/datacommons-aggregation-helper:$POSTPROCESSOR_TAG"
# Resulting Image URI:
# gcr.io/$PROJECT_ID/datacommons-aggregation-helper:<POSTPROCESSOR_TAG>cd /path/fork/of/datacommonsorg/import
cd pipeline/workflow/ingestion-helper
export PROJECT_ID="datcom-website-dev"
export INGESTION_HELPER_TAG="<username>-<feature>-$(date +%s)"
export INGESTION_HELPER_IMAGE="us-docker.pkg.dev/$PROJECT_ID/datacommons-artifacts/datacommons-ingestion-helper:$INGESTION_HELPER_TAG"
gcloud builds submit . \
--project="$PROJECT_ID" \
--tag="$INGESTION_HELPER_IMAGE"
# Resulting Image URI:
# us-docker.pkg.dev/$PROJECT_ID/datacommons-artifacts/datacommons-ingestion-helper:<INGESTION_HELPER_TAG>Dataflow executes as an Apache Beam Java Flex Template. Building the worker container image and staging the template JSON specification in Cloud Storage is handled through Cloud Build using pipeline/ingestion/cloudbuild.yaml:
cd /path/fork/of/datacommonsorg/import
export PROJECT_ID="datcom-website-dev"
export DATAFLOW_TAG="<username>-<feature>-$(date +%s)"
export TEMPLATE_BUCKET="<storage_artifacts_bucket_name>"
export IMAGE_GCR_PATH="us-docker.pkg.dev/$PROJECT_ID/datacommons-artifacts/dataflow-templates/ingestion"
# Build Dataflow worker image and stage Flex Template JSON specification in Cloud Storage:
gcloud builds submit . \
--config=pipeline/ingestion/cloudbuild.yaml \
--project="$PROJECT_ID" \
--substitutions="_VERSION=$DATAFLOW_TAG,_TEMPLATE_BUCKET=$TEMPLATE_BUCKET,_IMAGE_GCR_PATH=$IMAGE_GCR_PATH"
# Resulting Worker Image URI:
# us-docker.pkg.dev/$PROJECT_ID/datacommons-artifacts/dataflow-templates/ingestion:<DATAFLOW_TAG>
# Resulting Template GCS Path:
# gs://<storage_artifacts_bucket_name>/templates/flex/ingestion-<DATAFLOW_TAG>.jsonTo test custom container images or Dataflow templates on your deployed DCP instance, override the respective variables in ~/dcp-deployments/<namespace>/terraform.tfvars:
# Custom container image and Dataflow template overrides
datacommons_services_image = "gcr.io/<project_id>/datacommons-services:<custom_tag>"
ingestion_preprocessing_job_image = "gcr.io/<project_id>/datacommons-data:<custom_tag>"
ingestion_postprocessing_job_image = "gcr.io/<project_id>/datacommons-aggregation-helper:<custom_tag>"
ingestion_helper_service_image = "us-docker.pkg.dev/<project_id>/<repository>/ingestion-helper:<custom_tag>"
ingestion_dataflow_template_gcs_path = "gs://<storage_artifacts_bucket_name>/templates/flex/ingestion-<custom_tag>.json"Apply the updated configuration:
cd ~/dcp-deployments/<namespace>
terraform plan -out=tfplan
terraform apply tfplanTerraform updates the Cloud Run service, job, or Cloud Workflows definition to reference your custom image URI or template path and deploys a new revision without modifying persistent storage layers or Spanner databases.
When building custom container images in a development project (such as datcom-website-dev) and deploying them into a DCP instance running in another GCP project (such as datcom-dcp testbed environments), the target project's Cloud Run Service Agent must have read access to the source Artifact Registry or Container Registry:
# Retrieve target project number:
export TARGET_PROJECT_NUM=$(gcloud projects describe <TARGET_PROJECT_ID> --format="value(projectNumber)")
# Option A: Grant Artifact Registry Reader to target Cloud Run Service Agent:
gcloud artifacts repositories add-iam-policy-binding <REPOSITORY_NAME> \
--location=<repository_location> \
--project=<SOURCE_PROJECT_ID> \
--member="serviceAccount:service-${TARGET_PROJECT_NUM}@serverless-robot-prod.iam.gserviceaccount.com" \
--role="roles/artifactregistry.reader"
# Option B: Grant Cloud Storage Object Viewer for Google Container Registry (GCR):
gcloud storage buckets add-iam-policy-binding "gs://artifacts.<SOURCE_PROJECT_ID>.appspot.com" \
--member="serviceAccount:service-${TARGET_PROJECT_NUM}@serverless-robot-prod.iam.gserviceaccount.com" \
--role="roles/storage.objectViewer"When executing datacommons admin ingest start directly from a local workstation against a deployed instance, the CLI triggers Google Cloud Workflows using OAuth token impersonation. Before running ingestion commands against an instance, grant your user account the roles/iam.serviceAccountTokenCreator role on the provisioned Ingestion Workflow Service Account:
cd ~/dcp-deployments/<namespace>
export MY_USER="$(gcloud config get-value account)"
export PROJECT_ID="$(terraform output -raw project_id)"
export ORCHESTRATOR_SA="$(terraform output -raw ingestion_workflow_service_account_email)"
gcloud iam service-accounts add-iam-policy-binding "$ORCHESTRATOR_SA" \
--member="user:$MY_USER" \
--role="roles/iam.serviceAccountTokenCreator" \
--project="$PROJECT_ID"Important
Without this IAM binding, the CLI cannot generate OAuth tokens to authenticate with the Cloud Workflows API or Ingestion Helper service, resulting in HTTP 403 Forbidden errors.
When instances are deployed with datacommons_services_allow_unauthenticated_access = false (the secure default), you do not need to make services public or modify IAM policies to test HTTP endpoints. Establish an authenticated local proxy tunnel to the Cloud Run service:
cd ~/dcp-deployments/<namespace>
export PROJECT_ID="$(terraform output -raw project_id)"
export REGION="$(terraform output -raw region)"
export SERVICE_NAME="$(terraform output -raw datacommons_service_name)"
gcloud run services proxy "$SERVICE_NAME" \
--project="$PROJECT_ID" \
--region="$REGION" \
--port=8080This establishes an encrypted tunnel forwarding http://localhost:8080 to the private Cloud Run service, automatically attaching your gcloud credentials to every request. You can then query endpoints directly via cURL or your browser:
# Test the V2 Resolve endpoint through the proxy tunnel:
curl -s "http://localhost:8080/core/api/v2/resolve?nodes=california&resolver=place" | jq .
# Test natural language detection and fulfillment:
curl -s -X POST "http://localhost:8080/api/explore/detect-and-fulfill?q=population+in+california" \
-H "Content-Type: application/json" \
-d '{}' | jq .DCP organizes automated tests into two tiers:
- Unit Tests (Python and Terraform): Fast, offline tests that run automatically in pre-submit CI without live GCP credentials:
- Python Unit Tests: In-memory tests running via
pytestwhere external cloud APIs (Cloud Spanner, Cloud Workflows, Cloud Storage) and shell calls are mocked. Required for every bug fix, feature, and CLI subcommand. - Terraform Module Unit Tests: Hermetic module tests (
*.tftest.hcl) that runterraform planoffline with mocked providers (mock_provider "google").
- Python Unit Tests: In-memory tests running via
- Hermetic Integration Tests (End-to-End Validation): Local multi-service tests using Docker Compose to emulate Cloud Spanner, Cloud Storage, and serving containers offline without incurring GCP cloud costs. Integration tests validate end-to-end data ingestion, schema migrations, and query resolution before cutting releases or modifying cross-cutting data pipelines.
# Run all Python unit tests (or scope to a package, e.g. packages/datacommons-admin/tests/)
uv run pytest
# Run Terraform formatting, validation, and module unit tests
uv run infra/dcp/scripts/check_terraform.pySingle Source of Truth: For
check_terraform.pyflags (--fix,--fmt,--validate,--test) and instructions on authoring.tftest.hclunit tests, refer to the Infrastructure Guide.
The repository includes a hermetic testbed that uses Docker Compose to emulate Cloud Spanner, Cloud Storage, Ingestion Helper, and serving containers locally without incurring GCP cloud costs:
uv run pytest tests/integration/suites/ \
--instance emulated \
--test-config foobar_wagesSingle Source of Truth: For complete details on Docker Compose emulation, developer fast-iteration flags (such as
--reuse-datato skip re-ingestion), and running tests against live GCP sandbox projects, refer to the Integration Tests Guide.
CI measures test coverage on every pull request to main and every push to main, prints it in the build log, and uploads it to Codacy (report-only: coverage never fails the build).
To measure coverage locally:
uv run pytest --cov --cov-report=term --cov-report=xml
uv run diff-cover coverage.xml --compare-branch=origin/mainThis writes coverage.xml. Only code under packages/ is measured; Terraform and integration tests aren't included.
- Symptom: Integration tests or local serving queries fail with unauthorized or upstream RPC errors.
- Resolution: DCP federates queries against base Google Data Commons. Obtain a key from apikeys.datacommons.org and ensure
DC_API_KEYis exported in your shell:export DC_API_KEY="your-api-key"
- Symptom:
terraform applyfails with an error indicating that a BigQuery slot reservation already exists in the project and region. - Resolution: Google Cloud allows only one BigQuery slot reservation per project per region. When sharing a development project (such as
datcom-website-dev), setspanner_create_bigquery_reservation = falsein yourterraform.tfvars.
- Symptom: Hermetic integration tests report
Address already in useon emulator ports (such as port 9010 or 9020). - Resolution: Ensure no previous Docker Compose test containers are running:
docker compose -f tests/integration/emulated/docker-compose.yml down -v
- Symptom:
datacommons admin init-dbordatacommons admin ingest startfails with HTTP 403 / IAM permission denied when acquiring credentials. - Resolution: Ensure your GCP user account has
roles/iam.serviceAccountTokenCreatoron the workflow service account:gcloud iam service-accounts add-iam-policy-binding <workflow-sa-email> \ --member="user:$(gcloud config get-value account)" \ --role="roles/iam.serviceAccountTokenCreator" \ --project=<project-id>