Skip to content

Latest commit

 

History

History
485 lines (376 loc) · 29 KB

developer_guide.md

File metadata and controls

485 lines (376 loc) · 29 KB

Data Commons Platform Developer Guide

Overview

Welcome to the Data Commons Platform (DCP) developer guide. This document serves as the day-to-day workbench manual for engineers developing, testing, and debugging code within the datacommons monorepo.


Monorepo Topology and uv Workspace

This repository is structured as a Python monorepo managed by uv, alongside Terraform infrastructure configurations and integration test suites.

datacommons/
├── packages/                            # Python monorepo packages managed via uv
│   ├── datacommons-cli/                 # User-facing CLI entrypoint (datacommons)
│   ├── datacommons-admin/               # Administrative logic, scaffolding, and cloud clients
│   ├── datacommons-db/                  # Database models, Spanner client (DCGraph), migrations
│   ├── datacommons-schema/              # Pydantic models for JSON-LD and MCF parsing
│   └── datacommons-api/                 # API service layer components
│
├── infra/
│   └── dcp/                             # Terraform modules for provisioning GCP infrastructure
│       ├── main.tf                      # Root Terraform configuration
│       ├── variables.tf                 # Root variable definitions
│       ├── terraform.tfvars.template    # Configuration template for deployments
│       ├── scripts/                     # Test scripts (check_terraform.py)
│       └── modules/                     # Submodules (spanner, storage, auth, ingestion, etc.)
│
├── docs/                                # Centralized platform documentation
│   ├── README.md                        # Documentation architecture and standards
│   ├── developer_guide.md               # This document (workbench manual)
│   ├── user_guide.md                    # Master operational manual for DCP Admins
│   ├── codelabs/                        # Hands-on interactive tutorials
│   └── architecture/                    # Technical deep dives and specifications
│
├── tests/                               # Integration tests and automated cloud probers
│   └── integration/                     # Hermetic Docker Compose test suite and GCP testbed
│
└── experimental/                        # Experimental tools and artifact build scripts

Monorepo Packages Breakdown

Package Path Package Name Responsibility
packages/datacommons-cli datacommons-cli Thin distribution wrapper that exposes the datacommons console script and routes subcommands. Published to PyPI.
packages/datacommons-admin datacommons-admin Core administrative logic: template downloading for admin init, Spanner migration triggers for admin init-db, and Cloud Workflows API integration for admin ingest start. Published to PyPI.
packages/datacommons-db datacommons-db Database layer containing SQLAlchemy models, the DCGraph Spanner client, and versioned migration DDL scripts in migration_scripts/. Published to PyPI.
packages/datacommons-schema datacommons-schema Graph schema data models and format converters between MCF and compact JSON-LD (unpublished prototype).
packages/datacommons-api datacommons-api Internal API endpoints and service interfaces (unpublished prototype).

Note

Unpublished Prototype Packages: packages/datacommons-schema and packages/datacommons-api are not published to PyPI. They represent initial architecture explorations that the team pivoted away from, preserved in the workspace for potential future reuse. Only datacommons-cli, datacommons-admin, and datacommons-db are actively built, versioned, and published.

How uv Workspace Works

The repository root defines a unified workspace in pyproject.toml:

[tool.uv.workspace]
members = [
    "packages/*",
    "tools",
]

When you run commands using uv, all member packages are installed into a single shared virtual environment located at .venv/ in editable mode. Any changes you make to packages/datacommons-admin/ or packages/datacommons-db/ take effect immediately without requiring reinstallation.

Managing Package Dependencies

Always scope dependency additions to the specific package:

# Add a runtime dependency to datacommons-admin
uv add --package datacommons-admin <dependency-name>

# Add a development dependency to the root workspace
uv add --dev <dependency-name>

Local Development Recipes (The Workbench)

Working on the CLI (datacommons-cli and datacommons-admin)

1. Running Local CLI Code in Editable Mode

To run unreleased CLI code directly from your working tree without installing the package globally, use uv run --package datacommons-cli:

# Run admin init using local code
uv run --package datacommons-cli datacommons admin init

# Run database setup against an existing deployment
uv run --package datacommons-cli datacommons admin init-db

# Trigger data ingestion using local logic
uv run --package datacommons-cli datacommons admin ingest start --imports <dataset_name>

Adding a New CLI Command

  • Define the Click command in packages/datacommons-admin/datacommons_admin/<group>/<group>_cli.py.
  • Register the command on the group in packages/datacommons-admin/datacommons_admin/admin_cli.py.
  • If the command reads Terraform attributes, pass the root state flags (project_id, instance_name, tf_state_location) into get_terraform_outputs() in packages/datacommons-admin/datacommons_admin/core/terraform/state.py to load the validated TerraformOutputs dataclass defined in packages/datacommons-admin/datacommons_admin/core/terraform/models.py. This resolves outputs seamlessly from either local state (terraform output -json) or remote GCS backend state (--project-id and --instance-name, or --tf-state-location).
  • If adding new required Terraform outputs, export them in infra/dcp/modules/stack/outputs.tf (if delegated) and infra/dcp/outputs.tf, add the typed field to TerraformOutputs in core/terraform/models.py, and update test fixtures in packages/datacommons-admin/tests/conftest.py. Run the core unit and contract test suite to verify synchronization:
    uv run pytest packages/datacommons-admin/tests/core/

Working on the Database Layer (datacommons-db)

  • Entity Models: Graph models (NodeRecord, EdgeRecord, ObservationRecord) reside in packages/datacommons-db/datacommons_db/models/.
  • Schema Migrations: Schema alterations are managed as versioned Python migration scripts in packages/datacommons-db/datacommons_db/migrations/migration_scripts/.

Working on Infrastructure (infra/dcp)

  • Validation, Unit Testing, and Module Reference: For check_terraform.py usage, .tftest.hcl unit test authoring, and root variables.tf / outputs.tf references, consult the Infrastructure Guide.
  • Stack Architecture: For the dual entrypoint design, variable propagation pipeline, and cross-module IAM wiring, read Terraform Stack Architecture.

Testing Local Module Changes in a Scaffolded Workspace

When testing changes to infra/dcp/modules/ inside a personal deployment directory created by admin init without pushing commits to a remote Git branch:

  • Option A (Direct Local Path): In your deployment's main.tf, replace the remote Git reference with your local monorepo path:
    module "stack" {
      source = "/absolute/path/to/datacommons/infra/dcp/modules/stack"
  • Option B (Local Symlink): Create a symlink inside your deployment folder pointing to the local modules directory:
    ln -s /absolute/path/to/datacommons/infra/dcp/modules ./modules
    Then point main.tf to the local symlink:
    module "stack" {
      source = "./modules/stack"
  • Re-Initialize and Plan: Run terraform init -upgrade && terraform plan in your deployment directory so Terraform reads your local infra/dcp/modules/ directly.

Running the Stack on Latest (main and latest Builds)

When testing cross-repository features or validating unreleased changes against active development branches, deploy your instance against head builds (main branch and :latest containers) instead of pinned releases.

Running on latest involves four platform layers:

  • Terraform Infrastructure Modules (main branch): In ~/dcp-deployments/<namespace>/main.tf, point the root stack module to the main branch of datacommons (or use a local symlink to infra/dcp/modules/stack):

    module "stack" {
      source = "git::https://github.com/datacommonsorg/datacommons.git//infra/dcp/modules/stack?ref=main"
    }
  • Platform Containers & Dataflow Flex Template (dcp_version = "latest"): In ~/dcp-deployments/<namespace>/terraform.tfvars, set dcp_version to latest:

    dcp_version = "latest"

    Setting dcp_version = "latest" activates runtime behaviors across container images:

    • Container Image Resolution: Pins all four Cloud Run services and jobs (datacommons-services, datacommons-data, datacommons-aggregation-helper, datacommons-ingestion-helper) to the :latest tag in Container Registry (gcr.io/datcom-ci/...:latest) or Artifact Registry.
    • Cache-Busting Image Pulls (FORCE_RESTART and skip_container_restarts): Google Cloud Run resolves image tags to SHA-256 digests at deployment definition update time, not at request time. In infra/dcp/modules/stack/main.tf, Cloud Run services and jobs include an environment variable FORCE_RESTART = var.global.skip_container_restarts ? "" : timestamp(). When skip_container_restarts = false (the default), FORCE_RESTART evaluates to the current timestamp on every run, forcing Cloud Run to create a new revision and pull the newest :latest image digest. If your terraform.tfvars sets skip_container_restarts = true (as configured in dcp_developer_onboarding.md to prevent revision churn during shared development), FORCE_RESTART remains empty across applies. When testing against :latest, set skip_container_restarts = false so that terraform apply pulls fresh images.
    • Dataflow Flex Template: Directs the ingestion pipeline to the unpinned stable Beam template (gs://datcom-templates/templates/flex/ingestion-stable.json).
  • Admin CLI on Latest: Run the CLI against the latest main branch either on-the-fly via uvx or through a local repository clone:

    • Option A (uvx pointing to main): Execute the latest CLI directly from GitHub without installing packages globally. Because datacommons-cli pins datacommons-admin to a released version, supply --with for the sibling packages to ensure uvx runs all packages from the target Git ref instead of pulling released versions from PyPI:
      uvx --refresh \
        --from "git+https://github.com/datacommonsorg/datacommons.git@main#subdirectory=packages/datacommons-cli" \
        --with "git+https://github.com/datacommonsorg/datacommons.git@main#subdirectory=packages/datacommons-admin" \
        --with "git+https://github.com/datacommonsorg/datacommons.git@main#subdirectory=packages/datacommons-db" \
        --with "git+https://github.com/datacommonsorg/datacommons.git@main#subdirectory=packages/datacommons-schema" \
        datacommons admin <command>

      [!NOTE] If scaffolding a new deployment workspace with admin init while running against main, pass --tf-git-ref main (or manually set ?ref=main in main.tf). By default, admin init pins the Terraform module source in main.tf to the released version tag corresponding to the CLI version (for example, v1.x.x).

    • Option B (Local monorepo checkout): Because the uv workspace links all member packages in editable mode by default, run the CLI directly using uv:
      cd /path/fork/of/datacommonsorg/datacommons
      uv run --package datacommons-cli datacommons admin <command>
      Any local edits made in packages/datacommons-cli or packages/datacommons-admin take effect immediately without manual reinstallation.
  • Applying Latest Updates: Pull updated module commits and apply the plan:

    cd ~/dcp-deployments/<namespace>
    terraform init -upgrade
    terraform plan -out=tfplan
    terraform apply tfplan

Working with Container Images (Building and Overriding)

DCP microservices and batch pipelines run in serverless Google Cloud Run containers and Cloud Dataflow Apache Beam workers. The images originate from multiple repositories across the Data Commons ecosystem:

  • For high-level container topology, container roles, and end-to-end data flows, refer to Platform Architecture.
  • For the automated release candidate tagging, promotion, and publishing pipelines, refer to the Release Guide.
  • This section serves as the developer workbench guide for building custom development containers from source and overriding them in your deployment workspace.

Caution

Protected CI/CD Tags Rule: Never build, push, or overwrite tags that are reserved for CI/CD or platform automation, such as :latest, :stable, or version release tags (for example, release tags like v1.x.x or release candidates like 1.x.xrc1). Overwriting these tags corrupts automated integration tests, release candidate staging, and production deployments. Always use a descriptive, user-scoped tag for development builds (for example, <username>-<feature> or <username>-test-$(date +%s)).

Platform Container and Template Inventory

Component Name Role Dockerfile & Build Script Destination Registry (Dev) terraform.tfvars Override
datacommons-services Envoy, Mixer API, Website serving website:build/cdc_services/Dockerfile
website:scripts/push_cdc_services_image.sh
gcr.io/<project_id>/datacommons-services:<tag> datacommons_services_image
datacommons-data Preprocessor batch job import:simple/Dockerfile
import:simple/scripts/push_image.sh
gcr.io/<project_id>/datacommons-data:<tag> ingestion_preprocessing_job_image
datacommons-aggregation-helper Postprocessor aggregation job import:pipeline/workflow/aggregation-helper/Dockerfile gcr.io/<project_id>/datacommons-aggregation-helper:<tag> ingestion_postprocessing_job_image
datacommons-ingestion-helper Lock coordination & migrations import:pipeline/workflow/ingestion-helper/Dockerfile us-docker.pkg.dev/<project_id>/<repository>/datacommons-ingestion-helper:<tag> ingestion_helper_service_image
ingestion-flex Apache Beam Dataflow pipeline import:pipeline/ingestion/cloudbuild.yaml us-docker.pkg.dev/<project_id>/<repository>/dataflow-templates/ingestion:<tag>
gs://<storage_artifacts_bucket_name>/templates/flex/ingestion-<tag>.json
ingestion_dataflow_template_gcs_path

Building Images via Google Cloud Build

Build custom container images and push them to Google Container Registry (GCR) or Artifact Registry:

Serving Services (datacommons-services)

The website repository incorporates mixer as a Git submodule. If your changes involve code inside Mixer, align the submodule before triggering the build:

cd /path/fork/of/datacommonsorg/website

# (Optional) Align submodules:
# Option A: Checkout specific feature branches:
cd mixer && git checkout <mixer_feature_branch> && cd ..

# Option B: Sync submodules with upstream master:
git submodule update --remote --merge mixer

# Build and push custom datacommons-services image to development project:
export PROJECT_ID="datcom-website-dev"
export SERVICES_TAG="<username>-<feature>-$(date +%s)"
./scripts/push_cdc_services_image.sh "$SERVICES_TAG" "$PROJECT_ID"

# Resulting Image URI:
# gcr.io/$PROJECT_ID/datacommons-services:<SERVICES_TAG>

Note

scripts/push_cdc_services_image.sh invokes build/ci/cloudbuild.push_cdc_services_image.yaml, which resolves Git commit hashes for submodules and tags the container image. The second argument specifies the destination GCP project ($PROJECT_ID). If omitted, it defaults to the shared datcom-ci project (gcr.io/datcom-ci/datacommons-services:<SERVICES_TAG>). Images in both registries can be deployed to Cloud Run via datacommons_services_image in terraform.tfvars.

Preprocessor (datacommons-data)
cd /path/fork/of/datacommonsorg/import
git checkout <feature_branch>

# Build and push custom datacommons-data image to development project:
export PROJECT_ID="datcom-website-dev"
export PREPROCESSOR_TAG="<username>-<feature>-$(date +%s)"
./simple/scripts/push_image.sh "$PREPROCESSOR_TAG" "$PROJECT_ID"

# Resulting Image URI:
# gcr.io/$PROJECT_ID/datacommons-data:<PREPROCESSOR_TAG>

Note

simple/scripts/push_image.sh uploads only the simple/ directory from your working tree (including uncommitted changes) and builds it with simple/cloudbuild.yaml, using gcr.io/datcom-ci/datacommons-data:latest as the layer cache. The second argument specifies the destination GCP project ($PROJECT_ID); if omitted, it defaults to datcom-ci. To build and run the image locally instead, see Local Docker Build & Run.

Postprocessor (datacommons-aggregation-helper)
cd /path/fork/of/datacommonsorg/import
cd pipeline/workflow/aggregation-helper

export PROJECT_ID="datcom-website-dev"
export POSTPROCESSOR_TAG="<username>-<feature>-$(date +%s)"
gcloud builds submit . \
    --project="$PROJECT_ID" \
    --tag="gcr.io/$PROJECT_ID/datacommons-aggregation-helper:$POSTPROCESSOR_TAG"

# Resulting Image URI:
# gcr.io/$PROJECT_ID/datacommons-aggregation-helper:<POSTPROCESSOR_TAG>
Ingestion Helper Service (ingestion-helper)
cd /path/fork/of/datacommonsorg/import
cd pipeline/workflow/ingestion-helper

export PROJECT_ID="datcom-website-dev"
export INGESTION_HELPER_TAG="<username>-<feature>-$(date +%s)"
export INGESTION_HELPER_IMAGE="us-docker.pkg.dev/$PROJECT_ID/datacommons-artifacts/datacommons-ingestion-helper:$INGESTION_HELPER_TAG"
gcloud builds submit . \
    --project="$PROJECT_ID" \
    --tag="$INGESTION_HELPER_IMAGE"

# Resulting Image URI:
# us-docker.pkg.dev/$PROJECT_ID/datacommons-artifacts/datacommons-ingestion-helper:<INGESTION_HELPER_TAG>
Dataflow Flex Template and Ingestion Pipeline (ingestion-flex)

Dataflow executes as an Apache Beam Java Flex Template. Building the worker container image and staging the template JSON specification in Cloud Storage is handled through Cloud Build using pipeline/ingestion/cloudbuild.yaml:

cd /path/fork/of/datacommonsorg/import

export PROJECT_ID="datcom-website-dev"
export DATAFLOW_TAG="<username>-<feature>-$(date +%s)"
export TEMPLATE_BUCKET="<storage_artifacts_bucket_name>"
export IMAGE_GCR_PATH="us-docker.pkg.dev/$PROJECT_ID/datacommons-artifacts/dataflow-templates/ingestion"

# Build Dataflow worker image and stage Flex Template JSON specification in Cloud Storage:
gcloud builds submit . \
    --config=pipeline/ingestion/cloudbuild.yaml \
    --project="$PROJECT_ID" \
    --substitutions="_VERSION=$DATAFLOW_TAG,_TEMPLATE_BUCKET=$TEMPLATE_BUCKET,_IMAGE_GCR_PATH=$IMAGE_GCR_PATH"

# Resulting Worker Image URI:
# us-docker.pkg.dev/$PROJECT_ID/datacommons-artifacts/dataflow-templates/ingestion:<DATAFLOW_TAG>

# Resulting Template GCS Path:
# gs://<storage_artifacts_bucket_name>/templates/flex/ingestion-<DATAFLOW_TAG>.json

Overriding Images and Templates in terraform.tfvars

To test custom container images or Dataflow templates on your deployed DCP instance, override the respective variables in ~/dcp-deployments/<namespace>/terraform.tfvars:

# Custom container image and Dataflow template overrides
datacommons_services_image           = "gcr.io/<project_id>/datacommons-services:<custom_tag>"
ingestion_preprocessing_job_image    = "gcr.io/<project_id>/datacommons-data:<custom_tag>"
ingestion_postprocessing_job_image   = "gcr.io/<project_id>/datacommons-aggregation-helper:<custom_tag>"
ingestion_helper_service_image       = "us-docker.pkg.dev/<project_id>/<repository>/ingestion-helper:<custom_tag>"
ingestion_dataflow_template_gcs_path = "gs://<storage_artifacts_bucket_name>/templates/flex/ingestion-<custom_tag>.json"

Apply the updated configuration:

cd ~/dcp-deployments/<namespace>
terraform plan -out=tfplan
terraform apply tfplan

Terraform updates the Cloud Run service, job, or Cloud Workflows definition to reference your custom image URI or template path and deploys a new revision without modifying persistent storage layers or Spanner databases.

Granting Cross-Project Image Pull Permissions

When building custom container images in a development project (such as datcom-website-dev) and deploying them into a DCP instance running in another GCP project (such as datcom-dcp testbed environments), the target project's Cloud Run Service Agent must have read access to the source Artifact Registry or Container Registry:

# Retrieve target project number:
export TARGET_PROJECT_NUM=$(gcloud projects describe <TARGET_PROJECT_ID> --format="value(projectNumber)")

# Option A: Grant Artifact Registry Reader to target Cloud Run Service Agent:
gcloud artifacts repositories add-iam-policy-binding <REPOSITORY_NAME> \
    --location=<repository_location> \
    --project=<SOURCE_PROJECT_ID> \
    --member="serviceAccount:service-${TARGET_PROJECT_NUM}@serverless-robot-prod.iam.gserviceaccount.com" \
    --role="roles/artifactregistry.reader"

# Option B: Grant Cloud Storage Object Viewer for Google Container Registry (GCR):
gcloud storage buckets add-iam-policy-binding "gs://artifacts.<SOURCE_PROJECT_ID>.appspot.com" \
    --member="serviceAccount:service-${TARGET_PROJECT_NUM}@serverless-robot-prod.iam.gserviceaccount.com" \
    --role="roles/storage.objectViewer"

Local Workstation IAM Impersonation (Ingestion)

When executing datacommons admin ingest start directly from a local workstation against a deployed instance, the CLI triggers Google Cloud Workflows using OAuth token impersonation. Before running ingestion commands against an instance, grant your user account the roles/iam.serviceAccountTokenCreator role on the provisioned Ingestion Workflow Service Account:

cd ~/dcp-deployments/<namespace>

export MY_USER="$(gcloud config get-value account)"
export PROJECT_ID="$(terraform output -raw project_id)"
export ORCHESTRATOR_SA="$(terraform output -raw ingestion_workflow_service_account_email)"

gcloud iam service-accounts add-iam-policy-binding "$ORCHESTRATOR_SA" \
    --member="user:$MY_USER" \
    --role="roles/iam.serviceAccountTokenCreator" \
    --project="$PROJECT_ID"

Important

Without this IAM binding, the CLI cannot generate OAuth tokens to authenticate with the Cloud Workflows API or Ingestion Helper service, resulting in HTTP 403 Forbidden errors.

Debugging Private Cloud Run Services Locally

When instances are deployed with datacommons_services_allow_unauthenticated_access = false (the secure default), you do not need to make services public or modify IAM policies to test HTTP endpoints. Establish an authenticated local proxy tunnel to the Cloud Run service:

cd ~/dcp-deployments/<namespace>

export PROJECT_ID="$(terraform output -raw project_id)"
export REGION="$(terraform output -raw region)"
export SERVICE_NAME="$(terraform output -raw datacommons_service_name)"

gcloud run services proxy "$SERVICE_NAME" \
    --project="$PROJECT_ID" \
    --region="$REGION" \
    --port=8080

This establishes an encrypted tunnel forwarding http://localhost:8080 to the private Cloud Run service, automatically attaching your gcloud credentials to every request. You can then query endpoints directly via cURL or your browser:

# Test the V2 Resolve endpoint through the proxy tunnel:
curl -s "http://localhost:8080/core/api/v2/resolve?nodes=california&resolver=place" | jq .

# Test natural language detection and fulfillment:
curl -s -X POST "http://localhost:8080/api/explore/detect-and-fulfill?q=population+in+california" \
    -H "Content-Type: application/json" \
    -d '{}' | jq .

Testing Strategy and Execution

DCP organizes automated tests into two tiers:

  • Unit Tests (Python and Terraform): Fast, offline tests that run automatically in pre-submit CI without live GCP credentials:
    • Python Unit Tests: In-memory tests running via pytest where external cloud APIs (Cloud Spanner, Cloud Workflows, Cloud Storage) and shell calls are mocked. Required for every bug fix, feature, and CLI subcommand.
    • Terraform Module Unit Tests: Hermetic module tests (*.tftest.hcl) that run terraform plan offline with mocked providers (mock_provider "google").
  • Hermetic Integration Tests (End-to-End Validation): Local multi-service tests using Docker Compose to emulate Cloud Spanner, Cloud Storage, and serving containers offline without incurring GCP cloud costs. Integration tests validate end-to-end data ingestion, schema migrations, and query resolution before cutting releases or modifying cross-cutting data pipelines.

Unit Tests

# Run all Python unit tests (or scope to a package, e.g. packages/datacommons-admin/tests/)
uv run pytest

# Run Terraform formatting, validation, and module unit tests
uv run infra/dcp/scripts/check_terraform.py

Single Source of Truth: For check_terraform.py flags (--fix, --fmt, --validate, --test) and instructions on authoring .tftest.hcl unit tests, refer to the Infrastructure Guide.

Hermetic Integration Tests

The repository includes a hermetic testbed that uses Docker Compose to emulate Cloud Spanner, Cloud Storage, Ingestion Helper, and serving containers locally without incurring GCP cloud costs:

uv run pytest tests/integration/suites/ \
    --instance emulated \
    --test-config foobar_wages

Single Source of Truth: For complete details on Docker Compose emulation, developer fast-iteration flags (such as --reuse-data to skip re-ingestion), and running tests against live GCP sandbox projects, refer to the Integration Tests Guide.


Test coverage

CI measures test coverage on every pull request to main and every push to main, prints it in the build log, and uploads it to Codacy (report-only: coverage never fails the build).

To measure coverage locally:

uv run pytest --cov --cov-report=term --cov-report=xml
uv run diff-cover coverage.xml --compare-branch=origin/main

This writes coverage.xml. Only code under packages/ is measured; Terraform and integration tests aren't included.


Debugging Workflows and Common Gotchas

Missing DC_API_KEY

  • Symptom: Integration tests or local serving queries fail with unauthorized or upstream RPC errors.
  • Resolution: DCP federates queries against base Google Data Commons. Obtain a key from apikeys.datacommons.org and ensure DC_API_KEY is exported in your shell:
    export DC_API_KEY="your-api-key"

BigQuery Reservation Collision

  • Symptom: terraform apply fails with an error indicating that a BigQuery slot reservation already exists in the project and region.
  • Resolution: Google Cloud allows only one BigQuery slot reservation per project per region. When sharing a development project (such as datcom-website-dev), set spanner_create_bigquery_reservation = false in your terraform.tfvars.

Spanner Emulator Port Conflicts

  • Symptom: Hermetic integration tests report Address already in use on emulator ports (such as port 9010 or 9020).
  • Resolution: Ensure no previous Docker Compose test containers are running:
    docker compose -f tests/integration/emulated/docker-compose.yml down -v

Service Account Token Creator Missing

  • Symptom: datacommons admin init-db or datacommons admin ingest start fails with HTTP 403 / IAM permission denied when acquiring credentials.
  • Resolution: Ensure your GCP user account has roles/iam.serviceAccountTokenCreator on the workflow service account:
    gcloud iam service-accounts add-iam-policy-binding <workflow-sa-email> \
        --member="user:$(gcloud config get-value account)" \
        --role="roles/iam.serviceAccountTokenCreator" \
        --project=<project-id>