Add terraform script for complete deployment of the Registry in the Dev Account#475
Open
tloubrieu-jpl wants to merge 36 commits into
Open
Add terraform script for complete deployment of the Registry in the Dev Account#475tloubrieu-jpl wants to merge 36 commits into
tloubrieu-jpl wants to merge 36 commits into
Conversation
added 10 commits
February 5, 2026 18:56
…ds-infra repository
Merged
14 tasks
… with test data on AWS
…m their own terraform, integration test still fail.
added 2 commits
May 6, 2026 16:43
nutjob4life
reviewed
May 7, 2026
| common_tags = var.common_tags | ||
| } | ||
|
|
||
| # TL: not ready for integration as we have a deadline for the ISRO node creation in dev, step done manuallyt |
Member
There was a problem hiding this comment.
@tloubrieu-jpl I feel seen! 😁 (Which is also how some people pronounce my name 🤣)
Thank you for doing this 👍
added 5 commits
May 7, 2026 10:48
… from terragrunt call
Member
|
@tloubrieu-jpl to confirm the failing branch CI CD is expected? |
Contributor
There was a problem hiding this comment.
Pull request overview
Introduces a Terraform configuration (root module plus opensearch, lambda, and api_gateway sub-modules) for deploying the Registry stack to the dev AWS account, along with a Python helper script to populate the deployed registry with reference data, and a few small unrelated doc/data tweaks.
Changes:
- Adds a multi-module Terraform layout (OpenSearch Serverless collection + policies, Lambda for Cognito→AWS credentials exchange, API Gateway) with example
tfvars/backend config and a README walkthrough. - Adds
run-init-on-aws.pywhich reads Terraform outputs and runsregistry-loaderDocker containers to create indexes, harvest test data, and set archive statuses. - Misc: updates
docker/.envquoting,docker/README.mdintro, an install-docs note, and appends an entry to the staged-bundles CSV.
Reviewed changes
Copilot reviewed 29 out of 29 changed files in this pull request and generated 15 comments.
Show a summary per file
| File | Description |
|---|---|
| terraform/main.tf, data.tf, variables.tf, outputs.tf, provider.tf, backend.tf | Root module wiring the three sub-modules and SSM-sourced IAM roles. |
| terraform/opensearch/{main,variables,outputs,data}.tf | OpenSearch Serverless collection with encryption/network/data-access policies and per-node write rules. |
| terraform/lambda/{main,variables,outputs}.tf, src/get_awskeys_from_cognitojwt.py, src/layer/requirements.txt, build/layer/README.md | Lambda exchanging Cognito JWTs for temporary AWS credentials, packaged via local-exec layer build. |
| terraform/api_gateway/{main,variables,outputs}.tf | REST API exposing /credentials and integrating with the Lambda via request-mapping template. |
| terraform/run-init-on-aws.py, harvest-job-config.xml.template | Post-deploy initialization helper that loads reference data into the collection. |
| terraform/README.md, terraform.tfvars.example, backend-config.tfvars.example, backend_env.sh.example | Operator-facing documentation and example configs. |
| docker/README.md, docker/.env, docs/source/install/install.rst, docs/status/staged_bundles_in_registry.csv | Minor unrelated documentation/data updates. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| )['Groups'] | ||
|
|
||
| # Check if user is part of any allowed groups | ||
| if not user_groups or not any(group['GroupName'] in COGNITO_ALLOWED_GROUPS for group in user_groups): |
Contributor
There was a problem hiding this comment.
Fixed in 0e2e48a. COGNITO_ALLOWED_GROUPS is now split on commas into a set at assignment time, so group['GroupName'] in COGNITO_ALLOWED_GROUPS performs exact membership testing rather than substring matching.
Comment on lines
+13
to
+37
| # Option 1: Public access (easier for development/testing) | ||
| enable_public_access = true | ||
|
|
||
| # Option 2: VPC-only access via VPC endpoint (recommended for production) | ||
| # enable_public_access = false | ||
| # create_vpc_endpoint = true | ||
|
|
||
| # Needed for VPC end-point above and registry API load balancer and ECS service | ||
| vpc_id = "vpc-1234567890abcdef0" | ||
| subnet_ids = ["subnet-1234567890abcdef0", "subnet-0987654321fedcba0"] | ||
| public_subnet_ids = ["subnet-3234567890abcdef0", "subnet-3987654321fedcba0"] | ||
| security_group_id = [] # Leave empty to create a default security group | ||
|
|
||
| # Admin roles configuration (see DEPLOYMENT_STAGES.md for deployment workflow) | ||
| # Stage 1 (initial): Deploy without admin role | ||
| admin_console_role = [] | ||
| use_ssm_for_admin_role = false | ||
|
|
||
| # Stage 2 (after IAM role created in separate terraform): Enable SSM to read role ARN | ||
| # The IAM role ARN should be stored at: /pds/infra/iam/roles/pds_cognito_cds_admin_role_arn | ||
| # use_ssm_for_admin_role = true | ||
|
|
||
| # Alternative: Directly specify admin role ARNs for console access | ||
| # admin_console_role = ["arn:aws:iam::123456789012:role/pds-registry-admin"] | ||
|
|
| self.add_locally_defined_env_vars() | ||
| self.generate_auth_configs() | ||
|
|
||
| node_registry_with_ref_data = os.environ.get("NODE_REGISTRY_WITH_REF_DATA") |
Comment on lines
+140
to
+159
| def validate_jwt_token(token, JWSK_URL): | ||
| try: | ||
| # Download the JWSK | ||
| response = requests.get(JWSK_URL) | ||
| if response.status_code != 200: | ||
| raise ValueError("Unable to download JWSK") | ||
|
|
||
| jwsk = response.json() | ||
| headers = jwt.get_unverified_headers(token) | ||
| kid = headers['kid'] | ||
|
|
||
| # Search for the kid in the downloaded JWSK | ||
| public_key = None | ||
| for key in jwsk['keys']: | ||
| if kid == key.get('kid', ''): | ||
| public_key = jwk.construct(key) | ||
| break | ||
|
|
||
| if public_key is None: | ||
| raise ValueError("No keys found in JWSK.") |
Comment on lines
+128
to
+154
| "Principal" : var.admin_roles, | ||
| "Description" : "PDS - OpenSearch Admin Access" | ||
| }, | ||
| { | ||
| Rules = [ | ||
| { | ||
| "Resource" : [ | ||
| "collection/${var.collection_name}*" | ||
| ], | ||
| "Permission" : [ | ||
| "aoss:DescribeCollectionItems" | ||
| ], | ||
| "ResourceType" : "collection" | ||
| }, | ||
| { | ||
| "Resource" : [ | ||
| "index/*/*" | ||
| ], | ||
| "Permission" : [ | ||
| "aoss:ReadDocument", | ||
| "aoss:DescribeIndex" | ||
| ], | ||
| "ResourceType" : "index" | ||
| } | ||
| ], | ||
| "Principal" : var.readonly_roles, | ||
| "Description" : "PDS - OpenSearch Read-only Access" |
|
|
||
|
|
||
| docker_cmd = [ | ||
| "docker", "run", "--rm", "-it", |
Comment on lines
+216
to
+217
| for k , v in os.environ.items(): | ||
| self.env_vars[k] = v |
Comment on lines
+423
to
+425
| with tarfile.open(download_path, "r:gz") as tar: | ||
| tar.extractall(path=extract_dir) | ||
| print(f" Extracted {len(tar.getmembers())} files to: {extract_dir}\n") |
Comment on lines
+114
to
+126
| except subprocess.CalledProcessError: | ||
| print(f"Error: Terraform outputs not found. Have you run '{self.tf_cmd} apply'?") | ||
| return False | ||
|
|
||
| self.credentials_endpoint = self.run_command( | ||
| [self.tf_cmd, "output", "-raw", "credentials_endpoint"], | ||
| cwd=self.tf_working_dir, | ||
| ) | ||
|
|
||
| node_list_json = self.run_command( | ||
| [self.tf_cmd, "output", "-json", "node_list"], | ||
| cwd=self.tf_working_dir, | ||
| ) |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Member
|
@tloubrieu-jpl can you review the comments from Copilot and either either dismiss the proposed updates or assign copilot to fix them? |
This was referenced Jun 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🗒️ Summary
This PR initiates a terraform script which deployed opensearch and the needed policies to start.
A follow up action is going to be able to initialize the registry with registry-mgr and the reference dataset.
It is ready to be merged as it is used for deployment in dev with validated integration tests as a basis for the ISRO node creation.
⚙️ Test Data and/or Report
Integration test validated.
♻️ Related Issues
🤓 Reviewer Checklist
Reviewers: Please verify the following before approving this pull request.
Documentation and PR Content
Security & Quality
Testing & Validation
Maintenance