feat: Add bedrock-operation-review skill

[sean-nixon]

Description

This PR adds the bedrock-operation-review skill, a comprehensive Amazon Bedrock operational review aligned with the AWS Well-Architected Framework and Bedrock best practices. The skill evaluates workloads across five pillars — Security, Performance, Service Quotas, Cost Optimization, and Resilience — covering resource discovery (foundation models, guardrails, inference profiles, provisioned throughput, custom models, agents, knowledge bases, and Prompt Management), CloudWatch metric analysis with threshold-based severity classification, service quota utilization, and cost optimization opportunities such as prompt caching, model distillation, and batch inference. All data is collected through AWS control-plane APIs only (Bedrock, Bedrock Agent, CloudWatch, Service Quotas, EC2), with no data-plane model invocations and no prompt/response content read, and reviews produce a severity-ranked, shareable Markdown report.

Type of change

• New skill
• New custom agent
• Update to an existing skill or agent
• Documentation or infrastructure change

Testing

Tested using Agent Skill Eval (report in PR). Manually tested in DevOps Agent to confirm the skill triggered and output was valid. Full investigation produced correct findings with proper severity ratings.

License confirmation

• By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0.

[shekharsh]

Domain SME review complete. Accuracy, safety, and structure verified — all six items from my earlier review confirmed on the branch (model-access scoping, README pillar count, non-production disclaimer, CHANGELOG date, guardrails-quotas link, eval skill_path scrub), plus the metrics-thresholds Model Lifecycle correction. Read-only enforced, no internal refs or customer data, trigger evals 6/6, functional report passed. Approving.

[ams-thakkar]

Thanks for the thorough contribution and SME review. The overall direction is strong: read-only control-plane collection, five clear pillars, explicit thresholds, and a shareable report. I focused on behavior, data boundaries, repository integration, and evaluation evidence.

I have four items to address for v1; the last two are non-blocking recommendations for v2.

Must address for v1

1. Fix inactive-region detection. ListFoundationModels returns the regional catalog of models available to use; it is not evidence of account activity. The current probe will generally be non-empty in supported regions, causing inactive regions to receive a full review. Use account-owned resources and/or AWS/Bedrock metrics as the activity signal, and treat the foundation-model list only as catalog data. See ListFoundationModels.

2. Reconcile the no-prompt-content promise. SKILL.md calls GetPrompt and evaluates whether prompt templates are hardened against injection, while SKILL.md/README also promise that no prompt content is read. GetPrompt returns prompt variants/template configuration, so both statements cannot remain true. Either remove the content-level prompt-injection check and avoid processing prompt templates, or explicitly document what sensitive configuration is retrieved and how it is handled. See GetPrompt.

3. Fix repository/install integration. Please:

   • Add bedrock-operation-review to the root llms.txt Available Skills list.
   • Update the README zip command so SKILL.md is at the archive root and development-only files (README.md, CHANGELOG.md, .skilleval.yaml, evals/) are excluded. Running zip -r ... bedrock-operation-review/ from skills/ preserves the parent directory in the archive; build the zip from inside the skill directory instead.

4. Align the custom-agent contract. aws-operation-review/SYSTEM_PROMPT.md adds Bedrock in Step 2, but Step 1 still tells the agent to identify only EKS, RDS, or Aurora. Add Bedrock there. Also have the custom agent defer to the selected skill's report schema; its generic categories conflict with this skill's five pillars.

Recommended for v2 — non-blocking

5. Strengthen operational eval coverage. The current evals are useful smoke tests, but they validate context parsing, artifact naming, API boundaries, severity names, and pillar names—not a real operational-review path or threshold branch. Add a fixture-driven end-to-end review and at least one threshold/severity case in v2. I would not block v1 on this given the SME review and manual DevOps Agent run.

6. Tighten evidence requirements for heuristic findings. Some recommendations depend on workload intent that control-plane data may not establish—for example, interpreting guardrail intervention rates, prompt-routing suitability, or latency sensitivity. In v2, define minimum evidence per finding and report UNKNOWN / "needs workload context" when it is absent rather than presenting a heuristic as definitive.

Once the four v1 items are addressed, I am comfortable approving. Items 5–6 improve depth and confidence but are not merge blockers.

[sean-nixon]

Thank you @ams-thakkar. I've addressed your 4 blocking items in my latest commit. Can you take another look?

[ams-thakkar]

Thanks for addressing the v1 findings. I re-verified the current branch and am approving.

Resolved for v1:

• Inactive-region detection no longer uses ListFoundationModels as account activity; it uses account-owned resources and CloudWatch metrics.
• GetPrompt and the template-level prompt-injection assessment were removed; the check now uses guardrail prompt-attack filtering as the observable control.
• llms.txt entry added; the README zip command now puts SKILL.md at archive root and excludes development-only files.
• The custom agent now recognizes Bedrock in Step 1 and defers to the selected skill's report schema/pillars.

Validation completed: the generated zip contains only root-level SKILL.md plus the two references; eval JSON is valid; SKILL/CHANGELOG versions match at 1.0.0; and mkdocs build --strict passes.

The following are v2 follow-ups, not v1 blockers:

1. Make the activity probe explicit as ListInferenceProfiles(typeEquals="APPLICATION") so system-defined regional profiles cannot count as account activity.
2. Tighten the prompt-content boundary around GetAgentVersion, whose response can include instruction and basePromptTemplate: either remove the call or disclose that these fields may be returned and require that they are never analyzed, stored, or reproduced.
3. Add a fixture-driven end-to-end operational review and at least one threshold/severity eval.
4. Define minimum evidence for intent-dependent heuristic findings and emit UNKNOWN / needs-workload-context when it is absent.

Scope note: I reviewed behavior, data boundaries, repository integration, and validation evidence; relying on the Bedrock SME review for domain correctness.