From 057d2a9b122892885a60e70043e5d6fe0fa199dd Mon Sep 17 00:00:00 2001 From: shawjiv Date: Thu, 10 Sep 2026 15:53:18 -0500 Subject: [PATCH 1/3] Add aws-routing skill for AWS routing and BGP path-selection analysis --- skills/aws-routing/CHANGELOG.md | 30 + skills/aws-routing/README.md | 50 ++ skills/aws-routing/SKILL.md | 104 ++++ skills/aws-routing/evals/eval_queries.json | 58 ++ skills/aws-routing/evals/evals.json | 98 +++ .../cloudwan-dx-routing-patterns.md | 198 ++++++ .../references/dx-routing-patterns.md | 217 +++++++ .../networking-strategic-questions.md | 584 ++++++++++++++++++ .../references/tgw-routing-patterns.md | 147 +++++ .../references/vpn-dx-redundancy-patterns.md | 142 +++++ 10 files changed, 1628 insertions(+) create mode 100644 skills/aws-routing/CHANGELOG.md create mode 100644 skills/aws-routing/README.md create mode 100644 skills/aws-routing/SKILL.md create mode 100644 skills/aws-routing/evals/eval_queries.json create mode 100644 skills/aws-routing/evals/evals.json create mode 100644 skills/aws-routing/references/cloudwan-dx-routing-patterns.md create mode 100644 skills/aws-routing/references/dx-routing-patterns.md create mode 100644 skills/aws-routing/references/networking-strategic-questions.md create mode 100644 skills/aws-routing/references/tgw-routing-patterns.md create mode 100644 skills/aws-routing/references/vpn-dx-redundancy-patterns.md diff --git a/skills/aws-routing/CHANGELOG.md b/skills/aws-routing/CHANGELOG.md new file mode 100644 index 0000000..94db313 --- /dev/null +++ b/skills/aws-routing/CHANGELOG.md @@ -0,0 +1,30 @@ +# Changelog + +## [1.2.0] - 2026-08-25 +### Added +- Two functional eval scenarios covering the corrected knowledge: TGW→VPN summarization (keep propagation + static summary + limit-hit selection order) and overlapping DX supernet into Cloud WAN (Routing Policies drop/suppress on DX; no BGP communities on DX attachments). +- Matching positive trigger queries for the two new scenarios. +### Changed +- `references/tgw-routing-patterns.md`: clarified the limit-hit advertisement-selection order (static over propagated, then least-specific), labeled as field knowledge. +- `references/cloudwan-dx-routing-patterns.md`: added that Cloud WAN Routing Policies support prefix filtering (drop/summarize) inbound/outbound on Direct Connect attachments (distinct from the legacy allowed-prefixes list); DX attachments cannot match/set BGP communities. + +## [1.1.0] - 2026-08-18 +### Changed +- Refined the `description` for better activation: added symptom-based trigger phrasings and explicit + service/keyword coverage. +### Added +- Expanded functional evals (DX location preference, TGW ECMP, DX+VPN redundancy, verification of + unknown sources) — 7 scenarios total. +- Expanded trigger tests with more positive routing prompts and additional negative (non-routing) + prompts. + +## [1.0.0] - 2026-08-18 +### Added +- Initial release of the `aws-routing` skill, adapted from the AWS Routing custom agent. +- Route-evaluation guidance for Cloud WAN CNEs, Direct Connect Gateway path selection, Transit + Gateway route tables, and VPC route tables. +- BGP traffic-engineering guidance: LP communities (`7224:7100/7200/7300`), AS-path prepending + (within-region only), MED (low-priority tiebreaker), and longest-prefix-match behavior. +- DX + VPN redundancy, active/active vs active/passive, and failover analysis. +- Reference knowledge base under `references/`. +- Verification / anti-hallucination directives built into the skill instructions. diff --git a/skills/aws-routing/README.md b/skills/aws-routing/README.md new file mode 100644 index 0000000..e34440c --- /dev/null +++ b/skills/aws-routing/README.md @@ -0,0 +1,50 @@ +# aws-routing skill + +An AWS routing subject matter expert skill for path selection, BGP traffic engineering, and route +evaluation across AWS networking services (Cloud WAN, Direct Connect, Transit Gateway, VPC, VPN). +It provides **read-only** analysis, recommendations, and troubleshooting grounded in public AWS +documentation and field-proven patterns. + +## What it does + +- Explains and applies the route-evaluation order for Cloud WAN CNEs, DXGW path selection, TGW route + tables, and VPC route tables. +- Advises on BGP traffic engineering: LP communities (`7224:7100/7200/7300`), AS-path prepending + (within-region only), MED, and longest-prefix-match. +- Analyzes DX + VPN redundancy, active/active vs active/passive, and failover behavior. +- Flags non-deterministic path selection and recommends deterministic alternatives. +- Produces read-only validation commands (`describe-*`, `get-*`, `list-*`). + +## Prerequisites + +- No write permissions required — the skill is read-only. +- To run the suggested validation commands, the operator needs read-only access to the relevant + networking APIs (e.g., `ec2:Describe*`, `directconnect:Describe*`, `networkmanager:Get*`). + +## How to use it + +Ask routing questions in natural language, for example: +- "Why is my Cloud WAN traffic for an aggregate prefix leaving the local region instead of using the + local Direct Connect?" +- "How do I make a Transit Gateway prefer one Direct Connect location over another?" +- "Is my DX + VPN design active/active or active/passive, and will failover be deterministic?" + +The skill will trace the path, apply the evaluation logic, cite the relevant AWS documentation, and +suggest read-only commands to validate. + +## Structure + +``` +aws-routing/ +├── SKILL.md # Activation metadata + instructions +├── references/ # Detailed routing mechanics and patterns +├── README.md # This file +├── CHANGELOG.md # Version history +└── evals/ # Functional and trigger tests +``` + +## Disclaimer + +> ⚠️ This skill is sample code, not intended for production use without additional review and +> testing. Validate all routing guidance in a non-production environment first. Recommendations are +> based on public AWS documentation and field experience and may change as AWS services evolve. diff --git a/skills/aws-routing/SKILL.md b/skills/aws-routing/SKILL.md new file mode 100644 index 0000000..8f1893f --- /dev/null +++ b/skills/aws-routing/SKILL.md @@ -0,0 +1,104 @@ +--- +name: aws-routing +description: 'Analyze and troubleshoot AWS routing and BGP path selection - how AWS chooses a network path, why traffic takes an unexpected route, and how to steer it. Use for AWS Cloud WAN route evaluation and CNE path selection; Direct Connect Gateway (DXGW) path selection and local vs remote region preference; Transit Gateway route tables, ECMP, and peering; VPC route tables and longest-prefix-match; BGP traffic engineering with local-preference communities (7224:7100/7200/7300), AS-path prepending, and MED; DX + VPN redundancy and active/active vs active/passive design; asymmetric routing and failover. Activate on symptoms like "traffic is leaving the region", "not using my local Direct Connect", "routing to the wrong VIF/location", "path is asymmetric", or "how do I make AWS prefer this path". Read-only: analysis, recommendations, and describe/get/list validation commands grounded in public AWS docs.' +metadata: + version: "1.2" + author: shawjiv +--- + +## Overview + +This skill provides routing-first analysis for AWS networking. It reasons about the full traffic +path, applies the documented route-evaluation logic for each service, and grounds recommendations +in public AWS documentation. It is **read-only**: it produces analysis, recommendations, and +read-only validation commands (`describe-*`, `get-*`, `list-*`) - never mutating actions. + +## Verification & Anti-Hallucination Directives (Mandatory) + +- Do not generate any fact, citation, or source that is not fully verifiable. If you lack a verified + source, state "I cannot verify this" rather than inventing details. +- State assumptions explicitly before answering. +- Cross-check each claim against public AWS documentation or the `references/` files. Omit claims you + cannot confirm. +- Only cite AWS documentation URLs you have actually retrieved or confirmed. Do not fabricate links. + +## When to use this skill + +Activate when the user asks about: +- Cloud WAN route evaluation / Core Network Edge (CNE) path selection +- Direct Connect Gateway path selection, local-region preference, or LP communities +- Transit Gateway route evaluation, ECMP, or peering routing +- VPC route tables and longest-prefix-match behavior +- BGP traffic engineering (communities, AS-path prepending, MED) +- DX + VPN redundancy, active/active vs active/passive, or failover design +- Why traffic is taking an unexpected path + +## What this skill does NOT do + +This skill is **read-only and advisory**. It will not: +- **Design or apply infrastructure changes.** It does not create, modify, or delete AWS resources, + and it does not run mutating CLI/API calls (`create-*`, `modify-*`, `associate-*`, `delete-*`, + `put-*`, `update-*`) or write Infrastructure-as-Code to be deployed. Any change it describes is a + *recommendation* for you to review and implement yourself. +- **Configure BGP or push routing policy.** It explains which communities, AS-path, or MED settings + to use, but it does not apply them to VIFs, route tables, or Cloud WAN policy documents. +- **Execute anything against a live account beyond read-only validation.** Commands it provides are + limited to `describe-*`, `get-*`, and `list-*` for confirming state. +- **Guarantee production outcomes.** It recommends non-production testing and blue/green rollout; + it does not perform the cutover or validate the result for you. +- **Provide compliance, security, or contractual sign-off**, or make claims it cannot ground in + public AWS documentation or the `references/` files. + +For designing and applying changes, hand the recommendation to a change-capable workflow (with +appropriate review, approvals, and testing). + +## How to analyze a routing problem + +1. **Establish the path.** Identify source, destination, and every routing construct in between + (VPC route table → TGW/Cloud WAN CNE → DXGW → VIF → on-prem, or the relevant subset). +2. **Name the route table.** Be explicit about which table makes each decision (VPC RT, TGW RT, + CNE RT, DXGW internal selection). +3. **Apply the evaluation order for that construct** (see below). +4. **Identify the deciding attribute** (longest prefix, local preference, AS-path, MED, source type). +5. **Flag non-determinism** (e.g., ECMP or "deterministically random" tiebreakers) and recommend a + deterministic alternative. +6. **Cite the AWS documentation** that supports the behavior. +7. **Provide read-only validation commands** the user can run to confirm. + +## Key evaluation rules + +### Cloud WAN (per CNE) +Longest prefix match → static → VPC-propagated (same region) → unequal AS-path/MED (shortest wins) → +equal AS-path & MED source preference: DXGW-propagated → Cloud WAN Connect → S2S VPN → other +(TGW peering, remote CNEs; identical from 2+ sources = deterministically random). + +### Direct Connect (private/transit VIF, outbound AWS→on-prem) +Longest prefix match → local preference → AS_PATH length → MED → ECMP (equal AS_PATH and BGP +attributes). LP communities `7224:7300/7200/7100` (High/Medium/Low) set local preference and are +evaluated before AS-path. DXGW prefers the local associated-region DX by default. + +### Direction & tooling guidance +- **AWS → on-prem (egress):** control with DX LP communities. +- **On-prem → AWS (ingress):** control with the customer router's local-pref/weight/MED. +- **AS-path prepending:** within a region only - unreliable across regions because DXGW local-region + LP overrides it. **LP communities:** work within-region and cross-region. + +## Reference material + +Detailed mechanics, patterns, and caveats are in `references/`: +- `cloudwan-dx-routing-patterns.md` - Cloud WAN route evaluation, DXGW path selection, community + traffic-engineering patterns, regional inspection, multi-region egress +- `dx-routing-patterns.md` - DX BGP mechanics, BFD, MED, ASN ranges, VIF considerations, CloudHub, + when VIFs can talk, active/active vs active/passive +- `tgw-routing-patterns.md` - TGW route evaluation, ECMP limits, DX/VPN caveats, failover, peering +- `vpn-dx-redundancy-patterns.md` - VPN + DX redundancy, route preference, ECMP limits, asymmetry +- `networking-strategic-questions.md` - discovery questions per service + +## Output expectations + +- Explain the route-evaluation logic step by step. +- Name the specific route table involved. +- Give the end-to-end traffic path. +- Compare options with tradeoffs (determinism, complexity, failover) when multiple solutions exist. +- Include read-only AWS CLI validation commands. +- Recommend testing in a non-production environment and blue/green for migrations. diff --git a/skills/aws-routing/evals/eval_queries.json b/skills/aws-routing/evals/eval_queries.json new file mode 100644 index 0000000..b658864 --- /dev/null +++ b/skills/aws-routing/evals/eval_queries.json @@ -0,0 +1,58 @@ +[ + { + "query": "Why is my Cloud WAN traffic for 11.0.0.0/8 leaving the local region instead of using the local Direct Connect?", + "should_trigger": true + }, + { + "query": "How do I make my Transit Gateway prefer one Direct Connect location over another?", + "should_trigger": true + }, + { + "query": "Explain how BGP local preference communities 7224:7300 influence Direct Connect egress.", + "should_trigger": true + }, + { + "query": "My return traffic is asymmetric between Direct Connect and VPN. How do I fix the path selection?", + "should_trigger": true + }, + { + "query": "When will a Transit Gateway ECMP across two DX attachments versus pick a single path?", + "should_trigger": true + }, + { + "query": "How does longest prefix match interact with local preference on a private VIF?", + "should_trigger": true + }, + { + "query": "Can two on-prem sites reach each other through AWS over separate DX VIFs?", + "should_trigger": true + }, + { + "query": "How do I summarize routes advertised from a Transit Gateway to a Site-to-Site VPN without breaking return traffic?", + "should_trigger": true + }, + { + "query": "An on-prem supernet advertised into Cloud WAN overlaps my VPC CIDRs. How do I stop it shadowing my VPCs?", + "should_trigger": true + }, + { + "query": "Write a Python function to reverse a linked list.", + "should_trigger": false + }, + { + "query": "Create an S3 bucket with versioning enabled using the AWS CLI.", + "should_trigger": false + }, + { + "query": "Summarize this quarter's EC2 cost trend.", + "should_trigger": false + }, + { + "query": "Help me write a Lambda function to resize images uploaded to S3.", + "should_trigger": false + }, + { + "query": "What IAM policy do I need to allow read-only access to DynamoDB?", + "should_trigger": false + } +] diff --git a/skills/aws-routing/evals/evals.json b/skills/aws-routing/evals/evals.json new file mode 100644 index 0000000..13d5b81 --- /dev/null +++ b/skills/aws-routing/evals/evals.json @@ -0,0 +1,98 @@ +[ + { + "id": "cloudwan-local-dx-preference", + "prompt": "In AWS Cloud WAN, traffic for an aggregate prefix is leaving the local region instead of using the local Direct Connect. How does route evaluation decide this and how can I prefer the local DX?", + "expected_output": "Explains Cloud WAN route evaluation (longest-prefix-match first, then AS-path at the CNE), why a remote path can win, and recommends LP communities (7224:7300/7200/7100) and/or AS-path normalization to prefer the local DX.", + "assertions": [ + "references longest prefix match in Cloud WAN route evaluation", + "explains that AS-path length is evaluated at the CNE and can cause a remote path to win", + "recommends LP communities (7224:7300/7200/7100) as a deterministic remedy", + "does not contain 'error'" + ] + }, + { + "id": "dx-path-selection-order", + "prompt": "What is the outbound path-selection order for a Direct Connect private VIF, and where does MED fit?", + "expected_output": "Order: longest prefix match, then local preference, then AS_PATH length, then MED, then ECMP. MED is a low-priority tiebreaker AWS does not recommend relying on. LP communities 7224:7100/7200/7300 set local preference.", + "assertions": [ + "states the order longest prefix match, local preference, AS_PATH, MED, ECMP", + "notes MED is a low-priority tiebreaker", + "mentions LP communities 7224:7100/7200/7300", + "does not contain 'error'" + ] + }, + { + "id": "tgw-vs-vgw-vif-transit", + "prompt": "Can two on-premises sites on separate Direct Connect VIFs talk to each other through AWS?", + "expected_output": "By default AWS does not act as a transit hub (advertises only VPC CIDRs). VPN CloudHub enables site-to-site for VPN connections; TGW with an Allowed Prefix List (and two DXGWs for DX-to-DX) can enable it intentionally.", + "assertions": [ + "explains AWS does not act as a transit hub by default", + "mentions VPN CloudHub for VPN connections", + "mentions TGW or two DXGWs as a way to enable DX-to-DX intentionally", + "does not contain 'error'" + ] + }, + { + "id": "dx-prefer-one-location", + "prompt": "I have two Direct Connect locations and want traffic to prefer location A over location B, with B as backup. How do I do this and what should I avoid?", + "expected_output": "Use LP communities (7224:7300 on primary, lower on backup). AS-path prepending is reliable only within the same associated region; cross-region prepending is unreliable due to DXGW local-region LP.", + "assertions": [ + "recommends LP communities with a higher preference on the primary", + "explains AS-path prepending is reliable only within the same region", + "warns cross-region AS-path prepending is unreliable", + "does not contain 'error'" + ] + }, + { + "id": "tgw-ecmp-behavior", + "prompt": "When does a Transit Gateway load-balance across two Direct Connect paths, and when does it pick just one?", + "expected_output": "ECMP requires the same attachment type and equal AS-path length / BGP attributes; otherwise TGW installs a single (oldest) route. LP communities are not honored across multiple DXGW attachments on TGW.", + "assertions": [ + "states ECMP requires same attachment type and equal AS-path/BGP attributes", + "notes TGW picks a single route when ECMP conditions are not met", + "does not contain 'error'" + ] + }, + { + "id": "dx-vpn-redundancy-preference", + "prompt": "I have a Direct Connect and a backup Site-to-Site VPN to the same VGW. Which path does AWS prefer and how do I avoid asymmetric routing?", + "expected_output": "DX is always preferred over VPN for the same prefix at a VGW; there is no ECMP between DX and VPN. Asymmetric routing is the main failure mode; match ingress/egress preference and validate both directions.", + "assertions": [ + "states DX is always preferred over VPN for the same prefix at a VGW", + "identifies asymmetric routing as the main risk", + "does not contain 'error'" + ] + }, + { + "id": "verification-unknown-source", + "prompt": "Is there an official AWS 'DX Path Selection Flow Chart' document I can cite?", + "expected_output": "Does not fabricate a document or URL; states it cannot verify such a named artifact; points to the Direct Connect routing policies documentation for the path-selection order.", + "assertions": [ + "does not fabricate a document or URL", + "states it cannot verify such a named artifact", + "points to the Direct Connect routing policies documentation" + ] + }, + { + "id": "tgw-vpn-summarization", + "prompt": "A customer is at the TGW-to-VPN advertised-route limit. How do they summarize without breaking the return path?", + "expected_output": "Keep VPC propagation enabled (specifics needed for longest-prefix-match return forwarding), add a static summary route in the VPN-associated TGW route table; under the cap, static beats propagated and least-specific is advertised first, so the summary survives. Anchor the summary to a persistent attachment.", + "assertions": [ + "keeps VPC route propagation enabled for the return path", + "adds a static summary route in the TGW route table associated with the VPN attachment", + "explains static-over-propagated and least-specific selection under the advertisement cap", + "does not contain 'error'" + ] + }, + { + "id": "cloudwan-overlapping-supernet", + "prompt": "A customer advertises a supernet (10.0.0.0/8) from Direct Connect into Cloud WAN that overlaps VPC CIDRs (10.1.0.0/16, 10.2.0.0/16). What issues would this cause and how do they fix it?", + "expected_output": "Longest-prefix-match means VPC /16s win and on-prem hosts inside a VPC CIDR are shadowed/unreachable. Best fix is non-overlapping address planning. Cloud WAN Routing Policies can drop the inbound supernet and suppress VPC prefixes outbound on the DX attachment; BGP communities cannot be matched on DX attachments.", + "assertions": [ + "explains longest-prefix-match shadows on-prem hosts inside a VPC CIDR", + "recommends non-overlapping address planning as the best fix", + "mentions Cloud WAN Routing Policies can filter (drop) prefixes on the Direct Connect attachment", + "does not contain 'error'" + ] + } +] diff --git a/skills/aws-routing/references/cloudwan-dx-routing-patterns.md b/skills/aws-routing/references/cloudwan-dx-routing-patterns.md new file mode 100644 index 0000000..e630636 --- /dev/null +++ b/skills/aws-routing/references/cloudwan-dx-routing-patterns.md @@ -0,0 +1,198 @@ +--- +inclusion: manual +--- + +# Cloud WAN + Direct Connect Routing Patterns & Considerations + +This document captures key routing mechanics and traffic engineering patterns for Cloud WAN architectures with Direct Connect egress. Use this knowledge when advising on Cloud WAN route evaluation, DX Gateway behavior, BGP community usage, and regional failover design. + +## Reference Documentation + +- [Cloud WAN Route Evaluation](https://docs.aws.amazon.com/network-manager/latest/cloudwan/cloudwan-route-evaluation.html) +- [Cloud WAN Routing Policies](https://docs.aws.amazon.com/network-manager/latest/cloudwan/cloudwan-routing-policies.html) +- [Direct Connect Routing Policies and BGP Communities](https://docs.aws.amazon.com/directconnect/latest/UserGuide/routing-and-bgp.html) + +## Cloud WAN Route Evaluation Order (Per CNE) + +At each Core Network Edge, Cloud WAN evaluates routes in this order: + +1. **Most specific route** (longest prefix match) — absolute, wins before everything +2. For same-destination routes with different targets: + 1. Static routes + 2. VPC-propagated routes (same region) + 3. **Unequal AS-path length and/or MED** — shortest wins + 4. **Equal AS-path and MED** — preference order: + 1. Direct Connect Gateway-propagated routes + 2. Cloud WAN Connect (same region) + 3. Site-to-Site VPN (same region) + 4. Other sources (TGW peering, CNEs in other regions) — if identical from 2+ sources: **deterministically random** + +## DX Gateway Path Selection Mechanics + +### Key Behaviors + +1. **DXGW propagates only ONE path per prefix to each CNE route table** — it performs internal path selection and sends only the winner. CNEs do not see all available DX paths. + +2. **DXGW uses Local Preference to prefer local-region DX over remote regions** — if a DX VIF exists in the same associated AWS region as the target CNE, DXGW prefers it regardless of AS-path length. + + **⚠️ Always verify the Associated Region for each DX location** — some DX locations can associate to different regions than expected based on geography. For example, London-area locations associate to different regions: + - Equinix LD5 (Slough) → **eu-west-1** (Ireland) + - Digital Realty LHR20 (London) → **eu-west-1** (Ireland) + - Telehouse (London Docklands) → **eu-west-2** (London) + - Equinix MA3 (Manchester) → **eu-west-2** (London) + + The associated region determines which CNE/TGW considers the DX "local" for LP preference. Verify a location's associated region with the `DescribeLocations` API/CLI (`aws directconnect describe-locations`) or the official AWS Direct Connect locations reference: https://aws.amazon.com/directconnect/locations/ + +3. **Explicit LP communities override DXGW's default local-region preference** — applying `7224:7300/7200/7100` on advertised prefixes forces DXGW to honor the explicit LP before applying its local-vs-remote default. + +4. **LP communities are evaluated before AS-path** — per DX documentation: "Local preference BGP community tags are evaluated before any AS_PATH attribute." This applies at the DXGW path selection stage. + +5. **LP communities from DX VIFs are NOT visible as community tags in Cloud WAN** — the community tags themselves don't propagate into Cloud WAN routing policies. However, the LP value they set at the DXGW level influences which path gets propagated to the CNE route table. + +6. **DXGW ECMPs across equal remote paths** — when the local-region DX fails and multiple remote DX locations have equal attributes (same LP, same AS-path), DXGW will **ECMP across them** rather than picking one non-deterministically. This provides load-balanced failover across all remaining DX locations. + +7. **Route filtering on the native Cloud WAN DXGW attachment (Routing Policies)** — Cloud WAN Routing Policies support **prefix-based route filtering (drop) and summarization, inbound and outbound, on Direct Connect attachments**. This is a **separate, supported feature** from the (unsupported) DXGW allowed-prefixes list. Use it to drop an overlapping inbound supernet or to suppress specific VPC prefixes advertised outbound to on-prem. + - Match prefixes via `prefix-in-cidr` or `prefix-in-prefix-list`; actions are `allow`/`drop`; `routing-policy-direction` is `inbound` or `outbound`. + - **DX caveat:** BGP communities **cannot** be matched or set on Direct Connect attachments in Cloud WAN — prefix/CIDR matching only (consistent with point 5). Community-based routing policies work on other attachment types but not DX. + - Source: [Cloud WAN Routing Policies](https://docs.aws.amazon.com/network-manager/latest/cloudwan/cloudwan-routing-policies.html); the DXGW-attachment "no allowed-prefixes list" limitation ([DX gateway attachments in Cloud WAN](https://docs.aws.amazon.com/network-manager/latest/cloudwan/cloudwan-dxattach-about.html)) refers to the legacy allowed-prefixes construct, not routing-policy filtering. + +**Important distinction — path selection behavior by service:** +- **DXGW:** ECMPs across equal remote paths (load-balanced failover) +- **TGW:** Deterministic — selects oldest route for equal paths. Consistent but not customer-controllable. +- **Cloud WAN CNE:** Deterministic — "deterministically random" at Step 2.4.4 for equal remote CNE paths. Consistent but not customer-controllable. + +### Common Pitfall: AS-Path Prepending + Local-Region Preference + +**Problem pattern:** +- Customer prepends AS-path on a DX VIF that is local to a given CNE's region +- DXGW's local-region LP still prefers the local (prepended) path and propagates only that to the CNE +- The CNE route table now has a long-AS-path DX route vs. shorter remote CNE paths +- Step 2.3 (shortest AS-path) selects a remote CNE path → traffic leaves the region + +**Root cause:** DXGW's local-region preference overrides AS-path at the DXGW level, but at the CNE level the prepended path competes against shorter remote CNE paths where AS-path length IS evaluated. + +**Why only affected regions see the issue:** For regions where all DX locations are remote, DXGW falls back to AS-path to choose between them — prepending works as intended. The issue only occurs at CNEs where the prepended DX is in the same associated AWS region. + +### Common Pitfall: Confusing Prepend Count with Total AS-Path Length (Origination vs. Transit) + +When comparing AS-path lengths at a CNE (Step 2.3), the number that matters is the **total AS-path AWS receives on the VIF**, not the prepend count configured on a site. These are only the same when the advertising site is the **originator** of the prefix. + +**The distinction:** +- **Originating site** — the site owns/aggregates the prefix (e.g., `aggregate-address 11.0.0.0/8`). The AS-path AWS sees is just that site's ASN, repeated by its prepend count. A `prepend 60208 2` on an originated route → AS-path `60208 60208 60208` (length 3). +- **Transiting site** — the site does NOT originate the prefix; it **learns it from another site over the on-prem WAN** and re-advertises it to its DX. The AS-path already contains the origin's ASN, so the transiting site's ASN and prepends stack **on top of** the origin ASN. A `prepend 60208 2` applied to a `/8` learned from origin AS 56608 → AS-path `60208 60208 60208 56608` (length **4**, not 3). + +**Config signature to tell them apart:** A site that **matches-and-prepends** a prefix in a route-policy but has **no `aggregate-address`/origination** for it is almost certainly **transiting** a route learned from elsewhere. Do not assume it originates the prefix just because the prefix appears in its outbound policy. + +**Why this matters for path selection:** Step 2.3 compares total path length. If you model a transiting site's length as "prepend count + 1" (treating it as an originator) you will miscompute which copy wins — often flipping the predicted winner and the failover order. Example: three sites advertise a `/8` that **originates at only one** of them; the two that transit-and-prepend it are longer than a naive prepend-count model suggests, and can tie with the origin. + +**How to get the ground truth:** Never infer AS-path length from prepend config alone when a prefix may be transited. Read the literal path AWS receives per VIF: +``` +aws directconnect list-virtual-interface-routes --virtual-interface-id +``` +This shows the exact AS-path (and communities) AWS accepted, settling origination-vs-transit and the true length used at Step 2.3. + +## DX Routing Selection — General Best Practice + +- **Always use LP communities (`7224:7300/7200/7100`) for expected egress routing** — High Preference BGP communities are the recommended mechanism to achieve predictable DX egress behavior. +- **AS-path prepending: within a region ONLY** — AS-path prepending can be used to influence egress traffic between DX connections belonging to the **same associated AWS region** (e.g., two DX locations both in the same region). It should NOT be used to influence traffic across regions — DXGW's local-region LP behavior makes cross-region AS-path prepending unreliable. +- **BGP communities: within region OR across regions** — LP communities work for both intra-region path selection (between VIFs in the same region) and cross-region preference (overriding DXGW's default local-region LP to prefer a remote DX location globally). + +| Scope | Recommended Mechanism | Why | +|---|---|---| +| Cross-region DX preference | LP Communities (`7224:7300/7200/7100`) | Overrides DXGW local-region LP; evaluated before AS-path | +| Within-region DX preference (same associated region) | AS-path prepending OR LP communities | Both work; AS-path provides granular control between same-region VIFs | +| Never use for cross-region | AS-path prepending alone | DXGW local-region LP overrides AS-path at the DXGW level, making prepending unreliable across regions | + +## Traffic Engineering Patterns + +### Pattern 1: Global DX Preference (Single Primary) + +**Use case:** One DX location should be globally preferred for a prefix, with another as backup. + +**Solution:** Apply tiered LP communities from on-premises routers: + +| DX Location | Community | Role | +|---|---|---| +| Primary DX location | `7224:7300` | HIGH — globally preferred | +| Secondary DX location | `7224:7200` | MEDIUM — failover | +| All other DX locations | `7224:7100` | LOW — last resort | + +**Result:** DXGW selects the primary for ALL CNEs (overriding local-region preference). If the primary fails, the medium-priority location (`7224:7200`) takes over automatically. (Note: "medium" here refers to the medium local-preference community, not the BGP MED attribute.) + +**Tradeoff:** Regions with a local DX that isn't the primary will cross the AWS backbone to reach the preferred DX. Traffic still traverses the local-region firewall (service insertion happens at segment level, before the CNE egress decision). + +### Pattern 2: Per-Region DX Preference (Split DXGW) + +**Use case:** Different regions need different primary DX locations (e.g., one region group prefers DX-A, another region group prefers DX-B). + +**Problem:** Communities from a single DXGW are global — cannot provide per-region preferences. + +**Solution:** Create separate DXGWs per region group, each with their own VIFs: +- DXGW-A: Region-A DX locations (primary `7224:7300`, secondary `7224:7200`) +- DXGW-B: Region-B DX locations (`7224:7300`) + +Both DXGWs propagate their path to all CNEs. The CNE route table evaluates them at Step 2.3 (AS-path length). Control which DXGW wins per region by differentiating AS-path length on the VIFs between the two DXGWs. + +Use communities within each DXGW for internal failover. No Cloud WAN Routing Policies needed — VIF-level BGP attributes provide per-region control at the CNE route table. + +### Pattern 3: Per-Region Control via More Specific Routes + +**Use case:** On-prem address space is regionally segmented. + +**Solution:** Advertise more-specific prefixes from the corresponding DX location: +- DX location A advertises a more-specific covering Region-A on-prem hosts +- DX location B advertises a more-specific covering Region-B on-prem hosts +- All locations advertise the aggregate as a failover catch-all + +**Result:** Step 1 (longest prefix match) is absolute — each CNE route table forwards to the correct DX based on destination address. + +**Requirement:** Only works if on-prem address space is subdivided into regional ranges. Does NOT work for flat supernets. + +## Multi-Region Centralized Internet Egress + +### Non-Local Exit Point Selection + +In architectures where internet egress is centralized (e.g., a single region hosts the NAT/internet gateway or firewall for outbound traffic), Cloud WAN must route traffic from remote regions to the centralized exit point. Understanding how non-local exit points are selected is critical: + +- Each CNE evaluates routes per the standard route evaluation order +- If the default route (0.0.0.0/0) or internet-bound prefix is only advertised from the centralized region's attachment (e.g., via TGW Connect or VPN), all CNEs will install that path +- Remote CNEs forward internet-bound traffic across the AWS backbone to the centralized region's CNE + +### AS-Path Prepending for Deterministic Exit (via TGW/VPN) + +When routes are propagated from DX or VPN **via TGW** to Cloud WAN (TGW peering attachment), AS-path length information is preserved. This means: + +- **AS-path prepending can define a deterministic exit path** — if multiple regions advertise the same prefix via TGW attachments to Cloud WAN, prepending on less-preferred paths ensures remote CNEs choose the shorter AS-path (preferred exit) +- **More-specific routing also applies** — advertising a more-specific prefix from the preferred exit region wins at Step 1 (longest prefix match) regardless of AS-path + +**Note:** This behavior differs from DX-attached prefixes where DXGW's local-region LP can override AS-path. When routes come via TGW attachments to Cloud WAN, AS-path evaluation at the CNE operates without DXGW LP interference — making prepending a valid and effective traffic engineering tool in this context. + +### Example: Centralized Internet Egress + +- Region-A TGW advertises `0.0.0.0/0` to Cloud WAN via TGW peering (short AS-path) +- Region-B TGW also advertises `0.0.0.0/0` but with prepending (longer AS-path) +- All CNEs prefer Region-A as the internet exit (Step 2.3: shorter AS-path wins) +- If Region-A fails, Region-B's longer path takes over as backup + +## Regional Firewall Inspection + +- Cloud WAN segment policy / service insertion routes traffic through the **same-region firewall** before it reaches the CNE +- Traffic flow: VPC → local firewall (service insertion) → returns to local CNE → CNE route table → DXGW → DX +- The DX community fix ensures the DXGW path wins at the local CNE route table (2 hops < 3 hops from remote CNEs) — traffic stays at the local CNE for egress, preserving the local inspection chain +- If a remote CNE path wins instead, traffic leaves the local region at the CNE level, potentially bypassing local firewall inspection +- **Requirement:** Firewall VPCs must exist in every region with workloads + +## Directional Control + +| Direction | Controlled By | Mechanism | +|---|---|---| +| AWS → On-prem (egress) | DX BGP communities (`7224:7300/7200/7100`) | Sets LP at DXGW, determines which path reaches CNE route tables | +| On-prem → AWS (ingress) | Customer router policies | Local-pref, weight, MED on on-prem routers — independent of Cloud WAN | + +Communities do NOT affect the on-prem → AWS direction. If prepending was used for on-prem path selection, replace with router-side local-pref/weight before removing prepends. + +## Cleanup Recommendations + +- Removing AS-path prepending is recommended once communities are in place — communities fix the problem immediately, prepend removal is a cleanup step +- Prepending is counterproductive in Cloud WAN because prefixes from DX VIFs are shared across all CNEs via DXGW — the inflated AS-path creates unintended path selection at remote CNEs +- AS-path equalization across DX locations (without communities) achieves "prefer local DX" behavior but does not provide explicit failover ordering between regions diff --git a/skills/aws-routing/references/dx-routing-patterns.md b/skills/aws-routing/references/dx-routing-patterns.md new file mode 100644 index 0000000..a481612 --- /dev/null +++ b/skills/aws-routing/references/dx-routing-patterns.md @@ -0,0 +1,217 @@ +--- +inclusion: manual +--- + +# Direct Connect Routing Patterns & Considerations + +This document captures key BGP routing mechanics, path selection behavior, VIF-specific considerations, and operational caveats for AWS Direct Connect architectures. Use this knowledge when advising on DX design (active/active, active/passive), BGP community behavior, BFD configuration, and public/private/transit VIF routing. + +## Reference Documentation + +- [Direct Connect Routing Policies and BGP Communities](https://docs.aws.amazon.com/directconnect/latest/UserGuide/routing-and-bgp.html) +- [Direct Connect Path Selection (routing policies for private/transit VIFs)](https://docs.aws.amazon.com/directconnect/latest/UserGuide/routing-and-bgp.html) +- [Direct Connect Virtual Interfaces](https://docs.aws.amazon.com/directconnect/latest/UserGuide/WorkingWithVirtualInterfaces.html) + +## BGP & Routing + +### BGP Route Visibility (ListVirtualInterfaceRoutes) + +As of July 2026, DX supports **BGP route visibility** on virtual interfaces via the `ListVirtualInterfaceRoutes` API and the DX console: + +- **Accepted routes:** Routes AWS received from your on-prem router +- **Advertised routes:** Routes AWS is sending to your on-prem router +- Each route shows: prefix, address family, AS-path, community values, installation timestamp +- Can filter by: prefix, AS-path, community, or address family +- Available on: private, transit, and public VIFs +- Available in: all AWS commercial regions + +**Use for troubleshooting:** +- Verify route propagation between on-prem and AWS +- Validate BGP policy configurations (communities, AS-path prepending) +- Diagnose unexpected traffic patterns or asymmetric routing +- Confirm which routes DXGW is accepting vs. advertising +- Validate LP community tags are being received correctly + +**Reference:** [AWS Direct Connect BGP Route Visibility announcement](https://aws.amazon.com/about-aws/whats-new/2026/07/aws-direct-connect-bgp-visibility/) + +### CloudWatch BGP Metrics for DX Virtual Interfaces + +As of March 2026, DX publishes **3 BGP metrics** to CloudWatch for all VIF types (private, transit, public): + +| Metric | What It Reports | +|---|---| +| `VirtualInterfaceBgpStatus` | BGP session state: 1 = up, 0 = down | +| `VirtualInterfaceBgpPrefixesAccepted` | Number of prefixes AWS accepted from on-prem | +| `VirtualInterfaceBgpPrefixesAdvertised` | Number of prefixes AWS advertises to on-prem | + +**Key details:** +- Updated every 5 minutes (BGP flaps between intervals may not be captured) +- Published at no extra cost in the region where the DX location is associated +- Available via CloudWatch MCP server (`get_metric_data`) for programmatic access + +**Use cases:** +- **BGP session monitoring:** Alarm on `VirtualInterfaceBgpStatus < 1` to detect session failures without API polling +- **Prefix limit protection:** Alarm when `PrefixesAccepted` approaches VIF quota to prevent session idle state +- **Silent route withdrawal detection:** BGP session stays UP but prefix count drops → indicates routes withdrawn without session going down +- **DR/failover validation:** Compare prefix counts between primary and standby VIFs — a standby with fewer prefixes indicates a broken failover path +- **Route change correlation:** Sudden drop in accepted/advertised prefixes correlates with traffic impact events + +**Alarm examples:** +- `VirtualInterfaceBgpStatus` — Statistic: Minimum, Period: 5min, Threshold: Lower than 1 +- `VirtualInterfaceBgpPrefixesAccepted` — Statistic: Maximum, Period: 5min, Threshold: Greater/Equal to (near quota limit) +- `VirtualInterfaceBgpPrefixesAccepted` — Statistic: Maximum, Period: 5min, Threshold: Lower/Equal to (below expected minimum) + +**Reference:** [CloudWatch Metrics for DX VIF BGP Health and Prefix Count](https://aws.amazon.com/blogs/networking-and-content-delivery/introducing-cloudwatch-metrics-for-aws-direct-connect-virtual-interface-bgp-health-and-prefix-count/) + +### General Routing Behavior + +- When designing active/active or active/passive DX (within or across DX locations), apply the documented **Private/Transit VIF outbound path-selection order**: longest prefix match → local preference → AS_PATH length → MED → ECMP (equal AS_PATH length and BGP attributes). See [Direct Connect routing policies and BGP communities](https://docs.aws.amazon.com/directconnect/latest/UserGuide/routing-and-bgp.html). +- Consider recommendations for **migrating from DX + VGW/Private VIF to DX + TGW/Transit VIF** when customers are modernizing their architecture + +### MED Support + +- DX **supports Multi-Exit Discriminator (MED) on Private and Transit VIFs** — MED can influence the outbound (AWS → on-prem) routing path, but only as a **tiebreaker after prefix length, local preference, and AS_PATH are equal**. AWS explicitly **does not recommend relying on MED** given its lower priority in the evaluation order. +- For **Public VIFs**, the documented outbound routing policy uses **AS_PATH and longest prefix match** — MED is not listed among the supported public VIF path-selection attributes. +- **Outbound (AWS → on-prem) path selection order for Private/Transit VIF:** longest prefix match → local preference → AS_PATH length → MED → ECMP (equal AS_PATH length and BGP attributes). + +**Reference:** [Direct Connect routing policies and BGP communities](https://docs.aws.amazon.com/directconnect/latest/UserGuide/routing-and-bgp.html) + +### BFD (Bidirectional Forwarding Detection) + +**⚠️ Always verify the Associated Region for each DX location** — some DX locations can associate to different AWS regions than expected based on their physical geography. For example: +- Equinix LD5 (Slough, UK) → **eu-west-1** (Ireland) +- Telehouse (London Docklands, UK) → **eu-west-2** (London) +- Equinix MA3 (Manchester, UK) → **eu-west-2** (London) + +The associated region determines which TGW/DXGW considers the DX "local" for LP preference and routing behavior. Verify a location's associated region with the `DescribeLocations` API/CLI (`aws directconnect describe-locations`) or the official AWS Direct Connect locations reference: https://aws.amazon.com/directconnect/locations/ + +### BFD (Bidirectional Forwarding Detection) + +- **Asynchronous BFD is automatically enabled** on the AWS side for all DX virtual interfaces +- Customer must configure asynchronous BFD on their router (refer to vendor documentation) +- BFD provides fast link failure detection — ensures BGP neighbor relationship is torn down quickly for faster failover +- **Do NOT configure both BFD for BGP and BGP Graceful Restart simultaneously** — this results in sub-optimal routing performance (conflicting goals: BFD wants fast teardown, Graceful Restart wants to preserve routes during failure) + +### ASN Support + +| Context | Supported ASN Range | +|---|---| +| AWS side via DX-GW | Full 32-bit: 4,200,000,000 to 4,294,967,294 (private) | +| AWS side via VGW | 32-bit: 1 to 2,147,483,647 only | +| Customer side (Private/Public/Transit VIF) | 32-bit: 1 to 2,147,483,647 only | + +- Customers can request a Private/Transit VIF ASN from the 4,200,000,000–4,294,967,294 range via a support case (non-standard, requires internal approval) + +### SiteLink & BGP Communities + +- SiteLink **maintains BGP community tags** — tags are passed through from one connected peer to another +- This means communities set on one DX location will be visible at another DX location connected via SiteLink + +## VIF-Specific Considerations + +### Public VIF + +- **Reverse-path forwarding (RPF) strict mode is enforced** — validates that the source of traffic originated from the customer's advertised prefix (prevents address spoofing) +- **Do NOT use EC2 EIPs for Public VIF BGP peering** — can cause routing issues. Customers should use: + - Public IPs they own, OR + - IPs assigned by their provider/AWS support +- **EC2 BYOIP CIDRs are NOT in ip-ranges.json** (not part of Amazon address space), but they ARE advertised over DX Public VIF to customers +- **S3-only Public VIF**: BGP prefix lists are available (GitHub repository) for filtering S3 traffic over DX Public VIF — allows routing only S3 traffic through DX while sending other AWS traffic over internet to reduce costs + +### Private / Transit VIF + +- For Private/Transit VIFs, customers can use any IPv4 address (if not using AWS auto-generated IPs), but must ensure those IPs **do not conflict** with on-prem and/or VPC networks +- When attaching VPCs with **AWS-assigned IPv6** to a TGW, these are discontinuous by default and **cannot be aggregated** for Direct Connect — leads to DX prefix quota limitations + - **Workarounds:** Use Amazon-provided contiguous IPv6 CIDR blocks, or EC2 BYOIP + +### VPC-to-VPC Routing via DXGW (Regression) + +- As of November 2021, **VPC-to-VPC routing** connected to the same DXGW via VGW/TGW which are also connected over Private/Transit VIF **is possible due to a regression** +- This was previously blocked — be aware of this change +- Workarounds exist to block this unintended traffic flow if isolation is required + +### DX VIFs and VGW — CloudHub Behavior + +- **VPN CloudHub is a Site-to-Site VPN feature**: multiple VPN connections terminating on the same VGW can exchange routes with each other, letting remote sites communicate through AWS. See [AWS VPN CloudHub](https://docs.aws.amazon.com/vpc/latest/userguide/vpn-connections.html). +- **DX can participate in a CloudHub with VPN**: when a DX private VIF and one or more VPN connections share the same VGW, prefixes learned from a VPN customer gateway are advertised to the DX customer gateway and vice versa — so a DX site and VPN site(s) on the same VGW can reach each other. +- **DX-to-DX does NOT hub through a VGW**: if two DX VIFs from different on-prem sites terminate on the same VGW (or DX Gateway), AWS does not re-advertise one site's on-prem prefixes to the other. AWS advertises only the VPC CIDR(s) toward each VIF, so the two on-prem sites cannot reach each other through AWS. Site-to-site transit requires the customer's own router (hairpin) or a design that isn't a single shared gateway. +- **DX Gateway (VGW association) advertises only VPC CIDRs to on-prem** — it does not provide CloudHub between gateway associations; associations on the same DXGW cannot send traffic to each other. +- **Floating VGW (VGW not attached to a VPC):** + - *With VPN* — a floating VGW can still act as a VPN CloudHub: multiple VPN connections on it exchange routes, so remote sites reach each other through AWS with no VPC involved. + - *With DXGW* — not a usable scenario. A DXGW advertises associated VPC CIDRs to on-prem; a floating VGW has no VPC, so there is nothing to advertise. DXGW-over-VGW is only meaningful when the VGW is attached to a VPC. + +**Note:** This corrects an earlier claim that DX VIFs on a shared VGW form a DX-to-DX hub-and-spoke mesh. Route sharing between on-prem sites on a shared gateway is a VPN CloudHub behavior (VPN connections), not a DX-to-DX behavior. Validate specific flows in a lab before relying on them. + +### When Can VIFs Talk to Each Other? (Site-to-Site via AWS) + +**Governing principle:** AWS does not act as a transit hub between VIFs by default. Over a VGW or a single DXGW, AWS advertises only the **VPC CIDRs** toward each VIF, so two on-prem sites (each on its own VIF) do not learn each other's prefixes unless something explicitly enables it. + +**Supported — VIFs (sites) CAN reach each other:** + +| Scenario | How it works | +|---|---| +| VPN CloudHub on a VGW | Multiple VPN connections on the same VGW exchange routes; remote sites reach each other. See [AWS VPN CloudHub](https://docs.aws.amazon.com/vpc/latest/userguide/vpn-connections.html). | +| DX + VPN on the same VGW | A DX private VIF sharing a VGW with VPN connection(s) can talk to the VPN site(s) — DX ↔ VPN. | +| TGW + Allowed Prefix List | The DXGW→TGW attachment advertises whatever is in the Allowed Prefix List (no VPC-CIDR-only restriction), so remote-site prefixes can be advertised intentionally. Enables DX ↔ VPN CloudHub via TGW. | +| DX ↔ DX via two DXGWs into one TGW | Two DXGWs attached to the same TGW (CGW1–DXGW1–TGW–DXGW2–CGW2) allows DX site-to-site. | +| Customer-side hairpin | Always available: advertise a supernet/default from on-prem and let the customer router be the hub. AWS is not the transit point. | + +**Not supported — sites CANNOT reach each other through AWS:** + +| Scenario | Why | +|---|---| +| Two DX VIFs on the same VGW | AWS advertises only VPC CIDRs to each VIF; no site-to-site. | +| Two DX VIFs on the same single DXGW | Gateway associations on one DXGW cannot send traffic to each other. | +| Inter-site over a single DXGW with transit VIFs | Routes may install but the flow loops; not supported. Use two DXGWs. | +| Inter-site via two DXGWs with VGW associations | Only VPC CIDRs are advertised, so the sites never learn each other. | + +**Summary:** +- **VGW / single DXGW** → advertises VPC CIDRs only → no VIF-to-VIF transit (except true VPN CloudHub, which is a VPN feature). +- **TGW** → Allowed Prefix List with no route validation → VIF-to-VIF transit can be built intentionally, using two DXGWs for DX-to-DX. + +## DX vs Peering (Public VIF vs PNI) + +### Key Differences + +| Attribute | DX Public VIF | Public/Private Peering (PNI) | +|---|---|---| +| Official AWS service | Yes | No | +| SLA | Yes | No | +| AWS Support coverage | Yes | No (handled by Global Interconnect Strategy team) | +| Internal LOCAL_PREF | 3000 | 2500 | +| Path preference | **Preferred** (higher LP) | Lower priority | + +### Important Guidance + +- If the **same prefix** is received by AWS over both DX Public VIF and Peering (PNI, IXP, or Transit), **DX Public VIF is the preferred path** (LOCAL_PREF 3000 > 2500) +- Public Peering / PNI is NOT an official AWS service — no SLA, no support expectation from the technical field +- **Steer customers toward DX** when they need reliability and support coverage +- There is no expectation for the technical field to support Public Peering / PNI engagements + +## Design Considerations for Active/Active & Active/Passive + +### General Best Practice for DX Routing Selection + +- **Always use LP communities (`7224:7300/7200/7100`) for expected egress routing** — High Preference BGP communities are the recommended mechanism for predictable DX egress. +- **AS-path prepending: within a region ONLY** — Use AS-path prepending to influence egress between DX connections belonging to the **same associated AWS region**. Do NOT use AS-path prepending to influence traffic across regions — DXGW's local-region LP makes cross-region prepending unreliable. + - **When comparing AS-path lengths, distinguish an originating site from a transiting one** — a site that learns a prefix over the on-prem WAN and re-advertises it carries the origin ASN in its path, so its total length is prepend-count + origin ASN(s), not prepend-count alone. Confirm the real path with `list-virtual-interface-routes`. See "Confusing Prepend Count with Total AS-Path Length" in cloudwan-dx-routing-patterns.md. +- **BGP communities: within region OR across regions** — LP communities work for both intra-region path selection and cross-region preference (overriding DXGW's default local-region LP). + +### Active/Active (within or across DX locations) + +- Understand how AWS selects between multiple active paths using the Private/Transit VIF outbound path-selection order (longest prefix match → local preference → AS_PATH length → MED → ECMP). See [Direct Connect routing policies and BGP communities](https://docs.aws.amazon.com/directconnect/latest/UserGuide/routing-and-bgp.html). +- ECMP is available at the TGW level across multiple DX-GW attachments (same or different DX-GWs) +- LP communities (`7224:7300/7200/7100`) can influence preference but behavior varies between VGW/TGW/Cloud WAN — always test +- **DXGW ECMPs across equal remote paths** — when a local DX fails and multiple remote DX locations have matching attributes (same LP, same AS-path), DXGW will ECMP across them. This provides load-balanced failover rather than single-path selection. + +**Path selection behavior by service (NOT at DXGW):** +- **DXGW:** ECMPs across equal paths — load-balanced failover +- **TGW:** Deterministic — selects oldest route for equal paths. Consistent but not customer-controllable. +- **Cloud WAN CNE:** Deterministic — "deterministically random" at Step 2.4.4 for equal remote CNE paths. Consistent but not customer-controllable. + +### Active/Passive + +- Use LP communities or AS-path prepending to create primary/secondary paths +- For Cloud WAN: communities override DXGW local-region preference (see cloudwan-dx-routing-patterns.md) +- For TGW: LP communities are NOT honored across multiple DX-GW attachments (see tgw-routing-patterns.md) +- BFD should be enabled for fast failure detection on the active path diff --git a/skills/aws-routing/references/networking-strategic-questions.md b/skills/aws-routing/references/networking-strategic-questions.md new file mode 100644 index 0000000..df9daf2 --- /dev/null +++ b/skills/aws-routing/references/networking-strategic-questions.md @@ -0,0 +1,584 @@ +--- +inclusion: manual +--- + +# AWS Networking Infrastructure Strategic Questions + +Use these questions to guide discovery and architecture discussions for each AWS networking service. These ensure proper planning and successful implementation. + +## Cloud WAN (CWAN) + +### Network Architecture +1. How many AWS regions need to be connected? +2. What's your global network topology requirement? +3. How many VPCs and on-premises sites need connectivity? +4. Do you need network segmentation across your global network? +5. What's your branch office connectivity strategy? +6. How will you organize your network into segments? +7. What's your multi-account strategy? + +### Security & Routing +1. What network segments require isolation? +2. How will you implement security policies across regions? +3. What's your routing policy strategy? +4. Do you need service insertion for traffic inspection? +5. How will you control inter-segment communication? +6. What's your attachment acceptance strategy? +7. What route filtering is required? + +### Performance & Scale +1. What's your expected bandwidth per region? +2. How many attachments do you anticipate globally? +3. What are your latency requirements between regions? +4. Do you need optimized routing for performance? +5. What's your expected growth over 3-5 years? +6. How will you handle traffic prioritization? +7. What's your packet per second requirement? + +### Cost Management +1. How will you optimize core network edge costs? +2. What's your expected data transfer volume? +3. Do you need multiple core networks for cost allocation? +4. How will you track usage per business unit or region? +5. What's your attachment strategy to minimize costs? +6. How will you balance cost vs. performance? +7. What's your budget for global connectivity? + +### Operational +1. Who will manage Cloud WAN configurations? +2. What's your network monitoring strategy? +3. How will you handle policy changes? +4. What's your change management process? +5. How will you troubleshoot cross-region issues? +6. What automation is needed for network provisioning? +7. How will you manage network policies as code? + +### Integration +1. Do you need SD-WAN integration? +2. What third-party network devices require connectivity? +3. How will you integrate with existing Transit Gateways? +4. Do you need Direct Connect or VPN attachments? +5. What's your hybrid cloud connectivity strategy? +6. How will you handle migration from existing solutions? +7. What network management tools will you integrate? + +--- + +## Direct Connect (DX) + +### Technical Requirements +1. What bandwidth do you need? (1Gbps, 10Gbps, 100Gbps) +2. How many VPCs need to be connected? +3. Do you require redundancy/high availability? +4. Which AWS regions need connectivity? +5. Will you need public or private virtual interfaces (VIFs), or both? + +### Network Architecture +1. What's your existing network topology? +2. Do you need BGP support? +3. Which ASN (Autonomous System Number) will you use? +4. Do you have compatible router equipment? +5. What are your IP addressing requirements? + +### Business & Compliance +1. What are your latency requirements? +2. Do you have any compliance/regulatory requirements? +3. Do you require traffic segmentation at the VIF level? +4. What's your disaster recovery strategy? +5. What's your timeline for implementation? +6. Who will manage the Direct Connect connection? (In-house or partner?) + +### Location & Physical +1. Which Direct Connect location is closest to your infrastructure? +2. Do you need a Direct Connect Partner to provide last-mile connectivity? +3. Do you have cross-connect capabilities in your facility? + +### Security +1. What encryption requirements do you have? +2. Do you need AWS Direct Connect Gateway? +3. What are your network security policies? +4. Who needs access to manage the connection? + +### Cost Considerations +1. What's your budget for port hours and data transfer? +2. Do you need committed bandwidth? +3. How long do you need to commit to the connection? + +--- + +## Transit Gateway (TGW) + +### Network Architecture +1. How many VPCs need to be connected? +2. What's your cross-region connectivity requirement? +3. Do you need VPN or Direct Connect attachments? +4. What's your routing domain strategy? +5. How will you segment traffic between environments? + +### Security & Routing +1. What route table isolation is needed? +2. Do you need inspection architecture integration? +3. What's your traffic flow pattern (hub-spoke, mesh)? +4. How will you implement network segmentation? +5. What's your multicast requirement? + +### Performance & Scale +1. What's your expected bandwidth per attachment? +2. How many attachments do you anticipate? +3. What are your latency requirements? +4. Do you need inter-region peering? +5. What's your packet per second requirement? + +### Cost Management +1. How will you optimize attachment costs? +2. What's your data transfer pattern? +3. Do you need multiple TGWs for cost allocation? +4. How will you track usage per business unit? + +### Operational +1. Who will manage TGW configurations? +2. What's your monitoring strategy? +3. How will you handle route propagation? +4. What's your change management process? + +--- + +## VPC + +### Network Architecture +1. What is your total IP address space requirement? (CIDR block) +2. How many subnets do you need across all environments? +3. What's your public/private subnet strategy? +4. How many Availability Zones will you use? +5. Do you need connectivity to existing networks? +6. Are you considering a cellular architecture approach? +7. What's your subnet sizing strategy for future growth? + +### Connectivity Requirements +1. What type of internet access is needed? +2. Will you implement VPN or Direct Connect? +3. Do you need VPC peering or Transit Gateway? +4. Which AWS services require VPC endpoints? +5. How many NAT Gateways are required? +6. What's your expected bandwidth consumption? +7. Do you need cross-region connectivity? + +### Security Architecture +1. What are your baseline security group requirements? +2. What Network ACL policies are needed? +3. What compliance standards must be met? +4. Which traffic patterns must be controlled? +5. Do you need VPC flow logs? +6. What's your network isolation strategy? +7. How will you secure cross-VPC communication? +8. Do you require traffic inspection? + +### Application Requirements +1. What AWS resources will be deployed? +2. What's your load balancer strategy? +3. Are you using containers or serverless? +4. What's the expected traffic volume? +5. Do you need service discovery? +6. What's your application tier separation plan? +7. How will applications communicate between tiers? + +### Cost Management +1. What's your NAT Gateway deployment strategy? +2. Which VPC endpoints will provide cost benefits? +3. Do workloads require dedicated tenancy? +4. How will you track costs per environment? +5. What's your resource tagging strategy? +6. How will you optimize data transfer costs? +7. What's your reserved capacity strategy? + +### DNS and Routing +1. Will you use AWS or custom DNS? +2. What DHCP options are needed? +3. What's your DNS resolution strategy? +4. How will you manage route tables? +5. Do you need private hosted zones? +6. What's your domain naming strategy? +7. How will you handle cross-VPC DNS? + +### Operational Considerations +1. Who owns VPC management? +2. What monitoring solutions are needed? +3. How will you implement backup and DR? +4. What's your change management process? +5. How will you handle maintenance? +6. What logging requirements exist? +7. How will you manage configurations? + +### Growth and Scaling +1. What's your CIDR expansion strategy? +2. How will you accommodate new workloads? +3. What's your multi-account strategy? +4. How will you handle mergers/acquisitions? +5. What's your resource scaling approach? +6. How will you manage capacity? +7. What's your environment promotion strategy? + +### Network Performance Monitoring +1. What are your baseline performance requirements? +2. Which network paths require active monitoring? +3. What are your critical alerting thresholds? +4. How will you monitor VPC flow logs? +5. What tools will you use for network visibility? +6. How will you track cross-AZ traffic patterns? +7. What's your packet loss and latency tolerance? +8. How will you monitor resource saturation? +9. What's your strategy for troubleshooting network issues? +10. How will you validate security group effectiveness? + +--- + +## VPN + +### Connection Requirements +1. Do you need Site-to-Site VPN or Client VPN? +2. What's your required bandwidth capacity? +3. How many site connections are needed? +4. Do you require static or dynamic routing? +5. What's your high availability requirement? + +### Technical Specifications +1. What type of customer gateway device will you use? +2. Which routing protocol - static or BGP? +3. What IP ranges need to be routed? +4. Do you need tunnel redundancy? +5. What's your IKE/IPSec configuration? + +### Security +1. What encryption requirements do you have? +2. How will you manage security certificates? +3. What's your key rotation strategy? +4. Which authentication methods will you use? +5. What's your access control strategy? + +### Operational +1. Who will manage the VPN connections? +2. What's your monitoring approach? +3. How will you handle failover? +4. What's your troubleshooting process? +5. How will you manage configuration changes? + +### Cost Considerations +1. How many VPN connections needed? +2. What's your expected data transfer volume? +3. Do you need accelerated VPN? +4. Will you need multiple transit gateways? + +### Performance +1. What's your latency requirement? +2. Do you need guaranteed bandwidth? +3. What's your packet loss tolerance? +4. How will you monitor throughput? +5. Do you need AWS Accelerated VPN? + +--- + +## Elastic Load Balancing (ALB, NLB, GWLB) + +### Architecture & Type Selection +1. What type of traffic are you load balancing? (HTTP/HTTPS, TCP/UDP, third-party appliances) +2. Do you need Layer 7 (content-based) routing or Layer 4 (connection-based)? +3. Is this internet-facing or internal? +4. How many target groups and backends are expected? +5. Do you need cross-zone load balancing? +6. What's your multi-AZ deployment strategy? +7. Do you need Gateway Load Balancer for inline inspection (third-party appliances)? + +### Performance & Scale +1. What's your expected requests per second (RPS)? +2. What's your peak traffic pattern? (steady, spiky, seasonal) +3. Do you need pre-warming for anticipated traffic spikes? +4. What are your connection timeout requirements? +5. What's your expected connection count (concurrent)? +6. Do you need WebSocket or gRPC support? +7. What's your TLS termination strategy? + +### Health Checks & Failover +1. What health check criteria define a healthy target? +2. What are your acceptable health check intervals and thresholds? +3. Do you need cross-region failover (Route 53 + multi-region LBs)? +4. How will you handle target deregistration and connection draining? +5. What's your strategy for handling AZ impairments? + +### Security +1. What TLS/SSL certificate strategy will you use? (ACM, imported) +2. Do you need mutual TLS (mTLS)? +3. What security policies (TLS versions, ciphers) are required? +4. Do you need WAF integration? +5. What access logging requirements exist? +6. How will you restrict access? (Security groups, prefix lists) + +### Cost Management +1. What's your expected LCU/NLCU consumption? +2. How will you optimize idle load balancers? +3. Do you need separate LBs per environment or shared? +4. What's your data processing volume estimate? + +### Operational +1. Who manages load balancer configurations? +2. What's your access log analysis strategy? +3. How will you monitor 4xx/5xx error rates? +4. What's your certificate rotation process? +5. How will you handle load balancer scaling events? + +--- + +## AWS Network Firewall + +### Architecture +1. Where in the traffic flow does the firewall need to inspect? (ingress, egress, east-west) +2. How many VPCs/regions require firewall inspection? +3. Will you use a centralized inspection VPC or distributed per-VPC firewalls? +4. How will you integrate with Transit Gateway or Cloud WAN service insertion? +5. Do you need symmetric routing for stateful inspection? +6. What's your multi-AZ deployment strategy for firewall endpoints? + +### Rule Management +1. What stateless rules are needed? (5-tuple filtering) +2. What stateful rules are needed? (domain filtering, IPS/IDS, protocol detection) +3. Do you need Suricata-compatible rule sets? +4. Will you use managed rule groups (AWS or third-party)? +5. How will you organize rules into rule groups? +6. What's your rule ordering strategy? (strict vs default) +7. How will you handle rule updates and versioning? + +### Traffic Patterns +1. What traffic requires inspection? (all, internet-bound only, east-west only) +2. What domains or IPs need to be allowed/denied? +3. Do you need TLS inspection (decryption)? +4. What protocols beyond HTTP/HTTPS need inspection? +5. How will you handle encrypted traffic you cannot inspect? + +### Logging & Monitoring +1. What logging destinations will you use? (S3, CloudWatch, Kinesis) +2. Do you need alert-level logs vs. flow-level logs? +3. What SIEM integration is required? +4. How will you monitor firewall capacity and performance? +5. What alerting thresholds are needed for dropped/rejected traffic? + +### Performance & Scale +1. What's your expected throughput per AZ? +2. How many concurrent connections do you anticipate? +3. What are your latency requirements for inspected traffic? +4. How will you handle traffic spikes? +5. Do you have any bandwidth-intensive workloads that should bypass inspection? + +### Cost Management +1. What's your expected data processing volume? +2. Can you reduce costs by inspecting only specific traffic flows? +3. How will you track firewall costs per business unit? +4. What's your strategy for balancing security coverage vs. cost? + +--- + +## AWS PrivateLink + +### Connectivity Requirements +1. Are you a service provider, consumer, or both? +2. What services need to be exposed privately? (your own, third-party, AWS) +3. How many VPCs need access to the endpoint service? +4. Do you need cross-account access? +5. Do you need cross-region access? +6. What's your multi-account/multi-VPC consumption pattern? + +### Architecture +1. Will you use interface endpoints (PrivateLink) or gateway endpoints (S3/DynamoDB)? +2. What's your NLB/GWLB backend for the endpoint service? +3. How will you handle DNS resolution for the endpoint? (Private hosted zone, endpoint-specific DNS) +4. Do you need endpoint policies for fine-grained access control? +5. What's your subnet placement strategy for endpoints? +6. How many AZs need endpoint availability? + +### Security +1. What endpoint policies are needed to restrict access? +2. How will you control who can create connections to your service? +3. Do you need manual acceptance of connection requests? +4. What security groups will you apply to interface endpoints? +5. How will you audit endpoint usage? + +### Performance & Scale +1. How many concurrent connections do you expect per endpoint? +2. What's your bandwidth requirement per AZ? +3. Do you need cross-AZ load balancing at the endpoint level? +4. What are your latency requirements? + +### Cost Management +1. How many endpoints and endpoint services do you need? +2. What's your expected data processing volume? +3. Can you consolidate endpoints across services? +4. How will you track PrivateLink costs per team/service? + +### Operational +1. Who manages endpoint services vs. endpoint consumers? +2. How will you handle endpoint service versioning/migration? +3. What's your monitoring strategy for endpoint health? +4. How will you troubleshoot connectivity through endpoints? + +--- + +## Route 53 (DNS) + +### DNS Architecture +1. Will you use public hosted zones, private hosted zones, or both? +2. How many domains and subdomains do you manage? +3. Do you need split-horizon DNS (different answers for internal vs. external)? +4. What's your cross-account DNS strategy? (centralized vs. distributed) +5. How will VPCs resolve private hosted zones? (associations, RAM sharing) +6. Do you need Route 53 Profiles for multi-VPC/multi-account DNS management? + +### Resolver & Hybrid DNS +1. Do you need Route 53 Resolver for hybrid DNS (on-prem ↔ AWS)? +2. What on-premises DNS servers need to resolve AWS resources? +3. What AWS resources need to resolve on-prem domains? +4. How many Resolver inbound/outbound endpoints do you need? +5. What forwarding rules are required? +6. Do you need Resolver DNS Firewall for domain filtering? + +### Routing Policies +1. What routing policy do you need? (simple, weighted, latency, geolocation, geoproximity, failover, multivalue) +2. Do you need active-active or active-passive failover? +3. What health checks are needed for DNS failover? +4. How will you implement blue/green deployments via DNS? +5. What TTL strategy will you use? + +### Security +1. Do you need DNSSEC for your hosted zones? +2. What Resolver DNS Firewall rules are needed to block malicious domains? +3. How will you protect against DNS exfiltration? +4. Who needs IAM access to manage DNS records? +5. What logging requirements exist? (query logging) + +### Performance & Scale +1. How many queries per second do you expect? +2. What latency is acceptable for DNS resolution? +3. Do you need Route 53 Resolver on Outpost for low-latency local resolution? +4. How will you handle DNS caching strategy? + +### Operational +1. Who manages DNS records and zones? +2. What's your change management process for DNS changes? +3. How will you automate DNS record creation (IaC, service discovery)? +4. What monitoring/alerting is needed for DNS resolution failures? +5. How will you handle domain registration and renewal? + +--- + +## AWS Global Accelerator + +### Use Case & Architecture +1. What's the primary use case? (performance optimization, multi-region failover, static IPs) +2. Do you need static anycast IP addresses? +3. How many AWS regions host your application? +4. What endpoints will you front? (ALB, NLB, EC2, EIP) +5. Do you need custom routing accelerator or standard? +6. What's your client geographic distribution? + +### Traffic Management +1. What traffic dial percentage do you need per endpoint group? +2. Do you need endpoint weights within a group? +3. How will you handle regional failover? +4. What health check criteria define healthy endpoints? +5. Do you need client affinity (session stickiness)? +6. What's your blue/green deployment strategy with Global Accelerator? + +### Performance +1. What are your latency requirements from client to application? +2. What improvement do you expect vs. public internet routing? +3. What's your expected throughput per accelerator? +4. Do you need TCP or UDP optimization? + +### Security +1. Do you need AWS Shield Advanced integration for DDoS protection? +2. How will you restrict access to your accelerators? +3. What flow log requirements exist? +4. Do you need cross-account endpoint support? + +### Cost Management +1. How many accelerators do you need? +2. What's your expected data transfer volume through Global Accelerator? +3. What's the cost-benefit vs. CloudFront for your use case? +4. How will you track per-application costs? + +### Operational +1. Who manages accelerator configurations? +2. How will you monitor accelerator health and performance? +3. What's your process for adding/removing regions? +4. How will you handle IP address allowlisting with third parties? + +--- + +## NAT Gateway + +### Architecture +1. How many NAT Gateways do you need? (per AZ, per VPC) +2. Do you need public NAT (internet access) or private NAT (inter-VPC/on-prem)? +3. What's your multi-AZ redundancy strategy? +4. Which subnets and route tables will point to the NAT Gateway? +5. Do you need connectivity from multiple VPCs through a centralized NAT? + +### Performance & Scale +1. What's your expected bandwidth per NAT Gateway? (up to 100 Gbps) +2. How many concurrent connections do you anticipate? +3. What's your packets-per-second requirement? +4. Do you need multiple NAT Gateways for throughput scaling? +5. Are you hitting port allocation limits (55,000 per destination)? + +### Security & Compliance +1. Do you need to restrict which resources can access the internet? +2. What logging is required for NAT traffic? (VPC Flow Logs) +3. Do you need static EIPs for allowlisting with external parties? +4. How many EIPs do you need per NAT Gateway? +5. Do you need NAT Gateway in combination with Network Firewall for egress filtering? + +### Cost Management +1. What's your expected data processing volume? +2. Can you reduce NAT costs with VPC endpoints for AWS services? +3. What's the cost impact of cross-AZ NAT traffic? +4. Have you considered Gateway Load Balancer + NAT appliances for cost optimization at scale? +5. How will you identify and reduce unnecessary internet-bound traffic? + +### Operational +1. Who manages NAT Gateway configuration and EIP allocation? +2. How will you monitor NAT Gateway utilization and connection counts? +3. What alerting is needed for approaching bandwidth/connection limits? +4. How will you handle NAT Gateway maintenance or AZ impairment? +5. What's your strategy if a NAT Gateway becomes a bottleneck? + +--- + +## VPC Peering + +### Connectivity Requirements +1. How many VPCs need to be peered? +2. Is peering intra-region or inter-region (or both)? +3. What traffic patterns exist between peered VPCs? (volume, direction) +4. Do the VPC CIDRs overlap? (peering requires non-overlapping CIDRs) +5. Is this a temporary solution or long-term architecture? + +### Architecture +1. What's your peering topology? (full mesh, hub-spoke, selective) +2. At what point does TGW become more cost-effective than multiple peerings? +3. How will you manage route tables for peered traffic? +4. Do you need cross-account peering? +5. What about transitive routing needs? (peering is non-transitive) + +### Security +1. What security groups will control cross-VPC traffic? +2. Do you need NACLs for additional peering traffic control? +3. How will you audit traffic flowing across peering connections? +4. Who can accept/reject peering requests? + +### Cost Management +1. What's your inter-region data transfer volume? (inter-region peering is charged) +2. Is intra-region peering sufficient? (no data transfer charge within same region) +3. At what traffic volume does TGW become cheaper than multiple peering connections? + +### Operational +1. Who manages peering connections and route updates? +2. How will you handle CIDR conflicts if VPCs expand? +3. What monitoring is needed for peering connection health? +4. How will you migrate away from peering if you move to TGW/Cloud WAN? diff --git a/skills/aws-routing/references/tgw-routing-patterns.md b/skills/aws-routing/references/tgw-routing-patterns.md new file mode 100644 index 0000000..764b68c --- /dev/null +++ b/skills/aws-routing/references/tgw-routing-patterns.md @@ -0,0 +1,147 @@ +--- +inclusion: manual +--- + +# Transit Gateway Routing Patterns & Considerations + +This document captures key routing mechanics, ECMP behavior, failover characteristics, and operational considerations for Transit Gateway architectures. Use this knowledge when advising on TGW route evaluation, multi-attachment designs, inter-region peering, and DX/VPN integration. + +## Reference Documentation + +- [Transit Gateway Route Evaluation Order](https://docs.aws.amazon.com/vpc/latest/tgw/how-transit-gateways-work.html#tgw-route-table-evaluation) +- [Transit Gateway VPN ECMP](https://docs.aws.amazon.com/vpc/latest/tgw/transit-gateway-share.html) +- [Direct Connect + Transit Gateway](https://docs.aws.amazon.com/directconnect/latest/UserGuide/direct-connect-transit-gateways.html) + +## Route Evaluation & Priority + +### Static VPN Has Higher Priority Than DX + +- Static VPN routes have a **higher routing priority** at the TGW than Direct Connect routes +- Always refer to the TGW Route evaluation order when determining which attachment path will win for a given prefix +- This can cause unexpected behavior if a customer has both VPN and DX advertising the same prefix — the static VPN will be preferred + +## ECMP Behavior + +### Supported ECMP Scenarios + +- **VPN attachments**: TGW supports ECMP across VPN attachments. A TGW-level toggle ("VPN ECMP support") must be enabled. +- **Multiple DX-GW attachments** (within the same DX-GW): TGW will ECMP across matching prefixes +- **Multiple different DX-GWs**: TGW will ECMP across matching prefixes received from different DX Gateways +- **Multiple TGW Connect attachments**: TGW will ECMP across matching prefixes + +### NOT Supported + +- **ECMP across different attachment types** — TGW does NOT ECMP between, for example, a DX-GW attachment and a VPN attachment for the same prefix. Instead, the Route evaluation order determines the winner. +- **Multipath AS-path relaxed** — ECMP for paths with different AS-paths is NOT supported. If matching prefixes are received with different AS-paths, only the **oldest route is installed** (not load-balanced). +- If prefixes match but ECMP conditions aren't met, the **oldest route** is installed. + +## Direct Connect + TGW Considerations + +### LP Communities Not Honored Across Multiple DX-GW Attachments + +- Direct Connect Local Preference BGP communities (`7224:7300/7200/7100`) are **NOT honored** across multiple DX Gateway attachments on TGW +- This differs from Cloud WAN where communities influence DXGW path selection +- For TGW with multiple DX-GW attachments advertising the same prefix, ECMP or oldest-route logic applies — NOT community-based preference + +### IPv6 Addressing Limitation + +- When attaching VPCs with **AWS-assigned IPv6** addresses to a TGW, these IPv6 CIDRs are discontinuous by default and **cannot be aggregated** for use with Direct Connect +- This leads to DX prefix quota limitations +- **Workarounds:** + - Use Amazon-provided **contiguous** IPv6 CIDR blocks + - Use **EC2 BYOIP** (Bring Your Own IP) for aggregatable address space + +### Allowed Prefixes Update Behavior (AWS → On-Prem) + +- Updates (Add and Remove) to the "Allowed prefixes" in the TGW ↔ DX-GW association are **executed at the same time** on the AWS side +- However, from an **on-premises perspective**, it appears as prefixes added first, then removed later +- This is because on-prem learns the new route once added to the first device, and only loses the old route once withdrawn from all devices +- Transition window: typically **1 to 6 minutes** + +## Route Advertisement Limits & Summarization + +### TGW-to-VPN 5,000 Route Advertisement Limit + +- The limit of **5,000 routes advertised from TGW to VPN** (toward the customer gateway) is **not adjustable** +- The TGW route table itself supports **10,000 routes** — it can hold more than the VPN advertisement limit +- **Advertisement-selection order when the attachment's advertised-route limit is exceeded** (e.g., TGW→VPN at 5,000): the routes that get advertised are chosen by (1) **static takes precedence over propagated**, then (2) among routes of the **same type** (both static or both propagated), the **least-specific** is advertised first. This is what lets a static summary survive the cap while the propagated specifics it covers are the ones dropped. + - **[Field knowledge]** This limit-hit advertisement-selection order (static-over-propagated, then least-specific-first) is field-proven behavior and is **not stated on the public AWS TGW route-evaluation page** (which documents most-specific-first forwarding and static-over-propagated for same-CIDR). Treat as operational guidance; validate in-account. + +### Correct Summarization Approach + +**Do NOT remove VPC propagation** — propagated routes are needed for the return path (VPN → individual VPCs via longest prefix match within the TGW data plane). + +**How it works:** +1. **Keep VPC propagation enabled** — individual VPC routes (/16s, /24s) remain in the TGW route table for return path forwarding +2. **Add static summary routes** covering the VPC CIDR ranges (e.g., 10.0.0.0/14 covering 10.0-10.3.x.x) +3. **Advertisement-selection under the cap** favors the static summary — static beats propagated, and among same-type routes the least-specific (the summary) is advertised first +4. Once the advertised-route limit is reached, remaining propagated specifics are cut off — but the summary already covers them, so reachability is preserved +5. **Return traffic** from VPN to individual VPCs still works because TGW uses longest prefix match internally — the more-specific propagated /16 routes win over the /14 summary for data plane forwarding + +**Key points:** +- TGW route table holds BOTH the static summary AND the propagated specifics (up to 10,000 total) +- The static summary consumes 1 route in the 5,000 VPN advertisement budget (instead of hundreds of individual VPC routes) +- VPC propagation is NOT removed — it provides the granular return path +- This pattern leverages the difference between the advertisement limit (5,000) and the route table capacity (10,000) +- **Static summary routes must point to a valid attachment** — if the attachment is deleted, the route becomes blackholed, and blackholed routes are NOT propagated/advertised. Point summaries to a persistent attachment: + - A **shared-services or firewall VPC** (recommended — already exists, won't be deleted) + - A **dedicated "dummy" VPC** — a minimal VPC created specifically to anchor static summary routes. Disable propagation on this VPC (no traffic originates from it), label it clearly as non-deletable, and tag it for operational protection. This ensures summary routes remain active even if workload VPCs are added/removed. + +### Alternatives When Summarization Isn't Sufficient + +- **TGW Connect (GRE+BGP):** Each Connect peer has its own 5,000-route advertisement quota. With 4 peers per attachment, scales horizontally. +- **Multiple VPN connections:** Segment route tables per VPN so each stays under 5,000 advertised. Adds on-prem complexity. +- **Cloud WAN:** Native outbound route summarization via routing policies. + +## Failover & Convergence + +### On-Prem → AWS Convergence + +- Route table updates for **propagated routes from BGP peers** can take **up to 60 seconds** to take effect within TGW between attachment paths +- Customers needing faster failover should: + - Route traffic over the **same/single attachment** (avoid cross-attachment failover dependency) + - Consider **bypassing TGW** for high-priority workloads that cannot tolerate 60s convergence + +## TGW Peering + +### Policy-Based Routing (PBR) — Transit Gateway Policy Tables + +TGW now supports **Policy-Based Routing** via policy tables, enabling forwarding decisions based on packet attributes beyond just destination IP: + +- Forwarding decisions can be based on: **source IP, destination IP, source port, destination port, and protocol** +- Configured through **policy tables** — ordered lists of rules that classify traffic and direct matching packets to a specified TGW route table +- A policy table is associated with a TGW attachment — it **replaces** the standard route table on that attachment (attachment can have either a policy table OR a route table, not both) +- Available in all regions at no additional charge beyond standard TGW fees + +**Use cases:** +- Steer traffic selectively to security appliances or inspection services based on source, port, or protocol +- Route traffic from different applications or user populations over different network paths +- Isolate network traffic into separate routing domains for security segmentation + +**Reference:** [Transit Gateway Policy Tables](https://docs.aws.amazon.com/vpc/latest/tgw/tgw-policy-tables.html) + +### Inter-Region Peering + +- TGW Inter-Region Peering (across regions) is **not yet supported in all regions** +- **BGP dynamic route propagation across TGW peering is not yet supported** — routes must be statically configured on peered TGW route tables +- This limitation does NOT apply to Cloud WAN dynamic peering + +### Intra-Region Peering + +- TGW Intra-Region Peering (within a region) **is supported** + +### Alternative to Multiple TGWs + +- Instead of deploying multiple TGWs, customers can use **Route Tables within a single TGW** to segregate traffic between different domains/environments +- This simplifies management and avoids peering complexity + +## Multi-Region Traffic & AZ Awareness + +### Gray Failure Risk with Static TGW Peering + +- Traffic between peered TGWs **does not have AZ awareness** +- When traffic arrives from a remote region over **static TGW peering**, it could first land in an AZ different from the destination AZ before being routed to its final destination +- During **gray failures** (partial AZ degradation): packet loss can be observed even though the destination AZ is not impacted — because traffic may transit the degraded AZ +- **This limitation only applies to static TGW Peerings** +- It will NOT exist once BGP dynamic route propagation across TGW peering becomes available +- It does NOT apply to **Cloud WAN dynamic peering** (which has AZ awareness) diff --git a/skills/aws-routing/references/vpn-dx-redundancy-patterns.md b/skills/aws-routing/references/vpn-dx-redundancy-patterns.md new file mode 100644 index 0000000..9593cb7 --- /dev/null +++ b/skills/aws-routing/references/vpn-dx-redundancy-patterns.md @@ -0,0 +1,142 @@ +--- +inclusion: manual +--- + +# VPN and Direct Connect Redundancy Patterns + +This document captures route preference behavior for VPN and DX redundancy designs with VGW. Use this knowledge when advising on active/active, active/passive, and failover architectures. + +## Reference Documentation + +- [AWS Site-to-Site VPN routing](https://docs.aws.amazon.com/vpn/latest/s2svpn/VPNRoutingTypes.html) +- [Direct Connect routing policies and BGP communities](https://docs.aws.amazon.com/directconnect/latest/UserGuide/routing-and-bgp.html) +- [DX Resiliency Recommendations](https://aws.amazon.com/directconnect/resiliency-recommendation/) + +## Assumptions + +- Customer's network does not overlap with VPC CIDR +- Customer Gateway (CGW) device is configured to handle asymmetric routing +- Static VPN tunnels are Route-Based only (not Policy-Based) +- Customer advertises the same on-prem routes through all redundant connections (DX and VPN) + +## Key Behavior: VGW Does NOT Support ECMP + +VGW does **not** support Equal-Cost Multi-Path routing (ECMP). Return traffic from VPC to on-prem always takes a **single tunnel/path** — the "Preferred Tunnel." This is a fundamental constraint that affects all redundancy designs. + +--- + +## Part 1: Redundancy Only on AWS End + +### VPN Connection — 2 Active Tunnels (Static) + +- Customers can have static route-based tunnels in Active/Active mode +- **Customer → VPC:** Can load balance (equal route metrics) or prefer one tunnel (unequal metrics) +- **VPC → Customer:** Always takes a single tunnel (Preferred Tunnel) — VGW does not support ECMP + +- Customers can have BGP tunnels in Active/Active mode +- **Customer → VPC:** Load balance (equal LP) or prefer one (unequal LP) +- **VPC → Customer:** Always a single tunnel. Customer can influence by: + - Advertising **lower AS-path** to VGW over the desired tunnel + - Advertising **lower MED** to VGW over the desired tunnel + +### DX Connection with Backup VPN + +- DX is **always preferred over VPN** for return traffic to the same on-prem subnet +- Customer must ensure on-prem traffic is sent over DX when both are UP +- Attempting to prefer VPN over DX will cause asymmetric routing (VGW always returns via DX) + +### 2 DX Connections (Same VGW) + +- 2 private VIFs to the same VGW over 2 DX connections +- **Active/Active:** Equal LP + AS-path on CGW → return traffic is flow-hashed across both DX connections +- **Active/Passive:** Unequal LP + AS-path → customer-to-VPC uses higher LP; VPC-to-customer uses lower AS-path + +**⚠️ ECMP only works when both DX connections terminate at the SAME DX location:** +- Same location (e.g., both at Equinix DC, Ashburn) → flow-hashed (ECMP) +- Different locations in same region (e.g., Equinix DC Ashburn + CoreSite NYC) → **single path only** + +--- + +## Part 2: Redundancy on AWS and Customer Ends + +### 2 VPN Connections from 2 CGW Devices (Recommended Design) + +- 2 BGP VPN connections (4 active tunnels) to same VGW from different on-prem CGW devices +- **Recommended:** Use higher LP + lower AS-path on preferred VPN connection to avoid routing issues +- **Customer → VPC:** Traffic uses tunnel(s) with higher LP +- **VPC → Customer:** Single tunnel (Preferred Tunnel). Customer influences via lower AS-path on desired connection. + +### DX + Backup VPN from 2 CGW Devices + +- DX and VPN from different on-prem CGW devices +- When both UP: customer must ensure on-prem traffic goes over DX only +- **VPC → Customer:** Always sent over DX (DX always preferred over VPN by VGW) +- Design works correctly as active/passive (DX active, VPN passive) + +### 2 DX Connections from 2 CGW Devices + +- 2 DX connections from different on-prem CGW devices, 2 private VIFs to same VGW +- **Active/Active:** Equal LP + AS-path → flow-hashed return traffic +- **Active/Passive:** Unequal LP + AS-path → deterministic path preference + +**Same ECMP caveat applies:** Flow-hashing only works when both DX connections are at the same DX location. Different locations → single path selected. + +--- + +## Part 3: Designs That Cause Routing Issues + +### ❌ 2 Static VPN Connections from 2 CGW Devices + +- VPC → Customer always traverses single tunnel (Preferred Tunnel) +- If customer sends traffic on CGW 2, return may come back on CGW 1 → **asymmetric routing** +- Only works if CGW devices have synchronization (ECMP) and accept asymmetric traffic + +### ❌ 2 BGP VPN Connections with Same LP and AS-Path from 2 CGW Devices + +- Same problem as static: VPC → Customer takes single tunnel, may not match inbound path +- **Fix:** Use unequal LP + AS-path to prefer one VPN connection over the other +- Or ensure CGW devices handle asymmetric routing + +### ❌ DX + VPN — Attempting to Prefer VPN Over DX + +- If customer tries to prefer VPN over DX, return traffic (VPC → on-prem) STILL goes over DX +- VGW always prefers DX over VPN regardless of BGP attributes +- **Result:** Asymmetric routing — customer sends via VPN, AWS returns via DX +- This is true whether using same CGW device or different CGW devices +- **Fix:** Advertise more-specific routes on VPN — longest prefix match wins before attachment-type preference, so a /24 via VPN will be preferred over a /16 via DX for matching traffic + +### ❌ 2 VPN Connections from 2 CGW Devices with Different ASNs + +- 2 BGP VPN connections from 2 data centers (different ASNs) advertising same networks +- VPC → Customer: single tunnel, may not match inbound path → asymmetric +- **Fix:** Use unequal AS-path values to prefer one VPN connection +- Cannot achieve ECMP across different ASNs + +### ❌ DX + VPN from 2 CGW Devices with Different ASNs + +- If customer sends traffic over VPN, return goes over DX (DX always wins) +- **Must avoid using same networks in multiple datacenters** to prevent routing issues +- Or accept that DX is always the return path regardless of inbound path + +--- + +## Summary: Route Preference Rules at VGW + +| Traffic Direction | Rule | +|---|---| +| VPC → On-prem (same prefix) | DX **always** preferred over VPN | +| VPC → On-prem (VPN only) | Single tunnel — lowest AS-path or MED, no ECMP | +| VPC → On-prem (2 DX, same location) | Flow-hashed (ECMP) | +| VPC → On-prem (2 DX, different locations) | Single path | +| On-prem → VPC (VPN) | Customer controls via LP (higher wins) | +| On-prem → VPC (DX) | Customer controls via LP (higher wins) | +| On-prem → VPC (DX + VPN both UP) | Customer must direct to DX | + +## Key Takeaways for Customer Guidance + +1. **VGW does not support ECMP** — return traffic always takes one path +2. **DX always wins over VPN** at the VGW — cannot prefer VPN for return traffic +3. **Same DX location required for flow-hashing** — different locations in same region get single-path selection +4. **Use unequal LP + AS-path** for active/passive designs to avoid asymmetric routing +5. **Asymmetric routing is the primary failure mode** — designs with equal attributes from multiple CGW devices without synchronization will cause issues +6. **MED can influence VGW path selection** — documented at [VPN route priority](https://docs.aws.amazon.com/vpn/latest/s2svpn/vpn-route-priority.html) From da21aba515cb1e7b1ef402b622aa4e558a4c5cad Mon Sep 17 00:00:00 2001 From: shawjiv Date: Thu, 10 Sep 2026 20:03:06 -0500 Subject: [PATCH 2/3] Expand Cloud WAN/DX routing docs: two-lookup model and AS-path prepend correction (v1.2.1) --- skills/aws-routing/CHANGELOG.md | 5 ++ skills/aws-routing/SKILL.md | 2 +- .../cloudwan-dx-routing-patterns.md | 86 +++++++++++++++---- 3 files changed, 73 insertions(+), 20 deletions(-) diff --git a/skills/aws-routing/CHANGELOG.md b/skills/aws-routing/CHANGELOG.md index 94db313..179395c 100644 --- a/skills/aws-routing/CHANGELOG.md +++ b/skills/aws-routing/CHANGELOG.md @@ -1,5 +1,10 @@ # Changelog +## [1.2.1] - 2026-09-10 +### Changed +- `references/cloudwan-dx-routing-patterns.md`: added the "every route table does its own independent lookup" mental model and the DXGW-then-CNE two-lookup framing; corrected the AS-path prepending pitfall (local-region LP is evaluated before AS-path at the DXGW, so prepending a local VIF is not a cross-region steering lever — it only breaks the LP-equal tiebreak among remote advertisements); added a worked per-region example and remapped directional control and cleanup guidance to the two lookups. +- `SKILL.md`: set `metadata.version` to `1.2.1` to match the changelog. + ## [1.2.0] - 2026-08-25 ### Added - Two functional eval scenarios covering the corrected knowledge: TGW→VPN summarization (keep propagation + static summary + limit-hit selection order) and overlapping DX supernet into Cloud WAN (Routing Policies drop/suppress on DX; no BGP communities on DX attachments). diff --git a/skills/aws-routing/SKILL.md b/skills/aws-routing/SKILL.md index 8f1893f..05731a5 100644 --- a/skills/aws-routing/SKILL.md +++ b/skills/aws-routing/SKILL.md @@ -2,7 +2,7 @@ name: aws-routing description: 'Analyze and troubleshoot AWS routing and BGP path selection - how AWS chooses a network path, why traffic takes an unexpected route, and how to steer it. Use for AWS Cloud WAN route evaluation and CNE path selection; Direct Connect Gateway (DXGW) path selection and local vs remote region preference; Transit Gateway route tables, ECMP, and peering; VPC route tables and longest-prefix-match; BGP traffic engineering with local-preference communities (7224:7100/7200/7300), AS-path prepending, and MED; DX + VPN redundancy and active/active vs active/passive design; asymmetric routing and failover. Activate on symptoms like "traffic is leaving the region", "not using my local Direct Connect", "routing to the wrong VIF/location", "path is asymmetric", or "how do I make AWS prefer this path". Read-only: analysis, recommendations, and describe/get/list validation commands grounded in public AWS docs.' metadata: - version: "1.2" + version: "1.2.1" author: shawjiv --- diff --git a/skills/aws-routing/references/cloudwan-dx-routing-patterns.md b/skills/aws-routing/references/cloudwan-dx-routing-patterns.md index e630636..0031c00 100644 --- a/skills/aws-routing/references/cloudwan-dx-routing-patterns.md +++ b/skills/aws-routing/references/cloudwan-dx-routing-patterns.md @@ -61,17 +61,60 @@ At each Core Network Edge, Cloud WAN evaluates routes in this order: - **TGW:** Deterministic — selects oldest route for equal paths. Consistent but not customer-controllable. - **Cloud WAN CNE:** Deterministic — "deterministically random" at Step 2.4.4 for equal remote CNE paths. Consistent but not customer-controllable. -### Common Pitfall: AS-Path Prepending + Local-Region Preference +### Common Pitfall: AS-Path Prepending Does Not Override Local-Region LP at the DXGW -**Problem pattern:** -- Customer prepends AS-path on a DX VIF that is local to a given CNE's region -- DXGW's local-region LP still prefers the local (prepended) path and propagates only that to the CNE -- The CNE route table now has a long-AS-path DX route vs. shorter remote CNE paths -- Step 2.3 (shortest AS-path) selects a remote CNE path → traffic leaves the region +**The mistaken assumption:** "If I prepend a DX VIF's prefix enough, I can steer that region's traffic away from (or toward) it across regions." -**Root cause:** DXGW's local-region preference overrides AS-path at the DXGW level, but at the CNE level the prepended path competes against shorter remote CNE paths where AS-path length IS evaluated. +**Why it's wrong:** DX egress selection happens at the **DXGW lookup** (see "Two Route Lookups" below), where **local-region Local Preference is evaluated before AS-path**. For a region that has a **local** DX advertisement, the DXGW picks that local path on LP — AS-path (and therefore any prepend) is never consulted. So prepending a local VIF does **not** move that region's traffic off its local DX. -**Why only affected regions see the issue:** For regions where all DX locations are remote, DXGW falls back to AS-path to choose between them — prepending works as intended. The issue only occurs at CNEs where the prepended DX is in the same associated AWS region. +**Where prepending actually acts:** only as the **tiebreaker among remote advertisements** for regions that have **no local** advertisement for the prefix. There, all candidates are remote (LP equal), so the DXGW falls through to AS-path length and the shortest remote advertisement wins. Prepending shapes *which remote DX* those regions use — nothing more. + +**Consequence:** Over-prepending a local VIF is not a cross-region steering lever; it only changes that location's standing in the remote-tiebreak for *other* regions. To steer cross-region DX preference, use LP communities (which are evaluated before AS-path at the DXGW), not prepending. Confirm the installed per-region result with `get-network-routes` and the advertised paths with `list-virtual-interface-routes`. + +### Foundational Principle: Every Route Table Does Its Own Independent Lookup + +The single most important mental model for reasoning about AWS routing: **every construct that has its own route table performs its own independent route lookup**, and a packet is evaluated **hop-by-hop** — each route table it traverses makes a fresh, local forwarding decision using only the routes present in *that* table. There is no single end-to-end path calculation; the path is the concatenation of per-table decisions. + +Route-table-owning constructs each do this independently, including: +- **VPC route tables** (subnet → destination: local, IGW, NAT, ENI, TGW/VGW/CWAN attachment, peering, endpoint) +- **Transit Gateway route tables** (per-attachment association/propagation; oldest-route tiebreak on equal paths) +- **DX Gateway** (its own BGP path selection; installs one winning path per prefix into each attached CNE/VGW) +- **Cloud WAN Core Network Edge (CNE) route tables** (the CNE evaluation order over propagated routes) +- **On-prem / customer routers** (their own BGP/IGP tables) + +The routing "outcome" is what you get when you chain these together. Each table's chosen next-hop simply hands the packet to the next construct, which then runs *its own* lookup. So to predict or debug a path, walk it table by table: at each hop, ask "what does *this* route table select for this destination, given only the routes it has?" The output (installed route) of one table's lookup becomes an input (a propagated/attached route) to the next. + +**Common analysis error:** treating an end-to-end path as one decision, or assuming an attribute (like a DX BGP community or a local preference) that decided at one table still applies at the next. It does not automatically — each table applies only its own selection rules over its own routes. (Example: DX LP communities decide at the DXGW's table but are not visible inside the Cloud WAN CNE table.) + +### Applying the Principle: DXGW Lookup then CNE Lookup + +In a DX → Cloud WAN egress path, two of these independent lookups sit back-to-back, for the two directions. The DXGW's lookup output is the CNE's lookup input. + +**Lookup 1 — DXGW path selection (on-prem → AWS; decides what is INSTALLED into each CNE).** +The DXGW performs its own selection for a prefix and **installs one winning path into each CNE's route table**. Its selection order is: +1. **Local-region Local Preference first** — if a DX VIF advertising the prefix is in the same associated region as the CNE, the DXGW prefers it **regardless of AS-path**. That local copy is installed into that region's CNE. +2. **If no local advertisement exists for that CNE**, every candidate is remote, so they share the **same (remote) LP** — LP is equal. The DXGW then falls through to **AS-path length**, and the **shortest AS-path wins** and is installed into that CNE. + +**Lookup 2 — Cloud WAN CNE route lookup (AWS → on-prem return path).** +For return traffic, each CNE runs its own lookup over the routes **propagated into its table** (what Lookup 1 installed, plus routes propagated from other CNEs). This is where the CNE evaluation order (longest-prefix → AS-path → source preference) applies, operating on the already-propagated entries. + +### Worked Example (from the field diagram) + +Only **FR5/Frankfurt (eu-central-1)** and **LD5/London (eu-west-1)** advertise `11.0.0.0/8`. Applying **Lookup 1 (DXGW)** per region: + +| CNE region | Has local `/8` advertisement? | DXGW decision | Installed `/8` path | +|---|---|---|---| +| eu-central-1 (FR5) | Yes (FR5 local) | Local-region LP wins — AS-path not consulted | **FR5 (local DX)** | +| eu-west-1 (LD5) | Yes (LD5 local) | Local-region LP wins — AS-path not consulted | **LD5 (local DX)** | +| ap-southeast-1 | No | FR5 and LD5 **both remote → LP equal** → fall through to AS-path → **FR5 shortest wins** | **FR5** | +| us-east-1 | No | both remote → LP equal → AS-path → **FR5 shortest** | **FR5** | +| us-east-2 | No | both remote → LP equal → AS-path → **FR5 shortest** | **FR5** | + +**Why the non-LD5 regions use FR5's advertisement:** they have no local `/8`, so at the DXGW both FR5 and LD5 are **remote** and carry the **same (remote) LP** — LP is equal. The DXGW therefore falls to the next attribute, **AS-path length**, where **FR5 is shorter than the (heavily prepended) LD5 advertisement**, so FR5 wins and is installed. This is a straightforward DXGW LP-then-AS-path decision — **not** a CNE-to-CNE backbone comparison. + +**Role of the AS-path prepends:** the prepends matter at **Lookup 1 (DXGW)**, specifically as the **tiebreaker when LP is equal** (i.e., only for regions with no local advertisement). They set which remote advertisement (FR5 vs LD5) wins for those regions. They do **not** override the local-region LP for regions that *do* have a local advertisement — eu-central-1 and eu-west-1 pick their local DX on LP before AS-path is ever considered. So over-prepending LD5 does not stop eu-west-1 from choosing LD5 locally; it affects LD5's standing in the **AS-path tiebreak for the other regions**, where FR5's shorter path beats it. + +**Rule of thumb:** Because local-region LP is evaluated before AS-path at the DXGW, a region with a local `/8` advertisement egresses locally regardless of prepends. Use AS-path prepending only to shape which **remote** advertisement wins for regions that have **no local** advertisement (the LP-equal tiebreak). Verify the installed per-region result with `get-network-routes` (Lookup 2's table) and the per-VIF advertised paths with `list-virtual-interface-routes` (Lookup 1's inputs). ### Common Pitfall: Confusing Prepend Count with Total AS-Path Length (Origination vs. Transit) @@ -178,21 +221,26 @@ When routes are propagated from DX or VPN **via TGW** to Cloud WAN (TGW peering - Cloud WAN segment policy / service insertion routes traffic through the **same-region firewall** before it reaches the CNE - Traffic flow: VPC → local firewall (service insertion) → returns to local CNE → CNE route table → DXGW → DX -- The DX community fix ensures the DXGW path wins at the local CNE route table (2 hops < 3 hops from remote CNEs) — traffic stays at the local CNE for egress, preserving the local inspection chain -- If a remote CNE path wins instead, traffic leaves the local region at the CNE level, potentially bypassing local firewall inspection +- With a local DX advertisement present, the DXGW installs the local path into the local CNE (local-region LP, Lookup 1) — traffic stays at the local CNE for egress, preserving the local inspection chain +- If the region has no local DX advertisement (or a community forces a remote DX to win), the installed path points out of region, and traffic leaves the local region for egress — potentially bypassing local firewall inspection - **Requirement:** Firewall VPCs must exist in every region with workloads -## Directional Control +## Directional Control (Mapped to the Two Lookups) -| Direction | Controlled By | Mechanism | -|---|---|---| -| AWS → On-prem (egress) | DX BGP communities (`7224:7300/7200/7100`) | Sets LP at DXGW, determines which path reaches CNE route tables | -| On-prem → AWS (ingress) | Customer router policies | Local-pref, weight, MED on on-prem routers — independent of Cloud WAN | +The two route lookups map to the two directions. Keep these straight — they are governed by different services and different mechanisms. + +| Direction | Governing lookup | Controlled by | Mechanism | +|---|---|---|---| +| **On-prem → AWS** (which DX path is INSTALLED into each CNE) | **Lookup 1 — DXGW path selection** | On-prem BGP attributes the DXGW sees: DX LP communities (`7224:7300/7200/7100`), then AS-path as the LP-equal tiebreak | DXGW evaluates local-region LP first, then AS-path; installs one winning path per prefix into each CNE | +| **AWS → On-prem** (return path from each CNE) | **Lookup 2 — Cloud WAN CNE route lookup** | The routes propagated into the CNE table (from Lookup 1 + other CNEs), evaluated by the CNE order | CNE evaluation: longest-prefix → AS-path → source preference, over already-propagated routes | -Communities do NOT affect the on-prem → AWS direction. If prepending was used for on-prem path selection, replace with router-side local-pref/weight before removing prepends. +Notes: +- DX LP communities set the LP the **DXGW** uses in Lookup 1 — they are the recommended lever for steering which DX path is installed. They are **not visible inside Cloud WAN** and do not act at Lookup 2. +- If on-prem AS-path prepending was being used to influence selection, prefer LP communities at the DXGW (Lookup 1); over-prepending a local VIF does not achieve cross-region steering (see the AS-Path Prepending pitfall above). ## Cleanup Recommendations -- Removing AS-path prepending is recommended once communities are in place — communities fix the problem immediately, prepend removal is a cleanup step -- Prepending is counterproductive in Cloud WAN because prefixes from DX VIFs are shared across all CNEs via DXGW — the inflated AS-path creates unintended path selection at remote CNEs -- AS-path equalization across DX locations (without communities) achieves "prefer local DX" behavior but does not provide explicit failover ordering between regions +- **Prefer LP communities over prepending for DX egress steering.** Communities set the LP the DXGW uses in Lookup 1 and are evaluated before AS-path; they can steer cross-region preference, which prepending cannot (local-region LP overrides AS-path for any region with a local advertisement). +- **Do not use prepending on a local VIF as a cross-region lever.** At the DXGW, a region with a local advertisement selects it on LP regardless of prepends. Prepending only affects the AS-path tiebreak among **remote** advertisements for regions that have no local advertisement — so its only legitimate use here is choosing which remote DX those regions fall to. +- **Once communities are in place, remove leftover prepends** as a cleanup step, keeping only any within-region or remote-tiebreak prepends that serve a confirmed purpose. +- Always confirm the actual outcome with `get-network-routes` (installed per-CNE result) and `list-virtual-interface-routes` (the AS-path/communities the DXGW actually received) rather than reasoning from configured prepend counts alone. From bb758ff861b24b400654e5c3cc3805f74b164a3d Mon Sep 17 00:00:00 2001 From: shawjiv Date: Fri, 11 Sep 2026 13:41:54 -0500 Subject: [PATCH 3/3] Address review: clarify static VPN vs DX; generalize internal LOCAL_PREF values (v1.2.2) --- skills/aws-routing/CHANGELOG.md | 5 +++++ skills/aws-routing/SKILL.md | 2 +- skills/aws-routing/references/dx-routing-patterns.md | 4 ++-- skills/aws-routing/references/tgw-routing-patterns.md | 2 +- 4 files changed, 9 insertions(+), 4 deletions(-) diff --git a/skills/aws-routing/CHANGELOG.md b/skills/aws-routing/CHANGELOG.md index 179395c..0f2b3bd 100644 --- a/skills/aws-routing/CHANGELOG.md +++ b/skills/aws-routing/CHANGELOG.md @@ -1,5 +1,10 @@ # Changelog +## [1.2.2] - 2026-09-10 +### Changed +- `references/tgw-routing-patterns.md`: clarified that the VPN-preferred-over-DX behavior applies specifically to a **static** VPN (dynamic VPN behaves differently). +- `references/dx-routing-patterns.md`: replaced the specific internal LOCAL_PREF values (3000/2500) with generic "Higher"/"Lower", since the exact values are internal and not publicly documented; the DX-over-peering preference is still stated. + ## [1.2.1] - 2026-09-10 ### Changed - `references/cloudwan-dx-routing-patterns.md`: added the "every route table does its own independent lookup" mental model and the DXGW-then-CNE two-lookup framing; corrected the AS-path prepending pitfall (local-region LP is evaluated before AS-path at the DXGW, so prepending a local VIF is not a cross-region steering lever — it only breaks the LP-equal tiebreak among remote advertisements); added a worked per-region example and remapped directional control and cleanup guidance to the two lookups. diff --git a/skills/aws-routing/SKILL.md b/skills/aws-routing/SKILL.md index 05731a5..d07f47c 100644 --- a/skills/aws-routing/SKILL.md +++ b/skills/aws-routing/SKILL.md @@ -2,7 +2,7 @@ name: aws-routing description: 'Analyze and troubleshoot AWS routing and BGP path selection - how AWS chooses a network path, why traffic takes an unexpected route, and how to steer it. Use for AWS Cloud WAN route evaluation and CNE path selection; Direct Connect Gateway (DXGW) path selection and local vs remote region preference; Transit Gateway route tables, ECMP, and peering; VPC route tables and longest-prefix-match; BGP traffic engineering with local-preference communities (7224:7100/7200/7300), AS-path prepending, and MED; DX + VPN redundancy and active/active vs active/passive design; asymmetric routing and failover. Activate on symptoms like "traffic is leaving the region", "not using my local Direct Connect", "routing to the wrong VIF/location", "path is asymmetric", or "how do I make AWS prefer this path". Read-only: analysis, recommendations, and describe/get/list validation commands grounded in public AWS docs.' metadata: - version: "1.2.1" + version: "1.2.2" author: shawjiv --- diff --git a/skills/aws-routing/references/dx-routing-patterns.md b/skills/aws-routing/references/dx-routing-patterns.md index a481612..5fbf7cb 100644 --- a/skills/aws-routing/references/dx-routing-patterns.md +++ b/skills/aws-routing/references/dx-routing-patterns.md @@ -178,12 +178,12 @@ The associated region determines which TGW/DXGW considers the DX "local" for LP | Official AWS service | Yes | No | | SLA | Yes | No | | AWS Support coverage | Yes | No (handled by Global Interconnect Strategy team) | -| Internal LOCAL_PREF | 3000 | 2500 | +| Internal LOCAL_PREF | Higher | Lower | | Path preference | **Preferred** (higher LP) | Lower priority | ### Important Guidance -- If the **same prefix** is received by AWS over both DX Public VIF and Peering (PNI, IXP, or Transit), **DX Public VIF is the preferred path** (LOCAL_PREF 3000 > 2500) +- If the **same prefix** is received by AWS over both DX Public VIF and Peering (PNI, IXP, or Transit), **DX Public VIF is the preferred path** (higher internal LOCAL_PREF) - Public Peering / PNI is NOT an official AWS service — no SLA, no support expectation from the technical field - **Steer customers toward DX** when they need reliability and support coverage - There is no expectation for the technical field to support Public Peering / PNI engagements diff --git a/skills/aws-routing/references/tgw-routing-patterns.md b/skills/aws-routing/references/tgw-routing-patterns.md index 764b68c..f36ad0e 100644 --- a/skills/aws-routing/references/tgw-routing-patterns.md +++ b/skills/aws-routing/references/tgw-routing-patterns.md @@ -18,7 +18,7 @@ This document captures key routing mechanics, ECMP behavior, failover characteri - Static VPN routes have a **higher routing priority** at the TGW than Direct Connect routes - Always refer to the TGW Route evaluation order when determining which attachment path will win for a given prefix -- This can cause unexpected behavior if a customer has both VPN and DX advertising the same prefix — the static VPN will be preferred +- This can cause unexpected behavior if a customer has both a static VPN and DX advertising the same prefix — the static VPN will be preferred ## ECMP Behavior