Skip to content

Commit c87f271

Browse files
authored
Merge pull request #167 from Dstack-TEE/refactor/ingress-caa-flow
refactor(ingress): one CAA flow for every dns-01 setup
2 parents 55e0140 + a89d1c8 commit c87f271

9 files changed

Lines changed: 318 additions & 157 deletions

File tree

‎custom-domain/dstack-ingress/DNS_PROVIDERS.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ This guide explains how to configure dstack-ingress to work with different DNS p
2121

2222
- `DNS_PROVIDER` - DNS provider to use (`cloudflare`, `linode`, `namecheap`, `route53`). Detected from the provider's credential variable when unset; set it explicitly for `route53`
2323
- `ACME_EMAIL` - Contact address for the ACME account (`CERTBOT_EMAIL` still works). Published in the evidence, so leave it unset unless you want one
24-
- `SET_CAA` - Enable CAA record setup: `true` or `false` (default). Except on Cloudflare, this makes a non-wildcard domain's pointer an A record; see the README
24+
- `SET_CAA` - Enable CAA record setup: `true` or `false` (default `false`; `true` with `DELEGATION_ZONE`). Except on Cloudflare, this makes a non-wildcard domain's pointer an A record; see the README
2525
- `PORT` - HTTPS port (default: 443)
2626
- `TXT_PREFIX` - Prefix for TXT records (default: "_dstack-app-address")
2727

@@ -112,7 +112,7 @@ PolicyDocument:
112112
**Important Notes for Route53:**
113113
- The certbot plugin uses the format `certbot-dns-route53` package
114114
- CAA values are merged into the existing record set at the name: only values with the same tag are replaced, so `iodef` and other tags are kept
115-
- Where a CAA record shares the name (`DELEGATION_ZONE`, `SET_CAA=true` on a non-wildcard domain, or any CAA already there), the gateway pointer is an A record to the resolved gateway IP instead of a CNAME, replacing any CNAME or AAAA in the same change. It is refreshed every `RENEW_INTERVAL`
115+
- Where a CAA record shares the name (`SET_CAA=true` under `DELEGATION_ZONE`, where it is the default, or on a non-wildcard domain, or any CAA already there), the gateway pointer is an A record to the resolved gateway IP instead of a CNAME, replacing any CNAME or AAAA in the same change. It is refreshed every `RENEW_INTERVAL`
116116
- Wildcard record names are listed by Route53 as `\052.<domain>`; they are matched as `*.<domain>`
117117
- It is essential that the AWS service account used can only assume the limited role. See cloudformation example.
118118

‎custom-domain/dstack-ingress/README.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -175,7 +175,7 @@ otherwise arbitrary — the gateway answers every name under its domain.
175175
|----------|---------|-------------|
176176
| `PORT` | `443` | HAProxy listen port |
177177
| `ROUTING_MAP` | | Multi-domain routing: `domain=host:port` per line. A wildcard entry (`*.example.com=host:port`) matches one label, and exact entries take precedence over it. Unmatched names go to the first entry. It routes only: every name still needs to be in `DOMAIN`/`DOMAINS` to get a certificate |
178-
| `SET_CAA` | `false` | Enable CAA DNS record (dns-01 only; tls-alpn-01 cannot write DNS). Replaces existing values with the same tag; for a wildcard whose base is a zone apex this restricts the whole zone (see [Wildcard Domain Support](#wildcard-domain-support)). Except on Cloudflare, a non-wildcard domain is then published as an A record to the gateway's current address instead of a CNAME, because a CNAME would hide the CAA beside it; the address is refreshed every `RENEW_INTERVAL`. The same happens whenever a CAA record already exists at the name, whoever wrote it, so turning `SET_CAA` off later keeps the A record until you delete the CAA. Linode always uses an A record. With `DELEGATION_ZONE` the CAA is always published, whatever this says |
178+
| `SET_CAA` | `false` | Enable CAA DNS record (dns-01 only; tls-alpn-01 cannot write DNS). Replaces existing values with the same tag; for a wildcard whose base is a zone apex this restricts the whole zone (see [Wildcard Domain Support](#wildcard-domain-support)). Except on Cloudflare, a non-wildcard domain is then published as an A record to the gateway's current address instead of a CNAME, because a CNAME would hide the CAA beside it; the address is refreshed every `RENEW_INTERVAL`. The same happens whenever a CAA record already exists at the name, whoever wrote it, so turning `SET_CAA` off later keeps the A record until you delete the CAA. Linode always uses an A record. Defaults to `true` with `DELEGATION_ZONE`; turning it off there lets anyone who can write the delegation zone get certificates for the domain. Either way, the CAA in force is checked before every issuance and a domain it refuses is not sent to the CA |
179179
| `TXT_PREFIX` | `_dstack-app-address` | DNS TXT record prefix |
180180
| `ACME_EMAIL` | | ACME contact address, in either mode. `CERTBOT_EMAIL` is the historical name and still works. See [below](#the-acme-contact-address-is-optional-and-public) — it is optional, and published |
181181
| `ACME_STAGING` | `false` | Use Let's Encrypt staging, in either mode. `CERTBOT_STAGING` is the historical name and still works |
@@ -188,7 +188,7 @@ otherwise arbitrary — the gateway answers every name under its domain.
188188
| `DNS_WEBHOOK_URL` | | `DNS_SETUP_MODE=webhook`: endpoint to notify |
189189
| `DNS_WEBHOOK_TOKEN` | | Shared secret; the payload is HMAC-SHA256 signed with it |
190190
| `DNS_WEBHOOK_TIMEOUT` | `15` | Seconds per webhook POST attempt |
191-
| `DOH_RESOLVERS` | Google + Cloudflare | Comma-separated DoH endpoints used to verify records. `DOH_RESOLVER` is the historical name and still works |
191+
| `DOH_RESOLVERS` | Google + Cloudflare | Comma-separated DoH endpoints used to verify records, and to check CAA before issuing. `DOH_RESOLVER` is the historical name and still works |
192192
| `TLSALPN_ADDRESS` | `127.0.0.1` | Address lego's ACME responder binds to, and haproxy forwards `acme-tls/1` connections to |
193193
| `TLSALPN_PORT` | `9443` | Loopback port lego's ACME responder binds to |
194194
| `TLS_TERMINATE_PORT` | `9444` | Loopback port the TLS frontend moves to in tls-alpn-01 mode |
@@ -243,12 +243,12 @@ dstack-ingress publishes what they point at, in the delegation zone:
243243

244244
```
245245
svc.example.com.deleg.example.net A <address of GATEWAY_DOMAIN>
246-
svc.example.com.deleg.example.net CAA 0 issue "letsencrypt.org;validationmethods=dns-01;accounturi=…"
246+
svc.example.com.deleg.example.net CAA 0 issue "letsencrypt.org;validationmethods=dns-01;accounturi=…" (SET_CAA, on by default here)
247247
_dstack-app-address.svc.example.com.deleg.example.net TXT "<app-id>:<port>"
248248
_acme-challenge.svc.example.com.deleg.example.net TXT <challenge> (transient)
249249
```
250250

251-
The gateway pointer and the CAA share a name. Cloudflare answers a CAA query
251+
With `SET_CAA` the gateway pointer and the CAA share a name. Cloudflare answers a CAA query
252252
with the CAA beside a CNAME, so the pointer stays a CNAME there. Elsewhere it is
253253
an A record: by the DNS rules resolvers follow a CNAME without looking beside it,
254254
and Route53 and Linode refuse the pair. The address

‎custom-domain/dstack-ingress/TESTING.md‎

Lines changed: 15 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -284,8 +284,9 @@ These fail fast and are cheap, so run them on every change:
284284
| Scenario | Expected |
285285
|---|---|
286286
| tls-alpn-01 + a wildcard domain | Refused before any ACME call, citing RFC 8737 |
287-
| Delegation, CAA in the delegated zone changed to forbid the CA | Blocked before any ACME attempt — proves the published CAA is reachable through the CNAME |
288-
| Delegation with no CAA record at all | Proceeds, reporting `no CAA record set; issuance is unrestricted` — see below |
287+
| Delegation, `SET_CAA=false`, CAA in the delegated zone changed to forbid the CA | Blocked before any ACME attempt — proves a CAA there is reachable through the CNAME. With `SET_CAA` on, the next pass replaces it instead |
288+
| dns-01, `SET_CAA=false`, a CAA naming another CA or pinned to another account | Blocked before any ACME attempt with `CAA forbids issuing <domain> with this account` and the reason quoted back |
289+
| Delegation, `SET_CAA=false`, no CAA record at all | Proceeds, reporting `no CAA record set; issuance is unrestricted` — see below |
289290
| Delegation, later pass, gateway CNAME broken | Renewal proceeds — that record is for serving, and dns-01 does not use it |
290291
| A CAA record with `validationmethods=dns-01`, mode tls-alpn-01 | Blocked with the restriction quoted back |
291292
| TXT holding the wrong value | Reported as `TXT present but wrong value`, not "missing" |
@@ -298,7 +299,7 @@ it is observable rather than merely argued.
298299
had to create the CAA record, its absence was the state that mattered: nothing
299300
then stopped anyone else who could satisfy the delegated challenge, and the
300301
container held no token to create it. Once the container began publishing the
301-
delegated CAA itself, absence stopped meaning that — it means "not published
302+
delegated CAA itself (`SET_CAA`, the default under delegation), absence stopped meaning that — it means "not published
302303
yet", and the next pass fixes it — so the gate was removed along with
303304
`ALLOW_MISSING_CAA` and dnsguide's `--caa-required`. `check_caa` now follows
304305
RFC 8659 for delegation as it always did elsewhere: an absent CAA record set
@@ -310,19 +311,20 @@ attempt, which is what proves the published record is reachable through the
310311
operator's CNAME. Expect the reason quoted back, e.g. `issue allows
311312
'example.com', not letsencrypt.org`.
312313

313-
Note also that the first pass on a fresh deployment necessarily runs before any
314-
CAA exists: the record names the ACME account, so the account has to exist
315-
first. `process_domain` issues, writes the CAA, and issues again only if the
316-
first attempt failed and had to create the account.
314+
Note also that the CAA record names the ACME account, so `process_domain`
315+
registers the account on its own first. Every dns-01 pass runs the same steps
316+
in the same order — register, publish the records (CAA with `SET_CAA`), check
317+
that the CAA in force lets this account issue, issue once — so even a fresh
318+
deployment's first certificate is issued under the pin.
317319

318320
## Traps
319321

320-
**Start from a clean zone.** Records left over from an earlier run change which
321-
branch executes. A stale CAA record naming a previous ACME account makes dns-01's
322-
first issuance attempt fail and the second succeed — which happens to be the path
323-
where the code was correct, hiding a bug in the ordinary path where the first
324-
attempt succeeds. Delete test records between runs and verify against the
325-
authoritative nameserver, not a cached resolver.
322+
**Start from a clean zone.** Records left over from an earlier run change what a
323+
pass does. A stale CAA record naming a previous ACME account is rewritten with
324+
`SET_CAA=true`, but resolvers may keep the old one for its TTL, so the CAA check
325+
fails a pass or two before issuance; with `SET_CAA=false` it blocks for good.
326+
Delete test records between runs and verify against the authoritative
327+
nameserver, not a cached resolver.
326328

327329
**One resolver is not enough to trust either answer.** Public resolvers cache
328330
negative answers for the zone's SOA minimum, so a resolver queried before a

‎custom-domain/dstack-ingress/scripts/cert-lib.sh‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -90,10 +90,9 @@ run_pass() {
9090
# "nothing to do", and under `set -e` a bare non-zero would kill the
9191
# whole script on every quiet renewal pass.
9292
if ( process_domain "$domain" ); then status=0; else status=$?; fi
93-
# A step after issuance can still fail (a CAA write, a timeout that
94-
# kills the client after it saved the certificate). The new certificate
95-
# is on disk by then and the next pass will find it current, so apply
96-
# it now or never.
93+
# Issuance can fail after saving the certificate (a timeout that kills
94+
# the client). The new certificate is on disk by then and the next pass
95+
# will find it current, so apply it now or never.
9796
[ "$(cert_checksum "$domain")" = "$before" ] || changed=1
9897
case $status in
9998
0) changed=1 ;;

‎custom-domain/dstack-ingress/scripts/certman.py‎

Lines changed: 59 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -706,6 +706,51 @@ def apply_renewal_window(self, domain: str) -> None:
706706
return
707707
print(f"Renewal window for {domain} set to {days} days before expiry")
708708

709+
def register_account(self, email: str) -> bool:
710+
"""Create the ACME account without issuing, so the CAA can name it
711+
before the first order."""
712+
cmd = self._get_certbot_command() + [
713+
"register", "--non-interactive", "--agree-tos", "--no-eff-email",
714+
"--server", acme_server()]
715+
# Contactless unless an address is given; see _build_certbot_command.
716+
cmd += ["--email", email] if email else ["--register-unsafely-without-email"]
717+
try:
718+
result = subprocess.run(cmd, capture_output=True, text=True, timeout=120)
719+
except subprocess.TimeoutExpired:
720+
print("ACME account registration timed out", file=sys.stderr)
721+
return False
722+
if result.returncode != 0 or not self.acme_account_exists():
723+
print(f"ACME account registration failed: {result.stderr.strip()}", file=sys.stderr)
724+
return False
725+
print(f"✓ ACME account registered with {acme_server()}")
726+
return True
727+
728+
def _lineage_account_current(self, domain: str) -> bool:
729+
"""Whether `certbot renew` would find the account the lineage names.
730+
731+
certbot renews with the account recorded in the lineage and fails once
732+
that one is gone, and register_account runs before every pass, so
733+
"an account exists" no longer tells. Deterministic certificates are
734+
issued outside any lineage, with whatever account is current. A
735+
lineage that records no account is left to certbot rather than
736+
replaced on every pass.
737+
"""
738+
current = account_file()
739+
if current is None:
740+
return False
741+
if deterministic_key.enabled():
742+
return True
743+
recorded = None
744+
try:
745+
with open(self._renewal_conf_path(domain), encoding="utf-8") as fh:
746+
for line in fh:
747+
match = re.match(r"\s*account\s*=\s*(\S+)", line)
748+
if match:
749+
recorded = match.group(1)
750+
except OSError:
751+
pass
752+
return recorded is None or recorded == os.path.basename(os.path.dirname(current))
753+
709754
def acme_account_exists(self) -> bool:
710755
"""Check if an ACME account exists for the current server."""
711756
return account_file() is not None
@@ -731,14 +776,14 @@ def run_action(
731776
# certbot always records the server; a lineage without one has
732777
# a broken renewal config, which re-issuing would not repair.
733778
server = acme_server()
734-
if server == acme_server() and self.acme_account_exists():
779+
if server == acme_server() and self._lineage_account_current(domain):
735780
success, renewed = self.renew_certificate(domain, email)
736781
return success, renewed # Only generate evidence if actually renewed
737782
if server != acme_server():
738783
print(f"Certificate for {domain} came from {server or 'an unknown ACME server'}, "
739784
f"replacing it from {acme_server()}")
740785
else:
741-
print(f"Certificate exists for {domain} but the ACME account is missing, "
786+
print(f"Certificate for {domain} belongs to an ACME account that is gone, "
742787
f"re-obtaining")
743788
success = self.obtain_certificate(domain, email, replace=True)
744789
return success, success
@@ -758,7 +803,7 @@ def main():
758803
)
759804
parser.add_argument(
760805
"action", choices=["obtain", "renew", "auto", "setup", "cert-dir",
761-
"account-file"],
806+
"account-file", "register"],
762807
help="Action to perform"
763808
)
764809
parser.add_argument("--domain", help="Domain name")
@@ -775,6 +820,8 @@ def main():
775820
sys.exit(1)
776821
print(path)
777822
return
823+
if args.action == "register" and account_file():
824+
return # every pass asks; skip loading the provider SDKs
778825
if args.action == "cert-dir":
779826
if not args.domain:
780827
parser.error("--domain is required")
@@ -793,6 +840,15 @@ def main():
793840
print(f"Setup completed for {manager.provider_type} provider")
794841
return
795842

843+
# The contact address is optional; see _build_certbot_command.
844+
if not args.email:
845+
args.email = os.environ.get("ACME_EMAIL") or os.environ.get(
846+
"CERTBOT_EMAIL", ""
847+
)
848+
849+
if args.action == "register":
850+
sys.exit(0 if manager.register_account(args.email) else 1)
851+
796852
# Domain is required for certificate operations
797853
if not args.domain:
798854
print(
@@ -801,12 +857,6 @@ def main():
801857
)
802858
sys.exit(1)
803859

804-
# The contact address is optional; see _build_certbot_command.
805-
if not args.email:
806-
args.email = os.environ.get("ACME_EMAIL") or os.environ.get(
807-
"CERTBOT_EMAIL", ""
808-
)
809-
810860
success, needs_evidence = manager.run_action(
811861
args.domain, args.email, args.action
812862
)

0 commit comments

Comments
 (0)