From ccbd9cf2b05c8e445b392c2a14d8f06dd030f2b5 Mon Sep 17 00:00:00 2001 From: Mark C Date: Wed, 23 Sep 2026 12:28:38 +0100 Subject: [PATCH] Wave 8: UDP, QUIC, IKEv2, tunnels, the cleartext inventory, and the docs The last wave of the backlog. PR-40, PR-41, PR-42 and PR-43, plus a documentation pass, built by five agents concurrently and integrated here. 1149 tests -> 1508. pcapscan/datagrams.py -- the UDP seam (new, integrator) pcapscan/quic.py -- PR-40, QUIC Initial decryption (RFC 9001) pcapscan/ikev2.py -- PR-41, IKEv2 SA proposals over UDP pcapscan/tunnels.py -- PR-42, GRE/ERSPAN/VXLAN/GENEVE (#12, offline) pcapscan/cleartext.py -- PR-43, the inventory of what is not protected docs/ -- eight pages; README cut from 344 lines to 228 -------------------------------------------------------------------------- The blocker: this tool could not read a UDP packet -------------------------------------------------------------------------- `decode_frame` walked to TCP and returned None for everything else, with no counter and no mention in the stats. Measured over the capture corpus that was **30,530 of 159,929 packets -- 19% -- dropped in silence**, including 11,722 QUIC packets on port 443. A monitor whose purpose is to report what cryptography is in use was blind to the transport that now carries a growing share of the web's TLS 1.3 handshakes, and the three UDP protocols this wave adds would each have had to invent their own plumbing to get at it. So the seam came first, and it is the integrator's work rather than any one PR's: * **`decode_datagram`**, sharing the link-layer, VLAN, IPv4-options and IPv6-extension-chain walk with `decode_frame` through a new `_walk_network`. One walk, not two. Every defect that walk has ever had -- the fixed 14-byte Ethernet header, the fixed 20-byte IPv4 header, the unwalked IPv6 chain -- produced *plausible wrong offsets* rather than an error, and a second copy would be a second place to reintroduce them. It also refuses non-initial IPv4 fragments, which TCP never needed and a near-MTU QUIC Initial very much does. * **`pcapscan/datagrams.py`**, which keys UDP flows and *discovers* handlers rather than requiring registration: `HANDLER_MODULES` names the modules, each is imported if present and skipped if not. That is what let four people write four protocols at once without editing one shared list. * Wired into `SessionBuilder`, so the CLI, `iter_sessions` and the upload sandbox all get UDP without further change. -------------------------------------------------------------------------- PR-40 -- QUIC, and the post-quantum traffic nobody could see -------------------------------------------------------------------------- This is the headline and it is a large one. RFC 9001 derives Initial packet protection from a **published constant salt** and the client's Destination Connection ID, which is in the clear in the header. No keys are needed and none are used: the secrecy of a QUIC handshake never rested on the Initial keys. The handler walks long headers, removes header protection, opens the packet with AES-128-GCM, reassembles CRYPTO frames by offset across coalesced packets, and hands the result to **`pcapscan.sessions.Session`** -- the same object the TCP path builds. That reuse is the design. There is no second ClientHello parser and no second document shape, so JA4, the extension census, ECH, the key-share group, the resumption logic and the post-quantum classification all arrived with **no change to `cryptomon/analysis.py`, the CBOM, the CSV export or the dashboard**. Over the corpus: 86 QUIC flows claimed, 468 Initials seen, **468 decrypted, 0 undecryptable**, 86 ClientHellos and 60 ServerHellos recovered. 69 of the 86 hellos were still in flight across more than one packet, so a single-packet reader would have got a fragment of each. clients offering a post-quantum hybrid 69 / 86 = 80.2% (TCP: 71.6%) servers accepting one 35 / 60 = 58.3% (TCP: 20.4%) ECH offered 74 / 86 = 86.0% (TCP: 36.5%) ALPN h3 on all 86; no TCP hello offers it Whole-corpus effect, measured before and after: hybrid key exchanges 81 -> 116 (+43%) quantum-safe fraction 14.4% -> 18.6% TLS 1.3 sessions 499 -> 585 (+17%) **Sixty-nine post-quantum key exchanges were happening over QUIC and this tool could not see one of them.** Thirty-five of them completed. The honest bound on that: of the 11,722 QUIC packets, 10,984 are short-header 1-RTT and will never be read by any keyless tool. They are now counted, not read. The reachable population is 762 long-header packets and the 468 Initials inside them, and that is the whole handshake, which is the point. Verification was not straightforward and is worth recording. **tshark 3.4 on this machine cannot decrypt QUIC v1** -- it predates RFC 9000, carries draft-29's salt, and fails every Initial with a checktag error while calling version 1 "Unknown". Rather than take that as a disagreement, the agent decrypted one packet under three candidate salts to show which was right, then built a different oracle: it re-framed all 86 decrypted CRYPTO streams as ordinary TLS-over-TCP in a synthetic capture and let tshark's TLS dissector read them. 86/86 parsed as ClientHello, SNI agreed 86/86, and tshark reported key-share code points 0x6399 and 0x11EC exactly where we name X25519Kyber768Draft00 and X25519MLKEM768. RFC 9001 Appendix A's published test vectors are pinned in the test file as well. -------------------------------------------------------------------------- PR-41 -- IKEv2 -------------------------------------------------------------------------- IKEv2 is the one protocol here that *states* its cryptography instead of implying it: RFC 7296 puts the encryption algorithm, the PRF, the integrity algorithm and the key exchange on the wire as four separately numbered transforms, in the clear, before anything is encrypted. So almost nothing is inferred. Transform names are chosen to be exactly the strings `cryptomon/analysis.py` already knows, so **no additions to the marker tables were needed**. The RFC 9370 case is the good one: a `KE=x25519` with `ADDKE1=ml-kem-768` is emitted as the single name `x25519+ml-kem-768`, which normalises to `x25519mlkem768`, in which the existing table finds one post-quantum marker and one classical one and answers **hybrid** -- the true answer, and the one neither half gives alone. Port 4500 carries IKE, ESP and NAT keepalives on the same port, distinguished by a four-byte non-ESP marker. Getting that wrong means parsing ciphertext as an IKE header and reporting a proposal made of noise; a marker-shaped datagram is therefore tried both ways. ESP is reported as present-and-opaque, which is a useful finding rather than a failure -- and `opaque_flows` is a new readiness field so that "we could not see it" is never reported as "there was none". **There is no IKEv2 in the corpus**, verified independently rather than taken on trust: 0 packets on UDP 500 or 4500 in all 160,221. Everything quoted about its behaviour comes from eight generated fixtures. The one corpus-backed claim is the negative: 30,531 real datagrams offered to the detector, 0 claimed, 0 exceptions. The CBOM now fills `ikev2TransformTypes` -- `encr`, `prf`, `integ`, `ke` -- which is a CycloneDX 1.6 field this project has had a schema for and nothing to put in. `esn` and `auth` are deliberately left empty: ESN is not carried, and IKEv2's authentication method is negotiated inside the encrypted IKE_AUTH, so a passive observer never sees it. Emitting either would invent a value the traffic did not contain. -------------------------------------------------------------------------- PR-42 -- tunnels (#12, the offline half) -------------------------------------------------------------------------- Every measurement in this repository comes from a capture taken on an endpoint. That is not how anyone monitors a corporate network: they mirror a port, and the traffic arrives wrapped in GRE inside IP. To this tool, every one of those frames was unreadable. Unwrapping happens **before** framing, in its own module, not as recursion inside `decode_frame`. Peeling a tunnel does not yield a transport header -- it yields another whole frame needing the entire walk run over it again -- and putting that inside `decode_frame` would hand every caller, the live eBPF adapter included, a recursion it did not ask for with an attacker-chosen depth. The stage generalises: GRE (RFC 2784/2890, with the variable header its flag bits imply), ERSPAN Types I, II and III, VXLAN, GENEVE, IP-in-IP and 6in4, bounded to four levels. ERSPAN Type I is the trap and is handled: it is distinguished from Type II only by the GRE sequence flag, so a parser assuming a header is always present eats eight bytes of the real Ethernet frame and then decodes plausible nonsense. The proof is the one that matters: `tls12_certificate.pcap` wrapped frame-by-frame in ERSPAN Type II. Today's pipeline drops 19 of 19 frames and reports nothing. With the stage, the pipeline returns the **identical session document** -- `==` on the dict, same ciphersuite, same `sha384.badssl.com`, same timestamp to the microsecond. The ERSPAN truncation bit is carried through to a counter and onto the session, because a switch saying "I cut this frame short" is *labelled* missing data, and both times this project has reported a wrong answer that looked right -- truncated ClientHellos parsed as whole, tshark's desegmenter giving up in silence -- the data was missing without saying so. **The live eBPF path is still blind to tunnels**, and #12 should not be closed as though it covered both halves. -------------------------------------------------------------------------- PR-43 -- what is not protected at all -------------------------------------------------------------------------- Scope changed on evidence. The backlog's PR-43 leads with DTLS; there is not one DTLS record in the corpus, nor any WireGuard. What is there is 30,531 UDP datagrams, most of them entirely unencrypted. "This uses RSA-2048, which a quantum computer breaks" and "this uses nothing, which anyone on the segment breaks today" are both CBOM findings, and the second is actionable this afternoon. Fourteen protocols identified by content: DNS (with DNSSEC state read properly, from the EDNS0 DO bit and the RRSIG/DS/DNSKEY records), mDNS, LLMNR, NetBIOS-NS and -DGM, DHCP, NTP, CLDAP, STUN/TURN, HSRP v1 and v2, SNMP v1/v2c/v3, syslog, SSDP and BigFix. 1,058 flows claimed over the corpus, **zero false positives**, and no flow on port 443 taken. Of the non-QUIC UDP: **61.4% has no confidentiality and none of it is encrypted.** The four-rung protection ladder earns its place on the HSRP result -- all 15,867 hellos carry an all-zero plaintext auth field *and* a keyed-MD5 authentication TLV, so the gateway redundancy is authenticated by a 1991 hash. That is `obsolete`, which is neither `none` nor `authenticated only`, and the distinction is the finding. Also: **zero DNSSEC and zero EDNS0** in 2,134 DNS messages -- not one client on this network even asks for signatures. 764 of those queries (35.8%) are for HTTPS resource records, which is where the ECH configuration lives: this corpus asks for ECH keys in the clear 764 times. **Nothing identifying is recorded.** Not a query name, not an mDNS instance name, not a NetBIOS name, not a DHCP hostname. Service *types* and record *types* survive; names are counted via keyed BLAKE2 tokens under a per-process random key, because a stable digest of a hostname is a hostname when the name space is a corporate network. No community string, HSRP auth string or ICE username is ever written down -- only that one is present and in the clear. -------------------------------------------------------------------------- Documentation -------------------------------------------------------------------------- `README.md` restructured to route rather than explain (344 -> 228 lines), and `docs/` added: install, configuration, offline analysis, the service, the live sensor, reading a report, architecture, and a 623-line troubleshooting page mined from this repository's own commit history and module docstrings. Every command with a prompt was run; anything needing Linux, root, bcc or a live interface is marked "not verified on this machine" rather than implied. Thirteen inaccuracies in the old README are fixed, among them: port 990 documented as sftp when it is FTPS, "IPv6 support" still listed as a TODO four waves after it shipped, `http://0.0.0.0:8000/docs` offered as a browsable address, and an `enp0s1` interface default that does not exist. -------------------------------------------------------------------------- Defects found and fixed -------------------------------------------------------------------------- * **Every non-SSH record was counted as a TLS session.** `Summary.add` fell through to `_add_tls`, so the 1,058 cleartext UDP flows arrived in the readiness report as resumed TLS sessions -- moving the quantum-safe fraction by inventing sessions that never negotiated anything. Found independently by two agents. Fixed with an explicit branch, plus `_add_ikev2` for the protocol that does have a key exchange to classify. * **The upload sandbox was TCP-only.** `pcapscan/sandbox.py` never got the UDP seam, so a capture uploaded through the browser was analysed by a strictly weaker parser than the same file on the command line -- silently, because a report of nothing looks like a capture with nothing in it. * **`--no-certificates` lost the data it promised to keep.** `--help` said "keep the raw chain"; the flag passed `lambda _body: []`, so the chain was parsed to nothing and the DER discarded. Now a `KEEP_DER` sentinel. * **Four scripts were committed without their executable bit**, so `./ubuntu-setup.sh` was "permission denied" on a fresh clone -- and `deploy/README.md`'s own quickstart begins `sudo ./create-service.sh`. * **`BrokenPipeError` reached stderr** on `| head` and `| mongoimport`, both of which the README recommends: catching the write is not enough, because CPython flushes stdout again at shutdown where no `except` can reach it. * **The UDP flow counters did not sum.** `flows_unrecognised` was only incremented at the 64-datagram cap, so a two-datagram exchange was counted nowhere and 1,071 of 1,164 flows fell in the gap. A stats line whose parts do not add up is worse than no stats line. Now `flows_undetected` closes it: 1058 + 86 + 8 + 18 + 2 = 1172. * **`DatagramRouter.push` was unguarded** while `_detect` was guarded. One handler raising ended the capture for every other protocol sharing the loop. * **`DES`, `DES40` and the IKEv2 `ENCR_DES` family had no strength entry**, so they were reported with no symmetric strength at all. This also fixes `TLS_RSA_EXPORT_WITH_DES40_CBC_SHA` on the TLS side. * **The upload form read "up to 0 MB"** below a 1 MiB cap, and said MB for MiB. * And one of mine, in the seam: `self.router = router or None`, three lines below a comment warning that `DatagramRouter` defines `__len__` so an empty one is falsy. The comment now names the incident. -------------------------------------------------------------------------- Verified -------------------------------------------------------------------------- pytest -q 1508 passed, 19 skipped (was 1149, 18) pytest -m smoke -q 1370 passed (was 1032) with a corpus MongoDB 1523 passed, 4 skipped compileall clean fuzz, 11 targets 249,193 cases, 0 failures tshark oracle no drift The whole real corpus, end to end: 2,404 sessions (1,346 TLS, 1,058 cleartext), 624 key exchanges performed, 116 hybrid, 508 classical, 61 post-quantum offers refused, quantum-safe fraction 18.6%. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 511 ++--- create-service.sh | 0 cryptomon/analysis.py | 127 +- cryptomon/parsers/framing.py | 189 +- docs/README.md | 54 + docs/architecture.md | 326 +++ docs/configuration.md | 184 ++ docs/install.md | 255 +++ docs/live-sensor.md | 217 ++ docs/offline-analysis.md | 321 +++ docs/reading-a-report.md | 405 ++++ docs/service.md | 369 ++++ docs/troubleshooting.md | 611 ++++++ fapi/app/templates/upload.html | 2 +- parse-pcap.sh | 0 pcapscan/cbom.py | 95 +- pcapscan/cleartext.py | 2026 ++++++++++++++++++ pcapscan/cli.py | 46 +- pcapscan/datagrams.py | 287 +++ pcapscan/export.py | 27 +- pcapscan/ikev2.py | 1493 +++++++++++++ pcapscan/quic.py | 1029 +++++++++ pcapscan/sandbox.py | 24 +- pcapscan/sessions.py | 86 +- pcapscan/tunnels.py | 560 +++++ start_cryptomon.sh | 0 tests/README.md | 4 +- tests/fixtures/ikev2/ikev2_esp_only.pcap | Bin 0 -> 612 bytes tests/fixtures/ikev2/ikev2_ikev1.pcap | Bin 0 -> 284 bytes tests/fixtures/ikev2/ikev2_ipv6.pcap | Bin 0 -> 758 bytes tests/fixtures/ikev2/ikev2_legacy.pcap | Bin 0 -> 619 bytes tests/fixtures/ikev2/ikev2_natt_4500.pcap | Bin 0 -> 1123 bytes tests/fixtures/ikev2/ikev2_post_quantum.pcap | Bin 0 -> 906 bytes tests/fixtures/ikev2/ikev2_pq_refused.pcap | Bin 0 -> 798 bytes tests/fixtures/ikev2/ikev2_sa_init.pcap | Bin 0 -> 994 bytes tests/fixtures/tunnels/erspan2_tls12.pcap | Bin 0 -> 8199 bytes tests/fixtures/tunnels/tunnel_types.pcap | Bin 0 -> 1672 bytes tests/test_cleartext.py | 1039 +++++++++ tests/test_ikev2.py | 1109 ++++++++++ tests/test_quic.py | 1202 +++++++++++ tests/test_tunnels.py | 947 ++++++++ tests/tools/fuzz.py | 116 +- tests/tools/make_ikev2_fixtures.py | 461 ++++ ubuntu-setup.sh | 0 44 files changed, 13743 insertions(+), 379 deletions(-) mode change 100644 => 100755 create-service.sh create mode 100644 docs/README.md create mode 100644 docs/architecture.md create mode 100644 docs/configuration.md create mode 100644 docs/install.md create mode 100644 docs/live-sensor.md create mode 100644 docs/offline-analysis.md create mode 100644 docs/reading-a-report.md create mode 100644 docs/service.md create mode 100644 docs/troubleshooting.md mode change 100644 => 100755 parse-pcap.sh create mode 100644 pcapscan/cleartext.py create mode 100644 pcapscan/datagrams.py create mode 100644 pcapscan/ikev2.py create mode 100644 pcapscan/quic.py create mode 100644 pcapscan/tunnels.py mode change 100644 => 100755 start_cryptomon.sh create mode 100644 tests/fixtures/ikev2/ikev2_esp_only.pcap create mode 100644 tests/fixtures/ikev2/ikev2_ikev1.pcap create mode 100644 tests/fixtures/ikev2/ikev2_ipv6.pcap create mode 100644 tests/fixtures/ikev2/ikev2_legacy.pcap create mode 100644 tests/fixtures/ikev2/ikev2_natt_4500.pcap create mode 100644 tests/fixtures/ikev2/ikev2_post_quantum.pcap create mode 100644 tests/fixtures/ikev2/ikev2_pq_refused.pcap create mode 100644 tests/fixtures/ikev2/ikev2_sa_init.pcap create mode 100644 tests/fixtures/tunnels/erspan2_tls12.pcap create mode 100644 tests/fixtures/tunnels/tunnel_types.pcap create mode 100644 tests/test_cleartext.py create mode 100644 tests/test_ikev2.py create mode 100644 tests/test_quic.py create mode 100644 tests/test_tunnels.py create mode 100755 tests/tools/make_ikev2_fixtures.py mode change 100644 => 100755 ubuntu-setup.sh diff --git a/README.md b/README.md index 4c46ad6..6c7c67e 100644 --- a/README.md +++ b/README.md @@ -1,367 +1,246 @@ [![CodeQL](https://github.com/Santandersecurityresearch/CryptoMon/actions/workflows/github-code-scanning/codeql/badge.svg)](https://github.com/Santandersecurityresearch/CryptoMon/actions/workflows/github-code-scanning/codeql) [![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0) - # CryptoMon -Network Cryptography Monitor - using eBPF, written in python. - -**NB - This code is pre-production and is intended for demonstration purposes.** - -This is an demonstration service that allows the interception and analysis of over-the-wire TLS cryptography. - -Cryptomon looks for port 443 traffic, and if found, looks for the 'hello' packets from the client and server. It parses the packet data and then stores it in a MongoDB database that can later be analysed. - -The advantage of using network monitoring alongside the [CodeQL Source Code analysis](https://github.blog/2023-12-05-addressing-post-quantum-cryptography-with-codeql/) we have worked on previously, is that static analysis of code tells you what could be running, whilst over-the-wire monitoring tells you what is actually being negotiated. - -## What is supported - -Currently we support the following protocols and captures: +Network cryptography monitor, in Python and eBPF. -* TLS Handshake data for all TLS versions, inc. proposed ciphersuites and accepted ciphersuites, across several ports: - * 443 (https) - * 990 (sftp) - * 3389 (rdp) - * 8080 (proxy) - * 8443 (proxy) -* TLS Certificates - where they are complete and not affected by TCP fragmentation. -* SSH Handshakes - including kex, server algos, etc. +**This code is pre-production and is intended for demonstration purposes.** -We support a local FastAPI service, as well as logging to file via `TinyDB` or logging to a NoSQL document DB using MongoDB. +CryptoMon answers one question: *what cryptography is actually in use on this +network, and what fraction of it would survive a quantum computer.* It reads +TLS and SSH handshakes — off the wire with an eBPF filter, or out of a packet +capture — works out what each connection negotiated, and reports how much of +it rests on RSA and elliptic-curve Diffie-Hellman, both of which Shor's +algorithm breaks. -**TODO features** include: - -* SSH Key logging option -* IPv6 support - -## Quick start with Docker - -If you have Docker, you need none of the setup below. - -```bash -docker build -f docker/Dockerfile.offline -t cryptomon-offline . +It is the counterpart to the [CodeQL source-code +analysis](https://github.blog/2023-12-05-addressing-post-quantum-cryptography-with-codeql/) +we published previously. Static analysis tells you what your code *could* +negotiate. Traffic tells you what it *did* negotiate, against real servers, +with real middleboxes in the way, after whatever the deployment turned off. +The two disagree in both directions, and the disagreement is the interesting +part. -docker run --rm --network none --read-only \ - -v "$PWD:/captures:ro" cryptomon-offline /captures/your-capture.pcap -``` +## What it tells you -That prints what cryptography the handshakes in the capture negotiated: which -key exchanges would survive a quantum computer, which post-quantum offers the -server refused, and what is in the certificate chain. The image is 164MB, runs -as a non-root user with a read-only filesystem and no network at all, and -contains no database, no eBPF, no kernel headers and nothing from -`ubuntu-setup.sh`. +A committed 24-packet fixture, so this is exactly reproducible: -For the API, its browser upload UI and the dashboard, with a MongoDB beside it: +```console +$ python -m pcapscan tests/fixtures/streams/tls13_hello_retry.pcap +pcapscan: read tests/fixtures/streams/tls13_hello_retry.pcap -```bash -cd docker -cp env.example .env # then put a real password in MONGO_PASSWORD -docker compose up --build # http://127.0.0.1:8000/ -``` +Sessions 1 + tls 1 -Both ports bind `127.0.0.1`. The MongoDB password that ships in -`docker/compose.yaml` is `change-me-this-is-not-a-password` — a placeholder -written out in full so it cannot be mistaken for a generated secret. +Key exchange + performed 1 + none (resumed) 0 + post-quantum 0 (0.0%) + hybrid 0 (0.0%) + classical 1 (100.0%) + unknown 0 (0.0%) + quantum-safe 0.0% of key exchanges performed -The live eBPF sensor is a third image and an opt-in compose profile, because -it needs host networking and elevated capabilities. To find out whether eBPF -works on your machine at all: +Post-quantum offers refused by the server (1) + X25519Kyber768Draft00 -> secp256r1 x1 + unreadable (TLS 1.3) 1 sessions -```bash -docker build -f docker/Dockerfile.sensor -t cryptomon-sensor . -docker run --rm --cap-add BPF cryptomon-sensor -``` +TLS versions + TLSv1.3 1 +Algorithms observed + ciphersuite TLS_AES_256_GCM_SHA384 symmetric 1 + key-exchange secp256r1 classical 1 ``` -OK compiles -OK verifies as SOCKET_FILTER (fd=5) -OK verifies as SCHED_CLS (fd=5) -``` - -Then `SENSOR_IFACE=eth0 docker compose --profile sensor up`. See -`docker/README.md` for the capability list, what the host must provide, and -why `--privileged` is the lazy answer rather than the right one. -## Setup +One connection, and the finding this tool exists to produce. The client +offered a post-quantum hybrid group. The server refused it with a +HelloRetryRequest and named a classical curve instead, so the traffic that +followed can be decrypted by anyone who records it and waits for a quantum +computer. A reader that saw only the first ClientHello would have reported +"post-quantum key exchange offered" — the opposite of what happened. -This installs CryptoMon directly on a host, under Ubuntu 24.04 "Noble -Numbat". If you would rather not install anything, the containers above do -all of this and need none of it. +Point it at a real capture and you get the same report over thousands of +connections. Across the twelve corpus captures of everyday desktop +application traffic, measured with the parsers installed at the time of +writing: 624 key exchanges performed, 116 of them a post-quantum hybrid +(18.6%), 508 classical, and 61 post-quantum offers refused by the far end. +Every certificate key in the corpus is classical. -Firstly, `git clone` this repository. The `ubuntu-setup.sh` script will install all the necessary files. +[docs/reading-a-report.md](docs/reading-a-report.md) explains the rest of it, +starting with why `15.3%` is quoted *of key exchanges performed* and never on +its own. -If you wish to run this service all the time in the background, run -`create-service.sh`. It installs two systemd units -- the sensor, which -needs `CAP_BPF` and a network interface, and the API, which needs -neither -- and starts neither of them, so that you can read the unit -files and put the database password in place first. `deploy/README.md` -walks through that, and through putting the service behind nginx on a -subpath. +## Which of these are you? -You will also need to make sure that mongodb is installed and running. Once this is done, you should connect to the instance with `mongosh` and run the following: +Three ways to run this, and they have almost nothing in common. Read the +right-hand column and pick the first row that is true of you. -```python -use cryptomon -db.createCollection('cryptomon') -db.createUser({user: "cryptomonUser", pwd: passwordPrompt(), roles: [{ role: "readWrite", db: "cryptomon" }]}) -``` +| | You need | Use it when | +|---|---|---| +| **[Docker](docs/install.md#docker)** | Docker, and nothing else | You want an answer about a capture file and do not want to install anything. | +| **[The offline analyser](docs/install.md#the-offline-analyser)** | Python 3.10+, no root, no database | You have captures. This is the whole tool for most people and it sees more than the live sensor does. | +| **[The live sensor](docs/install.md#the-live-sensor)** | Linux, bcc, kernel headers, root, MongoDB | You want a continuous picture of a network rather than an answer about a file. | -This creates the `cryptomon` collection that the monitor will use to store information, as well as a read/write user for that database - this will prompt you to create a password. +If the live sensor is what you want and `bcc` will not install, that is +[issue #26](docs/troubleshooting.md#bcc-will-not-install-or-will-not-compile-issue-26), +and it is the most common way to get stuck here. The offline analyser needs +none of it and answers the same question about a capture. -Once this is done you may export these: +### A capture, analysed, with nothing installed ```bash -export DB_URL="mongodb://cryptomonUser:@:27017/cryptomon?retryWrites=true&w=majority" -export DB_NAME="cryptomon" -``` - -**OR** if you are using MongoDB Atlas or some other cloud service: +docker build -f docker/Dockerfile.offline -t cryptomon-offline . -```bash -export DB_URL="mongodb+srv:///cryptomon?retryWrites=true&w=majority" -export DB_NAME="cryptomon" +docker run --rm --network none --read-only \ + -v "$PWD:/captures:ro" cryptomon-offline /captures/your-capture.pcap ``` -The `fapi/config/__init__.py` should pick these settings up. If, for whatever reason, these environment variables are not picked up, you can edit that file manually. - -## Usage +The image runs as a non-root user with a read-only filesystem and no network +at all, and contains no database, no eBPF and no kernel headers. +[`docker/README.md`](docker/README.md) covers the other two images — the API +and the sensor — the compose stack, and the capability list the sensor needs. -Once everything is installed you can run the monitor and FastAPI with: +### A capture, analysed, with Python ```bash -sudo python3 ./cryptomon.py -i & -python3 ./api.py +pip install cryptography +python -m pcapscan your-capture.pcap ``` -Where `` should be replaced with the network interface to be monitored (`enp0s1` by default.) +No root, no eBPF, no interface, no database, on any platform Python runs on. +`cryptography` is the only third-party package it uses, and it is used for +one thing: parsing the X.509 chain. Without it the analyser still runs and +the certificate section of the report is silently empty — see +[docs/troubleshooting.md](docs/troubleshooting.md#the-report-has-no-certificates-in-it). +See [docs/offline-analysis.md](docs/offline-analysis.md) for the output +formats and the pipelines — NDJSON into MongoDB, CycloneDX CBOM, several +captures at once, reading from a pipe. -If you have installed `cryptomon` as a service, then you do not need to run the first line. To check the monitor is working you can run `db.cryptomon.count({})` from `mongosh` to see if the record count is increasing. - -## PCAP Files +### The service, the dashboard and the upload page ```bash -python3 -m pcapscan test.pcap -``` - -`pcapscan` reads the capture directly. It needs no root, no eBPF, no -interface and no database, and it runs on any platform Python does. - -It also sees considerably more than the live monitor can. The eBPF path -receives one packet at a time, so it can only read a handshake that starts at -the beginning of a TCP payload and finishes inside the same packet. `pcapscan` -reassembles the stream first, which means: - -* **Whole ClientHellos.** A modern hello with post-quantum key shares is - around 1.9KB and arrives in two segments; the live path reads the first one - and records whichever extensions happened to fit. -* **Whole certificate chains**, including intermediates. A chain is several - kilobytes and has never once fitted in a single packet. -* **Both halves of a HelloRetryRequest**, so a post-quantum key exchange that - the *server refused* is reported as refused rather than as offered. - -Measured against `tshark` over the project's capture corpus, the two agree -exactly on 441 ClientHellos and 93 certificate messages. - -### Output formats - -```bash -python3 -m pcapscan capture.pcap # readable report -python3 -m pcapscan capture.pcap -f csv -o out.csv # for a spreadsheet -python3 -m pcapscan *.pcapng -f ndjson | mongoimport --collection cryptomon -zcat big.pcap.gz | python3 -m pcapscan - -f json # from a pipe +export DB_URL="mongodb://127.0.0.1:27017/cryptomon" +export DB_NAME="cryptomon" +python api.py ``` -pcap and pcapng are both read, gzipped or not, and several captures can be -given at once and analysed as one body of traffic. The NDJSON records use the -same document shape the live monitor writes, so they load into the same -collection without translation. - -`--no-certificates` skips X.509 parsing; `--stats` writes the reader and -reassembler counters to stderr; `--max-stream-bytes` raises the per-direction -reassembly buffer for captures with unusually large certificate chains. +Then `http://127.0.0.1:8000/`. The dashboard is at `/`, the browser upload +page at `/analyse/`, the JSON rollups at `/stats/` and the raw documents at +`/data/`. It binds loopback, and writes are refused, until you say +otherwise. See [docs/service.md](docs/service.md), and +[deploy/README.md](deploy/README.md) before you put it in front of anybody. -### Replaying over loopback +### The live sensor ```bash -./parse-pcap.sh test.pcap -``` - -This replays the capture over the loopback interface for the live eBPF -monitor to parse, which exercises the same path production uses. It needs -root and the data environment variables set, and it sees only what a -single-packet reader can see -- prefer `pcapscan` unless you are specifically -testing the live path. - -## The dashboard - -Start the API and open `http://127.0.0.1:8000/`. It answers the question the -project exists for -- what fraction of this traffic would survive a quantum -computer -- with the denominator beside it, because 81 hybrid key exchanges -is 14% of the sessions that performed one and 6% of all sessions, and those -are different claims about the same estate. Below that: key exchange over -time by verdict, ciphersuites, TLS versions, certificate keys, JA4 client -fingerprints, Encrypted ClientHello uptake, and TLS alerts with the direction -they came from. - -The charts are server-rendered inline SVG. There is no JavaScript framework, -nothing vendored and nothing fetched from a CDN, and the page renders in full -with JavaScript switched off; the only script is a short polling loop that -refreshes a panel in place. - -**A word on what it shows.** The hosts panel lists server names taken from -SNI, which is browsing history. The service binds loopback by default and -that has not changed, but the first thing an exposed deployment serves at `/` -is a summary of who was talked to -- so put it behind `deploy/nginx/` with an -`API_KEY` set before exposing it. - -The same numbers are available as JSON under `/stats/` for anything that -would rather have them that way. - -## Analysing a capture from the browser - -Start the API and open `http://127.0.0.1:8000/analyse/`. Upload a pcap or -pcapng file and you get the same report `python -m pcapscan` prints, as a -page: what was negotiated, which key exchanges would survive a quantum -computer, and which post-quantum offers the server refused. - -The capture is read in a bounded subprocess — memory, CPU, wall-clock and -size are all capped — and **deleted as soon as it has been analysed**. Only -the report is kept. - -| setting | default | | -|---|---|---| -| `UPLOADS_ENABLED` | `true` | turn the feature off entirely | -| `UPLOAD_DIR` | `/tmp/cryptomon-uploads` | where reports are kept | -| `MAX_UPLOAD_BYTES` | `268435456` | 256 MB, enforced while reading | -| `ANALYSIS_TIMEOUT_SECONDS` | `120` | wall-clock ceiling for one capture | -| `REPORT_RETENTION_HOURS` | `24` | reports are swept after this; `0` keeps them | -| `RETENTION_SWEEP_MINUTES` | `15` | how often the sweep runs | -| `DATA_RETENTION_HOURS` | `0` | **the live collection**; `0` keeps everything | - -The service binds `127.0.0.1` by default, so this is a local tool unless you -put it behind something. **If you expose it, set `API_KEY`** — uploads then -require an `X-API-Key` header, because an open upload endpoint is an open -invitation to spend your CPU and disk. - -### What is kept, and for how long - -A capture's SNI field is browsing history: it records which hosts a machine -contacted and when, including every connection that happened to be in flight -at the time. The report keeps the server names it found. - -Uploaded captures are deleted as soon as they are analysed. Reports are -swept after `REPORT_RETENTION_HOURS`, and the upload form says so *before* -the file is chosen. - -The live MongoDB collection is a separate decision and **does not expire by -default**: silently discarding a monitoring database would destroy the -historical series this project exists to build. Set `DATA_RETENTION_HOURS` -to opt in, which installs a MongoDB TTL index on `expires_at` so the server -does the deleting whether or not the API is running. - -## FastAPI - -To access the FastAPI documentation go to `http://0.0.0.0:8000/docs` to find the documentation for the backend API. - -## Example Data - -### TLS Capture - -A TLS client capture example: - -```json -{ -"_id": "6682cd75393bb4e863fc0c65", -"eth": { - "src": { - "ipv4": "192.168.64.5" - }, - "dst": { - "ipv4": "3.210.189.242" - } -}, -"tls": { - "tls_versions": [ - "TLSv1.3", - "TLSv1.2" - ], - "ciphersuites": [ - "TLS_AES_128_GCM_SHA256", - "TLS_CHACHA20_POLY1305_SHA256", - "TLS_AES_256_GCM_SHA384", - "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", - "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", - "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256", - "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256", - "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", - "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", - "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA", - "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA", - "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA", - "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA", - "TLS_RSA_WITH_AES_128_GCM_SHA256", - "TLS_RSA_WITH_AES_256_GCM_SHA384", - "TLS_RSA_WITH_AES_128_CBC_SHA", - "TLS_RSA_WITH_AES_256_CBC_SHA" - ], - "EtM": false, - "hostname": "ping.chartbeat.net", - "groups": [ - "x25519", - "secp256r1", - "secp384r1", - "secp521r1", - "ffdhe2048", - "ffdhe3072" - ], - "kex_group": "x25519", - "sigalgs": [ - "ecdsa_secp256r1_sha256", - "ecdsa_secp384r1_sha384", - "ecdsa_secp521r1_sha512", - "rsa_pss_rsae_sha256", - "rsa_pss_rsae_sha384", - "rsa_pss_rsae_sha512", - "rsa_pkcs1_sha256", - "rsa_pkcs1_sha384", - "rsa_pkcs1_sha512", - "ecdsa_sha1", - "rsa_pkcs1_sha1" - ] -}, -"ptype": "client", -"ts": 1719848309.166212 -} -``` - -A TLS server hello capture example: - -```json -{ -"_id": "6682cd75393bb4e863fc0c66", -"eth": { - "src": { - "ipv4": "3.210.189.242" - }, - "dst": { - "ipv4": "192.168.64.5" - } -}, -"tls": { - "tls_versions": "TLSv1.2", - "ciphersuite": "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256" -}, -"ptype": "server", -"ts": 1719848309.26233 -} +sudo ./ubuntu-setup.sh # read it first; see docs/install.md +sudo python3 ./cryptomon.py -i eth0 ``` -## Software Bill of Materials (SBOM) -We are firm supporters of the SBOM movement, as it's a key building block in software security and software supply chain risk management. A SBOM is a nested inventory, a list of ingredients that make up software components and as such, here's our recipe: +Linux only. [docs/live-sensor.md](docs/live-sensor.md) covers what it can and +cannot see, and [deploy/README.md](deploy/README.md) covers running it as a +service under systemd with nginx in front. + +## What it reads + +TLS handshakes of every version, on any TCP port for the offline analyser and +on a [configurable list](docs/configuration.md#which-ports-the-sensor-watches) +for the live sensor: the versions and ciphersuites proposed and selected, the +key exchange group, the server name, ALPN, PSK modes, Encrypted ClientHello, +JA4/JA4S fingerprints, plaintext alerts and the direction they came from, and +X.509 certificate chains where the handshake is not encrypted. SSH KEXINIT +algorithm lists, banners, and host key type and size — though there is no SSH +traffic in this project's capture corpus, so that half is checked against +constructed fixtures and real OpenSSH-encoded keys rather than against +captured sessions. + +The offline analyser also reads what arrives over **UDP** — 30,530 datagrams, +19% of the corpus, used to be dropped at the framing layer and now reach a +dispatcher — and unwraps **mirrored and tunnelled traffic** (GRE, ERSPAN, +VXLAN, GENEVE, IP-in-IP) before it frames anything, so a capture taken off a +switch's mirror port reports what it carries. The individual UDP protocol +handlers are landing as this is written; see +[docs/offline-analysis.md](docs/offline-analysis.md#mirrored-and-tunnelled-traffic). +Neither applies to the live sensor, whose kernel filter reads TCP and nothing +else. + +## What it does not read + +Being clear about this is more useful than a feature list. + +* **Anything inside a TLS 1.3 encrypted flight.** The certificate in TLS 1.3 + travels under the handshake keys. CryptoMon records `certificates_unreadable` + for those sessions rather than reporting no certificate, because "not + readable" and "not sent" are different claims. +* **DTLS.** No support, in either path. +* **Encrypted traffic.** There is no decryption anywhere in this project. It + reads the parts of a handshake that are, by design, in the clear. +* **The inner name of an accepted ECH connection.** When a server accepts + Encrypted ClientHello the server name on the wire is a public outer name. + The record says `ech: offered` or `ech: accepted` so that the hostname + carries its own caveat. + +## Documentation + +| | | +|---|---| +| [docs/install.md](docs/install.md) | The three install paths, honestly, including what `ubuntu-setup.sh` actually does | +| [docs/configuration.md](docs/configuration.md) | Every setting, its default, its effect, and when to change it | +| [docs/offline-analysis.md](docs/offline-analysis.md) | `python -m pcapscan`: formats, flags, pipelines | +| [docs/service.md](docs/service.md) | The API, the dashboard, the upload page and `/stats` | +| [docs/live-sensor.md](docs/live-sensor.md) | The eBPF sensor: what it sees and what it costs | +| [docs/reading-a-report.md](docs/reading-a-report.md) | What `X25519MLKEM768`, `hybrid`, `t13d1516h2_…`, `resumed` and `ech: offered` mean | +| [docs/architecture.md](docs/architecture.md) | Why there are two parsing paths, and what each one can see | +| [docs/troubleshooting.md](docs/troubleshooting.md) | The things that go wrong, and what they look like | +| [deploy/README.md](deploy/README.md) | nginx, systemd, subpath mounting, hardening | +| [docker/README.md](docker/README.md) | The three images, capabilities, compose | +| [tests/README.md](tests/README.md) | How the test suite is built and why | + +## One number, two denominators + +The headline this tool produces is a percentage, and a percentage here is +meaningless without the base it is taken over. Across the project's twelve +capture corpus, 81 key exchanges used a post-quantum hybrid group. That is +**14.4% of the 564 sessions that performed a key exchange**, and **6.4% of +the 1260 TLS sessions**, because a resumed session performs no key exchange +at all and 696 of those 1260 performed none. Both numbers are true, they are +not the same claim, and which one is right depends on the question. + +The base has to be named, not assumed, and that is not a stylistic +preference — it is load-bearing. The same run also produces a record for +every cleartext UDP flow it identifies, and on this corpus those outnumber +the TLS sessions: dividing 116 by *that* total gives a smaller percentage +that measures nothing at all, because a DNS lookup was never going to +negotiate a key exchange. Every figure this project prints carries its base +beside it. +[docs/reading-a-report.md](docs/reading-a-report.md) explains why that +matters more than any other idea here. + +## Software Bill of Materials + +We are firm supporters of the SBOM movement, as it is a key building block in +software security and supply chain risk management. `bom.json` in this +repository is CryptoMon's own SBOM. + +Two things about it are worth knowing. It records the PyPI package `bcc`, +which is **not** what the eBPF sensor uses — the sensor uses the +distribution's `python3-bpfcc`, an unrelated package, as +[`docker/README.md`](docker/README.md#what-it-installs-for-the-sbom) records. +And a CBOM produced by this tool from observed traffic +(`python -m pcapscan capture.pcap -f cbom`) is a different document with a +different job: it inventories the cryptography on the *network*, not the +dependencies of *this software*. ![](img/sbom1.png) ![](img/sbom2.png) ![](img/sbom3.png) ![](img/sbom4.png) + +## Licence, citation and security + +GPL v3 — see [LICENSE](LICENSE). To cite this work, see +[CITATION.cff](CITATION.cff). To report a vulnerability, see +[SECURITY.md](SECURITY.md). diff --git a/create-service.sh b/create-service.sh old mode 100644 new mode 100755 diff --git a/cryptomon/analysis.py b/cryptomon/analysis.py index b933dd5..0704558 100644 --- a/cryptomon/analysis.py +++ b/cryptomon/analysis.py @@ -49,6 +49,23 @@ UNKNOWN = 'unknown' NOT_APPLICABLE = 'none' +# Verdicts about *protection*, as distinct from the verdicts above about +# quantum resistance. A protocol can be unbroken by Shor and still have no +# cryptography in it at all, and a report that answers only the first +# question says nothing about the 61% of this corpus's UDP that anyone on +# the segment can read today. `pcapscan/cleartext.py` produces these. +# +# `NOT_APPLICABLE` above is reused as the `none` rung rather than a fourth +# spelling of the same word -- but it is kept in its own field, never fed +# into `key_exchange_verdict`: "a resumed TLS session needed no key +# exchange" and "DHCP has no cryptography" are the same word about very +# different situations. +PROTECTION_OBSOLETE = 'obsolete' +PROTECTION_AUTHENTICATED = 'authenticated only' +PROTECTION_ENCRYPTED = 'encrypted' +PROTECTIONS = (NOT_APPLICABLE, PROTECTION_OBSOLETE, + PROTECTION_AUTHENTICATED, PROTECTION_ENCRYPTED) + # Substrings naming a post-quantum primitive. Matched case-insensitively # against the names this project's own tables produce, which follow the IANA # registry and the draft names still in use for the Kyber round-3 groups. @@ -78,14 +95,15 @@ ('AES_256', 256), ('AES_128', 128), ('CHACHA20', 256), ('CAMELLIA_256', 256), ('CAMELLIA_128', 128), ('ARIA_256', 256), ('ARIA_128', 128), - ('3DES', 112), ('DES_CBC', 56), ('RC4', 128), ('NULL', 0), + ('3DES', 112), ('DES_CBC', 56), ('DES40', 40), ('DES', 56), + ('RC4', 128), ('NULL', 0), ) # Ciphers that are broken today, by classical cryptanalysis, whatever their # nominal key length. Reporting RC4 as "128 bits, 64 after Grover" would be # an answer to the wrong question by a wide margin: it is not waiting for a # quantum computer. -SYMMETRIC_BROKEN = ('RC4', '3DES', 'DES_CBC', 'NULL') +SYMMETRIC_BROKEN = ('RC4', '3DES', 'DES_CBC', 'DES40', 'DES', 'NULL') # Anything at or below this after Grover is not a defensible symmetric # choice for data that has to stay secret. 128 pre-Grover leaves 64. @@ -245,6 +263,11 @@ def __init__(self): self.ech = collections.Counter() self.algorithms = {} self.ssh_kex = collections.Counter() + # Flows whose cryptography exists but cannot be read: a UDP- + # encapsulated ESP tunnel whose IKE_SA_INIT is not in the capture. + # Kept out of `key_exchange_verdict` so that "we could not see it" + # is never reported as "there was none". + self.opaque_flows = 0 # Distinct certificates, by fingerprint. A CBOM needs each one as its # own asset with its own subject and validity, which a count of # "RSA-2048: 274" cannot supply. @@ -256,6 +279,20 @@ def add(self, record): if 'ssh' in record: self._add_ssh(record) return + if 'ikev2' in record: + self._add_ikev2(record) + return + if 'tls' not in record: + # A record from a handler that negotiated neither TLS nor SSH. + # The UDP handlers emit these: an unprotected DNS or HSRP flow + # has no ciphersuite to put in a `tls` block, so it has none. + # Without this branch each one is counted as a TLS session that + # performed no key exchange, and on the capture corpus that is + # 1,058 cleartext UDP flows arriving in the report as resumed + # TLS sessions -- moving the quantum-safe fraction by inventing + # sessions that never negotiated anything. + self.protocols[_record_protocol(record)] += 1 + return tls = record.get('tls') or {} self.protocols['tls'] += 1 self._add_tls(record, tls) @@ -354,6 +391,67 @@ def _add_certificate(self, record, certificate): self._note(signature, 'signature', classify_algorithm(signature), record) + def _add_ikev2(self, record): + """ + An IPsec negotiation, from `pcapscan.ikev2`. + + IKEv2 is the one protocol here that states its cryptography instead of + implying it: RFC 7296 puts the encryption algorithm, the integrity + algorithm, the PRF and the key exchange on the wire as four separately + numbered transforms, in the clear. So almost nothing is inferred here. + + The one thing that still has to be got right is proposed against + selected. An IKE_SA_INIT request is a *menu* -- a strongSwan default + offers a dozen transforms -- and counting a menu as a deployment would + report every group on it as if it were in use. `pcapscan.ikev2` fills + `kex_group` only when a response carrying an SA payload was captured, + and this counts only that. + """ + ike = record.get('ikev2') or {} + self.protocols[_protocol_label(record).lower()] += 1 + if ike.get('opaque'): + # An ESP tunnel. The key exchange did happen -- in an IKE_SA_INIT + # this capture does not contain -- so counting it as `none` would + # file it beside a resumed TLS session, which performed no key + # exchange at all. Different fact, different counter. A readiness + # report should be able to say how much traffic it could not + # account for rather than quietly leaving it out of the totals. + self.opaque_flows += 1 + return + + group = ike.get('kex_group') + verdict = classify_key_exchange(group) + self.key_exchange[group or 'none'] += 1 + self.key_exchange_verdict[verdict] += 1 + if group: + self._note(group, 'key-exchange', verdict, record) + + cipher = ike.get('encryption') + if cipher: + self._note(cipher, 'cipher', SYMMETRIC, record) + label = symmetric_label(cipher) + if label: + self.symmetric_bits[label] += 1 + for name, kind in ((ike.get('prf'), 'prf'), + (ike.get('integrity'), 'integrity')): + if name and name != 'NONE': + self._note(name, kind, SYMMETRIC, record) + + # The finding a readiness report is for, in its IKEv2 spelling: the + # initiator asked for a KEM alongside its group (RFC 9370) and the + # responder answered INVALID_KE_PAYLOAD or picked a proposal without + # one. Same shape as the TLS HelloRetryRequest downgrade above. + offered = ike.get('offered_kex_group') + if offered and classify_algorithm(offered) in (POST_QUANTUM, HYBRID): + forced = ike.get('retry_kex_group') or group + if classify_algorithm(forced) == CLASSICAL: + self.downgrades.append({ + 'hostname': ike.get('responder'), + 'offered': offered, + 'forced': forced, + 'ts': record.get('ts'), + }) + def _add_ssh(self, record): self.protocols['ssh'] += 1 ssh = record.get('ssh') or {} @@ -416,6 +514,7 @@ def readiness(self): 'deprecated_tls_versions': sum( self.tls_versions[v] for v in DEPRECATED_TLS_VERSIONS), 'broken_symmetric_ciphers': weak_symmetric, + 'opaque_flows': self.opaque_flows, } def as_dict(self): @@ -450,10 +549,32 @@ def as_dict(self): MAX_CERTIFICATES = 512 +def _record_protocol(record): + """ + What a record with neither a `tls` nor an `ssh` block calls itself. + + Named from the record rather than assumed, so a handler added later does + not need this function changed to be counted correctly. + """ + cleartext = record.get('cleartext') + if isinstance(cleartext, dict) and cleartext.get('protocol'): + return str(cleartext['protocol']) + for name in ('quic', 'dtls', 'ikev2', 'cleartext'): + if name in record: + return name + return UNKNOWN + + def _protocol_label(record): - """'TLSv1.3', 'SSH' -- how this record's protocol should be named.""" + """'TLSv1.3', 'SSH', 'IKEv2' -- how this record's protocol is named.""" if 'ssh' in record: return 'SSH' + if 'ikev2' in record: + # Three protocols share one handler and one record key, because port + # 4500 carries all three; the CBOM wants them apart, since an ESP + # tunnel is `ipsec` and an IKE negotiation is `ike`. + return {'esp': 'ESP', 'ikev1': 'IKEv1'}.get( + (record.get('ikev2') or {}).get('kind'), 'IKEv2') versions = (record.get('tls') or {}).get('tls_versions') if isinstance(versions, list): # A client offering 1.3 and 1.2 that ends up on 1.3 lists both; the diff --git a/cryptomon/parsers/framing.py b/cryptomon/parsers/framing.py index ca22957..3c00643 100644 --- a/cryptomon/parsers/framing.py +++ b/cryptomon/parsers/framing.py @@ -56,6 +56,8 @@ MAX_VLAN_TAGS = 2 # one 802.1Q tag, or a QinQ pair; more is malformed IP_PROTO_TCP = 6 +IP_PROTO_UDP = 17 +UDP_HDR_LEN = 8 # fixed: src(2) dst(2) length(2) checksum(2) IPV6_HDR_LEN = 40 # fixed; options live in extension headers # Extension headers carrying a length in 8-octet units, not counting the @@ -78,6 +80,24 @@ class DecodedFrame(NamedTuple): payload_end: int # one past the last payload byte, padding excluded +class DecodedDatagram(NamedTuple): + """ + The UDP equivalent of DecodedFrame. No sequence number, no flags. + + That absence is the whole difference between the two transports and it + is deliberately visible in the type: UDP has no ordering to recover and + no connection to track, so anything reading this cannot accidentally + write code that assumes a stream. A protocol carried over UDP that + *does* have ordering -- QUIC does -- carries it in its own header, and + recovering it is that protocol's job rather than this module's. + """ + endpoints: dict # the same 'eth' block a TCP document carries + payload_offset: int # first byte after the UDP header + ip_total_len: int # whole IP datagram, header included, both families + version: int # 4 or 6 + payload_end: int # one past the last payload byte, padding excluded + + def _link_layer(raw, linktype): """ Resolve the link layer to (ethertype, offset of the network header). @@ -136,17 +156,20 @@ def _link_layer(raw, linktype): return None -def _walk_ipv6(raw, ip_offset): +def _walk_ipv6(raw, ip_offset, wanted=IP_PROTO_TCP): """ - Walk an IPv6 header and its extension header chain to the TCP header. + Walk an IPv6 header and its extension header chain to `wanted`. IPv6 moved options out of the fixed header into a chain, so unlike IPv4 there is no header-length field to read -- the chain has to be walked. - Returns (src, dst, tcp_offset, total_len), or None when the chain does - not end at TCP. - - A non-initial fragment is refused: it carries no TCP header, so anything - read at that offset would be payload bytes dressed up as one. + Returns (src, dst, transport_offset, total_len), or None when the chain + does not end at the protocol asked for. + + A non-initial fragment is refused: it carries no transport header, so + anything read at that offset would be payload bytes dressed up as one. + `wanted` is a parameter rather than a hard-coded 6 because UDP sits at + the end of exactly the same chain, and a second copy of this walk is a + second place for the fragment check to be forgotten. """ if len(raw) < ip_offset + IPV6_HDR_LEN: return None @@ -157,7 +180,7 @@ def _walk_ipv6(raw, ip_offset): offset = ip_offset + IPV6_HDR_LEN for _ in range(MAX_IPV6_EXT_HEADERS): - if next_header == IP_PROTO_TCP: + if next_header == wanted: return src, dst, offset, IPV6_HDR_LEN + payload_len if next_header == IPV6_NO_NEXT: return None @@ -173,28 +196,22 @@ def _walk_ipv6(raw, ip_offset): elif next_header in IPV6_OPTION_HEADERS: next_header, ext_len = raw[offset], (raw[offset + 1] + 1) * 8 else: - return None # ESP, ICMPv6, UDP, anything else + return None # ESP, ICMPv6, anything else offset += ext_len return None # chain too long to be genuine -def decode_frame(raw, linktype=LINKTYPE_ETHERNET): +def _walk_network(raw, linktype, wanted): """ - Walk link layer -> (VLAN tags) -> IPv4/IPv6 -> TCP. - - Returns a DecodedFrame, or None when the frame is not TCP or is too short - to walk. Returning None rather than guessing is the point: the callers - drop the packet, which is the honest outcome for a frame this parser - cannot read. - - Handles Ethernet, Linux cooked capture v1 and v2, BSD loopback and bare - IP, each with or without VLAN tags, over IPv4 and IPv6. Not handled: - ERSPAN and other tunnels, and IPv6 chains ending anywhere but TCP. - - Note the live path has its own copy of this problem, solved separately in - bpf.py -- the C walks VLAN tags and the IPv6 extension chain for itself, - because the kernel filter has to decide whether to forward a frame before - any of this code runs. + Link layer -> (VLAN tags) -> IPv4/IPv6 -> the transport named by `wanted`. + + Returns (src, dst, transport_offset, ip_total_len, ip_offset, version, + family), or None when the frame is not readable or does not carry + `wanted`. Shared by decode_frame and decode_datagram, because every + defect this walk has ever had -- the fixed 14-byte Ethernet header, the + fixed 20-byte IPv4 header, the unwalked IPv6 chain -- was a defect that + produced plausible wrong offsets rather than an error, and a second copy + for UDP would be a second place to reintroduce them. """ walked = _link_layer(raw, linktype) if walked is None: @@ -213,10 +230,10 @@ def decode_frame(raw, linktype=LINKTYPE_ETHERNET): # --- network layer -------------------------------------------------- if ethertype == ETHERTYPE_IPV6: - walked = _walk_ipv6(raw, ip_offset) + walked = _walk_ipv6(raw, ip_offset, wanted) if walked is None: return None - src, dst, tcp_offset, ip_total_len = walked + src, dst, transport_offset, ip_total_len = walked version, family = 6, 'ipv6' elif ethertype == ETHERTYPE_IPV4: if len(raw) < ip_offset + IP4_HDR_LEN: @@ -224,15 +241,118 @@ def decode_frame(raw, linktype=LINKTYPE_ETHERNET): ip_header_len = (raw[ip_offset] & 0x0F) * 4 if not IP4_HDR_LEN <= ip_header_len <= 60: return None # IHL below 5 or above 15 is malformed - if raw[ip_offset + 9] != IP_PROTO_TCP: + if raw[ip_offset + 9] != wanted: + return None + # A non-initial IPv4 fragment carries no transport header. IPv6 has + # always refused one (see _walk_ipv6); IPv4 did not, because a TCP + # handshake never fragments in practice. A UDP datagram does -- a + # QUIC Initial is close to the MTU by design -- so the same refusal + # belongs on both paths rather than only the one that noticed. + if lst2int(raw[ip_offset + 6:ip_offset + 8]) & 0x1FFF: + PARSE_STATS['ipv4_fragment'] += 1 return None ip_total_len = lst2int(raw[ip_offset + 2:ip_offset + 4]) src = bytes(raw[ip_offset + 12:ip_offset + 16]) dst = bytes(raw[ip_offset + 16:ip_offset + 20]) - tcp_offset = ip_offset + ip_header_len + transport_offset = ip_offset + ip_header_len version, family = 4, 'ipv4' else: return None + return (src, dst, transport_offset, ip_total_len, ip_offset, version, + family) + + +def _endpoints(raw, transport_offset, src, dst, family): + """ + The 'eth' block. Ports are the first four bytes of TCP *and* of UDP. + + Existing documents and queries use eth.src.ipv4, so IPv4 records keep + exactly the shape they had; IPv6 records carry eth.src.ipv6 rather than + overloading a field whose name would then be a lie. + """ + return { + 'src': {family: bytes_to_ip(src), + 'port': lst2int(raw[transport_offset:transport_offset + 2])}, + 'dst': {family: bytes_to_ip(dst), + 'port': lst2int(raw[transport_offset + 2:transport_offset + 4])}, + } + + +def _payload_end(raw, ip_offset, ip_total_len, floor): + """ + Where the payload really ends. + + Ethernet pads a frame out to 60 bytes, so for a short segment `len(raw)` + is several bytes past the last byte the sender transmitted. Reading to + the end of the buffer appends that padding to the stream, which a + single-frame parse never noticed and a reassembled one would carry into + the middle of a TLS record. + """ + end = ip_offset + ip_total_len + if end > len(raw) or end < floor: + return len(raw) # snaplen truncation, or a bad length field + return end + + +def decode_datagram(raw, linktype=LINKTYPE_ETHERNET): + """ + Walk link layer -> (VLAN tags) -> IPv4/IPv6 -> UDP. + + Returns a DecodedDatagram, or None when the frame is not UDP or is too + short to walk. The counterpart of decode_frame, and the seam every + UDP-carried protocol reads through -- QUIC, IKEv2, DTLS, DNS. + + Nineteen per cent of this project's capture corpus is UDP and until this + existed every byte of it was dropped at `decode_frame`'s `return None`, + silently and without a counter. A tool that reports what cryptography is + in use cannot be blind to the transport that carries QUIC. + """ + walked = _walk_network(raw, linktype, IP_PROTO_UDP) + if walked is None: + return None + src, dst, udp_offset, ip_total_len, ip_offset, version, family = walked + + if len(raw) < udp_offset + UDP_HDR_LEN: + PARSE_STATS['truncated_udp_header'] += 1 + return None + payload_offset = udp_offset + UDP_HDR_LEN + + # UDP carries its own length, and it is the tighter of the two bounds: + # the IP total length can be a lie from a bad NIC offload while the UDP + # length came from the sender. Take the smaller, and never past the + # buffer. + udp_len = lst2int(raw[udp_offset + 4:udp_offset + 6]) + end = _payload_end(raw, ip_offset, ip_total_len, payload_offset) + if UDP_HDR_LEN <= udp_len <= end - udp_offset: + end = udp_offset + udp_len + + return DecodedDatagram( + _endpoints(raw, udp_offset, src, dst, family), + payload_offset, ip_total_len, version, end) + + +def decode_frame(raw, linktype=LINKTYPE_ETHERNET): + """ + Walk link layer -> (VLAN tags) -> IPv4/IPv6 -> TCP. + + Returns a DecodedFrame, or None when the frame is not TCP or is too short + to walk. Returning None rather than guessing is the point: the callers + drop the packet, which is the honest outcome for a frame this parser + cannot read. + + Handles Ethernet, Linux cooked capture v1 and v2, BSD loopback and bare + IP, each with or without VLAN tags, over IPv4 and IPv6. For UDP, see + decode_datagram; for tunnels, the caller unwraps first. + + Note the live path has its own copy of this problem, solved separately in + bpf.py -- the C walks VLAN tags and the IPv6 extension chain for itself, + because the kernel filter has to decide whether to forward a frame before + any of this code runs. + """ + walked = _walk_network(raw, linktype, IP_PROTO_TCP) + if walked is None: + return None + src, dst, tcp_offset, ip_total_len, ip_offset, version, family = walked # --- transport layer ------------------------------------------------- if len(raw) < tcp_offset + TCP_HDR_LEN: @@ -277,6 +397,17 @@ def decode_frame(raw, linktype=LINKTYPE_ETHERNET): raw[tcp_offset + 13], payload_end) +# Public aliases. `pcapscan/tunnels.py` walks to IP protocol 47 (GRE) and to +# IP-in-IP through exactly this path: the link layer, stacked VLAN tags, the +# IPv4 header length from IHL, the IPv6 extension chain and the non-initial +# fragment refusal. Every defect that walk has had produced plausible wrong +# offsets rather than an error, so a second copy of it in another module +# would be a second place to reintroduce them. Better one caller reaching +# for it than two implementations of it. +walk_network = _walk_network +payload_end = _payload_end + + def decode_ipv4_tcp(raw): """ Ethernet-framed decode. Kept as the live path's entry point. diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..6cb908c --- /dev/null +++ b/docs/README.md @@ -0,0 +1,54 @@ +# CryptoMon documentation + +Start at the repository [README](../README.md) if you have not already — it +routes you to one of three quite different ways of running this. + +## Getting it running + +* **[install.md](install.md)** — Docker, the offline analyser, or the full + live sensor. What each one genuinely needs, and what `ubuntu-setup.sh` + actually does. +* **[configuration.md](configuration.md)** — every setting, its default, its + effect, and when you would change it. The environment variable names carry + **no prefix**; that is the first thing on the page. +* **[deploy/README.md](../deploy/README.md)** — nginx, systemd, secrets, + subpath mounting, hardening. Read this before anybody but you can reach the + service. +* **[docker/README.md](../docker/README.md)** — the three images, the + compose stack, and the sensor's capability list. + +## Using it + +* **[offline-analysis.md](offline-analysis.md)** — `python -m pcapscan`: all + five output formats, every flag, and the pipelines that make it useful. +* **[service.md](service.md)** — the FastAPI service and its three faces: the + dashboard at `/`, the upload page at `/analyse/`, and JSON at `/data` and + `/stats`. +* **[live-sensor.md](live-sensor.md)** — the eBPF sensor: what it sees, what + it cannot see, and what it costs. + +## Understanding it + +* **[reading-a-report.md](reading-a-report.md)** — what `X25519MLKEM768`, + `hybrid`, `t13d1516h2_8daaf6152771_02713d6af862`, `resumption: resumed` and + `ech: offered` mean, and which of them should worry you. Read the section + on denominators even if you read nothing else. +* **[architecture.md](architecture.md)** — why there are two parsing paths + and why the offline one sees several times more. + +## When it goes wrong + +* **[troubleshooting.md](troubleshooting.md)** — the failures this project + has actually hit, what each one looks like, and what to do. Including + issue #26. + +## A note on the measurements + +Numbers quoted in these pages come from the project's capture corpus: +`CryptomonData/` (eleven captures of common desktop applications on macOS and +Windows 11, December 2024 and November 2024), `sandbox/` (one 13MB capture) +and the trimmed fixtures under `tests/fixtures/`. Where a figure is not +measured, it says so. Where something could not be checked on the machine +these pages were written on — anything needing Linux, root, bcc or a live +interface — it says **not verified on this machine** rather than implying +otherwise. diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..ad10105 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,326 @@ +# Architecture + +There are two ways into this program and they read the same protocol very +differently. Understanding why is the difference between trusting a number +and misreading it. + +``` + the wire a capture file + │ │ + ┌─────▼──────┐ ┌─────▼──────┐ + │ eBPF filter│ in the kernel │ reader │ pcap / pcapng, + │ bpf.py │ one skb at a time │ reader.py │ gzip, stdin + └─────┬──────┘ └─────┬──────┘ + │ frames whose payload │ every frame + │ *starts with* a handshake ┌─────▼──────┐ + │ │ tunnels.py │ GRE / ERSPAN / + │ │ │ VXLAN / GENEVE + │ └─────┬──────┘ + │ ┌─────▼──────┐ + │ │ framing.py │ TCP ──┐ UDP ──┐ + │ └─────┬──────┘ │ │ + │ ┌─────▼──────┐ datagrams.py │ + │ │ reassembly │ │ │ + │ │ .py │ TCP ordering │ + │ └─────┬──────┘ per direction │ + │ ┌─────▼──────┐ │ + │ │ records.py │ TLS records, │ + │ │ │ then messages│ + │ └─────┬──────┘ │ + ┌─────▼───────────────────────────────────────▼──────┐ + │ cryptomon/parsers/ — one TLS parser │ + └─────┬───────────────────────────────────────┬──────┘ + │ one document per frame ┌─────▼──────┐ + │ │ sessions.py│ both directions + │ │ │ paired + │ └─────┬──────┘ + ┌─────▼───────────────────────────────────────▼──────┐ + │ cryptomon/analysis.py — one classification │ + └─────┬───────────────────────────────────────┬──────┘ + │ │ + MongoDB ──────► /stats, the dashboard export.py, cbom.py +``` + +The parsers and the judgement are shared. Only the framing differs, and the +framing is what decides how much there is to parse. + +Two stages on the right have no counterpart on the left, and both exist +because the live sensor's filter reads IP protocol 6 and nothing else. +`tunnels.py` unwraps mirrored traffic — GRE, ERSPAN, VXLAN, GENEVE, +IP-in-IP — *before* framing, because unwrapping a tunnel does not produce a +transport header, it produces another whole frame that has to be walked +again; putting that recursion inside `decode_frame` would hand every caller a +recursion it never asked for and make the depth an attacker-controlled +parameter in the hottest function in the project. `datagrams.py` takes the +UDP side, where there is no ordering to recover and no connection to track — +an absence deliberately visible in the type, so nothing downstream can write +code that assumes a stream. The protocol handlers that sit behind it are +landing as this is written and are documented separately. + +## Why the kernel filter cannot do better + +The eBPF program runs on one `skb` at a time. It has no memory between +packets and no way to acquire one that the verifier would accept, so it can +only forward a frame whose TCP payload *begins* with something recognisable +— a TLS record header (`0x16`, major 3, minor 1–4) or an SSH banner. Anything +that begins mid-message is invisible to it. + +That constraint is not negotiable and it is not a bug. A filter that buffered +would be a TCP stack in the kernel, written by us, on the packet path. The +right answer is to keep the filter cheap and do the reassembly where it is +safe to do it — which is what `pcapscan` is. + +The cost, measured across the eleven corpus captures: + +| | live path | offline path | +|---|---|---| +| ClientHellos reported | 411 | 441 | +| …of which whole | **139** | **441** | +| Certificate messages | **15**, every one a fragment | **93**, whole chains | + +A modern ClientHello with post-quantum key shares is around 1.9KB and arrives +in two segments — a 1238-byte one and a 660-byte one. The single-frame parser +reads the first, finds whatever extensions happened to fit, and records them +**as though that were the whole message**. Not missed: reported, incomplete, +and without saying so. Two thirds of the live tool's hello output has been +partial for as long as it has existed. A certificate chain is several +kilobytes and has never once fitted in a single packet. + +So "the offline path sees roughly five times as much" is a fair summary of +two numbers: 3.2× the whole ClientHellos and 6.2× the certificate messages. + +Three more things only reassembly can see: + +* **Both halves of a HelloRetryRequest.** A post-quantum key exchange the + server *refused* is reported as refused rather than as offered. Getting + this wrong reverses the finding. +* **Handshakes behind a ChangeCipherSpec.** TLS 1.3 sends a dummy CCS + mid-handshake so middleboxes see a familiar sequence (RFC 8446 D.4) and + then carries on in the clear. Reading that CCS the TLS 1.2 way — as "the + rest is encrypted" — loses every message after it, which is 108 + ClientHellos and 108 ServerHellos across the corpus, and they are the ones + that matter most. A measured 7% of corpus segments carry a CCS ahead of the + handshake in the same packet, which the kernel filter can never forward. +* **Traffic on unwatched ports.** The filter has to have a port list, because + the list is what stops it copying every packet on the interface to + userspace. The offline analyser has no port filter and identifies protocols + from the bytes: 13 of 1369 TLS flows in the corpus (0.95%) are on ports the + filter does not watch. + +## Checking the claim against somebody else's implementation + +A recall claim is worth exactly as much as its independent check. The offline +path is diffed against **tshark with desegmentation on** over eleven full +captures, in CI, and agrees on 441 ClientHellos and 93 certificate messages. + +There is one disagreement in the certificate column and it is ours: on one +flow tshark's desegmenter gives up, because the flow has TSO-offloaded +segments with zero checksums that trip its TCP analysis, and reports no +certificate. `pcapscan` recovers a well-formed 6721-byte chain which +`cryptography` parses without complaint. So the honest form of the claim is +"agrees with tshark on every handshake tshark found, and finds one more". + +The oracle has a second CI job that regenerates it with a current tshark and +fails on any difference, because a committed oracle is only an oracle while +it still matches what the reference says. + +## What is where + +### Shared + +| | | +|---|---| +| `cryptomon/parsers/framing.py` | Ethernet, VLAN, IPv4/IPv6, TCP. One decoder for both paths. | +| `cryptomon/parsers/tls.py` | Hello parsing, extensions, ECH. Takes the offset of a handshake message rather than assuming a record at the start of a frame, which is what let the offline walker reuse it unchanged. | +| `cryptomon/parsers/ssh.py` | KEXINIT algorithm lists. | +| `cryptomon/fingerprints.py` | JA4, JA4S, JA3. Pure functions, no I/O, so both paths compute the same string from the same code. | +| `cryptomon/analysis.py` | **The judgement.** Which algorithms survive a quantum computer, and the `Summary` both paths roll up into. | +| `cryptomon/alerts.py` | The IANA alert registry and which combinations are a finding. | +| `cryptomon/ports.py` | The watched port lists, and the generated C for them. | +| `cryptomon/data.py`, `utils.py` | Code-point tables and lookups. | + +`cryptomon/analysis.py` is the reason the two halves are one product rather +than two. Counting ciphersuites is arithmetic; saying which of them survive a +cryptographically relevant quantum computer is the question the tool was +built to answer, and it belongs in exactly one place. The CBOM exporter, the +report page, `/stats` and the dashboard all ask it rather than re-implementing +it. `/stats` in particular could have expressed the verdicts as a MongoDB +`$switch` and did not, for the same reason: a second copy in a second +language drifts, and this project has the worked example — the post-quantum +markers have to be matched *and removed* before the classical ones, or every +NIST signature selection reports as a hybrid of itself and DSA. That fix +lives in one function. A `$switch` would not have it. + +### The live path + +| | | +|---|---| +| `cryptomon/bpf.py` | The kernel C, generated from `ports.py` at import. | +| `cryptomon/__init__.py` | `CryptoMon`: attach, poll the perf buffer, parse, write. | +| `cryptomon.py` | The command line. | + +### The offline path + +| | | +|---|---| +| `pcapscan/reader.py` | pcap and pcapng, both byte orders, both timestamp resolutions, multiple interfaces and sections, gzip, truncated tails. Streams one frame at a time rather than `rdpcap`-ing a 700MB file into memory. | +| `pcapscan/tunnels.py` | Unwraps GRE, ERSPAN I/II/III, VXLAN, GENEVE and IP-in-IP before framing, so a capture from a switch mirror port reports what it carries. Borrows `framing.walk_network` rather than copying it: every defect that walk has had produced plausible wrong offsets rather than an error, and a second copy would be a second place to reintroduce them. | +| `pcapscan/datagrams.py` | The UDP side: no ordering to recover, no connection to track. | +| `pcapscan/reassembly.py` | Orders the bytes. Not a TCP stack: no windows, no ACK tracking, nothing emitted. Holes held until filled, retransmissions dropped, overlaps trimmed. | +| `pcapscan/records.py` | Walks TLS records, then the handshake messages inside them — two framings that do not line up. One record commonly holds ServerHello, Certificate, ServerKeyExchange and ServerHelloDone together, while one certificate chain routinely spans five records. | +| `pcapscan/protocols.py` | Identifies TLS, SSH, HTTP/1.x, HTTP/2, SMTP, IMAP, POP3, FTP and STUN from bytes, so a flow that carries none of them is abandoned early. | +| `pcapscan/sessions.py` | Pairs the two directions into one record per handshake. | +| `pcapscan/certificates.py` | X.509, via `cryptography`. Optional. | +| `pcapscan/export.py`, `cbom.py` | NDJSON, JSON, CSV; CycloneDX 1.6. | +| `pcapscan/sandbox.py` | Runs the whole pipeline in a subprocess with its own rlimits, for the upload path. | +| `pcapscan/cli.py` | The command line. | + +### The service + +| | | +|---|---| +| `api.py` | The application, and the router order — `/stats` is declared before the dashboard, because FastAPI matches in declaration order and a root mount declared first would swallow it. | +| `fapi/config/` | Settings. | +| `fapi/app/routers.py` | `/data`. | +| `fapi/app/query.py` | The filter allow-list for the count endpoints. | +| `fapi/app/security.py` | The write guard. | +| `fapi/app/uploads.py` | `/analyse`. | +| `fapi/app/stats.py` | `/stats`, ten aggregation pipelines. | +| `fapi/app/dashboard.py`, `charts.py` | `/`, and the inline SVG. | +| `fapi/app/indexes.py`, `retention.py` | Startup indexes and the expiry sweep. | + +## Bounds, everywhere + +An offline tool is handed files chosen by somebody else, and a service is +handed uploads chosen by somebody else, so almost every loop here has a +ceiling and almost every ceiling has a comment saying what it costs. The main +ones: + +| | | +|---|---| +| 16KB per direction | Reassembly buffer. The whole plaintext handshake fits with room to spare; past it a stream is bulk transfer. `--max-stream-bytes`. | +| 2048 flows | LRU flow table. `--max-flows`. | +| 1MB | Largest frame the reader will accept before it stops trusting the file. | +| 512 certificates, 50 hosts, 32 occurrences | What one `Summary` holds, so analysing a large capture costs a report rather than a second copy of the capture. | +| 2 GiB / 120 CPU-s / 180 wall-s | The upload sandbox, set against the measurement that the whole 265MB corpus analyses in 1.35s. | +| 2000 points | The timeline series, so `?bucket=minute&hours=8760` cannot ask for 525,600. | +| 64 ports | `TLS_PORTS`, so the generated C stays well inside the verifier's instruction limit. | +| 1000 in-flight inserts | The live sensor's write backpressure. | + +A flow the reassembler can tell carries nothing wanted is `abandon()`ed on +its first segment, which is what makes the memory ceiling real rather than +nominal: an 80MB capture runs in 0.1s holding 162 bytes at the end, because +62 of its 73 flows were dropped immediately. + +## Two document shapes in one collection + +This surprises people, so it is worth stating plainly. + +| | live sensor | `pcapscan` + `mongoimport` | +|---|---|---| +| `ptype` | `client` or `server` | `session` | +| granularity | one document per **frame** | one document per **handshake** | +| `ts` | set at insertion | from the capture | +| has | the single-frame view of `tls` | `certificates`, `alerts`, `resumption`, `proposed`, `selected`, `ja4s` | + +Both are valid and both can live in the same collection, which is +deliberate — the NDJSON export keeps the live path's field paths so that the +two feed one collection without translation. But a panel that assumes one +shape is wrong on somebody's deployment, so `/stats/overview` reports +`by_ptype` and every panel whose meaning changes between the two says so in +its `notes`. + +A live document, `ptype: client` — one ClientHello, as the sensor saw it: + +```json +{ + "_id": "6682cd75393bb4e863fc0c65", + "eth": {"src": {"ipv4": "192.168.64.5"}, + "dst": {"ipv4": "3.210.189.242"}}, + "tls": { + "tls_versions": ["TLSv1.3", "TLSv1.2"], + "ciphersuites": ["TLS_AES_128_GCM_SHA256", "…"], + "EtM": false, + "hostname": "ping.chartbeat.net", + "groups": ["x25519", "secp256r1", "secp384r1", "secp521r1", + "ffdhe2048", "ffdhe3072"], + "kex_group": "x25519", + "sigalgs": ["ecdsa_secp256r1_sha256", "…"] + }, + "ptype": "client", + "ts": 1719848309.166212 +} +``` + +The matching `ptype: server` document is a separate row, joined to it by +nothing: + +```json +{ + "_id": "6682cd75393bb4e863fc0c66", + "eth": {"src": {"ipv4": "3.210.189.242"}, + "dst": {"ipv4": "192.168.64.5"}}, + "tls": {"tls_versions": "TLSv1.2", + "ciphersuite": "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"}, + "ptype": "server", + "ts": 1719848309.26233 +} +``` + +That is enough to count what clients *offer* and not enough to say what was +*used*, which are different questions. An offline document, `ptype: session`, +carries both halves and what they imply: + +```json +{ + "ptype": "session", + "ts": 1733954736.226069, + "duration": 0.025676, + "eth": {"src": {"ipv4": "10.176.24.102", "port": 49492}, + "dst": {"ipv4": "152.199.2.76", "port": 443}}, + "tls": { + "hostname": "cdn.bizible.com", + "ech": "offered", + "ja4": "t13d1516h2_8daaf6152771_02713d6af862", + "ja4s": "t130200_1302_a56c5b993250", + "ciphersuite": "TLS_AES_256_GCM_SHA384", + "kex_group": "secp256r1", + "tls_versions": ["TLSv1.3"], + "resumption": "fresh", + "resumption_evidence": "TLS 1.3 without pre_shared_key", + "hello_retry_request": true, + "offered_kex_group": "X25519Kyber768Draft00", + "retry_kex_group": "secp256r1", + "certificates_unreadable": true, + "proposed": { … }, "selected": { … }, "messages": { … } + } +} +``` + +`tls.ciphersuite`, `tls.kex_group` and `tls.hostname` stay exactly where the +live tool has always put them, so existing queries and stored data keep +working; everything new hangs off `proposed`, `selected` and `certificates`. + +There are also two certificate parsers, for the same historical reason: the +live path uses `cryptomon.utils.cert_guess`, which needs `jc` and guesses at +a chain it can only see a fragment of; the offline path uses +`pcapscan.certificates`, which needs `cryptography` and parses a whole chain. + +## What is not here + +* **No DTLS.** +* **No decryption**, of anything, anywhere. +* **Nothing over UDP in the live path.** The kernel filter reads IP protocol + 6 and nothing else, so UDP never reaches userspace there. The offline + analyser does read UDP — see below. +* **No tunnels in the live path either.** Encapsulated traffic on a monitored + interface produces no events at all. The offline analyser unwraps GRE, + ERSPAN, VXLAN, GENEVE and IP-in-IP before framing. + +IPv6 *is* supported, in both paths, and any older note saying it is a planned +feature is stale. The kernel filter reads every offset from the packet, steps +over stacked VLAN tags, walks the IPv6 extension header chain and refuses +non-initial fragments; the offline framing decoder does the same in Python. +Single-tag VLAN happened to work before that change, because the kernel +strips one 802.1Q tag into `skb` metadata ahead of the filter — which is why +QinQ failed and a single tag did not. diff --git a/docs/configuration.md b/docs/configuration.md new file mode 100644 index 0000000..1c7d66e --- /dev/null +++ b/docs/configuration.md @@ -0,0 +1,184 @@ +# Configuration + +Everything is configured through environment variables. There is no +configuration file, no `--config` flag and no settings UI. + +## The variable names carry no prefix + +`READ_ONLY`, not `CRYPTOMON_READ_ONLY`. `API_KEY`, not `CRYPTOMON_API_KEY`. +The field names in `fapi/config/__init__.py` *are* the environment variable +names, and pydantic-settings is configured with no `env_prefix`. + +This is worth stating loudly because the project has already made the mistake +once: the error message the API returns for a refused write used to name +`CRYPTOMON_READ_ONLY` and `CRYPTOMON_API_KEY`, neither of which exists. An +operator following it exactly would have set two variables that do nothing, +seen no change, and concluded the guard could not be turned off. The message +was corrected; the lesson is in a comment at the top of `fapi/app/security.py`. + +```console +$ export DB_URL=mongodb://127.0.0.1:27017 DB_NAME=cryptomon # both required + +$ CRYPTOMON_READ_ONLY=false python -c 'from fapi.config import settings; print(settings.READ_ONLY)' +True +$ READ_ONLY=false python -c 'from fapi.config import settings; print(settings.READ_ONLY)' +False +``` + +Two consequences of how pydantic-settings reads them: + +* **Names are matched case-insensitively.** `read_only=false` works as well + as `READ_ONLY=false`. Use the upper-case spelling; it is what every unit + file and every example in this repository uses. +* **No `.env` file is read.** `Settings` declares no `env_file`, so a `.env` + sitting next to `api.py` is ignored entirely — verified. The `.env` that + `docker/env.example` tells you to create is read by *docker compose*, which + then passes the values into the container's environment, which is a + different mechanism arriving at the same place. Outside compose, export the + variables, or put them in a systemd `EnvironmentFile` as + [`deploy/systemd/api.env.example`](../deploy/systemd/api.env.example) does. + +## The API and the dashboard + +`DB_URL` and `DB_NAME` are **required and have no default**. Without them +nothing that imports `fapi.config` will start, including `api.py` and +`cryptomon.py`: + +``` +pydantic_core._pydantic_core.ValidationError: 2 validation errors for Settings +DB_URL + Field required [type=missing, ...] +DB_NAME + Field required [type=missing, ...] +``` + +| Variable | Default | What it does | When to change it | +|---|---|---|---| +| `DB_URL` | *required* | The MongoDB connection string, credentials included. | Always. | +| `DB_NAME` | *required* | The database inside it. The collection is always `cryptomon`. | Always. | +| `HOST` | `127.0.0.1` | The address uvicorn binds. | Only with something in front of it. Setting `0.0.0.0` publishes the API — and the dashboard, whose first panel is a list of server names taken from SNI — on every interface. An earlier release defaulted to `0.0.0.0` with unauthenticated `POST`, `PUT` and `DELETE` on `/data`; that is why the default is what it is. | +| `PORT` | `8000` | The port uvicorn binds. | When 8000 is taken. | +| `ROOT_PATH` | `""` (domain root) | The path prefix this service is mounted under behind a reverse proxy: `/cryptomon` for `https://host/cryptomon/`. | When nginx serves it on a subpath. It **must** equal the `location` prefix in the nginx config; [deploy/README.md](../deploy/README.md#the-subpath) explains why one of the pair alone is not enough. | +| `READ_ONLY` | `true` | Refuses `POST /data/`, `PUT /data/{id}` and `DELETE /data/{id}` with 403. Reads are always open. | Only if something genuinely needs to write records over HTTP. The sensor writes to MongoDB directly and does not use this API, so most deployments never need to change it. | +| `API_KEY` | `""` (off) | When set, mutating routes *and* the upload route require a matching `X-API-Key` header, compared with `secrets.compare_digest`. | Set it before the service is reachable from anywhere but localhost. Note the consequence: the browser upload form cannot send a header, so with a key set uploads have to come from `curl`. The form says so. | +| `UPLOADS_ENABLED` | `true` | The `/analyse/` capture-upload UI. On by default because the service binds loopback, so the thing to opt into is exposure rather than the feature. | Turn it off for a deployment that only ever serves the API and the dashboard. | +| `UPLOAD_DIR` | `/tmp/cryptomon-uploads` | Where captures are spooled and reports kept. Captures are deleted the moment they are analysed; only the report remains. | Under systemd, always. `cryptomon-api.service` sets `PrivateTmp=yes`, so `/tmp` is a namespace destroyed on every restart and reports would vanish while the form went on promising they were kept. The unit's `StateDirectory=` provides `/var/lib/cryptomon/uploads`. | +| `MAX_UPLOAD_BYTES` | `268435456` (256 MiB) | The upload cap, enforced *while the stream is read* — bytes are counted as they arrive and the partial file is removed the moment the cap is passed. | When your captures are bigger. Keep `client_max_body_size` in nginx strictly larger, or nginx refuses the upload with its own unstyled 413 before the application can answer. | +| `ANALYSIS_TIMEOUT_SECONDS` | `120` | Wall-clock ceiling for analysing one capture, passed to `pcapscan.sandbox`. | For very large captures. Keep nginx's `proxy_read_timeout` above it, or nginx answers 504 for work that finished. | +| `REPORT_RETENTION_HOURS` | `24` | How long an uploaded capture's report is kept before the sweep deletes it. `0` keeps them forever. | Shorten it if reports are sensitive; a report holds the SNI of every connection in the capture. The upload form states the value *before* the file is chosen. | +| `RETENTION_SWEEP_MINUTES` | `15` | How often that sweep runs. It runs inside the API process, so a deployment cannot end up serving a form that promises expiry while nothing expires anything. | Rarely. | +| `DATA_RETENTION_HOURS` | `0` (keep everything) | Expiry for the **live MongoDB collection**. When set, it becomes a MongoDB TTL index on `expires_at`, so the server does the deleting whether or not the API is running. | Only deliberately. The default is not an oversight: silently discarding a monitoring database would destroy the historical series this project exists to build. | +| `APP_NAME` | `Cryptomon API` | A label. Nothing depends on it. | Never, in practice. | +| `DEBUG_MODE` | `false` | Passed to uvicorn as `reload=`, which starts a file watcher and restarts the process on change. | Locally, while editing. Never on a host that anybody else can reach. Note that `config-secrets.sh` in the repository root sets `DEBUG_MODE=True`. | + +`ROOT_PATH` is validated rather than trusted. starlette strips the prefix +with a regular expression it builds by interpolation and without +`re.escape`, so a value containing `.`, `+` or `(` would silently match paths +it was never meant to. A bad value is a startup failure with a message +instead of a routing bug at run time: + +```console +$ ROOT_PATH=/crypto.mon python -c 'import fapi.config' +... +pydantic_core._pydantic_core.ValidationError: 1 validation error for Settings +ROOT_PATH + Value error, ROOT_PATH must be a plain absolute path such as '/cryptomon': + a leading slash, no trailing slash, and only letters, digits, underscore + and hyphen in each segment. Got '/crypto.mon'. [type=value_error, ...] +``` + +## Which ports the sensor watches + +Two more variables, read by `cryptomon/ports.py` and compiled into the eBPF +program. They also carry no prefix, for the same reason. + +| Variable | Default | What it does | +|---|---|---| +| `TLS_PORTS` | `443,990,3389,8080,8443` | TCP ports the kernel filter forwards TLS handshakes from. | +| `SSH_PORTS` | `22` | The same, for SSH. | + +Comma-separated, 1–65535, at most 64 entries. Unset means the defaults; +setting one to an empty string is an error rather than "watch nothing", +because the generated C needs at least one comparison to be C at all. + +```console +$ TLS_PORTS=443,8443,9443 python -c 'from cryptomon.ports import tls_ports; print(tls_ports())' +(443, 8443, 9443) +``` + +Parsing is deliberately strict, because the failure it prevents is silent: a +sensor watching the wrong ports reports no handshakes, and no handshakes +looks exactly like a quiet network. Every rejection names the variable and +the offending text. + +Each of these raises, with the traceback ending in the line shown: + +```console +$ TLS_PORTS=443,https python -c 'from cryptomon.ports import tls_ports; tls_ports()' +ValueError: TLS_PORTS: 'https' is not a port number +$ TLS_PORTS=4_43 python -c 'from cryptomon.ports import tls_ports; tls_ports()' +ValueError: TLS_PORTS: '4_43' is not a port number +$ TLS_PORTS=70000 python -c 'from cryptomon.ports import tls_ports; tls_ports()' +ValueError: TLS_PORTS: port 70000 is outside 1-65535 +$ TLS_PORTS= python -c 'from cryptomon.ports import tls_ports; tls_ports()' +ValueError: TLS_PORTS: empty. Unset the variable to use the defaults. +``` + +`4_43` is rejected on purpose: `int()` accepts underscores as digit +separators and non-ASCII digits, so `int('4_43')` is 443 and so is +`int('４４３')`. A port list is operator input that ends up compiled into a +kernel program. It should mean what it looks like or be refused. + +**Only 443 is justified by measurement.** Of 1260 sessions across the twelve +corpus captures, 1248 are to port 443 and the other 12 are to ephemeral +ports. Nothing in the corpus touches 990, 3389, 8080, 8443 or 22. The other +defaults are conventions, and 8080 is the weak one — it is conventionally +*plaintext* HTTP, and TLS there is a local habit. It stays because removing a +default changes what existing deployments see, and because the cost is +bounded: the program still requires a record beginning `0x16 0x03 0x0[1-4]` +before it submits anything, so plaintext HTTP on 8080 produces no events. It +is not free, though. On a host with busy 8080 traffic, `TLS_PORTS=443` is now +one variable away. + +The offline analyser has **no port filter at all**. It looks at every TCP +stream and decides from the bytes, which is why those 12 ephemeral-port +sessions appear in offline numbers and could never appear in live ones. + +## The sensor's interface + +`CRYPTOMON_IFACE` is read by systemd, not by Python. The unit interpolates it +into the command line: + +``` +ExecStart=/opt/cryptomon/.venv/bin/python /opt/cryptomon/cryptomon.py -i ${CRYPTOMON_IFACE} +``` + +There is no default and no sensible guess. If it is unset or empty, +`cryptomon.py` falls back to prompting with `input()` — and under systemd +there is no terminal, so the unit dies on every start with an `EOFError` +traceback that says nothing about a missing interface. `ip -br link` will +tell you the name. + +## uvicorn's own variables + +`FORWARDED_ALLOW_IPS` is read by uvicorn, not by `fapi.config`. It decides +which addresses uvicorn will believe `X-Forwarded-For` and `X-Forwarded-Proto` +from; the default, `127.0.0.1`, is right when nginx is on the same host. If +nginx is elsewhere and this is not set, every forwarded header is discarded +without a warning, every client appears to be the proxy, and the redirect +after an upload comes back as `http://` on an `https` site. Do not set it to +`*` on a host where anything else can reach port 8000. + +## Where to put these + +| | | +|---|---| +| A shell, for a moment | `export DB_URL=…` before `python api.py` | +| systemd | A mode-0600 `EnvironmentFile`. [`deploy/systemd/api.env.example`](../deploy/systemd/api.env.example) and [`sensor.env.example`](../deploy/systemd/sensor.env.example) are annotated line by line, and are the most complete worked configuration in the repository. | +| Docker | `docker/.env`, from `docker/env.example`, read by compose. | + +Secrets belong in a file the service account itself cannot open, not in +`Environment=` lines in a unit — those are world-readable through +`systemctl show`. `create-service.sh` and [deploy/README.md](../deploy/README.md) +cover that in detail; the previous version of that script wrote a literal +`` into a unit and started it. diff --git a/docs/install.md b/docs/install.md new file mode 100644 index 0000000..d204c53 --- /dev/null +++ b/docs/install.md @@ -0,0 +1,255 @@ +# Installing CryptoMon + +Three paths, and they are not variations on one install. Each needs a +different set of things from the machine, and the differences are large +enough that picking the wrong one costs an afternoon. + +| | Needs | Does not need | Runs on | +|---|---|---|---| +| **Docker** | Docker | anything else | anywhere Docker runs | +| **The offline analyser** | Python 3.10+ | root, eBPF, a kernel, a database, an interface | macOS, Linux, Windows, BSD | +| **The live sensor** | Linux 5.8+, bcc, kernel headers, root, MongoDB | — | Linux only | + +The question that decides it is whether you have **a capture file** or **a +network**. A capture file is answered by the first two, and the offline +analyser reads more out of a capture than the live sensor could ever have +seen (see [architecture.md](architecture.md)). A live network — a continuous +series rather than a snapshot — is the only reason to take the third path. + +--- + +## Docker + +Nothing to install but Docker itself, and three images to choose from. +[`docker/README.md`](../docker/README.md) is the reference; this is the +thirty-second version. + +```bash +# analyse a capture: 164MB, non-root, read-only filesystem, no network +docker build -f docker/Dockerfile.offline -t cryptomon-offline . +docker run --rm --network none --read-only \ + -v "$PWD:/captures:ro" cryptomon-offline /captures/your-capture.pcap + +# the API, the upload UI, the dashboard, and a MongoDB beside them +cd docker +cp env.example .env # then put a real password in MONGO_PASSWORD +docker compose up --build # http://127.0.0.1:8000/ + +# does eBPF work on this host at all? +docker build -f docker/Dockerfile.sensor -t cryptomon-sensor . +docker run --rm --cap-add BPF cryptomon-sensor +``` + +That last command is [issue #26](troubleshooting.md#bcc-will-not-install-or-will-not-compile-issue-26) +made answerable: it compiles the kernel filter and puts it past the verifier +in both attach modes, touching no interface and no database. If it prints +`OK compiles` and two `OK verifies`, the kernel half of CryptoMon works on +this machine and anything still wrong is configuration. + +> **Not verified on this machine.** The Docker commands above are quoted from +> `docker/README.md`, which records where and when they were checked. Building +> an image needs the network, and these pages were written with none. + +The password that ships in `docker/compose.yaml` is +`change-me-this-is-not-a-password` — a placeholder written out in full so it +cannot be mistaken for a generated secret. Both published ports bind +`127.0.0.1`. + +--- + +## The offline analyser + +The path most people want. It reads a capture file, needs no privileges and +touches no database. + +```bash +git clone https://github.com/Santandersecurityresearch/CryptoMon.git +cd CryptoMon +pip install cryptography +python -m pcapscan your-capture.pcap +``` + +Python 3.10 is the floor. `cryptography` is the only third-party package +involved, and it does exactly one job: parsing the X.509 chain out of the +handshake. Everything else — reading pcap and pcapng, reassembling TCP, +walking TLS records, classifying algorithms, writing NDJSON, CSV, JSON and a +CycloneDX CBOM — is the standard library. This was checked by blocking the +import of every other package in `requirements.txt` and running all five +output formats; they all work. + +Without `cryptography` the analyser still runs, and the certificate section +of the report is **silently empty** rather than absent. That failure has its +own entry in [troubleshooting.md](troubleshooting.md#the-report-has-no-certificates-in-it) +because an estate with no certificate keys and an estate whose certificates +could not be parsed look identical in the output. + +`pip install -r requirements.txt` installs the rest of the project too — +FastAPI, motor, uvicorn, scapy, TinyDB — and is what you want if you also +intend to run the service. It is not needed to analyse a capture. + +### A virtual environment, if your Python objects + +Recent distributions mark the system Python as externally managed (PEP 668) +and refuse `pip install` into it: + +```bash +python3 -m venv .venv +. .venv/bin/activate +pip install -r requirements.txt +``` + +--- + +## The live sensor + +Linux only, root, and a kernel that will load a BPF program. This is the half +of CryptoMon that watches an interface instead of reading a file. + +### What it needs + +| | | +|---|---| +| Linux kernel | 5.8 or newer, so `CAP_BPF` and `CAP_PERFMON` exist as their own capabilities. Older kernels can only be given `CAP_SYS_ADMIN`. | +| `bcc` | From the distribution — `bpfcc-tools`, which pulls in `python3-bpfcc`. **Not** the PyPI package called `bcc`, which is something else entirely. | +| Kernel headers | Matching the kernel that is actually running, because bcc compiles the program against them at load time. | +| `pyroute2` | `python3-pyroute2`, and only for the `-tc` traffic-control attach mode. | +| MongoDB | The sensor writes one document per parsed frame and has nowhere else to put them. | +| root | Or the capability set in [`docker/README.md`](../docker/README.md#capabilities-and-why-not---privileged). | + +`bcc` and `pyroute2` are deliberately absent from `requirements.txt`. They +are Linux-only and come from the distribution, and keeping them out is what +lets the test suite and the offline tools import on a machine that has +neither. + +### `ubuntu-setup.sh`, honestly + +The repository ships a 40-line `apt-get` script, and the README has always +said it "will install all the necessary files". Read it before you run it. +It has not aged well, and on Ubuntu 24.04 "Noble Numbat" — the release the +README names — several of its lines fail. + +```bash +sudo ./ubuntu-setup.sh +``` + +What is in it, and what to expect: + +* **It has no `set -e` and no root check.** Every line that fails is passed + over in silence and the script exits 0 regardless. That matters because + several lines do fail. +* **`linux-tools-5.15.0-41-generic` is pinned to one specific kernel.** That + is a 5.15 kernel — Ubuntu 22.04's. On 24.04, whose kernel is 6.8, the + package does not exist and the line fails. Nothing downstream needs it; + the later `linux-tools-$(uname -r)` line is the one that matters. +* **`llvm-12`, `clang-12` and `clang-format-12` are pinned to an LLVM + release that 24.04 does not carry.** Where those packages are absent the + three lines fail, and then the loop below them that strips the `-12` suffix + runs `which clang-12` on nothing and hands `ln -s` an empty path. Harmless, + noisy, and confusing to read in the output. Unversioned `clang` and `llvm` + are installed by the second line of the script regardless. +* **`sudo snap install --devmode bpftrace` carries a `# TODO - find out why + this doesn't work` beside it.** It has not worked and it does not need to: + nothing in this repository uses bpftrace. It can be ignored. +* **`apt-get install bpfcc-tools linux-headers-$(uname -r)` has no `-y`**, + unlike every other line, so it prompts. It is also the line that fails + inside a container or on a host whose running kernel has no matching + headers package — the single most common way the sensor fails to start. + `linux-headers-generic` is what the project's own container images install, + for exactly that reason. +* **It installs Python packages from `apt`**, not from `requirements.txt`: + `python3-scapy`, `python3-motor`, `python3-psutil`, `python3-pyroute2`, + `python3-pymongo`, `python3-fastapi`, `python3-tinydb`. Those are the + distribution's versions, which are not the versions `requirements.txt` + pins. +* **It does not install everything the API needs.** `uvicorn`, `jinja2`, + `python-multipart` and `cryptography` are in `requirements.txt` and are not + in this script, so `python api.py` will not start after running it alone. +* **It adds MongoDB 7.0 from the `jammy` (22.04) repository** regardless of + what you are running. +* **The final `python3 -m pip install jc` will be refused** on a PEP 668 + distribution. `jc` is used by one function on the live path + (`cryptomon.utils.cert_guess`), which prints a message and returns nothing + when it is missing. The apt line above it usually covers it. + +A reasonable reading of all that: run it for the eBPF toolchain, then install +the Python side properly. + +```bash +sudo ./ubuntu-setup.sh # the apt/eBPF half +python3 -m venv .venv && . .venv/bin/activate +pip install -r requirements.txt # the pinned Python half +``` + +> **Not verified on this machine.** Everything in this section is read from +> `ubuntu-setup.sh`, the package lists of the releases it names, and the +> project's own commit history. It was not executed: these pages were written +> on macOS, with no network. + +### MongoDB + +The sensor needs somewhere to write. Create the database and a user for it: + +```bash +mongosh +``` + +```javascript +use cryptomon +db.createCollection('cryptomon') +db.createUser({user: "cryptomonUser", pwd: passwordPrompt(), + roles: [{ role: "readWrite", db: "cryptomon" }]}) +``` + +Then point CryptoMon at it. The variable names carry no prefix — see +[configuration.md](configuration.md): + +```bash +export DB_URL="mongodb://cryptomonUser:@127.0.0.1:27017/cryptomon?retryWrites=true&w=majority" +export DB_NAME="cryptomon" +``` + +Or, for Atlas or another hosted service: + +```bash +export DB_URL="mongodb+srv:///cryptomon?retryWrites=true&w=majority" +export DB_NAME="cryptomon" +``` + +Worth giving the sensor its own user with write access only. The sensor and +the API have very different exposure, and there is no reason a compromise of +one should hand over the other's credential. + +### Running it + +```bash +sudo python3 ./cryptomon.py -i eth0 # the sensor +python3 ./api.py # the API, in another shell +``` + +See [live-sensor.md](live-sensor.md) for what it sees, and +[deploy/README.md](../deploy/README.md) for running both as systemd services +behind nginx. `create-service.sh` installs the units and deliberately starts +neither, so that you can read them and put the database password in place +first. + +--- + +## Checking the install + +```bash +python -m pcapscan tests/fixtures/streams/tls13_hello_retry.pcap +``` + +This is a committed 24-packet capture of one real handshake, and it exercises +the whole offline path — reader, reassembler, record walker, TLS parser, +certificate handling and the classification table. It should print one +session, one classical key exchange, and one post-quantum offer that the +server refused. + +```bash +python -m pytest -m smoke # ~5s, no captures, no database, no network +python -m pytest # everything, ~12s +``` + +See [tests/README.md](../tests/README.md) for what the two sets cover and why +they are split. diff --git a/docs/live-sensor.md b/docs/live-sensor.md new file mode 100644 index 0000000..7950c8d --- /dev/null +++ b/docs/live-sensor.md @@ -0,0 +1,217 @@ +# The live sensor + +```bash +sudo python3 ./cryptomon.py -i eth0 +``` + +The half of CryptoMon that watches an interface. It loads an eBPF program +into the kernel, has the kernel hand it the packets that look like the start +of a TLS or SSH handshake, parses those in Python, and writes one document +per parsed frame into MongoDB. + +Linux only, and it needs root (or the capability set below), bcc, kernel +headers matching the running kernel, and a database. [install.md](install.md) +covers getting those. If you want an answer about a *capture file*, use +[the offline analyser](offline-analysis.md) — it needs none of this and sees +more. + +> **Not verified on this machine.** Everything on this page that needs a +> kernel is read from the code, its comments, the project's commit history +> and [`docker/README.md`](../docker/README.md), which records where its own +> measurements were taken. These pages were written on macOS. The two +> failures quoted below with `$` prompts were reproduced here, because both +> happen before any kernel call. + +## Flags + +``` +python3 ./cryptomon.py [-i IFACE] [--pcap FILE] [-tc] +``` + +| | | +|---|---| +| `-i`, `--interface` | The interface to attach to. | +| `--pcap FILE` | Replay a capture over loopback instead of monitoring. Forces `-i lo`. | +| `-tc`, `--traffic-control` | Attach through traffic control instead of a raw socket. | + +**`-i` is effectively required.** Without it the script lists the interfaces +it found and prompts with `input()` for a number. That is fine at a terminal +and fatal under systemd, where there is no terminal and the prompt raises +`EOFError` — the unit then dies on every start with a traceback that says +nothing about a missing interface. `deploy/systemd/sensor.env.example` makes +`CRYPTOMON_IFACE` required for that reason. Note also that the README used to +say the interface defaults to `enp0s1`; it does not. The default lives on the +`CryptoMon` class, and `cryptomon.py` always passes whatever `-i` or the +prompt produced. + +### The two attach modes + +| | How | Needs | +|---|---|---| +| default (`library`) | `BPF.SOCKET_FILTER` on a raw `AF_PACKET` socket | `CAP_NET_RAW` | +| `-tc` | `BPF.SCHED_CLS` on a `clsact` qdisc, interface set `IFF_PROMISC` | `CAP_NET_ADMIN`, and `python3-pyroute2` | + +The comment in the code says most physical Ethernet devices need TC to sniff +properly; the raw-socket mode is the older path and the default. `-tc` also +puts the interface into promiscuous mode and adds a qdisc, and removes its +filter again on a clean shutdown. + +Without `pyroute2` the TC mode fails with a message that names the package: + +``` +Exception: pyroute2 is not available, so load_method='tc' cannot attach to +the interface. Install it from your distribution (`apt-get install +python3-pyroute2`), or use load_method='library' to attach a raw socket +instead. +``` + +### Capabilities, if you would rather not run it as root + +Measured — not read off `capability(7)` — in the project's sensor container: + +| | | +|---|---| +| `CAP_BPF` | loading the program and creating the perf-event map | +| `CAP_PERFMON` | `perf_event_open(2)`, the ring buffer that carries frames to userspace | +| `CAP_NET_RAW` | the `AF_PACKET` socket the default mode attaches to | +| `CAP_NET_ADMIN` | `-tc` only: the qdisc, the filter, and `IFF_PROMISC` | + +The row worth remembering is `CAP_BPF` + `CAP_NET_RAW` without +`CAP_PERFMON`: the program loads, attaches, and delivers nothing — which +looks exactly like a quiet network. `docker/README.md` has the full table, +including what each missing grant says when it fails. + +## What the kernel program does + +The filter is C, generated at import from `cryptomon/ports.py` so that +changing the watched ports is an environment variable rather than an edit to +a string literal. It: + +* reads every offset from the packet rather than assuming any of them — the + ethertype decides the family, stacked VLAN tags are stepped over, the IPv4 + header length comes from IHL, and for IPv6 the extension-header chain is + walked, because IPv6 moved options out of the fixed header and there is no + length field to read; +* refuses non-initial IPv6 fragments, which carry no transport header; +* is **TCP only**. It previously let UDP and ICMP through to the port checks + and then read a TCP data offset out of them, which decoded into nothing; +* checks the destination or source port against `TLS_PORTS` or `SSH_PORTS`; +* and forwards the frame only if the TCP payload *starts with* a TLS + handshake record (`0x16`, major 3, minor 1–4) or an SSH banner. + +That last condition is the whole reason the live path sees less than the +offline one. See [architecture.md](architecture.md). + +The filter has no test that pytest can reach, because a mistake in it does +not produce a wrong answer — it produces a service that will not start, which +is [issue #26](troubleshooting.md#bcc-will-not-install-or-will-not-compile-issue-26) +exactly. Two tools exist instead, both runnable in a container and both CI +jobs: `tests/tools/check_bpf.py` compiles the program and loads it past the +verifier in both attach modes, and `tests/tools/check_bpf_behaviour.py` +attaches it to loopback and replays the committed framing fixtures. Both read +`bpf.py` straight from its path rather than importing the package, so neither +needs the runtime dependencies. + +A malformed `TLS_PORTS` raises at import, before any of this — the traceback +ends: + +```console +$ TLS_PORTS=443,https python3 ./cryptomon.py -i eth0 +ValueError: TLS_PORTS: 'https' is not a port number +``` + +and without bcc, the monitor says so in as many words: + +```console +$ python3 ./cryptomon.py -i eth0 +Exception: bcc is not available, so the live monitor cannot start. It is +Linux-only and comes from your distribution rather than pip: run +ubuntu-setup.sh, or `apt-get install bpfcc-tools python3-bpfcc`. Parsing a +capture offline does not need it. +``` + +(Both need `DB_URL` and `DB_NAME` set, because `cryptomon.py` imports +`fapi.config`, which validates them first.) + +## What it writes + +One document per parsed frame, with `ptype` of `client` or `server`, and `ts` +set at insertion rather than from the packet. That is a different shape from +what the offline analyser produces — one document per *session* with `ptype` +of `session` and `ts` from the capture — and both shapes can share a +collection. `/stats/overview`'s `by_ptype` is how a reader tells which one +they are looking at. + +Writes are issued through motor without blocking the packet path, and the +future is kept so the result is retrieved. That matters: motor 3.5.1 returns +an already-scheduled future, so the write goes out even when nothing awaits +it — but the result was then discarded, and with it every authentication +failure, network error and duplicate key. Failures are now counted in +`cryptomon.WRITE_STATS` and the first of each kind is printed once. + +At most 1000 inserts may be in flight. Past that, new records are **dropped** +and counted as `dropped_backpressure`, because a burst of handshakes must not +be allowed to queue futures without bound. If you are seeing gaps, that +counter is the first thing to look at. + +`tag` — the field the dashboard's capture filter uses — comes from the +`data_tag` constructor argument. `cryptomon.py` passes an empty string, so +nothing the shipped command line writes is ever tagged. To tag a run you have +to drive `CryptoMon` as a library, or add the field during a `mongoimport` of +offline output (see [offline-analysis.md](offline-analysis.md#ndjson-and-the-point-of-it)). + +There is a TinyDB backend, selected by constructing `CryptoMon(mongodb=False)` +and writing to `cryptomon.json` in the working directory. `cryptomon.py` +always passes `mongodb=True`, so it is reachable from Python and not from the +command line. + +## What it cannot see + +The live path receives one frame at a time, with no reassembly, so it can +only read a handshake that starts at the beginning of a TCP payload and +finishes inside the same packet. Concretely, across the eleven corpus +captures it reported 411 "client hellos" of which **272 were truncated +records parsed as though they were whole** — not missed, reported, incomplete +and without saying so. Certificates were worse: 15 fragments against 93 whole +chains from the offline path. + +It also cannot see: + +* **Traffic on ports it is not watching.** 13 of 1369 TLS flows in the corpus + (0.95%) are on ports the filter does not watch — twelve on 53443, one on + 888. The offline analyser has no port filter and finds them. +* **A segment where a ChangeCipherSpec record sits ahead of the handshake**, + because the filter tests only the first three bytes of the payload. That is + a measured 7% of the corpus. +* **The second half of a HelloRetryRequest**, so a post-quantum key exchange + the server *refused* is indistinguishable from one it accepted. +* **Anything over UDP.** The kernel filter reads IP protocol 6 and nothing + else, so UDP never reaches userspace on this path. The offline analyser + does read it. +* **Encapsulated traffic.** A SPAN or ERSPAN feed arriving on a monitored + interface is IP protocol 47, which the filter refuses, so a mirror port + produces no events. The offline analyser unwraps it. +* **DTLS**, in either path. + +None of this is a defect in the filter. It is what a kernel filter that sees +one packet at a time can do, and it is why the offline analyser exists. + +## Running it as a service + +`create-service.sh` installs two systemd units — the sensor, which needs +`CAP_BPF`, `CAP_PERFMON` and `CAP_NET_ADMIN` and a network interface, and the +API, which needs neither — and starts neither of them, so that you can read +the unit files and put the database password in place first. + +```bash +sudo ./create-service.sh --nginx +``` + +[deploy/README.md](../deploy/README.md) is the reference for +what it installs and why, including the one systemd hardening directive that +is deliberately absent: `SystemCallFilter=~@resources` appears in nearly +every hardening guide and contains `setrlimit`, which `pcapscan.sandbox` +calls on itself to bound an upload. Filtered, every analysis would run with +no memory and no CPU ceiling, reports would still come out correct, and the +only symptom would be that the protection the upload path is built on is +gone. diff --git a/docs/offline-analysis.md b/docs/offline-analysis.md new file mode 100644 index 0000000..d1a0eb3 --- /dev/null +++ b/docs/offline-analysis.md @@ -0,0 +1,321 @@ +# Analysing captures offline + +```bash +python -m pcapscan capture.pcap +``` + +`pcapscan` reads a capture file directly. It needs no root, no eBPF, no +interface and no database, it runs anywhere Python 3.10 does, and it uses one +third-party package (`cryptography`, for the X.509 chain). It also sees +considerably more than the live sensor can, because it reassembles TCP before +it parses anything — see [architecture.md](architecture.md). + +It ships before any UI because it is the thing that gets automated. A web +form is used by whoever is sitting in front of it; a command that reads a +capture and writes NDJSON on standard output is used by a cron job, a CI step +and a pipeline somebody writes six months from now without asking. + +## The report + +The default output is a readable summary. This is the committed +HelloRetryRequest fixture — one real handshake, 24 packets: + +```console +$ python -m pcapscan tests/fixtures/streams/tls13_hello_retry.pcap +pcapscan: read tests/fixtures/streams/tls13_hello_retry.pcap + +Sessions 1 + tls 1 + +Key exchange + performed 1 + none (resumed) 0 + post-quantum 0 (0.0%) + hybrid 0 (0.0%) + classical 1 (100.0%) + unknown 0 (0.0%) + quantum-safe 0.0% of key exchanges performed + +Post-quantum offers refused by the server (1) + X25519Kyber768Draft00 -> secp256r1 x1 + unreadable (TLS 1.3) 1 sessions + +TLS versions + TLSv1.3 1 + +Algorithms observed + ciphersuite TLS_AES_256_GCM_SHA384 symmetric 1 + key-exchange secp256r1 classical 1 +``` + +That is the finding this tool exists to produce, in miniature. The client +offered a post-quantum hybrid group; the server refused it with a +HelloRetryRequest and named a classical curve instead. A reader that sees +only the first ClientHello reports "post-quantum key exchange offered" and +never records that it was turned down — which, for a tool measuring +post-quantum readiness, is the opposite of what happened. + +[reading-a-report.md](reading-a-report.md) explains every line of that +output, including why `quantum-safe` is a percentage of *key exchanges +performed* rather than of sessions. + +## Output formats + +| `-f` | What it is for | +|---|---| +| `summary` | The default. A person reading it. | +| `ndjson` | One JSON document per line, in the same shape the live sensor writes. Streams. The format to pipe into MongoDB. | +| `json` | One document: the records as an array, plus the analysis summary. Streams, with the summary appended at the end. | +| `csv` | One flat row per session. For a spreadsheet. | +| `cbom` | A CycloneDX 1.6 Cryptography Bill of Materials over the whole body of traffic. | + +```bash +python -m pcapscan capture.pcap # readable report +python -m pcapscan capture.pcap -f csv -o out.csv # for a spreadsheet +python -m pcapscan capture.pcap -f cbom > cbom.json # CycloneDX 1.6 +python -m pcapscan capture.pcap -f json -o report.json +``` + +Formats are *discovered* rather than listed: `cbom` appears in `--help` only +when its dependencies are present, so an option that would fail is absent +rather than offered and broken. + +### NDJSON, and the point of it + +The NDJSON records keep the document shape the live sensor already writes — +`ptype`, `eth.src.ipv4`, `tls.ciphersuite` and the rest in the same places. +That is not nostalgia. It means the offline path feeds the same collection as +the live path, with no translation step: + +```console +$ python -m pcapscan *.pcap -f ndjson -q \ + | mongoimport --uri "mongodb://127.0.0.1:27017/cryptomon" --collection cryptomon +connected to: mongodb://127.0.0.1:27017/cryptomon +1260 document(s) imported successfully. 0 document(s) failed to import. +``` + +(1260 is the whole project corpus. Yours will differ.) + +`-q` because the per-file progress goes to standard error and would otherwise +interleave with `mongoimport`'s own output. Note that `mongoimport` defaults +to `localhost:27017` and to the `test` database, so name the database in the +URI or with `--db`. + +**Tagging an import.** The dashboard and `/stats` can filter by a `tag` +field, which groups documents by capture run. Nothing in the pipeline writes +one — `tag` is only ever set by the live sensor's `data_tag`, and the shipped +`cryptomon.py` leaves that empty. Add it on the way past: + +```bash +python -m pcapscan capture.pcap -f ndjson -q \ + | python -c 'import sys, json +for line in sys.stdin: + if line.strip(): + print(json.dumps(dict(json.loads(line), tag="office-wifi")))' \ + | mongoimport --uri "mongodb://127.0.0.1:27017/cryptomon" --collection cryptomon +``` + +### CSV + +One row per session, 28 columns, ordered so that the identifying fields come +first and the verdict is visible without scrolling: + +``` +ts, time, duration, src, src_port, dst, dst_port, protocol, hostname, ech, +ja4, ja4s, tls_version, ciphersuite, kex_group, kex_verdict, resumption, +hello_retry_request, offered_kex_group, retry_kex_group, +certificate_subject, certificate_issuer, certificate_key, +certificate_not_after, certificate_count, proposed_groups, +proposed_ciphersuites, alerts +``` + +Flat and lossy, deliberately. It exists because the people who most need to +read "which of our connections would survive a quantum computer" open things +in a spreadsheet, and a nested document is not that. List-valued columns are +joined with `;`. The float epoch `ts` is kept *beside* a readable ISO `time` +rather than replaced by it, because a format that cannot round-trip is a +format that quietly loses data. + +### CBOM + +```bash +python -m pcapscan capture.pcap -f cbom -o cbom.json +``` + +A CycloneDX 1.6 document with one component per protocol version observed, +one per distinct algorithm, and one per distinct certificate, plus a +dependency graph built from what was seen *together*. "TLS 1.2 depends on +secp256r1" is then a statement about this capture; attaching every algorithm +to every version would be a statement about nothing. + +The serial number is derived from the content, so two runs over the same +traffic produce byte-identical documents and a CBOM can be diffed against +last month's — the difference is then a change in the estate rather than a +change in the clock. The timestamp lives in `metadata`, outside the hash. + +This is a different document from `bom.json` in the repository root. That one +inventories *this software's* dependencies. This one inventories the +cryptography on *your network*. + +## Mirrored and tunnelled traffic + +Nobody monitors a corporate network from an endpoint. They configure a SPAN, +RSPAN or ERSPAN session on a switch, or hang a TAP off a link, and the +mirrored traffic arrives somewhere else — encapsulated. The common +encapsulation is ERSPAN, which wraps the whole original Ethernet frame in GRE +inside IP. + +`pcapscan` unwraps that before it frames anything, so a capture taken off a +mirror port reports what it carries instead of reporting nothing: + +| | | +|---|---| +| GRE | RFC 2784, and everything built on it | +| ERSPAN | Types I, II and III | +| VXLAN | | +| GENEVE | | +| IP-in-IP | RFC 2003 | + +You do not ask for this and there is no flag: encapsulated frames are +resolved on the way in, and what a tunnel refuses is counted in the +`tunnel_*` family in `--stats` rather than dropped in silence. + +**There is no tunnelled traffic in this project's capture corpus** — zero +packets of IP protocol 47 in any of it, and no VXLAN or GENEVE either. The +fixtures this is checked against are built to the RFCs and committed. They +are not evidence about real switches, and should not be read as any. + +This is the offline path only. **The live eBPF sensor is still blind to +tunnels**: the kernel filter reads IP protocol 6 and nothing else, so +encapsulated traffic on a monitored interface produces no events at all. + +## Several captures, and captures from a pipe + +```bash +python -m pcapscan *.pcap *.pcapng # analysed as one body of traffic +python -m pcapscan mon-*.pcap -f ndjson # ditto, as documents +zcat big.pcap.gz | python -m pcapscan - # from standard input +python -m pcapscan capture.pcap.gz # gzip, sniffed from the magic +``` + +pcap and pcapng are both read, gzipped or not, and gzip is detected from the +file's magic bytes rather than its name: a `.pcap` written by +`tcpdump -z gzip` is gzipped, and a `.gz` that is not is a file somebody +renamed. + +Several captures given at once are fed to **one** session builder and +finished once at the end, not per file. `tcpdump -C` rotates a capture +mid-connection, and finishing after each part would report the two halves of +one handshake as two incomplete sessions. The practical consequence is worth +knowing: analysing twelve captures together and analysing them one at a time +give different session counts, because flows that appear in more than one +file are merged in the first case and counted twice in the second. The +per-file total is always the larger of the two. + +One unreadable file among several does not lose the others, and is not passed +over in silence either: + +```console +$ python -m pcapscan notacapture.pcap tests/fixtures/streams/tls13_hello_retry.pcap -f csv -q -o out.csv +pcapscan: notacapture.pcap: unrecognised capture magic b'hell'; expected pcap or pcapng +$ echo $? +1 +$ wc -l < out.csv +2 +``` + +Exit status is 0 for a clean run and 1 if any capture could not be read. + +## Flags + +| Flag | Default | What it does | +|---|---|---| +| `-f`, `--format` | `summary` | One of `summary`, `ndjson`, `json`, `csv`, `cbom`. | +| `-o`, `--output` | stdout | Write here instead. `-` also means stdout. | +| `-q`, `--quiet` | off | Suppress the per-file `pcapscan: read …` progress on stderr. Use it when piping. | +| `--stats` | off | Write the reader and reassembler counters to stderr when the run finishes. | +| `--no-certificates` | off | Skip X.509 parsing; keep the raw DER chain. | +| `--max-stream-bytes` | 16384 | Reassembly buffer per direction, in bytes. | +| `--max-flows` | 2048 | Connections tracked at once. | + +`--no-certificates` is the honest way to say "do not look at the +certificates": the record carries `tls.certificates_der`, a list of the raw +DER messages, in place of the parsed `tls.certificates`, so nothing on the +wire is thrown away and whoever reads the record can do as they like with it. +Use it when you do not have `cryptography`, when you want the analysis to +cost less, or when you intend to parse the chain with something else. + +`--max-stream-bytes` is 16KB because the whole plaintext handshake fits in +that with room to spare, and past it a stream is bulk transfer rather than +negotiation. Raise it only for a capture with unusually large certificate +chains; raising it raises the memory a hostile capture can make the process +hold. + +`--max-flows` bounds an LRU table of connections. A capture with more +simultaneous connections than this evicts the oldest, which is visible as +`evicted` in `--stats`. + +## `--stats`, and reading it + +```console +$ python -m pcapscan Firefox_Mac.pcap -q --stats -f ndjson -o /dev/null +pcapscan: {'capture_bytes': 5711078, 'capture_oversize_refused': 0, + 'capture_packets': 8582, 'capture_sections': 0, 'capture_truncated_tail': 0, + 'capture_unknown_blocks': 0, 'flows_http': 34, 'flows_not_tls_or_ssh': 30, + 'frames': 8312, 'frames_undecodable': 1, 'sessions': 114, + 'sessions_without_handshake': 12, 'udp_datagrams': 269, 'udp_flows': 19, + 'udp_payload_bytes': 135573, ...} +pcapscan: {'ignored_stateless': 115, 'flows': 262, 'segments': 8197, + 'payload_bytes': 5010119, 'reassembled_bytes': 433416, 'abandoned': 209, + 'resets': 51, 'open_streams': 262, 'held_bytes': 28506, + 'pending_segments': 0} +``` + +This is a dict, so the exact set of keys depends on what the capture +contained and on which protocol handlers this installation has. The `udp_*` +and `tunnel_*` families in particular are growing as handlers land; treat the +list as self-describing rather than fixed. + +The first line is the reader and the session builder, the second the +reassembler. Nothing here is refused silently: every frame that was dropped +is counted somewhere, so a run that quietly discarded data does not look like +a run that saw none. + +The counters worth knowing: + +| Counter | Means | +|---|---| +| `capture_truncated_tail` | The file was cut off mid-record. A normal thing for an interrupted `tcpdump`, not a corrupt file. | +| `capture_oversize_refused` | A length field said a frame was larger than 1MB. The stream is no longer trusted past that point. | +| `frames_undecodable` | The link layer could not be decoded, or the frame carried no transport header this tool reads. Now a small number, because UDP is decoded rather than dropped. | +| `sessions` vs `sessions_emitted` | TCP flows tracked, against records produced. `sessions_emitted` also includes the UDP records counted by `datagram_sessions_emitted`, so the two are not a like-for-like pair on a capture with UDP in it. | +| `flows_http`, `flows_not_tls_or_ssh`, `flows_stun` | Flows identified as something else, from their bytes, and abandoned. | +| `udp_datagrams`, `udp_flows`, `udp_payload_bytes` | What reached the UDP side. `udp_flows_capped` and `udp_flows_unrecognised` say what it declined to keep. | +| `tunnel_*` | Encapsulated traffic that was refused, and why — see [Mirrored and tunnelled traffic](#mirrored-and-tunnelled-traffic). | +| `abandoned` | Flows dropped on their first segment because they could be told not to carry a handshake. This is what makes the memory ceiling real. | +| `evicted` | Flows pushed out of the LRU table by `--max-flows`. If this is non-zero and you care about completeness, raise it. | +| `held_bytes`, `pending_segments` | What the reassembler was still holding when the run ended. Should be small. | + +## How fast, and how much memory + +The whole project corpus — twelve captures, 265MB on disk, 159,929 packets — +analyses in **under two seconds**, with certificate parsing on. + +The upload sandbox's limits are set against that measurement rather than by +taste: 2 GiB of address space and 120 CPU seconds leave roughly two orders of +magnitude of headroom and still stop a runaway well before it troubles the +host. See [service.md](service.md#what-happens-to-an-uploaded-capture). + +## Replaying a capture at the live sensor + +```bash +./parse-pcap.sh capture.pcap +``` + +This replays the capture over the loopback interface for the live eBPF sensor +to parse, which exercises the same code path production uses. It needs root +and the database variables set, and it sees only what a single-packet reader +can see. Prefer `pcapscan` unless you are specifically testing the live path. + +> **Not verified on this machine.** `parse-pcap.sh` needs Linux, root, an +> eBPF-capable kernel and bcc. diff --git a/docs/reading-a-report.md b/docs/reading-a-report.md new file mode 100644 index 0000000..d0358fa --- /dev/null +++ b/docs/reading-a-report.md @@ -0,0 +1,405 @@ +# Reading a report + +A CryptoMon report is full of strings like `X25519MLKEM768`, `hybrid`, +`t13d1516h2_8daaf6152771_02713d6af862`, `resumption: resumed` and +`ech: offered`. This page says what each of them means and which of them +should worry you. + +## First: a percentage without its base means nothing + +This is the most important idea in the project, and it is not a caveat — it +is the finding. + +Over the project's twelve capture corpus, 116 key exchanges used a +post-quantum hybrid group. Here are two true statements about that number: + +> **18.6%** of the 624 sessions that performed a key exchange were +> quantum-safe. +> +> **8.6%** of the 1346 TLS sessions were quantum-safe. + +They differ because **a resumed session performs no key exchange at all**. +722 of those 1346 performed none — they resumed an earlier one, or the +capture did not contain the exchange. There was no key agreement in them to +be quantum-safe or otherwise. Counting them in the denominator understates +readiness by the resumption rate. Leaving them out overstates the share of +traffic that is actually protected against a store-now-decrypt-later +adversary, because a resumed session's keys descend from the original +exchange, and that exchange was classical. + +Which number is right depends on the question: + +| The question | The denominator | +|---|---| +| "Are our clients and servers configured to negotiate PQ?" | key exchanges performed | +| "What fraction of this traffic is protected from a future quantum adversary?" | TLS sessions | +| "How much work is left?" | both, stated together | + +CryptoMon prints the base beside every figure and never on its own. The +summary says `18.6% of key exchanges performed`, not `18.6%`. +`readiness.quantum_safe_fraction` in the JSON is explicitly over +`key_exchanges_performed`, and `readiness.no_key_exchange` is right beside +it. + +If you quote one of these numbers to anybody, quote its base with it. + +(Those corpus figures move as protocol handlers are added — a QUIC handshake +is a TLS handshake, so it lands in both the numerator and the TLS +denominator. Re-run `python -m pcapscan` rather than trusting a number +written down here. The *shape* of the argument does not move.) + +### One base you should not use: `readiness.sessions` + +**`readiness['sessions']` counts every record the run produced, across every +protocol — and every other field in that dict is about TLS.** Over the same +corpus it reads 2404, because the cleartext UDP flows (DNS, NTP, mDNS, +NetBIOS and the rest) are now sessions too. The key-exchange buckets still +sum to the TLS sessions and not to that: + +``` +hybrid 116 + classical 508 + no_key_exchange 722 = 1346 (TLS sessions) +readiness['sessions'] = 2404 (all records) + unexplained 1058 +``` + +116 out of 2404 is 4.8%, and it is a number about nothing: a DNS lookup was +never going to negotiate a key exchange, so putting it in the base does not +measure a shortfall, it manufactures one. Use `protocols['tls']` — or the +`Sessions` breakdown at the top of the summary, which lists the protocols +separately for exactly this reason — as the "all sessions" base, and read +`readiness['sessions']` as "records produced". + +A third denominator effect worth knowing about: **analysing captures together +and separately gives different session counts.** Several captures fed to one +run are finished once at the end, so a flow that spans two files is one +session; the same captures analysed one at a time produce two partial ones. +The second total is always the larger. Neither is wrong. See +[offline-analysis.md](offline-analysis.md#several-captures-and-captures-from-a-pipe). + +## The five verdicts + +Every algorithm CryptoMon sees gets exactly one of these. The distinction +between the first three is not cosmetic. + +| Verdict | Means | Examples | +|---|---|---| +| `post-quantum` | Believed to resist Shor and Grover, standing alone. | ML-KEM-768, ML-DSA, SLH-DSA, Falcon | +| `hybrid` | A post-quantum algorithm combined with a classical one, so it is no weaker than either. | `X25519MLKEM768`, `X25519Kyber768Draft00`, `sntrup761x25519-sha512@openssh.com` | +| `classical` | Broken by Shor. | RSA, ECDSA, EdDSA, DSA, `x25519`, `secp256r1`, `ffdhe2048` | +| `symmetric` | Names no asymmetric primitive at all. | `TLS_AES_256_GCM_SHA384` | +| `unknown` | Not recognised. Counted, never guessed at. | — | + +**`hybrid` is its own answer and not a rounding of either neighbour.** Every +post-quantum key exchange in the corpus is hybrid — there is no +post-quantum-only exchange anywhere in it. Reporting those as `post-quantum` +would overstate deployment; reporting them as `classical` would erase the +work. Hybrid is what the industry is actually shipping, because neither half +is trusted alone yet, and it is what your report will be full of. + +**`symmetric` is not "unknown".** A TLS 1.3 ciphersuite name says nothing +about the key exchange, because in TLS 1.3 the suite no longer carries it — +the key exchange moved into an extension. `TLS_AES_256_GCM_SHA384` is a +complete and correct suite name that describes only the record-layer cipher +and the hash. Reporting it as unrecognised would read as a gap in the +classification table, which it is not. + +**`unknown` is a gap in CryptoMon's table, not a finding about your +traffic.** It is never folded into `classical`. + +There is also `none`, which appears where a *session* is being classified +rather than an algorithm: it means no key exchange was performed. It is kept +separate from every verdict for the reason at the top of this page. + +### What to do about each + +| You see | It means | Priority | +|---|---|---| +| `classical` key exchange | A quantum adversary recording this traffic today can read it once it has a machine. | This is the population you are trying to shrink. | +| `hybrid` key exchange | Already protected against that adversary. | None. Count it. | +| `post-quantum` refused | Your client offered PQ; the far end said no. | **High.** The fix is somebody else's configuration and they may not know. | +| `classical` certificate key | Signature, not confidentiality. A quantum adversary cannot retroactively forge a signature made today. | Lower than key exchange, but it is the migration with the longest tail. | +| `unknown` | CryptoMon does not recognise the name. | Worth reporting as an issue. | + +## Key exchange groups + +The `kex_group` field, and the "Key exchange" section of the summary. + +| Name | Verdict | What it is | +|---|---|---| +| `x25519` | classical | Curve25519 ECDH. The most common by far. | +| `secp256r1`, `secp384r1`, `secp521r1` | classical | NIST P-256/384/521 ECDH. | +| `X25519MLKEM768` | hybrid | X25519 combined with ML-KEM-768 (the standardised Kyber). The one that is winning. | +| `X25519Kyber768Draft00` | hybrid | The same idea with the pre-standard Kyber draft. Its presence dates a client. | +| `ffdhe2048`, `ffdhe3072` | classical | Named finite-field Diffie–Hellman groups. | +| `ffdhe512`, `ffdhe1024` | classical | Not a named group — the size read off a TLS 1.2 ServerKeyExchange, where the prime is sent inline, so the prime's length is the only thing identifying the strength. **512 bits is broken today, by a classical computer.** | +| `finite field DH` | classical | Finite-field DH whose prime length could not be read. | +| `explicit EC parameters` | classical | A server sending curve parameters inline rather than naming a curve. Deprecated for good reason; rare and worth a look. | +| `none` | — | No key exchange. The session resumed. | + +Hybrid is decided by finding both a post-quantum and a classical marker in +the same name — `X25519MLKEM768` is X25519 and ML-KEM-768. The post-quantum +markers are matched **and removed** before the classical ones are looked for, +because ML-DSA and SLH-DSA both end in "dsa" and a naive substring match +would report every NIST signature selection as a hybrid of itself and DSA. + +## Certificates + +``` +Certificate keys + RSA-2048 274 + RSA-4096 84 + EC-384 18 + EC-256 13 + quantum-vulnerable 389 of 389 + unreadable (TLS 1.3) 336 sessions +``` + +The size travels with the algorithm because it is the only thing that makes +the verdict actionable. Every RSA key is broken by Shor, and the ones that +have to be replaced first are not chosen at random. + +**`unreadable (TLS 1.3)` is not "no certificate".** In TLS 1.3 the server's +Certificate message travels inside the encrypted handshake flight, so a +passive observer cannot read it. CryptoMon records `certificates_unreadable` +for those sessions rather than reporting nothing, because "not readable" and +"not sent" are different claims and a readiness report that conflated them +would undercount the certificates in use. Expect this number to be large and +to grow. + +An `unreadable` label in the *key* column is different again: it means the +certificate was on the wire and did not decode far enough to have a key. + +Certificate keys are a slower-burning problem than key exchange. A signature +made today cannot be retroactively forged by a quantum computer in ten years' +time — the signature only has to be unforgeable while it is being relied on. +Key exchange is the opposite: traffic recorded today can be decrypted later. +That is why the headline on the dashboard is about key exchange. + +## Symmetric strength + +``` +"AES_256 (256 bits, 128 after Grover)": 321 +"AES_128 (128 bits, 64 after Grover)": 838 +"RC4 (broken independently of any quantum computer)": 10 +``` + +Symmetric ciphers are judged separately, because Grover's algorithm halves an +exhaustive search rather than breaking the algorithm. AES-256 retains 128 +bits against it; AES-128 retains 64. Calling AES-128 "quantum-vulnerable" +alongside RSA would be wrong by many orders of magnitude. + +RC4, 3DES, single DES and NULL get their own label, because reporting RC4 as +"128 bits, 64 after Grover" would answer the wrong question by a wide margin: +it is not waiting for a quantum computer. `broken_symmetric_ciphers` in the +readiness block counts these, and any non-zero value there is a finding today. + +## Resumption + +`resumption` is one of `resumed`, `fresh` or `unknown`, and +`resumption_evidence` says how it was decided: + +| `resumption` | `resumption_evidence` | Strength | +|---|---|---| +| `resumed` | `server selected pre_shared_key` | Unambiguous. In TLS 1.3 that *is* the acceptance. | +| `resumed` | `inferred: abbreviated handshake, no certificate` | Inferred. A TLS 1.2 abbreviated handshake sends no Certificate. Reliable in practice, and labelled so you know which you got. | +| `fresh` | `TLS 1.3 without pre_shared_key` | Unambiguous. | +| `fresh` | `certificate sent` | Unambiguous. | +| `unknown` | `no server hello` / `handshake incomplete in the capture` | The capture did not contain enough. | + +Roughly half the corpus resumes (625 resumed, 535 fresh, 100 unknown out of +1260). If your resumption rate is high, the gap between your two +quantum-safe percentages will be wide, and the "of all sessions" figure is +the more honest one to plan against — resumed sessions inherit their security +from an earlier classical exchange. + +## `ech` — Encrypted ClientHello + +This field sits directly under `hostname` on purpose, because it is the +qualifier on the hostname. + +| Value | Means | +|---|---| +| `offered` | The ClientHello carried the extension with the outer type byte. This is what you will see. | +| `accepted` | A ServerHello or HelloRetryRequest echoed it back. | +| `inner` | An inner-hello type byte seen in plaintext. The message is not what it claims to be. | +| `malformed` | The extension did not parse. | + +**While servers decline ECH, the hostname you see is the real one. Once they +accept it, it is a public outer name and the real destination is inside the +encrypted inner hello.** A hostname that quietly stops being the hostname is +unlabelled missing data, which is the defect this whole field exists to +prevent: `www.example.com` alone is a claim, `www.example.com, ech=offered` +is an observation with its uncertainty attached. + +Two honest negatives: + +* **Absence of `accepted` is not evidence of rejection.** A server accepting + ECH in a ServerHello does not echo the extension at all — the acceptance + signal is eight bytes of `ServerHello.random` derived from the inner + transcript, which a passive observer cannot check without the inner hello + it is derived from. `accepted` is only reachable through a + HelloRetryRequest echo. Across the corpus, 465 sessions offer ECH and + **zero** of 1170 ServerHellos echo it. +* **`grease` is never emitted.** A GREASE ECH is by design a well-formed + outer hello with a random config id and payload, which is exactly what a + real one looks like from outside. In aggregate this corpus is plainly + GREASE — all 465 offers carry a plausible plaintext SNI, 150 distinct + names, none of them a public outer name — but aggregate is not per-session, + so no session is labelled `grease` rather than labelled wrongly. + +## `ja4`, `ja4s` and `ja3` — which client, and which server + +`cryptomon.analysis` answers "what was negotiated". It cannot answer "by +whom", and the second question is what turns a number into an action: *"37% +of sessions offer no post-quantum key share"* is a finding, *"and all of it +is two builds of one browser"* is a ticket. + +### Reading a JA4 + +``` +t13d1516h2_8daaf6152771_02713d6af862 +│││ │ │ │ │ └─ sha256 of the extension list (SNI and ALPN +│││ │ │ │ │ removed, sorted) + "_" + signature algorithms +│││ │ │ │ │ in wire order, truncated to 12 hex characters +│││ │ │ │ └─ sha256 of the sorted ciphersuite list, truncated +│││ │ │ └─ first and last character of the first ALPN value ("h2") +│││ │ └─ 16 extensions, GREASE excluded, SNI and ALPN included +│││ └─ 15 ciphersuites, GREASE excluded +││└─ "d" = SNI present; "i" = absent +│└─ highest version the client *offers* in supported_versions: TLS 1.3 +└─ transport: "t" = TCP +``` + +That exact string is FoxIO's published Chrome fingerprint, reproduced from a +committed fixture — which is the point of it. A fingerprint that matches what +everybody else records is a joinable identifier; one that is merely +self-consistent is not. + +Four details of the specification are easy to read past, and each is pinned +by a golden value in the test suite. The extension *count* includes SNI and +ALPN while the extension *hash* excludes them — the count says how talkative +the client is, the hash must not change when it visits a different host over +a different protocol. Signature algorithms are appended in wire order, not +sorted, because a client's preference order is signal while its extension +order (which Chrome randomises per connection) is not. The version is the +highest offered in `supported_versions`, not the legacy version in the +handshake header, which every modern hello sets to TLS 1.2. + +### JA4S + +``` +t130200_1302_a56c5b993250 +│││ │ │ └─ sha256 of the server's extensions, in the order it sent them +│││ │ └─ the one cipher the server chose, in hex (0x1302 = +│││ │ TLS_AES_256_GCM_SHA384) +│││ └─ first/last of ALPN, or "00" for none +││└─ 2 extensions +│└─ TLS 1.3 +└─ TCP +``` + +Servers are far less varied than clients — 45 distinct JA4S across 1170 +ServerHellos in the corpus — so JA4S is weak alone and strong paired with the +JA4 of the hello it answered. One server answering two clients differently is +the interesting shape. + +### Why JA4 and not JA3 + +Over the same 1260 hellos: **32 distinct JA4 against 393 distinct JA3.** The +340 Chromium sessions that share one JA4 produce 340 distinct JA3s — one per +connection — because Chrome shuffles its extension order and JA3 hashes that +order. `ja3` is still emitted, because Zeek, Suricata, NetworkMiner and a +decade of threat-intel feeds are keyed on it and a record that cannot be +joined to what you already have is less useful than one that can. It is not +to be trusted for anything security-bearing. + +**Neither is authentication.** A hello can be replayed byte for byte, and +several tools exist that do exactly that. A JA4 match is evidence about a +client, never proof about a peer. + +## Alerts + +``` +handshake_failure (server) 29 +certificate_unknown (client) 11 +inappropriate_fallback (server) 2 +``` + +Plaintext TLS alerts seen during a handshake, with the side that sent them. +The direction is the half that carries the finding: merged, "a middlebox +refused our key share" and "we refused their parameters" are the same row. + +**The correlation this exists for** is a post-quantum group offered followed +by a fatal `handshake_failure`. Where that is a server — or, more often, a +middlebox between the two — that cannot cope with a ClientHello carrying a +kilobyte of ML-KEM key share, it is the most actionable thing a readiness +survey produces, because the fix is somebody else's configuration and they do +not know yet. + +In this corpus it is not that. All 24 correlated failures are +`dh512.badssl.com` and `dh1024.badssl.com`, deliberately weak test servers, +and the post-quantum offer is incidental — 448 of the 1260 sessions offer +one, because current browsers offer one on every connection. The correlation +is the right thing to compute and this corpus does not contain the +intolerance it looks for. Yours might. + +**An empty alerts panel is not evidence that connections closed cleanly.** +`close_notify` is sent under the negotiated keys, and CryptoMon stops reading +a stream at the first encrypted record, so every plaintext alert it can see +is by construction one sent *during* a handshake. `close_notify` never +appearing is a property of where the observer stands, not of the network. + +## `hello_retry_request`, `offered_kex_group`, `retry_kex_group` + +When these three appear together, you are looking at the single most useful +finding in a readiness survey: + +``` +hello_retry_request: true +offered_kex_group: X25519Kyber768Draft00 +retry_kex_group: secp256r1 +kex_group: secp256r1 +``` + +The client offered a post-quantum hybrid. The server sent a +HelloRetryRequest naming a classical curve instead, and the connection +completed on that. In the summary these are collected under: + +``` +Post-quantum offers refused by the server (61) + X25519Kyber768Draft00 -> secp384r1 x48 + X25519Kyber768Draft00 -> secp256r1 x11 + X25519MLKEM768 -> secp256r1 x1 + X25519MLKEM768 -> secp384r1 x1 +``` + +A tool that sees only the first ClientHello reports "post-quantum key +exchange offered" and never records that it was turned down. That is the +opposite of what happened, and it is why the offline analyser reassembles +before it parses. Only `pcapscan` can see this; the live sensor cannot. + +## TLS versions + +`deprecated (RFC 8996)` counts TLS 1.0 and TLS 1.1, which are deprecated +outright. The corpus has 30 such sessions. This is an immediate finding, not +a quantum one, and it usually travels with the weak symmetric ciphers above. + +## The readiness block, field by field + +`summary.readiness` in the JSON and the report page. Every number here is a +count of sessions unless it says otherwise. + +| Field | Means | +|---|---| +| `sessions` | Every record the run produced, **across every protocol** — including cleartext UDP flows that were never going to negotiate anything. Not a base for any percentage here; use `protocols['tls']`. | +| `key_exchanges_performed` | TLS sessions where a key exchange happened. **The denominator for `quantum_safe_fraction`.** | +| `no_key_exchange` | The rest of the TLS sessions. Resumed, or the capture did not contain the exchange. `key_exchanges_performed + no_key_exchange` is the TLS session count, not `sessions`. | +| `post_quantum`, `hybrid`, `classical`, `unknown` | Sessions by key-exchange verdict. | +| `quantum_safe_fraction` | `(post_quantum + hybrid) / key_exchanges_performed`. `null` when nothing performed one. | +| `post_quantum_refused` | HelloRetryRequests that downgraded a PQ offer to a classical group. | +| `certificates_quantum_vulnerable` | Certificate keys judged classical. Counted per certificate, not per session — a chain contributes several. | +| `certificates_post_quantum` | The same for PQ signature keys. Expect 0 for now. | +| `certificates_unreadable` | Sessions whose certificate was inside a TLS 1.3 encrypted flight. | +| `ech_offered`, `ech_accepted` | See above. `ech_accepted` is almost always 0 and that is not evidence of rejection. | +| `deprecated_tls_versions` | TLS 1.0 and 1.1 sessions. | +| `broken_symmetric_ciphers` | Sessions using RC4, 3DES, DES or NULL. Broken today. | diff --git a/docs/service.md b/docs/service.md new file mode 100644 index 0000000..9f28a26 --- /dev/null +++ b/docs/service.md @@ -0,0 +1,369 @@ +# The service + +```bash +export DB_URL="mongodb://127.0.0.1:27017/cryptomon" +export DB_NAME="cryptomon" +python api.py +``` + +``` +INFO: Started server process [24436] +INFO: Waiting for application startup. +INFO: Application startup complete. +INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit) +``` + +One FastAPI application with three faces on the same port, and they are aimed +at different people. + +| Path | What it is | +|---|---| +| `/` | The dashboard. What the collection contains, as a page. | +| `/analyse/` | Upload a capture in a browser, get a report. Needs no database. | +| `/stats/*` | The dashboard's numbers, as JSON. Read-only. | +| `/data/*` | The raw documents. Reads open, writes guarded. | +| `/docs` | The generated OpenAPI documentation. | + +It binds `127.0.0.1` by default and writes are refused by default, so out of +the box this is a local tool. Both of those are one variable away from +changing — see [configuration.md](configuration.md) — and +[deploy/README.md](../deploy/README.md) is what to read before anybody else +can reach it. + +`DB_URL` and `DB_NAME` are required even for the parts that do not use the +database: `fapi.config` validates them at import. + +--- + +## The dashboard + +`http://127.0.0.1:8000/` + +It answers the project's question with the denominator beside it, because +that is the only way the number means anything. Rendered from a collection of +1,342 imported session documents: + +``` +81 quantum-safe key exchanges +13.5% of 598 of key exchanges performed + 6.0% of 1,342 of all sessions + 744 performed no key exchange +``` + +Read "of all sessions" as "of all documents in this window". That is the +right base for a collection of TLS sessions and the wrong one for a +collection that also holds cleartext UDP records, which an import of the +current `pcapscan` output will contain — see +[reading-a-report.md](reading-a-report.md#one-base-you-should-not-use-readinesssessions). +`/stats/overview` reports `protocols` and `by_ptype` beside the total so that +the mix is visible rather than assumed. + +Below that: key exchange by group and verdict, the same over time, negotiated +ciphersuites, TLS versions, certificate signing keys, JA4 client +fingerprints, server names, Encrypted ClientHello uptake, and TLS alerts with +the direction they came from. + +A window control (`?hours=`) and a capture-tag filter (`?tag=`) apply to +every panel. `hours=0` means all of time, which is a collection scan by +definition; the ceiling is 8760 (a year). + +**There is no JavaScript framework, nothing vendored and nothing fetched from +a CDN.** The charts are server-rendered inline SVG. The page renders in full +with JavaScript switched off — verified: the served HTML for the whole-corpus +view contains zero `