Add architecture, code map and glossary

A map of the code: processes and threads, the life of a call in each
context mode, the paths Python uses to call Erlang, which code is live and
which is legacy, and one meaning per overloaded word (worker, context,
pool). The stale file list in the NIF headers now points at these pages.
Also completes the 4.2.0 changelog.

Author: benoitc

CHANGELOG.md

@@ -34,7 +34,8 @@
   such a region with ring backpressure, usable as `wsgi.input` in isolated
   contexts. Handles are plain terms and travel inside any argument or result;
   `py_shm:read_only/1` and `new(Size, #{writable => false})` hand Python a
-  read-only mapping.
+  read-only mapping. `py_buffer:write/3` takes a timeout for the case where
+  the ring is full and nobody reads (default 30 s).
 - `py:python_executable/0`, `py:kill/1`, `py_nif:os_kill/2`.
 - `py_isolated` is a `gen_statem` (states `idle`, `{busy, Id}`, `looping`,
   `stopping_loop`, `{restarting, Reason}`): `sys:get_state/1` and
@@ -48,6 +49,11 @@
   storms, loop churn, 60 s mixed workload with resource counters checked.
 - Guide: `docs/isolated.md`, with what each of the three modes guarantees.
 
+### Fixed
+
+- `pthread_timedjoin_np` was called without `_GNU_SOURCE`, an implicit
+  declaration on Linux that newer compilers reject.
+
 ## 4.1.0 (2026-08-15)
 
 ### Added

README.md

@@ -654,6 +654,7 @@ py:execution_mode().  %% => worker | owngil
 
 ## Documentation
 
+- [Architecture](docs/architecture.md), [Code map](docs/code-map.md), [Glossary](docs/glossary.md) - how the pieces fit
 - [Getting Started](docs/getting-started.md)
 - [Process-Bound Environments](docs/process-bound-envs.md) - Isolated Python state per Erlang process
 - [AI Integration Guide](docs/ai-integration.md)

c_src/README.md

@@ -0,0 +1,71 @@
+# c_src
+
+The NIF that embeds CPython in the VM. One translation unit: `py_nif.c`
+`#include`s every other `.c` file (see the "Include module implementations"
+section near line 265), so build with `rebar3 compile` (CMake through
+`do_cmake.sh` / `do_build.sh`), never a single file. Headers declare what
+the included files share.
+
+Read `docs/architecture.md` first for how a call travels; this file says
+where things are.
+
+## Files
+
+| File | What it owns | Notes |
+|---|---|---|
+| `py_nif.h` | All shared types: `py_context_t` and its request queue, request types, callback and suspension state, runtime state machine, atoms, globals, declarations | 2.4k lines. The struct comments carry the locking rules; read `py_context_t` before touching threads |
+| `py_nif.c` | Runtime init and finalize, resource types, context create/destroy, the request queue, `worker_context_thread_main` and `owngil_context_thread_main`, `owngil_execute_*` (used by both thread kinds), the `nif_context_*` NIFs, process-local envs, `py_ref`, the NIF function table at the end | Sections are banner-separated; `grep -n '^ \* ===\|^/\* ==='` lists them |
+| `py_convert.c` | `py_to_term`, `term_to_py`, depth limits, tagged tuples (`{bytes, B}`, `{'$py_shm', ...}`), error tuples `{error, {Type, Msg}}` | The type mapping tables in the comments are the reference for `_etf.py` |
+| `py_exec.c` | Executing a call/eval/exec with suspension support; the legacy single executor thread | |
+| `py_callback.c` | The `erlang` Python module: `call`, `send`, `whereis`, `schedule*`, `Atom`/`Pid`/`Ref` types, callback delivery paths (suspension, blocking pipe, async pipe), channel and shared-dict methods, callback name registry | `erlang_call_impl` documents the path precedence |
+| `py_thread_worker.c` | Python threads calling Erlang through `py_thread_handler` | |
+| `py_subinterp_thread.c/.h` | Thread pool of sub-interpreters (owngil contexts, loop pools) | |
+| `py_event_loop.c/.h` | `ErlangEventLoop` support: `enif_select` readers and writers, timers, task injection into loops, reactor dispatch, fd registry, Python module `py_event_loop`; plus test-only fd/TCP/UDP NIFs (section "Test Helper Functions") | Largest file |
+| `py_channel.c/.h`, `py_buffer.c/.h`, `py_reactor_buffer.c/.h`, `py_shared_dict.c` | Resources with a Python-facing object each | |
+| `py_logging.c` | Logging and tracing NIFs | |
+| `py_mem_limit.c` | obmalloc arena accounting for owngil memory caps | |
+| `py_worker_pool.c/.h` | Legacy pool, no caller in `src/` | Candidate for removal |
+| `py_util.c/.h` | Macros, small helpers | |
+
+## Where the live paths are
+
+- `py:call/3` in worker or owngil mode: `nif_context_call_async` (`py_nif.c`)
+  enqueues; `worker_context_thread_main` or `owngil_context_thread_main`
+  dequeues and calls `owngil_execute_request`; the reply goes out as
+  `{py_result, Ref, Result}`.
+- `erlang.call` from Python: `erlang_call_impl` (`py_callback.c`).
+- Interrupt: `nif_context_interrupt` (`py_nif.c`), `interrupt_mutex` rules on
+  `py_context_t`.
+- Type conversion: `py_to_term` / `term_to_py` (`py_convert.c`).
+- Isolated mode has no C code of its own: `os_kill` is the only NIF it uses.
+
+## Rules that are easy to break
+
+- Only the context's thread touches the context's Python objects. NIFs
+  called from Erlang processes enqueue requests and return; they do not run
+  Python for a context that has a thread.
+- `Py_BEGIN_ALLOW_THREADS` around every blocking wait; never block on an
+  Erlang-side resource with the GIL held.
+- Never call a Future method while holding `async_futures_mutex`
+  (`py_callback.c`, comment above the struct explains the pattern).
+- `interrupt_mutex` is taken only by threads that do not hold the GIL.
+- `queue_mutex` protects the request queue of a context and is taken before
+  a request's own mutex (`ctx_queue_cancel_all`), never the other way.
+- A NIF that can block or run Python is registered with a dirty scheduler
+  flag in the table at the end of `py_nif.c`.
+
+## Adding a NIF
+
+1. Implement `static ERL_NIF_TERM nif_x(ErlNifEnv*, int, const ERL_NIF_TERM[])`
+   next to related code.
+2. Add `{"x", Arity, nif_x, Flags}` to `nif_funcs[]` at the end of `py_nif.c`.
+3. Add the stub and its `-spec` and doc to `src/py_nif.erl`.
+4. Cover it in a suite; `rebar3 dialyzer` and `rebar3 xref` must stay clean.
+
+## Legacy code, for orientation
+
+Not on the path of contexts created today: the `worker_*` NIFs and
+"Worker management" section, `async_worker_*` NIFs (return `deprecated`),
+the "Legacy mode" inline branches in `nif_context_call/eval/exec`,
+`py_worker_pool.c`, and the `cancel_reader/writer` aliases. When in doubt,
+follow `nif_context_call_async` and ignore the rest.

c_src/py_nif.c

@@ -29,11 +29,10 @@
  * - Resource types for Python objects to ensure proper cleanup
  * - Dirty NIF flags for GIL-holding operations
  *
- * This file is the main entry point. It includes the following modules:
- * - py_nif.h: Shared header with types and declarations
- * - py_convert.c: Type conversion (Python <-> Erlang)
- * - py_exec.c: Python execution and GIL management
- * - py_callback.c: Callback system and asyncio support
+ * This file is the main entry point and the single translation unit: it
+ * includes the other .c files (see "Include module implementations"). The
+ * file map is c_src/README.md; the request lifecycle per context mode is
+ * docs/architecture.md.
  */
 
 /* pthread_timedjoin_np (used to bound the owngil worker join on Linux)

c_src/py_nif.h

@@ -21,54 +21,19 @@
  *
  * @mainpage Python-Erlang NIF Integration
  *
- * @section intro_sec Introduction
- *
- * This NIF (Native Implemented Function) library provides seamless integration
- * between Erlang/OTP and Python. It embeds a Python interpreter within the
- * Erlang VM and provides bidirectional communication capabilities.
- *
- * @section arch_sec Architecture
- *
- * The implementation follows a modular design with four main components:
- *
- * - **py_nif.h** - Shared types, macros, and declarations
- * - **py_convert.c** - Bidirectional type conversion (Python ↔ Erlang)
- * - **py_exec.c** - Python execution engine and GIL management
- * - **py_callback.c** - Erlang callback support and asyncio integration
- *
- * @section modes_sec Execution Modes
- *
- * The library supports three execution modes based on Python version:
- *
- * | Mode | Python Version | Description |
- * |------|----------------|-------------|
- * | FREE_THREADED | 3.13+ (no-GIL) | Direct execution without GIL |
- * | SUBINTERP | 3.12+ | Per-interpreter GIL isolation |
- * | MULTI_EXECUTOR | Any | Multiple executor threads with GIL |
- *
- * @section gil_sec GIL Management
- *
- * The GIL (Global Interpreter Lock) is managed following PyO3/Granian patterns:
- *
- * - `Py_BEGIN_ALLOW_THREADS` / `Py_END_ALLOW_THREADS` around blocking ops
- * - Executor threads hold the GIL and process queued requests
- * - Dirty I/O schedulers are used for Python-calling NIFs
- *
- * @section callback_sec Callback Mechanism
- *
- * Python code can call back to Erlang using a suspension/resume pattern:
- *
- * 1. Python calls `erlang.call('func', args)`
- * 2. NIF raises `SuspensionRequired` exception
- * 3. Dirty scheduler is released, callback sent to Erlang
- * 4. Erlang processes callback, calls `resume_callback/2`
- * 5. Python execution resumes with cached result
- *
- * @section mem_sec Memory Management
- *
- * - Erlang resources wrap Python objects (prevent GC)
- * - Thread-local storage for callback context
- * - Proper cleanup in resource destructors
+ * This NIF embeds CPython in the Erlang VM. The map of the code, the life
+ * of a call in each context mode, the callback paths and the locking rules
+ * are documented in docs/architecture.md, c_src/README.md and
+ * docs/glossary.md; keep those current instead of this comment.
+ *
+ * In one paragraph: py_nif.c is the single translation unit and includes
+ * the other .c files; a context (py_context_t) owns a request queue and a
+ * pthread that runs Python for it (worker mode: main interpreter, shared
+ * GIL; owngil mode: a sub-interpreter with its own GIL); Erlang enqueues
+ * through nif_context_call_async and receives {py_result, Ref, Result};
+ * Python calls Erlang through erlang_call_impl (py_callback.c), by
+ * suspension in worker mode and a blocking pipe in owngil mode. Isolated
+ * mode runs Python in a child process and uses no C code beyond os_kill.
  */
 
 #ifndef PY_NIF_H