Merge pull request #74 from benoitc/docs-architecture

Add architecture, code map and glossary

Author: benoitc

CHANGELOG.md

@@ -34,7 +34,8 @@
   such a region with ring backpressure, usable as `wsgi.input` in isolated
   contexts. Handles are plain terms and travel inside any argument or result;
   `py_shm:read_only/1` and `new(Size, #{writable => false})` hand Python a
-  read-only mapping.
+  read-only mapping. `py_buffer:write/3` takes a timeout for the case where
+  the ring is full and nobody reads (default 30 s).
 - `py:python_executable/0`, `py:kill/1`, `py_nif:os_kill/2`.
 - `py_isolated` is a `gen_statem` (states `idle`, `{busy, Id}`, `looping`,
   `stopping_loop`, `{restarting, Reason}`): `sys:get_state/1` and
@@ -48,6 +49,11 @@
   storms, loop churn, 60 s mixed workload with resource counters checked.
 - Guide: `docs/isolated.md`, with what each of the three modes guarantees.
 
+### Fixed
+
+- `pthread_timedjoin_np` was called without `_GNU_SOURCE`, an implicit
+  declaration on Linux that newer compilers reject.
+
 ## 4.1.0 (2026-08-15)
 
 ### Added

README.md

@@ -654,6 +654,7 @@ py:execution_mode().  %% => worker | owngil
 
 ## Documentation
 
+- [Architecture](docs/architecture.md), [Code map](docs/code-map.md), [Glossary](docs/glossary.md) - how the pieces fit
 - [Getting Started](docs/getting-started.md)
 - [Process-Bound Environments](docs/process-bound-envs.md) - Isolated Python state per Erlang process
 - [AI Integration Guide](docs/ai-integration.md)

c_src/README.md

@@ -0,0 +1,71 @@
+# c_src
+
+The NIF that embeds CPython in the VM. One translation unit: `py_nif.c`
+`#include`s every other `.c` file (see the "Include module implementations"
+section near line 265), so build with `rebar3 compile` (CMake through
+`do_cmake.sh` / `do_build.sh`), never a single file. Headers declare what
+the included files share.
+
+Read `docs/architecture.md` first for how a call travels; this file says
+where things are.
+
+## Files
+
+| File | What it owns | Notes |
+|---|---|---|
+| `py_nif.h` | All shared types: `py_context_t` and its request queue, request types, callback and suspension state, runtime state machine, atoms, globals, declarations | 2.4k lines. The struct comments carry the locking rules; read `py_context_t` before touching threads |
+| `py_nif.c` | Runtime init and finalize, resource types, context create/destroy, the request queue, `worker_context_thread_main` and `owngil_context_thread_main`, `owngil_execute_*` (used by both thread kinds), the `nif_context_*` NIFs, process-local envs, `py_ref`, the NIF function table at the end | Sections are banner-separated; `grep -n '^ \* ===\|^/\* ==='` lists them |
+| `py_convert.c` | `py_to_term`, `term_to_py`, depth limits, tagged tuples (`{bytes, B}`, `{'$py_shm', ...}`), error tuples `{error, {Type, Msg}}` | The type mapping tables in the comments are the reference for `_etf.py` |
+| `py_exec.c` | Executing a call/eval/exec with suspension support; the legacy single executor thread | |
+| `py_callback.c` | The `erlang` Python module: `call`, `send`, `whereis`, `schedule*`, `Atom`/`Pid`/`Ref` types, callback delivery paths (suspension, blocking pipe, async pipe), channel and shared-dict methods, callback name registry | `erlang_call_impl` documents the path precedence |
+| `py_thread_worker.c` | Python threads calling Erlang through `py_thread_handler` | |
+| `py_subinterp_thread.c/.h` | Thread pool of sub-interpreters (owngil contexts, loop pools) | |
+| `py_event_loop.c/.h` | `ErlangEventLoop` support: `enif_select` readers and writers, timers, task injection into loops, reactor dispatch, fd registry, Python module `py_event_loop`; plus test-only fd/TCP/UDP NIFs (section "Test Helper Functions") | Largest file |
+| `py_channel.c/.h`, `py_buffer.c/.h`, `py_reactor_buffer.c/.h`, `py_shared_dict.c` | Resources with a Python-facing object each | |
+| `py_logging.c` | Logging and tracing NIFs | |
+| `py_mem_limit.c` | obmalloc arena accounting for owngil memory caps | |
+| `py_worker_pool.c/.h` | Legacy pool, no caller in `src/` | Candidate for removal |
+| `py_util.c/.h` | Macros, small helpers | |
+
+## Where the live paths are
+
+- `py:call/3` in worker or owngil mode: `nif_context_call_async` (`py_nif.c`)
+  enqueues; `worker_context_thread_main` or `owngil_context_thread_main`
+  dequeues and calls `owngil_execute_request`; the reply goes out as
+  `{py_result, Ref, Result}`.
+- `erlang.call` from Python: `erlang_call_impl` (`py_callback.c`).
+- Interrupt: `nif_context_interrupt` (`py_nif.c`), `interrupt_mutex` rules on
+  `py_context_t`.
+- Type conversion: `py_to_term` / `term_to_py` (`py_convert.c`).
+- Isolated mode has no C code of its own: `os_kill` is the only NIF it uses.
+
+## Rules that are easy to break
+
+- Only the context's thread touches the context's Python objects. NIFs
+  called from Erlang processes enqueue requests and return; they do not run
+  Python for a context that has a thread.
+- `Py_BEGIN_ALLOW_THREADS` around every blocking wait; never block on an
+  Erlang-side resource with the GIL held.
+- Never call a Future method while holding `async_futures_mutex`
+  (`py_callback.c`, comment above the struct explains the pattern).
+- `interrupt_mutex` is taken only by threads that do not hold the GIL.
+- `queue_mutex` protects the request queue of a context and is taken before
+  a request's own mutex (`ctx_queue_cancel_all`), never the other way.
+- A NIF that can block or run Python is registered with a dirty scheduler
+  flag in the table at the end of `py_nif.c`.
+
+## Adding a NIF
+
+1. Implement `static ERL_NIF_TERM nif_x(ErlNifEnv*, int, const ERL_NIF_TERM[])`
+   next to related code.
+2. Add `{"x", Arity, nif_x, Flags}` to `nif_funcs[]` at the end of `py_nif.c`.
+3. Add the stub and its `-spec` and doc to `src/py_nif.erl`.
+4. Cover it in a suite; `rebar3 dialyzer` and `rebar3 xref` must stay clean.
+
+## Legacy code, for orientation
+
+Not on the path of contexts created today: the `worker_*` NIFs and
+"Worker management" section, `async_worker_*` NIFs (return `deprecated`),
+the "Legacy mode" inline branches in `nif_context_call/eval/exec`,
+`py_worker_pool.c`, and the `cancel_reader/writer` aliases. When in doubt,
+follow `nif_context_call_async` and ignore the rest.

c_src/py_nif.c

@@ -29,11 +29,10 @@
  * - Resource types for Python objects to ensure proper cleanup
  * - Dirty NIF flags for GIL-holding operations
  *
- * This file is the main entry point. It includes the following modules:
- * - py_nif.h: Shared header with types and declarations
- * - py_convert.c: Type conversion (Python <-> Erlang)
- * - py_exec.c: Python execution and GIL management
- * - py_callback.c: Callback system and asyncio support
+ * This file is the main entry point and the single translation unit: it
+ * includes the other .c files (see "Include module implementations"). The
+ * file map is c_src/README.md; the request lifecycle per context mode is
+ * docs/architecture.md.
  */
 
 /* pthread_timedjoin_np (used to bound the owngil worker join on Linux)

c_src/py_nif.h

@@ -21,54 +21,19 @@
  *
  * @mainpage Python-Erlang NIF Integration
  *
- * @section intro_sec Introduction
- *
- * This NIF (Native Implemented Function) library provides seamless integration
- * between Erlang/OTP and Python. It embeds a Python interpreter within the
- * Erlang VM and provides bidirectional communication capabilities.
- *
- * @section arch_sec Architecture
- *
- * The implementation follows a modular design with four main components:
- *
- * - **py_nif.h** - Shared types, macros, and declarations
- * - **py_convert.c** - Bidirectional type conversion (Python ↔ Erlang)
- * - **py_exec.c** - Python execution engine and GIL management
- * - **py_callback.c** - Erlang callback support and asyncio integration
- *
- * @section modes_sec Execution Modes
- *
- * The library supports three execution modes based on Python version:
- *
- * | Mode | Python Version | Description |
- * |------|----------------|-------------|
- * | FREE_THREADED | 3.13+ (no-GIL) | Direct execution without GIL |
- * | SUBINTERP | 3.12+ | Per-interpreter GIL isolation |
- * | MULTI_EXECUTOR | Any | Multiple executor threads with GIL |
- *
- * @section gil_sec GIL Management
- *
- * The GIL (Global Interpreter Lock) is managed following PyO3/Granian patterns:
- *
- * - `Py_BEGIN_ALLOW_THREADS` / `Py_END_ALLOW_THREADS` around blocking ops
- * - Executor threads hold the GIL and process queued requests
- * - Dirty I/O schedulers are used for Python-calling NIFs
- *
- * @section callback_sec Callback Mechanism
- *
- * Python code can call back to Erlang using a suspension/resume pattern:
- *
- * 1. Python calls `erlang.call('func', args)`
- * 2. NIF raises `SuspensionRequired` exception
- * 3. Dirty scheduler is released, callback sent to Erlang
- * 4. Erlang processes callback, calls `resume_callback/2`
- * 5. Python execution resumes with cached result
- *
- * @section mem_sec Memory Management
- *
- * - Erlang resources wrap Python objects (prevent GC)
- * - Thread-local storage for callback context
- * - Proper cleanup in resource destructors
+ * This NIF embeds CPython in the Erlang VM. The map of the code, the life
+ * of a call in each context mode, the callback paths and the locking rules
+ * are documented in docs/architecture.md, c_src/README.md and
+ * docs/glossary.md; keep those current instead of this comment.
+ *
+ * In one paragraph: py_nif.c is the single translation unit and includes
+ * the other .c files; a context (py_context_t) owns a request queue and a
+ * pthread that runs Python for it (worker mode: main interpreter, shared
+ * GIL; owngil mode: a sub-interpreter with its own GIL); Erlang enqueues
+ * through nif_context_call_async and receives {py_result, Ref, Result};
+ * Python calls Erlang through erlang_call_impl (py_callback.c), by
+ * suspension in worker mode and a blocking pipe in owngil mode. Isolated
+ * mode runs Python in a child process and uses no C code beyond os_kill.
  */
 
 #ifndef PY_NIF_H