diff --git a/README.md b/README.md index 208b78a56..c5810d82b 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,8 @@ # Speakeasy -Speakeasy is a Windows malware emulation framework that executes binaries, drivers, and shellcode in a modeled Windows runtime instead of a full VM. It emulates APIs, process/thread behavior, filesystem, registry, and network activity so samples can keep moving through realistic execution paths. You can run it from the `speakeasy` CLI for fast triage or embed it as a Python library and consume structured JSON reports. +Speakeasy is a Windows malware emulation framework that executes binaries, drivers and shellcode in a modeled Windows runtime instead of a full VM. It emulates the Windows APIs, processes, threads, file system, registry and network that a sample uses, so the sample runs further than in a bare CPU emulator. The `speakeasy` CLI writes the report as JSON for triage, and the Python library returns the same report to your code. -Background context: [Mandiant's overview post](https://cloud.google.com/blog/topics/threat-intelligence/emulation-of-malicious-shellcode-with-speakeasy/). +For background, see Mandiant's [introduction to Speakeasy](https://cloud.google.com/blog/topics/threat-intelligence/emulation-of-malicious-shellcode-with-speakeasy/). ## Quick start @@ -12,55 +12,26 @@ Install from PyPI: python3 -m pip install speakeasy-emulator ``` -Run a sample and inspect high-level report fields (replace `sample.dll` with your target): +Emulate a sample and print some fields of the report. Replace `sample.dll` with your file: ```console speakeasy -t sample.dll --no-mp -o report.json 2>/dev/null jq '{sha256, arch, file_type, entry_points: (.entry_points | length)}' report.json ``` +The values depend on the sample: + ```json { - "sha256": "30ec092d122a90441a2560f6778ef8233c98079cd34b7633f7bbc2874c8d7a45", + "sha256": "…", "arch": "x86", "file_type": "dll", "entry_points": 3 } ``` -## Documentation map - -### Start here - -- [Installation](https://github.com/mandiant/speakeasy/blob/master/doc/install.md) -- [Python library usage](https://github.com/mandiant/speakeasy/blob/master/doc/library.md) -- [Help and troubleshooting](https://github.com/mandiant/speakeasy/blob/master/doc/help.md) -- [Documentation index](https://github.com/mandiant/speakeasy/blob/master/doc/index.md) - -### CLI usage - -- [CLI reference](https://github.com/mandiant/speakeasy/blob/master/doc/cli-reference.md) -- [CLI analysis recipes](https://github.com/mandiant/speakeasy/blob/master/doc/cli-analysis-recipes.md) -- [CLI environment overrides](https://github.com/mandiant/speakeasy/blob/master/doc/cli-environment-overrides.md) -- [CLI execution controls](https://github.com/mandiant/speakeasy/blob/master/doc/cli-execution-controls.md) - -### Reports, configuration, and runtime behavior - -- [Configuration walkthrough](https://github.com/mandiant/speakeasy/blob/master/doc/configuration.md) -- [Report walkthrough](https://github.com/mandiant/speakeasy/blob/master/doc/reporting.md) -- [Memory management](https://github.com/mandiant/speakeasy/blob/master/doc/memory.md) -- [Limitations](https://github.com/mandiant/speakeasy/blob/master/doc/limitations.md) - -### Debugging and extension - -- [GDB debugging reference](https://github.com/mandiant/speakeasy/blob/master/doc/gdb.md) -- [Mounting host files with `--volume`](https://github.com/mandiant/speakeasy/blob/master/doc/volumes.md) -- [Adding API handlers](https://github.com/mandiant/speakeasy/blob/master/doc/api-handlers.md) -- [Examples directory](https://github.com/mandiant/speakeasy/tree/master/examples) -- [Speakeasy 2 walkthrough outline](https://github.com/mandiant/speakeasy/blob/master/doc/speakeasy2-walkthrough.md) - -## Questions and help +## Documentation -Start with [doc/help.md](https://github.com/mandiant/speakeasy/blob/master/doc/help.md). +Start with [installation](https://github.com/mandiant/speakeasy/blob/master/doc/install.md), [Python library usage](https://github.com/mandiant/speakeasy/blob/master/doc/library.md) and the [CLI reference](https://github.com/mandiant/speakeasy/blob/master/doc/cli-reference.md). The [documentation index](https://github.com/mandiant/speakeasy/blob/master/doc/index.md) lists all pages. -If you still need help, open an issue at [github.com/mandiant/speakeasy/issues](https://github.com/mandiant/speakeasy/issues). +For help, see [doc/help.md](https://github.com/mandiant/speakeasy/blob/master/doc/help.md) or open an issue at [github.com/mandiant/speakeasy/issues](https://github.com/mandiant/speakeasy/issues). diff --git a/doc/api-handlers.md b/doc/api-handlers.md index 51e356c96..f974f9248 100644 --- a/doc/api-handlers.md +++ b/doc/api-handlers.md @@ -1,28 +1,28 @@ # Adding API handlers -Like most emulators, Speakeasy handles OS API calls in framework code. You can add a handler by defining a function with the expected API name in the corresponding emulated module. +Speakeasy emulates Windows APIs with Python handlers. Each emulated DLL has a handler class in `speakeasy/winenv/api/usermode` or `speakeasy/winenv/api/kernelmode`. To add a handler, add a method with the `@apihook` decorator and the API name to the class of the DLL. -Handler rules: +Follow these rules when you write a handler: -- specify `argc` so stack cleanup is correct -- if calling convention is omitted, stdcall is assumed -- `argv` contains the raw integer argument slots; handlers do not change it +- give `argc`, so that the emulator removes the correct number of arguments from the stack +- give `conv` if the API does not use stdcall. The decorator uses stdcall by default +- `argv` has the raw integer argument slots. Do not change it - annotate `argv` as `list[int]` and `emu` as the emulator that runs the handler: `Win32Emulator` in `usermode` modules and `WinKernelEmulator` in `kernelmode` modules. msvcrt and ntoskrnl handlers run in both modes, so they use `WindowsEmulator` -- unpack `argv` into locals with the parameter names of the prototype, as in the example below. When a name is a Python keyword or hides a builtin that the handler uses, use a close name (`in_` for `in`, `blen` for `len`) +- unpack `argv` into locals with the parameter names of the prototype, as in the example below. When a name is a Python keyword or hides a builtin that the handler uses, use a similar name (`in_` for `in`, `blen` for `len`) - to show a decoded value in the report, set the `display` of the argument in `ctx.args` (for example `ctx.args["lpFileName"].display = path`) -- import Windows constants from `speakeasy/winenv/defs`, and do not define them again in a handler module. An SDK enum or flag set is an `IntEnum` or `IntFlag` class in `speakeasy/winenv/defs/enums.py` with the name of its win32json enum, such as `PAGE_PROTECTION_FLAGS`. `scripts/gen_win32_enums.py` writes that module from win32json; to add a class, add its name to `scripts/win32_enums.json` and run the script again. To get the name of a value, call `decode(PAGE_PROTECTION_FLAGS, flProtect)` from `speakeasy.winenv.defs.lookup`. It uses the rules of the signature database, so one value gets one name in all places -- a struct class in `speakeasy/winenv/defs` has the size and the field offsets of the Windows struct on x86 and x64. Declare each field at class level in the C order, as `cb: Annotated[int, ct.c_uint32]`, or with its struct class for a nested struct (`Name: UNICODE_STRING`), so that mypy checks the handlers that write the field. Declare a pointer-sized field (`HANDLE`, `SIZE_T`, `ULONG_PTR`) as `Ptr`. A field whose type depends on `ptr_size` or on a constructor parameter gets its Python type at class level and its ctypes type in `__init__` through `add_fields` (see the `EmuStruct` docstring). `tests/test_struct_layouts.py` checks the layouts against win32json (`deps/win32json`) and against a table of kernel structs. +- import Windows constants from `speakeasy/winenv/defs`, and do not define them again in a handler module. An SDK enum or flag set is an `IntEnum` or `IntFlag` class in `speakeasy/winenv/defs/enums.py` with the name of its win32json enum, such as `PAGE_PROTECTION_FLAGS`. `scripts/gen_win32_enums.py` writes that module from win32json. To add a class, add its name to `scripts/win32_enums.json` and run the script again. To get the name of a value, call `decode(PAGE_PROTECTION_FLAGS, flProtect)` from `speakeasy.winenv.defs.lookup`. `decode` uses the rules of the signature database, so a value gets the same name in the handlers and in the report +- a struct class in `speakeasy/winenv/defs` must have the size and the field offsets of the Windows struct on x86 and x64. Declare each field at class level in the C order, as `cb: Annotated[int, ct.c_uint32]`, or with its struct class for a nested struct (`Name: UNICODE_STRING`), so that mypy checks the handlers that write the field. Declare a pointer-sized field (`HANDLE`, `SIZE_T`, `ULONG_PTR`) as `Ptr`. If the type of a field depends on `ptr_size` or on a constructor parameter, declare its Python type at class level and give its ctypes type in `__init__` with `add_fields` (see the `EmuStruct` docstring). `tests/test_struct_layouts.py` checks the layouts against win32json (`deps/win32json`) and against a table of kernel structs. - write integers to guest memory with `self.write_dword`, `self.write_qword` or `self.write_ptr`, and read them with `self.read_dword` or `self.read_ptr`. The write helpers call `ApiHandler.mem_write`, which also writes to the other views of a shared file mapping -- return the value expected by the sample path -- read and write `A` strings with `read_mem_string`, `write_mem_string`, `decode_string` and `encode_string`, which use the ANSI code page (1252) for a character width of 1. Do not select a Python codec for guest text -- order two strings with `get_string_order` from `speakeasy.winenv.api.compare`, and give the case fold of the function (`Fold`). It returns -1, 0 or 1 -- start the docstring with the C prototype from the SDK, DDK, phnt or CRT headers, as in the example below: one parameter per line, no calling convention, linkage or SAL annotations, and the closing `);` on its own line. The function name is the hook name: a hook without the `A` or `W` suffix uses the generic name and TCHAR types (`LPCTSTR`). Parameter names come from the Microsoft documentation. When the signature database has other names, use those, because the report and `ctx.args` use them. Notes about the emulation go after a blank line. A handler for exported data (`@impdata`) gives the C declaration of the variable +- return the value that the API returns on Windows for the same input +- read and write `A` strings with `read_mem_string`, `write_mem_string`, `decode_string` and `encode_string`, These functions use the ANSI code page (1252) for a character width of 1. Do not select a Python codec for guest text +- to compare two strings, call `get_string_order` from `speakeasy.winenv.api.compare` with the case fold (`Fold`) of the API. `get_string_order` returns -1, 0 or 1 +- start the docstring with the C prototype from the SDK, DDK, phnt or CRT headers, as in the example below. Put one parameter on each line, leave out the calling convention, the linkage and the SAL annotations, and put the closing `);` on its own line. Use the hook name as the function name. A hook without the `A` or `W` suffix uses the generic name and TCHAR types (`LPCTSTR`). Take the parameter names from the Microsoft documentation. When the signature database has other names, use those, because the report and `ctx.args` use them. Put notes about the emulation after a blank line. In a handler for exported data (`@impdata`), give the C declaration of the variable -For some APIs, returning a success code is enough to keep execution on a useful path. +For some APIs, a handler that only returns a success code is sufficient for the sample to continue. -A handler gets the object of a handle with a typed lookup, for example `emu.get_handle_table().get(hEvent, objman.Event)`. The lookup returns None when the handle is not open or refers to an object of a different type. Then the handler returns the failure value of its API and sets the last error as Windows does. For example, `SetEvent` sets `ERROR_INVALID_HANDLE` and returns 0, `ResumeThread` returns `0xFFFFFFFF`, a registry function returns `ERROR_INVALID_HANDLE`, and a kernel-mode handler returns `STATUS_INVALID_HANDLE`. +A handler gets the object of a handle with a typed lookup, for example `emu.get_handle_table().get(hEvent, objman.Event)`. The lookup returns None when the handle is not open or when the object has a different type. Then the handler returns the failure value of its API and sets the last error as Windows does. For example, `SetEvent` sets `ERROR_INVALID_HANDLE` and returns 0, `ResumeThread` returns `0xFFFFFFFF`, a registry function returns `ERROR_INVALID_HANDLE`, and a kernel-mode handler returns `STATUS_INVALID_HANDLE`. -When a module has no handler for a name, Speakeasy tries related names. A handler without the `A` or `W` suffix serves both variants, `Nt` and `Zw` names share a handler, ntdll falls back to the ntoskrnl handlers, and the registry functions that kernel32 exports (`Reg*`) fall back to the advapi32 handlers. All C runtime DLLs use the msvcrt handlers, and an `_o_` export of the private CRT uses the handler of the name without that prefix. kernelbase falls back to the kernel32 handlers, and ntoskrnl falls back to the msvcrt handlers for the C runtime functions that it exports. +When a module has no handler for a name, Speakeasy tries related names. `get_handler_candidates` in `speakeasy/windows/api_resolution.py` gives them in order. For example, a handler without the `A` or `W` suffix serves both variants, and `Nt` and `Zw` names share a handler. ntdll uses the ntoskrnl and msvcrt handlers. kernel32 uses the advapi32 handlers for the registry functions (`Reg*`) and the psapi handlers for the `K32*` functions. All C runtime DLLs use the msvcrt handlers, and an `_o_` export of the private CRT uses the handler of the name without that prefix. kernelbase uses the kernel32 handlers, and ntoskrnl uses the msvcrt handlers for the C runtime functions that it exports. An API set uses the handlers of the module that implements it. `DLL_HOSTS` in `speakeasy/windows/common.py` lists the modules for each name prefix. For example, `api-ms-win-core-com-*` uses ole32, `api-ms-win-core-crt-*` uses msvcrt, `api-ms-win-core-version-*` uses version, `api-ms-win-security-*` uses advapi32, and the other `api-ms-win-core-*` sets use kernel32. Some sets have functions in more than one module. For these, an import uses the first module that has a handler for the name. For example, imports from `api-ms-win-core-processthreads-*` use kernel32, but `OpenProcessToken` uses advapi32. `api-ms-win-core-libraryloader-*` uses kernel32 and user32 in the same way. `LoadLibrary` and `GetModuleHandle` with an API set name return the main module only. A function that win32metadata declares only under an API set counts as a declaration of the main module of the set. @@ -49,19 +49,28 @@ def HeapAlloc(self, emu: "Win32Emulator", argv: list[int], ctx: api.ApiContext = ## What happens without a handler -Imports that have no `@apihook` are not necessarily fatal. Speakeasy ships a signature database generated from Microsoft's [win32metadata](https://github.com/microsoft/win32metadata) (via the [win32json](https://github.com/marlersoft/win32json) export, vendored as the `deps/win32json` submodule). When an import misses every handler, `WindowsEmulator.dispatch_api_call` looks the function up there and, if it is declared, emulates the call from its prototype: it reads the right number of argument slots, decodes `PSTR`/`PWSTR`/`BOOL` arguments for the trace, renders enum and flag parameters symbolically (`dwCreationDisposition: CREATE_ALWAYS`, `dwShareMode: FILE_SHARE_WRITE|FILE_SHARE_READ`), expands pointers to known structs into a JSON-like rendering with typed fields (`lpSecurityAttributes: {nLength: 0xc, lpSecurityDescriptor: 0x0, bInheritHandle: TRUE}`, following nested pointers one level so `OBJECT_ATTRIBUTES.ObjectName` shows its string), zero-fills `Out` buffers whose size the prototype declares, returns a type-appropriate success value, and cleans up the stack according to the calling convention. Each argument in the event of such a call has the parameter name, its type, the raw value, and the rendering (see [reporting](reporting.md)). +A call to an import without an `@apihook` does not always stop the run. Speakeasy has a signature database that it generates from Microsoft's [win32metadata](https://github.com/microsoft/win32metadata), through the [win32json](https://github.com/marlersoft/win32json) export in the `deps/win32json` submodule. When no handler matches an import, `WindowsEmulator.dispatch_api_call` looks for the function in the database. If the database declares it, the emulator emulates the call from the prototype: -Calls served by a handler use the same signature to name and render their arguments when its slot count equals the handler's `argc`. The arguments are rendered before the handler runs. `ctx.args` gives the handler one entry per parameter, by name (`ctx.args["hKey"]`) or by parameter index. A handler that sets `display` replaces the rendering of that parameter. The type of the argument becomes `str` if the parameter is declared as a string, and `text` otherwise. A display for a parameter that the signature renders as an enum or flags does not replace that rendering, so all such values have one format. A value that is not a string, such as `None` from a failed lookup or a NULL pointer the handler did not decode, keeps the current display. +- It reads the correct number of argument slots. +- It decodes `PSTR`, `PWSTR` and `BOOL` arguments for the trace. +- It shows enum and flag parameters by name (`dwCreationDisposition: CREATE_ALWAYS`, `dwShareMode: FILE_SHARE_WRITE|FILE_SHARE_READ`). +- It shows a pointer to a known struct as its typed fields (`lpSecurityAttributes: {nLength: 0xc, lpSecurityDescriptor: 0x0, bInheritHandle: TRUE}`). It follows nested pointers one level, so `OBJECT_ATTRIBUTES.ObjectName` shows its string. +- It writes zeros to the `Out` buffers whose size the prototype declares. +- It returns a success value for the return type and removes the arguments from the stack as the calling convention requires. -Handlers without a matching signature (the C runtime, kernel-mode APIs, COM methods) record one unnamed argument per slot. These handlers use the slot index (`ctx.args[0]`), because a name finds no argument. A name or index that matches no argument gives a detached entry, and its changes are not recorded. Variadic handlers add entries with `ctx.args.append(text)` and remove all entries with `ctx.args.clear()`. `tests/test_handler_arg_access.py` checks that each name a handler uses is a parameter of every signature of that function, and that each index means the same parameter on x86 and x64. +Each argument in the event of such a call has the parameter name, the type, the raw value and the rendering (see [reporting](reporting.md)). -Hand-written handlers always take precedence, and are still required whenever a sample depends on the *behavior* of an API (output parameters, objects, files, network). The fallback only keeps emulation coherent and the trace informative. +When the slot count of the signature is equal to the `argc` of a handler, the emulator also uses the signature to name and show the arguments of the handler call. It renders the arguments before the handler runs. `ctx.args` gives the handler one entry per parameter, by name (`ctx.args["hKey"]`) or by parameter index. A handler that sets `display` replaces the rendering of that parameter. The type of the argument becomes `str` if the parameter is declared as a string, and `text` otherwise. A display for a parameter that the signature renders as an enum or flags does not replace that rendering, so all such values have one format. A value that is not a string, such as `None` from a failed lookup or a NULL pointer the handler did not decode, keeps the current display. -Undocumented native APIs (`Nt*`/`Zw*`, `Rtl*`, `Ldr*`, `Csr*`, `Dbg*` ...) come from a second source generated from the [phnt](https://github.com/winsiderss/phnt) headers (MIT, vendored as `deps/phnt`) by `scripts/gen_phnt_signatures.py`: about 2,500 prototypes with SAL-derived direction and buffer-size annotations and the information-class enums (`SystemInformationClass: SystemProcessInformation`). phnt struct layouts are not parsed; a `ps:` pointer resolves against every loaded source, so structs win32metadata also declares (`OBJECT_ATTRIBUTES`, `UNICODE_STRING`, `LARGE_INTEGER`, `CLIENT_ID`) still render and phnt-only structs show as pointers. The win32metadata source is consulted first. +If no signature matches a handler, for example a C runtime, kernel mode or COM method handler, the emulator records one unnamed argument per slot. These handlers use the slot index (`ctx.args[0]`), because a name finds no argument. A name or index that matches no argument gives a detached entry, and the report does not record its changes. Variadic handlers add entries with `ctx.args.append(text)` and remove all entries with `ctx.args.clear()`. `tests/test_handler_arg_access.py` checks that each name a handler uses is a parameter of every signature of that function, and that each index means the same parameter on x86 and x64. + +A handler always comes before the signature. A sample that depends on the behavior of an API (output parameters, objects, files, network) needs a handler. The signature only keeps the stack correct and the trace readable. + +The signatures of undocumented native APIs (`Nt*`, `Zw*`, `Rtl*`, `Ldr*`, `Csr*`, `Dbg*` ...) come from a second database. `scripts/gen_phnt_signatures.py` generates it from the [phnt](https://github.com/winsiderss/phnt) headers (MIT license, in the `deps/phnt` submodule). This database has the parameter directions and buffer sizes from the SAL annotations, and the information class enums (`SystemInformationClass: SystemProcessInformation`). The generator does not read the phnt struct layouts. The emulator looks for the struct of a pointer in all databases, so the structs that win32metadata also declares (`OBJECT_ATTRIBUTES`, `UNICODE_STRING`, `LARGE_INTEGER`, `CLIENT_ID`) show their fields, and the structs that only phnt declares show as pointers. The emulator looks in the win32metadata database first. ### Regenerating the signature databases -The databases live at `speakeasy/resources/win32/win32json_signatures.json.gz` (win32metadata) and `speakeasy/resources/win32/phnt_signatures.json.gz` (phnt). They are build artifacts, not committed: `python -m build` regenerates them (see `setup.py`), and for a source checkout run +The databases are `speakeasy/resources/win32/win32json_signatures.json.gz` (win32metadata) and `speakeasy/resources/win32/phnt_signatures.json.gz` (phnt). Git does not track them. `python -m build` generates them (see `setup.py`). In a source checkout, run ```console just gen-signatures @@ -71,15 +80,15 @@ python scripts/gen_win32_signatures.py --stats python scripts/gen_phnt_signatures.py --stats ``` -Without them Speakeasy still works; it logs a warning per missing database, and the imports that database would have covered are unknown APIs (see [Unknown APIs](limitations.md#unknown-apis)). +If a database is missing, Speakeasy logs a warning and continues. The imports that the database declares are then unknown APIs (see [Unknown APIs](limitations.md#unknown-apis)). -win32metadata records neither calling conventions nor variadic parameters, so those are supplied by hand in `scripts/win32_overrides.json` (`cdecl_dlls`, `cdecl`, `variadic`, plus `dll_aliases`/`name_prefixes` for forwarders such as `psapi!EnumProcesses` -> `kernel32!K32EnumProcesses`). For prototypes whose header declares no parameter names, win32metadata records `param0`, `param1`, and so on; `param_names` gives the names from the documentation. Edit that file and regenerate to correct a declaration; never patch the generated file. +win32metadata does not record calling conventions or variadic parameters. `scripts/win32_overrides.json` gives them (`cdecl_dlls`, `cdecl`, `variadic`), and also the forwarders (`dll_aliases`, `name_prefixes`), such as `psapi!EnumProcesses` -> `kernel32!K32EnumProcesses`. When a header declares no parameter names, win32metadata records `param0`, `param1` and so on, and `param_names` gives the names from the documentation. To correct a declaration, edit that file and generate the database again. Do not edit the generated file. -`tests/test_apihook_signatures.py` cross-checks every `@apihook`'s `argc` against the database, so a hook that disagrees with the documented prototype fails CI. +`tests/test_apihook_signatures.py` compares the `argc` of each `@apihook` with the database. If a hook does not agree with the documented prototype, CI fails. -The rendering logic lives in `speakeasy.winenv.api.sigfmt.ArgFormatter`; it caps nesting depth, array length and total output size so a trace line stays readable. +`speakeasy.winenv.api.sigfmt.ArgFormatter` renders the arguments. It limits the nesting depth, the array length and the total length, so that each trace line is short enough to read. -Additional signature sources (for example vendor-specific DLLs) can be plugged in by implementing `speakeasy.winenv.api.sigdb.SignatureSource` and adding it to `emu.get_signature_db()`. +To add signatures from another source, for example for a third-party DLL, write a subclass of `speakeasy.winenv.api.sigdb.SignatureSource` and add it with `emu.get_signature_db().add_source()`. ## Related docs diff --git a/doc/architecture.md b/doc/architecture.md index f05943ee8..b5fe47821 100644 --- a/doc/architecture.md +++ b/doc/architecture.md @@ -1,85 +1,85 @@ # Architecture of the Windows emulator -This page tells you where the main parts of the Windows emulator are, and how a call to a Windows API gets to a Python handler. +This page tells you where the main parts of the Windows emulator are, and how a call to a Windows API reaches a Python handler. ## Classes -`BinaryEmulator` (`speakeasy/binemu.py`) wraps Unicorn and capstone. It has the memory manager and the hook lists. It does not know about Windows. +`BinaryEmulator` (`speakeasy/binemu.py`) wraps Unicorn and capstone. It has the memory manager and the hook lists, and no Windows-specific code. -`WindowsEmulator` (`speakeasy/windows/winemu.py`) adds processes, threads, modules, runs, API calls and SEH. It also gives access to the object, file, registry and network managers. The file, registry and network managers record the file, registry and network events of the report, so each handler that uses a manager method gets the event. +`WindowsEmulator` (`speakeasy/windows/winemu.py`) adds processes, threads, modules, runs, API calls and SEH. It also gives access to the object, file, registry and network managers. The file, registry and network managers record the report events, so a handler does not add these events itself. `Win32Emulator` (`speakeasy/windows/win32.py`) runs user mode executables, DLLs and shellcode. `WinKernelEmulator` (`speakeasy/windows/kernel.py`) runs drivers. The `Speakeasy` class (`speakeasy/speakeasy.py`) is the public interface. It creates the emulator for the input. -API handlers are in `speakeasy/winenv/api/`, one class for each DLL. Each handler has an `@apihook` decorator with the export name, the number of arguments and the calling convention. See [Adding API handlers](api-handlers.md). +Each emulated DLL has one handler class in `speakeasy/winenv/api/`. Each handler has an `@apihook` decorator with the export name, the number of arguments and the calling convention. See [Adding API handlers](api-handlers.md). ## Runs A `Run` is one call into guest code: the entry point, a DLL export, a thread, a TLS callback or a DllMain. The emulator keeps a queue of runs. `start()` adds the DllMain and TLS callbacks of the imported DLLs to the front of the queue, and `_execute_runs` runs the queue in order. `_prepare_next_run` sets up the stack and registers for the next run. -`_execute_current_run` runs one run. Unicorn runs guest code until a hook stops it. A hook does not call API handlers while Unicorn is running. It records a native exit with `_record_exit` and stops Unicorn. If Unicorn stops and no hook recorded an exit, `_get_exit_at_pc` gets the exit from the address where Unicorn stopped. Then `_handle_exit` does the work for the exit, and the loop starts Unicorn again. A native exit tells why Unicorn stopped. It is one of these types: +`_execute_current_run` runs one run. Unicorn runs guest code until a hook stops it. A hook does not run API handlers or guest exception handlers. It records a native exit with `_record_exit` and stops Unicorn. If Unicorn stops and no hook recorded an exit, `_get_exit_at_pc` gets the exit from the address where Unicorn stopped. Then `_handle_exit` handles the exit, for example it calls the API handler, and the loop starts Unicorn again. A native exit tells why Unicorn stopped. It is one of these types: -- `SuspendedApiCall`: the guest called an API. The handler of the API runs next. +- `SuspendedApiCall`: the guest called an API. The loop calls the handler of the API next. - `RunReturn`: the guest returned from the run. The run is complete. - `CallbackReturn`: a guest callback returned. The API handler that called it continues. - `FaultReturn`: a fault ended the run. - `ExecRecovery`: the guest ran code in a guest PE page without execute permission. The loop adds the permission. -- `RaisedCpuException`: a CPU exception or a memory fault for guest SEH. +- `RaisedCpuException`: a CPU exception or a memory fault that the loop sends to the guest SEH handlers. -The emulator keeps one native exit. If a second hook records an exit before Unicorn stops, the first exit stays. +If two hooks record an exit before Unicorn stops, the emulator keeps the first. `RunLimits` has the time and instruction limits of a run. The loop checks all limits before each start of Unicorn and before each API handler. Before `RunReturn`, `CallbackReturn`, `FaultReturn` and `ExecRecovery`, it checks only the total time. ## API calls -Each export of an emulated DLL has 32 bytes of code in the DLL image. The code jumps to a token in a reserved range that is never mapped. When the guest calls the API, Unicorn fails to fetch the token, and `_intercept_api_trap` finds the API for the token. The export table, the IAT and `GetProcAddress` give the address of the export code. The guest can read, patch and hook this code. See [Unified Windows API addresses](unified-api-addresses.md). +Each export of an emulated DLL has 32 bytes of code in the DLL image. The code jumps to a token in a reserved range that the emulator never maps. When the guest calls the API, Unicorn fails to fetch the token, and `_intercept_api_trap` finds the API for the token. The export table, the IAT and `GetProcAddress` give the address of the export code. The guest can read, patch and hook this code. See [Unified Windows API addresses](unified-api-addresses.md). `dispatch_api_call(dll, name)` calls `resolve_api_implementation` to find the code to run. `speakeasy/windows/api_resolution.py` has one class for each result, in this order: 1. `HandlerImplementation`: the handler for the name, or for a related name. User hooks for the API run first, and they can call the handler. -2. `UserHookImplementation`: a user hook without a handler. The last hook that was added runs. +2. `UserHookImplementation`: a user hook without a handler. The emulator runs the hook that the caller added last. 3. `SignatureImplementation`: a function from the signature database. It logs the arguments and returns a success value. 4. `StubImplementation`: with `functions_always_exist`, a stub that returns 1. 5. `UnsupportedImplementation`: the run ends with an `unsupported_api` error. -Each result has its own method: `_handle_api_with_handler`, `_handle_api_hook`, `_handle_api_from_signature`, `_handle_api_stub` and `_handle_unsupported_api`. - -`get_handler_candidates(dll, name)` gives the related names, in order. For example, the name without the `A` or `W` suffix, the `Zw` name for an `Nt` name, or the modules that implement an API set (`get_dll_hosts`). `get_api_handler` uses the first related name that has a handler. `get_proc` gets the module for an import from `get_import_host`. For an API set with functions in more than one module, this is the first module that has a handler for the name. +`get_handler_candidates(dll, name)` gives the related names, in order: for example, the name without the `A` or `W` suffix, the `Zw` name for an `Nt` name, or the modules that implement an API set (`get_dll_hosts`). `get_api_handler` uses the first related name that has a handler. `get_proc` asks `get_import_host` for the module of an import. When more than one module implements an API set, `get_import_host` returns the first module that has a handler for the name. A handler can call guest code, for example a window procedure, with `setup_callback` or by yielding `GuestCall`. `speakeasy/windows/guest_callbacks.py` keeps track of these calls. ## Machine setup -`init_machine(arch)` creates the machine before an image loads. The `Speakeasy` class calls it with the architecture of the input. It creates the Unicorn engine, the disassembler and the API layer. Then `init_machine_structures` creates the object manager, the GDT and the segments, `KUSER_SHARED_DATA`, the system modules and the user modules of the config. In user mode, it also adds the processes of the config to the process list, except the `is_main_exe` entry. The process enumeration APIs read this list. The main process comes after them: `load_container` adds the container process, and `run_module` adds the process of an EXE input. +`init_machine(arch)` creates the machine before an image loads. The `Speakeasy` class calls it with the architecture of the input. It creates the Unicorn engine, the disassembler and the API layer. Then `init_machine_structures` creates the object manager, the GDT and the segments, `KUSER_SHARED_DATA`, the system modules and the user modules of the config. In user mode, it also adds the processes of the config to the process list, except the `is_main_exe` entry. The process enumeration APIs read this list. Later, `load_container` adds the container process, or `run_module` adds the process of an EXE input, at the end of the list. + +If the caller adds a hook before the load, the `Speakeasy` class keeps it in a list, because the engine does not exist yet. The `Speakeasy` class adds these hooks to the engine right after `init_machine`, before the load maps the input. So a memory map hook added before the load sees the maps of the load, but not the maps that `init_machine` makes. -A hook that the caller adds before the load waits in a list of the `Speakeasy` class, because the engine does not exist yet. The `Speakeasy` class adds these hooks right after `init_machine`, before the load maps the input. So a memory map hook added before the load sees the maps of the load, but not the maps that `init_machine` makes. +In user mode, when the input is a DLL or shellcode, the `Speakeasy` class then calls `load_container`, before the input loads. `load_container` loads the container image and makes its process the current process. The container image is a placeholder for the EXE that loaded the DLL or that runs the shellcode, and it is the main image of the process. The entry in `processes` of the config that has `is_main_exe` set defines it (see [Configuration walkthrough](configuration.md)). The container image is a synthetic module, unless the module directory has a file with its name, such as `main.exe`. The `ImageBaseAddress` of the PEB and `GetModuleHandle(NULL)` give its base. The container process uses the session of the entry, but not its `pid`. -In user mode, when the input is a DLL or shellcode, the `Speakeasy` class then calls `load_container`, before the input loads. This step loads the container image and makes its process the current process. The container image is a placeholder module for the EXE that loaded the DLL or that runs the shellcode. It is the main image of the process in which the input runs. It comes from the entry in `processes` of the config that has `is_main_exe` set. By default, its name is `main` and its path is `C:\Windows\system32\svchost.exe`. It is a synthetic module, unless the module directory has a file with its name, such as `main.exe`. The `ImageBaseAddress` of the PEB and `GetModuleHandle(NULL)` give its base. The container process gets the session of the entry, by default 1, but not its `pid`. When the input is an EXE, there is no container image, and the input is the main image of the process. When the config has no `is_main_exe` entry, `load_container` does nothing. Then a DLL runs in a process whose main image is the DLL, and shellcode runs in an empty process. +When the input is an EXE, there is no container image, and the input is the main image of the process. When the config has no `is_main_exe` entry, `load_container` does nothing. Then a DLL runs in a process whose main image is the DLL, and shellcode runs in an empty process. -The machine uses fixed addresses, so that the input image can use its preferred base. `init_machine` creates the machine in an `allocations_from(MACHINE_BASE)` block. In this block, each allocation that does not give an address starts its search for free memory at `MACHINE_BASE` (0x7D000000). `load_container` loads the container image in the same way, at the first free base above `MACHINE_BASE`. On x86, the container image must fit below the TEB, or `load_container` raises `OutOfMemoryError`. The user modules of the config load at their configured bases. In user mode, the TEB of the first thread is at 0x7FFDE000 and the PEB of the main process is at 0x7FFDF000, as in 32-bit Windows. On x64, the TEB also stays below 4 GB, because the GS base comes from a 32-bit GDT descriptor. `KUSER_SHARED_DATA` is at 0x7FFE0000. The host has the same architecture as the input, so the process is never a WOW64 process. `IsWow64Process`, `IsWow64Process2`, `IsOS(OS_WOW6432)` and `ImageNumberLow`/`ImageNumberHigh` in `KUSER_SHARED_DATA` all match this. In kernel mode, the TEB and the PEB do not have fixed addresses. +`init_machine` puts the machine at fixed high addresses, so that the input image can load at its preferred base. It creates the machine in an `allocations_from(MACHINE_BASE)` block. In this block, when an allocation gives no address, the memory manager searches for free memory from `MACHINE_BASE` (0x7D000000). `load_container` loads the container image in the same way, at the first free base above `MACHINE_BASE`. On x86, the container image must fit below the TEB, or `load_container` raises `OutOfMemoryError`. The emulator loads the user modules of the config at their configured bases. In user mode, the TEB of the first thread is at 0x7FFDE000 and the PEB of the main process is at 0x7FFDF000, as in 32-bit Windows. On x64, the emulator also puts the TEB below 4 GB, because the GS base comes from a 32-bit GDT descriptor. `KUSER_SHARED_DATA` is at 0x7FFE0000. The emulated system has the same architecture as the input, so the process is never a WOW64 process, and the WOW64 query APIs report this. In kernel mode, the TEB and the PEB do not have fixed addresses. ## Loader -`load_image` loads a PE file, such as an .exe or a .dll. If a dependency fails to load, it removes everything that it added. `ModuleLoader` (`speakeasy/windows/module_loader.py`, the `module_loader` attribute of the emulator) keeps the data that it must restore: the import bindings, the DLLs whose DllMain and TLS callbacks must run, and the DLLs that each new process gets in its PEB. +`load_image` loads a PE file, such as an .exe or a .dll. If a dependency fails to load, it removes everything that it added. `ModuleLoader` (`speakeasy/windows/module_loader.py`, the `module_loader` attribute of the emulator) keeps the loader data that `load_image` restores after a failure: the import bindings, the DLLs whose DllMain and TLS callbacks must run, and the DLLs that each new process gets in its PEB. -For each image, `_load_image` maps it, binds its imports and adds it to the PEB. The caller marks the input image with `is_input`. The input image is the first module in `modules`, and only the input image allocates the stack. If a machine module or the container image uses the preferred base of the input image, the input image loads at the next free base. +For each image, `_load_image` maps it, binds its imports and adds it to the PEB. The caller marks the input image with `is_input`. The input image is the first module in `modules`, and the emulator allocates the stack only for the input image. If a machine module or the container image uses the preferred base of the input image, `_load_image` loads the input image at the next free base. -A loader (`PeLoader`, `ShellcodeLoader` or `ApiModuleLoader`) returns a `LoadedImage` with RVAs: the regions, the import slots, the exports, the entry points, the TLS callbacks and the TLS and resource directories. Its `image_base` is the preferred base. `PeLoader` also returns the base relocations of the PE, or None if the PE has no relocation data. Synthetic API modules and shellcode have no absolute addresses, so they can load at any base. `relocate_image(image, base)` returns a copy of the image at a base. It applies the relocations, writes the base in the ImageBase field of the header and adds the base to each RVA. It applies the HIGHLOW and DIR64 relocation types, skips ABSOLUTE and raises `ValueError` for the other types. It also raises `ValueError` if the base is not the preferred base and the image has no relocation data. +A loader (`PeLoader`, `ShellcodeLoader` or `ApiModuleLoader`) returns a `LoadedImage` with RVAs: the regions, the import slots, the exports, the entry points, the TLS callbacks and the TLS and resource directories. Its `image_base` is the preferred base. `PeLoader` also returns the base relocations of the PE, or None if the PE has no relocation data. Synthetic API modules and shellcode have no absolute addresses, so they can load at any base. `relocate_image(image, base)` returns a copy of the image rebased to `base`. `_load_image` selects the base as the Windows loader does. It uses the base that the caller gives, for example the `base_addr` of a module in the config, else the preferred base. If that range is not free, it uses the next free range after it. An image with preferred base 0, such as shellcode, loads at any free address. `ensure_pe_import_hooks` binds the imports of a PE that the guest maps itself, for example by process hollowing. `speakeasy/windows/mapped_imports.py` reads the import table from guest memory. -DllMain and TLS callbacks run in two ways. For the DLLs that the input imports, `_queue_startup_initializers` adds a run for each callback before the first run in the queue. For `LoadLibrary`, `load_library` calls them as guest callbacks. +The emulator runs DllMain and TLS callbacks in two ways. For the DLLs that the input imports, `_queue_startup_initializers` adds a run for each callback before the first run in the queue. For `LoadLibrary`, `load_library` calls them as guest callbacks. -Usually there are few runs before the entry point. Only a DLL with a real PE image has a DllMain or TLS callbacks: a DLL from the module directory (`module_directory_x86` or `module_directory_x64`, not set by default), or a DLL that the guest loads with `LoadLibrary` from the emulated file system. The other DLLs, such as kernel32, are synthetic API modules without initializers. Imports never load from the emulated file system, so a DLL that the guest drops and a child process imports is also a synthetic module. All processes share one set of modules, and a child process gets in its PEB all the DLLs that are mapped. If the parent loaded a dropped DLL with `LoadLibrary`, its initializers run again before the entry point of the child. +Usually there are few runs before the entry point. Only a DLL with a real PE image has a DllMain or TLS callbacks: a DLL from the module directory (`module_directory_x86` or `module_directory_x64`, not set by default), or a DLL that the guest loads with `LoadLibrary` from the emulated file system. The other DLLs, such as kernel32, are synthetic API modules without initializers. The loader never loads imports from the emulated file system. All processes share one set of modules. Both ways use the same data. `Process.peb_modules` has the modules in the loader lists of a process. `RuntimeModule.load_states` has the state of the module in each process, with names from the Windows `LDR_DDAG_STATE` values: `INITIALIZING` while its DllMain and TLS callbacks run, then `READY_TO_RUN` or `INIT_ERROR`. `ModuleLoader.collect_initializers` returns an `InitRoutine` for each DllMain and TLS callback to run. `InitRoutine.args` has the arguments of the call for `DLL_PROCESS_ATTACH`. After each callback, `ModuleLoader.finish_initializer` decides if it succeeded. A DllMain fails if it faults, or if the low byte of its result is zero. A TLS callback fails only if it faults. After the last callback of a module succeeds, its state is `READY_TO_RUN`. -The two ways do different things when a callback fails. At start, `_finish_run_initializers` sets the module state to `INIT_ERROR` and removes the other runs of the module from the queue. The module stays mapped, and the process continues. In `LoadLibrary`, `_fail_library_initializer` makes the call return NULL with error 1114 (`ERROR_DLL_INIT_FAILED`) and unloads the DLL. If a callback faults in `LoadLibrary`, the run ends. Before the next run, `_finish_run_initializers` undoes the loads that the run did not complete. +The two ways handle a failed callback differently. At start, `_finish_run_initializers` sets the module state to `INIT_ERROR`, removes the other runs of the module from the queue and keeps the module mapped. The process continues. In `LoadLibrary`, `_fail_library_initializer` makes the call return NULL with error 1114 (`ERROR_DLL_INIT_FAILED`) and unloads the DLL. If a callback faults in `LoadLibrary`, the run ends. Before the next run, `_finish_run_initializers` undoes the loads that the run did not complete. -If the emulator cancels the runs in the queue, for example when the total time limit is reached, `ModuleLoader.cancel_initializers` clears the `INITIALIZING` state of the callbacks that did not run. +If the emulator cancels the runs in the queue, for example when emulation reaches the total time limit, `ModuleLoader.cancel_initializers` clears the `INITIALIZING` state of the callbacks that did not run. ## Hooks and faults @@ -91,10 +91,10 @@ If the emulator cancels the runs in the queue, for example when the total time l - `_hook_code_coverage` when `analysis.coverage` is set. - `_hook_code_deferred_work`, which runs at the next instruction after an API call, a fault or a request to end the run, and then disables itself. -If a hook gets an exception, it calls `_end_run_on_hook_error`. This saves the error on the run and stops emulation. +If a hook raises a Python exception, the hook catches it and calls `_end_run_on_hook_error`, which saves the error on the run and stops emulation. -A hook does not send a fault to the guest while Unicorn runs. It calls `raise_cpu_exception`, which records the exception as a native exit and stops Unicorn. Unmapped reads, writes and fetches use this path, and so do invalid instructions, interrupts and most protection faults. An access to the reserved range of the API tokens that is not an API call also uses it. The emulator does not map a page at the fault address. +A hook that detects a fault calls `raise_cpu_exception`, which records the exception as a native exit and stops Unicorn. Unmapped reads, writes and fetches use this path, and so do invalid instructions, interrupts and most protection faults. An access to the reserved range of the API tokens that is not an API call also uses it. -After Unicorn stops, `_handle_exit` calls `_dispatch_cpu_exception`, which sends the exception to the guest handlers with `dispatch_seh`. If no handler takes the exception, `guest_callbacks.fail` ends the run with the fault error. This function keeps the guest callback frames of the run. Thus, if a DllMain faults in `LoadLibrary`, `_finish_run_initializers` can undo the load before the next run. See [Exception dispatch](exceptions.md). +After Unicorn stops, `_handle_exit` calls `_dispatch_cpu_exception`, which sends the exception to the guest handlers with `dispatch_seh`. If no handler takes the exception, `guest_callbacks.fail` ends the run with the fault error. `guest_callbacks.fail` keeps the guest callback frames of the run. So if a DllMain faults in `LoadLibrary`, `_finish_run_initializers` can undo the load before the next run. See [Exception dispatch](exceptions.md). `speakeasy/windows/thread_context.py` has the tables that `get_thread_context` and `load_thread_context` use to copy registers to and from a `CONTEXT`. Unicorn does not have debug registers, so the emulator keeps them on the thread. diff --git a/doc/cli-analysis-recipes.md b/doc/cli-analysis-recipes.md index 1999b8a68..791886628 100644 --- a/doc/cli-analysis-recipes.md +++ b/doc/cli-analysis-recipes.md @@ -1,121 +1,92 @@ # CLI analysis recipes -This page focuses on practical flag combinations for common analysis outputs. +Each recipe gives the flags for one kind of analysis output and a command that checks the output. ## In-report memory snapshots (`--snapshot-memory-regions`) -Command: - ```bash speakeasy -t sample.exe --snapshot-memory-regions -o report.json ``` -Expected artifact: -- `entry_points[*].memory.layout[*].data_ref` populated with SHA-256 refs -- top-level `data` populated with `base64(zlib(raw_bytes))` entries - -Quick verification: +The memory regions in `entry_points[*].memory.layout` get a `data_ref`, the SHA-256 of their bytes. Stack and heap regions do not get one. The top-level `data` object maps each SHA-256 to the bytes, compressed with zlib and encoded with base64. To list the regions that have data: ```bash jq '.entry_points[].memory.layout[] | select(.data_ref != null) | .tag' report.json ``` -Tradeoff: -- report size increases, but repeated payloads deduplicate across runs +The report gets larger. Regions with the same bytes share one entry in `data`, also across runs. ## Coverage collection (`--analysis-coverage`) -Command: - ```bash speakeasy -t sample.exe --analysis-coverage -o report.json ``` -Expected artifact: -- `entry_points[*].coverage` contains executed instruction addresses - -Quick verification: +`entry_points[*].coverage` has the addresses of the instructions that each run executed. To count them: ```bash jq '.entry_points[] | {start_address, coverage_count: (.coverage // [] | length)}' report.json ``` -Tradeoff: -- extra tracing overhead increases runtime +The emulator runs a hook for each instruction, so emulation is slower. ## Memory tracing (`--analysis-memory-tracing`) -Command: - ```bash speakeasy -t sample.exe --analysis-memory-tracing -o report.json ``` -Expected artifact: -- per-region access counters in `memory.layout[*].accesses` -- symbol access summaries in `symbol_accesses` - -Quick verification: +The regions in `entry_points[*].memory.layout` that the run accessed get `accesses`, with the counts of reads, writes and executes. `entry_points[*].symbol_accesses` has the counts for each symbol. To count the symbols that each run accessed: ```bash jq '.entry_points[] | {start_address, symbol_accesses: (.symbol_accesses // [] | length)}' report.json ``` -Tradeoff: -- substantial runtime impact on memory-heavy samples +The emulator runs a hook for each instruction, memory read and memory write. A sample that accesses much memory runs much slower. ## String extraction controls (`--analysis-strings` / `--no-analysis-strings`) -Enable: +String extraction is on by default, so `--analysis-strings` has an effect only when a `--config` file turns it off: ```bash speakeasy -t sample.exe --analysis-strings -o report.json ``` -Disable: +`--no-analysis-strings` turns it off: ```bash speakeasy -t sample.exe --no-analysis-strings -o report.json ``` -Quick verification: +The strings are in the top-level `strings` object. Without string extraction, the report has no `strings`: ```bash jq '.strings' report.json ``` -Tradeoff: -- disabling strings reduces report size and post-processing time +Without string extraction, the report is smaller and Speakeasy finishes sooner. ## Dropped files archive (`--dropped-files-path`) -Command: - ```bash speakeasy -t sample.exe --dropped-files-path dropped.zip ``` -Expected artifact: -- `dropped.zip` with files written during emulation and a manifest - -Quick verification: +`dropped.zip` has the files that the sample wrote and `speakeasy_manifest.json`, which gives the emulated path, size and SHA-256 of each file. If the sample wrote no files, the CLI does not make the archive. To list the archive: ```bash unzip -l dropped.zip ``` -Tradeoff: -- captures useful payload artifacts but adds archive creation overhead - ## Combined triage profile -Command: +This command collects all the outputs of this page in one run. `--max-run-time 30` limits each run to 30 seconds: ```bash speakeasy -t sample.exe \ @@ -127,8 +98,6 @@ speakeasy -t sample.exe \ -o report.json ``` -Use this profile when you want broad telemetry and artifact capture in one run. - ## Related docs - [Project README](../README.md) diff --git a/doc/cli-environment-overrides.md b/doc/cli-environment-overrides.md index a5a896736..ae8bb62f7 100644 --- a/doc/cli-environment-overrides.md +++ b/doc/cli-environment-overrides.md @@ -1,18 +1,10 @@ # CLI environment overrides -This page covers flags that shape the emulated host environment and behavior determinism. +These flags set the identity of the emulated host, the process environment, DNS answers, module loading and API hammering mitigation. At the start of each emulation, Speakeasy logs the active config to stderr, so you can check the values in the log. -## Host, user, and OS identity +## Host, user and OS identity -Primary flags: -- `--hostname` -- `--domain` -- `--user-name` -- `--user-is-admin` / `--no-user-is-admin` -- `--user-sid` -- `--os-ver-major`, `--os-ver-minor`, `--os-ver-build` - -Example: enterprise-like identity profile +The flags are `--hostname`, `--domain`, `--user-name`, `--user-is-admin` / `--no-user-is-admin`, `--user-sid`, `--os-ver-major`, `--os-ver-minor` and `--os-ver-build`. This example gives a workstation in a corporate domain, with a user that is not an administrator: ```bash speakeasy -t sample.exe \ @@ -27,7 +19,7 @@ speakeasy -t sample.exe \ -v -o report.json 2> run.log ``` -Quick verification: +To check the values: ```bash rg "(hostname|domain|user\.name|user\.sid|user\.is_admin|os_ver\.)" run.log @@ -35,18 +27,11 @@ rg "(hostname|domain|user\.name|user\.sid|user\.is_admin|os_ver\.)" run.log ## Process context and environment variables -Primary flags: -- `--current-dir` -- `--command-line` -- `--env KEY=VALUE` (repeatable) - -Names are not case-sensitive, and the config stores them in lower case, so `--env TEMP=...` replaces the default `temp`. - -Example: +The flags are `--current-dir`, `--command-line` and `--env KEY=VALUE`. `--env` is repeatable, and each value adds or replaces one variable. The config stores variable names in lower case, because Windows variable names are not case-sensitive. So `--env TEMP=...` replaces the default `temp`. ```bash speakeasy -t sample.exe \ - --current-dir 'C:\\ProgramData\\Microsoft' \ + --current-dir 'C:\ProgramData\Microsoft' \ --command-line 'svchost.exe -k netsvcs -p' \ --env TEMP=C:\\Windows\\Temp \ --env APPDATA=C:\\Users\\jdoe\\AppData\\Roaming \ @@ -54,7 +39,7 @@ speakeasy -t sample.exe \ -v -o report.json 2> run.log ``` -Quick verification: +To check the values: ```bash rg "(current_dir|command_line|env =)" run.log @@ -62,10 +47,7 @@ rg "(current_dir|command_line|env =)" run.log ## DNS override mappings -Primary flag: -- `--network-dns-names HOST=IP` (repeatable) - -Example: force known C2 host resolutions +`--network-dns-names HOST=IP` sets the address that a DNS lookup gives for a host name. The flag is repeatable. This example gives addresses for two C2 hosts: ```bash speakeasy -t sample.exe \ @@ -74,7 +56,7 @@ speakeasy -t sample.exe \ -o report.json ``` -Quick verification: +Each lookup gives a `net_dns` event in the report: ```bash jq '.entry_points[].events[]? | select(.event == "net_dns") | {query, response}' report.json @@ -82,15 +64,13 @@ jq '.entry_points[].events[]? | select(.event == "net_dns") | {query, response}' ## Module load policy and module directories -Some samples (especially shellcode) parse PE export tables directly to resolve API pointers. When expected modules or exports are missing, these controls let you choose strict or permissive behavior. +By default, `LoadLibrary` fails for a DLL that Speakeasy does not know. With `--modules-always-exist`, the emulator makes a synthetic module for the DLL. + +By default, `GetProcAddress` fails for a function that Speakeasy does not know, and a call to such a function ends the run with an `unsupported_api` error. With `--modules-functions-always-exist`, `GetProcAddress` gives an address for the function, and a call to it returns 1. -Primary flags: -- `--modules-always-exist` / `--no-modules-always-exist` -- `--modules-functions-always-exist` / `--no-modules-functions-always-exist` -- `--module-directory-x86` -- `--module-directory-x64` +`--module-directory-x86` and `--module-directory-x64` give the directories of real x86 and x64 DLLs. The emulator loads a DLL from the directory of the input architecture in place of its synthetic API module. -Example: relaxed unresolved-import policy with native DLL directories +This example allows unknown modules and functions, and loads real DLLs from two directories: ```bash speakeasy -t sample.exe \ @@ -101,16 +81,11 @@ speakeasy -t sample.exe \ -o report.json ``` -Use this when triaging samples that otherwise stop early on missing modules/APIs. +Use these flags when a sample stops early because a module or a function is missing. ## API hammering controls -Primary flags: -- `--api-hammering-enabled` / `--no-api-hammering-enabled` -- `--api-hammering-threshold` -- `--api-hammering-allow-list VALUE` (repeatable) - -Example: +Some samples call an API many times in a loop to make emulation slow. With `--api-hammering-enabled`, the emulator counts the calls to each API from each call site. When the count reaches `--api-hammering-threshold`, the emulator patches an x86 call site so that the call gives 0 and does not run the API. The emulator does not patch x64 code. `--api-hammering-allow-list` gives an API in the form `dll.name` that the emulator does not patch. The flag is repeatable, and its values replace the default allow list. ```bash speakeasy -t sample.exe \ @@ -121,7 +96,7 @@ speakeasy -t sample.exe \ -o report.json ``` -This is useful when balancing anti-loop containment with legitimate hot API usage. +Increase the threshold if the emulator patches calls that the sample needs. ## Related docs diff --git a/doc/cli-execution-controls.md b/doc/cli-execution-controls.md index 27c2b4245..d9dcb3f08 100644 --- a/doc/cli-execution-controls.md +++ b/doc/cli-execution-controls.md @@ -1,18 +1,19 @@ # CLI execution controls -This page covers runtime limits, execution mode, and debugging controls. +This page tells you how to limit a run, select the execution mode and debug a run. ## Stopping conditions -Primary flags: -- `--max-run-time`: active execution cap in seconds per run (0 disables it) -- `--max-total-time`: active execution cap in seconds across all runs (0, the default, disables it) -- `--max-api-calls`: cap API calls per run -- `--max-instructions`: cap executed instructions per run +These flags stop a run: -Use these together to bound long-running or looping samples. +- `--max-run-time`: the active execution time of one run, in seconds. 0 disables the limit. +- `--max-total-time`: the active execution time of all runs, in seconds. 0 disables the limit. +- `--max-api-calls`: the number of API calls in one run. +- `--max-instructions`: the number of instructions in one run. -The CLI emulates in a child process. It stops that process after the larger of `--max-run-time` and `--max-total-time` and saves the report that the process gives within 5 seconds of the stop. When both are 0, the CLI does not stop the process. +Use them together for samples that run for a long time or loop. + +The CLI emulates in a child process. It stops the child process after the larger of `--max-run-time` and `--max-total-time`. Then it saves the report that the child process sends in the next 5 seconds. When both limits are 0, the CLI does not stop the child process. `--max-run-time` and `--max-total-time` measure host time, so the point where they stop a run depends on the speed of the host. Two runs with the same limits can stop at different instructions and give different reports. For a repeatable report, set `--max-run-time 0` and stop the run with `--max-instructions` or `--max-api-calls`: @@ -24,16 +25,17 @@ speakeasy -t sample.exe --max-run-time 0 --max-instructions 3000000 --max-api-ca ## Execution mode controls -Primary flags: -- `--raw`: emulate target as raw bytes -- `--arch`: shellcode architecture (`x86`, `x64`, `amd64`), required with `--raw` -- `--raw-offset`: raw-mode execution start offset (decimal, or hex with `0x`) -- `--emulate-children`: emulate spawned child processes -- `--no-mp`: run in current process instead of worker process +These flags select what the emulator runs and where: + +- `--raw`: emulate the input as raw bytes, for example shellcode. +- `--arch`: the architecture of the shellcode (`x86`, `x64` or `amd64`). `--raw` needs it. +- `--raw-offset`: the offset in the raw input where execution starts (decimal, or hex with `0x`). +- `--emulate-children`: emulate the child processes that the sample starts. +- `--no-mp`: emulate in the CLI process instead of a child process. -With `--emulate-children`, a child process runs after the process that started it stops. All processes share one address space and the loaded DLLs. Before a child runs, the image, PEB and heap blocks of the process that ran before it are unmapped. The child gets its own PEB, with its image first in the loader list, and a new TEB. A child whose image base is used by other memory is skipped with a warning. The child gets a copy of the current directory and the environment variables of its parent, and it starts with no vectored exception handlers, no unhandled exception filter and no function tables. When the guest calls `CreateProcess` with a `lpCurrentDirectory`, the child starts in that directory instead. The API handlers keep one state for all processes, for example the process heap handle and the C runtime data. +With `--emulate-children`, a child process runs after the process that started it stops. All processes share one address space and the loaded DLLs. Before a child runs, the emulator unmaps the image, the PEB and the heap blocks of the process that ran before it. The child gets its own PEB, with its image first in the loader list, and a new TEB. If the child image cannot load, for example because other memory uses its base and it has no relocation data, the emulator skips the child and logs a warning. The child gets a copy of the current directory and the environment variables of its parent. If the guest gives `lpCurrentDirectory` to `CreateProcess`, the child starts in that directory. The child does not get the vectored exception handlers, the unhandled exception filter or the function tables of its parent. The API handlers keep one state for all processes, for example the process heap handle and the C runtime data. -Example: raw shellcode run with bounded execution +This command runs raw shellcode with limits: ```bash speakeasy -t shellcode.bin \ @@ -44,17 +46,14 @@ speakeasy -t shellcode.bin \ ## Debugging controls -Primary flags: -- `--verbose`: DEBUG logging -- `--gdb`: start GDB stub and pause before first instruction -- `--gdb-host`: GDB stub bind host (default `127.0.0.1`) -- `--gdb-port`: GDB stub port +These flags control logging and the GDB stub: -Notes: -- `--gdb` implies `--no-mp` automatically. -- Use `gdb` or `gdb-multiarch` to connect to `localhost:`. +- `--verbose`: log at the DEBUG level. +- `--gdb`: start the GDB stub and pause before the first instruction. +- `--gdb-host`: the address where the GDB stub listens. +- `--gdb-port`: the port of the GDB stub. -Example: debugger-first startup profile +`--gdb` sets `--no-mp`. Connect `gdb` or `gdb-multiarch` to `localhost:`. This command starts the stub and waits for the debugger: ```bash speakeasy -t sample.dll --gdb --gdb-port 1234 --verbose @@ -62,7 +61,7 @@ speakeasy -t sample.dll --gdb --gdb-port 1234 --verbose ## Profiles -### Anti-loop containment profile +### Fast triage of looping samples ```bash speakeasy -t sample.exe \ @@ -73,9 +72,9 @@ speakeasy -t sample.exe \ -o report.json ``` -Use this as a fast triage baseline for samples that often spin in loops. +Use this profile first for samples that often loop. -### Deep-debug profile +### Debugging one run with full telemetry ```bash speakeasy -t sample.exe \ @@ -87,23 +86,11 @@ speakeasy -t sample.exe \ -o report.json ``` -Use this when you want step control and richer telemetry during one run. +Use this profile when you want to step through one run and get more data in the report. ## Performance and telemetry tuning -Speakeasy is Python-based, so runtime is sensitive to telemetry scope and stop conditions. - -Practical tuning: - -- disable heavy collectors unless needed: - - `--no-analysis-memory-tracing` - - `--no-analysis-coverage` - - `--no-snapshot-memory-regions` -- set hard stop limits for unstable or looping samples: - - `--max-run-time` - - `--max-api-calls` - - `--max-instructions` -- run fast triage and deep telemetry as separate profiles +Speakeasy is written in Python, so the data that it collects and the stop limits have a large effect on the run time. Memory tracing, coverage and memory region snapshots make a run slower and the report larger. They are off by default. If your config file enables them, `--no-analysis-memory-tracing`, `--no-analysis-coverage` and `--no-snapshot-memory-regions` disable them for one run. For samples that loop or crash, set `--max-run-time`, `--max-api-calls` and `--max-instructions`. Run fast triage and the full telemetry as two different runs. ## Related docs diff --git a/doc/cli-reference.md b/doc/cli-reference.md index 47b090a13..f87a82449 100644 --- a/doc/cli-reference.md +++ b/doc/cli-reference.md @@ -2,17 +2,13 @@ ## Invocation and required input -Basic emulation invocation: +To emulate a sample, give its path with `--target`: ```bash speakeasy --target [flags] ``` -Rules: -- `--target` is required for emulation runs. -- `--dump-default-config` and `--version` are the only modes that do not require `--target`. - -Default config dump: +An emulation run needs `--target`. Only `--dump-default-config` and `--version` run without it. `--dump-default-config` prints the default config: ```bash speakeasy --dump-default-config > default-config.json @@ -20,31 +16,28 @@ speakeasy --dump-default-config > default-config.json ## Runtime-only flags -These flags are not generated from the config schema. +These flags do not come from the config schema. - `-t, --target`: input file to emulate -- `-o, --output`: output report JSON path. Without it, the CLI writes the report to stdout -- `--argv`: argv values for the emulated process (quoted string, parsed with shell rules). For an EXE they replace `command_line`, and the process command line is the quoted program path followed by these values -- `-c, --config`: JSON config overlay file -- `--dump-default-config`: print built-in default config and exit -- `--raw`: treat input as raw bytes/shellcode -- `--raw-offset`: raw execution start offset +- `-o, --output`: path of the report JSON. Without it, the CLI writes the report to stdout +- `--argv`: arguments of the emulated process, as one quoted string that the CLI splits with shell rules. For an EXE, they replace `command_line`, and the process command line is the quoted program path followed by these arguments +- `-c, --config`: JSON config file that changes the defaults +- `--dump-default-config`: print the default config as JSON and exit +- `--raw`: emulate the input as raw bytes, such as shellcode +- `--raw-offset`: offset in the raw input where emulation starts - `--entry-point`: RVA where a PE starts, in place of its entry point -- `--arch`: shellcode architecture (`x86`, `x64` or `amd64`, in any case). `--raw` needs it; other modes ignore it -- `--dropped-files-path`: dropped-files archive output path -- `-k, --emulate-children`: emulate child processes spawned by the sample -- `--no-mp`: run in current process instead of worker process +- `--arch`: shellcode architecture (`x86`, `x64` or `amd64`, in any case). `--raw` needs it, and other modes ignore it +- `--dropped-files-path`: path of the dropped-files archive +- `-k, --emulate-children`: after the input finishes, emulate the processes that it created with the CreateProcess APIs +- `--no-mp`: run the emulation in the CLI process instead of a child process - `-v, --verbose`: DEBUG logging -- `--gdb`: enable GDB stub and pause before first instruction -- `--gdb-host`: GDB stub bind host (default `127.0.0.1`) -- `--gdb-port`: GDB stub port (default `1234`) -- `-V, --volume`: host_path:guest_path mapping (repeatable) +- `--gdb`: start a GDB server and stop before the first instruction. See [GDB debugging](gdb.md) +- `--gdb-host`: address where the GDB server listens (default `127.0.0.1`) +- `--gdb-port`: port of the GDB server (default `1234`) +- `-V, --volume`: `host_path:guest_path` mapping (repeatable). See [Volumes](volumes.md) - `--version`: print the speakeasy version and exit -Notes: -- `--gdb` implies `--no-mp`; Speakeasy enables this automatically. -- `--raw-offset` and `--entry-point` take a decimal number, or a hex number with `0x`: `--raw-offset 32` and `--raw-offset 0x20` are the same. -- option abbreviations are disabled; pass full flag names. +With `--gdb`, the CLI also sets `--no-mp`. `--raw-offset`, `--entry-point` and the integer config flags take a decimal number, or a hex number with `0x`, so `--raw-offset 32` and `--raw-offset 0x20` are the same. The CLI does not accept abbreviated flag names. ## Output and exit status @@ -54,72 +47,34 @@ The CLI writes the report JSON to the `--output` file, or to stdout without `--o speakeasy --target sample.exe | jq '.entry_points[].error' ``` -The CLI exits 0 when it writes a report, also when an entry point in the report has an `error`, for example an invalid instruction or a timeout. A batch script reads `error` in each entry point to see that emulation stopped early. The CLI exits 1 when it cannot make a report, for example because the loader rejects the input, and writes the error message to stderr. When an error stops the emulation outside an entry point, the CLI writes the report that it has and also exits 1. An argument error, for example `--raw` without `--arch` or a `--volume` host path that does not exist, gives exit status 2. - -## Schema-derived config flags - -Most scalar/toggle/list/mapping fields in `SpeakeasyConfig` are exposed as CLI flags. - -Naming rules: -- config path `a.b_c` maps to `--a-b-c` -- booleans use dual form: `--flag` and `--no-flag` -- dict mappings use repeatable `KEY=VALUE` -- list values use repeatable `VALUE` - -### Current schema-derived flags (complete list) - -Boolean toggles: -- `--analysis-memory-tracing` / `--no-analysis-memory-tracing` -- `--analysis-strings` / `--no-analysis-strings` -- `--analysis-coverage` / `--no-analysis-coverage` -- `--keep-memory-on-free` / `--no-keep-memory-on-free` -- `--snapshot-memory-regions` / `--no-snapshot-memory-regions` -- `--exceptions-dispatch-handlers` / `--no-exceptions-dispatch-handlers` -- `--user-is-admin` / `--no-user-is-admin` -- `--api-hammering-enabled` / `--no-api-hammering-enabled` -- `--modules-always-exist` / `--no-modules-always-exist` -- `--modules-functions-always-exist` / `--no-modules-functions-always-exist` - -Scalars: -- `--max-run-time` -- `--max-total-time` -- `--max-api-calls` -- `--max-instructions` -- `--stack-size` -- `--random-seed` -- `--start-time` -- `--os-ver-major` -- `--os-ver-minor` -- `--os-ver-build` -- `--current-dir` -- `--command-line` -- `--domain` -- `--hostname` -- `--user-name` -- `--user-sid` -- `--api-hammering-threshold` -- `--module-directory-x86` -- `--module-directory-x64` - -Mappings/lists: -- `--env KEY=VALUE` (repeatable) -- `--network-dns-names KEY=VALUE` (repeatable) -- `--api-hammering-allow-list VALUE` (repeatable) +The CLI exits 0 when it writes a report. This is also true when an entry point in the report has an `error`, for example an invalid instruction or a timeout. To find the runs that stopped early, read `error` in each entry point. The CLI exits 1 when it cannot make a report, for example because the loader rejects the input. It then writes the error message to stderr. If an error stops the emulation outside an entry point, the CLI still gives the partial report and exits 1. An argument error gives exit status 2, for example `--raw` without `--arch`, or a `--volume` host path that does not exist. + +## Config flags + +The CLI makes a flag for each config field with a boolean, number, string, string list or string-to-string mapping value. The [config-file-only fields](#config-file-only-fields) have no flag. `speakeasy --help` shows the description and the default of each flag. + +The CLI makes the flag name from the config path. `analysis.memory_tracing` gives `--analysis-memory-tracing`. If a path part starts with the same word as the part before it, the CLI uses that word one time: `modules.modules_always_exist` gives `--modules-always-exist`. A boolean field has two flags, such as `--analysis-coverage` and `--no-analysis-coverage`. + +A mapping flag takes `KEY=VALUE` and is repeatable. Each value adds the key to the mapping or replaces its value, and the other keys of the mapping do not change. A list flag takes one `VALUE` and is repeatable. The values replace the whole list. + +Boolean flags: `--analysis-memory-tracing`, `--analysis-strings`, `--analysis-coverage`, `--keep-memory-on-free`, `--snapshot-memory-regions`, `--exceptions-dispatch-handlers`, `--user-is-admin`, `--api-hammering-enabled`, `--modules-always-exist` and `--modules-functions-always-exist`, each also with a `--no-` form. + +Value flags: `--max-run-time`, `--max-total-time`, `--max-api-calls`, `--max-instructions`, `--stack-size`, `--random-seed`, `--start-time`, `--os-ver-major`, `--os-ver-minor`, `--os-ver-build`, `--current-dir`, `--command-line`, `--domain`, `--hostname`, `--user-name`, `--user-sid`, `--api-hammering-threshold`, `--module-directory-x86` and `--module-directory-x64`. + +Mapping flags: `--env KEY=VALUE` and `--network-dns-names KEY=VALUE`. List flag: `--api-hammering-allow-list VALUE`. ## Config precedence -Active runtime config is built in this order: -1. built-in defaults (`SpeakeasyConfig`) -2. optional `--config` JSON overlay -3. explicit CLI overrides +The CLI builds the config of a run in this order: -The CLI reads the `--config` file in the same way as `Speakeasy(config=...)` reads a dict: -- a key that the file gives replaces the default value. This is also true for a mapping such as `env` or `network.dns.names`, and for a list -- a key that the file leaves out gets its default, also in a nested object. For example, `{"user": {"name": "bob"}}` keeps `user.is_admin` at `true` +1. the defaults of `SpeakeasyConfig` +2. the `--config` file +3. the `--volume` mappings +4. the config flags -Speakeasy does not currently ship named built-in config profiles beyond the default baseline. +The CLI reads the `--config` file in the same way as `Speakeasy(config=...)` reads a dict. A key in the file replaces the default value. This is also true for a mapping such as `env` or `network.dns.names`, and for a list. A key that the file leaves out keeps its default, also in a nested object. For example, `{"user": {"name": "bob"}}` keeps `user.is_admin` at `true`. -Conflict example: +In this example, the flags change two values of the config file: ```bash speakeasy --target sample.exe \ @@ -129,37 +84,35 @@ speakeasy --target sample.exe \ --output report.json ``` -If `profile.json` sets `max_run_time=120` and `analysis.strings=true`, effective runtime values are `max_run_time=20` and `analysis.strings=false`. +If `profile.json` sets `max_run_time` to 120 and `analysis.strings` to `true`, the run uses `max_run_time` 20 and `analysis.strings` `false`. -## Unsupported complex fields on CLI +## Config-file-only fields -The following fields are config-file-only: -- schema/meta: `config_version`, `emu_engine`, `os_ver.name` -- object lists and nested structures: - - `symlinks`, `drives` - - `filesystem.files` - - `registry.keys` - - `network.adapters`, `network.dns.txt`, `network.http.responses`, `network.winsock.responses` - - `processes` - - `modules.user_modules`, `modules.system_modules` +These fields hold nested objects or lists of objects, or describe the config itself. They have no flag, so set them in a `--config` file: -Rationale: these are nested or large structures and are not ergonomic as CLI arguments. +- `config_version`, `emu_engine`, `os_ver.name` +- `symlinks`, `drives` +- `filesystem.files` +- `registry.keys` +- `network.adapters`, `network.dns.txt`, `network.http.responses`, `network.winsock.responses` +- `processes` +- `modules.user_modules`, `modules.system_modules` -## Concrete examples +## Examples -Simple PE run: +Run a PE: ```bash speakeasy --target sample.exe --output report.json ``` -Raw shellcode run: +Run raw shellcode: ```bash speakeasy --target shellcode.bin --raw --arch x86 --raw-offset 0x20 --output report.json ``` -Memory snapshots and dropped-files archive: +Save memory snapshots in the report and write the dropped-files archive: ```bash speakeasy --target sample.exe --snapshot-memory-regions --dropped-files-path dropped.zip --output report.json diff --git a/doc/configuration.md b/doc/configuration.md index 0b035df95..9451f97c5 100644 --- a/doc/configuration.md +++ b/doc/configuration.md @@ -1,10 +1,6 @@ # Configuration walkthrough -Source of truth: -- `speakeasy/config.py` -- behavior references in `speakeasy/windows/*.py`, `speakeasy/binemu.py`, and `speakeasy/winenv/api/*` - -The example below is JSONC (JSON with comments). Remove comment lines for machine parsing. +`speakeasy/config.py` defines the config fields and their defaults. `speakeasy --dump-default-config` prints the default config. The example below shows the fields with comments. It is JSONC (JSON with comments), so remove the comments before you give it to Speakeasy. Addresses and sizes (`base_addr`, `stack_size`, `byte_fill.size`) are ints. JSON has no hex numbers, so these fields also take a string that starts with `0x`, such as `"0x400000"`. `--dump-default-config` writes each `base_addr` as a `0x` string. @@ -14,33 +10,36 @@ Addresses and sizes (`base_addr`, `stack_size`, `byte_fill.size`) are ints. JSON // other values, such as the 0.2 of Speakeasy 1.x configs. Leave it out to use "2.0". "config_version": "2.0", - // Emulation backend. Only "unicorn" is currently supported. + // Emulation engine. The only value is "unicorn". "emu_engine": "unicorn", - // Active execution cap in seconds for each run. 0 disables it. - // If exceeded, the run ends with a timeout error. - // Raise for slow/staged samples. Lower for quick triage. + // Limit of the execution time of each run, in seconds. 0 disables it. + // At the limit, the emulator ends the run with a timeout error. + // Increase it for slow samples. Decrease it for quick triage. "max_run_time": 60, - // Active execution cap in seconds across all runs of one run_module, - // run_shellcode, or call invocation. 0 disables it. + // Limit of the execution time of all runs of one run_module, run_shellcode, + // call or resume call, in seconds. 0 disables it. + // At the limit, the emulator ends the current run with a max_total_time error + // and cancels the other runs. "max_total_time": 0, - // Per-run API call cap. If exceeded, the run ends with a max_api_count error. - // Raise to allow very API-heavy behavior; lower to cut anti-analysis loops earlier. + // Limit of the API calls of each run. A run that makes more API calls ends + // with a max_api_count error. Decrease it to stop anti-analysis loops sooner. "max_api_calls": 10000, - // Instruction cap forwarded to the emulator. - // -1 means effectively unbounded from Speakeasy policy. + // Limit of the instructions of each run. -1 disables it. + // The emulator stops a run at the limit and records a max_instructions error. "max_instructions": -1, - // Override the stack size in bytes for all emulated threads. - // 0 (default) uses the PE header's SizeOfStackReserve or the built-in default (0x12000). - // Set to e.g. 0x100000 (1 MiB) for samples that allocate a large stack. + // Size in bytes of the stack that the emulator allocates for the input. + // 0 uses the stack size of the input: the SizeOfStackReserve of a PE, or + // the stack_commit argument of run_shellcode. + // Set it, for example to 0x100000 (1 MiB), for a sample that uses a large stack. "stack_size": 0, - // Seed of the random number generator behind RtlGenRandom, CryptGenRandom, - // BCryptGenRandom, ProcessPrng, CoCreateGuid and UuidCreate. + // Seed of the random number generator of the random APIs, such as + // RtlGenRandom and CoCreateGuid. // Runs with the same seed give the same random bytes and GUIDs to the sample. "random_seed": 0, @@ -53,61 +52,64 @@ Addresses and sizes (`base_addr`, `stack_size`, `byte_fill.size`) are ints. JSON "start_time": "2026-10-31T15:15:00Z", "analysis": { - // Enables per-region memory access counters and symbol access telemetry. - // Turn on for unpacking/PEB-walk/import-resolver analysis. + // Count the reads, writes and executes of each symbol, section and memory map + // in the report. Use it to examine unpackers, PEB walks and import resolvers. "memory_tracing": false, - // Enables static + in-memory string extraction in the report. - // Disable for speed when string data is not needed. + // If true, the report has the strings of the input file and of emulated memory. + // Set it to false to make the emulation faster when you do not need the strings. "strings": true, - // Enables instruction-address coverage collection per entry point. - // Useful for diffing runs and rough execution mapping. + // Record the address of each instruction that a run executes. + // Use it to compare runs or to see which code ran. "coverage": false }, - // If true, frees do not immediately remove memory maps. - // Useful when you want to inspect post-free data artifacts. + // If true, the memory manager keeps a freed region mapped, so you can read + // the data that the sample freed. "keep_memory_on_free": false, - // If true, report memory regions include deduplicated snapshot refs in the top-level data store. - // This increases report size but keeps memory forensics self-contained. + // If true, the report has the contents of the memory regions at the end of each run. + // The report keeps each content one time in its top-level data store. + // The report is larger, but you do not need the emulator to examine the memory. "snapshot_memory_regions": false, "exceptions": { - // If true, emulate exception-handler dispatch (SEH/VEH-like paths, see exceptions.md). - // If false, faulting paths stop sooner and are easier to localize. + // If true, the emulator sends CPU exceptions to the VEH and SEH handlers of + // the guest (see exceptions.md). If false, a fault ends the run, so you can + // find the fault more easily. "dispatch_handlers": true }, "os_ver": { - // OS family marker. Keep as "windows". + // The only value is "windows". "name": "windows", - // The version that the PEB, KUSER_SHARED_DATA and the version functions - // (GetVersionEx, RtlGetVersion, RtlGetNtVersionNumbers and others) give, and - // the values of HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion (6.1, 6.2, - // 6.3 and 10.0). - // registry.keys values for that key take precedence. - // Align these with the environment your sample expects. + // The version that the PEB, KUSER_SHARED_DATA and the version functions, + // such as GetVersionEx and RtlGetVersion, give. For 6.1, 6.2, 6.3 and 10.0, + // the emulator also sets the values of + // HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion. + // A value for that key in registry.keys replaces the value of the emulator. + // Set the version that your sample expects. "major": 6, "minor": 1, - // Build number used by version checks. + // Build number that the version functions give. "build": 7601 }, - // Current directory exposed to process APIs and relative path resolution. + // Current directory of the process. The emulator resolves relative paths from it. "current_dir": "C:\\Windows\\system32", - // Process command line exposed to query APIs. + // Command line of the process, for example for GetCommandLine. "command_line": "svchost.exe -k netsvcs", - // Environment variables visible to the emulated process. - // Change when malware reads env values for path decisions or anti-sandbox checks. + // Environment variables of the process. Change them when a sample selects + // paths or detects sandboxes from the environment. // Names are not case-sensitive, and the config stores them in lower case. // programfiles and commonprogramfiles also give the ProgramFilesDir and CommonFilesDir - // values of HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion; registry.keys values take precedence. + // values of HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion. A value for that + // key in registry.keys replaces the value of the environment. "env": { "comspec": "C:\\Windows\\system32\\cmd.exe", "systemroot": "C:\\Windows", @@ -116,39 +118,40 @@ Addresses and sizes (`base_addr`, `stack_size`, `byte_fill.size`) are ints. JSON "userprofile": "C:\\Users\\analyst" }, - // Domain/workgroup identity used by account/network APIs. - // Set to match expected enterprise or non-domain environments. + // Domain or workgroup of the user, for example for LookupAccountName. + // Set it to match the domain that the sample expects. "domain": "LAB", - // Hostname returned by networking/system identity APIs. + // Host name, for example for GetComputerName. "hostname": "WIN7-LAB", "user": { - // Username used by account/profile APIs. + // User name, for example for GetUserName. "name": "analyst", - // Privilege context for is-admin checks. + // If true, the token of the user is elevated and has a high integrity level. "is_admin": true, - // Optional explicit SID. Set when SID-sensitive logic is present. + // SID of the user. "sid": "S-1-5-21-1111111111-2222222222-3333333333-1001" }, "api_hammering": { - // Enables anti-API-hammering mitigation. + // If true, the emulator patches API hammering loops in x86 code. "enabled": false, - // Repeat count before mitigation patches trigger. + // When the guest calls one API from one call site this number of times, + // the emulator patches the call to return 0. "threshold": 2000, - // APIs exempt from mitigation. Use for hot APIs that are legitimately noisy. + // APIs that the emulator does not patch. An empty list uses a built-in list. "allow_list": [ "kernel32.WriteFile", "kernel32.ReadFile" ] }, - // Object-manager symbolic links, used by path/device resolution. + // Symbolic links of the object manager. The emulator uses them to resolve device paths. "symlinks": [ { "name": "\\??\\C:", @@ -160,7 +163,7 @@ Addresses and sizes (`base_addr`, `stack_size`, `byte_fill.size`) are ints. JSON } ], - // Virtual drive metadata used by drive APIs. + // Drives that the drive APIs, such as GetLogicalDrives, give. "drives": [ { "root_path": "C:\\", @@ -177,55 +180,55 @@ Addresses and sizes (`base_addr`, `stack_size`, `byte_fill.size`) are ints. JSON "filesystem": { "files": [ { - // Mode discriminator. full_path = exact/wildcard path matching. + // full_path matches the full path of the file. "mode": "full_path", - // Emulated file path pattern to match (supports wildcard matching). + // Path in the emulated file system. It can have wildcards, such as *. "emu_path": "c:\\programdata\\seed\\config.bin", - // Host file to serve for reads. - // Use $ROOT$ for repo-relative assets in packaged configs. + // Host file that gives the content of the file. + // $ROOT$ is the directory of the installed speakeasy package. "path": "$ROOT$/resources/files/default.bin" }, { "mode": "full_path", "emu_path": "c:\\programdata\\seed\\padding.bin", - // Synthetic content source when no host file is desired. + // Content that the emulator makes, in place of a host file. "byte_fill": { - // Byte to repeat for generated data. + // Byte that fills the content. "byte": "0x41", - // Generated size in bytes. + // Size of the content in bytes. "size": 512 } }, { - // by_ext = apply to all paths with this extension. + // by_ext matches all paths with this extension, if no full_path entry matches. "mode": "by_ext", "ext": "txt", "path": "$ROOT$/resources/files/default.bin" }, { - // default = catch-all fallback if no full_path/by_ext matches. + // default matches all paths, if no full_path or by_ext entry matches. "mode": "default", "path": "$ROOT$/resources/files/default.bin" } ] }, - // Optional registry seed data. - // Keep only keys/values that your sample reads to reduce maintenance. + // Registry keys and values that exist when emulation starts. "registry": { "keys": [ { "path": "HKEY_LOCAL_MACHINE\\Software\\Microsoft\\Windows\\CurrentVersion\\Run", "values": [ { - // Value name. Use "default" for unnamed default values. + // Value name. "default" is the unnamed default value. "name": "Updater", - // Registry type token consumed by registry emulation. + // Value type, such as REG_SZ, REG_DWORD or REG_MULTI_SZ. "type": "REG_SZ", - // Value payload as string form. + // Data as a string. REG_DWORD and REG_QWORD data is a number, such as "0x3". + // REG_MULTI_SZ data separates the strings with "\u0000". "data": "C:\\ProgramData\\updater.exe" } ] @@ -235,22 +238,21 @@ Addresses and sizes (`base_addr`, `stack_size`, `byte_fill.size`) are ints. JSON "network": { "dns": { - // A-record style domain -> IP mappings. - // Include "default" for fallback name lookups. + // IP address of each domain name. "default" gives the address of all other names. "names": { "default": "10.10.10.10", "example.org": "93.184.216.34" }, - // TXT response handlers loaded from files. + // Host files that give the TXT record of a domain name. "txt": [ { - // Domain-specific TXT mapping. + // TXT record of one domain name. "name": "bootstrap.example.org", "path": "$ROOT$/resources/web/default.bin" }, { - // Fallback TXT response if no exact name matches. + // TXT record of all other names. "name": "default", "path": "$ROOT$/resources/web/default.bin" } @@ -260,7 +262,8 @@ Addresses and sizes (`base_addr`, `stack_size`, `byte_fill.size`) are ints. JSON "http": { "responses": [ { - // Verb this response set applies to. + // HTTP verb of the requests that these files answer: GET, POST or HEAD. + // by_ext matches the extension of the URL path. "verb": "GET", "files": [ { @@ -287,7 +290,7 @@ Addresses and sizes (`base_addr`, `stack_size`, `byte_fill.size`) are ints. JSON }, "winsock": { - // Raw socket recv seed payloads. A default handler is typically sufficient. + // Data that recv gives on a socket. The emulator uses only the default entry. "responses": [ { "mode": "default", @@ -296,7 +299,7 @@ Addresses and sizes (`base_addr`, `stack_size`, `byte_fill.size`) are ints. JSON ] }, - // Adapter inventory returned by networking APIs such as GetAdaptersInfo. + // Network adapters that APIs such as GetAdaptersInfo give. "adapters": [ { "name": "{00000000-0000-0000-0000-000000000000}", @@ -310,11 +313,13 @@ Addresses and sizes (`base_addr`, `stack_size`, `byte_fill.size`) are ints. JSON ] }, - // Process inventory visible to process enumeration APIs. - // Include one is_main_exe=true process for shellcode/DLL container use. - // Process enumeration gives the other entries in this order, then the main process. - // The container image loads at the first free base above 0x7D000000, not at its base_addr. - // The container process gets the session of that entry, but not its pid. + // Processes that the process enumeration APIs give, in this order, then the + // main process. + // The is_main_exe entry defines the container image, which is the main image + // of the process when the input is a DLL or shellcode (see architecture.md). + // The emulator ignores the base_addr of this entry and loads the container image + // at the first free base above 0x7D000000. + // The container process gets the session of the entry, but not its pid. "processes": [ { "name": "System", @@ -337,22 +342,23 @@ Addresses and sizes (`base_addr`, `stack_size`, `byte_fill.size`) are ints. JSON ], "modules": { - // If true, loads of unknown modules create empty synthetic API modules - // instead of failing. + // If true, when the guest loads an unknown module, the emulator creates an + // empty synthetic API module. If false, the load fails. "modules_always_exist": false, - // If true, unresolved API imports that are in neither the handlers nor the - // bundled signature database resolve to dynamic entries that act as stubs - // (4 stdcall arguments, return value 1). Imports with a known signature are - // always emulated from it, see doc/api-handlers.md. + // If true, an API that has no handler and no entry in the signature database + // resolves to a stub. The stub takes 4 stdcall arguments and returns 1. + // An API with an entry in the signature database always uses that entry + // (see api-handlers.md). "functions_always_exist": false, - // Directories of native DLLs, by architecture. A module with a matching - // file name loads from the directory instead of as a synthetic API module. + // Directories of native DLLs, by architecture. If a directory has a file + // with the name of a module, the emulator loads that file in place of the + // synthetic API module. "module_directory_x86": null, "module_directory_x64": null, - // User-mode module inventory. + // User mode modules that the emulator loads at their base_addr. "user_modules": [ { "name": "ntdll", @@ -366,14 +372,15 @@ Addresses and sizes (`base_addr`, `stack_size`, `byte_fill.size`) are ints. JSON } ], - // Kernel/system module inventory. + // Kernel mode modules. "system_modules": [ { "name": "ntoskrnl", "base_addr": "0x803D0000", "path": "C:\\Windows\\system32\\ntoskrnl.exe", - // Optional driver/device object declaration for kernel object namespace realism. + // Optional driver object and device objects of the module, in the + // object manager namespace. "driver": { "name": "\\Driver\\ntoskrnl", "devices": [ @@ -388,40 +395,22 @@ Addresses and sizes (`base_addr`, `stack_size`, `byte_fill.size`) are ints. JSON } ``` -## CLI mapping and precedence - -CLI config overrides are generated from `SpeakeasyConfig` field definitions. - -Mapping rules: -- config path `a.b_c` maps to `--a-b-c` -- a path part is left out when the next part starts with the same word, so `modules.modules_always_exist` maps to `--modules-always-exist` and `modules.module_directory_x86` maps to `--module-directory-x86` -- booleans map to dual flags (`--flag` / `--no-flag`) -- int flags take a decimal value or a hex value that starts with `0x` -- `dict[str, str]` maps to repeatable `--flag KEY=VALUE` -- `list[str]` maps to repeatable `--flag VALUE` - -Examples: -- `analysis.coverage` -> `--analysis-coverage` -- `analysis.strings` -> `--analysis-strings` / `--no-analysis-strings` -- `network.dns.names` -> `--network-dns-names c2.example=203.0.113.10` -- `api_hammering.allow_list` -> `--api-hammering-allow-list kernel32.WriteFile` +## CLI flags -Runtime precedence: -1. built-in model defaults -2. optional `--config` overlay -3. explicit CLI flags +The CLI has a flag for most config fields. `speakeasy/cli_config.py` makes the flags from the fields of `SpeakeasyConfig`. A flag joins the parts of the config path with `-` and replaces `_` with `-`, so `analysis.coverage` gives `--analysis-coverage`. The CLI leaves out a path part when the next part starts with the same word, so `modules.modules_always_exist` gives `--modules-always-exist` and `modules.module_directory_x86` gives `--module-directory-x86`. -The CLI reads the `--config` file in the same way as `Speakeasy(config=...)` reads a dict: -- a key that the file gives replaces the default value. This is also true for a mapping such as `env` or `network.dns.names`, and for a list -- a key that the file leaves out gets its default, also in a nested object. For example, `{"user": {"name": "bob"}}` keeps `user.is_admin` at `true` +A boolean field has two flags, such as `--analysis-strings` and `--no-analysis-strings`. An int flag takes a decimal value or a hex value that starts with `0x`. A list of strings takes one value for each flag, such as `--api-hammering-allow-list kernel32.WriteFile`, and the flag values replace the list. A mapping takes one `KEY=VALUE` for each flag, such as `--network-dns-names c2.example=203.0.113.10`, and each flag adds or replaces one key. A list of objects, such as `processes`, `filesystem.files` or `registry.keys`, has no flag, so set it in a config file. -`--volume` is a dedicated shortcut for filesystem mappings and is applied before schema-derived CLI overrides. +The CLI builds the config in this order, and each step changes the result of the step before it: -Use `--dump-default-config` to emit the built-in default profile. +1. the built-in defaults +2. the `--config` file +3. the `--volume` mounts, which add entries to `filesystem.files` (see [volumes](volumes.md)) +4. the config flags -Speakeasy does not currently ship named built-in config profiles beyond that default baseline. If you use `--config`, you are supplying your own overlay file. +The CLI reads the `--config` file in the same way as `Speakeasy(config=...)` reads a dict. A key that the file gives replaces the default value, also when the value is a mapping such as `env` or a list. A key that the file leaves out gets its default, also in a nested object. For example, `{"user": {"name": "bob"}}` keeps `user.is_admin` at `true`. -Complex nested structures remain config-file-only (for example `filesystem.files`, `registry.keys`, `processes`, `modules.user_modules`, `modules.system_modules`, and structured network response lists). +Speakeasy has no named config profiles. To make a config file, start from the output of `--dump-default-config`. ## Operational notes diff --git a/doc/errors.md b/doc/errors.md index 6a5f97ac6..71fd6f96e 100644 --- a/doc/errors.md +++ b/doc/errors.md @@ -27,8 +27,8 @@ Each method that uses the emulator raises `SpeakeasyError` if no input is loaded | Method | Raises | |---|---| | `Speakeasy(...)` | `ConfigError` if the configuration is not valid, a volume is malformed or its host path does not exist, or `volumes` is set together with a `SpeakeasyConfig` | -| `load_module` | `InputError` if neither `path` nor `data` is given, the file is missing, the input is not a valid PE, or its architecture is not x86 or x64; `NotSupportedError` for a .NET assembly | -| `load_shellcode` | `InputError` if neither `fpath` nor `data` is given, the file is missing, or `arch` is not x86, x64 or amd64 | +| `load_module` | `InputError` if neither `path` nor `data` is given, the file is missing, the input is not a valid PE or its architecture is not x86 or x64; `NotSupportedError` for a .NET assembly | +| `load_shellcode` | `InputError` if neither `fpath` nor `data` is given, the file is missing or `arch` is not x86, x64 or amd64 | | `run_shellcode` | `SpeakeasyError` if the kernel mode emulator is loaded | | `disasm`, `mem_read`, `mem_write` | `SpeakeasyError` if the operation fails | | `mem_alloc` | `OutOfMemoryError` if no free range below the address limit can hold the block | diff --git a/doc/exceptions.md b/doc/exceptions.md index 9294dd207..01ec046d6 100644 --- a/doc/exceptions.md +++ b/doc/exceptions.md @@ -1,73 +1,81 @@ # Exception dispatch -With `exceptions.dispatch_handlers` enabled (the default), exceptions reach the -sample's own handlers the way they would on Windows, so code that relies on -exceptions for control flow or anti-analysis keeps running. Each exception that -is offered to a handler is reported as an `exception` event. +When `exceptions.dispatch_handlers` is set (the default), the emulator sends +exceptions to the handlers of the sample as Windows does. So code that uses +exceptions for control flow or anti-analysis continues to run. The report has +an `exception` event for each exception that the emulator sends to a handler. ## What raises exceptions -- CPU faults: access violations, including reads of no-access pages and writes - to read-only pages, and division by zero. -- Trap instructions, following Wine: `int3` and `int 3`, `int 2d` (or nothing, - for the debug service codes 1, 3, 4 and 5), other `int n` as access - violations, `ud2`, `icebp`, `hlt`, `in` and `out`, and the trap flag. The - system service gates (`int 2e`, and `int 2a` and `int 2b` on x86) are not - emulated and end the run as an unhandled interrupt. -- `PAGE_GUARD` pages, once, with `STATUS_GUARD_PAGE_VIOLATION`. -- Execute breakpoints in the debug registers of the current thread, set with - `SetThreadContext` or by a handler that edits its context record. +- CPU faults. These are access violations, which include reads of no-access + pages and writes to read-only pages, and division by zero. +- Trap instructions, as in Wine. These are `int3` and `int 3`, `int 2d`, other + `int n` as access violations, `ud2`, `icebp`, `hlt`, `in`, `out` and the trap + flag. `int 2d` raises nothing for the debug service codes 1, 3, 4 and + 5. The emulator does not emulate the system service gates (`int 2e`, and + `int 2a` and `int 2b` on x86). They end the run as an unhandled interrupt. +- `PAGE_GUARD` pages, one time, with `STATUS_GUARD_PAGE_VIOLATION`. +- Execute breakpoints in the debug registers of the current thread. The sample + sets them with `SetThreadContext`, or a handler sets them in its context + record. - `RaiseException` and `RtlRaiseException`. -- An API handler that touches an invalid guest pointer: the access violation is - raised in the API's caller, as the API's own code would fault. +- An API handler that accesses a guest pointer that is not valid. The emulator + raises the access violation in the caller of the API, at the same point as + the fault of the real API code. ## Who handles them -Vectored handlers run first, then frame-based handlers: +Vectored handlers run first, then frame-based handlers. -- x64 walks the stack with the unwind data of images (`.pdata`), or of tables - registered with `RtlAddFunctionTable`, and calls language handlers such as - `__C_specific_handler`. `RtlUnwind` and `RtlUnwindEx` run `__finally` blocks - on the way to the target frame. -- x86 walks the chain of registrations at `fs:[0]`. The MSVC language handlers - `_except_handler3` and `_except_handler4_common` run filters and `__finally` - blocks; `RtlUnwind` unwinds registrations. +On x64, the emulator walks the stack with the unwind data (`.pdata`) of the +images, or of the tables that the sample adds with `RtlAddFunctionTable`. It +calls language handlers such as `__C_specific_handler`. `RtlUnwind` and +`RtlUnwindEx` run the `__finally` blocks between the current frame and the +target frame. -When no frame handles an exception, the filter of `SetUnhandledExceptionFilter` -runs, and may continue execution. Otherwise the run ends with the error of the -exception, as Windows would end the process, whether or not a handler saw it; -an unhandled access violation is `invalid_read`, `invalid_write` or -`invalid_fetch`. As an exception to that, `RaiseException` returns to its -caller when nothing takes a continuable exception, so that emulation goes on. +On x86, the emulator walks the chain of registrations at `fs:[0]`. The MSVC +language handlers `_except_handler3` and `_except_handler4_common` run filters +and `__finally` blocks, and `RtlUnwind` unwinds registrations. -A fault that repeats at one address without progress ends the run after a few -attempts, rather than looping until the timeout. +When no handler takes an exception, the emulator calls the filter that the +sample set with `SetUnhandledExceptionFilter`. The filter can continue +execution. If it does not, or if the sample set no filter, the run ends with the +error of the exception, as Windows ends the process. An unhandled access violation gives the error +`invalid_read`, `invalid_write` or `invalid_fetch`. `RaiseException` is +different: when no handler takes a continuable exception, it returns to its +caller, so emulation continues. + +If a fault occurs again and again at one address and the code does not move +forward, the emulator ends the run after a small number of attempts. It does +not loop until the time limit. ## Where the code is -- `speakeasy/windows/guest_callbacks.py`: how a Python handler runs guest code, - such as a filter, and gets its result, without nesting emulation. -- `speakeasy/windows/seh.py`: what both architectures share; `seh64.py` and - `unwind.py` for x64, `seh32.py` for x86. -- `speakeasy/windows/cpu_exceptions.py`: the exception a CPU event becomes. -- `speakeasy/windows/thread_context.py` and `hardware_breakpoints.py`: thread - contexts and debug registers. -- `speakeasy/winenv/api/exception_apis.py`: the exception and unwind APIs that - kernel32 and ntdll share. +- `speakeasy/windows/guest_callbacks.py` lets a Python handler run guest code, + such as a filter, and get its result, without nested emulation. +- `speakeasy/windows/seh.py` has the code for both architectures. `seh64.py` + and `unwind.py` are for x64, and `seh32.py` is for x86. +- `speakeasy/windows/cpu_exceptions.py` converts a CPU event to an exception. +- `speakeasy/windows/thread_context.py` and `hardware_breakpoints.py` have the + thread contexts and the debug registers. +- `speakeasy/winenv/api/exception_apis.py` has the exception and unwind APIs + that kernel32 and ntdll share. ## Limitations -- Unicorn does not enforce privilege levels, so `cli`, `sti`, `mov cr0` and - `rdmsr` run without raising an exception. -- Data breakpoints (a Dr7 R/W field other than execute) are not emulated. -- Callbacks registered with `RtlInstallFunctionTableCallback` are not called; - code in their range unwinds as leaf functions. -- XMM6-XMM15 are not restored when continuing at a context. -- On x64, a fault in a function's epilogue unwinds as if it were in the body. -- Security cookies of SEH4 frames are not checked, and x86 registrations are - not validated against the stack or SafeSEH. -- C++ exceptions are not emulated: `_CxxThrowException` returns, and the C++ +- Unicorn does not apply privilege levels, so `cli`, `sti`, `mov cr0` and + `rdmsr` do not raise an exception. +- Data breakpoints (a Dr7 R/W field other than execute) have no effect. +- The emulator does not call the callbacks that the sample adds with + `RtlInstallFunctionTableCallback`. Code in their range unwinds as leaf + functions. +- When execution continues at a context, the emulator does not restore + XMM6-XMM15. +- On x64, a fault in the epilogue of a function unwinds as a fault in the body. +- The emulator does not check the security cookies of SEH4 frames, or check + x86 registrations against the stack or SafeSEH. +- The emulator does not support C++ exceptions. `_CxxThrowException` returns, and the C++ frame handlers do not run catch blocks. -- Continuing a non-continuable exception ends the run with - `STATUS_NONCONTINUABLE_EXCEPTION`, where Windows would raise that as a new - exception. +- If a handler continues a non-continuable exception, the run ends with + `STATUS_NONCONTINUABLE_EXCEPTION`. Windows raises a new exception in this + case. diff --git a/doc/gdb.md b/doc/gdb.md index 5a8d11b45..b156fc304 100644 --- a/doc/gdb.md +++ b/doc/gdb.md @@ -1,10 +1,8 @@ # GDB Debugging -Speakeasy supports interactive debugging of emulated binaries via the GDB Remote Serial Protocol. When enabled, the emulator pauses before the first instruction and waits for a GDB client to connect. You can then set breakpoints, inspect registers and memory, single-step, and continue execution — all through a standard GDB interface. +Speakeasy has a GDB server for the GDB Remote Serial Protocol (RSP). With the server on, the emulator stops before the first instruction and waits for a GDB client. In the client, you can set breakpoints, read and write registers and memory, step and continue. -Synthesized Windows API pointers returned by `GetProcAddress`, bound imports, and export-table walks refer to mapped executable entries. Set a breakpoint directly on that pointer; stepping executes its public bytes before handler dispatch. See [Unified Windows API addresses](unified-api-addresses.md) for entry layouts, dynamic-only symbols and unsupported-call stops. - ---- +`GetProcAddress`, the IAT and the export table give the address of a Windows API as the address of code in the DLL image. You can set a breakpoint on this address. When you step into the API, the guest runs this code before the emulator calls the Python handler. See [Unified Windows API addresses](unified-api-addresses.md) for the layout of this code. ## Quick Start @@ -14,7 +12,7 @@ Synthesized Windows API pointers returned by `GetProcAddress`, bound imports, an speakeasy -t sample.exe --gdb ``` -This starts the GDB server on the default port (1234). Speakeasy will print a message and block until a GDB client connects: +The GDB server listens on port 1234. Speakeasy logs this message and waits until a GDB client connects: ``` Native GDB server listening on 127.0.0.1:1234, waiting for connection... @@ -26,21 +24,19 @@ To use a different port: speakeasy -t sample.exe --gdb --gdb-port 9999 ``` -The server binds to loopback by default. To allow a debugger on another machine to connect, explicitly choose a bind address: +The server listens on loopback. To let a debugger on a different machine connect, give a bind address: ```console speakeasy -t sample.exe --gdb --gdb-host 0.0.0.0 --gdb-port 9999 ``` -> **Warning:** GDB RSP has no authentication and permits target memory and register modification. Only use a non-loopback `--gdb-host` on a trusted network or behind an appropriate firewall/tunnel. -> -> `--gdb` automatically enables `--no-mp` because the server must remain attached to the same Unicorn instance. +RSP has no authentication, and a client can change the memory and registers of the target. Use a `--gdb-host` other than loopback only on a trusted network, or behind a firewall or a tunnel. Speakeasy logs a warning for such a host. With `--gdb`, the CLI also sets `--no-mp`. ### 2. Connect with GDB -In another terminal, connect using `gdb-multiarch` (or your platform's GDB): +In a different terminal, connect with `gdb-multiarch` or the GDB of your platform. -**For a 32-bit x86 binary:** +For a 32-bit x86 binary: ```console $ gdb-multiarch @@ -48,7 +44,7 @@ $ gdb-multiarch (gdb) target remote localhost:1234 ``` -**For a 64-bit x86-64 binary:** +For a 64-bit x86-64 binary: ```console $ gdb-multiarch @@ -56,11 +52,11 @@ $ gdb-multiarch (gdb) target remote localhost:1234 ``` -Once connected, GDB will show the current stop position. From here you can debug normally. +GDB then shows the address where the emulator stopped. ### 3. Debug -Standard GDB commands work: +Use the usual GDB commands: ``` (gdb) info registers @@ -71,8 +67,6 @@ Standard GDB commands work: (gdb) x/32xb $esp ``` ---- - ## Examples ### Debugging a DLL @@ -119,22 +113,18 @@ $ gdb-multiarch (gdb) continue ``` ---- - ## Using with IDA Pro -IDA Pro's built-in GDB debugger can connect to the speakeasy GDB stub: +The Remote GDB debugger of IDA Pro can connect to the Speakeasy GDB server: 1. Start speakeasy with `--gdb` 2. In IDA, go to **Debugger > Select debugger > Remote GDB debugger** -3. Set **Hostname** to `localhost` and **Port** to `1234` (or your chosen port) +3. Set **Hostname** to `localhost` and **Port** to `1234` (or your port) 4. Click **Debugger > Attach to process** or **Start process** ---- - ## Programmatic Usage -The GDB server can also be enabled when using speakeasy as a library: +To start the GDB server from the library, give `gdb_port` to `Speakeasy`: ```python import speakeasy @@ -145,49 +135,44 @@ module = se.load_module("sample.dll") se.run_module(module) ``` ---- - ## How It Works -Speakeasy implements the all-stop GDB Remote Serial Protocol directly. The server provides register and memory access, software and hardware breakpoints, watchpoints, single-step, asynchronous Ctrl-C pause, and the Windows process metadata used by IDA: +Speakeasy implements the all-stop mode of RSP itself. A reader thread reads the RSP packets from the socket. When it gets Ctrl-C, it stops Unicorn. Breakpoint and watchpoint hooks record the stop reason and stop Unicorn. The emulator thread handles the packets only after Unicorn stops. -- `qXfer:libraries:read` returns all mapped Speakeasy PE modules; mapping one while the target runs (for example through `LoadLibrary`) halts it with the `library` stop reason so the client re-reads the list -- `qXfer:exec-file:read` returns the emulated executable path -- `qXfer:threads:read`, `qfThreadInfo`, and `qGetTIBAddr` describe the active emulated thread -- `qXfer:features:read` reports a register layout matching the emulated architecture -- `qXfer:memory-map:read` reports Unicorn's mapped memory regions -- `vCont`, legacy resume packets, register/memory writes, detach, and process exit are supported -- when the last run ends without a fault, the server halts the target with a `T05` stop reply before it sends `W00`, even if an earlier run faulted. The client can then read the final memory, for example to dump an unpacked image after `ExitProcess`. When the last run faults, the server sends `X0b` after the fault stop. +The server gives the Windows process data that IDA uses: -A socket reader handles RSP framing and can stop Unicorn asynchronously when it receives Ctrl-C. Breakpoint and watchpoint hooks only record a stop reason and stop Unicorn; protocol processing happens after `emu_start()` returns, rather than from inside a Unicorn callback. +- `qXfer:libraries:read` gives all mapped PE modules. When a module maps or unmaps while the target runs, for example in `LoadLibrary`, the server stops the target with the `library` stop reason. The client then reads the list again. +- `qXfer:exec-file:read` gives the path of the module of the current run. +- `qXfer:threads:read` and `qfThreadInfo` give the active thread, and `qGetTIBAddr` gives the address of its TEB. +- `qXfer:features:read` gives the registers of the emulated architecture. +- `qXfer:memory-map:read` gives the mapped memory regions of Unicorn. ---- +When the last run ends without a fault, the server stops the target with a `T05` stop reply, and then sends `W00`. This is also true when an earlier run faulted. So the client can read the final memory, for example to dump an unpacked image after `ExitProcess`. When the last run faults, the server sends `X0b` after the fault stop. ## Supported protocol subset -The server implements the all-stop RSP subset exercised by GDB and IDA: +The server implements the all-stop RSP subset that GDB and IDA use: | Area | Packets | |---|---| | Transport | checksums, ACK/NACK retransmission, `QStartNoAckMode`, binary escaping, raw Ctrl-C | | Target discovery | `qSupported`, `qXfer:features:read`, `qXfer:memory-map:read` | | Windows metadata | `qXfer:libraries:read`, `qXfer:exec-file:read`, `qXfer:threads:read`, `qGetTIBAddr` | -| Thread selection | `qC`, `qfThreadInfo`, `qsThreadInfo`, `H`, `T` for the synthetic active thread | +| Thread selection | `qC`, `qfThreadInfo`, `qsThreadInfo`, `H`, `T` for the one active thread | | Registers | `g`, `G`, `p`, `P` | | Memory | `m`, `M`, `X` | -| Breakpoints | `Z0`/`z0`, `Z1`/`z1`, and read/write/access watchpoints | -| Execution | `c`, `s`, `C`, `S` (the signal is ignored), `vCont`, asynchronous Ctrl-C, `?`, `swbreak`/`hwbreak`/watchpoint/`library` stop replies | -| Lifecycle | `D`, `k`, a final `T05` stop before exit, and `Wxx` exit replies | +| Breakpoints | `Z0`/`z0`, `Z1`/`z1` and the read, write and access watchpoints (`Z2` to `Z4`) | +| Execution | `c`, `s`, `C`, `S` (the server ignores the signal), `vCont`, asynchronous Ctrl-C, `?`, `swbreak`/`hwbreak`/watchpoint/`library` stop replies | +| Lifecycle | `D`, `k`, `vKill`, a final `T05` stop before exit and `Wxx` exit replies | -Unsupported optional packets receive an empty response as required by RSP. Non-stop mode, multiprocess mode, reverse execution, and remote file I/O are not advertised. +The server sends an empty reply to an optional packet that it does not support, as RSP requires. The server does not implement non-stop mode, multiprocess mode, reverse execution or remote file I/O. ## Limitations -- one debugger client can connect to a Speakeasy instance -- Speakeasy models Windows thread objects but executes one Unicorn CPU context at a time; the server therefore presents the currently active execution context as one synthetic GDB thread -- breakpoints persist across runs: Speakeasy executes multiple runs (for example TLS callbacks, the module entry point, and exports) -- modules stay mapped for the life of the emulation, so the server reports library loads but never unloads -- reverse execution, non-stop mode, and multiprocess RSP are not implemented +- One debugger client can connect to a Speakeasy instance. +- Speakeasy has Windows thread objects, but Unicorn runs one CPU context at a time. The server shows the active context as one GDB thread. +- The server keeps breakpoints across runs. A run is one call into guest code, such as a TLS callback, the entry point or an export. +- `FreeLibrary` does not unmap a module, so the library list does not change after it. ## Related docs diff --git a/doc/help.md b/doc/help.md index ed9ba9d06..c4d373a0b 100644 --- a/doc/help.md +++ b/doc/help.md @@ -1,65 +1,37 @@ # Help and troubleshooting -Use this page when you are blocked and need the fastest path to the right documentation. +This page tells you which page to read for a problem. -## First step: route to the right page +## Where to look -- Unsure where to look: start at the [documentation index](index.md) -- Installation/runtime setup: [install.md](install.md) -- CLI flags and invocation rules: [cli-reference.md](cli-reference.md) -- Environment shaping and determinism: [cli-environment-overrides.md](cli-environment-overrides.md) -- Runtime controls and stop conditions: [cli-execution-controls.md](cli-execution-controls.md) -- Report schema and field semantics: [reporting.md](reporting.md) +- All pages: [documentation index](index.md) +- Overview and quick start: [project README](../README.md) +- Installation: [install.md](install.md) +- CLI flags: [cli-reference.md](cli-reference.md) +- The emulated environment and repeatable runs: [cli-environment-overrides.md](cli-environment-overrides.md) +- Limits and stop conditions: [cli-execution-controls.md](cli-execution-controls.md) +- Report fields: [reporting.md](reporting.md) - Interactive debugging: [gdb.md](gdb.md) ## Common issues -### Unsupported API errors +If the log shows `Unsupported API: .` and the report has an `unsupported_api` error, the sample called an API that has no handler and no signature. [limitations.md](limitations.md) tells you why the run stops, and [api-handlers.md](api-handlers.md) tells you how to add a handler. -Symptom: -- `Unsupported API: .` in the log, and an `unsupported_api` error in the report +If the sample exits too early or takes a path that you do not expect, it can need a file, a registry key or another part of the environment. Read [configuration.md](configuration.md) and [cli-environment-overrides.md](cli-environment-overrides.md). -Read: -- [limitations.md](limitations.md) -- [api-handlers.md](api-handlers.md) +If a run is too slow or the report is too large, read [cli-analysis-recipes.md](cli-analysis-recipes.md) and [performance and telemetry tuning](cli-execution-controls.md#performance-and-telemetry-tuning). -### Sample exits too early or behaves differently than expected +To step through the code of a sample, read [gdb.md](gdb.md). To give host files to the sample, read [volumes.md](volumes.md) and [cli-reference.md](cli-reference.md). -Check: -- [configuration.md](configuration.md) -- [cli-environment-overrides.md](cli-environment-overrides.md) +## Before you open an issue -### Run is too slow or produces very large artifacts +Include these items, so that others can reproduce the problem: -Check: -- [cli-analysis-recipes.md](cli-analysis-recipes.md) -- [cli-execution-controls.md](cli-execution-controls.md) -- [cli-execution-controls.md#performance-and-telemetry-tuning](cli-execution-controls.md#performance-and-telemetry-tuning) +- the Speakeasy version (`speakeasy --version`) and how you installed it +- the full command line +- the config file, if you used one +- the type of the input (`exe`, `dll`, `sys` or raw shellcode) +- the `--verbose` log near the failure +- the `errors` and `entry_points[*].error` fields of the report -### Need to step through execution interactively - -Check: -- [gdb.md](gdb.md) - -### Need mounted files in emulated filesystem - -Check: -- [volumes.md](volumes.md) -- [cli-reference.md](cli-reference.md) - -## Before opening an issue - -Include the following so others can reproduce quickly: - -- Speakeasy version (`speakeasy --version`) and install method -- full command line used -- config overlay (if any) -- target type (`exe`, `dll`, `sys`, raw shellcode) -- `--verbose` log excerpt around failure -- report excerpt (`errors`, `entry_points[*].error`) - -## Need more help - -- Documentation hub: [index.md](index.md) -- Project landing page: [../README.md](../README.md) -- Issue tracker: [github.com/mandiant/speakeasy/issues](https://github.com/mandiant/speakeasy/issues) +Report issues in the [issue tracker](https://github.com/mandiant/speakeasy/issues). diff --git a/doc/index.md b/doc/index.md index a52875392..77dd7b947 100644 --- a/doc/index.md +++ b/doc/index.md @@ -1,12 +1,13 @@ # Documentation index -This page maps common tasks to the right Speakeasy documentation. +This page gives the Speakeasy documentation for each task. ## Start here - [Project README](../README.md) - [Installation](install.md) - [Help and troubleshooting](help.md) +- [What's new in 2.0](speakeasy2-walkthrough.md) - [Migrating to 2.0](migrating-to-2.0.md) ## CLI usage @@ -15,6 +16,7 @@ This page maps common tasks to the right Speakeasy documentation. - [CLI analysis recipes](cli-analysis-recipes.md) - [CLI environment overrides](cli-environment-overrides.md) - [CLI execution controls](cli-execution-controls.md) +- [GDB debugging reference](gdb.md) ## Configuration and reports @@ -25,10 +27,6 @@ This page maps common tasks to the right Speakeasy documentation. - [Python exceptions that Speakeasy raises](errors.md) - [Limitations](limitations.md) -## Debugging - -- [GDB debugging reference](gdb.md) - ## Extending and embedding - [Architecture of the Windows emulator](architecture.md) @@ -37,5 +35,3 @@ This page maps common tasks to the right Speakeasy documentation. - [Adding API handlers](api-handlers.md) - [Unified Windows API addresses](unified-api-addresses.md) - [Examples directory](../examples/) -- [Speakeasy 2 walkthrough outline](speakeasy2-walkthrough.md) - diff --git a/doc/install.md b/doc/install.md index 3ccf77f84..d5878885b 100644 --- a/doc/install.md +++ b/doc/install.md @@ -8,7 +8,7 @@ Speakeasy requires Python 3.10+. python3 -m pip install speakeasy-emulator ``` -Verify installation: +Check the installation: ```console speakeasy -h @@ -24,9 +24,9 @@ python3 scripts/gen_win32_signatures.py python3 scripts/gen_phnt_signatures.py ``` -The last two steps build the API signature databases from the `deps/win32json` and `deps/phnt` submodules (see [Adding API handlers](api-handlers.md)). Wheels built with `python -m build` include them automatically; an editable install needs them generated once, and again after updating the submodules. +The last two steps build the API signature databases from the `deps/win32json` and `deps/phnt` submodules (see [Adding API handlers](api-handlers.md)). `python -m build` builds the databases and puts them in the wheel. For an editable install, run the two scripts once, and again after you update the submodules. -GDB support ([GDB debugging](gdb.md)) needs no extra packages. +[GDB debugging](gdb.md) needs no extra packages. ## Related docs diff --git a/doc/library.md b/doc/library.md index c1a05af16..57ff7e54c 100644 --- a/doc/library.md +++ b/doc/library.md @@ -1,12 +1,12 @@ # Python library usage -You can use Speakeasy as a Python library through the `speakeasy.Speakeasy` interface. +To use Speakeasy from Python, create a `speakeasy.Speakeasy` object. ## Public interface The public interface is the names in `speakeasy.__all__`, and the methods of `Win32Emulator` and `WinKernelEmulator` (including the methods they inherit) whose names do not start with `_`. Speakeasy follows semantic versioning for this interface: only a major release removes a name or changes a signature in a way that breaks a caller. Import these names from `speakeasy`, for example `from speakeasy import Speakeasy, InputError`. -All other modules, such as `speakeasy.windows.winemu` or `speakeasy.winenv.api`, are internal, and a minor release can change them. `speakeasy.defs` and `speakeasy.arch` stay available for the structures and constants of the Windows API, but they are internal too. +All other modules, such as `speakeasy.windows.winemu` or `speakeasy.winenv.api`, are internal, and a minor release can change them. You can use `speakeasy.defs` and `speakeasy.arch` for the structures and constants of the Windows API, but they are internal too. The boolean parameters of the public methods are keyword-only, for example `se.run_module(module, all_entrypoints=True)` and `se.disasm(addr, size, fast=False)`. @@ -39,16 +39,13 @@ report = se.get_report() se.shutdown() ``` -## Library options you may want +## Options -- pass `config=` to control environment and analysis -- pass `argv=` to populate emulated process arguments -- pass `volumes=` to mount host paths into emulated filesystem -- pass `gdb_port=` to block for GDB attach before execution; `gdb_host=` controls the bind address and defaults to `127.0.0.1` +The `Speakeasy` constructor takes these arguments. `config=` sets the emulated environment and the analysis options, as a dict or a `SpeakeasyConfig` (see [Configuration walkthrough](configuration.md)). `argv=` sets the arguments of the emulated program. `volumes=` mounts host paths into the emulated file system, and it requires a dict or no `config=`. With `gdb_port=`, the first run waits for a GDB client before it starts (see [GDB debugging reference](gdb.md)). `gdb_host=` sets the address on which the GDB server listens, and its default is `127.0.0.1`. ## Hooks -Each `add_*_hook` method calls its callback with the emulator as the first argument. You can add a hook before or after `load_module`. The method returns a hook object, and `se.remove_hook(hook)` removes the hook. `speakeasy` has a `Protocol` for each callback type, for example `CodeHookCallback`. +The emulator calls each hook callback with the emulator as the first argument. You can add a hook before or after `load_module`. Each `add_*_hook` method returns a hook object, and `se.remove_hook(hook)` removes the hook. `speakeasy` has a `Protocol` for each callback type, for example `CodeHookCallback`. | Method | Callback | Return value of the callback | |---|---|---| @@ -64,7 +61,7 @@ Each `add_*_hook` method calls its callback with the emulator as the first argum | `add_in_instruction_hook(cb, begin, end)` | `cb(emu, port, size)` | the value that the IN instruction reads | | `add_syscall_instruction_hook(cb, begin, end)` | `cb(emu)` | not used | -The SYSCALL hook fires for SYSCALL and SYSENTER. Only one IN hook runs for an instruction: the first hook that is enabled and covers the address. When `exceptions.dispatch_handlers` is set, the user mode emulator adds its own IN hook at the start of the first run. That hook raises a privileged instruction exception, as Windows does. An IN hook that you add before the run comes first, so it replaces the exception. +The emulator calls a SYSCALL hook for SYSCALL and SYSENTER. For an IN instruction, the emulator calls only the first IN hook that is enabled and covers the address. When `exceptions.dispatch_handlers` is set, the user mode emulator adds its own IN hook at the start of the first run. That hook raises a privileged instruction exception, as Windows does. If you add an IN hook before the run, your hook comes first, so the exception does not occur. This code hook runs one time, then removes itself: @@ -77,7 +74,7 @@ def on_first_instruction(emu, address, size): hook = se.add_code_hook(on_first_instruction) ``` -`se.add_api_hook(cb, "user32", "MessageBox*")` calls `cb(emu, api_name, func, params)` in place of the handler. `params` holds the raw argument slots, and `func(params)` runs the handler. After that call, `emu.get_api_args()` returns the arguments as the report shows them, with the values the handler decoded: +After `se.add_api_hook(cb, "user32", "MessageBox*")`, the emulator calls `cb(emu, api_name, func, params)` in place of the handler. `params` holds the raw argument slots, and `func(params)` runs the handler. `func` is `None` when the API has no handler. After `func(params)`, `emu.get_api_args()` returns the arguments as the report shows them, with the values that the handler decoded: ```python def hook_messagebox(emu, api_name, func, params): diff --git a/doc/limitations.md b/doc/limitations.md index 11485bdc5..e79eb87ea 100644 --- a/doc/limitations.md +++ b/doc/limitations.md @@ -1,39 +1,30 @@ # Limitations -Speakeasy does not delegate API calls, object management, or I/O to a real Windows kernel. Those behaviors are modeled by the emulator, so unsupported paths or environment mismatches can stop execution early. +Speakeasy does not send API calls, object operations or I/O to a real Windows kernel. The emulator models them, so an API without a model or a difference in the environment can stop execution early. Exceptions reach the handlers of the sample, with the limits in [Exception dispatch](exceptions.md). ## Unimplemented APIs -Samples call APIs with assumptions about arguments, return values, side effects, and stack behavior. Speakeasy has hand-written handlers for a curated subset of the Windows API; everything else is handled in one of two ways. +A sample expects a specific argument layout, return value, side effects and stack cleanup from each API that it calls. Speakeasy has handlers for a part of the Windows API. For the other APIs, Speakeasy uses one of the two methods below. ### Documented APIs without a handler -If the import is declared in the bundled Win32 API signature database (generated from Microsoft's win32metadata, see [Adding API handlers](api-handlers.md)), Speakeasy emulates it from the declaration alone: +If the bundled signature database declares the import (see [Adding API handlers](api-handlers.md)), Speakeasy emulates the call from the declaration: -- the correct number of argument slots is consumed and, for stdcall on x86, cleaned up -- the call is traced with parameter names and decoded strings/booleans, e.g. `kernel32.MoveFileExW(lpExistingFileName: "C:\old.txt", lpNewFileName: "C:\new.txt", dwFlags: 0x1)` -- a type-appropriate success value is returned (`TRUE`, `S_OK`, a fresh handle, ...) +- It reads the correct number of argument slots and, for stdcall on x86, removes them from the stack. +- It logs the call with parameter names and decoded strings and booleans, for example `kernel32.MoveFileExW(lpExistingFileName: "C:\old.txt", lpNewFileName: "C:\new.txt", dwFlags: 0x1)`. +- It returns a success value for the return type (`TRUE`, `S_OK`, a new handle, ...). -What the fallback does **not** do: it has no real side effects. `Out`-only pointer parameters are zero-filled for as many bytes as the prototype declares (the pointed-to scalar or struct, a buffer sized by a sibling count parameter, or just the terminator of an unsized string), so a sample that reads them back sees empty strings, NULL handles and zero counts rather than stale stack contents; but no genuine data is ever produced. A run continues after a call to an API that has a declaration but no handler. Treat calls in the trace that have parameter names (`name: value`) as unimplemented stubs when interpreting a report. +The call has no real side effects. Speakeasy writes zeros to each `Out`-only pointer parameter, for the size that the prototype declares: the scalar or struct that the pointer points to, a buffer with the size of a related count parameter, or only the terminator of a string without a size. So a sample that reads these parameters gets empty strings, NULL handles and zero counts, and not old stack data. But it never gets real data. The run continues after the call. The report does not mark these calls, and handler calls also have parameter names. To find them, use `--verbose`: the log has a `no handler for .` line for each of these calls. ### Unknown APIs -If neither a handler nor a signature exists (the C runtime, third-party DLLs, natives that neither win32metadata nor phnt declares), the current run stops and records an error. - -Expected error pattern: +If the API has no handler and no signature, the current run stops with an error. Examples are a C runtime function without a handler, a function of a third-party DLL or a native function that neither win32metadata nor phnt declares. The run also stops when the signature has a form that the emulator cannot call, for example a variadic function. The log shows: - `Unsupported API: .` -Why execution stops: - -- for unknown APIs, argument count/calling convention cannot be trusted -- continuing may corrupt stack state and generate misleading report data - -Queued runs (for example additional entry points) can still execute. `modules.functions_always_exist` forces these calls through as 4-argument stdcall stubs returning 1. +The emulator does not know the argument count or the calling convention of an unknown API. If the run continued, the stack could be wrong and the report could have incorrect data. -## Exceptions - -Exceptions reach the sample's own handlers, with the limits listed in [Exception dispatch](exceptions.md). +The runs in the queue, for example other entry points, still run. With `modules.functions_always_exist`, Speakeasy emulates a call to an API without a declaration as a stdcall function with 4 arguments that returns 1. ## Threads @@ -43,13 +34,11 @@ Threads do not run at the same time. Each thread is a run in the run queue, and The ANSI code page is 1252 and the OEM code page is 437. Speakeasy reads and writes `A` strings in code page 1252, so each byte is one character. A character that is not in code page 1252 becomes `?`. For some of these characters Windows uses a best-fit mapping (`ā` to `a`), but Speakeasy does not. `GetCPInfo`, `MultiByteToWideChar` and `WideCharToMultiByte` also support `CP_UTF8` and code page 437, and fail with `ERROR_INVALID_PARAMETER` for other code pages. `IsValidCodePage` returns `TRUE` only for 437, 1252 and 65001. The C runtime uses the C locale, so `mbstowcs` and `wcstombs` map each byte to the wide character with the same value. -`lstrcmp`, `lstrcmpi` and `CompareString` approximate the linguistic sort of Windows: they compare the strings without case first, and when these are equal, lower case sorts before upper case. Punctuation and symbols sort by their code values, without the Windows weights. The case-insensitive C runtime functions, such as `_stricmp`, fold only `A` to `Z`. +`lstrcmp`, `lstrcmpi` and `CompareString` approximate the linguistic sort of Windows. They compare the strings without case first, and when these are equal, lower case sorts before upper case. Punctuation and symbols sort by their code values, without the Windows weights. The case-insensitive C runtime functions, such as `_stricmp`, fold only `A` to `Z`. ## Environmental requirements -A sample may expect files, registry keys, network responses, loaded modules, or runtime structures that are not present in the active profile. These misses can look like anti-analysis behavior even when the issue is configuration drift. - -Use config and CLI overrides to model the expected environment before concluding a sample is unsupported. +A sample can expect files, registry keys, network responses, loaded modules or system structures that the config does not have. When one is missing, the sample can act as if it found an analysis environment, but the cause is the config. Before you decide that Speakeasy does not support a sample, add the missing items with the config file and the CLI overrides. ## Repeatable runs diff --git a/doc/memory.md b/doc/memory.md index aee807651..006600cb4 100644 --- a/doc/memory.md +++ b/doc/memory.md @@ -1,36 +1,22 @@ # Memory management -Speakeasy layers a memory manager on top of the emulator engine and tracks each mapped region with tags and metadata that can be exported in reports. +The Speakeasy memory manager wraps the memory functions of the CPU engine. It gives each mapped region a tag, and the report can include the regions with their tags. -## Memory tagging +## Memory tags -Each allocation is tagged with this namespace format: +A tag starts with the origin, then names the object and ends with the base address, for example `api.VirtualAlloc.0x450000` or `emu.stack.0x1200000`. The origin `emu` marks memory that the emulator maps, such as images, stacks and system structures. The origin `api` marks memory that an API handler allocates, for example for `VirtualAlloc`. -`...` +## Keeping freed memory -Typical origins: +Set `keep_memory_on_free` (or `--keep-memory-on-free`) to keep a region mapped after the sample frees it. Use this option when a sample frees a buffer soon after it writes the buffer, and you want to examine the buffer. -- `emu`: memory mapped by emulator internals (images, stacks, core runtime structures) -- `api`: memory allocated via API handlers (for example `VirtualAlloc` paths) +## Memory snapshots -## Memory freezing - -Set `keep_memory_on_free` (or `--keep-memory-on-free`) to keep mappings after free operations. This is useful when samples allocate, populate, and free buffers quickly but you still need to inspect resulting artifacts. - -## Memory acquisition mode - -In-report snapshots: - -- `--snapshot-memory-regions` stores region payload refs in the report -- resolve those refs through the top-level `data` section, which stores `base64(zlib(raw_bytes))` keyed by SHA-256 - -This keeps memory snapshots self-contained while deduplicating repeated regions across runs. +With `--snapshot-memory-regions`, the report includes the content of each memory region at the end of a run. Each region in the report has a `data_ref`, which is a SHA-256 key in the top-level `data` section. The `data` section keeps the bytes as `base64(zlib(raw_bytes))`. The report keeps one copy of bytes that repeat across regions or runs, and it does not need other files. ## Memory tracing -Set `analysis.memory_tracing` (or `--analysis-memory-tracing`) to track per-region read/write/execute counters and symbol access summaries. - -This adds overhead, especially on memory-heavy samples. +Set `analysis.memory_tracing` (or `--analysis-memory-tracing`) to count the reads, writes and executes of each region and to record the accesses to symbols. Memory tracing makes the run slower, mostly for samples that access much memory. ## Related docs diff --git a/doc/migrating-to-2.0.md b/doc/migrating-to-2.0.md index 8e050d761..ca699e67e 100644 --- a/doc/migrating-to-2.0.md +++ b/doc/migrating-to-2.0.md @@ -1,12 +1,12 @@ # Migrating to 2.0 -This page is for code and scripts that use Speakeasy as a Python library or from the command line, and move from 1.x to 2.0. Each section names a change to the interface and gives the code or command to change. +This page tells you how to move code and scripts from Speakeasy 1.x to 2.0. It covers the Python library and the command line. Each section names a change to the interface and tells you what to change in your code or command. ## Report schema 6.0.0 The report of 2.0 has `report_version` `6.0.0`. The report of 1.x has `1.1.0`. `Speakeasy.get_report` returns a `Report`, a pydantic model from `speakeasy.report`. In 1.x it returned a plain dict. -In 1.x, each entry point had one list for each kind of event: `apis`, `file_access`, `registry_access`, `network_events` (with `dns` and `traffic`), `process_events` and `handled_exceptions`. It also had `mem_access`, `sym_accesses`, `dynamic_code_segments` and `dropped_files`. In 2.0, an entry point has one `events` list in execution order in place of the event lists. The `event` key of each event gives its kind, and `pos` gives the tick, thread, process and `pc` of the event. `mem_access` moves to `memory`, `sym_accesses` has a new name, and `dynamic_code_segments` and `dropped_files` stay. +In 1.x, each entry point had one list for each kind of event: `apis`, `file_access`, `registry_access`, `network_events` (with `dns` and `traffic`), `process_events` and `handled_exceptions`. It also had `mem_access`, `sym_accesses`, `dynamic_code_segments` and `dropped_files`. In 2.0, an entry point has one `events` list in execution order in place of the event lists. The `event` key of each event gives its kind, and `pos` gives the tick, thread, process and `pc` of the event. `mem_access` moves to `memory`, and `sym_accesses` has a new name. `dynamic_code_segments` and `dropped_files` do not change. Before: @@ -102,7 +102,7 @@ These keys take one value from a fixed set. The schema lists them: | none | `compression` of a `data` entry | `zlib` | | none | `encoding` of a `data` entry | `base64` | -`ep_type` was one string with the kind of run and a detail. In 2.0, `type` holds the kind and `name` holds the detail. The JSON report leaves out `name` when the run has no detail. +In 1.x, `ep_type` was one string with the kind of run and a detail. In 2.0, `type` holds the kind and `name` holds the detail. The JSON report leaves out `name` when the run has no detail. | 1.x `ep_type` | 2.0 `type` | 2.0 `name` | |---|---|---| @@ -123,13 +123,13 @@ These keys take one value from a fixed set. The schema lists them: | `driver_reinit` | `driver_reinit` | none | | `irp_mj_`, for example `irp_mj_create` | `irp` | `IRP_MJ_`, for example `IRP_MJ_CREATE` | -2.0 also adds data. The top-level `image_base` gives the address where Speakeasy loaded the input. Each entry point has `pid`, `tid` and `memory`, the memory layout and the modules at the end of the run, and `coverage` with `--analysis-coverage`. The `dependency_dll_entry` and `dependency_tls_callback` runs execute the initializers of a DLL that the input loads. The new events are `mem_free`, `module_load`, `file_delete`, `directory_create`, `directory_delete`, `reg_write_value`, `reg_delete_key` and `reg_delete_value`. +2.0 also adds data. The top-level `image_base` gives the address where Speakeasy loaded the input. Each entry point has `pid`, `tid` and `memory`, the memory layout and the modules at the end of the run, and `coverage` with `--analysis-coverage`. The `dependency_dll_entry` and `dependency_tls_callback` runs execute the initializers of a DLL that the input loads. The new events are `file_delete`, `directory_create`, `directory_delete`, `reg_write_value`, `reg_delete_key` and `reg_delete_value`. The schema also has `mem_free` and `module_load`, but Speakeasy does not set them. The models reject unknown keys (`extra="forbid"`), so `Report.model_validate_json` fails for a 1.x report. `speakeasy/resources/report.schema.json` is the JSON schema of 6.0.0, and the [report walkthrough](reporting.md) describes each key. ## Exceptions -The exceptions in `speakeasy.errors` now have a tree with `InputError`, `ConfigError`, `EmulationError` and `EngineError` below `SpeakeasyError`. See [Python exceptions](errors.md). +The exceptions in `speakeasy.errors` form a tree. `SpeakeasyError` is the root, and `InputError`, `ConfigError`, `EmulationError` and `EngineError` are its subclasses. See [Python exceptions](errors.md). Two classes have new names: @@ -140,7 +140,7 @@ Two classes have new names: The old names are aliases of the new classes for one release, so `except EmuException` still works. Change it to the new name. -The `Speakeasy` constructor, `load_module` and `load_shellcode` check their arguments, and do not let exceptions of other packages go to the caller. +The `Speakeasy` constructor, `load_module` and `load_shellcode` check their arguments. They convert the exceptions of other packages to `InputError` or `ConfigError`: | Call | 1.x | 2.0 | |---|---|---| @@ -154,7 +154,7 @@ The `Speakeasy` constructor, `load_module` and `load_shellcode` check their argu | `Speakeasy(volumes=...)` with a malformed volume | `ValueError` | `ConfigError` | | `Speakeasy(volumes=...)` with a missing host path | `FileNotFoundError` | `ConfigError` | -`load_shellcode` now takes the architecture name in any case, so `"X64"` works. +`load_shellcode` accepts the architecture name in upper or lower case, for example `"X64"`. Before: @@ -220,19 +220,19 @@ These changes can break your code: | `get_registry_key(handle, path)` | `reg_get_key(handle, path)` | | `get_memory_dumps()`, `create_file_archive()` | none, keep a reference to the `Speakeasy` object | - `Speakeasy.mem_read` and `Speakeasy.mem_write` raise `SpeakeasyError` when the access fails. The emulator methods do not change the exception. + `Speakeasy.mem_read` and `Speakeasy.mem_write` raise `SpeakeasyError` when the access fails. The emulator methods raise `unicorn.UcError`. - A dynamic code callback gets `(emu, region)`. In 1.x it got only `region`. - `remove_code_hook` is removed. Use `remove_hook`. - `add_IN_instruction_hook` and `add_SYSCALL_instruction_hook` are now `add_in_instruction_hook` and `add_syscall_instruction_hook`. The old names give a `DeprecationWarning` for one release. - The IN hook now calls its callback with `(emu, port, size)`, and the IN instruction reads the return value. In 1.x the callback did not run. The SYSCALL hook now fires for SYSCALL and SYSENTER. In 1.x it fired only for SYSENTER. Both hooks now use `begin` and `end`. -Code that uses the internals of hooks also changes. The emulator `add_*_hook` methods do not take the `emu` argument. `Speakeasy` does not have the `api_hooks`, `code_hooks` and other lists of hooks that wait for the load, and `pending_hooks` holds them in the order that you added them. The `Hook` classes in `speakeasy.common` do not take `se_obj` and `emu_eng`, and the `se_obj` attribute is now `emu`. +Code that uses the internals of hooks also changes. The emulator `add_*_hook` methods do not take the `emu` argument. `Speakeasy` keeps the hooks that you add before the load in one list, `pending_hooks`, in the order that you add them. It has no `api_hooks`, `code_hooks` or other list for each hook type. The `Hook` classes in `speakeasy.common` do not take `se_obj` and `emu_eng`, and the `se_obj` attribute is now `emu`. ## Public interface and keyword-only flags `speakeasy.__all__` lists the public names, and [Python library usage](library.md#public-interface) tells which methods are public. Import the public names from `speakeasy`. The exceptions, `SpeakeasyConfig`, `Report` and the report models, `Hook` and the hook callback types are all available there. -`speakeasy` does not export `_PeParser` now. It is internal. Import it from `speakeasy.windows.common` if you must use it, but a minor release can change it. +`speakeasy` does not export `_PeParser`. It is internal. If you must use it, import it from `speakeasy.windows.common`. A minor release can change it. The boolean parameters of the public methods of `Speakeasy`, `Win32Emulator` and `WinKernelEmulator`, and the parameters after them, are keyword-only. A call that gives them by position raises `TypeError`. In this table, a method without a class name is a method of both emulators: @@ -282,7 +282,7 @@ se.run_module(module, all_entrypoints=True, emulate_children=False) ## Command line -Many flags have new names, and some short flags are removed. The CLI does not accept the 1.x names or an abbreviation of a flag, so an old flag is an argument error, and the CLI exits 2. +Many flags have new names, and some short flags are removed. The CLI rejects the 1.x names and abbreviated flags with an argument error and exit status 2. | 1.x | 2.0 | |---|---| diff --git a/doc/reporting.md b/doc/reporting.md index 1b50ad614..cb1c253c3 100644 --- a/doc/reporting.md +++ b/doc/reporting.md @@ -1,41 +1,40 @@ # Report walkthrough -Source of truth: -- `speakeasy/report.py` -- `speakeasy/profiler_events.py` -- population logic in `speakeasy/profiler.py`, `speakeasy/windows/winemu.py`, `speakeasy/windows/win32.py` and `speakeasy/windows/kernel.py` +The report models are in `speakeasy/report.py` and `speakeasy/profiler_events.py`. `speakeasy/profiler.py` builds the report from the data that the emulator records in `speakeasy/windows/winemu.py`, `win32.py` and `kernel.py`. -`speakeasy/resources/report.schema.json` is the JSON schema of the report. It ships in the package. Run `just gen-report-schema` after a change to the report models. The models reject unknown keys. +`speakeasy/resources/report.schema.json` is the JSON schema of the report, and the package includes it. After you change a report model, run `just gen-report-schema`. The models reject unknown keys. For the changes from the 1.x report, see [Migrating to 2.0](migrating-to-2.0.md#report-schema-600). -The examples use JSONC for explanation. Remove comments for machine parsing. +The examples on this page are JSONC: JSON with comments that explain the fields. Remove the comments before you parse an example as JSON. -## How report fields are populated +## Report fields -A key is absent when Speakeasy did not collect its data, for example `coverage` when `analysis.coverage=false`. An empty list or object means that Speakeasy collected the data and found nothing. +If Speakeasy did not collect the data of a key, the report leaves out the key, for example `coverage` when `analysis.coverage=false`. An empty list or object means that Speakeasy collected the data and found nothing. -The report writes each machine word as a hex string with a `0x` prefix and lowercase digits: addresses, pointers, handles, raw argument values, return values, register values, exception codes and coverage. A 64-bit word can be above 2\*\*53 or 2\*\*63. JavaScript, Go `interface{}`, jq and int64 stores such as SQLite lose such a value as a JSON number, and they give no error. In Python, `int(value, 16)` reads the string. The report writes each quantity, such as a size, a count, an offset, a tick or an ID, as a JSON number. The schema gives the pattern `^0x[0-9a-f]+$` for each hex string. +The report writes each machine word as a hex string with a `0x` prefix and lowercase digits. Machine words are addresses, pointers, handles, raw argument values, return values, register values, exception codes and coverage addresses. A 64-bit word can be above 2\*\*53 or 2\*\*63. JavaScript, Go `interface{}`, jq and int64 stores such as SQLite lose such a value when it is a JSON number, and they give no error. In Python, `int(value, 16)` reads the string. The schema gives the pattern `^0x[0-9a-f]+$` for each hex string. The report writes each quantity, such as a size, a count, an offset, a tick or an ID, as a JSON number. -- `report_version`: from `speakeasy.report.REPORT_VERSION`. -- `emulation_total_runtime`: profiler start/stop wall-clock duration. -- `timestamp`: profiler start time epoch seconds. -- `arch` (`x86` or `x64`), `path`, `sha256`, `size`, `file_type` (`exe`, `dll`, `driver` or `shellcode`), `image_base`: input metadata that the loader records. `path` is the host path of the input file. It is absent when the input came from bytes in memory. -- `errors`: top-level session errors recorded by profiler. The loader adds an `invalid_image` error, with the defect in `context_summary`, when the entry point or the import directory of the input PE is outside the image, or when the raw data of a section is past the end of the file. Emulation then continues. -- `strings`: present when string extraction is enabled (`analysis.strings=true`). -- `data`: payloads that the report stores once, keyed by SHA-256. The `data_ref` fields refer to its keys. -- `entry_points`: one per run (`Run` object) in execution order. +`report_version` is the value of `speakeasy.report.REPORT_VERSION`. `timestamp` is the start time of the profiler in epoch seconds, and `emulation_total_runtime` is the wall-clock time from the start to the stop of the profiler in seconds. The other top-level fields are: -Entry-point highlights: -- `type` and `name`: the kind of code that the run executes, and a detail for that kind. The JSON report leaves out `name` when the kind has no detail (`None` in Python). See the table below. -- `api_hash`: SHA-256 over unique API names, lowercased, first-seen order. -- `events`: built by profiler recorders (`record_api_event`, `record_file_access_event`, `record_process_event`, `record_registry_access_event`, `record_dns_event`, `record_http_event`, `record_network_event`, `record_exception_event`). -- `symbol_accesses`: symbol counters from memory tracing, present only if `analysis.memory_tracing=true`. -- `dynamic_code_segments`: populated when dynamically mapped code executes. -- `coverage`: executed instruction addresses as hex strings, present only if `analysis.coverage=true`. -- `dropped_files`: populated from filesystem manager fully-written files. -- `memory`: populated from run-end memory/module capture, with optional region payload refs when `snapshot_memory_regions=true`. -- `error`: present when the run ended abnormally. `type` is an identifier such as `invalid_read`, `unsupported_api` or `timeout`. The report schema lists all values. A Python exception in Speakeasy itself, such as a bug in an API handler, gives `internal_error`, with the exception text in `message` and the Python traceback in `traceback`. +- `arch` (`x86` or `x64`), `path`, `sha256`, `size`, `file_type` (`exe`, `dll`, `driver` or `shellcode`) and `image_base`: the input metadata that the loader records. `path` is the host path of the input file. If the input came from bytes in memory, the report has no `path`. +- `errors`: the errors of the session that do not belong to one run. If the input PE has a defect, for example an entry point outside the image, the loader adds an `invalid_image` error with the defect in `context_summary`, and emulation continues. `PeLoader` in `speakeasy/windows/loaders.py` has the checks. +- `strings`: the strings from the input file and from memory, only if `analysis.strings=true`. +- `data`: the payloads that the report stores once, keyed by SHA-256. Each `data_ref` field holds a key of `data`. +- `entry_points`: a list of the runs (`Run` objects), in the order that the runs started. + +Each entry point has these fields: + +- `api_hash`: the SHA-256 of the lowercase names of the APIs that the run called, each name once, in the order of the first call. +- `events`: the events of the run, in order. The `record_*_event` methods of `Profiler` add them. +- `symbol_accesses`: the read, write and execute counts of each symbol, only if `analysis.memory_tracing=true`. +- `dynamic_code_segments`: the memory regions with dynamic code, for example from `VirtualAlloc`, where code ran. +- `coverage`: the addresses of the instructions that ran, only if `analysis.coverage=true`. +- `dropped_files`: in user mode, the files that the guest wrote completely, up to the end of the run. +- `memory`: in user mode, the memory regions and the modules at the end of the run. If `snapshot_memory_regions=true`, each region also has a `data_ref` to its contents. + +If the run ended abnormally, the entry point has an `error`. Its `type` is an identifier such as `invalid_read`, `unsupported_api` or `timeout`. The report schema lists all values. A Python exception in Speakeasy itself, such as a bug in an API handler, gives `internal_error`, with the exception text in `message` and the Python traceback in `traceback`. + +The `type` of an entry point is the kind of code that the run executes, and `name` is a detail for that kind. If the kind has no detail (`None` in Python), the JSON report leaves out `name`. | `type` | `name` | |---|---| @@ -51,20 +50,16 @@ Entry-point highlights: | `driver_entry`, `driver_unload`, `driver_reinit` | none | | `irp` | IRP major function, for example `IRP_MJ_DEVICE_CONTROL` | -The schema also has the type `callback`, which Speakeasy does not set. +The schema also has the run type `callback` and the event types `mem_free` and `module_load`. Speakeasy does not set them. + +## API arguments and handles -An `api` event records its arguments in `args`, a list of objects with `name`, `type`, `value`, -and `display`. When the signature database declares the function and its prototype consumes -the same argument slots as the call, `args` has one entry per parameter, with the parameter -name in `name` and `display` rendered from the declared type. Strings are decoded, enums and -flags are symbolic, and struct pointers are expanded. Without a usable signature, `args` has -one entry per argument slot, `name` is absent, and `display` is the hex value. `type` tells a -consumer how to read `display`. +An `api` event records its arguments in `args`, a list of objects with `name`, `type`, `value` and `display`. If the signature database declares the function, and its prototype uses the same argument slots as the call, `args` has an entry per parameter. Then `name` is the parameter name, and Speakeasy renders `display` from the declared type. It decodes strings, shows enum and flag values as names and expands struct pointers. Without a usable signature, `args` has one entry for each argument slot, with no `name`, and `display` is the hex value. `type` tells you how to read `display`. | Type | `display` | |---|---| | `int` | hex integer | -| `ptr` | address in hex (also null, out-only, and unreadable pointers) | +| `ptr` | address in hex (also null, out-only and unreadable pointers) | | `handle` | handle value in hex | | `str` | decoded string, without quotes | | `bool` | `TRUE` or `FALSE` | @@ -76,34 +71,19 @@ consumer how to read `display`. | `float` | decimal number | | `text` | a value that a handler decoded itself | -`value` holds the raw value the caller passed, as a hex string, so a consumer can match a flag -or constant without parsing `display`. A parameter that uses two argument slots on x86 (a 64-bit integer, -for example) has the combined value. For a call served by a handler, a value the handler -decoded itself (such as the key path behind a registry handle) replaces the `display` of that -parameter, and `value` still holds what the caller passed. An entry that a variadic handler -adds, such as the formatted output of `printf`, has no `value`. - -A handle value refers to one object for the whole run. Speakeasy gives out the handles of all -object types from one counter that starts at `0x80` and adds 4, and it does not use the value -of a closed handle again. Thus a `handle` in an event identifies the file, key or other object -that the call used. - -Every event carries `pos`, the position of the actor that performed the operation, so an -event always sits in the timeline of the thread that caused it. Process-scoped events -(`mem_alloc`, `mem_write`, `mem_read`, `mem_protect`, `mem_free`, `thread_create`, -`thread_inject`, `process_create`) also carry the target process as a `pid` payload. That -payload is omitted for purely local operations, so a populated `pid` means the operation -crossed a process boundary. Thread events still report `tid` for the created or injected -thread, even for a local create. - -## Report fields to recipe flags - -- top-level `strings` is controlled by `--analysis-strings` / `--no-analysis-strings` ([recipe](cli-analysis-recipes.md#recipe-analysis-strings)). -- `entry_points[*].coverage` is populated by `--analysis-coverage` ([recipe](cli-analysis-recipes.md#recipe-analysis-coverage)). -- `entry_points[*].symbol_accesses` and `entry_points[*].memory.layout[*].accesses` are populated by `--analysis-memory-tracing` ([recipe](cli-analysis-recipes.md#recipe-memory-tracing)). -- `entry_points[*].memory.layout[*].data_ref` is populated by `--snapshot-memory-regions` ([recipe](cli-analysis-recipes.md#recipe-memory-snapshots)). -- `entry_points[*].dropped_files[*].data_ref` resolves through the top-level `data` store; large files keep metadata only. Dropped-files archive export is still available via `--dropped-files-path` ([recipe](cli-analysis-recipes.md#recipe-dropped-files)). -- report-wide binary payload deduplication lives in the top-level `data` mapping, keyed by SHA-256. +`value` is the raw value that the caller passed, as a hex string. With it, you can match a flag or a constant without parsing `display`. If a parameter uses two argument slots on x86, for example a 64-bit integer, `value` has the combined value. If a handler serves the call and decodes a parameter itself, for example the key path of a registry handle, the decoded text replaces the `display` of that parameter. `value` still has what the caller passed. If a variadic handler adds an entry, such as the formatted output of `printf`, that entry has no `value`. + +A handle value refers to one object for the whole emulation. Speakeasy gives out the handles of all object types from one counter, and it does not give out the value of a closed handle again. So a `handle` in an event identifies the file, key or other object that the call used. + +Each event has `pos`, the position of the thread that did the operation, so the event is in the timeline of that thread. The events about a process, its memory or its threads (`mem_alloc`, `mem_write`, `mem_read`, `mem_protect`, `mem_free`, `thread_create`, `thread_inject` and `process_create`) also have the `pid` of the target process. If the target is the process of that thread, the event has no `pid`. So a `pid` in these events means that the operation crossed a process boundary. `thread_create` and `thread_inject` have the `tid` of the new thread, also when the thread is in the same process. + +## Report fields and CLI flags + +- `--analysis-strings` and `--no-analysis-strings` control the top-level `strings` ([recipe](cli-analysis-recipes.md#recipe-analysis-strings)). +- `--analysis-coverage` fills `entry_points[*].coverage` ([recipe](cli-analysis-recipes.md#recipe-analysis-coverage)). +- `--analysis-memory-tracing` fills `entry_points[*].symbol_accesses` and `entry_points[*].memory.layout[*].accesses` ([recipe](cli-analysis-recipes.md#recipe-memory-tracing)). +- `--snapshot-memory-regions` fills `entry_points[*].memory.layout[*].data_ref` ([recipe](cli-analysis-recipes.md#recipe-memory-snapshots)). +- `entry_points[*].dropped_files[*].data_ref` holds a key of the top-level `data`. A file above `MAX_EMBEDDED_FILE_SIZE` (`speakeasy/artifacts.py`) has no `data_ref`. To also write the dropped files to an archive, use `--dropped-files-path` ([recipe](cli-analysis-recipes.md#recipe-dropped-files)). ## Annotated report schema example @@ -112,13 +92,13 @@ thread, even for a local create. // Report format version. "report_version": "6.0.0", - // Total wall-clock runtime in seconds. + // Wall-clock seconds from the last start of the run queue to the end. "emulation_total_runtime": 1.234, - // Emulation start timestamp (epoch seconds). + // Unix time in seconds of the last start of the run queue. "timestamp": 1760000000, - // Input metadata (may be null if unavailable in a custom flow). + // Input metadata. A key is absent when the value is unknown. "arch": "x86", "path": "/samples/payload.bin", "sha256": "1111111111111111111111111111111111111111111111111111111111111111", @@ -415,7 +395,7 @@ thread, even for a local create. } ], - // Symbol-level R/W/X counts, typically from memory tracing. + // Symbol-level R/W/X counts from memory tracing. "symbol_accesses": [ { "symbol": "kernel32.LoadLibraryA", diff --git a/doc/speakeasy2-walkthrough.md b/doc/speakeasy2-walkthrough.md index 3cf84b416..d1749c88e 100644 --- a/doc/speakeasy2-walkthrough.md +++ b/doc/speakeasy2-walkthrough.md @@ -1,24 +1,22 @@ -# Speakeasy 2 walkthrough outline +# What's new in Speakeasy 2.0 -Here are the major new features/changes in Speakeasy v2. +This page shows the main new features and changes in Speakeasy 2. For the changes that can break your code or scripts, see [Migrating to 2.0](migrating-to-2.0.md). -## Runtime baseline updates: Python 3.10 and Unicorn 2.1.4+ +## Python 3.10 and Unicorn 2.1.4 -Speakeasy now targets Python 3.10+ and Unicorn 2.1.4+. Unicorn 2.1.3 and below have significant issues and should not be used. +Speakeasy 2 needs Python 3.10 or later and Unicorn 2.1.4 or later. -## Unified loader and module model +## Loader and module model -The module system was reorganized around explicit Loader types, a LoadedImage data model, and a consistent RuntimeModule representation. This replaces multiple parallel module registries and consolidates image loading, import hookup, and bookkeeping into one path. The change makes behavior more predictable across PE files, shellcode, and API modules. +Speakeasy loads PE files, shellcode and the synthetic API modules through one code path. A loader (`PeLoader`, `ShellcodeLoader` or `ApiModuleLoader`) reads its input and returns a `LoadedImage`. A `LoadedImage` holds only data: the architecture, the preferred base and size, the regions to map, the imports and exports, the sections and the TLS data. A loader does not change the state of the emulator. -Under the hood, Loader is just the source adapter layer. Each loader type (PE, shellcode, API-generated module, etc.) takes its own input format and normalizes it into one common LoadedImage object. LoadedImage is intentionally “just data”: arch, base/size, regions to map, imports/exports, section metadata, TLS info, and visibility flags, with no emulator side effects yet. +`load_image` puts the `LoadedImage` into the emulator. It maps and writes the regions, binds the IAT slots to the API addresses, registers the exports of the module, sets the section protections and adds the module to the PEB. It returns a `RuntimeModule`. The rest of Speakeasy, for example the `modules` list and the PEB code, uses the `RuntimeModule` and does not need to know the kind of image. See [Architecture](architecture.md#loader). -The actual side effects happen in `load_image()`, which materializes that data into emulator state: map/write memory, bind IAT entries to mapped API entry addresses, register module exports in the session API registry, apply section protections, and publish symbols. The returned RuntimeModule is the stable runtime handle used everywhere else (`self.modules`, module lookups, PEB population), so the rest of Speakeasy can treat PE files, shellcode, and synthetic modules the same way. +## Host mounts with --volume -## Docker-style host mounts with --volume +`-V host_path:guest_path` (or `--volume`) maps a host file or directory into the emulated file system. For a directory, Speakeasy mounts all files below it. Speakeasy puts the volume entries before the `filesystem.files` entries of the config and uses the first entry that matches a path. So you can give a sample its dependencies without a change to the config file. See [Mounting host files with --volume](volumes.md). -A new -V/--volume CLI option maps host files or directories into the emulated Windows filesystem using host_path:guest_path syntax. Volume entries are expanded and injected with precedence so mounted data is resolved before default file mappings. This gives analysts a direct, repeatable way to stage sample dependencies without rewriting config files. - -Here is a CLI run using `-V` to mount a host file as `c:\myfile.txt` inside the emulator. I ran with `-v` so you can see where the mapping goes and what the sample reads. +This run mounts a host file as `c:\myfile.txt`. With `-v`, the log shows the mapping and the file reads of the sample. ```console speakeasy -t /tmp/se_vol_demo2/target/file_access_test_x86.exe --no-mp -v -V /tmp/se_vol_demo2/mount/myfile.txt:c:\myfile.txt @@ -33,31 +31,27 @@ INFO 0x40119b: 'ntdll.NtReadFile("c:\\myfile.txt", 0x0, 0x0, 0x0, 0x0, 0x12f INFO 0x40102e: 'api-ms-win-crt-stdio-l1-1-0.__stdio_common_vfprintf(0x0, 0x1, "File contained: 0x7266206f6c6c6568\\n")' -> 0x23 ``` -The key line is `emu_path=c:\myfile.txt host_path=...`: that is the `--volume` mapping being injected as a `full_path` rule. Then you can see the sample open and read `c:\myfile.txt`, and print back bytes from the mounted file, which confirms the mount is active during emulation. - -## Automatic target-directory mounting +The `emu_path=c:\myfile.txt host_path=...` line shows the volume as a `full_path` entry of the emulated file system. Then the sample opens and reads `c:\myfile.txt`, and prints bytes from the mounted file. -When a target sample is loaded from disk, Speakeasy now auto-mounts immediate sibling files from that directory into the emulated current directory. This covers common malware assumptions that companion payloads or config files sit next to the executable. The result is fewer setup failures caused by missing local artifacts. +## Automatic mount of the target directory -I ran a concrete CLI check with `file_access_test_x86.exe` and a sibling `myfile.txt`, using `--no-mp -v` and a config with `current_dir` set to `c:\`. Startup logs showed `Auto-mounted 3 file(s) ... into c:\` and debug mappings for each direct child file, including `c:\myfile.txt -> /tmp/.../myfile.txt`. Right after that, file manager logged the mounted entry in the `full_path` list, which confirms the auto-mount is applied before execution starts. +When you load a PE file from a host path, Speakeasy mounts the other files in the same host directory into the emulated current directory (`current_dir` in the config). Malware often expects payloads or config files next to the executable. Speakeasy does not mount subdirectories. It also does not mount a directory for shellcode or for a PE that you load from bytes. Speakeasy puts these entries before the volume and config entries, so they take priority when more than one entry matches a path. -The sample then opened and read `c:\myfile.txt` without any `--volume` flag (`file_open: c:\myfile.txt`, followed by successful `NtReadFile` and `File contained: ...` output). So the behavior is: this only triggers when you load by host path (not raw bytes), it mounts only immediate child files (not recursive), it maps into `current_dir`, and these entries are prepended so they win first-match file resolution. +For example, put `file_access_test_x86.exe` and `myfile.txt` in one directory, set `current_dir` to `c:\` in the config and run with `--no-mp -v`. At startup, the log shows an `Auto-mounted ... into c:\` line and a debug line for each file, such as `c:\myfile.txt -> /tmp/.../myfile.txt`. The file manager then lists the file in its `full_path` entries. The sample opens and reads `c:\myfile.txt` without a `--volume` flag: the log shows `file_open: c:\myfile.txt`, a successful `NtReadFile` and the `File contained: ...` output. -## AMD64 thread context get/set +## AMD64 thread context -AMD64 thread context retrieval and restoration now covers full register state instead of returning empty or unimplemented results. This includes general-purpose registers, RIP, segment registers, and EFLAGS. The change improves correctness for x64 samples that manipulate thread context during process replacement. +`GetThreadContext` and `SetThreadContext` support the x64 `CONTEXT`: the general purpose registers, `Rip`, the segment registers, `EFlags`, `MxCsr` and the debug registers. In 1.x these APIs returned an empty context or were not implemented for x64. x64 samples that do process hollowing read and write the thread context of the target, for example to set the new entry point. -When Speakeasy was first written, many malware samples were still 32-bits. These days 64-bit samples are much more prevalent. So we have better support for that now. +## Import hooks for injected and replaced child processes -## Hook imports for injected/replaced child processes +When a sample writes a PE into a child process with `WriteProcessMemory`, the loader does not see that PE, so 1.x did not bind its imports. In 2.0, when the sample resumes the modified child process with `ResumeThread`, the emulator reads the import directory of the PE from emulated memory. It binds the IAT slots to the same API addresses that the loader uses. Speakeasy then intercepts and reports the API calls of the hollowed payload. -Injected PEs written with WriteProcessMemory can bypass normal loader IAT patching, so imports were not always hooked. Speakeasy now parses the in-memory PE import directory and binds IAT slots to the same mapped API entries used by runtime resolution before resumed execution. This restores API interception and trace visibility for hollowed payloads. +## Memory snapshots in the report -## Config gate for memory byte capture +The `snapshot_memory_regions` config field (`--snapshot-memory-regions` on the command line) adds the memory content at the end of each run to the report. At the end of each run, for example a module entry, a TLS callback or an export, Speakeasy records the memory layout. When this field is set, each region gets a `data_ref`, except the stack and heap regions. The `data_ref` is a key of the top-level `data` mapping, which stores each payload one time, compressed with zlib and encoded with base64. -A new `snapshot_memory_regions` configuration option controls whether run-end memory content is included in reports. This capture happens at the end of each entry-point run (module entry, TLS callback, export run, etc.) when Speakeasy snapshots memory layout for that run. If enabled, each non-excluded region gets a `data_ref` field that points into the top-level `data` store, where payloads are saved once as `base64(zlib(raw_bytes))` keyed by SHA-256; module-backed regions are emitted as headers + per-section slices, while non-module maps are emitted as one region each. - -The same top-level `data` store is also used for dropped-file content and other captured binary previews, so repeated payloads deduplicate naturally across runs. +Speakeasy stores dropped files and other binary data in the same `data` mapping, so the report holds each payload one time, even when many runs or files contain it. ```json { @@ -69,13 +63,11 @@ The same top-level `data` store is also used for dropped-file content and other speakeasy -t sample.exe --snapshot-memory-regions -o report.json ``` -## Per-section memory layout and broader dump coverage - -Memory layout reporting now emits module headers and sections as distinct regions instead of a single module-wide block. Dump capture also includes non-excluded regions even when write counters are absent, which covers API-populated memory not tracked as direct writes. This makes reported layout and content closer to actual runtime state. +## Memory layout for each section -Here is what that looks like in a real report from `dll_test_x86.dll` with `analysis.memory_tracing=true` and `snapshot_memory_regions=true`. In `entry_points[0].memory.layout`, the sample module is split into `headers`, `.text`, `.rdata`, and `.data` records, each with its own protection, size, and counters instead of one merged module region. In this run, `.text` recorded only execution (`execs: 677`), `.rdata` recorded reads (`reads: 15`), and `.data` recorded both reads and writes (`reads: 30`, `writes: 26`). +The memory layout of a run has one region for the headers of a module and one region for each section, in place of one region for the whole module. With `snapshot_memory_regions`, Speakeasy captures each region that is not excluded, also when no write was counted. This includes memory that an API handler wrote, because the memory hooks do not count these writes. -The same snapshot also shows the broader capture rule: non-excluded regions get `data_ref` even if `writes == 0`, while excluded regions like stack still keep counters but skip raw bytes. So you end up with section-level activity you can reason about directly in JSON, plus deduplicated payloads in the top-level `data` store when capture is enabled. +This excerpt comes from a run of `dll_test_x86.dll` with `analysis.memory_tracing` and `snapshot_memory_regions` set. In `entry_points[0].memory.layout`, the module has separate `headers`, `.text`, `.rdata` and `.data` regions. Each region has its own protection, size and counters. In this run, code in `.text` executed, the sample read `.rdata` and it read and wrote `.data`. The stack region has counters but no `data_ref`, because Speakeasy does not capture the stack. ```json { @@ -129,13 +121,11 @@ The same snapshot also shows the broader capture rule: non-excluded regions get } ``` -## First-class sections and per-section memory protections +## Sections and section protections -Section metadata is now first-class on loaded images/modules across loader types, and PE section permissions are applied with per-section mem_protect calls. Permission mapping was expanded so NONE, EXEC, RX, and related combinations are handled explicitly. This improves protection fidelity and supports section-aware analysis logic. +`LoadedImage` and `RuntimeModule` have the sections of the image. When `load_image` loads a PE, it sets the protection of each section from the section characteristics with `mem_protect`. A section can have any combination of read, write and execute access, or no access. If two sections share a page, the page gets the protections of both. -There is no dedicated config toggle for this one: section metadata and per-section protections are applied automatically for PE images during load. To make it visible in the report, run with normal settings plus `analysis.memory_tracing=true`. - -Here is an actual report excerpt from `tests/bins/dll_test_x86.dll.xz` showing section-level protections in both `memory.modules[*].segments` and `memory.layout`: +You do not need a config setting for this. The report always has the sections in `memory.modules[*].segments` and the regions in `memory.layout`. This excerpt from `tests/bins/dll_test_x86.dll.xz` shows the section protections in both places: ```json { @@ -164,11 +154,9 @@ Here is an actual report excerpt from `tests/bins/dll_test_x86.dll.xz` showing s } ``` -## Per-section read/write/execute access tracking - -Memory hooks now track access counters at section granularity for module-backed memory, with map-level fallback for non-module regions. Report generation reads these counters from section-specific state instead of module-wide aggregates. Analysts can now see which sections were executed or modified, not just which module was touched. +## Access counts for each section -There is no separate toggle for this feature; it is part of `analysis.memory_tracing`. If you run with memory tracing enabled, Speakeasy installs code/read/write hooks and records access counts per section key `(module_base, section_rva)`, then writes them into each section entry in `entry_points[*].memory.layout[*].accesses`. +With `analysis.memory_tracing`, the memory hooks count reads, writes and executes for each section of a module. Outside modules, they count for each memory map. Speakeasy writes the counts to `entry_points[*].memory.layout[*].accesses`, so the report shows which sections of a module ran code or had writes. ```json { @@ -179,7 +167,7 @@ There is no separate toggle for this feature; it is part of `analysis.memory_tra } ``` -A concrete run against `tests/bins/dll_test_x86.dll.xz` produced section-level counters like these: `.text` had execution only (`execs: 677`), `.rdata` had reads only (`reads: 15`), and `.data` had reads and writes (`reads: 30`, `writes: 26`). That is the key behavior change: different sections of one module now show different activity profiles instead of one module-wide aggregate. +In a run of `tests/bins/dll_test_x86.dll.xz`, each section of the module has different counts: ```json [ @@ -201,11 +189,11 @@ A concrete run against `tests/bins/dll_test_x86.dll.xz` produced section-level c ] ``` -## Unique TID/PID attribution per entry-point run +## Thread and process ID for each run -Entry-point records include pid/tid metadata. TLS callbacks and the module entry run on the main thread, so the entry sees the TLS slots and static TLS data that the callbacks set, as on Windows. Each export run gets its own thread, so export calls can be separated by execution context in timelines of multi-run samples. +Each entry in `entry_points` has a `pid` and a `tid`. The TLS callbacks and the module entry run on the main thread, as on Windows, so the module entry sees the TLS slots and static TLS data that the callbacks set. Speakeasy runs each export on a new thread. So you can tell the export calls apart in the timeline of a sample with many runs. -Here is a real `entry_points` excerpt from a run of `dll_test_x86.dll` with `all_entrypoints=True`. The important part is that each run has its own `tid`, so module entry and each export call can be separated cleanly in timelines and downstream tooling. +This excerpt comes from a run of `dll_test_x86.dll` with `all_entrypoints=True`: ```json [ @@ -233,19 +221,19 @@ Here is a real `entry_points` excerpt from a run of `dll_test_x86.dll` with `all ] ``` -How to read this: `tid` changes on each queued run (`1076 -> 1200 -> 1212`), while the export runs stay in the same process (`pid=1056`). +Each run has a different `tid` (`1076`, `1200`, `1212`), and all runs are in process `1056`. -## Interactive debugging with --gdb +## Debugging with --gdb -Speakeasy provides a native GDB remote stub and can pause before the first instruction until a client connects. The flow is available from both CLI and library usage, with explicit port selection. This enables interactive pausing, stepping, breakpoints, watchpoints, and memory/register inspection during emulation. +`--gdb` starts a GDB remote stub. Speakeasy stops before the first instruction and waits for a client. Then you can pause, step, set breakpoints and watchpoints, and read memory and registers. In Python, the `gdb_port` argument of `Speakeasy` starts the same stub. -Quick way to use it: in terminal 1 run `speakeasy -t sample.dll --gdb --gdb-port 1234` and wait for the listener message; Speakeasy will force `--no-mp` automatically so the stub stays attached to the same Unicorn instance. In terminal 2 run `gdb-multiarch`, then `set architecture i386` (or `i386:x86-64` for 64-bit), and `target remote localhost:1234`; from there standard commands like `info registers`, `x/10i $pc`, `break *0x...`, `stepi`, and `continue` work as expected. +In one terminal, run `speakeasy -t sample.dll --gdb --gdb-port 1234` and wait for the listener message. `--gdb` turns on `--no-mp`, because the stub must stay attached to the Unicorn engine in the same process. In a second terminal, start `gdb-multiarch`, run `set architecture i386` (or `i386:x86-64` for x64) and then `target remote localhost:1234`. The usual commands work, such as `info registers`, `x/10i $pc`, `break *0x...`, `stepi` and `continue`. -For the full reference and caveats, use [doc/gdb.md](gdb.md). That page covers the practical details this summary skips, including IDA Remote GDB setup, one-session-per-process limits, and how breakpoints behave across multi-run flows like DllMain plus exports. +[GDB debugging](gdb.md) tells you how to connect IDA with its Remote GDB debugger. It also gives the limits, for example one client for each Speakeasy instance and breakpoints that stay set across runs such as DllMain and the exports. -## Timeout enforcement across multi-run and parent supervision +## Time limits across runs -Timeout handling was tightened so each run has its own active-time budget and `max_total_time` caps active time across chained entry-point runs and parent-process control logic. This closes cases where retry loops or queue waits could effectively outlive the requested timeout. Users should now see more predictable stop behavior on long or stalled analyses. +`max_run_time` limits the active time of each run. `max_total_time` limits the active time of all runs of one call, such as `run_module`. Retry loops and waits in the run queue cannot make emulation run longer than these limits. The CLI also stops the emulation process after the larger of the two limits. See [CLI execution controls](cli-execution-controls.md). ## Related docs @@ -254,7 +242,6 @@ Timeout handling was tightened so each run has its own active-time budget and `m - [CLI reference](cli-reference.md) - [Configuration walkthrough](configuration.md) - [Report walkthrough](reporting.md) -- [GDB debugging reference](gdb.md) +- [Memory management](memory.md) +- [Unified Windows API addresses](unified-api-addresses.md) - [Help and troubleshooting](help.md) - -See [Unified Windows API addresses](unified-api-addresses.md) for export catalogs, private dispatch and debugger behavior. diff --git a/doc/unified-api-addresses.md b/doc/unified-api-addresses.md index 71aef52a8..23cbdd2d8 100644 --- a/doc/unified-api-addresses.md +++ b/doc/unified-api-addresses.md @@ -1,8 +1,8 @@ # Unified Windows API addresses -Speakeasy gives each synthesized API function a stable, readable executable address in its owning module. The export table, bound imports, `GetProcAddress`, `LdrGetProcedureAddress`, and kernel routine lookup use the same session registry. Registering a handler or user hook changes dispatch behavior without changing an existing address. +Speakeasy gives each API function of a synthetic module one fixed address in that module. The guest can read and execute the code at this address. The export table, the bound imports, `GetProcAddress`, `LdrGetProcedureAddress` and the kernel routine lookup all get the address from one API registry (`ApiRegistry` in `speakeasy/windows/api_registry.py`). If you add a handler or a user hook, the emulator runs different code for the call, but the address does not change. -This makes the returned pointer a useful debugger breakpoint and symbol address. A guest can read or patch the entry; its bytes execute before Python handles the call. Memory tracing observes execution and never decides whether to intercept a function. +So you can use the address of an API as a debugger breakpoint and as a symbol address. The guest can read or patch the code at the address, and this code runs before the Python handler. Memory tracing only counts accesses. It does not change which calls go to Python. ```mermaid flowchart LR @@ -17,85 +17,85 @@ flowchart LR Scheduler --> Handler[Handler / user hook / supported ABI] ``` -Forwarder strings follow PE forwarding rules to the destination entry. Dynamic-only requests enter through the registry and resolver; they deliberately have no export-table entry. +A forwarder string leads to the entry in the destination module, as in a PE. A dynamic entry is an entry that the emulator adds at run time, when the guest asks for a function that the export table does not have. The registry and `GetProcAddress` find dynamic entries, but the export table does not get them. -## Modules and export surfaces +## Modules and export tables -`ApiModuleLoader` takes the names and ordinals of a module from its physical export manifest when one exists. Otherwise it combines exact DLL declarations from the signature database with explicit function and data handlers. Enumeration is independent of generic ABI support: an unsupported declaration still appears in the EAT and has a breakpointable entry. Architecture-ineligible declarations are excluded. The loader does not manufacture A/W variants. +`ApiModuleLoader` takes the names and ordinals of a module from its export manifest, if the module has one. Otherwise it uses the declarations of the DLL in the signature database and the function and data handlers of the DLL. The export table also has the declarations that the emulator cannot call from the signature, and each of them has an entry where you can set a breakpoint. The loader leaves out the declarations for the other architecture. It does not add `A` or `W` variants. -The generated PE has a read-only header and export directory, executable `.text` and `.dyn` sections, and writable `.data`. Name pointers are sorted, name ordinals index the EAT correctly, and explicit sparse ordinals retain real holes. Code and data are different export kinds. An empty placeholder is still a mapped PE with valid headers and a reserved dynamic arena. +The generated PE has a read-only header and export directory, executable `.text` and `.dyn` sections and a writable `.data` section. The name pointers are sorted, the name ordinals point to the correct EAT entries, and explicit ordinals keep the gaps between them. Code exports and data exports are different kinds. A placeholder for an unknown module is a mapped PE with valid headers and a `.dyn` section, but no exports. -Each file `speakeasy/resources/win32/exports//.json` records the export directory of one real Windows binary: the directory name, ordinals, names, forwarder strings and code or data kind, with the file version and SHA-256 of the source binary. The bundled manifests come from Windows Server 2025 (10.0.26100), with x64 modules from System32 and x86 modules from SysWOW64. `speakeasy/resources/win32/exports/README.md` describes their source and format. When a manifest exists, the generated export directory takes its name from the manifest, for example `KERNEL32.dll`, and the manifest names and ordinals define the EAT, including ordinal-only entries, so an ordinal import reaches the same function as on that build. Handler names that the manifest does not list get ordinals after the highest physical ordinal. A handler name whose A or W variant is in the manifest is not exported, because it exists only for dispatch. A manifest data export gets 0x100 zeroed writable bytes unless a data handler initializes it. Manifest forwarders are recorded but not applied: a forwarded name is a local entry. +Each file `speakeasy/resources/win32/exports//.json` records the export directory of one real Windows binary: the directory name, the ordinals, the names, the forwarder strings and the code or data kind. It also records the file version and the SHA-256 of the binary. `speakeasy/resources/win32/exports/README.md` tells where the binaries come from and gives the format. When a module has a manifest, the generated export directory gets its name from the manifest, for example `KERNEL32.dll`. The manifest names and ordinals define the EAT, with the ordinal-only entries, so an import by ordinal gets the same function as on the Windows build of the manifest. A handler name that is not in the manifest gets an ordinal after the highest manifest ordinal. If the manifest has the `A` or `W` variant of a handler name, the emulator does not export the handler name, because it uses that name only to find the handler. A data export of the manifest gets 0x100 bytes of writable memory set to zero, unless a data handler sets the value. The emulator records the manifest forwarders but does not follow them: a forwarded name gets an entry in the module itself. -Kernel modules and user modules without a manifest use the signature catalog. Win32metadata records import-library declarations, which omit undocumented physical exports, and PHNT declarations do not supply ordinals or forwarders. For these modules, generated ordinals are deterministic emulator identifiers unless a handler supplies an explicit ordinal. CRT `_fmode`, `_commode`, `__initenv`, `_iob` and `_sys_nerr` are explicit writable data exports; their pointer accessor functions, such as `__iob_func` and `__acrt_iob_func`, return the same shared storage. Known PHNT architecture restrictions are curated; remaining unqualified declarations have inferred availability. +Kernel modules and the user modules without a manifest use the signature database. win32metadata records the declarations of the import libraries, so it does not have the undocumented exports. phnt does not give ordinals or forwarders. For these modules, the emulator gives the ordinals in a fixed order, unless a handler gives an explicit ordinal. The CRT variables `_fmode`, `_commode`, `__initenv`, `_iob` and `_sys_nerr` are writable data exports. Their accessor functions, such as `__iob_func` and `__acrt_iob_func`, return the same memory. The phnt generator does not read the architecture guards of the headers. `FUNCTION_ARCHITECTURES` in `scripts/gen_phnt_signatures.py` lists the functions that only one architecture has, and the other phnt declarations are for both architectures. -`PeLoader` preserves native export RVAs, ordinals, data, aliases and forwarder strings. Its exports execute guest code, even when a same-named Python handler exists. Native files selected through configured module paths use this guest policy; there is no automatic mixture of native instructions and handler interception. A guest image cannot acquire invented exports. Rebase requests require relocation data, and architecture mismatches fail before mapping. +`PeLoader` keeps the export RVAs, ordinals, data exports, aliases and forwarder strings of a real PE. The exports of a real PE run its own code, also when a handler with the same name exists. A DLL from the module directory of the config is a real PE, so the emulator runs its code and not the handlers. The emulator does not add dynamic entries to a real PE. To load a real PE at a different base, the PE must have relocation data. The loader rejects a PE for the wrong architecture before it maps it. ## Public entries and private dispatch -Each synthesized function occupies a 32-byte slot. The first 16 bytes are NOP padding for patches, and the public entry starts at offset 16. When the original entry reaches its jump, the stack, registers and flags are as they were at the entry: +Each function of a synthetic module has a 32-byte slot. The first 16 bytes are NOPs, so that a patch has space, and the public entry starts at offset 16. When the entry code gets to its jump, the stack, the registers and the flags are the same as at the entry: | Architecture | Public bytes | | --- | --- | | x86 | `8B FF 55 8B EC` (`mov edi,edi; push ebp; mov ebp,esp`), `5D` (`pop ebp`), then `E9 rel32` | | x64 | `66 90`, followed by `FF 25 00 00 00 00` and an embedded 64-bit target | -The jump goes to a unique, permanently unmapped private token. The private reservation is 1 MiB with 16-byte token spacing. It is separate from the SEH and other control addresses. Allocation is monotonic, including after a failed load, so retired tokens cannot become a different API. +The jump goes to a token, which is a unique address that the emulator never maps. The tokens are in a reserved range of 1 MiB, 16 bytes apart. This range is not the range of the SEH and the other return addresses of the emulator. The emulator never uses a token again, also after a failed load, so a token cannot change to a different API. -The invalid-memory dispatcher recognizes private controls before user fault hooks. An expected API fetch stops Unicorn and returns control to the outer execution loop. Python dispatch then runs outside the native callback. This avoids mapping fake API pages and makes breakpoint, step, interrupt and library-notification ordering explicit. Unregistered targets or data accesses in the reservation suspend native execution and deliver an ordinary access violation to guest SEH outside the callback. Unhandled accesses produce `invalid_read`, `invalid_write` or `invalid_fetch`. SEH can redirect PC or repair registers, but exception recovery never maps the reservation. The first five x86 bytes are the Windows hot-patch prologue. Inline hooks that match this prologue copy it into a trampoline and resume at `pop ebp`, which restores the stack and EBP before the jump. +The emulator checks for API tokens before it calls the user hooks for invalid memory. When the guest fetches a token, the emulator stops Unicorn and the run loop continues. Then the Python handler runs outside the Unicorn callback. So the emulator does not map pages for the API code, and the order of breakpoints, steps, interrupts and loader notifications is clear. If the guest jumps to an address in the range that is not a token, or reads or writes the range, the emulator stops Unicorn and gives an access violation to the guest SEH, outside the callback. If no handler takes it, the run ends with `invalid_read`, `invalid_write` or `invalid_fetch`. A guest handler can change PC or registers, but the emulator never maps the range. The first five x86 bytes are the hot-patch prologue of Windows. An inline hook that knows this prologue copies it to a trampoline and continues at `pop ebp`, which restores the stack and EBP before the jump. -A host or debugger patch invalidates Unicorn's translated code cache. Patching a public entry to return or jump elsewhere executes those guest bytes and can bypass handler dispatch. A suspended call retains its entry bytes, SP and return slot; editing its frame or public code abandons the retained dispatch and executes the revised entry. +When the host or a debugger patches guest code, Unicorn discards its translated code. If a patch makes the public entry return or jump to a different address, the guest runs the patched code, and the handler can be skipped. A suspended call keeps its entry bytes, SP and return slot. If you change its frame or the entry code, the emulator drops the suspended call and runs the changed entry code. -Real forwarders resolve through destination modules, names or ordinals, with cycle and depth checks. Missing destinations are not silently synthesized. Synthetic forwarder specifications retain real EAT forwarder strings rather than overwriting them with code. A forwarder may resolve to guest code or shared data storage. +A real forwarder resolves through the destination module and the name or ordinal. The emulator checks for loops and for chains that are too long. It does not make an entry for a destination that does not exist. When a synthetic module has a forwarder of a real DLL, the EAT keeps the forwarder string, and the emulator does not write code over it. A forwarder can lead to guest code or to shared data. -## Dynamic-only requests +## Dynamic entries -Each synthetic image reserves a 64 KiB `.dyn` arena, enough for 2,048 additional entries. Name and ordinal requests have separate keys. Repeated resolution reuses the same address. New entries never change the module's EAT, `SizeOfImage`, section layout or existing pointers. +Each synthetic image has a `.dyn` section of 64 KiB, with space for 2,048 dynamic entries. A name and an ordinal give different entries. When the guest asks for the same function again, it gets the same address. A new dynamic entry does not change the EAT of the module, its `SizeOfImage`, its section layout or the existing addresses. -An empty unknown placeholder can resolve dynamic-only functions. For a populated known surface, a procedure lookup for a name that is not exported fails unless the name has a supported declaration for that DLL, `functions_always_exist` is set, or an explicit user hook permits synthesis. A declared name therefore resolves even when the physical manifest lacks it, because other Windows builds export it. A failed `GetProcAddress` sets `ERROR_PROC_NOT_FOUND`. Static/injected import binding and the explicit `get_proc` interface permit dynamic placeholders for missing synthetic exports, preserving a meaningful address and later unsupported-call diagnostic. Guest PEs always remain strict. Capacity exhaustion or guest modifications to arena bytes/protections fail allocation without moving existing entries. +A placeholder for an unknown module can give dynamic entries. For a module with known exports, a lookup of a name that the module does not export fails, except when the signature database declares the name for that DLL, `functions_always_exist` is set or a user hook exists for the name. So a declared name resolves also when the manifest does not have it, because other Windows builds export it. A failed `GetProcAddress` sets `ERROR_PROC_NOT_FOUND`. The import binding of the loader and the `get_proc` interface can make dynamic entries for missing exports of a synthetic module. So the import gets an address, and a call to it later gives an unsupported API error that names the function. The emulator never adds entries to a real PE. If the `.dyn` section is full, or the guest changed its bytes or protection, the lookup fails, and the existing entries do not move. -`functions_always_exist` permits dynamic resolution. It also enables the fallback for completely undeclared functions: treat the call as a 4-argument stdcall function, log those arguments, and return 1. This guess lets execution continue. On x86 it corrupts the stack when the real argument count differs. Known declarations rejected by the execution gate never enter the unknown-function fallback. Other unsupported calls log a diagnostic and stop during ordinary dispatch. Under GDB, preflight stops before recording an API call and the original frame remains suspended so a caller can inspect it, add a hook, redirect execution, or patch the entry. +`functions_always_exist` lets the emulator make dynamic entries. It also lets the emulator call a function that has no declaration: the emulator calls it as a stdcall function with 4 arguments, logs these arguments and returns 1. This is a guess, so that execution can continue. On x86 it corrupts the stack when the real argument count is different. A function that has a declaration that the emulator cannot call does not get this guess. A call to it, or to another unsupported function, logs an error and stops the run. Under GDB, the debugger stops before the emulator records the call, and the emulator holds the call. You can then examine it, add a hook, change PC or patch the entry. ## Signature and handler dispatch -Generic dispatch uses an exact module/name/architecture declaration with source precedence. Permissive cross-DLL signature lookup remains available for formatting arguments of implemented handlers, but cannot select an unknown function's execution ABI. +To call a function without a handler, the emulator uses the declaration for the same module, name and architecture. If more than one database declares it, the emulator uses the database with the highest priority. To show the arguments of a handler call, the emulator can also use the declaration of the name in a different DLL, but it never uses that declaration to call a function without a handler. -When several export names share one address, dispatch and telemetry use the first registered name, because an address alone does not identify the name a caller used to find it. Hooks match that name. +When more than one export name has the same address, the emulator uses the name that it registered first for the call and for the report, because the address does not tell which name the caller used. Hooks match that name. -The generic execution gate supports scalar/pointer cdecl and stdcall declarations it can transport correctly, with the corresponding Win64 integer/pointer ABI on x64. A by-value aggregate argument occupies its stack slots on x86 and one slot on x64. The gate rejects skipped declarations, variadic calls, unsupported conventions, floating-point transport, aggregate returns, and x86 64-bit returns. These functions remain visible and can have explicit handlers or hooks. Exceptions raised by providers during dispatch lookup are reported with the requested DLL and function rather than silently selecting another source's ABI. Bundled archive loading treats unreadable or malformed archives as unavailable sources. +The emulator can call a declaration without a handler when all arguments are integers or pointers and the convention is cdecl or stdcall, or the Win64 convention on x64. An argument that is a struct passed by value uses its stack slots on x86 and one slot on x64. The emulator cannot call skipped declarations, variadic functions, other conventions, floating-point arguments, struct return values or 64-bit return values on x86. These functions are still in the export table, and you can add handlers or hooks for them. If a signature database raises an exception during the lookup, the emulator reports the DLL and function of the call, and does not use the declaration of a different database. If a bundled database file cannot be read or is not valid, the emulator skips it. -Dispatch records API events against the originating run. A handler that changes PC, SP, return address or run ownership is not automatically returned a second time. Stack-transforming handlers perform their own return. A handler that changes SP while leaving PC on the dispatch trap, without scheduling a callback or lifecycle transfer, ends the run once with `api_handler_did_not_return`; it is not redispatched in a loop. Typed callback frames preserve the originating API's return slot, stack, calling convention and result across queued and nested guest callbacks. API telemetry reflects the result after a deferred initialization failure. +The emulator records each API event on the run that made the call. If a handler changes PC, SP, the return address or the current run, the emulator does not return from the call again. A handler that changes the stack does its own return. If a handler changes SP but leaves PC on the token, and it does not start a callback or another run, the run ends with `api_handler_did_not_return`. The emulator does not call the handler again. A callback frame keeps the return slot, the stack, the calling convention and the result of the API, also across nested guest callbacks and callbacks in the queue. If a DLL initializer fails later, the API event shows the final result. ## Loader and process integration -An image owns one contiguous mapped span. Regions are validated and written inside that span, exports are registered, imports are bound, and section permissions are applied before publication. Loader graph failures remove new module mappings, registry records, import bindings and PEB attachments while preserving surviving addresses and entries. Public module-change notifications occur after a coherent load graph is available. Listener exceptions are logged independently and do not undo committed loads or prevent other listeners from running. +An image uses one continuous range of memory. Before the loader adds the module to the module list, it checks the regions and writes them in that range, registers the exports, binds the imports and sets the section permissions. If a module or one of its dependencies fails to load, the loader removes the new mappings, registry records, import bindings and PEB entries. It keeps the addresses and entries that were there before. The emulator sends the module change notifications after the complete load. If a listener raises an exception, the emulator logs it, keeps the load and calls the other listeners. -`PeLoader` defaults to lenient parsing of optional import/export inventories: malformed delay directories, invalid export entries, and malformed forwarders produce warnings and are omitted from the inventory. Non-ASCII DLL names use lossless Latin-1 decoding. Guest header bytes remain available for inspection. Image mapping bounds, architecture checks and relocation requirements remain mandatory. Mandatory checks reject impossible mappings (including section extents beyond `SizeOfImage`), inconsistent machine/optional-header architectures and rebasing without relocations; dropping optional inventories cannot make those mappings coherent. +By default, `PeLoader` does not reject a PE for errors in the import and export data. It logs a warning and leaves out a malformed delay import directory, an export entry that is not valid or a malformed forwarder. It decodes DLL names that are not ASCII as Latin-1, so that no byte is lost. The guest can still read the header bytes. The loader always checks that the image fits its range, that the architecture is correct and that the image has relocation data if it must move. It rejects a section that goes past `SizeOfImage`, a machine type that does not agree with the optional header, and a move without relocations. -Static imports retain their original thunk/IAT positions when pefile filters a malformed function name. Skipped slots remain unchanged; valid later entries bind to their own slots. The inventory honors pefile's validated IAT fallback for an empty/unreadable OriginalFirstThunk, rejects reserved ordinal bits, and bounds raw nonzero-slot work across descriptors while allowing terminating NUL lookahead. Synthetic module construction retains a non-ASCII module's actual name, path and registry identity, using an ASCII label only inside its generated PE export directory. +When pefile drops a malformed function name, the other static imports keep their thunk and IAT positions. The skipped slots do not change, and each valid entry after them binds to its own slot. If OriginalFirstThunk is empty or cannot be read, the loader uses the IAT, as pefile does. The loader rejects ordinals with reserved bits set. It limits the number of import slots that it reads for each image, but it can always read the terminating NUL. A synthetic module for a name that is not ASCII keeps its real name, path and registry key. It uses an ASCII name only in its generated export directory. -Ordinary and delay imports share the import inventory and binder. Native delay imports are bound eagerly; legacy VA-form descriptors are handled when rebasing. Injected import repair validates architecture and bounds descriptor, thunk and string reads by the mapped image. Injected repair resolves and validates all import slots before committing writes. Invalid slots keep their original bytes. Binding bookkeeping is separate from callable address ownership. +Normal imports and delay imports use the same import list and binder. The loader binds the delay imports of a real PE at load time. It also rebases old delay import descriptors that hold virtual addresses. When the guest maps a PE itself, the emulator checks the architecture, and it reads the descriptors, thunks and strings only inside the mapped image. It checks all import slots before it writes any of them, and it does not change a slot that is not valid. The loader keeps the import bindings apart from the API registry, which owns the addresses. -Each process has its own loader entries. List heads are real sentinels with reciprocal forward/back links, including empty lists. The main image is first in load/memory order and excluded from initialization order. Core/default DLLs are attached during PEB allocation; subsequent reinitialization preserves that process's actual membership. Cached loads attach only to the selected process, and kernel/invisible modules stay out of user PEBs. Allocating another PEB does not replace the running thread's FS/GS PEB pointer. +Each process has its own loader entries. The list heads are real sentinel entries with forward and back links to each other, also when a list is empty. The main image is first in the load order and the memory order, and it is not in the initialization order. The emulator adds the core and default DLLs when it allocates the PEB, and keeps the modules of each process when it initializes the PEB again. A module that is already loaded goes only into the PEB of the selected process. Kernel modules and hidden modules are not in the PEB of a user process. When the emulator allocates another PEB, the PEB pointer in FS or GS of the running thread does not change. -Native DLL dependencies initialize under the outer scheduler, in completed dependency-load order, with TLS callbacks before DllMain. Only modules attached to the current process are eligible, with initialization state tracked per process. Runtime library loading uses deferred guest callback frames. Reentrant loads see initialization state and do not schedule the same module twice. Runtime failed DllMain returns roll back new attachments and unowned new mappings while preserving another process's surviving cached module; `LdrLoadDll` also clears its output handle and returns `STATUS_DLL_INIT_FAILED`. Synthetic APIs have no invented initializer. +The run loop initializes the dependencies of a real DLL in the order in which they completed the load, with the TLS callbacks before DllMain. It initializes only the modules of the current process, and it keeps the initialization state for each process. `LoadLibrary` runs the initializers as guest callbacks. A nested load sees the initialization state and does not queue the same module two times. When DllMain fails during `LoadLibrary`, the emulator removes the new PEB entries and the new mappings that no other process uses. It keeps a module that another process has. `LdrLoadDll` also clears its output handle and returns `STATUS_DLL_INIT_FAILED`. Synthetic API modules have no initializer. -Speakeasy uses a shared emulated address space. Per-process PEB membership is not virtual-address-space isolation; module data can remain shared. Complete Windows reference counting, API-set schema/build fidelity, native TLS semantics, special `LoadLibraryEx` mapping modes and DLL unload notifications are not modeled. +All processes share one emulated address space. Each process has its own PEB module list, but the address spaces are not separate, and processes can share module data. The emulator does not model the full Windows reference counting, the API set schema of each Windows build, the native TLS behavior, the special mapping modes of `LoadLibraryEx` or the notifications when a DLL unloads. ## Debugger limits and symbols -A resume or step is one logical action across private yields. Stepping the final trampoline jump completes the intercepted call or guest transfer without executing a caller instruction. Pending watchpoints or interrupts stop before dispatch. Memory watchpoints stop at a completed instruction boundary so resuming does not repeat a guest write. +A resume or a step is one action, also when the run passes through API calls. When you step over the last jump of a trampoline, the intercepted call or the guest jump completes, and no instruction of the caller runs. A pending watchpoint or interrupt stops the debugger before the handler runs. A memory watchpoint stops after the instruction completes, so a resume does not do the guest write again. -Instruction limits are per run and persist across debugger actions and API yields. Synthesized guest instructions count; Python handling does not. Active execution time is cumulative within one run across actions and API yields, including Python handling, and excludes time paused in the debugger. Every queued entry point, fresh shellcode run, and public `Speakeasy.call()` starts a new per-run timeout budget. Separately, `max_total_time` caps active execution across all queued runs and nested starts in one public `run_module`, `run_shellcode`, or `call` invocation (default 0, which disables it). Direct emulator `start()` creates the same scope. A fresh public invocation receives a fresh aggregate budget; debugger pauses are excluded. Aggregate exhaustion stops the remaining queue with `max_total_time`. Native execution uses the earlier of the two deadlines. Python handlers are measured cooperatively after returning, so a blocking host hook can overshoot the deadline. Initial loading and preparation outside execution are excluded; runtime loads and scheduler work inside active execution consume the budget. `max_runs` caps queue length, while the aggregate cap bounds successive runs. Timeout/instruction exhaustion produces a typed stop rather than passing zero to Unicorn as an unlimited budget. Completed callback/run-return controls are finalized even at an exact instruction boundary. Cancellation is checked before restarting native execution. Capped execution uses exact Python instruction accounting across API yields, with Unicorn's native count also bounding each invocation. This adds overhead when `max_instructions` is set. Unicorn has no interface that reports the consumed instruction count, and block-count estimates miscount REP iterations and partial blocks. [Unicorn 2.1.4 maintains and resets that counter internally](https://github.com/unicorn-engine/unicorn/blob/2.1.4/uc.c#L987-L1024). +The instruction limit is for each run, and it continues across debugger actions and API calls. Guest instructions count, and Python code does not. The active time of a run includes the Python handlers, also across debugger actions and API calls, but not the time while the debugger is paused. Each entry point in the queue, each new shellcode run and each call of `Speakeasy.call()` starts with a new time limit. `max_total_time` limits the active time of all runs in the queue and all nested runs in one call of `run_module`, `run_shellcode` or `call`. 0 disables it. A direct call of the emulator `start()` uses the same scope. Each new public call starts a new total time, and the time while the debugger is paused does not count. When the total time runs out, the emulator stops the runs in the queue with the error `max_total_time`. Unicorn stops at the earlier of the two limits. The emulator measures the time of a Python handler after it returns, so a host hook that blocks can go past the limit. The initial load and setup do not count, but loads and run loop work during execution count. `max_runs` limits the length of the queue, and the total time limits the runs that follow each other. When a run has no time or instructions left, the emulator stops it with the limit error. It does not start Unicorn with a limit of zero, because Unicorn reads zero as no limit. If a run or a callback returns at the instruction where a limit is reached, the emulator completes the return. The emulator checks for cancellation before each start of Unicorn. When `max_instructions` is set, the emulator counts each instruction in Python across API calls, and it also gives the remaining count to Unicorn for each start. This makes the run slower. Unicorn has no interface that gives the number of instructions that it ran, and a count of blocks is wrong for REP iterations and partial blocks. [Unicorn 2.1.4 keeps and resets that counter inside](https://github.com/unicorn-engine/unicorn/blob/2.1.4/uc.c#L987-L1024). -`Speakeasy.get_symbols()` returns a `{address: (dll, name)}` snapshot that combines registry exports and dynamic entries with auxiliary kernel symbols. Registry ownership takes precedence at a duplicate address. `get_symbol_from_address(address)` resolves public entries, trampoline interiors and exact auxiliary kernel symbols. `Speakeasy.get_api_symbols()` returns a snapshot of mapped function/data labels, including dynamic-only entries and internal callback entries. It excludes private tokens and EAT forwarder strings. Take a new snapshot after resolving additional modules or functions; PE exports alone intentionally omit dynamic-only symbols. +`Speakeasy.get_symbols()` returns a `{address: (dll, name)}` snapshot with the registry exports, the dynamic entries and the other kernel symbols. If two symbols have the same address, the registry entry wins. `get_symbol_from_address(address)` finds public entries, addresses inside a trampoline and exact kernel symbols. `Speakeasy.get_api_symbols()` returns a snapshot of the function and data labels, with the dynamic entries and the internal callback entries. It does not include the tokens or the EAT forwarder strings. Take a new snapshot after the emulator resolves more modules or functions. The PE exports do not have the dynamic entries. -Genuine memory faults remain terminal for the current run after debugger inspection. Register or PC edits at those fault stops do not make the run resumable. Analysis recovers from execution of non-executable anonymous and guest-module memory, including packers that execute decrypted `.data` bytes. This is a coverage policy rather than a complete Windows DEP policy model; it applies independently of architecture and NXCOMPAT. Recovery grants execute permission to the faulting guest PE page after Unicorn unwinds, retaining its read/write permissions. This avoids repeated protected-fetch faults without changing protections inside a native callback. Debugger fault stops and synthetic API section permissions remain authoritative. Fault completion waits for native execution to unwind so errors cannot be attributed to the next queued run. The resumable unsupported-API preflight described above is a separate path. +Under the debugger, a memory fault stops the debugger, and the run ends when you resume. A change of the registers or PC at this stop does not let the run continue. Without a debugger, when the guest runs code in memory without execute permission, the emulator continues. This occurs for allocated memory and for the pages of a real PE, for example when a packer runs decrypted bytes in `.data`. This is not a full model of the Windows DEP rules, and it does not depend on the architecture or NXCOMPAT. For a page of a real PE, the emulator adds execute permission to the page after Unicorn stops, and keeps the read and write permissions. So the same fetch does not fault again, and the permission change occurs outside the Unicorn callback. A debugger stop at a fault and the permissions of the synthetic API sections always apply. The emulator waits for Unicorn to stop before it completes a fault, so it does not give the error to the next run in the queue. The stop at an unsupported API under GDB, described above, is a different path. -The SEH repeated-fault limit holds across handler returns. Handler/filter execution does not establish progress: the guard resets after resumed guest execution advances, a different fault key, or a fresh run. An unrepaired identical fault ends with its ordinary memory error rather than repeatedly dispatching handlers until timeout. This bounds replay; it does not implement complete C++ exception semantics or recovery-page lifetime. Same-PC partial progress such as REP remains a fidelity limit. A `RaiseException` call that no handler or unhandled-exception filter takes returns to the caller when the exception is continuable. A non-continuable exception ends the run with `unhandled_exception`, because the caller cannot continue after it on Windows. See [Exception dispatch](exceptions.md). +The limit on repeated SEH faults applies across handler returns. When a handler or filter runs, the emulator does not count this as progress. The count starts again when guest execution moves forward after the resume, when a different fault occurs or when a new run starts. If the same fault occurs again without a repair, the run ends with its normal memory error, and the handlers do not run again and again until the time limit. This limits the repeats. It does not give full C++ exception behavior or a model of the lifetime of recovery pages. Progress at the same PC, for example in a REP instruction, does not count. When no handler and no unhandled exception filter takes the exception of a `RaiseException` call, the call returns to its caller if the exception is continuable. A non-continuable exception ends the run with `unhandled_exception`, because on Windows the caller cannot continue after it. See [Exception dispatch](exceptions.md). ## Fidelity limits -Synthetic stubs are identical except for their private targets. Nt/Zw names are not inferred to be same-address aliases, and no real syscall numbering or ntdll layout is modeled. FreshyCalls, Halo’s Gate and similar layout-based resolution require versioned native fixtures or explicit handlers. A PE export ordinal base is a DWORD and the image builder can represent that range; GetProcAddress-style ordinal requests and forwarded ordinal syntax use the Windows 16-bit ordinal interface. Generated ordinals in the bundled catalogs stay within that callable range. +All synthetic entries have the same code, except for the token. The emulator does not make `Nt` and `Zw` names aliases at the same address, and it does not model real system call numbers or the layout of ntdll. FreshyCalls, Halo's Gate and other methods that read the layout of ntdll need real native files or explicit handlers. The ordinal base of a PE export directory is a DWORD, and the image builder supports that range. `GetProcAddress` and the forwarder syntax for ordinals use the 16-bit ordinals of Windows. The generated ordinals in the bundled databases fit in that range. -The private reservation is not a guest code or data surface; data probes fault and may be handled by existing guest SEH. A failed startup dependency initializer emits a warning, retains its mapped exports and sets the module state to `init_error` (`LoadState.INIT_ERROR`). If DllMain returns FALSE and the run has no other error, the run error type is `dll_initialization_failed`. Independent dependencies and the main sample continue. It is not treated as initialized or automatically retried at startup. Runtime load failures return the corresponding failure and roll back new attachments. +The guest cannot use the token range for code or data. A read or write in it faults, and the guest SEH can take the fault. When a DLL initializer fails at start, the emulator logs a warning, keeps the exports of the DLL mapped and sets the module state to `init_error` (`LoadState.INIT_ERROR`). If DllMain returns FALSE and the run has no other error, the error type of the run is `dll_initialization_failed`. The other dependencies and the main sample continue. The emulator does not mark the module as initialized and does not try the initializer again at start. When a load fails at run time, the API returns its failure value, and the emulator removes the new PEB entries. diff --git a/doc/volumes.md b/doc/volumes.md index 702362199..f8868b8fc 100644 --- a/doc/volumes.md +++ b/doc/volumes.md @@ -1,17 +1,12 @@ # Mounting host files with --volume -The `-V` / `--volume` flag maps host files or directories into the emulated Windows filesystem. The interface is this flag and the `volumes` parameter of `Speakeasy`. The functions in `speakeasy.volumes` are internal, and a minor release can change them (see [Public interface](library.md#public-interface)). +The `-V` / `--volume` flag maps host files or directories into the emulated Windows file system. The `volumes` parameter of `Speakeasy` does the same in Python. The functions in `speakeasy.volumes` are internal, and a minor release can change them (see [Public interface](library.md#public-interface)). -Format: - -- `host_path:guest_path` -- repeat `-V` to provide multiple mappings - -Both paths can start with a drive letter, for example `C:\samples\mal.exe:D:\staging\mal.exe`. The separator is the first `:` that is not part of a drive letter. +A volume has the format `host_path:guest_path`. Repeat `-V` for more than one volume. Both paths can start with a drive letter, for example `C:\samples\mal.exe:D:\staging\mal.exe`. The separator is the first `:` that is not part of a drive letter. ## Mount a directory recursively -A directory mount maps each file below the host directory, at any depth, to the same relative path below the guest path. With `-V /tmp/vol_demo:c:\appdata`, the emulated filesystem has these files: +A directory mount maps each file below the host directory, at any depth, to the same relative path below the guest path. With `-V /tmp/vol_demo:c:\appdata`, the emulated file system has these files: ```text c:\appdata\configs\settings.ini <- /tmp/vol_demo/configs/settings.ini @@ -32,7 +27,7 @@ Mount one directory: speakeasy -t malware.exe -V /data/samples:c:\windows\temp ``` -Mount multiple paths: +Mount more than one path: ```console speakeasy -t malware.exe \ @@ -40,7 +35,7 @@ speakeasy -t malware.exe \ -V /data/payload.dll:c:\windows\system32\payload.dll ``` -## Programmatic usage +## Python example ```python import speakeasy @@ -55,10 +50,11 @@ module = se.load_module("malware.exe") se.run_module(module) ``` -## Notes +## How Speakeasy applies volumes + +Speakeasy adds a `full_path` file entry to the config for each mounted file. It puts these entries before the file entries of the config, so a volume file wins over a config entry for the same path. -- Volume entries are injected as filesystem `full_path` rules. -- When loading from a host file path, Speakeasy can also auto-mount immediate sibling files from the target directory into the emulated current directory. +When `load_module` loads the input from a host path, Speakeasy also maps each file in the directory of the input into the emulated current directory. It does not map the subdirectories. ## Related docs @@ -66,5 +62,5 @@ se.run_module(module) - [Documentation index](index.md) - [CLI reference](cli-reference.md) - [Configuration walkthrough](configuration.md) -- [Speakeasy 2 walkthrough outline](speakeasy2-walkthrough.md) +- [What's new in 2.0](speakeasy2-walkthrough.md) - [Help and troubleshooting](help.md) diff --git a/speakeasy/binemu.py b/speakeasy/binemu.py index 637a351b3..54c54c52e 100644 --- a/speakeasy/binemu.py +++ b/speakeasy/binemu.py @@ -89,7 +89,6 @@ def fp80_to_double_bits(mantissa, sign_exponent): HookT = TypeVar("HookT", bound=common.Hook) -# Generic emulator class for binary code class BinaryEmulator(MemoryManager, ABC): """ Base class for emulating binaries. @@ -111,17 +110,17 @@ class BinaryEmulator(MemoryManager, ABC): @abstractmethod def _unmap_reserved_range(self) -> None: - """Unmap the reserved memory range. Subclasses must implement.""" + """Unmap the reserved range, which holds the API tokens and the return hooks.""" ... @abstractmethod def on_emu_complete(self) -> None: - """Called when emulation completes. Subclasses must implement.""" + """Finish the emulation after the last run.""" ... @abstractmethod def get_current_run(self) -> Any: - """Get the current run context. Subclasses must implement.""" + """Get the current run.""" ... def __init__(self, config): @@ -138,19 +137,24 @@ def __init__(self, config): self.runtime: float = 0 def get_profiler(self) -> Profiler: - """Get the current event profiler object (if any).""" + """Get the profiler, which records the report events.""" return self.profiler def get_report(self) -> Report: - """Get the emulation report for all runs that were executed.""" + """Get the report of the runs.""" return self.profiler.get_report() def get_json_report(self) -> str: - """Get the emulation report for all runs that were executed formatted as a JSON string.""" + """Get the report of the runs as a JSON string.""" return self.profiler.get_json_report() def _parse_config(self, config: SpeakeasyConfig): - """Parse the config to be used for emulation.""" + """ + Store the config and create the engine that it names. + + Raises: + CpuError: if the config names an engine that is not supported. + """ self.config = config _eng = config.emu_engine @@ -187,13 +191,13 @@ def get_bytes(self, obj): return obj.get_bytes() def stop(self) -> None: - """Stop emulation completely.""" + """Stop the engine and the run clock.""" self.emu_eng.stop() if self.profiler: self.profiler.stop_run_clock() def start(self, addr, size): - """Begin emulation.""" + """Add the hooks and start the engine at ``addr``. An engine error ends the emulation.""" self.set_hooks() self._unmap_reserved_range() if self.profiler: @@ -240,7 +244,7 @@ def reg_read(self, reg) -> int: return value def set_hooks(self): - """Set instruction level hooks.""" + """Add the code, memory and interrupt hooks that the engine does not have yet.""" for ht in ( common.HOOK_CODE, common.HOOK_MEM_READ, @@ -276,7 +280,7 @@ def disasm(self, mem: bytes, addr: int, *, fast: bool = True) -> tuple[str, str, return self._cs_disasm(mem, addr, fast=fast) def get_register_state(self) -> dict[str, int]: - """Get the current state of registers from the emulator.""" + """Get the general-purpose registers and the instruction pointer by name.""" regs = {} if self.get_arch() == e_arch.ARCH_X86: for name, reg in ( @@ -320,19 +324,14 @@ def get_disasm(self, addr: int, size: int, *, fast: bool = True) -> tuple[str, s def set_func_args(self, stack_addr, ret_addr, *args, conv, home_space=True): """ - Set the arguments before an emulated function call. + Set the arguments of a function call, then push ``ret_addr`` and set the stack pointer. - Lays out ``args`` according to ``conv`` and the current architecture, - writes ``ret_addr`` at the top of the resulting stack frame, and - updates the stack pointer register. - - On x86 fastcall the first two arguments go into ECX and EDX. On x64 - the first four go into RCX, RDX, R8 and R9. The remaining arguments - go onto the stack. On x64 a 32-byte shadow (home) space is - reserved above the return address when ``home_space`` is ``True``. + On x86 fastcall, the first two arguments go in ECX and EDX. On x64, the first four go + in RCX, RDX, R8 and R9. The other arguments go on the stack. On x64, if ``home_space`` + is true, the method also reserves the 32-byte home space above the return address. Args: - stack_addr: Base address of the stack region. + stack_addr: Stack pointer before the call. ret_addr: Return address to place on the stack. *args: Argument values to pass to the callee. conv: Calling convention (for example, ``CALL_CONV_STDCALL``). @@ -377,7 +376,7 @@ def set_func_args(self, stack_addr, ret_addr, *args, conv, home_space=True): self.reg_write(sp_reg, curr_sp) def get_func_argv(self, callconv, argc): - """Get the arguments for a function given the supplied calling convention.""" + """Read the arguments of a function at its entry, for a calling convention.""" argv = [] ptr_size = self.get_ptr_size() arch = self.get_arch() @@ -417,7 +416,7 @@ def get_func_argv(self, callconv, argc): else: raise CpuError("Unsupported architecture") - # Skip past the saved ret addr + # Skip the return address sp += ptr_size for _ in range(nargs): ptr = self.mem_read(sp, ptr_size) @@ -427,7 +426,11 @@ def get_func_argv(self, callconv, argc): return argv def do_call_return(self, argc, ret_addr=None, ret_value=None, conv=None): - """Set the emulation state after a call has completed.""" + """ + Return from a call: set the pc to ``ret_addr`` and the return register to ``ret_value``. + + If ``conv`` makes the callee remove its stack arguments, also remove them. + """ if self.get_arch() == e_arch.ARCH_X86: sp = e_arch.X86_REG_ESP pc = e_arch.X86_REG_EIP @@ -453,9 +456,8 @@ def do_call_return(self, argc, ret_addr=None, ret_value=None, conv=None): else: self.reg_write(rr, ret_value) - # Cleanup the stack if conv in (e_arch.CALL_CONV_CDECL, e_arch.CALL_CONV_FLOAT): - # If cdecl, the emu engine will clean the stack + # The caller removes the arguments. pass elif conv == e_arch.CALL_CONV_FASTCALL: if self.get_arch() == e_arch.ARCH_X86: @@ -487,7 +489,7 @@ def get_ret_address(self) -> int: return int.from_bytes(self.mem_read(sp, self.ptr_size), "little") def set_ret_address(self, addr) -> None: - """Set the return address on the stack.""" + """Write ``addr`` at the top of the stack.""" sp = self.get_stack_ptr() self.mem_write(sp, addr.to_bytes(self.get_ptr_size(), "little")) @@ -611,7 +613,12 @@ def reset_cpu_context(self): self.reg_write(e_arch.X86_REG_MXCSR, 0x1F80) def reset_stack(self, base) -> tuple[int, int]: - """Reset stack to the supplied base address.""" + """ + Set the stack pointer and the frame pointer below ``base``, with room for stack arguments. + + Returns: + A tuple of the stack base and the stack pointer. + """ arch = self.get_arch() ptr = base @@ -621,7 +628,8 @@ def reset_stack(self, base) -> tuple[int, int]: self.reg_write(e_arch.X86_REG_ESP, ptr) self.reg_write(e_arch.X86_REG_EBP, ptr) elif arch == e_arch.ARCH_AMD64: - # Save room for stack arguments beyond the ones a run supplies; an odd count keeps rsp % 16 + # Save room for stack arguments beyond the ones a run supplies. An odd count makes rsp + 8 + # 16-byte aligned, as at the entry of a function. ptr -= self.ptr_size * 65 self.reg_write(e_arch.AMD64_REG_RSP, ptr) self.reg_write(e_arch.AMD64_REG_RBP, ptr) @@ -630,12 +638,11 @@ def reset_stack(self, base) -> tuple[int, int]: def alloc_stack(self, size): """Allocate memory to use for the program stack.""" - # Allocate memory for our stack - # Stack grows down chunk = self.get_valid_ranges(size, addr=0x1200000) addr, block_size = chunk self.mem_map(block_size, base=addr, tag="emu.stack", perms=common.PERM_MEM_RW) + # The stack grows down, so its base is the end of the map. base = addr + block_size self.mem_reserve(size, base=base) @@ -644,7 +651,7 @@ def alloc_stack(self, size): return base, ptr def clean_stack_args(self, argc): - """Adjust the stack for arguments that were supplied.""" + """Remove ``argc`` arguments from the stack on x86. On x64, the caller removes them.""" ptr_size = self.get_ptr_size() arch = self.get_arch() @@ -699,7 +706,7 @@ def read_mem_string(self, address, width=1, max_chars=0) -> str: return text.replace("\x00", "") def mem_string_len(self, address, width=1): - """Get the length of a string from emulated memory.""" + """Get the length in characters of a null-terminated string in emulated memory.""" slen = -1 char = b"\xff" @@ -710,7 +717,7 @@ def mem_string_len(self, address, width=1): return slen def get_ansi_strings(self, data, min_len=4): - """Get all ansi strings from a supplied memory blob.""" + """Get the offset and the text of each ANSI string in a buffer.""" astrs = [] pat = b"[\x20-\x7f]{%d,}" % (min_len) res = re.compile(pat) @@ -723,7 +730,7 @@ def get_ansi_strings(self, data, min_len=4): return astrs def get_unicode_strings(self, data, min_len=4): - """Get all unicode strings from a supplied memory blob.""" + """Get the offset and the text of each UTF-16LE string in a buffer.""" wstrs = [] pat = b"(?:[\x20-\x7f]\x00){%d,}" % (min_len) res = re.compile(pat) @@ -749,7 +756,7 @@ def write_mem_string(self, string, address, width=1): """ Write string data to an emulated memory address. - Appends a terminating zero byte if the string does not end with one. + Appends a null character if the string does not end with one. Raises: ValueError: if ``width`` is not 1 or 2. @@ -819,7 +826,7 @@ def set_ptr_size(self, arch): raise CpuError("Unsupported architecture") def get_module_from_addr(self, addr): - """If the supplied address belongs to a module, return it.""" + """Get the module that contains an address, or ``None``.""" for mod in self.modules: base = mod.base size = mod.image_size @@ -828,7 +835,7 @@ def get_module_from_addr(self, addr): return None def get_api_hooks(self, mod_name, func_name) -> list[common.ApiHook]: - """If an API hook has been set, return it here.""" + """Get the enabled API hooks whose module and API names or wildcard patterns match.""" mod_name = mod_name.lower() func_name = func_name.lower() try: @@ -860,11 +867,13 @@ def add_hook(self, hook: HookT) -> HookT: """ Bind a hook to this emulator and add it. - When multiple hooks are registered for the same event, they run in - FIFO order (first registered, first called). All hooks in the chain run, - and the return value of the last hook is used. This order applies to all - hook types in Speakeasy. An IN instruction is the exception: only the first - IN hook that is enabled and covers the address runs. + The hooks of one type run in the order that the caller added them, and all of them run, + with these exceptions: + + - An invalid memory hook that returns ``False`` stops the hooks after it. + - An invalid instruction hook that returns ``True`` stops the hooks after it. + - For an IN instruction, only the first IN hook that is enabled and covers the address runs. + - For an API that has no handler, only the API hook that the caller added last runs. Raises: ValueError: if the engine rejects the hook, for example an instruction hook on an @@ -926,17 +935,14 @@ def _add_api_hook(self, hook: common.ApiHook) -> None: api_dictionary = ({api_name: [hook]}, wildcard_api) if not _hooks: - # First addition obj = ({module: api_dictionary}, wildcard_module) else: module_dict, previous_wildcard_module = _hooks try: api_dict, previous_wildcard_api = module_dict[module] except KeyError: - # The module asked is not present, so we just add the api dictionary module_dict[module] = api_dictionary else: - # The module asked is present, so we can just add the hook api_dict.setdefault(api_name, []).append(hook) module_dict[module] = (api_dict, previous_wildcard_api | wildcard_api) obj = (module_dict, previous_wildcard_module | wildcard_module) @@ -947,7 +953,7 @@ def add_api_hook(self, cb: ApiHookCallback, module="", api_name="", argc=0, call return self.add_hook(common.ApiHook(cb, module, api_name, argc, call_conv)) def add_code_hook(self, cb: CodeHookCallback, begin=1, end=0) -> common.CodeHook: - """Add a hook that will fire for every CPU instruction.""" + """Add a hook that fires for each instruction from ``begin`` to ``end``.""" return self.add_hook(common.CodeHook(cb, begin, end)) def _fire_dyn_code_hooks(self, addr): @@ -963,7 +969,7 @@ def _fire_dyn_code_hooks(self, addr): h.cb(h.emu, mm) def _set_dyn_code_hook(self, addr, size): - """Set the top level dispatch hook for dynamic code execution.""" + """Add a code hook on the start of a new region, which fires the dynamic code hooks one time.""" max_hook_size = 0x10 if size > max_hook_size: size = max_hook_size @@ -978,26 +984,30 @@ def _dynamic_code_cb(emu, addr, size): hook_ref[0] = self.add_code_hook(cb=_dynamic_code_cb, begin=addr, end=addr + size) def add_dyn_code_hook(self, cb: DynCodeHookCallback) -> common.DynCodeHook: - """Add a hook that will fire when dynamically generated/copied code is executed.""" + """Add a hook that fires when the guest runs code in memory that ``VirtualAlloc`` or a similar API allocated.""" return self.add_hook(common.DynCodeHook(cb)) def add_mem_read_hook(self, cb: MemAccessHookCallback, begin=1, end=0) -> common.ReadMemHook: - """Add a hook that will fire for memory reads.""" + """Add a hook that fires when the guest reads mapped memory.""" return self.add_hook(common.ReadMemHook(cb, begin, end)) def add_mem_write_hook(self, cb: MemAccessHookCallback, begin=1, end=0) -> common.WriteMemHook: - """Add a hook that will fire for memory writes.""" + """Add a hook that fires when the guest writes mapped memory.""" return self.add_hook(common.WriteMemHook(cb, begin, end)) def add_mem_map_hook(self, cb: MemMapHookCallback, begin=1, end=0) -> common.MapMemHook: - """Add a hook that will fire for memory maps.""" + """Add a hook that fires when the emulator maps memory.""" return self.add_hook(common.MapMemHook(cb, begin, end)) def _intercept_api_trap(self, access, address, size): return False def _hook_mem_invalid_dispatch(self, emu, access, address, size, value): - """This handler will dispatch other invalid memory hooks.""" + """ + Call the invalid memory hooks, unless the access is an API call. + + A hook that returns ``False`` stops the hooks after it. + """ if self._intercept_api_trap(access, address, size): return False hl = self.hooks.get(common.HOOK_MEM_INVALID, []) @@ -1011,18 +1021,18 @@ def _hook_mem_invalid_dispatch(self, emu, access, address, size, value): return rv def add_mem_invalid_hook(self, cb: MemAccessHookCallback) -> common.InvalidMemHook: - """Add a hook that will fire for invalid memory access.""" + """Add a hook that fires when the guest accesses unmapped or protected memory.""" return self.add_hook(common.InvalidMemHook(cb)) def add_interrupt_hook(self, cb: InterruptHookCallback) -> common.InterruptHook: - """Add a hook that will fire for software interrupts.""" + """Add a hook that fires for interrupts and CPU exceptions.""" return self.add_hook(common.InterruptHook(cb)) def add_instruction_hook( self, cb: InHookCallback | OutHookCallback | SyscallHookCallback, begin=1, end=0, *, insn: int ) -> common.InstructionHook: """ - Add a hook that will fire for an IN, OUT, SYSCALL, or SYSENTER instruction. + Add a hook that fires for an IN, OUT, SYSCALL, or SYSENTER instruction. Raises: ValueError: if the engine does not support hooks on ``insn``. @@ -1038,5 +1048,5 @@ def _hook_in_dispatch(self, uc_engine, port, size, user_data): return 0 def add_invalid_instruction_hook(self, cb: InvalidInstructionHookCallback) -> common.InvalidInstructionHook: - """Add a hook that will fire for invalid instructions.""" + """Add a hook that fires when the guest runs an invalid instruction.""" return self.add_hook(common.InvalidInstructionHook(cb)) diff --git a/speakeasy/cli.py b/speakeasy/cli.py index 63b8c576c..24898cc98 100644 --- a/speakeasy/cli.py +++ b/speakeasy/cli.py @@ -261,16 +261,14 @@ def run_main(parser: argparse.ArgumentParser, args: argparse.Namespace, config_s def main(): parser = argparse.ArgumentParser(description="Emulate a Windows binary with speakeasy", allow_abbrev=False) - parser.add_argument( - "-t", "--target", action="store", dest="target", required=False, help="Path to input file to emulate" - ) + parser.add_argument("-t", "--target", action="store", dest="target", required=False, help="Path of the input file") parser.add_argument( "-o", "--output", action="store", dest="output", required=False, - help="Path to output file to save report (default: write the report to stdout)", + help="Path of the report file (default: write the report to stdout)", ) parser.add_argument( "--argv", @@ -278,25 +276,25 @@ def main(): default="", dest="argv", required=False, - help="Commandline parameters to supply to emulated process, as a quoted string " + help="Command line arguments of the emulated process, as one quoted string " "(e.g. --argv=\"-log -path 'C:\\\\path with spaces\\\\'\")", ) parser.add_argument( - "-c", "--config", action="store", dest="config", required=False, help="Path to emulator config file" + "-c", "--config", action="store", dest="config", required=False, help="Path of a JSON config file" ) parser.add_argument( "--dump-default-config", action="store_true", dest="dump_default_config", required=False, - help="Print built-in default config JSON and exit", + help="Print the default config as JSON and exit", ) parser.add_argument( "--raw", action="store_true", dest="do_raw", required=False, - help="Attempt to emulate file as-is with no parsing (e.g. shellcode)", + help="Emulate the input as shellcode, without PE parsing", ) parser.add_argument( "--raw-offset", @@ -304,7 +302,7 @@ def main(): default=0, required=False, dest="raw_offset", - help="When in raw mode, offset to start emulating: decimal, or hex with 0x", + help="Offset in the shellcode at which emulation starts: decimal, or hex with 0x", ) parser.add_argument( "--entry-point", @@ -312,8 +310,8 @@ def main(): default=None, required=False, dest="entry_point", - help="RVA to use as entry point instead of the PE's default: decimal, or hex with 0x. " - "Only applies to PE modules, not raw/shellcode.", + help="RVA at which emulation starts, in place of the entry point of the PE: decimal, or hex with 0x. " + "Applies only to PE input.", ) parser.add_argument( "--arch", @@ -322,14 +320,14 @@ def main(): type=str.lower, choices=["x86", "x64", "amd64"], required=False, - help="Architecture of the shellcode. Required with --raw, and only applies to raw mode.", + help="Architecture of the shellcode. Required with --raw.", ) parser.add_argument( "--dropped-files-path", action="store", dest="dropped_files_path", required=False, - help="Path to store files created during emulation", + help="Path of a zip archive for the files that the sample writes, with a speakeasy_manifest.json", ) parser.add_argument( "-k", @@ -337,14 +335,14 @@ def main(): action="store_true", dest="emulate_children", required=False, - help="Emulate any processes created with CreateProcess APIs after the input file finishes emulating", + help="After the PE input finishes, emulate the processes that the input creates", ) parser.add_argument( "--no-mp", action="store_true", dest="no_mp", required=False, - help="Run emulation in the current process instead of a child process", + help="Run the emulation in the current process, in place of a child process", ) parser.add_argument( "-v", @@ -359,7 +357,7 @@ def main(): action="store_true", dest="gdb", required=False, - help="Enable GDB server stub (pauses before first instruction)", + help="Start a GDB server and wait for a client before the first instruction. Implies --no-mp.", ) parser.add_argument( "--gdb-host", @@ -367,7 +365,7 @@ def main(): dest="gdb_host", default="127.0.0.1", required=False, - help="GDB server bind host (default: 127.0.0.1)", + help="Address on which the GDB server listens (default: 127.0.0.1)", ) parser.add_argument( "--gdb-port", @@ -376,7 +374,7 @@ def main(): type=int, default=1234, required=False, - help="GDB server port (default: 1234)", + help="Port on which the GDB server listens (default: 1234)", ) parser.add_argument( "-V", @@ -384,7 +382,7 @@ def main(): action="append", dest="volumes", default=[], - help="Mount a host path into the emulated filesystem (host_path:guest_path). May be repeated.", + help="Map a host path into the emulated file system (host_path:guest_path). Can be given more than once.", ) parser.add_argument("--version", action="version", version=f"speakeasy {__version__}") diff --git a/speakeasy/common.py b/speakeasy/common.py index 00368be60..82cdb99dc 100644 --- a/speakeasy/common.py +++ b/speakeasy/common.py @@ -63,7 +63,7 @@ class Hook: Base class for all emulator hooks. A hook has no emulator until ``bind`` gives it one. ``Speakeasy`` keeps the hooks - that a caller adds before the load, and binds them before the emulator runs. + that a caller adds before the load, and binds them when it creates the machine. """ htype: int @@ -76,9 +76,9 @@ def __init__(self, cb, *, native_hook=False): Args: cb: Python callback function. Its first argument is the emulator. - native_hook: If true, register a new, raw callback with the underlying - emulation engine, which calls it directly. Otherwise, dispatch the hook - through a wrapper hook (for example ``_wrap_code_cb``). + native_hook: If true, ``add`` adds an engine hook that calls the callback through + a wrapper such as ``_wrap_code_cb``. If false, a dispatch hook of the emulator + calls the callback, for example ``_hook_mem_invalid_dispatch``. """ self.cb: Callable[..., Any] = cb self.handle: int = 0 @@ -150,7 +150,7 @@ def _wrap_invalid_insn_cb(self, emu, ctx=None): class ApiHook(Hook): - """This hook type is used when using a specific API (e.g. kernel32.CreateFile).""" + """Hook that fires when the guest calls an API, for example ``kernel32.CreateFile``.""" htype = HOOK_API @@ -164,16 +164,17 @@ def __init__(self, cb, module="", api_name="", argc=0, call_conv=None): class DynCodeHook(Hook): """ - Hook that fires when dynamically created or copied code runs. + Hook that fires when the guest runs code in memory that ``VirtualAlloc`` or a similar API allocated. - It fires once per dynamic code mapping. This is useful for unpacking. + It fires the first time that the start of the region runs, and each time that code in the + region calls an API. Use it to find unpacked code. """ htype = HOOK_DYN_CODE class CodeHook(Hook): - """This hook callback will fire for every CPU instruction.""" + """Hook that fires for each instruction in its address range.""" htype = HOOK_CODE @@ -189,7 +190,7 @@ def add(self): class ReadMemHook(Hook): - """This hook will fire each time a valid chunk of memory is read from.""" + """Hook that fires when the guest reads mapped memory in its address range.""" htype = HOOK_MEM_READ @@ -207,7 +208,7 @@ def add(self): class WriteMemHook(Hook): - """This hook will fire each time a valid chunk of memory is written to.""" + """Hook that fires when the guest writes mapped memory in its address range.""" htype = HOOK_MEM_WRITE @@ -225,7 +226,7 @@ def add(self): class MapMemHook(Hook): - """This hook will fire each time a chunk of memory is mapped.""" + """Hook that fires when the emulator maps memory with a base in the address range of the hook.""" htype = HOOK_MEM_MAP @@ -236,7 +237,7 @@ def __init__(self, cb, begin=1, end=0): class InvalidMemHook(Hook): - """This hook will fire each time a invalid chunk of memory is accessed.""" + """Hook that fires when the guest accesses unmapped or protected memory.""" htype = HOOK_MEM_INVALID @@ -247,7 +248,7 @@ def add(self): class InterruptHook(Hook): - """This hook will fire each time a a software interrupt is triggered.""" + """Hook that fires for an interrupt or a CPU exception.""" htype = HOOK_INTERRUPT @@ -264,7 +265,7 @@ class InstructionHook(Hook): """ Hook that fires when a selected instruction executes. - Unicorn supports only the instructions IN, OUT, SYSCALL, and SYSENTER. The emulator + The engine supports only the instructions IN, OUT, SYSCALL, and SYSENTER. The emulator calls the IN hooks through one dispatch hook (see ``BinaryEmulator._hook_in_dispatch``). """ @@ -281,7 +282,7 @@ def covers_pc(self) -> bool: """Get whether the pc is in the range of the hook. ``begin > end`` means all addresses.""" # Unicorn does not check the range of an IN or OUT hook, and checks a SYSENTER hook # against a pc that is not up to date, so the wrappers check the range here. - # A hook with no range does not read the pc, because the read costs about 1 us. + # A hook with no range does not read the pc, because a register read is slow in a hot hook. return self.begin > self.end or self.begin <= self.emu.get_pc() <= self.end def add(self): diff --git a/speakeasy/config.py b/speakeasy/config.py index 439e0d3a8..d66f89b97 100644 --- a/speakeasy/config.py +++ b/speakeasy/config.py @@ -291,15 +291,23 @@ def _parse_config_int(value: Any) -> Any: class AnalysisConfig(BaseModel): model_config = ConfigDict(extra="forbid", frozen=True) - memory_tracing: bool = Field(default=False, description="Enable memory access tracing in reports.") - strings: bool = Field(default=True, description="Extract strings from input and emulated memory.") - coverage: bool = Field(default=False, description="Collect executed instruction addresses per run.") + memory_tracing: bool = Field( + default=False, + description="Count the reads, writes and executes of each symbol, section and memory map for the report.", + ) + strings: bool = Field( + default=True, description="Add the strings of the input and of emulated memory to the report." + ) + coverage: bool = Field(default=False, description="Record the address of each instruction that a run executes.") class ExceptionsConfig(BaseModel): model_config = ConfigDict(extra="forbid", frozen=True) - dispatch_handlers: bool = Field(default=True, description="Dispatch configured exception handlers during faults.") + dispatch_handlers: bool = Field( + default=True, + description="Send CPU exceptions to the VEH and SEH handlers of the guest. If false, a fault ends the run.", + ) class OsVersionConfig(BaseModel): @@ -314,17 +322,21 @@ class OsVersionConfig(BaseModel): class UserConfig(BaseModel): model_config = ConfigDict(extra="forbid", frozen=True) - name: str = Field(default="speakeasy_user", description="Username exposed to account and profile APIs.") - is_admin: bool = Field(default=True, description="Expose elevated privileges to admin checks.") + name: str = Field(default="speakeasy_user", description="User name, for example for GetUserName.") + is_admin: bool = Field(default=True, description="If true, the token of the user is elevated.") sid: str = Field(default="S-1-5-21-1111111111-2222222222-3333333333-1001", description="SID of the emulated user.") class ApiHammeringConfig(BaseModel): model_config = ConfigDict(extra="forbid", frozen=True) - enabled: bool = Field(default=False, description="Enable API hammering mitigation.") - threshold: int = Field(default=2000, description="Repetition threshold that triggers mitigation.") - allow_list: list[str] = Field(default_factory=list, description="API names exempt from mitigation.") + enabled: bool = Field(default=False, description="Patch API hammering loops in x86 code.") + threshold: int = Field( + default=2000, description="Number of calls to one API from one call site before the emulator patches the call." + ) + allow_list: list[str] = Field( + default_factory=list, description="APIs that the emulator does not patch. An empty list uses a built-in list." + ) class SymlinkConfig(BaseModel): @@ -337,42 +349,46 @@ class SymlinkConfig(BaseModel): class DriveConfig(BaseModel): model_config = ConfigDict(extra="forbid", frozen=True) - root_path: str | None = Field(default=None, description="Drive root path returned by drive APIs.") - drive_type: str | None = Field(default=None, description="Drive type token used by drive APIs.") - volume_guid_path: str | None = Field(default=None, description="Volume GUID path mapped to this drive.") + root_path: str | None = Field(default=None, description="Root path of the drive, such as C:\\.") + drive_type: str | None = Field(default=None, description="Drive type, such as DRIVE_FIXED.") + volume_guid_path: str | None = Field(default=None, description="Volume GUID path of the drive.") class ByteFillConfig(BaseModel): model_config = ConfigDict(extra="forbid", frozen=True) - byte: str = Field(description="Byte value used to synthesize file content.") - size: _Size = Field(description="Number of bytes to synthesize.") + byte: str = Field(description="Byte that fills the content, in hex, such as 0x41.") + size: _Size = Field(description="Size of the content in bytes.") + + +_HOST_PATH_DESCRIPTION = "Host file that gives the data. $ROOT$ is the directory of the speakeasy package." +_BYTE_FILL_DESCRIPTION = "Data that the emulator makes, in place of a host file." class FileEntryFullPath(BaseModel): model_config = ConfigDict(extra="forbid", frozen=True) - mode: Literal["full_path"] = Field(description="File entry matcher mode.") - emu_path: str = Field(description="Emulated path pattern matched by this entry.") - path: str | None = Field(default=None, description="Host file path served for matched reads.") - byte_fill: ByteFillConfig | None = Field(default=None, description="Synthetic content source for matched reads.") + mode: Literal["full_path"] = Field(description="full_path matches the full path.") + emu_path: str = Field(description="Path that this entry matches. It can have wildcards, such as *.") + path: str | None = Field(default=None, description=_HOST_PATH_DESCRIPTION) + byte_fill: ByteFillConfig | None = Field(default=None, description=_BYTE_FILL_DESCRIPTION) class FileEntryByExt(BaseModel): model_config = ConfigDict(extra="forbid", frozen=True) - mode: Literal["by_ext"] = Field(description="File entry matcher mode.") - ext: str = Field(description="File extension matched by this entry.") - path: str | None = Field(default=None, description="Host file path served for matched reads.") - byte_fill: ByteFillConfig | None = Field(default=None, description="Synthetic content source for matched reads.") + mode: Literal["by_ext"] = Field(description="by_ext matches all paths with the extension ext.") + ext: str = Field(description="File extension that this entry matches, without the dot, such as exe.") + path: str | None = Field(default=None, description=_HOST_PATH_DESCRIPTION) + byte_fill: ByteFillConfig | None = Field(default=None, description=_BYTE_FILL_DESCRIPTION) class FileEntryDefault(BaseModel): model_config = ConfigDict(extra="forbid", frozen=True) - mode: Literal["default"] = Field(description="File entry matcher mode.") - path: str | None = Field(default=None, description="Fallback host file path served for reads.") - byte_fill: ByteFillConfig | None = Field(default=None, description="Fallback synthetic content source.") + mode: Literal["default"] = Field(description="default matches all paths that no other entry matches.") + path: str | None = Field(default=None, description=_HOST_PATH_DESCRIPTION) + byte_fill: ByteFillConfig | None = Field(default=None, description=_BYTE_FILL_DESCRIPTION) FileEntry = Annotated[ @@ -387,16 +403,16 @@ class FilesystemConfig(BaseModel): files: list[FileEntry] = Field( default_factory=lambda: [_FILE_ENTRY.validate_python(item) for item in _DEFAULT_FILES], - description="Filesystem response mapping rules.", + description="Entries that give the content of the files in the emulated file system.", ) class RegistryValueConfig(BaseModel): model_config = ConfigDict(extra="forbid", frozen=True) - name: str = Field(description="Registry value name.") - type: str = Field(description="Registry type token.") - data: str = Field(description="Registry value payload.") + name: str = Field(description="Value name. default is the unnamed default value.") + type: str = Field(description="Value type, such as REG_SZ, REG_DWORD or REG_MULTI_SZ.") + data: str = Field(description="Value data as a string. REG_DWORD and REG_QWORD data is a number, such as 0x3.") class RegistryKeyConfig(BaseModel): @@ -411,26 +427,29 @@ class RegistryConfig(BaseModel): keys: list[RegistryKeyConfig] = Field( default_factory=lambda: [RegistryKeyConfig.model_validate(item) for item in _DEFAULT_REGISTRY_KEYS], - description="Seed registry key definitions.", + description="Registry keys and values that exist when emulation starts.", ) class DnsTxtConfig(BaseModel): model_config = ConfigDict(extra="forbid", frozen=True) - name: str | None = Field(default=None, description="Domain name for this TXT response mapping.") - path: str | None = Field(default=None, description="Host file path used as TXT response payload.") + name: str | None = Field( + default=None, description="Domain name of this TXT record. default gives the TXT record of all other names." + ) + path: str | None = Field(default=None, description="Host file that gives the TXT record.") class DnsConfig(BaseModel): model_config = ConfigDict(extra="forbid", frozen=True) names: dict[str, str] = Field( - default_factory=lambda: dict(_DEFAULT_DNS_NAMES), description="Domain-to-IP mappings used by DNS lookups." + default_factory=lambda: dict(_DEFAULT_DNS_NAMES), + description="IP address of each domain name. default gives the address of all other names.", ) txt: list[DnsTxtConfig] = Field( default_factory=lambda: [DnsTxtConfig.model_validate(item) for item in _DEFAULT_DNS_TXT], - description="TXT response mapping entries.", + description="Host files that give the TXT records.", ) @@ -438,9 +457,12 @@ class HttpResponseConfig(BaseModel): model_config = ConfigDict(extra="forbid", frozen=True) verb: Literal["GET", "POST", "HEAD"] | None = Field( - default=None, description="HTTP verb matched by this response set." + default=None, description="HTTP verb of the requests that these files answer." + ) + files: list[FileEntry] = Field( + default_factory=list, + description="Files that give the response data. by_ext matches the extension of the URL path.", ) - files: list[FileEntry] = Field(default_factory=list, description="File mapping rules for this HTTP response set.") class HttpConfig(BaseModel): @@ -448,7 +470,7 @@ class HttpConfig(BaseModel): responses: list[HttpResponseConfig] = Field( default_factory=lambda: [HttpResponseConfig.model_validate(item) for item in _DEFAULT_HTTP_RESPONSES], - description="HTTP response mapping entries.", + description="Files that give the HTTP response data, by verb.", ) @@ -457,7 +479,7 @@ class WinsockConfig(BaseModel): responses: list[FileEntry] = Field( default_factory=lambda: [_FILE_ENTRY.validate_python(item) for item in _DEFAULT_WINSOCK_RESPONSES], - description="Winsock receive response mapping entries.", + description="Data that recv gives on a socket. The emulator uses only the default entry.", ) @@ -470,7 +492,7 @@ class NetworkAdapterConfig(BaseModel): type: str | None = Field(default=None, description="Adapter media type.") ip_address: str | None = Field(default=None, description="Adapter IPv4 address.") subnet_mask: str | None = Field(default=None, description="Adapter subnet mask.") - dhcp_enabled: bool | None = Field(default=None, description="Whether DHCP is enabled for this adapter.") + dhcp_enabled: bool | None = Field(default=None, description="If true, the adapter uses DHCP.") class NetworkConfig(BaseModel): @@ -481,7 +503,7 @@ class NetworkConfig(BaseModel): winsock: WinsockConfig | None = Field(default_factory=WinsockConfig, description="Winsock emulation settings.") adapters: list[NetworkAdapterConfig] = Field( default_factory=lambda: [NetworkAdapterConfig.model_validate(item) for item in _DEFAULT_ADAPTERS], - description="Network adapters returned by APIs.", + description="Network adapters that APIs such as GetAdaptersInfo give.", ) @@ -489,12 +511,15 @@ class ProcessConfig(BaseModel): model_config = ConfigDict(extra="forbid", frozen=True) name: str = Field(description="Process name.") - base_addr: _Address = Field(description="Process image base address.") - path: str = Field(description="Process executable path.") - pid: int | None = Field(default=None, description="Optional process identifier.") - command_line: str | None = Field(default=None, description="Optional process command line.") - is_main_exe: bool | None = Field(default=None, description="Whether this process is the main executable container.") - session: int | None = Field(default=None, description="Optional session identifier.") + base_addr: _Address = Field(description="Base address of the process image.") + path: str = Field(description="Path of the process image.") + pid: int | None = Field(default=None, description="Process ID.") + command_line: str | None = Field(default=None, description="Command line of the process.") + is_main_exe: bool | None = Field( + default=None, + description="If true, this entry gives the container process, in which a DLL or shellcode input runs.", + ) + session: int | None = Field(default=None, description="Session ID of the process.") class DeviceConfig(BaseModel): @@ -507,13 +532,13 @@ class DriverConfig(BaseModel): model_config = ConfigDict(extra="forbid", frozen=True) name: str = Field(description="Driver object name.") - devices: list[DeviceConfig] = Field(default_factory=list, description="Device objects exposed by this driver.") + devices: list[DeviceConfig] = Field(default_factory=list, description="Device objects of the driver.") class UserModuleConfig(BaseModel): model_config = ConfigDict(extra="forbid", frozen=True) - name: str = Field(description="User-mode module short name.") + name: str = Field(description="Module name without the extension.") base_addr: _Address = Field(description="Module base address.") path: str = Field(description="Module path.") @@ -521,32 +546,36 @@ class UserModuleConfig(BaseModel): class SystemModuleConfig(BaseModel): model_config = ConfigDict(extra="forbid", frozen=True) - name: str = Field(description="System module short name.") + name: str = Field(description="Module name without the extension.") base_addr: _Address = Field(description="Module base address.") path: str = Field(description="Module path.") - driver: DriverConfig | None = Field(default=None, description="Optional driver and device object metadata.") + driver: DriverConfig | None = Field(default=None, description="Driver object and device objects of the module.") class ModulesConfig(BaseModel): model_config = ConfigDict(extra="forbid", frozen=True) modules_always_exist: bool = Field( - default=False, description="Synthesize unknown modules instead of failing loads." + default=False, + description="If true, the emulator loads an unknown module as an empty API module.", + ) + functions_always_exist: bool = Field( + default=False, + description="If true, an API without a handler or a declaration resolves to a stub that returns 1.", ) - functions_always_exist: bool = Field(default=False, description="Treat unresolved imports as existing stubs.") module_directory_x86: str | None = Field( - default=None, description="Directory of x86 DLLs to load instead of synthetic API modules." + default=None, description="Directory of x86 DLLs that the emulator loads in place of the synthetic API modules." ) module_directory_x64: str | None = Field( - default=None, description="Directory of x64 DLLs to load instead of synthetic API modules." + default=None, description="Directory of x64 DLLs that the emulator loads in place of the synthetic API modules." ) user_modules: list[UserModuleConfig] = Field( default_factory=lambda: [UserModuleConfig.model_validate(item) for item in _DEFAULT_USER_MODULES], - description="Configured user-mode module inventory.", + description="User-mode modules that the emulator loads at their base addresses.", ) system_modules: list[SystemModuleConfig] = Field( default_factory=lambda: [SystemModuleConfig.model_validate(item) for item in _DEFAULT_SYSTEM_MODULES], - description="Configured kernel/system module inventory.", + description="Kernel-mode modules.", ) @@ -567,26 +596,32 @@ def lowercase_env_names(cls, v: dict[str, str]) -> dict[str, str]: return {name.lower(): value for name, value in v.items()} config_version: Literal["2.0"] = Field(default="2.0", description="Configuration schema version.") - emu_engine: Literal["unicorn"] = Field(default="unicorn", description="Emulation backend identifier.") + emu_engine: Literal["unicorn"] = Field( + default="unicorn", description="Emulation engine. The only value is unicorn." + ) max_run_time: float = Field( default=60, - description="Maximum active execution seconds per run. 0 disables this cap.", + description="Limit of the execution time of each run, in seconds. 0 disables the limit.", ) max_total_time: float = Field( default=0, ge=0, allow_inf_nan=False, - description="Maximum active execution seconds per public invocation, across all runs. 0 disables this cap.", + description="Limit of the execution time of all runs in one call of run_module, run_shellcode, call or resume," + " in seconds. 0 disables the limit.", + ) + max_api_calls: int = Field(default=10000, description="Limit of the API calls of each run.") + max_instructions: int = Field( + default=-1, description="Limit of the instructions of each run. 0 or less disables the limit." ) - max_api_calls: int = Field(default=10000, description="Maximum API calls per run.") - max_instructions: int = Field(default=-1, description="Maximum instructions to execute per run.") stack_size: _Size = Field( default=0, - description="Override stack size in bytes. 0 uses the PE header value or the built-in default (0x12000).", + description="Size of the stack of the input in bytes. 0 uses max(SizeOfStackReserve, 0x12000) for a PE," + " and the stack_commit argument of run_shellcode (default 0x4000) for shellcode.", ) random_seed: int = Field( default=0, - description="Seed of the random number generator that gives random bytes and GUIDs to emulated code.", + description="Seed of the random number generator that gives random bytes and GUIDs to the guest.", ) start_time: str = Field( default="2026-10-31T15:15:00Z", @@ -595,41 +630,41 @@ def lowercase_env_names(cls, v: dict[str, str]) -> dict[str, str]: ) analysis: AnalysisConfig = Field( default_factory=AnalysisConfig, - description="Analysis and telemetry collection settings.", + description="Data that the report collects.", ) keep_memory_on_free: bool = Field( default=False, - description="Retain freed memory maps for post-free inspection.", + description="If true, the memory manager keeps a freed region mapped, so that its data can be read.", ) snapshot_memory_regions: bool = Field( default=False, - description="Include run-end memory region snapshots in the report data store.", + description="If true, the report has the contents of the memory regions at the end of each run.", ) exceptions: ExceptionsConfig = Field( default_factory=ExceptionsConfig, - description="Exception dispatch behavior.", + description="Exception dispatch settings.", ) os_ver: OsVersionConfig = Field( default_factory=OsVersionConfig, - description="OS version values exposed to emulated code.", + description="Windows version that the PEB and the version APIs give.", ) current_dir: str = Field( default="C:\\Windows\\system32", - description="Current working directory for emulated process APIs.", + description="Current directory of the process.", ) command_line: str = Field( default="svchost.exe myarg1 myarg2", - description="Command line exposed to emulated process APIs.", + description="Command line of the process, for example for GetCommandLine.", ) env: dict[str, str] = Field( default_factory=lambda: dict(_DEFAULT_ENV), - description="Environment variables visible to the emulated process.", + description="Environment variables of the process. Names are not case-sensitive.", ) - domain: str | None = Field(default="speakeasy_domain", description="Domain or workgroup identity.") - hostname: str = Field(default="speakeasy_host", description="Hostname exposed to emulated system APIs.") + domain: str | None = Field(default="speakeasy_domain", description="Domain or workgroup of the user.") + hostname: str = Field(default="speakeasy_host", description="Host name, for example for GetComputerName.") user: UserConfig = Field( default_factory=UserConfig, - description="Primary emulated user account.", + description="User account of the process.", ) api_hammering: ApiHammeringConfig | None = Field( default_factory=ApiHammeringConfig, @@ -641,15 +676,15 @@ def lowercase_env_names(cls, v: dict[str, str]) -> dict[str, str]: ) drives: list[DriveConfig] = Field( default_factory=lambda: [DriveConfig.model_validate(item) for item in _DEFAULT_DRIVES], - description="Virtual drive metadata.", + description="Drives that the drive APIs, such as GetLogicalDrives, give.", ) filesystem: FilesystemConfig = Field( default_factory=FilesystemConfig, - description="Filesystem mapping rules.", + description="Content of the files in the emulated file system.", ) registry: RegistryConfig | None = Field( default_factory=RegistryConfig, - description="Registry key and value seed data.", + description="Registry keys and values that exist when emulation starts.", ) network: NetworkConfig = Field( default_factory=NetworkConfig, @@ -657,11 +692,11 @@ def lowercase_env_names(cls, v: dict[str, str]) -> dict[str, str]: ) processes: list[ProcessConfig] = Field( default_factory=lambda: [ProcessConfig.model_validate(item) for item in _DEFAULT_PROCESSES], - description="Process inventory visible to enumeration APIs.", + description="Processes that the process enumeration APIs give, and the container process.", ) modules: ModulesConfig = Field( default_factory=ModulesConfig, - description="Module loading and inventory settings.", + description="Module loading settings and module lists.", ) diff --git a/speakeasy/engines/unicorn_eng.py b/speakeasy/engines/unicorn_eng.py index 38bc5b1c9..776071e49 100644 --- a/speakeasy/engines/unicorn_eng.py +++ b/speakeasy/engines/unicorn_eng.py @@ -1,7 +1,5 @@ # Copyright (C) 2020 FireEye, Inc. All Rights Reserved. -# Unicorn specific wrappers and abstraction implemented here - from __future__ import annotations import ctypes as ct @@ -87,7 +85,12 @@ def get_tsc_offset_location(mode): class ToggleableHook: - """Hook than can be toggled on/off at arbitrary times.""" + """ + Hold the ctypes callback of a hook that ``hook_add`` added through the C API. + + The reference keeps the callback alive while Unicorn can call it. The engine does not read ``enabled``. + The ``Hook`` objects in ``common`` check their own flag. + """ def __init__(self, cb): self.cb: ct._CFunctionType = cb @@ -103,7 +106,7 @@ def disable(self): class EmuEngine: - """Wrapper class for underlying cpu emulation engines.""" + """Wrap a Unicorn instance for the emulator.""" def __init__(self): self.name: str = "unicorn" @@ -196,7 +199,7 @@ def __init__(self): def init_engine(self, eng_arch, mode): """ - Initialize cpu engine. + Create the Unicorn instance for an architecture and a bitness. Raises: EngineError: if ``eng_arch`` is not x86 or AMD64, or ``mode`` is not 32 or 64 bits. @@ -220,18 +223,18 @@ def init_engine(self, eng_arch, mode): get_tsc_offset_location(_mode) def mem_map(self, base, size, perms=common.PERM_MEM_RWX): - """Allocate memory in the cpu engine.""" + """Map memory in the engine.""" assert self.emu is not None perm = self.perms.get(perms, uc.UC_PROT_ALL) return self.emu.mem_map(base, size, perm) def mem_unmap(self, addr, size): - """Free memory in the cpu engine.""" + """Unmap memory in the engine.""" assert self.emu is not None return self.emu.mem_unmap(addr, size) def mem_regions(self): - """Get current memory allocations from the engine.""" + """Get the mapped regions of the engine.""" assert self.emu is not None return self.emu.mem_regions() @@ -240,7 +243,7 @@ def mem_write(self, addr, data): assert self.emu is not None result = self.emu.mem_write(addr, data) if data: - # Host/debugger writes must invalidate already translated stub bytes. + # Drop the translated code of the range, so that the guest runs the new bytes. self.emu.ctl_remove_cache(addr, addr + len(data)) return result @@ -250,7 +253,7 @@ def mem_read(self, addr, size): return self.emu.mem_read(addr, size) def mem_protect(self, addr, size, perms): - """Change the memory protections for pages in the emu engine.""" + """Change the protection of pages in the engine.""" assert self.emu is not None perm = self.perms.get(perms, uc.UC_PROT_ALL) result = self.emu.mem_protect(addr, size, perm) @@ -286,18 +289,18 @@ def advance_tsc(self, ticks): self.emu.context_restore(context) def invalidate_translations(self, begin, end): - """Drop the translated code of [begin, end), e.g. so that a code hook added since applies to it.""" + """Drop the translated code of [begin, end), so that the next run of it uses the current hooks.""" assert self.emu is not None self.emu.ctl_remove_cache(begin, end) def flush_translations(self): - """Drop all translated code, e.g. so that a code hook added since applies to it.""" + """Drop all translated code, so that the next run uses the current hooks.""" assert self.emu is not None self.emu.ctl(uc.UC_CTL_TB_FLUSH, uc.UC_CTL_IO_WRITE) def reg_write(self, reg, val): """ - Modify register values. + Write a register. Raises: EngineError: if ``reg`` is not a known register. @@ -310,7 +313,7 @@ def reg_write(self, reg, val): def reg_read(self, reg): """ - Read register values. + Read a register. Raises: EngineError: if ``reg`` is not a known register. @@ -322,12 +325,12 @@ def reg_read(self, reg): return self.emu.reg_read(ereg) def stop(self): - """Stop the emulation engine.""" + """Stop the engine.""" assert self.emu is not None return self.emu.emu_stop() def start(self, addr, timeout=0, count=0): - """Start the emulation engine. A timeout of 0 seconds disables it.""" + """Start the engine at ``addr``. A ``timeout`` of 0 seconds and a ``count`` of 0 or -1 set no limit.""" assert self.emu is not None if count == -1: count = 0 @@ -341,7 +344,7 @@ def start(self, addr, timeout=0, count=0): def hook_add(self, addr=None, cb=None, htype=None, begin=1, end=0, arg1=0): """ - Add a callback function for a specific event type or address. + Add a callback for a hook type and an address range, and return the hook handle. Raises: EngineError: if ``htype`` is not a known hook type. @@ -355,15 +358,14 @@ def hook_add(self, addr=None, cb=None, htype=None, begin=1, end=0, arg1=0): handle = self.emu._uch - # The unicorn bindings have a default python wrapper. We want to use - # our own wrapper and don't need the extra overhead. Add callbacks directly - # to the unicorn library here. + # The Python wrapper of the Unicorn bindings adds a call to each hook. For the code, memory, + # invalid memory and instruction hooks, add the callback to the C library directly. if hook_type == uc.UC_HOOK_INSN: - if arg1 == u.UC_X86_INS_IN: # IN instruction + if arg1 == u.UC_X86_INS_IN: cb = ct.cast(UC_HOOK_INSN_IN_CB(cb), UC_HOOK_INSN_IN_CB) - elif arg1 == u.UC_X86_INS_OUT: # OUT instruction + elif arg1 == u.UC_X86_INS_OUT: cb = ct.cast(UC_HOOK_INSN_OUT_CB(cb), UC_HOOK_INSN_OUT_CB) - elif arg1 in (u.UC_X86_INS_SYSCALL, u.UC_X86_INS_SYSENTER): # SYSCALL/SYSENTER + elif arg1 in (u.UC_X86_INS_SYSCALL, u.UC_X86_INS_SYSENTER): cb = ct.cast(UC_HOOK_INSN_SYSCALL_CB(cb), UC_HOOK_INSN_SYSCALL_CB) else: raise ValueError(f"unsupported instruction for an instruction hook: {arg1}") @@ -376,7 +378,7 @@ def hook_add(self, addr=None, cb=None, htype=None, begin=1, end=0, arg1=0): else: return self.emu.hook_add(htype=hook_type, callback=cb, begin=begin, end=end) ptr = ct.cast(cb, ct.c_void_p) - # uc_hook_add requires an additional paramter for the hook type UC_HOOK_INSN + # For UC_HOOK_INSN, uc_hook_add takes the instruction as an extra argument. if hook_type == uc.UC_HOOK_INSN: insn = ct.c_int(arg1) rv = _uc.uc_hook_add(handle, ct.byref(hook_id), hook_type, ptr.value, None, begin, end, insn) @@ -391,13 +393,13 @@ def hook_add(self, addr=None, cb=None, htype=None, begin=1, end=0, arg1=0): return hook_id.value def hook_enable(self, hook_handle): - """Enable a previously disabled hook.""" + """Set the ``enabled`` flag of a hook that ``hook_add`` added through the C API.""" hook = self._callbacks.get(hook_handle) if hook: return hook.enable() def hook_disable(self, hook_handle): - """Disable a previously enabled hook.""" + """Clear the ``enabled`` flag of a hook that ``hook_add`` added through the C API.""" hook = self._callbacks.get(hook_handle) if hook: return hook.disable() diff --git a/speakeasy/errors.py b/speakeasy/errors.py index 52e7b150f..0554fa90f 100644 --- a/speakeasy/errors.py +++ b/speakeasy/errors.py @@ -18,7 +18,7 @@ class EmulationError(SpeakeasyError): class NotSupportedError(EmulationError): - """Sample is not currently supported.""" + """Raised when the emulator does not support the input, for example a .NET assembly.""" class ApiEmuError(EmulationError): @@ -30,7 +30,7 @@ class CpuError(EmulationError): class OutOfMemoryError(EmulationError): - """No free range of the address space can hold an allocation.""" + """Raised when no free range of the address space can hold an allocation.""" class WindowsEmuError(EmulationError): @@ -61,6 +61,6 @@ class EngineError(SpeakeasyError): """Raised when the CPU engine rejects a request, for example an unknown register or hook type.""" -# Old names, kept for one release. +# Deprecated aliases. EmuException = CpuError EmuEngineError = EngineError diff --git a/speakeasy/gdb.py b/speakeasy/gdb.py index 255329e8b..96c699b58 100644 --- a/speakeasy/gdb.py +++ b/speakeasy/gdb.py @@ -1,10 +1,9 @@ """ -Speakeasy-native GDB Remote Serial Protocol server. +A GDB Remote Serial Protocol (RSP) server for the emulator. -The server deliberately implements the all-stop subset used by GDB and IDA. -Target operations are performed by the emulation thread; a small reader thread -only parses transport packets and interrupts a running Unicorn instance on -Ctrl-C. +The server implements the all-stop subset of RSP that GDB and IDA use. The emulator thread runs +the commands that read or change the target. A reader thread reads the packets from the socket, +sends the acknowledgements, and stops Unicorn when it gets Ctrl-C. """ from __future__ import annotations @@ -77,7 +76,7 @@ class RegisterInfo: class GdbServer: - """An all-stop RSP server backed directly by a WindowsEmulator.""" + """An all-stop RSP server that reads and changes the state of a WindowsEmulator.""" def __init__(self, emu: Any, port: int, host: str): self.emu: Any = emu @@ -100,7 +99,8 @@ def __init__(self, emu: Any, port: int, host: str): # breakpoint, so a resume from a breakpoint stop does not stop again at once. self._resume_pc: int | None = None self._exec_breakpoints: dict[tuple[int, int, int], int] = {} - # The code hook runs for each instruction, so it looks up one-byte breakpoints by address. + # ``_on_code`` runs for each instruction in a hooked range, so it finds a one-byte breakpoint by + # a dict lookup and scans only the larger breakpoints. self._exec_breakpoint_addresses: dict[int, int] = {} self._exec_breakpoint_ranges: tuple[tuple[int, int, int], ...] = () self._read_watchpoints: dict[tuple[int, int], int] = {} @@ -183,8 +183,8 @@ def close(self) -> None: hook.remove() except Exception: pass - # Prevent BinaryEmulator.set_hooks() from restoring debugger hooks if - # emulation is started again after this session closes. + # ``BinaryEmulator.set_hooks`` adds each removed hook in ``emu.hooks`` again when an emulation + # starts, so remove the debugger hooks from these lists. for registered_hooks in getattr(self.emu, "hooks", {}).values(): if isinstance(registered_hooks, list): registered_hooks[:] = [hook for hook in registered_hooks if hook not in self._hooks] @@ -247,7 +247,8 @@ def _read_loop(self) -> None: pass finally: self._commands.put(None) - # A client that disconnects while the target runs detaches it. + # If the client disconnects while the target runs, stop the run. ``command_loop`` then reads + # None and detaches. self._request_stop(StopReason(signal=2, kind="interrupt")) def _send_raw(self, data: bytes) -> None: @@ -294,8 +295,8 @@ def command_loop(self, reason: StopReason | None = None) -> ResumeAction: continue if action is not None: if not action.detach and not action.kill: - # Mark the resume transition so Ctrl-C arriving before - # begin_run() becomes a pending interrupt, not a queued '?'. + # Set ``_running`` now, so that a Ctrl-C before ``begin_run`` gives a pending + # interrupt and not a ``?`` command. with self._state_lock: self._running = True self._resume_pc = self.emu.get_pc() @@ -331,10 +332,10 @@ def _request_stop(self, reason: StopReason) -> None: return self._stop_reason = reason self._stop_pending = True - # Memory hooks run before the instruction has finished. Stopping here - # can leave a committed write at an unchanged PC, so resuming repeats - # its side effect. Watch stops are delivered at the next code boundary - # or when native single-step/private-trap execution returns naturally. + # A memory hook runs before the instruction completes. If the server stopped Unicorn here, the + # write could complete at an unchanged pc, and a resume would run the instruction again. So + # ``_on_code`` stops a watch at the next instruction, or the run loop stops when Unicorn + # returns by itself, for example after a single step or at a native exit. if reason.kind not in ("watch", "rwatch", "awatch") and self.emu.emu_eng is not None: self.emu.emu_eng.stop() @@ -371,19 +372,18 @@ def _set_hook_enabled(hook: Any | None, enabled: bool) -> None: if hook is None: return if enabled and not hook.added: - # A removed hook is disabled, and add() keeps that state. + # A removed hook is disabled, and ``add`` keeps that state. hook.add() hook.enable() elif not enabled and hook.added: - # Hook.disable() still enters Python for every native event. Remove - # the Unicorn hook entirely to avoid that callback overhead. + # A disabled hook still calls Python for each event, so remove the Unicorn hook. hook.remove() def _breakpoint_ranges(self) -> set[tuple[int, int]]: """ Get the inclusive address ranges of the exec breakpoints. - Overlapping and adjacent breakpoints merge, so that no instruction calls two hooks. + This method merges the overlapping and adjacent breakpoints, so that no instruction calls two hooks. """ ranges: list[tuple[int, int]] = [] for begin, size, _ in sorted(self._exec_breakpoints): @@ -396,8 +396,8 @@ def _breakpoint_ranges(self) -> set[tuple[int, int]]: def _refresh_hooks(self) -> None: code_hook_added = self._code_hook is not None and self._code_hook.added - # A watch stop waits for the next instruction, wherever it is. Otherwise - # only breakpoint addresses call Python. + # ``_on_code`` stops a watch at the next instruction, so while a watchpoint is set, the code hook + # runs for each instruction. Otherwise only the breakpoint ranges have code hooks. watching = bool(self._read_watchpoints or self._write_watchpoints or self._access_watchpoints) self._set_hook_enabled(self._code_hook, watching) hook_added = self._code_hook is not None and self._code_hook.added and not code_hook_added @@ -412,7 +412,7 @@ def _refresh_hooks(self) -> None: self._hooks.append(hook) hook_added = True if hook_added: - # Code translated while a hook was absent does not call it. + # Unicorn does not call a new hook from code that it translated before the hook was added. self.emu.emu_eng.flush_translations() self._set_hook_enabled(self._read_hook, bool(self._read_watchpoints or self._access_watchpoints)) self._set_hook_enabled(self._write_hook, bool(self._write_watchpoints or self._access_watchpoints)) @@ -420,8 +420,8 @@ def _refresh_hooks(self) -> None: def _on_code(self, _emu: Any, address: int, _size: int) -> bool: reason = self.pending_stop_reason() if reason is not None and reason.kind in ("watch", "rwatch", "awatch"): - # The watched instruction completed; leave this next instruction - # unexecuted and preserve the original watch reason/address. + # The watched instruction completed. Stop before the next instruction, and keep the + # reason and the address of the watch stop. self.emu.emu_eng.stop() return True if self._resume_pc == address: @@ -518,7 +518,7 @@ def _handle_command(self, payload: bytes) -> ResumeAction | None: elif payload.startswith(b"vCont;"): return self._resume_command(payload[6:]) elif payload[:1] in (b"c", b"s", b"C", b"S"): - # The target has no signal delivery, so C and S ignore the signal. + # The emulator does not deliver signals, so the server ignores the signal of C and S. address = payload[1:] if payload[:1] in (b"c", b"s") else payload[1:].partition(b";")[2] if address: try: @@ -531,8 +531,8 @@ def _handle_command(self, payload: bytes) -> ResumeAction | None: self._reply(b"OK") return ResumeAction(detach=True) elif payload == b"k": - # The protocol gives k no reply, but lldb waits for an exit reply - # as lldb-server sends one. gdb does not read it. + # RSP gives no reply to k, but lldb waits for an exit reply, because lldb-server sends one. + # GDB does not read the reply. self._reply(b"X09") return ResumeAction(kill=True) elif payload.startswith(b"vKill"): @@ -640,8 +640,8 @@ def _make_registers(self) -> list[RegisterInfo]: ) ) definitions.extend((f"st{i}", getattr(ux, f"UC_X86_REG_FP{i}"), 10, "float") for i in range(8)) - # Unicorn does not expose the x87 segment selectors or the operand - # offset. Keep their standard RSP slots so clients retain the canonical layout. + # The server gives 0 for the x87 segment selectors and the operand offset, and ignores a write + # to them. These registers keep their slots in the GDB i386 register layout. definitions.extend( (name, reg, 4, "float") for name, reg in ( @@ -801,7 +801,8 @@ def _handle_xfer(self, payload: bytes) -> bytes | None: if offset >= len(data): return b"l" - # Reserve one payload byte for the m/l marker. _reply will escape data. + # Leave one byte for the m/l marker, and half of the rest, because the escapes of ``_reply`` + # can double the size. chunk = data[offset : offset + min(requested, (_PACKET_SIZE - 1) // 2)] marker = b"l" if offset + len(chunk) >= len(data) else b"m" return marker + chunk @@ -862,8 +863,8 @@ def _memory_map_xml(self) -> str: lines = [""] if self.emu.emu_eng is not None: for begin, end, perms in self.emu.emu_eng.mem_regions(): - # GDB's memory-map DTD has no permissions attribute. IDA maps - # RAM as RWX and ROM as RX, which is the closest representation. + # The memory-map DTD of GDB has no permissions attribute. IDA shows RAM as RWX and ROM + # as RX, so a writable region is RAM and the other regions are ROM. memory_type = "ram" if perms & 2 else "rom" lines.append(f'') lines.append("") @@ -890,8 +891,9 @@ def _stop_reply(self, reason: StopReason) -> bytes: reply += reason.kind.encode("ascii") + b":;" elif reason.kind in ("watch", "rwatch", "awatch") and reason.address is not None: reply += f"{reason.kind}:{reason.address:x};".encode("ascii") - # A library change is a flag rather than a stop kind, so it can coalesce - # with another pending stop reason or with a whole dependency chain. + # The server keeps a library change as a flag and not as a stop kind. So one stop reply can + # report it with another stop reason, and one ``library`` stop can cover a module and the + # modules that it loads. with self._state_lock: library_list_changed = self._library_list_changed self._library_list_changed = False diff --git a/speakeasy/memmgr.py b/speakeasy/memmgr.py index ba0d79eea..93d3818f6 100644 --- a/speakeasy/memmgr.py +++ b/speakeasy/memmgr.py @@ -15,7 +15,12 @@ class MemMap: - """Class that defines a memory mapping (e.g. heap/pool alloc, binary image, etc.).""" + """ + Describe one allocation, such as a heap block, a pool block or an image. + + Allocations smaller than a page share a page block. ``block_base`` and ``block_size`` give the block + that holds the allocation. + """ def __init__(self, base, size, tag, prot, flags, block_base, block_size, shared=False, process=None): self.base: int = base @@ -57,11 +62,10 @@ def __ne__(self, other): class MemoryManager: """ - Primitive memory manager that blocks OS sized allocation units into smaller ones. + Keep the mapped and reserved memory of the emulator, and find free ranges for allocations. - Subclasses must define the following attributes: - hooks: Dictionary of hooks. - config: Emulation configuration. + An allocation smaller than a page that gives no address goes into a shared page block. The class that + inherits this class must set ``hooks``, ``emu_eng`` and ``config``. """ hooks: dict[int, Any] @@ -90,8 +94,8 @@ def allocations_from(self, base: int) -> Iterator[None]: """Start the search for free memory at ``base`` for allocations in this block that give no address.""" previous = self.alloc_base self.alloc_base = base - # Start a new shared block for small allocations, here and after the block, - # so that small allocations also come from the current search base + # Start a new shared block for small allocations, here and after the block, so that small + # allocations also come from the current search base. self.block_base = self.block_offset = 0 try: yield @@ -104,7 +108,6 @@ def _hook_mem_map_dispatch(self, mm): # Copy the list, because a callback can remove its hook. for mem_map_hook in list(hl): if mem_map_hook.enabled: - # the mapped memory region's base address falls within the hook's bounds if mem_map_hook.begin <= mm.base: if not mem_map_hook.end or mem_map_hook.end > mm.base: mem_map_hook.cb(mem_map_hook.emu, mm.base, mm.size, mm.tag, mm.prot, mm.flags) @@ -120,7 +123,16 @@ def mem_map( shared: bool = False, process: Any = None, ) -> int: - """Map a block of memory with specified permissions and a tag.""" + """ + Map memory and return its address. + + If ``base`` is ``None`` and ``size`` is smaller than a page, the allocation goes into the shared page + block, which the emulator maps RWX. Otherwise the emulator maps a page-aligned range at or after + ``base``. + + Raises: + OutOfMemoryError: if no free range fits below the address limit. + """ if not process and tag and not tag.startswith("emu"): process = self.get_current_process() @@ -159,17 +171,19 @@ def mem_map( return base def mem_free(self, base) -> None: - """Free a block of memory, if all blocks in a block are set to free, unmap the entire block.""" + """ + Mark the allocation at ``base`` as free. + + When all allocations in its block are free, unmap the block, unless the config keeps freed memory. + """ mm = self.get_address_map(base) if mm: mm.free = True - # If we want to freeze memory, just return if self.config.keep_memory_on_free: return ml = [m for m in self.get_mem_maps() if m.block_base == mm.block_base] - # if all blocks are free in the current block, free it from the emu engine if all([m.free for m in ml]): self.block_base = 0 self.mem_unmap(mm.block_base, mm.block_size) @@ -177,7 +191,7 @@ def mem_free(self, base) -> None: self.maps.remove(m) def mem_unmap(self, base, size): - """Free a block of emulated memory.""" + """Unmap a range in the engine. The ``MemMap`` objects of the range stay.""" self.emu_eng.mem_unmap(base, size) def mem_write(self, addr, data) -> None: @@ -189,27 +203,27 @@ def mem_read(self, addr, size) -> bytes: return bytes(self.emu_eng.mem_read(addr, size)) def mem_protect(self, addr, size, perms): - """Change memory protections.""" + """Change the protection of a range.""" self.emu_eng.mem_protect(addr, size, perms) def _mem_unmap_region(self, base, size): - """Remove an entire memory region that may not have blocks allocated within it.""" + """Unmap a range in the engine, also a range that has no allocations.""" self.emu_eng.mem_unmap(base, size) def get_address_map(self, address): - """Get the "MemMap" object associated with a specific address.""" + """Get the ``MemMap`` of the mapped allocation that contains an address, or ``None``.""" for m in self.maps: if m.base <= address <= (m.base + m.size) - 1: return m def get_reserve_map(self, address): - """Get the "MemMap" object that was only reserved for a specific address.""" + """Get the ``MemMap`` of the reserved range that contains an address, or ``None``.""" for m in self.mem_reserves: if m.base <= address <= (m.base + m.size) - 1: return m def is_address_valid(self, address) -> bool: - """Was this address previously reserved or mapped?""" + """Return whether an address is in a mapped allocation or a reserved range.""" if self.get_address_map(address): return True if self.get_reserve_map(address): @@ -217,13 +231,13 @@ def is_address_valid(self, address) -> bool: return False def get_address_tag(self, address): - """Get the tag for a supplied memory address.""" + """Get the tag of the allocation that contains an address, or ``None``.""" for m in self.maps: if address >= m.base and address <= (m.base + m.size) - 1: return m.tag def mem_reserve(self, size, base=None, perms=None, tag=None, flags=0, *, shared=False): - """Reserve (but do not map) a block of memory.""" + """Reserve a range without a map, so that other allocations do not use it, and return its address.""" if base is None: block = self.get_valid_ranges(size) base, size = block @@ -234,15 +248,15 @@ def mem_reserve(self, size, base=None, perms=None, tag=None, flags=0, *, shared= return base def mem_unreserve(self, base): - """Release a reserved block of memory without mapping it.""" + """Release the reserved range at ``base`` without a map.""" self.mem_reserves = [r for r in self.mem_reserves if r.base != base] def get_mem_maps(self) -> list[MemMap]: - """Get the listing of current memory maps.""" + """Get the ``MemMap`` objects of the mapped allocations.""" return self.maps def mem_map_reserve(self, mapped_base): - """Map a previously reserved block of memory.""" + """Map the reserved range at ``mapped_base`` with its permissions and tag, and return its address.""" for r in self.mem_reserves: if mapped_base == r.base: self.mem_reserves.remove(r) @@ -251,12 +265,12 @@ def mem_map_reserve(self, mapped_base): return None def get_mem_regions(self): - """Get the current regions of mapped memory.""" + """Get the mapped regions of the engine.""" return self.emu_eng.mem_regions() def get_valid_ranges(self, size: int, addr: int | None = None) -> tuple[int, int]: """ - Get a valid address range that can satisfy the requested size. + Get the first free page-aligned range at or after ``addr`` that has no mapped or reserved page. Args: size: Number of bytes to allocate. @@ -270,10 +284,8 @@ def get_valid_ranges(self, size: int, addr: int | None = None) -> tuple[int, int """ page_size = self.page_size - # mem_map needs to be page aligned total = size - # alloced address needs to also be on a page boundary if addr is None: addr = self.alloc_base base = addr - (addr % page_size) @@ -287,7 +299,6 @@ def get_valid_ranges(self, size: int, addr: int | None = None) -> tuple[int, int for m in self.get_mem_regions(): curr += range(m[0], m[1], page_size) - # Add reserved memory so we don't accidentally allocate it for res in self.mem_reserves: curr += range(res.base, (res.base + res.size), page_size) diff --git a/speakeasy/profiler.py b/speakeasy/profiler.py index 60c3cab4a..fdad641fa 100644 --- a/speakeasy/profiler.py +++ b/speakeasy/profiler.py @@ -69,7 +69,7 @@ class MemAccess: - """Represent a symbolicated chunk of memory that can be tracked.""" + """Read, write and execute counts of one memory range or symbol.""" def __init__(self, base=None, sym=None, size=0): self.base: int | None = base @@ -82,7 +82,7 @@ def __init__(self, base=None, sym=None, size=0): @dataclass class GuestCallbackFrame: - """State retained while guest callbacks run: the API call that started them, if any.""" + """State of a guest callback sequence, and of the API call that started it, if any.""" stack_pointer: int return_address: int @@ -97,9 +97,9 @@ class GuestCallbackFrame: event: ApiEvent | None = None created_modules: list[Any] = field(default_factory=list) peb_module_snapshot: list[tuple[Any, dict[int, Any]]] = field(default_factory=list) - # Set while a generator handler (speakeasy.windows.guest_callbacks) runs on this frame + # The generator handler (speakeasy.windows.guest_callbacks) that runs on this frame, if any. handler: Any = None - # (register, value) pairs of the nonvolatile registers when the first guest call starts + # (register, value) pairs of the nonvolatile registers before the first guest call. saved_registers: tuple[tuple[int, int], ...] | None = None @@ -116,10 +116,9 @@ class DynCode(TypedDict): class Run: """ - Basic execution primitive for the emulation engine. + One unit of execution in the run queue, and the report data that the run collects. - A run can represent any form of execution: a thread, a callback, an exported function, - or even a child process. + A run starts at one address: an entry point, an export, a TLS callback, a thread or a child process. """ def __init__(self): @@ -166,7 +165,7 @@ def get_label(self) -> str: class Profiler: - """Generate an execution report for all runs that occur within a binary emulation.""" + """Collect the events of all runs and build the report.""" def __init__(self, analysis: AnalysisConfig | None = None): super().__init__() @@ -184,34 +183,34 @@ def __init__(self, analysis: AnalysisConfig | None = None): self.artifact_store: ArtifactStore = ArtifactStore() def add_input_metadata(self, meta): - """Add top-level profiler fields with metadata for the module to emulate.""" + """Set the metadata of the input, such as the path and the SHA-256, for the report.""" self.meta = meta def set_start_time(self): - """Record the start time of a sample to time the execution length.""" + """Set the start time of the report clock to now.""" self.start_time = time.time() def get_run_time(self): - """Get the time spent emulating a specific run.""" + """Get the number of seconds since the last call of ``set_start_time``.""" return time.time() - self.start_time def stop_run_clock(self): - """Stop the runtime clock to include in the report.""" + """Save the time since the last call of ``set_start_time`` as the runtime of the report.""" self.runtime = self.get_run_time() def add_run(self, run: Run) -> None: - """Add a new run to the captured run list.""" + """Add a run to the report.""" self.runs.append(run) def put_binary_data(self, data: bytes, limit: int | None = None) -> str | None: - """Store binary data and return its artifact reference.""" + """Store the data, cut to ``limit`` bytes, and return its artifact reference.""" if not data: return None payload = data[:limit] if limit else data return self.artifact_store.put_bytes(payload) def merge_binary_data(self, artifact_ref: str | None, data: bytes, limit: int | None = None) -> str | None: - """Append raw bytes to an existing artifact payload and store the merged result.""" + """Append the data to an artifact, cut the result to ``limit`` bytes, and return the new reference.""" if not data and artifact_ref: return artifact_ref if not artifact_ref: @@ -222,7 +221,7 @@ def merge_binary_data(self, artifact_ref: str | None, data: bytes, limit: int | return self.artifact_store.put_bytes(merged) def record_error_event(self, error: ErrorInfo) -> None: - """Log a top level emulator error for the emulation report.""" + """Add an error to the top-level ``errors`` list of the report.""" if not self.meta.get("errors"): self.meta["errors"] = [] self.meta["errors"].append(error) @@ -248,12 +247,12 @@ def record_api_event( deduplicate: bool = True, ) -> ApiEvent: """ - Log a call to an OS API. This includes arguments, return address, and return value. + Record a call to an API, with the arguments and the return value. - A call is recorded once when it repeats the API, address, arguments and - return value of one of the last few events. Raw argument values do not - count, so a loop that passes the same string from another buffer stays - one event. + If ``deduplicate`` is true, the profiler does not add a call that has the same API, pc, + shown arguments and return value as one of the last 3 events of the run. The shown + argument values are compared, not the raw values, so a loop that passes the same string + from a different buffer gives one event. """ run.num_apis += 1 @@ -289,9 +288,10 @@ def record_file_access_event( size=None, ): """ - Log file access events. + Record a file event: create, open, read, write or delete of a file, or create or delete of a directory. - This includes handles being opened, data reads, and data writes. + A read or a write merges with the last event of the same type and path in the run. The + data preview keeps the first 1024 bytes. Raises: ValueError: if ``event_type`` is not a file event type. @@ -376,9 +376,7 @@ def record_registry_access_event( size=None, ): """ - Log registry access events. - - This includes values and keys being accessed, read, and written. + Record a registry event: open, create, list or delete of a key, or read, write or delete of a value. Raises: ValueError: if ``event_type`` is not a registry event type. @@ -455,16 +453,14 @@ def record_registry_access_event( def record_process_event(self, run, pos: TracePosition, proc, event_type: EventType, kwargs): """ - Log events related to a process accessing another process. + Record a process, memory or thread event. - This includes creating a child process, reading or writing to a process, and creating - a thread within another process. + ``pos`` gives the process and the thread that do the operation, and ``proc`` is the target + process. The ``pid`` of the event is the target process ID, or ``None`` when the target is + the process of the actor. - ``pos`` is the actor's execution position (the process/thread that performed - the operation); ``proc`` is the target. The target process id is carried as - event payload (``pid``) so the event stays in the actor's timeline. When the - target is the actor's own process, the payload ``pid`` is left NULL, so a - populated ``pid`` always means the operation crossed a process boundary. + A memory read or write merges with the last process event when that event is of the same + type and its range ends at the start of the new range. Raises: ValueError: if ``event_type`` is not a process, memory or thread event type. @@ -579,7 +575,7 @@ def record_process_event(self, run, pos: TracePosition, proc, event_type: EventT self.last_event = event def record_dns_event(self, run, pos: TracePosition, domain, ip=""): - """Log DNS name lookups for the emulation report.""" + """Record a DNS lookup. The profiler adds one event for each query and response in a run.""" for evt in run.events: if isinstance(evt, NetDnsEvent) and evt.query == domain and evt.response == ip: return @@ -592,7 +588,7 @@ def record_dns_event(self, run, pos: TracePosition, domain, ip=""): run.events.append(event) def record_http_event(self, run, pos: TracePosition, server, port, headers="", body=b"", secure=False): - """Log HTTP traffic that occurs during emulation.""" + """Record an HTTP request. The profiler adds one event for each server, port, protocol and headers in a run.""" event = NetHttpEvent( pos=pos, server=server, @@ -615,7 +611,7 @@ def record_http_event(self, run, pos: TracePosition, server, port, headers="", b run.events.append(event) def record_dyn_code_event(self, run, tag, base, size): - """Log code that the emulated program generates at runtime and then executes.""" + """Record a dynamic code region that the guest runs. The profiler adds each base address once per run.""" if base not in run.dyn_code["base_addrs"]: entry: DynCodeMapping = {"tag": tag, "base": base, "size": size} run.dyn_code["mmap"].append(entry) @@ -632,7 +628,7 @@ def record_network_event( data=b"", method="", ): - """Log network activity for an emulation run.""" + """Record socket traffic.""" data_ref = self.put_binary_data(data or b"", limit=0x3000) event = NetTrafficEvent( @@ -659,7 +655,7 @@ def record_exception_event( pc_offset=None, stack=None, ): - """Log a handled exception event.""" + """Record an exception that the emulator sends to a guest handler.""" event = ExceptionEvent( pos=pos, instruction=instruction, @@ -674,12 +670,12 @@ def record_exception_event( run.events.append(event) def get_json_report(self) -> str: - """Get the execution profile of the emulator as a JSON string.""" + """Get the report as JSON. Fields with a ``None`` value are left out.""" report = self.get_report() return report.model_dump_json(indent=4, exclude_none=True) def get_report(self) -> Report: - """Get the execution profile of the emulator.""" + """Build the report from the runs, the strings and the artifacts.""" entry_points = [] for r in self.runs: diff --git a/speakeasy/profiler_events.py b/speakeasy/profiler_events.py index 4e444e015..93ab11dbf 100644 --- a/speakeasy/profiler_events.py +++ b/speakeasy/profiler_events.py @@ -10,7 +10,7 @@ class EventType(str, Enum): """ Event type discriminators, as they appear in the ``event`` field of a report event. - Use ``StrEnum`` when the minimum Python version is 3.11. + The class uses ``str, Enum`` because ``StrEnum`` needs Python 3.11. """ API = "api" @@ -54,65 +54,28 @@ class StackEntry(BaseModel): class TracePosition(BaseModel): - """ - Locates where an event occurred in execution time and context. - - Every emitted event carries this structure so consumers can reconstruct - chronology and thread/process ownership. - - Use ``tick`` ordering for timeline reconstruction and ``pc`` for direct - disassembly alignment. - """ + """Position of an event: the instruction count, the thread, the process and the pc.""" model_config = ConfigDict(extra="forbid") tick: int = Field( - description=( - "Instruction-count tick at the time of the event.\n\n" - "Use this as the primary event ordering key within a run." - ) - ) - tid: int = Field( - description=( - "Thread identifier active when the event was emitted.\n\n" - "Correlate this with thread creation/injection activity." - ) - ) - pid: int = Field( - description=( - "Process identifier active when the event was emitted.\n\n" - "Use this to separate behavior across process boundaries." - ) + description="Instruction count of the run at the time of the event. The events of a run are in tick order." ) + tid: int = Field(description="ID of the thread that did the operation.") + pid: int = Field(description="ID of the process that did the operation.") pc: HexIntOptional = Field( default=None, - description=( - "Program counter at event emission time, when available.\n\n" - "Some synthetic events may omit this if no precise PC context exists." - ), + description="Address of the instruction that did the operation. For an API event, the address of the call.", ) class Event(BaseModel): - """ - Base class shared by all report event records. - - Concrete event types extend this with event-specific payload fields. - - Consumers should discriminate event payloads using the ``event`` field. - """ + """Base class of the report events. The ``event`` field gives the event type.""" model_config = ConfigDict(extra="forbid") - pos: TracePosition = Field( - description=( - "Execution position metadata for this event.\n\n" - "Used to align event payloads with timeline and execution context." - ) - ) - event: EventType = Field( - description=("Event type discriminator.\n\nDetermines which concrete payload schema applies.") - ) + pos: TracePosition = Field(description="Position of the thread that did the operation.") + event: EventType = Field(description="Type of the event. The type sets which other fields the event has.") class ApiArgType(str, Enum): @@ -166,589 +129,416 @@ class ApiArg(BaseModel): class ApiEvent(Event): """ - Records one intercepted API invocation. + One call to an API. - API events are emitted by import-call handlers and represent core behavioral - telemetry for most runs. - - Use these entries to understand control flow and argument-level intent. + The profiler leaves out a call that repeats the API, pc, shown arguments and return value of + one of the last 3 events of the run. """ - event: Literal[EventType.API] = Field(default=EventType.API, description="Discriminator for API call events.") - api_name: str = Field( - description=( - "Resolved API name in ``module.function`` form.\n\n" - "Used for behavior signatures, clustering, and family comparison." - ) - ) + event: Literal[EventType.API] = Field(default=EventType.API, description="Event type of an API call.") + api_name: str = Field(description="API name in ``module.function`` form, such as ``kernel32.CreateFileA``.") args: list[ApiArg] = Field( description=( - "Arguments of the call. With a usable API signature there is one entry per parameter; " - "otherwise one entry per argument slot, plus any entries a variadic handler adds." + "Arguments of the call. With a usable API signature, one entry for each parameter. " + "Else, one entry for each argument slot. A variadic handler can add entries." ) ) return_value: HexIntOptional = Field( default=None, - description=( - "Return value of the API call, when available.\n\n" - "Use this to identify failure codes and branch-driving outcomes." - ), + description="Return value of the API, when the handler gives one.", ) class ProcessCreateEvent(Event): - """ - Records creation of a child process context. - - Emitted when process-creation APIs are invoked and accepted by emulation. - - These events indicate lateral execution or staged payload spawning. - """ + """A process that the guest creates, such as with ``CreateProcess`` or ``ShellExecute``.""" event: Literal[EventType.PROCESS_CREATE] = Field( default=EventType.PROCESS_CREATE, - description="Discriminator for process creation events.", - ) - path: str = Field( - description=( - "Target process image path.\n\nRepresents the emulated executable location requested by the sample." - ) - ) - command_line: str = Field( - description=( - "Command line used for process creation.\n\nUse this to recover execution parameters and staged arguments." - ) + description="Event type of a process creation.", ) + path: str = Field(description="Emulated path of the image of the new process.") + command_line: str = Field(description="Command line of the new process.") pid: int | None = Field( default=None, - description=( - "Target process identifier.\n\nThe actor (event pos) creates this process; the target may differ." - ), + description="ID of the new process.", ) class MemAllocEvent(Event): - """ - Records remote or explicit process memory allocation activity. - - Emitted by memory allocation APIs that target process virtual memory. - - Commonly seen in injection and unpacking workflows. - """ + """Memory that the guest allocates with ``VirtualAlloc`` or ``VirtualAllocEx``.""" event: Literal[EventType.MEM_ALLOC] = Field( - default=EventType.MEM_ALLOC, description="Discriminator for memory allocation events." + default=EventType.MEM_ALLOC, description="Event type of a memory allocation." ) - path: str = Field(description="Target process path associated with the allocation event.") - base: HexInt = Field(description="Allocated base address.") - size: int = Field(description="Allocation size in bytes.") + path: str = Field(description="Image path of the target process.") + base: HexInt = Field(description="Base address of the allocation.") + size: int = Field(description="Size that the caller asked for, in bytes.") protect: list[str] | None = Field( default=None, description="Names of the requested page protection flags, such as ``PAGE_READWRITE``.", ) pid: int | None = Field( default=None, - description=( - "Target process identifier.\n\nThe actor (event pos) allocates in this process; NULL when the " - "target is the actor itself." - ), + description="ID of the target process. Absent when the target is the process in ``pos``.", ) class MemWriteEvent(Event): """ - Records writes into process memory. + Data that the guest writes to process memory with an API, such as ``WriteProcessMemory``. - Emitted for API-mediated memory writes and may merge adjacent writes in the - profiler for compactness. - - Use with thread/process events to identify code injection flows. + The profiler merges a write into the last process event when that event is a write that ends + at the start of the new write. """ event: Literal[EventType.MEM_WRITE] = Field( - default=EventType.MEM_WRITE, description="Discriminator for memory write events." + default=EventType.MEM_WRITE, description="Event type of a memory write." ) - path: str = Field(description="Target process path associated with the write event.") - base: HexInt = Field(description="Write start address.") + path: str = Field(description="Image path of the target process.") + base: HexInt = Field(description="Start address of the write.") size: int = Field(description="Number of bytes written.") data_ref: str | None = Field( default=None, - description=("Reference into the top-level report ``data`` store for the captured write preview bytes."), + description="Key in the top-level ``data`` of the first bytes of the write.", ) pid: int | None = Field( default=None, - description=( - "Target process identifier.\n\nThe actor (event pos) writes into this process; NULL when the " - "target is the actor itself." - ), + description="ID of the target process. Absent when the target is the process in ``pos``.", ) class MemReadEvent(Event): """ - Records reads from process memory. + Data that the guest reads from process memory with an API, such as ``ReadProcessMemory``. - Emitted for API-mediated memory reads and may merge adjacent reads in the - profiler for compactness. - - Useful for identifying memory scraping or credential theft behavior. + The profiler merges a read into the last process event when that event is a read that ends at + the start of the new read. """ - event: Literal[EventType.MEM_READ] = Field( - default=EventType.MEM_READ, description="Discriminator for memory read events." - ) - path: str = Field(description="Target process path associated with the read event.") - base: HexInt = Field(description="Read start address.") + event: Literal[EventType.MEM_READ] = Field(default=EventType.MEM_READ, description="Event type of a memory read.") + path: str = Field(description="Image path of the target process.") + base: HexInt = Field(description="Start address of the read.") size: int = Field(description="Number of bytes read.") data_ref: str | None = Field( default=None, - description="Reference into the top-level report ``data`` store for the captured read preview bytes.", + description="Key in the top-level ``data`` of the first bytes of the read.", ) pid: int | None = Field( default=None, - description=( - "Target process identifier.\n\nThe actor (event pos) reads from this process; NULL when the " - "target is the actor itself." - ), + description="ID of the target process. Absent when the target is the process in ``pos``.", ) class MemProtectEvent(Event): - """ - Records memory protection changes. - - Emitted for API paths that alter page permissions in process memory. - - This is a key indicator for RWX transitions and shellcode staging. - """ + """A change of page protection with ``VirtualProtectEx``.""" event: Literal[EventType.MEM_PROTECT] = Field( default=EventType.MEM_PROTECT, - description="Discriminator for memory protection change events.", + description="Event type of a page protection change.", ) - path: str = Field(description="Target process path associated with the protection change.") - base: HexInt = Field(description="Protection-change base address.") - size: int = Field(description="Protection-change size in bytes.") + path: str = Field(description="Image path of the target process.") + base: HexInt = Field(description="Base address of the range.") + size: int = Field(description="Size of the range in bytes.") protect: list[str] | None = Field( default=None, description="Names of the new page protection flags, such as ``PAGE_EXECUTE_READ``.", ) pid: int | None = Field( default=None, - description=( - "Target process identifier.\n\nThe actor (event pos) changes protection in this process; NULL when " - "the target is the actor itself." - ), + description="ID of the target process. Absent when the target is the process in ``pos``.", ) class MemFreeEvent(Event): - """ - Records memory free/decommit operations. + """Memory that the guest frees.""" - Emitted when memory free APIs are invoked against process regions. - - Use this to track lifecycle of allocation artifacts. - """ - - event: Literal[EventType.MEM_FREE] = Field( - default=EventType.MEM_FREE, description="Discriminator for memory free events." - ) - path: str = Field(description="Target process path associated with the free event.") - base: HexInt = Field(description="Freed region base address.") - size: int = Field(description="Freed region size in bytes.") + event: Literal[EventType.MEM_FREE] = Field(default=EventType.MEM_FREE, description="Event type of a memory free.") + path: str = Field(description="Image path of the target process.") + base: HexInt = Field(description="Base address of the freed region.") + size: int = Field(description="Size of the freed region in bytes.") pid: int | None = Field( default=None, - description=( - "Target process identifier.\n\nThe actor (event pos) frees memory in this process; NULL when the " - "target is the actor itself." - ), + description="ID of the target process. Absent when the target is the process in ``pos``.", ) class ModuleLoadEvent(Event): - """ - Records dynamic module loading activity. - - Emitted when module load operations map additional images during runtime. - - Useful for dependency discovery and staged import behavior. - """ + """A module that the guest loads at run time.""" event: Literal[EventType.MODULE_LOAD] = Field( - default=EventType.MODULE_LOAD, description="Discriminator for module load events." + default=EventType.MODULE_LOAD, description="Event type of a module load." ) - name: str = Field(description="Loaded module name.") - path: str = Field(description="Loaded module path in emulated filesystem space.") - base: HexInt = Field(description="Module base address.") - size: int = Field(description="Mapped module size in bytes.") + name: str = Field(description="Name of the module.") + path: str = Field(description="Emulated path of the module.") + base: HexInt = Field(description="Base address of the module.") + size: int = Field(description="Size of the module image in bytes.") class ThreadCreateEvent(Event): - """ - Records thread creation in a process context. - - Emitted for thread creation APIs and often accompanies process injection - workflows. - - Combine with memory-write and start-address analysis for attribution. - """ + """A thread that the guest creates in its own process.""" event: Literal[EventType.THREAD_CREATE] = Field( default=EventType.THREAD_CREATE, - description="Discriminator for thread creation events.", + description="Event type of a thread creation.", ) - path: str = Field(description="Target process path where the thread is created.") - start_address: HexInt = Field(description="Thread start address.") - param: HexInt = Field(description="Thread parameter pointer or value.") + path: str = Field(description="Image path of the target process.") + start_address: HexInt = Field(description="Start address of the thread.") + param: HexInt = Field(description="Parameter that the thread routine gets.") pid: int | None = Field( default=None, - description=( - "Target process identifier.\n\nThe actor (event pos) creates the thread in this process; NULL when " - "the target is the actor itself." - ), + description="ID of the target process. Absent when the target is the process in ``pos``.", ) tid: int | None = Field( default=None, - description="Target thread identifier of the newly created thread, when known.", + description="ID of the new thread.", ) class ThreadInjectEvent(Event): - """ - Records thread injection activity into another process. - - Emitted for injection-oriented thread APIs. - - Treat these as high-signal indicators of code injection behavior. - """ + """A thread that the guest creates in another process, such as with ``CreateRemoteThread``.""" event: Literal[EventType.THREAD_INJECT] = Field( default=EventType.THREAD_INJECT, - description="Discriminator for thread injection events.", + description="Event type of a thread injection.", ) - path: str = Field(description="Target process path receiving the injected thread.") - start_address: HexInt = Field(description="Injected thread start address.") - param: HexInt = Field(description="Injected thread parameter pointer or value.") + path: str = Field(description="Image path of the target process.") + start_address: HexInt = Field(description="Start address of the thread.") + param: HexInt = Field(description="Parameter that the thread routine gets.") pid: int | None = Field( default=None, - description=( - "Target process identifier receiving the injected thread.\n\nNULL when the target is the actor itself." - ), + description="ID of the target process. Absent when the target is the process in ``pos``.", ) tid: int | None = Field( default=None, - description="Target thread identifier of the injected thread, when known.", + description="ID of the new thread.", ) class FileCreateEvent(Event): - """ - Records file creation/open-with-create semantics. - - Emitted when file APIs request creation semantics for a path. - - Use with follow-up write events to track dropped artifacts. - """ + """A file that the guest creates, or opens with a disposition that can create the file.""" event: Literal[EventType.FILE_CREATE] = Field( - default=EventType.FILE_CREATE, description="Discriminator for file create events." + default=EventType.FILE_CREATE, description="Event type of a file creation." ) - path: str = Field(description="File path targeted by creation request.") - handle: HexIntOptional = Field(default=None, description="Assigned file handle, when available.") + path: str = Field(description="Emulated path of the file.") + handle: HexIntOptional = Field(default=None, description="File handle that the API returns.") open_flags: list[str] | None = Field( default=None, - description="Creation/open disposition flags supplied by the API call.", + description="Names of the disposition flags of the call, such as ``CREATE_ALWAYS``.", ) access_flags: list[str] | None = Field( default=None, - description="Requested access rights for the file operation.", + description="Names of the access rights that the caller asks for, such as ``GENERIC_READ``.", ) class FileOpenEvent(Event): - """ - Records file open operations without explicit create intent. + """A file that the guest opens without a create disposition.""" - Emitted for open-style file API calls. - - Use to reconstruct file-read dependency chains. - """ - - event: Literal[EventType.FILE_OPEN] = Field( - default=EventType.FILE_OPEN, description="Discriminator for file open events." - ) - path: str = Field(description="File path targeted by open request.") - handle: HexIntOptional = Field(default=None, description="Assigned file handle, when available.") + event: Literal[EventType.FILE_OPEN] = Field(default=EventType.FILE_OPEN, description="Event type of a file open.") + path: str = Field(description="Emulated path of the file.") + handle: HexIntOptional = Field(default=None, description="File handle that the API returns.") open_flags: list[str] | None = Field( default=None, - description="Open disposition flags supplied by the API call.", + description="Names of the disposition flags of the call, such as ``OPEN_EXISTING``.", ) access_flags: list[str] | None = Field( default=None, - description="Requested access rights for the file operation.", + description="Names of the access rights that the caller asks for, such as ``GENERIC_READ``.", ) class FileReadEvent(Event): """ - Records reads from a file handle. + Data that the guest reads from a file. - Emitted when file-read APIs pull bytes from emulated files. - - Useful for identifying configuration ingestion and payload loading. + The profiler merges all reads of one path in a run into one event. """ - event: Literal[EventType.FILE_READ] = Field( - default=EventType.FILE_READ, description="Discriminator for file read events." - ) - path: str = Field(description="File path associated with the read operation.") - handle: HexIntOptional = Field(default=None, description="File handle used by the read operation.") - size: int | None = Field(default=None, description="Number of bytes requested/read.") + event: Literal[EventType.FILE_READ] = Field(default=EventType.FILE_READ, description="Event type of a file read.") + path: str = Field(description="Emulated path of the file.") + handle: HexIntOptional = Field(default=None, description="File handle of the first read.") + size: int | None = Field(default=None, description="Total number of bytes that the guest read.") data_ref: str | None = Field( default=None, - description="Reference into the top-level report ``data`` store for the captured file bytes.", + description="Key in the top-level ``data`` of the first bytes that the guest read.", ) buffer: HexIntOptional = Field( default=None, - description="Destination buffer pointer.", + description="Address of the buffer of the first read.", ) class FileWriteEvent(Event): """ - Records writes to a file handle. - - Emitted when file-write APIs append or overwrite emulated file data. + Data that the guest writes to a file. - Combine with dropped-file summaries to recover persisted payloads. + The profiler merges all writes to one path in a run into one event. """ event: Literal[EventType.FILE_WRITE] = Field( - default=EventType.FILE_WRITE, description="Discriminator for file write events." + default=EventType.FILE_WRITE, description="Event type of a file write." ) - path: str = Field(description="File path associated with the write operation.") - handle: HexIntOptional = Field(default=None, description="File handle used by the write operation.") - size: int | None = Field(default=None, description="Number of bytes written.") + path: str = Field(description="Emulated path of the file.") + handle: HexIntOptional = Field(default=None, description="File handle of the first write.") + size: int | None = Field(default=None, description="Total number of bytes that the guest wrote.") data_ref: str | None = Field( default=None, - description="Reference into the top-level report ``data`` store for the captured file bytes.", + description="Key in the top-level ``data`` of the first bytes that the guest wrote.", ) buffer: HexIntOptional = Field( default=None, - description="Source buffer pointer.", + description="Address of the buffer of the first write.", ) class FileDeleteEvent(Event): - """ - Records file deletion. - - Emitted when file APIs delete a file. - - Use with file create events to track files that a sample removes after use. - """ + """A file that the guest deletes.""" event: Literal[EventType.FILE_DELETE] = Field( - default=EventType.FILE_DELETE, description="Discriminator for file delete events." + default=EventType.FILE_DELETE, description="Event type of a file deletion." ) - path: str = Field(description="File path targeted by the delete request.") + path: str = Field(description="Emulated path of the file.") class DirectoryCreateEvent(Event): - """ - Records directory creation. - - Emitted when file APIs create a directory. - - Use with file create events to track where a sample drops files. - """ + """A directory that the guest creates.""" event: Literal[EventType.DIRECTORY_CREATE] = Field( - default=EventType.DIRECTORY_CREATE, description="Discriminator for directory create events." + default=EventType.DIRECTORY_CREATE, description="Event type of a directory creation." ) - path: str = Field(description="Directory path targeted by the create request.") + path: str = Field(description="Emulated path of the directory.") class DirectoryDeleteEvent(Event): - """ - Records directory removal. - - Emitted when file APIs remove a directory. - - Use with directory create events to track staging directories. - """ + """A directory that the guest deletes.""" event: Literal[EventType.DIRECTORY_DELETE] = Field( - default=EventType.DIRECTORY_DELETE, description="Discriminator for directory delete events." + default=EventType.DIRECTORY_DELETE, description="Event type of a directory deletion." ) - path: str = Field(description="Directory path targeted by the remove request.") + path: str = Field(description="Emulated path of the directory.") class RegOpenKeyEvent(Event): - """ - Records registry key open operations. - - Emitted when registry APIs open existing keys. - - Use these to identify configuration and persistence probing paths. - """ + """A registry key that the guest opens.""" event: Literal[EventType.REG_OPEN_KEY] = Field( default=EventType.REG_OPEN_KEY, - description="Discriminator for registry key open events.", + description="Event type of a registry key open.", ) - path: str = Field(description="Registry key path targeted by the open operation.") - handle: HexIntOptional = Field(default=None, description="Registry handle returned for the key, when available.") + path: str = Field(description="Path of the key.") + handle: HexIntOptional = Field(default=None, description="Key handle that the API returns.") open_flags: list[str] | None = Field( default=None, - description="Open/disposition flags associated with the operation.", + description="Names of the open flags of the call.", ) access_flags: list[str] | None = Field( default=None, - description="Requested registry access rights.", + description="Names of the access rights that the caller asks for.", ) class RegCreateKeyEvent(Event): - """ - Records registry key creation operations. - - Emitted when registry APIs create new keys. - - Strong signal for persistence and installation behavior. - """ + """A registry key that the guest creates or opens with a create API, such as ``RegCreateKeyEx``.""" event: Literal[EventType.REG_CREATE_KEY] = Field( default=EventType.REG_CREATE_KEY, - description="Discriminator for registry key create events.", + description="Event type of a registry key creation.", ) - path: str = Field(description="Registry key path targeted by the create operation.") - handle: HexIntOptional = Field(default=None, description="Registry handle returned for the key, when available.") + path: str = Field(description="Path of the key.") + handle: HexIntOptional = Field(default=None, description="Key handle that the API returns.") open_flags: list[str] | None = Field( default=None, - description="Create/open disposition flags associated with the operation.", + description="Names of the disposition flags of the call.", ) access_flags: list[str] | None = Field( default=None, - description="Requested registry access rights.", + description="Names of the access rights that the caller asks for.", ) class RegReadValueEvent(Event): - """ - Records registry value read operations. - - Emitted when value-query APIs request data from a key. - - Use this to identify configuration keys and anti-analysis checks. - """ + """A registry value that the guest reads.""" event: Literal[EventType.REG_READ_VALUE] = Field( default=EventType.REG_READ_VALUE, - description="Discriminator for registry value read events.", + description="Event type of a registry value read.", ) - path: str = Field(description="Registry key path containing the queried value.") - handle: HexIntOptional = Field(default=None, description="Registry key handle used for the read, when available.") - value_name: str | None = Field(default=None, description="Registry value name requested by the API call.") - size: int | None = Field(default=None, description="Requested/returned value size in bytes.") + path: str = Field(description="Path of the key that has the value.") + handle: HexIntOptional = Field(default=None, description="Key handle of the call.") + value_name: str | None = Field(default=None, description="Name of the value.") + size: int | None = Field(default=None, description="Size of the value data in bytes.") data_ref: str | None = Field( default=None, - description="Reference into the top-level report ``data`` store for the captured registry value bytes.", + description="Key in the top-level ``data`` of the first bytes of the value data.", ) - buffer: HexIntOptional = Field(default=None, description="Destination buffer pointer.") + buffer: HexIntOptional = Field(default=None, description="Address of the output buffer.") class RegWriteValueEvent(Event): - """ - Records registry value write operations. - - Emitted when APIs set or overwrite a registry value. - - Use this to capture persistence and configuration mutations. - """ + """A registry value that the guest writes.""" event: Literal[EventType.REG_WRITE_VALUE] = Field( default=EventType.REG_WRITE_VALUE, - description="Discriminator for registry value write events.", + description="Event type of a registry value write.", ) - path: str = Field(description="Registry key path containing the written value.") - handle: HexIntOptional = Field(default=None, description="Registry key handle used for the write, when available.") - value_name: str | None = Field(default=None, description="Registry value name written by the API call.") - size: int | None = Field(default=None, description="Written value size in bytes.") + path: str = Field(description="Path of the key that has the value.") + handle: HexIntOptional = Field(default=None, description="Key handle of the call.") + value_name: str | None = Field(default=None, description="Name of the value.") + size: int | None = Field(default=None, description="Size of the value data in bytes.") data_ref: str | None = Field( default=None, - description="Reference into the top-level report ``data`` store for the written registry value bytes.", + description="Key in the top-level ``data`` of the first bytes of the value data.", ) - buffer: HexIntOptional = Field(default=None, description="Source buffer pointer.") + buffer: HexIntOptional = Field(default=None, description="Address of the input buffer.") class RegListSubkeysEvent(Event): - """ - Records enumeration of registry subkeys. - - Emitted when APIs list child keys beneath a registry path. - - Helps reveal discovery behavior for installed software and services. - """ + """A registry key whose subkeys the guest lists.""" event: Literal[EventType.REG_LIST_SUBKEYS] = Field( default=EventType.REG_LIST_SUBKEYS, - description="Discriminator for registry subkey enumeration events.", + description="Event type of a subkey list.", ) - path: str = Field(description="Registry key path being enumerated.") - handle: HexIntOptional = Field(default=None, description="Registry handle used for enumeration, when available.") + path: str = Field(description="Path of the key.") + handle: HexIntOptional = Field(default=None, description="Key handle of the call.") class RegDeleteKeyEvent(Event): - """ - Records registry key deletion. - - Emitted when registry APIs delete a key. - - Use this to track persistence cleanup and anti-forensics behavior. - """ + """A registry key that the guest deletes.""" event: Literal[EventType.REG_DELETE_KEY] = Field( default=EventType.REG_DELETE_KEY, - description="Discriminator for registry key delete events.", + description="Event type of a registry key deletion.", ) - path: str = Field(description="Registry key path targeted by the delete operation.") - handle: HexIntOptional = Field(default=None, description="Registry handle used for the delete, when available.") + path: str = Field(description="Path of the key.") + handle: HexIntOptional = Field(default=None, description="Key handle of the call.") class RegDeleteValueEvent(Event): - """ - Records registry value deletion. - - Emitted when registry APIs delete a value from a key. - - Use this to track persistence cleanup and anti-forensics behavior. - """ + """A registry value that the guest deletes.""" event: Literal[EventType.REG_DELETE_VALUE] = Field( default=EventType.REG_DELETE_VALUE, - description="Discriminator for registry value delete events.", + description="Event type of a registry value deletion.", ) - path: str = Field(description="Registry key path containing the deleted value.") - handle: HexIntOptional = Field(default=None, description="Registry key handle used for the delete, when available.") - value_name: str | None = Field(default=None, description="Registry value name deleted by the API call.") + path: str = Field(description="Path of the key that had the value.") + handle: HexIntOptional = Field(default=None, description="Key handle of the call.") + value_name: str | None = Field(default=None, description="Name of the value.") class NetDnsEvent(Event): """ - Records DNS name resolution activity. + A host name that the guest looks up. - Emitted when DNS APIs request host resolution or related lookups. - - Use for C2 endpoint discovery and fallback behavior analysis. + The profiler adds one event for each query and response in a run. """ - event: Literal[EventType.NET_DNS] = Field(default=EventType.NET_DNS, description="Discriminator for DNS events.") - query: str = Field(description="Queried domain or hostname.") + event: Literal[EventType.NET_DNS] = Field(default=EventType.NET_DNS, description="Event type of a DNS lookup.") + query: str = Field(description="Host name of the lookup.") response: str | None = Field( default=None, - description="Resolved IP response, when a mapping was available.", + description="IP address from ``network.dns.names`` in the config. Absent when the config has no address.", ) @@ -779,72 +569,56 @@ class NetHttpProtocol(str, Enum): class NetTrafficEvent(Event): - """ - Records non-HTTP network traffic and socket-level activity. - - Emitted for connect/bind/send/recv style network events. - - Use this for protocol flow reconstruction outside HTTP abstractions. - """ + """A socket operation of the guest: connect, bind, accept, send or receive.""" event: Literal[EventType.NET_TRAFFIC] = Field( - default=EventType.NET_TRAFFIC, description="Discriminator for socket traffic events." + default=EventType.NET_TRAFFIC, description="Event type of a socket operation." ) - server: str = Field(description="Remote endpoint hostname or IP address.") - port: int = Field(description="Remote endpoint port number.") + server: str = Field(description="Host name or IP address of the socket peer.") + port: int = Field(description="Port of the socket peer.") proto: NetTrafficProtocol = Field( description="Protocol of the socket type: ``tcp``, ``udp``, ``raw``, or ``unknown`` for another socket type." ) - type: NetTrafficType | None = Field(default=None, description="Socket operation that produced the event.") + type: NetTrafficType | None = Field(default=None, description="Socket operation of the event.") data_ref: str | None = Field( default=None, - description="Reference into the top-level report ``data`` store for the captured traffic payload bytes.", + description="Key in the top-level ``data`` of the first bytes that the guest sent or received.", ) - method: str | None = Field(default=None, description="Source API path or method that produced the event.") + method: str | None = Field(default=None, description="Name of the operation, such as ``winsock.connect``.") class NetHttpEvent(Event): """ - Records HTTP request telemetry. + An HTTP request that the guest sends with WinINet or WinHTTP. - Emitted by WinInet/WinHTTP handlers when requests are issued. - - Use these records to inspect request metadata and transmitted body content. + The profiler adds one event for each server, port, protocol and headers in a run. """ - event: Literal[EventType.NET_HTTP] = Field(default=EventType.NET_HTTP, description="Discriminator for HTTP events.") - server: str = Field(description="HTTP server hostname or address.") - port: int = Field(description="HTTP server port.") + event: Literal[EventType.NET_HTTP] = Field(default=EventType.NET_HTTP, description="Event type of an HTTP request.") + server: str = Field(description="Host name or address of the server.") + port: int = Field(description="Port of the server.") proto: NetHttpProtocol = Field(description="``http``, or ``https`` for a secure request.") - headers: str | None = Field(default=None, description="Serialized HTTP request headers, when available.") + headers: str | None = Field(default=None, description="Headers of the request, as text.") data_ref: str | None = Field( default=None, - description="Reference into the top-level report ``data`` store for the captured HTTP body bytes.", + description="Key in the top-level ``data`` of the first bytes of the request body.", ) class ExceptionEvent(Event): - """ - Records handled exception dispatch activity. + """An exception that the emulator sends to a guest handler.""" - Emitted when exception handling telemetry is captured during execution. - - Use this to understand exception-driven control flow and anti-analysis logic. - """ - - event: Literal[EventType.EXCEPTION] = Field( - default=EventType.EXCEPTION, description="Discriminator for exception events." - ) - instruction: str = Field(description="Instruction text associated with the exception.") + event: Literal[EventType.EXCEPTION] = Field(default=EventType.EXCEPTION, description="Event type of an exception.") + instruction: str = Field(description="Disassembly of the instruction at the exception address.") exception_code: HexInt = Field(description="Exception code, such as ``0xc0000005``.") - handler_address: HexInt = Field(description="Address of the selected exception handler.") - registers: dict[str, HexInt] = Field(description="Register snapshot captured at exception time.") + handler_address: HexInt = Field(description="Address of the handler that gets the exception.") + registers: dict[str, HexInt] = Field(description="Register values at the time of the exception.") address: HexIntOptional = Field( default=None, description="Memory address that caused an access violation or a guard page violation." ) pc_module: str | None = Field(default=None, description="Name of the module that contains the faulting PC.") pc_offset: int | None = Field(default=None, description="Offset of the faulting PC from the base of ``pc_module``.") - stack: list[StackEntry] | None = Field(default=None, description="Stack slots at exception time.") + stack: list[StackEntry] | None = Field(default=None, description="Stack slots at the time of the exception.") AnyEvent = Annotated[ diff --git a/speakeasy/report.py b/speakeasy/report.py index 947d418e2..86ff72146 100644 --- a/speakeasy/report.py +++ b/speakeasy/report.py @@ -9,75 +9,47 @@ class StringCollection(BaseModel): - """ - Represents one categorized bucket of extracted strings. - - String extraction runs both on static image bytes and post-emulation memory - snapshots. This model keeps ANSI and UTF-16 findings separated. - - Compare these lists to identify runtime-decoded strings that were not present - in the original file bytes. - """ + """ANSI and UTF-16 strings from one source.""" model_config = ConfigDict(extra="forbid") ansi: list[str] = Field( default_factory=list, - description=( - "ASCII/ANSI strings extracted for this bucket.\n\n" - "These are populated by static string scans or memory string extraction " - "depending on the parent field." - ), + description="ASCII strings.", ) unicode: list[str] = Field( default_factory=list, - description=( - "UTF-16 style strings extracted for this bucket.\n\n" - "Use these to spot wide-character configuration values and command " - "artifacts." - ), + description="UTF-16LE strings.", ) class StringsReport(BaseModel): - """ - Groups static and in-memory string extraction results. - - Static strings come from the original loaded image, while in-memory strings - are collected after execution and often include decoded/decrypted content. - - Use this section to quickly triage embedded versus runtime materialized - indicators. - """ + """Strings of the input file, and strings that the guest made in memory.""" model_config = ConfigDict(extra="forbid") static: StringCollection = Field( default_factory=StringCollection, - description=( - "Strings extracted from the initial file image bytes.\n\n" - "Primarily influenced by the module loader and static string scanner." - ), + description="Strings in the data of the input file.", ) in_memory: StringCollection = Field( default_factory=StringCollection, description=( - "Strings extracted from memory after emulation executes.\n\n" - "Populated when string analysis is enabled and runtime decoding created " - "new string material." + "Strings in the stack and the heaps at the end of the emulation that are not in ``static``. " + "These strings often are decoded data." ), ) class RegionInfo(BaseModel): - """Describes a memory region for exception context.""" + """A memory region near a fault.""" model_config = ConfigDict(extra="forbid") - tag: str = Field(description="Memory-map tag identifying the region.") + tag: str = Field(description="Tag of the memory region.") base: HexInt = Field(description="Base address of the region.") - size: int = Field(description="Region size in bytes.") - perms: str | None = Field(default=None, description="Permission string (e.g. ``r-x``).") + size: int = Field(description="Size of the region in bytes.") + perms: str | None = Field(default=None, description="Permissions, such as ``r-x``.") class ErrorType(str, Enum): @@ -118,50 +90,38 @@ class AccessType(str, Enum): class ErrorInfo(BaseModel): """ - Captures failure context for top-level or per-entry-point errors. - - Error records can represent unsupported API termination, instruction faults, - explicit runtime limits, or unhandled exceptions. + Why a run, or the emulation, ended with an error. - Inspect this structure first when a run exits unexpectedly. + The emulator fills only the fields that apply to the error ``type``. """ model_config = ConfigDict(extra="forbid") type: ErrorType = Field( description=( - "Error category identifier describing the failure mode.\n\n" - "Examples include unsupported API paths, max API limits, and fault " - "conditions raised by emulator hooks. A Python exception in Speakeasy " - "itself, such as a bug in an API handler, is ``internal_error``." + "Kind of error, such as ``unsupported_api``, ``max_api_count`` or ``invalid_read``. " + "A Python exception in Speakeasy itself, such as a bug in an API handler, is ``internal_error``." ) ) pc: HexIntOptional = Field( default=None, - description=( - "Program counter associated with the error, when available.\n\n" - "Use this address to align failures with disassembly and event traces." - ), + description="Address of the instruction that caused the error. For ``unsupported_api``, the return address.", ) instruction: str | None = Field( default=None, - description=( - "Faulting instruction text, when known.\n\n" - "Useful for quickly identifying invalid memory operations or bad control " - "flow transitions." - ), + description="Disassembly of the instruction at ``pc``.", ) address: HexIntOptional = Field( default=None, - description="Faulting memory address that triggered the error.", + description="Memory address that the faulting access used.", ) access_type: AccessType | None = Field( default=None, - description="Memory access type that faulted: read, write, or fetch.", + description="Kind of the faulting access: read, write or fetch.", ) registers: dict[str, HexInt] | None = Field( default=None, - description="Register state snapshot at the time of the error.", + description="Register values at the time of the error.", ) stack: list[StackEntry] | None = Field( default=None, @@ -177,21 +137,21 @@ class ErrorInfo(BaseModel): ) address_region: RegionInfo | None = Field( default=None, - description="Memory region containing the faulting address, if mapped.", + description="Memory region that contains ``address``. Absent when ``address`` is not mapped.", ) nearby_regions: list[RegionInfo] | None = Field( default=None, - description="Up to 2 nearest memory regions when the faulting address is unmapped.", + description="The memory regions nearest to ``address`` when ``address`` is not mapped.", ) tid: int | None = Field(default=None, description="Thread ID at the time of the error.") pid: int | None = Field(default=None, description="Process ID at the time of the error.") context_summary: str | None = Field( default=None, - description="Human-readable one-line triage summary of the error.", + description="One line of text that tells what went wrong.", ) traceback: str | None = Field( default=None, - description="Python traceback when the error originated from an internal exception.", + description="Python traceback of an ``internal_error``.", ) message: str | None = Field( default=None, @@ -199,93 +159,49 @@ class ErrorInfo(BaseModel): ) api_name: str | None = Field( default=None, - description="API function name for unsupported-API errors.", + description="Name of the API that has no handler, for ``unsupported_api``.", ) count: int | None = Field( default=None, - description="API call count that triggered a max-API-count limit.", + description=( + "Limit that ended the run: ``max_api_calls`` of the config for ``max_api_count``, " + "or the instruction limit for ``max_instructions``." + ), ) last_api: str | None = Field( default=None, - description="Last API name invoked before max-API-count termination.", + description="Name of the API call that went above the limit, for ``max_api_count``.", ) interrupt_num: int | None = Field( default=None, - description="Interrupt number for unhandled-interrupt errors.", + description="Interrupt number, for ``unhandled_interrupt``.", ) class SymAccessReport(BaseModel): - """ - Summarizes memory accesses to symbol-resolved addresses. - - Symbol access accounting is mainly populated when memory tracing is active - and the emulator can resolve touched addresses back to known symbols. - - Use these counters to identify import-table walking and export resolution - behavior. - """ + """Read, write and execute counts of one symbol address, from memory tracing.""" model_config = ConfigDict(extra="forbid") - symbol: str = Field( - description=( - "Resolved symbol name in ``module.symbol`` form.\n\n" - "This identifies the logical target accessed by code during tracing." - ) - ) - reads: int = Field( - description=( - "Number of read accesses observed for this symbol.\n\n" - "High values often indicate lookup loops or pointer chasing." - ) - ) - writes: int = Field( - description=( - "Number of write accesses observed for this symbol.\n\n" - "Non-zero writes can indicate tampering or patching behavior." - ) - ) - execs: int = Field( - description=( - "Number of execution accesses observed for this symbol.\n\n" - "Execution hits often represent direct calls/jumps through symbol-backed " - "addresses." - ) - ) + symbol: str = Field(description="Symbol name in ``module.symbol`` form.") + reads: int = Field(description="Number of reads of the symbol address.") + writes: int = Field(description="Number of writes to the symbol address.") + execs: int = Field(description="Number of times that the guest ran code at the symbol address.") class DynamicCodeSegment(BaseModel): """ - Describes one dynamically generated code region that executed. - - Dynamic code segments are added when execution reaches memory previously - registered as dynamic code (for example after VirtualAlloc/WriteProcessMemory - style paths). + A dynamic code region where code ran. - This section is useful for unpacking and staged payload detection. + The emulator watches the memory that the guest allocates, for example with ``VirtualAlloc``, and + adds a region when the guest runs code in the region. """ model_config = ConfigDict(extra="forbid") - tag: str = Field( - description=( - "Memory-map tag assigned to the dynamic region.\n\n" - "Tags help correlate the segment with allocation provenance in traces." - ) - ) - base: HexInt = Field( - description=( - "Base address of the dynamic code region.\n\n" - "Use this with memory layout data to inspect surrounding mapped content." - ) - ) - size: int = Field( - description=( - "Region size of the dynamic code segment.\n\n" - "Large regions may indicate unpacked module images or staged blobs." - ) - ) + tag: str = Field(description="Tag of the memory region.") + base: HexInt = Field(description="Base address of the region.") + size: int = Field(description="Size of the region in bytes.") class ArtifactCompression(str, Enum): @@ -302,238 +218,113 @@ class ArtifactEncoding(str, Enum): class DataArtifact(BaseModel): """ - Stores one deduplicated binary payload referenced elsewhere in a report. + One payload of the report. - Payloads are keyed by SHA-256 in the top-level ``Report.data`` mapping. - Other models store the SHA-256 key as a reference instead of embedding bytes - inline repeatedly. + The top-level ``data`` of the report keys each payload by the SHA-256 of its bytes. A ``data_ref`` + field elsewhere in the report holds such a key. """ model_config = ConfigDict(extra="forbid") - compression: ArtifactCompression = Field(description="Compression algorithm applied to the stored payload.") - encoding: ArtifactEncoding = Field(description="Text encoding used for serialized payload bytes.") - size: int = Field(description="Uncompressed payload size in bytes.") - data: str = Field(description="Encoded payload bytes after compression and serialization.") + compression: ArtifactCompression = Field(description="Compression of the payload.") + encoding: ArtifactEncoding = Field(description="Text encoding of the compressed payload.") + size: int = Field(description="Size of the payload before compression, in bytes.") + data: str = Field(description="Payload, compressed and then encoded.") class DroppedFile(BaseModel): - """ - Represents one file artifact written by emulated execution. - - Dropped-file entries are created from filesystem manager state when a run - completes and fully written files are detected. - - Use these records to recover secondary payloads and configuration artifacts. - """ + """A file that the guest wrote completely, up to the end of the run.""" model_config = ConfigDict(extra="forbid") - path: str = Field( - description=( - "Emulated destination path of the written file.\n\n" - "Interpret this as the malware-intended filesystem location." - ) - ) - size: int = Field(description="Dropped file size in bytes.") - sha256: str = Field( - description=( - "SHA-256 hash of the dropped file bytes.\n\nUse this for sample correlation and deduplication workflows." - ) - ) + path: str = Field(description="Emulated path of the file.") + size: int = Field(description="Size of the file in bytes.") + sha256: str = Field(description="SHA-256 of the file data.") data_ref: str | None = Field( default=None, - description=( - "Optional reference into the top-level report ``data`` store.\n\n" - "When present, resolve this SHA-256 key in ``Report.data`` to recover the compressed file bytes." - ), + description="Key in the top-level ``data`` of the file data. Absent for a file above MAX_EMBEDDED_FILE_SIZE.", ) class MemoryAccesses(BaseModel): - """ - Aggregates read/write/execute counters for one memory region. - - These counters are produced by runtime memory tracing hooks when enabled. - - Use them to prioritize regions with meaningful behavioral activity. - """ + """Read, write and execute counts of one memory region, from memory tracing.""" model_config = ConfigDict(extra="forbid") - reads: int = Field( - description=( - "Total read operations observed for the region.\n\n" - "High read density often indicates lookup tables or parsed structures." - ) - ) - writes: int = Field( - description=( - "Total write operations observed for the region.\n\n" - "High write density often indicates decoded payload construction." - ) - ) - execs: int = Field( - description=( - "Total execute operations observed for the region.\n\n" - "Non-zero execution marks code-bearing pages and trampoline regions." - ) - ) + reads: int = Field(description="Number of reads in the region.") + writes: int = Field(description="Number of writes in the region.") + execs: int = Field(description="Number of instructions that ran in the region.") class MemoryRegion(BaseModel): """ - Represents one memory layout entry captured for an entry point. - - Memory layout capture runs at run completion and may include section-split - module regions plus generic heap/stack mappings. + One memory region at the end of a run. - Use this data to correlate execution artifacts with mapped memory topology. + The emulator splits the image of a module into one region for the headers and one region + for each section. """ model_config = ConfigDict(extra="forbid") - tag: str = Field( - description=( - "Region tag assigned by the memory manager.\n\n" - "Tags encode provenance such as module sections, heaps, or emulator " - "internal structures." - ) - ) - base: HexInt = Field( - description=( - "Base address of the memory region.\n\n" - "Treat this as the start of the region for disassembly and dump slicing." - ) - ) - size: int = Field( - description=("Region size in bytes.\n\nCombine with ``base`` to derive half-open address ranges.") - ) - perms: str = Field( - description=( - "Permission string (for example ``r-x`` or ``rw-``).\n\n" - "Interpret this as the final effective page protection at capture time." - ) - ) + tag: str = Field(description="Tag of the region. The tag of a module region starts with ``emu.module.``.") + base: HexInt = Field(description="Base address of the region.") + size: int = Field(description="Size of the region in bytes.") + perms: str = Field(description="Permissions at the end of the run, such as ``r-x`` or ``rw-``.") is_free: bool = Field( default=False, - description=( - "Whether the region had been freed at capture time.\n\n" - "Freed regions can remain visible when memory retention settings keep old " - "maps for analysis." - ), + description="If true, the guest freed the region. Only ``keep_memory_on_free`` keeps a freed region.", ) accesses: MemoryAccesses | None = Field( default=None, - description=( - "Optional read/write/execute counters for the region.\n\n" - "Populated primarily when memory tracing was enabled for the run." - ), + description="Access counts of the region. Absent when memory tracing is off or the run did not use the region.", ) data_ref: str | None = Field( default=None, description=( - "Optional reference into the top-level report ``data`` store.\n\n" - "When present, resolve this SHA-256 key in ``Report.data`` to recover the compressed region bytes." + "Key in the top-level ``data`` of the region data. Only with ``snapshot_memory_regions``, " + "and never for the stack and the heaps." ), ) class ModuleSegment(BaseModel): - """ - Describes one section-like segment within a loaded module. - - Segment data is collected from module metadata during memory layout capture. - - Use this to align addresses with section boundaries and permissions. - """ + """One section of a loaded module.""" model_config = ConfigDict(extra="forbid") - name: str = Field( - description=( - "Segment or section name within the module image.\n\n" - "Useful for mapping behavior to familiar PE sections such as ``.text``." - ) - ) - base: HexInt = Field( - description=( - "Absolute virtual address of the segment start.\n\nUse with ``size`` to identify covered address ranges." - ) - ) - size: int = Field( - description=("Segment size in bytes.\n\nLarge executable segments usually contain primary code paths.") - ) - perms: str = Field( - description=( - "Effective memory permissions for the segment, such as ``r-x``.\n\n" - "Use this to identify writable-executable anomalies or data-only regions." - ) - ) + name: str = Field(description="Name of the section, such as ``.text``.") + base: HexInt = Field(description="Virtual address of the section.") + size: int = Field(description="Virtual size of the section in bytes.") + perms: str = Field(description="Permissions from the section header, such as ``r-x``.") class LoadedModule(BaseModel): - """ - Captures one module visible in the run's loaded-module inventory. - - Module entries are assembled from runtime module objects at run completion. - - This section provides a concise map of what binaries were present and where. - """ + """A module that is loaded at the end of a run.""" model_config = ConfigDict(extra="forbid") - name: str = Field( - description=( - "Module file name or canonical module identifier.\n\n" - "Use this to correlate module events and API import usage." - ) - ) - path: str = Field( - description=("Emulated module path.\n\nUseful for distinguishing system modules from payload modules.") - ) - base: HexInt = Field( - description=( - "Module base address in virtual memory.\n\n" - "Use this as the relocation anchor for offsets and symbol lookups." - ) - ) - size: int = Field( - description=("Total mapped module size in bytes.\n\nHelps identify unusually large or tiny mapped images.") - ) + name: str = Field(description="Name of the module.") + path: str = Field(description="Emulated path of the module.") + base: HexInt = Field(description="Base address of the module.") + size: int = Field(description="Size of the module image in bytes.") segments: list[ModuleSegment] = Field( default_factory=list, - description=( - "Section/segment breakdown for the module mapping.\n\n" - "Use these entries for section-aware interpretation of memory behavior." - ), + description="Sections of the module.", ) class MemoryLayout(BaseModel): - """ - Bundles memory-region and loaded-module snapshots for one run. - - This object is populated when the emulator captures run-end memory state. - - Use it to understand both raw mappings and higher-level module context. - """ + """Memory regions and loaded modules at the end of a run.""" model_config = ConfigDict(extra="forbid") layout: list[MemoryRegion] = Field( default_factory=list, - description=( - "Flat list of captured memory regions for the run.\n\n" - "Entries may include module headers/sections and generic maps." - ), + description="Memory regions, with one region for the headers and one for each section of a module.", ) modules: list[LoadedModule] = Field( default_factory=list, - description=( - "Loaded modules present when the run completed.\n\n" - "Use alongside ``layout`` to interpret address ownership and segment roles." - ), + description="Modules that are loaded at the end of the run.", ) @@ -559,124 +350,79 @@ class EntryPointType(str, Enum): class EntryPoint(BaseModel): """ - Represents one emulation execution primitive captured as a run. - - A run may be a module entry point, callback, exported function, injected - thread, or other execution start context. + One run, and the events and data that the run collected. - Most behavioral telemetry is attached at this level, so treat each entry as - a self-contained execution narrative. + A run starts at one address, such as the module entry point, an export, a TLS callback or a + thread. """ model_config = ConfigDict(extra="forbid") type: EntryPointType = Field( - description=( - "Kind of code that this run executes.\n\n" - "Examples include the module entry point, an export, a TLS callback, and a thread." - ) + description="Kind of code that the run executes, such as ``module_entry``, ``export`` or ``thread``." ) name: str | None = Field( default=None, description=( - "Detail for ``type``, or null when the type has no detail.\n\n" - "For example the export name, the TLS callback index, the dependency module, or the IRP major function." + "Detail for ``type``, such as the export name, the TLS callback index, the dependency module or " + "the IRP major function. Absent when the type has no detail." ), ) - start_address: HexInt = Field( - description=( - "Address where this run began executing.\n\n" - "Use this as the anchor when following control flow in disassembly." - ) - ) + start_address: HexInt = Field(description="Address where the run started.") args: list[HexInt] = Field( default_factory=list, - description="Argument values supplied at run entry.", + description="Arguments that the emulator gives to the code at ``start_address``.", ) pid: int | None = Field( default=None, - description=( - "Process identifier associated with the run context.\n\n" - "Populated when process context is known for this entry point." - ), + description="ID of the process of the run.", ) tid: int | None = Field( default=None, - description=( - "Thread identifier associated with the run context.\n\n" - "Useful for correlating with per-event ``pos.tid`` values." - ), + description="ID of the thread of the run.", ) instruction_count: int | None = Field( default=None, - description=( - "Instruction count executed for this run.\n\n" - "Can be limited by configuration caps or early termination conditions." - ), + description="Number of instructions that the run executed. Absent when the run executed none.", ) api_hash: str = Field( default="", description=( - "SHA-256 hash over unique API names invoked in first-seen order.\n\n" - "Use this for coarse behavioral clustering across samples and architectures." + "SHA-256 of the lowercase names of the APIs that the run called, each name once, " + "in the order of the first call." ), ) return_value: HexIntOptional = Field( default=None, - description=( - "Return value produced when the run ended normally.\n\n" - "Interpret this in the calling convention context of ``type``." - ), + description="Value of the return register when the run completed.", ) error: ErrorInfo | None = Field( default=None, - description=( - "Run-local error details when execution did not complete cleanly.\n\n" - "Consult this with ``events`` to identify the exact failure transition." - ), + description="Error that ended the run. Absent when the run completed.", ) events: list[AnyEvent] = Field( default_factory=list, - description=( - "Chronological event stream emitted during this run.\n\n" - "Event contents are produced by API, process, file, registry, network, " - "and exception recorders." - ), + description="Events of the run, in order.", ) symbol_accesses: list[SymAccessReport] | None = Field( default=None, - description=( - "Symbol-level access counters observed during tracing.\n\n" - "Present only when memory tracing is enabled in config." - ), + description="Access counts of each API symbol. Only with ``analysis.memory_tracing``.", ) dynamic_code_segments: list[DynamicCodeSegment] = Field( default_factory=list, - description=( - "Dynamically generated code regions that executed in this run.\n\n" - "Indicates unpacking, JIT-like behavior, or in-memory staging activity." - ), + description="Dynamic code regions where code ran in the run.", ) coverage: list[HexInt] | None = Field( default=None, - description=( - "Sorted list of executed instruction addresses for coverage analysis.\n\n" - "Present only when coverage collection is enabled in config." - ), + description="Sorted addresses of the instructions that ran. Only with ``analysis.coverage``.", ) dropped_files: list[DroppedFile] = Field( default_factory=list, - description=( - "File artifacts detected as fully written during the run.\n\n" - "Use this list to extract secondary payloads and logs." - ), + description="Files that the guest wrote completely, up to the end of the run.", ) memory: MemoryLayout | None = Field( default=None, - description=( - "Optional memory and module snapshot captured at run completion.\n\n" - "Populate this when memory-layout capture is active for the emulator mode." - ), + description="Memory regions and modules at the end of the run. User mode only.", ) @@ -698,12 +444,7 @@ class FileType(str, Enum): class Report(BaseModel): """ - Top-level Speakeasy output report model. - - This object combines run telemetry, metadata for the emulated input, and - optional analysis artifacts. - - Treat this as the canonical machine-readable output for downstream tooling. + Report of one emulation: the input metadata, the runs and the payloads. Machine words are ``0x``-prefixed hex strings: addresses, pointers, handles, raw argument values, return values, register values and exception @@ -720,119 +461,66 @@ class Report(BaseModel): report_version: str = Field( default=REPORT_VERSION, - description=( - "Schema/report format version generated by the profiler.\n\n" - "Use this to route parsing logic when consuming multiple report versions." - ), + description="Version of the report format.", ) emulation_total_runtime: float = Field( description=( - "Total wall-clock runtime in seconds for the emulation session.\n\n" - "Derived from profiler start/stop timing and rounded to milliseconds." - ) - ) - timestamp: int = Field( - description=( - "Epoch timestamp (seconds) captured at emulation start time.\n\n" - "Use this for timeline correlation with external logs." + "Wall-clock time in seconds, rounded to milliseconds, from the last start of the run queue " + "to the end of the emulation." ) ) + timestamp: int = Field(description="Unix time in seconds of the last start of the run queue.") arch: Arch | None = Field( default=None, - description=("Architecture of the input sample, when known.\n\nPopulated from loader metadata."), + description="Architecture of the input.", ) path: str | None = Field( default=None, - description=( - "Host path of the input file, when the input was loaded from a file.\n\n" - "Useful for provenance tracking in batch pipelines." - ), + description="Host path of the input file. Absent when the input came from bytes.", ) sha256: str | None = Field( default=None, - description=( - "SHA-256 hash of the emulated input bytes, when available.\n\n" - "Use this as the primary stable sample identifier." - ), + description="SHA-256 of the input.", ) size: int | None = Field( default=None, - description=( - "Input file size in bytes, when available.\n\n" - "Helps validate sample identity and triage packaging anomalies." - ), + description="Size of the input in bytes.", ) file_type: FileType | None = Field( default=None, - description=("Kind of input file, when known.\n\nDerived from loader inspection before emulation starts."), + description="Kind of input file.", ) image_base: HexIntOptional = Field( default=None, description=( - "Base address at which the primary module was loaded during emulation.\n\n" - "Use this to rebase analysis tools (e.g. IDA Pro) to match emulated addresses." + "Base address of the input image in the emulator. Rebase a disassembler database to this " + "address to match the addresses of the report." ), ) errors: list[ErrorInfo] = Field( default_factory=list, - description=( - "Top-level emulator errors not tied to a specific run.\n\n" - "These usually reflect session-level failures rather than per-entry-point " - "termination." - ), + description="Errors that do not belong to one run, such as an ``invalid_image`` defect of the input PE.", ) strings: StringsReport | None = Field( default=None, - description=( - "Optional static and in-memory string extraction output.\n\n" - "Present only when string analysis is enabled in config." - ), + description="Strings of the input and of memory. Only with ``analysis.strings``.", ) data: dict[str, DataArtifact] = Field( default_factory=dict, - description=( - "Deduplicated binary payload store keyed by SHA-256.\n\n" - "Resolve ``*_ref`` fields in the report against this mapping to recover compressed payload bytes." - ), + description="Payloads of the report, keyed by SHA-256. Each ``data_ref`` field holds a key of this mapping.", ) entry_points: list[EntryPoint] = Field( default_factory=list, - description=( - "Ordered list of all runs executed during emulation.\n\n" - "This is the primary behavioral timeline for the report." - ), + description="Runs of the emulation, in the order that the runs started.", ) class FileManifestEntry(BaseModel): - """ - Manifest entry used by dropped-file archive exports. - - These records are stored in ``speakeasy_manifest.json`` inside file archive - bundles created by the high-level API. - - Use this manifest to map archive members back to emulated file paths. - """ + """One file in ``speakeasy_manifest.json`` of the dropped-files archive.""" model_config = ConfigDict(extra="forbid") - path: str = Field( - description=( - "Emulated file path associated with the archived artifact.\n\n" - "Represents where the sample wrote the file in emulated space." - ) - ) - file_name: str = Field( - description=( - "Archive member file name for the artifact bytes.\n\n" - "Use this to locate the corresponding entry inside the zip package." - ) - ) - size: int = Field( - description=( - "Artifact size in bytes.\n\nUseful for quick filtering of tiny markers versus payload-like binaries." - ) - ) - sha256: str = Field( - description=("SHA-256 hash of the archived file bytes.\n\nSupports integrity checks and IOC correlation.") - ) + path: str = Field(description="Emulated path of the file.") + file_name: str = Field(description="Name of the zip member that has the file data.") + size: int = Field(description="Size of the file in bytes.") + sha256: str = Field(description="SHA-256 of the file data.") diff --git a/speakeasy/report_types.py b/speakeasy/report_types.py index fb2b32eeb..a0eccd09b 100644 --- a/speakeasy/report_types.py +++ b/speakeasy/report_types.py @@ -4,15 +4,7 @@ def hex_serializer(v: int | None) -> str | None: - """ - Serialize integer values as ``0x``-prefixed hexadecimal strings. - - Args: - v: Integer value to serialize. - - Returns: - Hexadecimal string, or ``None`` when the input is ``None``. - """ + """Serialize an int as a ``0x``-prefixed hex string. ``None`` does not change.""" if v is None: return None return hex(v) @@ -20,13 +12,9 @@ def hex_serializer(v: int | None) -> str | None: def _parse_hex_int(v: Any) -> Any: """ - Parse integer-like strings accepted by report models. - - Args: - v: Raw input value. + Convert a string, such as ``"0x1f"``, to an int. Other values do not change. - Returns: - Parsed integer when ``v`` is a numeric string, otherwise ``v`` unchanged. + The prefix of the string gives the base, so a string without a prefix is decimal. """ if isinstance(v, str): return int(v, 0) diff --git a/speakeasy/resources/report.schema.json b/speakeasy/resources/report.schema.json index d8d38fdc4..5e765a5b0 100644 --- a/speakeasy/resources/report.schema.json +++ b/speakeasy/resources/report.schema.json @@ -79,26 +79,26 @@ }, "ApiEvent": { "additionalProperties": false, - "description": "Records one intercepted API invocation.\n\nAPI events are emitted by import-call handlers and represent core behavioral\ntelemetry for most runs.\n\nUse these entries to understand control flow and argument-level intent.", + "description": "One call to an API.\n\nThe profiler leaves out a call that repeats the API, pc, shown arguments and return value of\none of the last 3 events of the run.", "properties": { "pos": { "$ref": "#/$defs/TracePosition", - "description": "Execution position metadata for this event.\n\nUsed to align event payloads with timeline and execution context." + "description": "Position of the thread that did the operation." }, "event": { "const": "api", "default": "api", - "description": "Discriminator for API call events.", + "description": "Event type of an API call.", "title": "Event", "type": "string" }, "api_name": { - "description": "Resolved API name in ``module.function`` form.\n\nUsed for behavior signatures, clustering, and family comparison.", + "description": "API name in ``module.function`` form, such as ``kernel32.CreateFileA``.", "title": "Api Name", "type": "string" }, "args": { - "description": "Arguments of the call. With a usable API signature there is one entry per parameter; otherwise one entry per argument slot, plus any entries a variadic handler adds.", + "description": "Arguments of the call. With a usable API signature, one entry for each parameter. Else, one entry for each argument slot. A variadic handler can add entries.", "items": { "$ref": "#/$defs/ApiArg" }, @@ -116,7 +116,7 @@ } ], "default": null, - "description": "Return value of the API call, when available.\n\nUse this to identify failure codes and branch-driving outcomes.", + "description": "Return value of the API, when the handler gives one.", "title": "Return Value" } }, @@ -155,23 +155,23 @@ }, "DataArtifact": { "additionalProperties": false, - "description": "Stores one deduplicated binary payload referenced elsewhere in a report.\n\nPayloads are keyed by SHA-256 in the top-level ``Report.data`` mapping.\nOther models store the SHA-256 key as a reference instead of embedding bytes\ninline repeatedly.", + "description": "One payload of the report.\n\nThe top-level ``data`` of the report keys each payload by the SHA-256 of its bytes. A ``data_ref``\nfield elsewhere in the report holds such a key.", "properties": { "compression": { "$ref": "#/$defs/ArtifactCompression", - "description": "Compression algorithm applied to the stored payload." + "description": "Compression of the payload." }, "encoding": { "$ref": "#/$defs/ArtifactEncoding", - "description": "Text encoding used for serialized payload bytes." + "description": "Text encoding of the compressed payload." }, "size": { - "description": "Uncompressed payload size in bytes.", + "description": "Size of the payload before compression, in bytes.", "title": "Size", "type": "integer" }, "data": { - "description": "Encoded payload bytes after compression and serialization.", + "description": "Payload, compressed and then encoded.", "title": "Data", "type": "string" } @@ -187,21 +187,21 @@ }, "DirectoryCreateEvent": { "additionalProperties": false, - "description": "Records directory creation.\n\nEmitted when file APIs create a directory.\n\nUse with file create events to track where a sample drops files.", + "description": "A directory that the guest creates.", "properties": { "pos": { "$ref": "#/$defs/TracePosition", - "description": "Execution position metadata for this event.\n\nUsed to align event payloads with timeline and execution context." + "description": "Position of the thread that did the operation." }, "event": { "const": "directory_create", "default": "directory_create", - "description": "Discriminator for directory create events.", + "description": "Event type of a directory creation.", "title": "Event", "type": "string" }, "path": { - "description": "Directory path targeted by the create request.", + "description": "Emulated path of the directory.", "title": "Path", "type": "string" } @@ -215,21 +215,21 @@ }, "DirectoryDeleteEvent": { "additionalProperties": false, - "description": "Records directory removal.\n\nEmitted when file APIs remove a directory.\n\nUse with directory create events to track staging directories.", + "description": "A directory that the guest deletes.", "properties": { "pos": { "$ref": "#/$defs/TracePosition", - "description": "Execution position metadata for this event.\n\nUsed to align event payloads with timeline and execution context." + "description": "Position of the thread that did the operation." }, "event": { "const": "directory_delete", "default": "directory_delete", - "description": "Discriminator for directory delete events.", + "description": "Event type of a directory deletion.", "title": "Event", "type": "string" }, "path": { - "description": "Directory path targeted by the remove request.", + "description": "Emulated path of the directory.", "title": "Path", "type": "string" } @@ -243,20 +243,20 @@ }, "DroppedFile": { "additionalProperties": false, - "description": "Represents one file artifact written by emulated execution.\n\nDropped-file entries are created from filesystem manager state when a run\ncompletes and fully written files are detected.\n\nUse these records to recover secondary payloads and configuration artifacts.", + "description": "A file that the guest wrote completely, up to the end of the run.", "properties": { "path": { - "description": "Emulated destination path of the written file.\n\nInterpret this as the malware-intended filesystem location.", + "description": "Emulated path of the file.", "title": "Path", "type": "string" }, "size": { - "description": "Dropped file size in bytes.", + "description": "Size of the file in bytes.", "title": "Size", "type": "integer" }, "sha256": { - "description": "SHA-256 hash of the dropped file bytes.\n\nUse this for sample correlation and deduplication workflows.", + "description": "SHA-256 of the file data.", "title": "Sha256", "type": "string" }, @@ -270,7 +270,7 @@ } ], "default": null, - "description": "Optional reference into the top-level report ``data`` store.\n\nWhen present, resolve this SHA-256 key in ``Report.data`` to recover the compressed file bytes.", + "description": "Key in the top-level ``data`` of the file data. Absent for a file above MAX_EMBEDDED_FILE_SIZE.", "title": "Data Ref" } }, @@ -284,21 +284,21 @@ }, "DynamicCodeSegment": { "additionalProperties": false, - "description": "Describes one dynamically generated code region that executed.\n\nDynamic code segments are added when execution reaches memory previously\nregistered as dynamic code (for example after VirtualAlloc/WriteProcessMemory\nstyle paths).\n\nThis section is useful for unpacking and staged payload detection.", + "description": "A dynamic code region where code ran.\n\nThe emulator watches the memory that the guest allocates, for example with ``VirtualAlloc``, and\nadds a region when the guest runs code in the region.", "properties": { "tag": { - "description": "Memory-map tag assigned to the dynamic region.\n\nTags help correlate the segment with allocation provenance in traces.", + "description": "Tag of the memory region.", "title": "Tag", "type": "string" }, "base": { - "description": "Base address of the dynamic code region.\n\nUse this with memory layout data to inspect surrounding mapped content.", + "description": "Base address of the region.", "pattern": "^0x[0-9a-f]+$", "title": "Base", "type": "string" }, "size": { - "description": "Region size of the dynamic code segment.\n\nLarge regions may indicate unpacked module images or staged blobs.", + "description": "Size of the region in bytes.", "title": "Size", "type": "integer" } @@ -313,11 +313,11 @@ }, "EntryPoint": { "additionalProperties": false, - "description": "Represents one emulation execution primitive captured as a run.\n\nA run may be a module entry point, callback, exported function, injected\nthread, or other execution start context.\n\nMost behavioral telemetry is attached at this level, so treat each entry as\na self-contained execution narrative.", + "description": "One run, and the events and data that the run collected.\n\nA run starts at one address, such as the module entry point, an export, a TLS callback or a\nthread.", "properties": { "type": { "$ref": "#/$defs/EntryPointType", - "description": "Kind of code that this run executes.\n\nExamples include the module entry point, an export, a TLS callback, and a thread." + "description": "Kind of code that the run executes, such as ``module_entry``, ``export`` or ``thread``." }, "name": { "anyOf": [ @@ -329,17 +329,17 @@ } ], "default": null, - "description": "Detail for ``type``, or null when the type has no detail.\n\nFor example the export name, the TLS callback index, the dependency module, or the IRP major function.", + "description": "Detail for ``type``, such as the export name, the TLS callback index, the dependency module or the IRP major function. Absent when the type has no detail.", "title": "Name" }, "start_address": { - "description": "Address where this run began executing.\n\nUse this as the anchor when following control flow in disassembly.", + "description": "Address where the run started.", "pattern": "^0x[0-9a-f]+$", "title": "Start Address", "type": "string" }, "args": { - "description": "Argument values supplied at run entry.", + "description": "Arguments that the emulator gives to the code at ``start_address``.", "items": { "pattern": "^0x[0-9a-f]+$", "type": "string" @@ -357,7 +357,7 @@ } ], "default": null, - "description": "Process identifier associated with the run context.\n\nPopulated when process context is known for this entry point.", + "description": "ID of the process of the run.", "title": "Pid" }, "tid": { @@ -370,7 +370,7 @@ } ], "default": null, - "description": "Thread identifier associated with the run context.\n\nUseful for correlating with per-event ``pos.tid`` values.", + "description": "ID of the thread of the run.", "title": "Tid" }, "instruction_count": { @@ -383,12 +383,12 @@ } ], "default": null, - "description": "Instruction count executed for this run.\n\nCan be limited by configuration caps or early termination conditions.", + "description": "Number of instructions that the run executed. Absent when the run executed none.", "title": "Instruction Count" }, "api_hash": { "default": "", - "description": "SHA-256 hash over unique API names invoked in first-seen order.\n\nUse this for coarse behavioral clustering across samples and architectures.", + "description": "SHA-256 of the lowercase names of the APIs that the run called, each name once, in the order of the first call.", "title": "Api Hash", "type": "string" }, @@ -403,7 +403,7 @@ } ], "default": null, - "description": "Return value produced when the run ended normally.\n\nInterpret this in the calling convention context of ``type``.", + "description": "Value of the return register when the run completed.", "title": "Return Value" }, "error": { @@ -416,10 +416,10 @@ } ], "default": null, - "description": "Run-local error details when execution did not complete cleanly.\n\nConsult this with ``events`` to identify the exact failure transition." + "description": "Error that ended the run. Absent when the run completed." }, "events": { - "description": "Chronological event stream emitted during this run.\n\nEvent contents are produced by API, process, file, registry, network, and exception recorders.", + "description": "Events of the run, in order.", "items": { "discriminator": { "mapping": { @@ -557,11 +557,11 @@ } ], "default": null, - "description": "Symbol-level access counters observed during tracing.\n\nPresent only when memory tracing is enabled in config.", + "description": "Access counts of each API symbol. Only with ``analysis.memory_tracing``.", "title": "Symbol Accesses" }, "dynamic_code_segments": { - "description": "Dynamically generated code regions that executed in this run.\n\nIndicates unpacking, JIT-like behavior, or in-memory staging activity.", + "description": "Dynamic code regions where code ran in the run.", "items": { "$ref": "#/$defs/DynamicCodeSegment" }, @@ -582,11 +582,11 @@ } ], "default": null, - "description": "Sorted list of executed instruction addresses for coverage analysis.\n\nPresent only when coverage collection is enabled in config.", + "description": "Sorted addresses of the instructions that ran. Only with ``analysis.coverage``.", "title": "Coverage" }, "dropped_files": { - "description": "File artifacts detected as fully written during the run.\n\nUse this list to extract secondary payloads and logs.", + "description": "Files that the guest wrote completely, up to the end of the run.", "items": { "$ref": "#/$defs/DroppedFile" }, @@ -603,7 +603,7 @@ } ], "default": null, - "description": "Optional memory and module snapshot captured at run completion.\n\nPopulate this when memory-layout capture is active for the emulator mode." + "description": "Memory regions and modules at the end of the run. User mode only." } }, "required": [ @@ -637,11 +637,11 @@ }, "ErrorInfo": { "additionalProperties": false, - "description": "Captures failure context for top-level or per-entry-point errors.\n\nError records can represent unsupported API termination, instruction faults,\nexplicit runtime limits, or unhandled exceptions.\n\nInspect this structure first when a run exits unexpectedly.", + "description": "Why a run, or the emulation, ended with an error.\n\nThe emulator fills only the fields that apply to the error ``type``.", "properties": { "type": { "$ref": "#/$defs/ErrorType", - "description": "Error category identifier describing the failure mode.\n\nExamples include unsupported API paths, max API limits, and fault conditions raised by emulator hooks. A Python exception in Speakeasy itself, such as a bug in an API handler, is ``internal_error``." + "description": "Kind of error, such as ``unsupported_api``, ``max_api_count`` or ``invalid_read``. A Python exception in Speakeasy itself, such as a bug in an API handler, is ``internal_error``." }, "pc": { "anyOf": [ @@ -654,7 +654,7 @@ } ], "default": null, - "description": "Program counter associated with the error, when available.\n\nUse this address to align failures with disassembly and event traces.", + "description": "Address of the instruction that caused the error. For ``unsupported_api``, the return address.", "title": "Pc" }, "instruction": { @@ -667,7 +667,7 @@ } ], "default": null, - "description": "Faulting instruction text, when known.\n\nUseful for quickly identifying invalid memory operations or bad control flow transitions.", + "description": "Disassembly of the instruction at ``pc``.", "title": "Instruction" }, "address": { @@ -681,7 +681,7 @@ } ], "default": null, - "description": "Faulting memory address that triggered the error.", + "description": "Memory address that the faulting access used.", "title": "Address" }, "access_type": { @@ -694,7 +694,7 @@ } ], "default": null, - "description": "Memory access type that faulted: read, write, or fetch." + "description": "Kind of the faulting access: read, write or fetch." }, "registers": { "anyOf": [ @@ -710,7 +710,7 @@ } ], "default": null, - "description": "Register state snapshot at the time of the error.", + "description": "Register values at the time of the error.", "title": "Registers" }, "stack": { @@ -765,7 +765,7 @@ } ], "default": null, - "description": "Memory region containing the faulting address, if mapped." + "description": "Memory region that contains ``address``. Absent when ``address`` is not mapped." }, "nearby_regions": { "anyOf": [ @@ -780,7 +780,7 @@ } ], "default": null, - "description": "Up to 2 nearest memory regions when the faulting address is unmapped.", + "description": "The memory regions nearest to ``address`` when ``address`` is not mapped.", "title": "Nearby Regions" }, "tid": { @@ -819,7 +819,7 @@ } ], "default": null, - "description": "Human-readable one-line triage summary of the error.", + "description": "One line of text that tells what went wrong.", "title": "Context Summary" }, "traceback": { @@ -832,7 +832,7 @@ } ], "default": null, - "description": "Python traceback when the error originated from an internal exception.", + "description": "Python traceback of an ``internal_error``.", "title": "Traceback" }, "message": { @@ -858,7 +858,7 @@ } ], "default": null, - "description": "API function name for unsupported-API errors.", + "description": "Name of the API that has no handler, for ``unsupported_api``.", "title": "Api Name" }, "count": { @@ -871,7 +871,7 @@ } ], "default": null, - "description": "API call count that triggered a max-API-count limit.", + "description": "Limit that ended the run: ``max_api_calls`` of the config for ``max_api_count``, or the instruction limit for ``max_instructions``.", "title": "Count" }, "last_api": { @@ -884,7 +884,7 @@ } ], "default": null, - "description": "Last API name invoked before max-API-count termination.", + "description": "Name of the API call that went above the limit, for ``max_api_count``.", "title": "Last Api" }, "interrupt_num": { @@ -897,7 +897,7 @@ } ], "default": null, - "description": "Interrupt number for unhandled-interrupt errors.", + "description": "Interrupt number, for ``unhandled_interrupt``.", "title": "Interrupt Num" } }, @@ -939,21 +939,21 @@ }, "ExceptionEvent": { "additionalProperties": false, - "description": "Records handled exception dispatch activity.\n\nEmitted when exception handling telemetry is captured during execution.\n\nUse this to understand exception-driven control flow and anti-analysis logic.", + "description": "An exception that the emulator sends to a guest handler.", "properties": { "pos": { "$ref": "#/$defs/TracePosition", - "description": "Execution position metadata for this event.\n\nUsed to align event payloads with timeline and execution context." + "description": "Position of the thread that did the operation." }, "event": { "const": "exception", "default": "exception", - "description": "Discriminator for exception events.", + "description": "Event type of an exception.", "title": "Event", "type": "string" }, "instruction": { - "description": "Instruction text associated with the exception.", + "description": "Disassembly of the instruction at the exception address.", "title": "Instruction", "type": "string" }, @@ -964,7 +964,7 @@ "type": "string" }, "handler_address": { - "description": "Address of the selected exception handler.", + "description": "Address of the handler that gets the exception.", "pattern": "^0x[0-9a-f]+$", "title": "Handler Address", "type": "string" @@ -974,7 +974,7 @@ "pattern": "^0x[0-9a-f]+$", "type": "string" }, - "description": "Register snapshot captured at exception time.", + "description": "Register values at the time of the exception.", "title": "Registers", "type": "object" }, @@ -1031,7 +1031,7 @@ } ], "default": null, - "description": "Stack slots at exception time.", + "description": "Stack slots at the time of the exception.", "title": "Stack" } }, @@ -1047,21 +1047,21 @@ }, "FileCreateEvent": { "additionalProperties": false, - "description": "Records file creation/open-with-create semantics.\n\nEmitted when file APIs request creation semantics for a path.\n\nUse with follow-up write events to track dropped artifacts.", + "description": "A file that the guest creates, or opens with a disposition that can create the file.", "properties": { "pos": { "$ref": "#/$defs/TracePosition", - "description": "Execution position metadata for this event.\n\nUsed to align event payloads with timeline and execution context." + "description": "Position of the thread that did the operation." }, "event": { "const": "file_create", "default": "file_create", - "description": "Discriminator for file create events.", + "description": "Event type of a file creation.", "title": "Event", "type": "string" }, "path": { - "description": "File path targeted by creation request.", + "description": "Emulated path of the file.", "title": "Path", "type": "string" }, @@ -1076,7 +1076,7 @@ } ], "default": null, - "description": "Assigned file handle, when available.", + "description": "File handle that the API returns.", "title": "Handle" }, "open_flags": { @@ -1092,7 +1092,7 @@ } ], "default": null, - "description": "Creation/open disposition flags supplied by the API call.", + "description": "Names of the disposition flags of the call, such as ``CREATE_ALWAYS``.", "title": "Open Flags" }, "access_flags": { @@ -1108,7 +1108,7 @@ } ], "default": null, - "description": "Requested access rights for the file operation.", + "description": "Names of the access rights that the caller asks for, such as ``GENERIC_READ``.", "title": "Access Flags" } }, @@ -1121,21 +1121,21 @@ }, "FileDeleteEvent": { "additionalProperties": false, - "description": "Records file deletion.\n\nEmitted when file APIs delete a file.\n\nUse with file create events to track files that a sample removes after use.", + "description": "A file that the guest deletes.", "properties": { "pos": { "$ref": "#/$defs/TracePosition", - "description": "Execution position metadata for this event.\n\nUsed to align event payloads with timeline and execution context." + "description": "Position of the thread that did the operation." }, "event": { "const": "file_delete", "default": "file_delete", - "description": "Discriminator for file delete events.", + "description": "Event type of a file deletion.", "title": "Event", "type": "string" }, "path": { - "description": "File path targeted by the delete request.", + "description": "Emulated path of the file.", "title": "Path", "type": "string" } @@ -1149,21 +1149,21 @@ }, "FileOpenEvent": { "additionalProperties": false, - "description": "Records file open operations without explicit create intent.\n\nEmitted for open-style file API calls.\n\nUse to reconstruct file-read dependency chains.", + "description": "A file that the guest opens without a create disposition.", "properties": { "pos": { "$ref": "#/$defs/TracePosition", - "description": "Execution position metadata for this event.\n\nUsed to align event payloads with timeline and execution context." + "description": "Position of the thread that did the operation." }, "event": { "const": "file_open", "default": "file_open", - "description": "Discriminator for file open events.", + "description": "Event type of a file open.", "title": "Event", "type": "string" }, "path": { - "description": "File path targeted by open request.", + "description": "Emulated path of the file.", "title": "Path", "type": "string" }, @@ -1178,7 +1178,7 @@ } ], "default": null, - "description": "Assigned file handle, when available.", + "description": "File handle that the API returns.", "title": "Handle" }, "open_flags": { @@ -1194,7 +1194,7 @@ } ], "default": null, - "description": "Open disposition flags supplied by the API call.", + "description": "Names of the disposition flags of the call, such as ``OPEN_EXISTING``.", "title": "Open Flags" }, "access_flags": { @@ -1210,7 +1210,7 @@ } ], "default": null, - "description": "Requested access rights for the file operation.", + "description": "Names of the access rights that the caller asks for, such as ``GENERIC_READ``.", "title": "Access Flags" } }, @@ -1223,21 +1223,21 @@ }, "FileReadEvent": { "additionalProperties": false, - "description": "Records reads from a file handle.\n\nEmitted when file-read APIs pull bytes from emulated files.\n\nUseful for identifying configuration ingestion and payload loading.", + "description": "Data that the guest reads from a file.\n\nThe profiler merges all reads of one path in a run into one event.", "properties": { "pos": { "$ref": "#/$defs/TracePosition", - "description": "Execution position metadata for this event.\n\nUsed to align event payloads with timeline and execution context." + "description": "Position of the thread that did the operation." }, "event": { "const": "file_read", "default": "file_read", - "description": "Discriminator for file read events.", + "description": "Event type of a file read.", "title": "Event", "type": "string" }, "path": { - "description": "File path associated with the read operation.", + "description": "Emulated path of the file.", "title": "Path", "type": "string" }, @@ -1252,7 +1252,7 @@ } ], "default": null, - "description": "File handle used by the read operation.", + "description": "File handle of the first read.", "title": "Handle" }, "size": { @@ -1265,7 +1265,7 @@ } ], "default": null, - "description": "Number of bytes requested/read.", + "description": "Total number of bytes that the guest read.", "title": "Size" }, "data_ref": { @@ -1278,7 +1278,7 @@ } ], "default": null, - "description": "Reference into the top-level report ``data`` store for the captured file bytes.", + "description": "Key in the top-level ``data`` of the first bytes that the guest read.", "title": "Data Ref" }, "buffer": { @@ -1292,7 +1292,7 @@ } ], "default": null, - "description": "Destination buffer pointer.", + "description": "Address of the buffer of the first read.", "title": "Buffer" } }, @@ -1316,21 +1316,21 @@ }, "FileWriteEvent": { "additionalProperties": false, - "description": "Records writes to a file handle.\n\nEmitted when file-write APIs append or overwrite emulated file data.\n\nCombine with dropped-file summaries to recover persisted payloads.", + "description": "Data that the guest writes to a file.\n\nThe profiler merges all writes to one path in a run into one event.", "properties": { "pos": { "$ref": "#/$defs/TracePosition", - "description": "Execution position metadata for this event.\n\nUsed to align event payloads with timeline and execution context." + "description": "Position of the thread that did the operation." }, "event": { "const": "file_write", "default": "file_write", - "description": "Discriminator for file write events.", + "description": "Event type of a file write.", "title": "Event", "type": "string" }, "path": { - "description": "File path associated with the write operation.", + "description": "Emulated path of the file.", "title": "Path", "type": "string" }, @@ -1345,7 +1345,7 @@ } ], "default": null, - "description": "File handle used by the write operation.", + "description": "File handle of the first write.", "title": "Handle" }, "size": { @@ -1358,7 +1358,7 @@ } ], "default": null, - "description": "Number of bytes written.", + "description": "Total number of bytes that the guest wrote.", "title": "Size" }, "data_ref": { @@ -1371,7 +1371,7 @@ } ], "default": null, - "description": "Reference into the top-level report ``data`` store for the captured file bytes.", + "description": "Key in the top-level ``data`` of the first bytes that the guest wrote.", "title": "Data Ref" }, "buffer": { @@ -1385,7 +1385,7 @@ } ], "default": null, - "description": "Source buffer pointer.", + "description": "Address of the buffer of the first write.", "title": "Buffer" } }, @@ -1398,31 +1398,31 @@ }, "LoadedModule": { "additionalProperties": false, - "description": "Captures one module visible in the run's loaded-module inventory.\n\nModule entries are assembled from runtime module objects at run completion.\n\nThis section provides a concise map of what binaries were present and where.", + "description": "A module that is loaded at the end of a run.", "properties": { "name": { - "description": "Module file name or canonical module identifier.\n\nUse this to correlate module events and API import usage.", + "description": "Name of the module.", "title": "Name", "type": "string" }, "path": { - "description": "Emulated module path.\n\nUseful for distinguishing system modules from payload modules.", + "description": "Emulated path of the module.", "title": "Path", "type": "string" }, "base": { - "description": "Module base address in virtual memory.\n\nUse this as the relocation anchor for offsets and symbol lookups.", + "description": "Base address of the module.", "pattern": "^0x[0-9a-f]+$", "title": "Base", "type": "string" }, "size": { - "description": "Total mapped module size in bytes.\n\nHelps identify unusually large or tiny mapped images.", + "description": "Size of the module image in bytes.", "title": "Size", "type": "integer" }, "segments": { - "description": "Section/segment breakdown for the module mapping.\n\nUse these entries for section-aware interpretation of memory behavior.", + "description": "Sections of the module.", "items": { "$ref": "#/$defs/ModuleSegment" }, @@ -1441,32 +1441,32 @@ }, "MemAllocEvent": { "additionalProperties": false, - "description": "Records remote or explicit process memory allocation activity.\n\nEmitted by memory allocation APIs that target process virtual memory.\n\nCommonly seen in injection and unpacking workflows.", + "description": "Memory that the guest allocates with ``VirtualAlloc`` or ``VirtualAllocEx``.", "properties": { "pos": { "$ref": "#/$defs/TracePosition", - "description": "Execution position metadata for this event.\n\nUsed to align event payloads with timeline and execution context." + "description": "Position of the thread that did the operation." }, "event": { "const": "mem_alloc", "default": "mem_alloc", - "description": "Discriminator for memory allocation events.", + "description": "Event type of a memory allocation.", "title": "Event", "type": "string" }, "path": { - "description": "Target process path associated with the allocation event.", + "description": "Image path of the target process.", "title": "Path", "type": "string" }, "base": { - "description": "Allocated base address.", + "description": "Base address of the allocation.", "pattern": "^0x[0-9a-f]+$", "title": "Base", "type": "string" }, "size": { - "description": "Allocation size in bytes.", + "description": "Size that the caller asked for, in bytes.", "title": "Size", "type": "integer" }, @@ -1496,7 +1496,7 @@ } ], "default": null, - "description": "Target process identifier.\n\nThe actor (event pos) allocates in this process; NULL when the target is the actor itself.", + "description": "ID of the target process. Absent when the target is the process in ``pos``.", "title": "Pid" } }, @@ -1511,32 +1511,32 @@ }, "MemFreeEvent": { "additionalProperties": false, - "description": "Records memory free/decommit operations.\n\nEmitted when memory free APIs are invoked against process regions.\n\nUse this to track lifecycle of allocation artifacts.", + "description": "Memory that the guest frees.", "properties": { "pos": { "$ref": "#/$defs/TracePosition", - "description": "Execution position metadata for this event.\n\nUsed to align event payloads with timeline and execution context." + "description": "Position of the thread that did the operation." }, "event": { "const": "mem_free", "default": "mem_free", - "description": "Discriminator for memory free events.", + "description": "Event type of a memory free.", "title": "Event", "type": "string" }, "path": { - "description": "Target process path associated with the free event.", + "description": "Image path of the target process.", "title": "Path", "type": "string" }, "base": { - "description": "Freed region base address.", + "description": "Base address of the freed region.", "pattern": "^0x[0-9a-f]+$", "title": "Base", "type": "string" }, "size": { - "description": "Freed region size in bytes.", + "description": "Size of the freed region in bytes.", "title": "Size", "type": "integer" }, @@ -1550,7 +1550,7 @@ } ], "default": null, - "description": "Target process identifier.\n\nThe actor (event pos) frees memory in this process; NULL when the target is the actor itself.", + "description": "ID of the target process. Absent when the target is the process in ``pos``.", "title": "Pid" } }, @@ -1565,32 +1565,32 @@ }, "MemProtectEvent": { "additionalProperties": false, - "description": "Records memory protection changes.\n\nEmitted for API paths that alter page permissions in process memory.\n\nThis is a key indicator for RWX transitions and shellcode staging.", + "description": "A change of page protection with ``VirtualProtectEx``.", "properties": { "pos": { "$ref": "#/$defs/TracePosition", - "description": "Execution position metadata for this event.\n\nUsed to align event payloads with timeline and execution context." + "description": "Position of the thread that did the operation." }, "event": { "const": "mem_protect", "default": "mem_protect", - "description": "Discriminator for memory protection change events.", + "description": "Event type of a page protection change.", "title": "Event", "type": "string" }, "path": { - "description": "Target process path associated with the protection change.", + "description": "Image path of the target process.", "title": "Path", "type": "string" }, "base": { - "description": "Protection-change base address.", + "description": "Base address of the range.", "pattern": "^0x[0-9a-f]+$", "title": "Base", "type": "string" }, "size": { - "description": "Protection-change size in bytes.", + "description": "Size of the range in bytes.", "title": "Size", "type": "integer" }, @@ -1620,7 +1620,7 @@ } ], "default": null, - "description": "Target process identifier.\n\nThe actor (event pos) changes protection in this process; NULL when the target is the actor itself.", + "description": "ID of the target process. Absent when the target is the process in ``pos``.", "title": "Pid" } }, @@ -1635,26 +1635,26 @@ }, "MemReadEvent": { "additionalProperties": false, - "description": "Records reads from process memory.\n\nEmitted for API-mediated memory reads and may merge adjacent reads in the\nprofiler for compactness.\n\nUseful for identifying memory scraping or credential theft behavior.", + "description": "Data that the guest reads from process memory with an API, such as ``ReadProcessMemory``.\n\nThe profiler merges a read into the last process event when that event is a read that ends at\nthe start of the new read.", "properties": { "pos": { "$ref": "#/$defs/TracePosition", - "description": "Execution position metadata for this event.\n\nUsed to align event payloads with timeline and execution context." + "description": "Position of the thread that did the operation." }, "event": { "const": "mem_read", "default": "mem_read", - "description": "Discriminator for memory read events.", + "description": "Event type of a memory read.", "title": "Event", "type": "string" }, "path": { - "description": "Target process path associated with the read event.", + "description": "Image path of the target process.", "title": "Path", "type": "string" }, "base": { - "description": "Read start address.", + "description": "Start address of the read.", "pattern": "^0x[0-9a-f]+$", "title": "Base", "type": "string" @@ -1674,7 +1674,7 @@ } ], "default": null, - "description": "Reference into the top-level report ``data`` store for the captured read preview bytes.", + "description": "Key in the top-level ``data`` of the first bytes of the read.", "title": "Data Ref" }, "pid": { @@ -1687,7 +1687,7 @@ } ], "default": null, - "description": "Target process identifier.\n\nThe actor (event pos) reads from this process; NULL when the target is the actor itself.", + "description": "ID of the target process. Absent when the target is the process in ``pos``.", "title": "Pid" } }, @@ -1702,26 +1702,26 @@ }, "MemWriteEvent": { "additionalProperties": false, - "description": "Records writes into process memory.\n\nEmitted for API-mediated memory writes and may merge adjacent writes in the\nprofiler for compactness.\n\nUse with thread/process events to identify code injection flows.", + "description": "Data that the guest writes to process memory with an API, such as ``WriteProcessMemory``.\n\nThe profiler merges a write into the last process event when that event is a write that ends\nat the start of the new write.", "properties": { "pos": { "$ref": "#/$defs/TracePosition", - "description": "Execution position metadata for this event.\n\nUsed to align event payloads with timeline and execution context." + "description": "Position of the thread that did the operation." }, "event": { "const": "mem_write", "default": "mem_write", - "description": "Discriminator for memory write events.", + "description": "Event type of a memory write.", "title": "Event", "type": "string" }, "path": { - "description": "Target process path associated with the write event.", + "description": "Image path of the target process.", "title": "Path", "type": "string" }, "base": { - "description": "Write start address.", + "description": "Start address of the write.", "pattern": "^0x[0-9a-f]+$", "title": "Base", "type": "string" @@ -1741,7 +1741,7 @@ } ], "default": null, - "description": "Reference into the top-level report ``data`` store for the captured write preview bytes.", + "description": "Key in the top-level ``data`` of the first bytes of the write.", "title": "Data Ref" }, "pid": { @@ -1754,7 +1754,7 @@ } ], "default": null, - "description": "Target process identifier.\n\nThe actor (event pos) writes into this process; NULL when the target is the actor itself.", + "description": "ID of the target process. Absent when the target is the process in ``pos``.", "title": "Pid" } }, @@ -1769,20 +1769,20 @@ }, "MemoryAccesses": { "additionalProperties": false, - "description": "Aggregates read/write/execute counters for one memory region.\n\nThese counters are produced by runtime memory tracing hooks when enabled.\n\nUse them to prioritize regions with meaningful behavioral activity.", + "description": "Read, write and execute counts of one memory region, from memory tracing.", "properties": { "reads": { - "description": "Total read operations observed for the region.\n\nHigh read density often indicates lookup tables or parsed structures.", + "description": "Number of reads in the region.", "title": "Reads", "type": "integer" }, "writes": { - "description": "Total write operations observed for the region.\n\nHigh write density often indicates decoded payload construction.", + "description": "Number of writes in the region.", "title": "Writes", "type": "integer" }, "execs": { - "description": "Total execute operations observed for the region.\n\nNon-zero execution marks code-bearing pages and trampoline regions.", + "description": "Number of instructions that ran in the region.", "title": "Execs", "type": "integer" } @@ -1797,10 +1797,10 @@ }, "MemoryLayout": { "additionalProperties": false, - "description": "Bundles memory-region and loaded-module snapshots for one run.\n\nThis object is populated when the emulator captures run-end memory state.\n\nUse it to understand both raw mappings and higher-level module context.", + "description": "Memory regions and loaded modules at the end of a run.", "properties": { "layout": { - "description": "Flat list of captured memory regions for the run.\n\nEntries may include module headers/sections and generic maps.", + "description": "Memory regions, with one region for the headers and one for each section of a module.", "items": { "$ref": "#/$defs/MemoryRegion" }, @@ -1808,7 +1808,7 @@ "type": "array" }, "modules": { - "description": "Loaded modules present when the run completed.\n\nUse alongside ``layout`` to interpret address ownership and segment roles.", + "description": "Modules that are loaded at the end of the run.", "items": { "$ref": "#/$defs/LoadedModule" }, @@ -1821,32 +1821,32 @@ }, "MemoryRegion": { "additionalProperties": false, - "description": "Represents one memory layout entry captured for an entry point.\n\nMemory layout capture runs at run completion and may include section-split\nmodule regions plus generic heap/stack mappings.\n\nUse this data to correlate execution artifacts with mapped memory topology.", + "description": "One memory region at the end of a run.\n\nThe emulator splits the image of a module into one region for the headers and one region\nfor each section.", "properties": { "tag": { - "description": "Region tag assigned by the memory manager.\n\nTags encode provenance such as module sections, heaps, or emulator internal structures.", + "description": "Tag of the region. The tag of a module region starts with ``emu.module.``.", "title": "Tag", "type": "string" }, "base": { - "description": "Base address of the memory region.\n\nTreat this as the start of the region for disassembly and dump slicing.", + "description": "Base address of the region.", "pattern": "^0x[0-9a-f]+$", "title": "Base", "type": "string" }, "size": { - "description": "Region size in bytes.\n\nCombine with ``base`` to derive half-open address ranges.", + "description": "Size of the region in bytes.", "title": "Size", "type": "integer" }, "perms": { - "description": "Permission string (for example ``r-x`` or ``rw-``).\n\nInterpret this as the final effective page protection at capture time.", + "description": "Permissions at the end of the run, such as ``r-x`` or ``rw-``.", "title": "Perms", "type": "string" }, "is_free": { "default": false, - "description": "Whether the region had been freed at capture time.\n\nFreed regions can remain visible when memory retention settings keep old maps for analysis.", + "description": "If true, the guest freed the region. Only ``keep_memory_on_free`` keeps a freed region.", "title": "Is Free", "type": "boolean" }, @@ -1860,7 +1860,7 @@ } ], "default": null, - "description": "Optional read/write/execute counters for the region.\n\nPopulated primarily when memory tracing was enabled for the run." + "description": "Access counts of the region. Absent when memory tracing is off or the run did not use the region." }, "data_ref": { "anyOf": [ @@ -1872,7 +1872,7 @@ } ], "default": null, - "description": "Optional reference into the top-level report ``data`` store.\n\nWhen present, resolve this SHA-256 key in ``Report.data`` to recover the compressed region bytes.", + "description": "Key in the top-level ``data`` of the region data. Only with ``snapshot_memory_regions``, and never for the stack and the heaps.", "title": "Data Ref" } }, @@ -1887,37 +1887,37 @@ }, "ModuleLoadEvent": { "additionalProperties": false, - "description": "Records dynamic module loading activity.\n\nEmitted when module load operations map additional images during runtime.\n\nUseful for dependency discovery and staged import behavior.", + "description": "A module that the guest loads at run time.", "properties": { "pos": { "$ref": "#/$defs/TracePosition", - "description": "Execution position metadata for this event.\n\nUsed to align event payloads with timeline and execution context." + "description": "Position of the thread that did the operation." }, "event": { "const": "module_load", "default": "module_load", - "description": "Discriminator for module load events.", + "description": "Event type of a module load.", "title": "Event", "type": "string" }, "name": { - "description": "Loaded module name.", + "description": "Name of the module.", "title": "Name", "type": "string" }, "path": { - "description": "Loaded module path in emulated filesystem space.", + "description": "Emulated path of the module.", "title": "Path", "type": "string" }, "base": { - "description": "Module base address.", + "description": "Base address of the module.", "pattern": "^0x[0-9a-f]+$", "title": "Base", "type": "string" }, "size": { - "description": "Mapped module size in bytes.", + "description": "Size of the module image in bytes.", "title": "Size", "type": "integer" } @@ -1934,26 +1934,26 @@ }, "ModuleSegment": { "additionalProperties": false, - "description": "Describes one section-like segment within a loaded module.\n\nSegment data is collected from module metadata during memory layout capture.\n\nUse this to align addresses with section boundaries and permissions.", + "description": "One section of a loaded module.", "properties": { "name": { - "description": "Segment or section name within the module image.\n\nUseful for mapping behavior to familiar PE sections such as ``.text``.", + "description": "Name of the section, such as ``.text``.", "title": "Name", "type": "string" }, "base": { - "description": "Absolute virtual address of the segment start.\n\nUse with ``size`` to identify covered address ranges.", + "description": "Virtual address of the section.", "pattern": "^0x[0-9a-f]+$", "title": "Base", "type": "string" }, "size": { - "description": "Segment size in bytes.\n\nLarge executable segments usually contain primary code paths.", + "description": "Virtual size of the section in bytes.", "title": "Size", "type": "integer" }, "perms": { - "description": "Effective memory permissions for the segment, such as ``r-x``.\n\nUse this to identify writable-executable anomalies or data-only regions.", + "description": "Permissions from the section header, such as ``r-x``.", "title": "Perms", "type": "string" } @@ -1969,21 +1969,21 @@ }, "NetDnsEvent": { "additionalProperties": false, - "description": "Records DNS name resolution activity.\n\nEmitted when DNS APIs request host resolution or related lookups.\n\nUse for C2 endpoint discovery and fallback behavior analysis.", + "description": "A host name that the guest looks up.\n\nThe profiler adds one event for each query and response in a run.", "properties": { "pos": { "$ref": "#/$defs/TracePosition", - "description": "Execution position metadata for this event.\n\nUsed to align event payloads with timeline and execution context." + "description": "Position of the thread that did the operation." }, "event": { "const": "net_dns", "default": "net_dns", - "description": "Discriminator for DNS events.", + "description": "Event type of a DNS lookup.", "title": "Event", "type": "string" }, "query": { - "description": "Queried domain or hostname.", + "description": "Host name of the lookup.", "title": "Query", "type": "string" }, @@ -1997,7 +1997,7 @@ } ], "default": null, - "description": "Resolved IP response, when a mapping was available.", + "description": "IP address from ``network.dns.names`` in the config. Absent when the config has no address.", "title": "Response" } }, @@ -2010,26 +2010,26 @@ }, "NetHttpEvent": { "additionalProperties": false, - "description": "Records HTTP request telemetry.\n\nEmitted by WinInet/WinHTTP handlers when requests are issued.\n\nUse these records to inspect request metadata and transmitted body content.", + "description": "An HTTP request that the guest sends with WinINet or WinHTTP.\n\nThe profiler adds one event for each server, port, protocol and headers in a run.", "properties": { "pos": { "$ref": "#/$defs/TracePosition", - "description": "Execution position metadata for this event.\n\nUsed to align event payloads with timeline and execution context." + "description": "Position of the thread that did the operation." }, "event": { "const": "net_http", "default": "net_http", - "description": "Discriminator for HTTP events.", + "description": "Event type of an HTTP request.", "title": "Event", "type": "string" }, "server": { - "description": "HTTP server hostname or address.", + "description": "Host name or address of the server.", "title": "Server", "type": "string" }, "port": { - "description": "HTTP server port.", + "description": "Port of the server.", "title": "Port", "type": "integer" }, @@ -2047,7 +2047,7 @@ } ], "default": null, - "description": "Serialized HTTP request headers, when available.", + "description": "Headers of the request, as text.", "title": "Headers" }, "data_ref": { @@ -2060,7 +2060,7 @@ } ], "default": null, - "description": "Reference into the top-level report ``data`` store for the captured HTTP body bytes.", + "description": "Key in the top-level ``data`` of the first bytes of the request body.", "title": "Data Ref" } }, @@ -2084,26 +2084,26 @@ }, "NetTrafficEvent": { "additionalProperties": false, - "description": "Records non-HTTP network traffic and socket-level activity.\n\nEmitted for connect/bind/send/recv style network events.\n\nUse this for protocol flow reconstruction outside HTTP abstractions.", + "description": "A socket operation of the guest: connect, bind, accept, send or receive.", "properties": { "pos": { "$ref": "#/$defs/TracePosition", - "description": "Execution position metadata for this event.\n\nUsed to align event payloads with timeline and execution context." + "description": "Position of the thread that did the operation." }, "event": { "const": "net_traffic", "default": "net_traffic", - "description": "Discriminator for socket traffic events.", + "description": "Event type of a socket operation.", "title": "Event", "type": "string" }, "server": { - "description": "Remote endpoint hostname or IP address.", + "description": "Host name or IP address of the socket peer.", "title": "Server", "type": "string" }, "port": { - "description": "Remote endpoint port number.", + "description": "Port of the socket peer.", "title": "Port", "type": "integer" }, @@ -2121,7 +2121,7 @@ } ], "default": null, - "description": "Socket operation that produced the event." + "description": "Socket operation of the event." }, "data_ref": { "anyOf": [ @@ -2133,7 +2133,7 @@ } ], "default": null, - "description": "Reference into the top-level report ``data`` store for the captured traffic payload bytes.", + "description": "Key in the top-level ``data`` of the first bytes that the guest sent or received.", "title": "Data Ref" }, "method": { @@ -2146,7 +2146,7 @@ } ], "default": null, - "description": "Source API path or method that produced the event.", + "description": "Name of the operation, such as ``winsock.connect``.", "title": "Method" } }, @@ -2184,26 +2184,26 @@ }, "ProcessCreateEvent": { "additionalProperties": false, - "description": "Records creation of a child process context.\n\nEmitted when process-creation APIs are invoked and accepted by emulation.\n\nThese events indicate lateral execution or staged payload spawning.", + "description": "A process that the guest creates, such as with ``CreateProcess`` or ``ShellExecute``.", "properties": { "pos": { "$ref": "#/$defs/TracePosition", - "description": "Execution position metadata for this event.\n\nUsed to align event payloads with timeline and execution context." + "description": "Position of the thread that did the operation." }, "event": { "const": "process_create", "default": "process_create", - "description": "Discriminator for process creation events.", + "description": "Event type of a process creation.", "title": "Event", "type": "string" }, "path": { - "description": "Target process image path.\n\nRepresents the emulated executable location requested by the sample.", + "description": "Emulated path of the image of the new process.", "title": "Path", "type": "string" }, "command_line": { - "description": "Command line used for process creation.\n\nUse this to recover execution parameters and staged arguments.", + "description": "Command line of the new process.", "title": "Command Line", "type": "string" }, @@ -2217,7 +2217,7 @@ } ], "default": null, - "description": "Target process identifier.\n\nThe actor (event pos) creates this process; the target may differ.", + "description": "ID of the new process.", "title": "Pid" } }, @@ -2231,21 +2231,21 @@ }, "RegCreateKeyEvent": { "additionalProperties": false, - "description": "Records registry key creation operations.\n\nEmitted when registry APIs create new keys.\n\nStrong signal for persistence and installation behavior.", + "description": "A registry key that the guest creates or opens with a create API, such as ``RegCreateKeyEx``.", "properties": { "pos": { "$ref": "#/$defs/TracePosition", - "description": "Execution position metadata for this event.\n\nUsed to align event payloads with timeline and execution context." + "description": "Position of the thread that did the operation." }, "event": { "const": "reg_create_key", "default": "reg_create_key", - "description": "Discriminator for registry key create events.", + "description": "Event type of a registry key creation.", "title": "Event", "type": "string" }, "path": { - "description": "Registry key path targeted by the create operation.", + "description": "Path of the key.", "title": "Path", "type": "string" }, @@ -2260,7 +2260,7 @@ } ], "default": null, - "description": "Registry handle returned for the key, when available.", + "description": "Key handle that the API returns.", "title": "Handle" }, "open_flags": { @@ -2276,7 +2276,7 @@ } ], "default": null, - "description": "Create/open disposition flags associated with the operation.", + "description": "Names of the disposition flags of the call.", "title": "Open Flags" }, "access_flags": { @@ -2292,7 +2292,7 @@ } ], "default": null, - "description": "Requested registry access rights.", + "description": "Names of the access rights that the caller asks for.", "title": "Access Flags" } }, @@ -2305,21 +2305,21 @@ }, "RegDeleteKeyEvent": { "additionalProperties": false, - "description": "Records registry key deletion.\n\nEmitted when registry APIs delete a key.\n\nUse this to track persistence cleanup and anti-forensics behavior.", + "description": "A registry key that the guest deletes.", "properties": { "pos": { "$ref": "#/$defs/TracePosition", - "description": "Execution position metadata for this event.\n\nUsed to align event payloads with timeline and execution context." + "description": "Position of the thread that did the operation." }, "event": { "const": "reg_delete_key", "default": "reg_delete_key", - "description": "Discriminator for registry key delete events.", + "description": "Event type of a registry key deletion.", "title": "Event", "type": "string" }, "path": { - "description": "Registry key path targeted by the delete operation.", + "description": "Path of the key.", "title": "Path", "type": "string" }, @@ -2334,7 +2334,7 @@ } ], "default": null, - "description": "Registry handle used for the delete, when available.", + "description": "Key handle of the call.", "title": "Handle" } }, @@ -2347,21 +2347,21 @@ }, "RegDeleteValueEvent": { "additionalProperties": false, - "description": "Records registry value deletion.\n\nEmitted when registry APIs delete a value from a key.\n\nUse this to track persistence cleanup and anti-forensics behavior.", + "description": "A registry value that the guest deletes.", "properties": { "pos": { "$ref": "#/$defs/TracePosition", - "description": "Execution position metadata for this event.\n\nUsed to align event payloads with timeline and execution context." + "description": "Position of the thread that did the operation." }, "event": { "const": "reg_delete_value", "default": "reg_delete_value", - "description": "Discriminator for registry value delete events.", + "description": "Event type of a registry value deletion.", "title": "Event", "type": "string" }, "path": { - "description": "Registry key path containing the deleted value.", + "description": "Path of the key that had the value.", "title": "Path", "type": "string" }, @@ -2376,7 +2376,7 @@ } ], "default": null, - "description": "Registry key handle used for the delete, when available.", + "description": "Key handle of the call.", "title": "Handle" }, "value_name": { @@ -2389,7 +2389,7 @@ } ], "default": null, - "description": "Registry value name deleted by the API call.", + "description": "Name of the value.", "title": "Value Name" } }, @@ -2402,21 +2402,21 @@ }, "RegListSubkeysEvent": { "additionalProperties": false, - "description": "Records enumeration of registry subkeys.\n\nEmitted when APIs list child keys beneath a registry path.\n\nHelps reveal discovery behavior for installed software and services.", + "description": "A registry key whose subkeys the guest lists.", "properties": { "pos": { "$ref": "#/$defs/TracePosition", - "description": "Execution position metadata for this event.\n\nUsed to align event payloads with timeline and execution context." + "description": "Position of the thread that did the operation." }, "event": { "const": "reg_list_subkeys", "default": "reg_list_subkeys", - "description": "Discriminator for registry subkey enumeration events.", + "description": "Event type of a subkey list.", "title": "Event", "type": "string" }, "path": { - "description": "Registry key path being enumerated.", + "description": "Path of the key.", "title": "Path", "type": "string" }, @@ -2431,7 +2431,7 @@ } ], "default": null, - "description": "Registry handle used for enumeration, when available.", + "description": "Key handle of the call.", "title": "Handle" } }, @@ -2444,21 +2444,21 @@ }, "RegOpenKeyEvent": { "additionalProperties": false, - "description": "Records registry key open operations.\n\nEmitted when registry APIs open existing keys.\n\nUse these to identify configuration and persistence probing paths.", + "description": "A registry key that the guest opens.", "properties": { "pos": { "$ref": "#/$defs/TracePosition", - "description": "Execution position metadata for this event.\n\nUsed to align event payloads with timeline and execution context." + "description": "Position of the thread that did the operation." }, "event": { "const": "reg_open_key", "default": "reg_open_key", - "description": "Discriminator for registry key open events.", + "description": "Event type of a registry key open.", "title": "Event", "type": "string" }, "path": { - "description": "Registry key path targeted by the open operation.", + "description": "Path of the key.", "title": "Path", "type": "string" }, @@ -2473,7 +2473,7 @@ } ], "default": null, - "description": "Registry handle returned for the key, when available.", + "description": "Key handle that the API returns.", "title": "Handle" }, "open_flags": { @@ -2489,7 +2489,7 @@ } ], "default": null, - "description": "Open/disposition flags associated with the operation.", + "description": "Names of the open flags of the call.", "title": "Open Flags" }, "access_flags": { @@ -2505,7 +2505,7 @@ } ], "default": null, - "description": "Requested registry access rights.", + "description": "Names of the access rights that the caller asks for.", "title": "Access Flags" } }, @@ -2518,21 +2518,21 @@ }, "RegReadValueEvent": { "additionalProperties": false, - "description": "Records registry value read operations.\n\nEmitted when value-query APIs request data from a key.\n\nUse this to identify configuration keys and anti-analysis checks.", + "description": "A registry value that the guest reads.", "properties": { "pos": { "$ref": "#/$defs/TracePosition", - "description": "Execution position metadata for this event.\n\nUsed to align event payloads with timeline and execution context." + "description": "Position of the thread that did the operation." }, "event": { "const": "reg_read_value", "default": "reg_read_value", - "description": "Discriminator for registry value read events.", + "description": "Event type of a registry value read.", "title": "Event", "type": "string" }, "path": { - "description": "Registry key path containing the queried value.", + "description": "Path of the key that has the value.", "title": "Path", "type": "string" }, @@ -2547,7 +2547,7 @@ } ], "default": null, - "description": "Registry key handle used for the read, when available.", + "description": "Key handle of the call.", "title": "Handle" }, "value_name": { @@ -2560,7 +2560,7 @@ } ], "default": null, - "description": "Registry value name requested by the API call.", + "description": "Name of the value.", "title": "Value Name" }, "size": { @@ -2573,7 +2573,7 @@ } ], "default": null, - "description": "Requested/returned value size in bytes.", + "description": "Size of the value data in bytes.", "title": "Size" }, "data_ref": { @@ -2586,7 +2586,7 @@ } ], "default": null, - "description": "Reference into the top-level report ``data`` store for the captured registry value bytes.", + "description": "Key in the top-level ``data`` of the first bytes of the value data.", "title": "Data Ref" }, "buffer": { @@ -2600,7 +2600,7 @@ } ], "default": null, - "description": "Destination buffer pointer.", + "description": "Address of the output buffer.", "title": "Buffer" } }, @@ -2613,21 +2613,21 @@ }, "RegWriteValueEvent": { "additionalProperties": false, - "description": "Records registry value write operations.\n\nEmitted when APIs set or overwrite a registry value.\n\nUse this to capture persistence and configuration mutations.", + "description": "A registry value that the guest writes.", "properties": { "pos": { "$ref": "#/$defs/TracePosition", - "description": "Execution position metadata for this event.\n\nUsed to align event payloads with timeline and execution context." + "description": "Position of the thread that did the operation." }, "event": { "const": "reg_write_value", "default": "reg_write_value", - "description": "Discriminator for registry value write events.", + "description": "Event type of a registry value write.", "title": "Event", "type": "string" }, "path": { - "description": "Registry key path containing the written value.", + "description": "Path of the key that has the value.", "title": "Path", "type": "string" }, @@ -2642,7 +2642,7 @@ } ], "default": null, - "description": "Registry key handle used for the write, when available.", + "description": "Key handle of the call.", "title": "Handle" }, "value_name": { @@ -2655,7 +2655,7 @@ } ], "default": null, - "description": "Registry value name written by the API call.", + "description": "Name of the value.", "title": "Value Name" }, "size": { @@ -2668,7 +2668,7 @@ } ], "default": null, - "description": "Written value size in bytes.", + "description": "Size of the value data in bytes.", "title": "Size" }, "data_ref": { @@ -2681,7 +2681,7 @@ } ], "default": null, - "description": "Reference into the top-level report ``data`` store for the written registry value bytes.", + "description": "Key in the top-level ``data`` of the first bytes of the value data.", "title": "Data Ref" }, "buffer": { @@ -2695,7 +2695,7 @@ } ], "default": null, - "description": "Source buffer pointer.", + "description": "Address of the input buffer.", "title": "Buffer" } }, @@ -2708,10 +2708,10 @@ }, "RegionInfo": { "additionalProperties": false, - "description": "Describes a memory region for exception context.", + "description": "A memory region near a fault.", "properties": { "tag": { - "description": "Memory-map tag identifying the region.", + "description": "Tag of the memory region.", "title": "Tag", "type": "string" }, @@ -2722,7 +2722,7 @@ "type": "string" }, "size": { - "description": "Region size in bytes.", + "description": "Size of the region in bytes.", "title": "Size", "type": "integer" }, @@ -2736,7 +2736,7 @@ } ], "default": null, - "description": "Permission string (e.g. ``r-x``).", + "description": "Permissions, such as ``r-x``.", "title": "Perms" } }, @@ -2793,10 +2793,10 @@ }, "StringCollection": { "additionalProperties": false, - "description": "Represents one categorized bucket of extracted strings.\n\nString extraction runs both on static image bytes and post-emulation memory\nsnapshots. This model keeps ANSI and UTF-16 findings separated.\n\nCompare these lists to identify runtime-decoded strings that were not present\nin the original file bytes.", + "description": "ANSI and UTF-16 strings from one source.", "properties": { "ansi": { - "description": "ASCII/ANSI strings extracted for this bucket.\n\nThese are populated by static string scans or memory string extraction depending on the parent field.", + "description": "ASCII strings.", "items": { "type": "string" }, @@ -2804,7 +2804,7 @@ "type": "array" }, "unicode": { - "description": "UTF-16 style strings extracted for this bucket.\n\nUse these to spot wide-character configuration values and command artifacts.", + "description": "UTF-16LE strings.", "items": { "type": "string" }, @@ -2817,15 +2817,15 @@ }, "StringsReport": { "additionalProperties": false, - "description": "Groups static and in-memory string extraction results.\n\nStatic strings come from the original loaded image, while in-memory strings\nare collected after execution and often include decoded/decrypted content.\n\nUse this section to quickly triage embedded versus runtime materialized\nindicators.", + "description": "Strings of the input file, and strings that the guest made in memory.", "properties": { "static": { "$ref": "#/$defs/StringCollection", - "description": "Strings extracted from the initial file image bytes.\n\nPrimarily influenced by the module loader and static string scanner." + "description": "Strings in the data of the input file." }, "in_memory": { "$ref": "#/$defs/StringCollection", - "description": "Strings extracted from memory after emulation executes.\n\nPopulated when string analysis is enabled and runtime decoding created new string material." + "description": "Strings in the stack and the heaps at the end of the emulation that are not in ``static``. These strings often are decoded data." } }, "title": "StringsReport", @@ -2833,25 +2833,25 @@ }, "SymAccessReport": { "additionalProperties": false, - "description": "Summarizes memory accesses to symbol-resolved addresses.\n\nSymbol access accounting is mainly populated when memory tracing is active\nand the emulator can resolve touched addresses back to known symbols.\n\nUse these counters to identify import-table walking and export resolution\nbehavior.", + "description": "Read, write and execute counts of one symbol address, from memory tracing.", "properties": { "symbol": { - "description": "Resolved symbol name in ``module.symbol`` form.\n\nThis identifies the logical target accessed by code during tracing.", + "description": "Symbol name in ``module.symbol`` form.", "title": "Symbol", "type": "string" }, "reads": { - "description": "Number of read accesses observed for this symbol.\n\nHigh values often indicate lookup loops or pointer chasing.", + "description": "Number of reads of the symbol address.", "title": "Reads", "type": "integer" }, "writes": { - "description": "Number of write accesses observed for this symbol.\n\nNon-zero writes can indicate tampering or patching behavior.", + "description": "Number of writes to the symbol address.", "title": "Writes", "type": "integer" }, "execs": { - "description": "Number of execution accesses observed for this symbol.\n\nExecution hits often represent direct calls/jumps through symbol-backed addresses.", + "description": "Number of times that the guest ran code at the symbol address.", "title": "Execs", "type": "integer" } @@ -2867,32 +2867,32 @@ }, "ThreadCreateEvent": { "additionalProperties": false, - "description": "Records thread creation in a process context.\n\nEmitted for thread creation APIs and often accompanies process injection\nworkflows.\n\nCombine with memory-write and start-address analysis for attribution.", + "description": "A thread that the guest creates in its own process.", "properties": { "pos": { "$ref": "#/$defs/TracePosition", - "description": "Execution position metadata for this event.\n\nUsed to align event payloads with timeline and execution context." + "description": "Position of the thread that did the operation." }, "event": { "const": "thread_create", "default": "thread_create", - "description": "Discriminator for thread creation events.", + "description": "Event type of a thread creation.", "title": "Event", "type": "string" }, "path": { - "description": "Target process path where the thread is created.", + "description": "Image path of the target process.", "title": "Path", "type": "string" }, "start_address": { - "description": "Thread start address.", + "description": "Start address of the thread.", "pattern": "^0x[0-9a-f]+$", "title": "Start Address", "type": "string" }, "param": { - "description": "Thread parameter pointer or value.", + "description": "Parameter that the thread routine gets.", "pattern": "^0x[0-9a-f]+$", "title": "Param", "type": "string" @@ -2907,7 +2907,7 @@ } ], "default": null, - "description": "Target process identifier.\n\nThe actor (event pos) creates the thread in this process; NULL when the target is the actor itself.", + "description": "ID of the target process. Absent when the target is the process in ``pos``.", "title": "Pid" }, "tid": { @@ -2920,7 +2920,7 @@ } ], "default": null, - "description": "Target thread identifier of the newly created thread, when known.", + "description": "ID of the new thread.", "title": "Tid" } }, @@ -2935,32 +2935,32 @@ }, "ThreadInjectEvent": { "additionalProperties": false, - "description": "Records thread injection activity into another process.\n\nEmitted for injection-oriented thread APIs.\n\nTreat these as high-signal indicators of code injection behavior.", + "description": "A thread that the guest creates in another process, such as with ``CreateRemoteThread``.", "properties": { "pos": { "$ref": "#/$defs/TracePosition", - "description": "Execution position metadata for this event.\n\nUsed to align event payloads with timeline and execution context." + "description": "Position of the thread that did the operation." }, "event": { "const": "thread_inject", "default": "thread_inject", - "description": "Discriminator for thread injection events.", + "description": "Event type of a thread injection.", "title": "Event", "type": "string" }, "path": { - "description": "Target process path receiving the injected thread.", + "description": "Image path of the target process.", "title": "Path", "type": "string" }, "start_address": { - "description": "Injected thread start address.", + "description": "Start address of the thread.", "pattern": "^0x[0-9a-f]+$", "title": "Start Address", "type": "string" }, "param": { - "description": "Injected thread parameter pointer or value.", + "description": "Parameter that the thread routine gets.", "pattern": "^0x[0-9a-f]+$", "title": "Param", "type": "string" @@ -2975,7 +2975,7 @@ } ], "default": null, - "description": "Target process identifier receiving the injected thread.\n\nNULL when the target is the actor itself.", + "description": "ID of the target process. Absent when the target is the process in ``pos``.", "title": "Pid" }, "tid": { @@ -2988,7 +2988,7 @@ } ], "default": null, - "description": "Target thread identifier of the injected thread, when known.", + "description": "ID of the new thread.", "title": "Tid" } }, @@ -3003,20 +3003,20 @@ }, "TracePosition": { "additionalProperties": false, - "description": "Locates where an event occurred in execution time and context.\n\nEvery emitted event carries this structure so consumers can reconstruct\nchronology and thread/process ownership.\n\nUse ``tick`` ordering for timeline reconstruction and ``pc`` for direct\ndisassembly alignment.", + "description": "Position of an event: the instruction count, the thread, the process and the pc.", "properties": { "tick": { - "description": "Instruction-count tick at the time of the event.\n\nUse this as the primary event ordering key within a run.", + "description": "Instruction count of the run at the time of the event. The events of a run are in tick order.", "title": "Tick", "type": "integer" }, "tid": { - "description": "Thread identifier active when the event was emitted.\n\nCorrelate this with thread creation/injection activity.", + "description": "ID of the thread that did the operation.", "title": "Tid", "type": "integer" }, "pid": { - "description": "Process identifier active when the event was emitted.\n\nUse this to separate behavior across process boundaries.", + "description": "ID of the process that did the operation.", "title": "Pid", "type": "integer" }, @@ -3031,7 +3031,7 @@ } ], "default": null, - "description": "Program counter at event emission time, when available.\n\nSome synthetic events may omit this if no precise PC context exists.", + "description": "Address of the instruction that did the operation. For an API event, the address of the call.", "title": "Pc" } }, @@ -3045,21 +3045,21 @@ } }, "additionalProperties": false, - "description": "Top-level Speakeasy output report model.\n\nThis object combines run telemetry, metadata for the emulated input, and\noptional analysis artifacts.\n\nTreat this as the canonical machine-readable output for downstream tooling.\n\nMachine words are ``0x``-prefixed hex strings: addresses, pointers,\nhandles, raw argument values, return values, register values and exception\ncodes. A 64-bit word can be above 2**53 or 2**63, and JavaScript, Go\n``interface{}``, jq and int64 stores such as SQLite lose such a value as a\nJSON number with no error. Quantities, such as sizes, counts, offsets,\nticks and IDs, are JSON numbers.\n\nA key is absent when Speakeasy did not collect its data. An empty list or\nobject means that Speakeasy collected the data and found nothing.", + "description": "Report of one emulation: the input metadata, the runs and the payloads.\n\nMachine words are ``0x``-prefixed hex strings: addresses, pointers,\nhandles, raw argument values, return values, register values and exception\ncodes. A 64-bit word can be above 2**53 or 2**63, and JavaScript, Go\n``interface{}``, jq and int64 stores such as SQLite lose such a value as a\nJSON number with no error. Quantities, such as sizes, counts, offsets,\nticks and IDs, are JSON numbers.\n\nA key is absent when Speakeasy did not collect its data. An empty list or\nobject means that Speakeasy collected the data and found nothing.", "properties": { "report_version": { "default": "6.0.0", - "description": "Schema/report format version generated by the profiler.\n\nUse this to route parsing logic when consuming multiple report versions.", + "description": "Version of the report format.", "title": "Report Version", "type": "string" }, "emulation_total_runtime": { - "description": "Total wall-clock runtime in seconds for the emulation session.\n\nDerived from profiler start/stop timing and rounded to milliseconds.", + "description": "Wall-clock time in seconds, rounded to milliseconds, from the last start of the run queue to the end of the emulation.", "title": "Emulation Total Runtime", "type": "number" }, "timestamp": { - "description": "Epoch timestamp (seconds) captured at emulation start time.\n\nUse this for timeline correlation with external logs.", + "description": "Unix time in seconds of the last start of the run queue.", "title": "Timestamp", "type": "integer" }, @@ -3073,7 +3073,7 @@ } ], "default": null, - "description": "Architecture of the input sample, when known.\n\nPopulated from loader metadata." + "description": "Architecture of the input." }, "path": { "anyOf": [ @@ -3085,7 +3085,7 @@ } ], "default": null, - "description": "Host path of the input file, when the input was loaded from a file.\n\nUseful for provenance tracking in batch pipelines.", + "description": "Host path of the input file. Absent when the input came from bytes.", "title": "Path" }, "sha256": { @@ -3098,7 +3098,7 @@ } ], "default": null, - "description": "SHA-256 hash of the emulated input bytes, when available.\n\nUse this as the primary stable sample identifier.", + "description": "SHA-256 of the input.", "title": "Sha256" }, "size": { @@ -3111,7 +3111,7 @@ } ], "default": null, - "description": "Input file size in bytes, when available.\n\nHelps validate sample identity and triage packaging anomalies.", + "description": "Size of the input in bytes.", "title": "Size" }, "file_type": { @@ -3124,7 +3124,7 @@ } ], "default": null, - "description": "Kind of input file, when known.\n\nDerived from loader inspection before emulation starts." + "description": "Kind of input file." }, "image_base": { "anyOf": [ @@ -3137,11 +3137,11 @@ } ], "default": null, - "description": "Base address at which the primary module was loaded during emulation.\n\nUse this to rebase analysis tools (e.g. IDA Pro) to match emulated addresses.", + "description": "Base address of the input image in the emulator. Rebase a disassembler database to this address to match the addresses of the report.", "title": "Image Base" }, "errors": { - "description": "Top-level emulator errors not tied to a specific run.\n\nThese usually reflect session-level failures rather than per-entry-point termination.", + "description": "Errors that do not belong to one run, such as an ``invalid_image`` defect of the input PE.", "items": { "$ref": "#/$defs/ErrorInfo" }, @@ -3158,18 +3158,18 @@ } ], "default": null, - "description": "Optional static and in-memory string extraction output.\n\nPresent only when string analysis is enabled in config." + "description": "Strings of the input and of memory. Only with ``analysis.strings``." }, "data": { "additionalProperties": { "$ref": "#/$defs/DataArtifact" }, - "description": "Deduplicated binary payload store keyed by SHA-256.\n\nResolve ``*_ref`` fields in the report against this mapping to recover compressed payload bytes.", + "description": "Payloads of the report, keyed by SHA-256. Each ``data_ref`` field holds a key of this mapping.", "title": "Data", "type": "object" }, "entry_points": { - "description": "Ordered list of all runs executed during emulation.\n\nThis is the primary behavioral timeline for the report.", + "description": "Runs of the emulation, in the order that the runs started.", "items": { "$ref": "#/$defs/EntryPoint" }, diff --git a/speakeasy/speakeasy.py b/speakeasy/speakeasy.py index 9fb92d8d6..14e17b832 100644 --- a/speakeasy/speakeasy.py +++ b/speakeasy/speakeasy.py @@ -63,7 +63,13 @@ class Speakeasy: - """Wrapper class for invoking the speakeasy emulators.""" + """ + Emulate a Windows PE file or shellcode. + + ``load_module``, ``load_shellcode`` or ``load_image`` creates the emulator for the input: + ``WinKernelEmulator`` for a driver, else ``Win32Emulator``. Use the object as a context + manager, or call ``shutdown`` at the end. + """ config: SpeakeasyConfig @@ -91,16 +97,17 @@ def __init__( gdb_host: str = "127.0.0.1", ) -> None: """ - Create a wrapper for the emulators. + Validate the configuration. The emulator does not exist until a load creates it. Args: - config: Emulator configuration as a raw dict or a validated ``SpeakeasyConfig``. - ``None`` selects the default configuration. + config: Configuration as a dict or a ``SpeakeasyConfig``. ``None`` selects the + default configuration. argv: Command line arguments for the emulated program. - debug: Whether to log each instruction and the registers at the DEBUG level. + debug: If true, log each instruction and the general-purpose registers at the DEBUG level. exit_event: Event that stops the emulation when it is set. - gdb_port: Port on which to listen for a GDB client. ``None`` disables the GDB server. - volumes: Host paths to mount into the emulated file system. + gdb_port: Port on which to listen for a GDB client. The first run waits for the client. + ``None`` disables the GDB server. + volumes: Host paths to mount into the emulated file system, in the ``--volume`` format. gdb_host: Address on which the GDB server listens. Raises: @@ -157,11 +164,7 @@ def __exit__( def _init_config(self, config: dict | SpeakeasyConfig | None) -> None: """ - Validate the configuration and store it. - - Args: - config: The configuration data to set up the emulator environment. - If ``None``, the default configuration is used. + Validate the configuration and store it. ``None`` selects the default configuration. Raises: ConfigError: if the configuration is not valid. @@ -180,10 +183,9 @@ def _init_config(self, config: dict | SpeakeasyConfig | None) -> None: def _init_pe_emulator(self, path: str | None = None, data: bytes | None = None) -> None: """ - Create the emulator that fits the PE metadata and its machine. + Create the kernel mode emulator for a driver, else the user mode emulator, and its machine. - Picks the user mode emulator or the kernel mode emulator and creates the - machine for the architecture of the PE. + For a user mode input that is not an EXE, also load the container image. Raises: InputError: if the PE is not valid, or its architecture is not x86 or x64. @@ -193,7 +195,6 @@ def _init_pe_emulator(self, path: str | None = None, data: bytes | None = None) pe = _PeParser(path=path, data=data) except (PEFormatError, ValueError) as err: raise InputError(f"Target file is not a valid PE: {err}") from err - # Get the machine type we only support x86/x64 atm machine = pe.FILE_HEADER.Machine mach = MACHINE_TYPE.get(machine, f"0x{machine:x}").split("_")[-1].lower() if mach not in ("amd64", "i386"): @@ -234,8 +235,7 @@ def _init_machine(self, emu: WindowsEmulator, arch: int) -> None: """ Create the machine of the emulator, then add the pending hooks. - The engine exists only after the machine is created, and the hooks must be in place before the load maps - the input. + The engine does not exist before ``init_machine``, and the hooks must see the maps of the load. """ emu.init_machine(arch) self._init_hooks() @@ -272,12 +272,12 @@ def disasm(self, addr: int, size: int, *, fast: bool = True) -> tuple[str, str, Args: addr: Address at which to begin the disassembly. - size: Number of bytes to include. - fast: Whether to use the fast disassembly path. + size: Number of bytes to read. + fast: If true, disassemble only the first instruction, with capstone's lite mode. Returns: - A tuple of the mnemonic, the operands, and the full instruction. If ``fast`` is false, - a list of capstone instructions. + If ``fast`` is true, a tuple of the mnemonic, the operands, and the full instruction. + If ``fast`` is false, a list of the capstone instructions in the bytes. Raises: SpeakeasyError: if the disassembly fails or the emulator does not exist. @@ -289,18 +289,16 @@ def disasm(self, addr: int, size: int, *, fast: bool = True) -> tuple[str, str, def is_pe(self, data: bytes) -> bool: """Check whether the data starts with the PE ``MZ`` header.""" - # Check for the PE header if data[:2] == b"MZ": return True return False def _auto_mount_target_directory(self, path: str) -> None: """ - Mount direct-child files in the target's host directory into the emulated current directory. + Mount the files of the host directory of the input into the emulated current directory. - This makes sibling files (for example data files the sample expects to find - in its working directory) available to the emulated program. Only - immediate children are mounted, not subdirectories. + Then the emulated program finds the files next to it, for example data files that it + reads from its working directory. The method does not mount subdirectories. """ target_dir = Path(path).resolve().parent if not target_dir.is_dir(): @@ -334,17 +332,21 @@ def load_module( self, path: str | None = None, data: bytes | None = None, filename: str | None = None ) -> RuntimeModule: """ - Load a PE module into the emulator. + Create the emulator for a PE file and load the file as the input image. + + A driver gets the kernel mode emulator. Other PE files get the user mode emulator, and a + DLL loads into the container process. If ``path`` is given, the emulator also mounts the + files of its directory into the emulated current directory. Args: - path: File path of the PE module. - data: Bytes of the PE module. - filename: Override for the module filename that the emulated program sees. - Useful when emulating from bytes and the malware checks its - own filename (for example through ``GetModuleFileName``). + path: File path of the PE file. + data: Bytes of the PE file. + filename: File name of the module that the emulated program sees. Use it when you + load from bytes and the sample checks its own file name, for example with + ``GetModuleFileName``. Returns: - The loaded module object from the emulator. + The module of the input image. Raises: InputError: if neither ``path`` nor ``data`` is given, the file does not exist, @@ -384,7 +386,8 @@ def load_image(self, image: LoadedImage) -> RuntimeModule: """ Load a ``LoadedImage`` into the emulator. - If there is no emulator yet, this method creates it and the image is the input image. + If there is no emulator yet, ``load_image`` creates the user mode emulator, and the image is + the input image. An image that is not an EXE then loads into the container process. """ is_input = self._emu is None if is_input: @@ -406,16 +409,18 @@ def run_module( entry_point: int | None = None, ) -> None: """ - Run a previously loaded module through the configured emulator. + Add the runs of a loaded module to the run queue, and run the queue. + + The runs are the TLS callbacks and the entry point of the module, and its exports if + ``all_entrypoints`` is true. Args: - module: The module whose entry point to run. - all_entrypoints: If true, emulate all exports. Otherwise, emulate - only the main PE entry point. - emulate_children: If true, emulate the child processes that this - module creates. Otherwise, emulate only this module. - entry_point: RVA to use as entry point instead of the - default PE entry point. + module: The module that ``load_module`` returned. + all_entrypoints: If true, also run the exports. For a driver, also send IRPs to its + dispatch functions and run its unload routine. + emulate_children: If true, run the child processes that the module creates, after + the module. The kernel mode emulator ignores this argument. + entry_point: RVA at which to start, in place of the entry point of the PE. Raises: SpeakeasyError: if no module is loaded yet. @@ -439,7 +444,7 @@ def load_shellcode( filename: str | None = None, ) -> int: """ - Load a shellcode blob into the emulation space. + Create the user mode emulator and load shellcode into the container process. Args: fpath: File path of the shellcode blob. @@ -474,12 +479,12 @@ def load_shellcode( @execution_scope def run_shellcode(self, sc_addr: int, stack_commit: int = 0x4000, offset: int = 0) -> None: """ - Run a previously loaded shellcode blob by address. + Run shellcode that ``load_shellcode`` loaded. Args: - sc_addr: Address of the loaded shellcode to emulate. - stack_commit: Number of bytes of stack to commit. - offset: Offset within the blob at which to begin emulation. + sc_addr: Address that ``load_shellcode`` returned. + stack_commit: Size of the stack in bytes, if the ``stack_size`` of the config is not set. + offset: Offset in the shellcode at which to start. Raises: SpeakeasyError: if no shellcode is loaded or the emulator is the kernel mode emulator. @@ -517,18 +522,23 @@ def add_api_hook( self, cb: ApiHookCallback, module: str = "", api_name: str = "", argc: int = 0, call_conv: int | None = None ) -> common.ApiHook: """ - Set a callback to fire when the emulated program calls a specified API. + Set a callback to fire when the emulated program calls an API. The emulator calls ``cb(emu, api_name, func, argv)`` in place of the API handler. ``argv`` holds the raw arguments, and ``func(argv)`` runs the handler. ``func`` is ``None`` when the API has no handler. The callback returns the return value of the API. + If more than one hook matches an API that has a handler, all of them run in the order + that you added them, and the API returns the value of the last hook. If the API has no + handler, only the hook that you added last runs. + Args: cb: Python function to call. - module: Name of the module that contains the target API. - api_name: Name of the API to hook. Wild cards (for example ``*``) are supported. - argc: Force the emulator to account for this number of arguments (for stack cleanup). - call_conv: Force the emulator to use this calling convention for the hook. + module: Name of the module, for example ``kernel32``. Wildcards such as ``*`` are supported. + api_name: Name of the API. Wildcards such as ``*`` are supported. + argc: Number of arguments of an API that has no handler. The emulator reads this + number of arguments and removes them from the stack. + call_conv: Calling convention of an API that has no handler. The default is stdcall. Returns: The hook object. Pass it to ``remove_hook`` to remove the hook. @@ -538,10 +548,10 @@ def add_api_hook( @execution_scope def resume(self, addr: int, count: int = -1) -> None: """ - Resume emulating at the specified address. + Start emulation at an address in the current run. Args: - addr: Address at which to begin emulation. + addr: Address at which to start. count: Number of instructions to emulate. ``-1`` means no limit. """ output_active_config(self.config, logger) @@ -555,13 +565,11 @@ def stop(self) -> None: def shutdown(self) -> None: """ - Close the emulation instance and release the underlying engine resources. + Remove the hooks that the engine added through the Unicorn C API. - Removes all Unicorn hooks so the garbage collector can finalize the native - engine without it accessing stale callbacks. The Uc handle stays alive on - purpose: Unicorn's ``uc_close`` has process-global side effects that can - corrupt other live engine instances. The garbage collector and the weak-ref - finalizer release it when the object graph is unreachable. + Then Unicorn does not call a hook after Python frees its callback. ``shutdown`` does not + call ``uc_close``, because ``uc_close`` changes state that the other Unicorn instances in + the process use. The garbage collector releases the Unicorn instance. """ if self._emu is not None: self._emu.emu_eng.close() @@ -569,11 +577,15 @@ def shutdown(self) -> None: @execution_scope def call(self, addr: int, params: list[int] | None = None) -> None: """ - Call the function at the specified address. + Run the function at an address as a new run. + + If the run queue is empty, the run starts at once. Otherwise, the emulator adds the run + to the end of the queue and does not start it. Args: addr: Address of the function. - params: Arguments to push onto the stack for the call. + params: Arguments of the call, with the stdcall convention on x86 and the Microsoft + x64 convention on x64. """ output_active_config(self.config, logger) self._init_hooks() @@ -581,7 +593,7 @@ def call(self, addr: int, params: list[int] | None = None) -> None: def add_code_hook(self, cb: CodeHookCallback, begin: int = 1, end: int = 0) -> common.CodeHook: """ - Set a callback to fire for every CPU instruction that is emulated. + Set a callback to fire for each instruction in an address range. The emulator calls ``cb(emu, address, size)`` before the instruction at ``address`` runs. The return value is not used. @@ -589,7 +601,8 @@ def add_code_hook(self, cb: CodeHookCallback, begin: int = 1, end: int = 0) -> c Args: cb: Python function to call. begin: Start of the address range to hook. - end: End of the address range to hook. + end: Last address of the range to hook. If ``begin`` is greater than ``end``, + the hook covers all addresses. Returns: The hook object. Pass it to ``remove_hook`` to remove the hook. @@ -598,10 +611,11 @@ def add_code_hook(self, cb: CodeHookCallback, begin: int = 1, end: int = 0) -> c def add_dyn_code_hook(self, cb: DynCodeHookCallback) -> common.DynCodeHook: """ - Set a callback to fire when dynamically generated or copied code runs. + Set a callback to fire when the guest runs code in memory that ``VirtualAlloc`` or a similar API allocated. - The emulator calls ``cb(emu, region)`` with the ``MemMap`` of the code. - The return value is not used. + The hook fires the first time that the start of the region runs, and each time that code + in the region calls an API. The emulator calls ``cb(emu, region)`` with the ``MemMap`` of + the code. The return value is not used. Args: cb: Python function to call. @@ -613,7 +627,7 @@ def add_dyn_code_hook(self, cb: DynCodeHookCallback) -> common.DynCodeHook: def add_mem_read_hook(self, cb: MemAccessHookCallback, begin: int = 1, end: int = 0) -> common.ReadMemHook: """ - Set a callback to fire when a memory address is read from. + Set a callback to fire when the guest reads mapped memory. The emulator calls ``cb(emu, access, address, size, value)``. The return value is not used. @@ -629,7 +643,7 @@ def add_mem_read_hook(self, cb: MemAccessHookCallback, begin: int = 1, end: int def add_mem_write_hook(self, cb: MemAccessHookCallback, begin: int = 1, end: int = 0) -> common.WriteMemHook: """ - Set a callback to fire when a memory address is written to. + Set a callback to fire when the guest writes mapped memory. The emulator calls ``cb(emu, access, address, size, value)``, where ``value`` is the value that the instruction writes. The return value is not used. @@ -706,7 +720,7 @@ def add_SYSCALL_instruction_hook( def add_invalid_instruction_hook(self, cb: InvalidInstructionHookCallback) -> common.InvalidInstructionHook: """ - Set a callback to fire when the emulator hits an invalid instruction. + Set a callback to fire when the guest runs an invalid instruction. The emulator calls ``cb(emu)``. The callback returns ``True`` if it handled the instruction, so that the hooks after it do not run, or ``False`` to let the next hook @@ -722,7 +736,7 @@ def add_invalid_instruction_hook(self, cb: InvalidInstructionHookCallback) -> co def add_mem_invalid_hook(self, cb: MemAccessHookCallback) -> common.InvalidMemHook: """ - Set a callback to fire when a memory access violation occurs. + Set a callback to fire when the guest accesses unmapped or protected memory. The emulator calls ``cb(emu, access, address, size, value)``. The callback returns ``False`` to stop the hooks after it and fail the access. Any other value lets the next @@ -738,7 +752,7 @@ def add_mem_invalid_hook(self, cb: MemAccessHookCallback) -> common.InvalidMemHo def add_interrupt_hook(self, cb: InterruptHookCallback) -> common.InterruptHook: """ - Set a callback to fire for software interrupts. + Set a callback to fire for interrupts and CPU exceptions. The emulator calls ``cb(emu, number)`` with the interrupt number. The return value is not used. @@ -752,24 +766,15 @@ def add_interrupt_hook(self, cb: InterruptHookCallback) -> common.InterruptHook: def get_registry_key(self, handle: int = 0, path: str = "") -> RegKey | None: """ - Get a registry key by path or handle. - - Args: - handle: Handle of the registry key. - path: Path of the registry key. + Get a registry key by path, or by handle if ``path`` is empty. Returns: - The registry key object, if the handle or path is valid. + The registry key, or ``None`` if no key has the path or the handle. """ return self.emu.reg_get_key(handle=handle, path=path) def get_address_map(self, addr: int) -> MemMap | None: - """ - Get the memory map that holds the specified address. - - Args: - addr: Address in the emulation space. - """ + """Get the memory map that holds an address, or ``None`` if the address is not mapped.""" mm: MemMap | None = self.emu.get_address_map(addr) return mm @@ -778,10 +783,10 @@ def mem_alloc(self, size: int, base: int | None = None, tag: str = "speakeasy.No Allocate a block of memory in the emulation space. Args: - size: Size of the requested memory block. - base: Requested base address. If it is in use, the emulator picks - the next nearest address. - tag: Tag to assign to the new memory mapping. + size: Size of the block in bytes. + base: Requested base address. If the range is not free, the emulator uses the next + free range above it. + tag: Tag of the new memory map. Returns: Address of the newly allocated memory block. @@ -822,14 +827,10 @@ def mem_write(self, addr: int, data: bytes) -> None: def mem_cast(self, obj: EmuStruct, addr: int) -> Any: """ - Cast an address as an object for easy access. - - Args: - obj: Object type to cast into. - addr: Address of the data to cast into ``obj``. + Read a structure from memory. Returns: - Python object based on the data at ``addr``. + ``obj``, with its fields set from the bytes at ``addr``. """ return self.emu.mem_cast(obj, addr) @@ -861,8 +862,8 @@ def create_file_archive(self) -> bytes: """ Create a zip archive of the dropped files. - The archive contains a manifest that matches the metadata of the dropped - files with the acquired files. + The archive has ``speakeasy_manifest.json``, which gives the emulated path, the size and + the SHA-256 of each file. Returns: The bytes of the zip archive, or empty bytes if the sample dropped no files. @@ -898,11 +899,11 @@ def get_mem_maps(self) -> list[MemMap]: def get_memory_dumps(self) -> Generator[tuple[str | None, int, int, bool, Any, bytes], None, None]: """ - Yield the contents of all readable memory maps with context information. + Yield the contents of the memory maps. Yields: - Tuples of tag, base, size, free flag, process, and data. Maps that cannot be read - are skipped. + Tuples of tag, base, size, free flag, process, and data. The method skips the maps + that it cannot read. """ for mm in self.emu.get_mem_maps(): base = mm.base @@ -932,19 +933,20 @@ def read_mem_string(self, address: int, width: int = 1, max_chars: int = 0) -> s def get_symbols(self) -> dict[int, tuple[str, str]]: """ - Get a snapshot of public addresses mapped to ``(dll, name)`` tuples. + Get a snapshot of the symbols, as ``(dll, name)`` tuples by address. - Includes mapped API entries and auxiliary symbols. Take a new snapshot after - the emulator resolves new modules or functions. Private dispatch tokens are excluded. + The symbols are the exports of the emulated DLLs and the other labels that the emulator + adds, such as ``KiSystemCall64``. API tokens and forwarder strings are not symbols. Get + a new snapshot after the emulator loads a module or resolves a function. """ return self.emu.get_symbols() def get_ret_address(self) -> int: - """Get the value stored at the top of the stack.""" + """Get the value at the top of the stack, which is the return address at the start of a call.""" return self.emu.get_ret_address() def set_ret_address(self, addr: int) -> None: - """Set the return address on the stack.""" + """Write a value at the top of the stack, which is the return address at the start of a call.""" return self.emu.set_ret_address(addr) def push_stack(self, val: int) -> None: @@ -978,7 +980,7 @@ def set_pc(self, addr: int) -> None: def reset_stack(self, base: int) -> tuple[int, int]: """ - Reset the stack to the supplied base address. + Set the stack pointer and the frame pointer below ``base``, with room for stack arguments. Returns: A tuple of the stack base and the stack pointer. @@ -998,7 +1000,7 @@ def get_ptr_size(self) -> int: return self.emu.ptr_size def get_all_registers(self) -> dict[str, str]: - """Get the state of all registers as a dict of zero-padded hex strings.""" + """Get the general-purpose registers and the instruction pointer as zero-padded hex strings.""" width = 2 + self.get_ptr_size() * 2 return {name: f"{value:#0{width}x}" for name, value in self.emu.get_register_state().items()} @@ -1012,21 +1014,16 @@ def get_symbol_from_address(self, address: int) -> str | None: return self.emu.get_symbol_from_address(address) def get_api_symbols(self) -> dict[int, str]: - """ - Get public addresses mapped to ``dll.name`` labels, including auxiliary symbols. - - Take a new snapshot after the emulator resolves new modules or functions. Private - dispatch tokens and export-directory forwarder strings are excluded. - """ + """Get a snapshot of the symbols of ``get_symbols``, as ``dll.name`` strings by address.""" return self.emu.get_api_symbols() def is_address_valid(self, address: int) -> bool: - """Check whether the address is reserved or mapped.""" + """Check whether the memory manager mapped or reserved the address.""" return self.emu.is_address_valid(address) def add_mem_map_hook(self, cb: MemMapHookCallback, begin: int = 1, end: int = 0) -> common.MapMemHook: """ - Set a callback to fire when a memory address is mapped. + Set a callback to fire when the emulator maps memory. The emulator calls ``cb(emu, base, size, tag, perms, flags)`` for each new map with a base in the range. The return value is not used. diff --git a/speakeasy/struct.py b/speakeasy/struct.py index e46598a13..4e0486b47 100644 --- a/speakeasy/struct.py +++ b/speakeasy/struct.py @@ -8,14 +8,14 @@ class EmuStructException(Exception): - """Container class for struct exceptions.""" + """``create_struct`` raises this exception when it cannot construct a nested struct.""" class Enum: """ - Basic Python object that serves as a C style enum. + A C-style enum, with the constants as attributes. - It supports arbitrary attribute access for dynamic enum-style constants. + ``__setattr__`` and ``__getattr__`` let mypy accept any attribute name. """ def __setattr__(self, name: str, value: Any) -> None: @@ -26,20 +26,20 @@ def __getattr__(self, name: str) -> Any: class PtrMeta(type): - """Metaclass for pointer types.""" + """Metaclass that makes ``Ptr * n`` declare an array of ``n`` pointers.""" def __mul__(self, mult): return tuple((self, mult)) class Ptr(metaclass=PtrMeta): - """Generic object that identifies pointer variables, which expand to the ``ptr_size`` passed to ``__init__``.""" + """A pointer field type. ``create_struct`` gives the field the width of ``ptr_size``.""" _points_to_ = None class CMeta(type): - """Metaclass that hooks ``__call__`` and keeps ``__dict__`` ordered on all Python versions.""" + """Metaclass that builds the ctypes struct after ``__init__``, and makes ``Struct * n`` declare an array.""" @classmethod def __prepare__(metacls, name: str, bases: tuple[type, ...], /, **kwds: Any) -> MutableMapping[str, object]: @@ -87,14 +87,11 @@ def __init__(self, ptr_size, width): (``self.Apc = KAPC(ptr_size).get_cstruct()``), and mypy gives these fields their ctypes type. """ - # Save the unique types we create here - # This is necessary since the ctypes metaclass won't allow us - # to assign structures even if the types are identical on the surface. - # ctypes will test the id (address) of the type to make sure they match - # upon assignment. Otherwise we will hit spurious TypeErrors. + # ctypes compares struct types by identity when it assigns a struct to a field, so two types with the + # same layout are not compatible. ``__struct_factory`` caches each type and reuses it for the same name and layout. __types__: dict[tuple, type] = {} - # The fields that each class declares at class level, resolved at its first instance + # The cache of ``get_declared_fields``, by class. __declared__: dict[type, list[tuple[str, Any]]] = {} class FilteredStruct(ct.Structure): @@ -102,7 +99,7 @@ def __hash__(self): return hash(repr(self)) def __init__(self, ptr_size=0, pack=0): - # Set __dict__ directly here to avoid __getattribute__ loops + # ``__setattr__`` raises AttributeError until ``__struct__`` exists, so write ``__dict__`` directly. self.__dict__["__pack__"] = pack self.__dict__["__struct__"] = None self.__dict__["__fields__"] = [] @@ -116,8 +113,8 @@ def get_declared_fields(cls) -> list[tuple[str, Any]]: Get the fields of the class-level declarations, with the fields of the base classes first. A field is ``Annotated`` with its type (``x: Annotated[int, ct.c_uint32]``), or is a nested struct - (``x: UNICODE_STRING``). The annotations are resolved at the first instance, so a field can name a - struct that the module defines later. + (``x: UNICODE_STRING``). This method resolves the annotations at the first instance, so a field can + name a struct that the module defines later. """ fields = EmuStruct.__declared__.get(cls) if fields is None: @@ -134,9 +131,9 @@ def add_fields(self, **fields: Any) -> None: """ Add fields at the end of the struct, in the order of the arguments. - ``__init__`` calls it for the fields whose ctypes type depends on ``ptr_size`` or on a constructor - parameter, and for the fields that come after them. A nested struct field also has a class-level - declaration, which puts it with the declared fields, so this method moves it to the end. + The ``__init__`` of a subclass calls this method for the fields whose ctypes type depends on ``ptr_size`` + or on a constructor parameter, and for the fields that come after them. If a field is also in the + declared fields, for example a nested struct, this method moves the field to the end. """ for name, ctype in fields.items(): self.__dict__.pop(name, None) @@ -149,12 +146,13 @@ def _is_ctype(self, obj): def create_struct(self): """ - Walk each attribute and build the ctypes structure. + Build the ctypes struct from the field attributes in ``__dict__``. - ``ctypes.Structure`` uses a metaclass, so this method builds ``_fields_`` dynamically through a factory. + ctypes reads ``_fields_`` when it creates the class, so this method collects the fields first, and then + ``__struct_factory`` creates the class. Raises: - EmuStructException: if a nested structure cannot be built. + EmuStructException: if the constructor of a nested struct does not accept ``ptr_size``. """ if self.__struct__: return @@ -188,17 +186,12 @@ def create_struct(self): continue try: if self._is_ctype(obj): - # Simply append ctypes since they will be handled - # automatically self.__fields__.append((d, obj)) elif issubclass(obj, Ptr): - # Expand pointers to the required width self.__fields__.append((d, self.get_ptr_field())) elif issubclass(obj, EmuStruct): - # Allow nesting of this class which we are calling a - # "filter class". That is, when fields are accessed in the - # underlying ctypes struct, we pass the getattr/setattr through + # ``__getattribute__`` returns the nested EmuStruct from ``__filtermap__``, not its ctypes struct. if obj.__name__ != self.__class__.__name__: try: filt = obj(self.__ptrsize__, self.__pack__) @@ -216,7 +209,7 @@ def create_struct(self): self.__init_struct() def get_ptr_field(self): - """Get ctypes value for the required pointer size.""" + """Get the ctypes type for a pointer of ``ptr_size``.""" if self.__ptrsize__ == 4: return ct.c_uint32 if self.__ptrsize__ == 8: @@ -224,7 +217,11 @@ def get_ptr_field(self): return ct.c_void_p def get_pack(self): - """Get the required structure pack (defaults to pointer size).""" + """ + Get ``pack`` if it is set, else twice ``ptr_size``, else 1. + + MSVC packs structs to 8 bytes on x86 and to 16 bytes on x64 by default. + """ if self.__pack__: return self.__pack__ if self.__ptrsize__: @@ -232,7 +229,7 @@ def get_pack(self): return 1 def _link_cstructs(self, obj): - """Link nested ctypes structures together so that the buffer API can convert them to bytes.""" + """Copy the ctypes struct of each nested EmuStruct into its field of the parent ctypes struct.""" for fname, subobj in obj.__filtermap__.items(): if isinstance(subobj, list): x = getattr(obj.__struct__, fname) @@ -246,7 +243,7 @@ def _link_cstructs(self, obj): setattr(obj.__struct__, fname, subobj.__struct__) def get_bytes(self): - """Convert the structure to bytes and respecting endianness.""" + """Get the bytes of the struct, with the current values of the nested structs.""" self._link_cstructs(self) struct = self.__struct__ @@ -273,7 +270,7 @@ def _deep_cast(self, obj, bytez, offset): self._deep_cast(subobj, bytez, field_offset) def cast(self, bytez): - """Convert a bytes object to the C structure by "casting" them.""" + """Set the struct and its nested structs from ``bytez``.""" self._deep_cast(self, bytez, 0) return self @@ -281,7 +278,7 @@ def get_cstruct(self): return self.__struct__.__class__ def __struct_factory(self, name): - """Factory used to generate ctypes structures using the ctypes metaclass.""" + """Get the ctypes struct type for ``__fields__``, from ``__types__`` if the layout is already there.""" _type_name = f"ct{name}{self.__ptrsize__}" key = (_type_name, self.get_pack(), tuple(self.__fields__)) _type = EmuStruct.__types__.get(key) @@ -299,7 +296,11 @@ def __init_struct(self): return self.__struct__ def __setattr__(self, name, value): - """Hook ``setattr`` to handle access to the underlying ctypes structure.""" + """ + Write the ctypes struct field if ``name`` is a field, else set a Python attribute. + + A bytes value overwrites the start of an array field. + """ struct = self.__struct__ if struct: fields = struct._fields_ @@ -314,7 +315,11 @@ def __setattr__(self, name, value): super().__setattr__(name, value) def __getattribute__(self, name): - """Hook ``getattribute`` to handle access to the underlying ctypes structure.""" + """ + Read the ctypes struct field if ``name`` is a field. + + For a nested struct or an array of structs, return the EmuStruct or the list in ``__filtermap__``. + """ try: struct = super().__getattribute__("__struct__") if struct: diff --git a/speakeasy/volumes.py b/speakeasy/volumes.py index e4a05d354..71ba11981 100644 --- a/speakeasy/volumes.py +++ b/speakeasy/volumes.py @@ -1,4 +1,4 @@ -"""Docker-style --volume support for mounting host files into the emulated filesystem.""" +"""The ``--volume`` option: add host files to the emulated file system, as Docker volumes do.""" from __future__ import annotations @@ -9,9 +9,8 @@ def parse_volume_spec(spec: str) -> tuple[Path, PureWindowsPath]: r""" Parse a ``host_path:guest_path`` volume specification. - Handles Windows drive-letter colons on both sides (for example - ``C:\samples:C:\guest``). The separator is the first ``:`` - that is not the second character of a drive-letter prefix. + Each path can start with a Windows drive letter, for example ``C:\samples:C:\guest``. The + separator is the first ``:`` after the drive letter of the host path. Raises: ValueError: if the specification is empty or malformed. @@ -19,11 +18,8 @@ def parse_volume_spec(spec: str) -> tuple[Path, PureWindowsPath]: if not spec: raise ValueError("Empty volume specification") - # Find the separator colon. Skip a leading drive letter (X:) on the - # host side, then look for the next colon. start = 0 if len(spec) >= 2 and spec[1] == ":": - # Host path starts with a drive letter — skip past it. start = 2 idx = spec.find(":", start) @@ -86,10 +82,14 @@ def expand_volume_to_entries(host_path: Path, guest_path: PureWindowsPath) -> li def apply_volumes(config: dict, volume_specs: list[str]) -> dict: """ - Parse *volume_specs* and prepend the resulting file entries to *config*. + Parse ``volume_specs`` and put the file entries before the file entries of ``config``. - Entries are prepended so they win first-match resolution in - ``get_emu_file()``. + ``FileManager.get_emu_file`` uses the first ``full_path`` entry that matches, so the volume + entries come first. + + Raises: + ValueError: if a specification is empty or malformed. + FileNotFoundError: if a host path does not exist. """ if not volume_specs: return config diff --git a/speakeasy/windows/api_image.py b/speakeasy/windows/api_image.py index 825f7f614..11e51ed06 100644 --- a/speakeasy/windows/api_image.py +++ b/speakeasy/windows/api_image.py @@ -1,8 +1,8 @@ """ -Deterministic, mapped PE images for synthetic API modules. +The PE images of the synthetic API modules. -Raw section offsets equal RVAs, so the regions also form a parseable PE file. -The registry patches the faulting public entries after allocating private traps. +The raw offset of each section is equal to its RVA, so the regions are also a valid PE file. +The API registry writes the entry code of each function export when it gives the export an API token. """ from __future__ import annotations @@ -32,9 +32,9 @@ class ApiExportSpec: def encode_api_stub(arch: int, entry_address: int, trap_address: int) -> bytes: - """Encode a stack/flags-neutral jump from a public entry to its private trap.""" + """Encode the entry code that jumps to the API token. The code does not change the stack or the flags.""" if arch == _arch.ARCH_X86: - # Inline hooks match the Windows prologue; pop ebp undoes it before the jump. + # Inline hooks look for the Windows hot-patch prologue. pop ebp undoes it before the jump. return b"\x8b\xff\x55\x8b\xec\x5d\xe9" + struct.pack(" LoadedImage: """ - Build a valid PE with sparse EAT ordinals and independently sorted names. + Build a valid PE image with the given exports. - base is the preferred base. The image has no absolute addresses, so it can load at any base. - Explicit ordinal aliases share a target. Unknown ordinals use the lowest - available positive value in byte-sorted name order. Export spans are bounded - by the WORD indices in the export name ordinal table. + ``base`` is the preferred base. The image has no absolute addresses, so it can load at any base. + Exports with the same explicit ordinal share one EAT entry. An export without an + ordinal gets the lowest free ordinal, in the byte order of the names. The EAT can + have gaps between the ordinals. + + Raises: + ValueError: if a name is not ASCII, the exports do not fit the WORD indices of the + export directory, two exports with one ordinal are different, or the image + does not fit the address space. """ _ascii_string(name, "module name") specs = sorted(exports, key=lambda e: (e.name is None, (e.name or "").encode("ascii"), e.ordinal or 0)) @@ -257,7 +262,8 @@ def make_image(self) -> LoadedImage: if handler is None: continue for key, func in handler.funcs.items(): - # Handlers register each ordinal under its name as well. + # Handlers register each ordinal under its name too. ntdll exports only the Nt and Zw + # handlers of ntoskrnl. if isinstance(key, int) or (handler is nt_handler and not key.startswith(("Nt", "Zw"))): continue handler_ordinals[key] = func[4] @@ -269,10 +275,10 @@ def kind(name: str | None, physical_kind: str = "function") -> str: manifest = get_export_manifest(self._name.lower(), arch_name) if manifest is not None: - # The physical table owns names and ordinals. Handler-only names - # follow the highest physical ordinal so that they never take an - # ordinal that the real module leaves unused. A handler name whose - # A or W variant is physical only serves dispatch of that variant. + # The manifest defines the names and ordinals. A handler name that is not in the + # manifest gets an ordinal after the highest manifest ordinal, so that it does not + # take an ordinal that the real module does not use. If the manifest has the A or W + # variant of a handler name, the emulator uses the handler name only to find the handler. exports = [ApiExportSpec(e.name, e.ordinal, kind(e.name, e.kind)) for e in manifest.exports] physical = {e.name for e in manifest.exports} next_ordinal = max((e.ordinal for e in manifest.exports), default=0) + 1 @@ -281,13 +287,13 @@ def kind(name: str | None, physical_kind: str = "function") -> str: exports.append(ApiExportSpec(name, next_ordinal, kind(name))) next_ordinal += 1 else: - # The catalog owns surface membership. Permissive lookup serves - # ABI reuse and must never determine exported names. + # The declarations of this DLL and the handlers give the export names. A declaration + # of the name in a different DLL can give the arguments of a call, but not an export. specs = {name: ApiExportSpec(name) for name in self._signature_db.get_functions(self._name, arch_name)} specs.update({name: ApiExportSpec(name, ordinal) for name, ordinal in handler_ordinals.items()}) if self._name == "ntoskrnl": - # The kernel exports native services under both prefixes, and - # dispatch folds each pair onto one handler. + # ntoskrnl exports the system services with the Nt and the Zw prefix, and one + # handler serves both names. for name in list(handler_ordinals): if name.startswith(("Nt", "Zw")): alias = ("Zw" if name.startswith("Nt") else "Nt") + name[2:] @@ -298,9 +304,8 @@ def kind(name: str | None, physical_kind: str = "function") -> str: try: image_name.encode("ascii") except UnicodeEncodeError: - # The PE export-directory label need not be the loaded filename. - # Keep that label ASCII while retaining the actual Unicode module - # identity in the registry, loader lists and emulated path. + # The name in the export directory can differ from the module name. Use an ASCII + # name there, and keep the real name for the registry, the loader lists and the path. image_name = "speakeasy" image = build_api_image( name=image_name, diff --git a/speakeasy/windows/api_registry.py b/speakeasy/windows/api_registry.py index 08600e380..a25cdaeb7 100644 --- a/speakeasy/windows/api_registry.py +++ b/speakeasy/windows/api_registry.py @@ -1,4 +1,4 @@ -"""Session-local ownership of public API entries and private dispatch traps.""" +"""The API registry: the export entries of the loaded modules, and the API tokens of the synthetic exports.""" from __future__ import annotations @@ -20,8 +20,7 @@ def module_name(value: str) -> str: - # Strip only image extensions so that repeated normalization keeps dotted - # module names such as windows.storage intact. + # Strip only image extensions, so that a second call keeps a dotted name such as windows.storage. name = ntpath.basename(value).lower() root, extension = ntpath.splitext(name) return root if extension in MODULE_EXTENSIONS else name @@ -89,7 +88,7 @@ def _allocate_trap(self) -> int: return self.trap_base + offset def unregister_module(self, module: RuntimeModule) -> None: - """Discard an unsuccessful load without reusing published trap tokens.""" + """Remove the entries of a module that failed to load. The emulator never uses their tokens again.""" addresses = [address for address, entry in self.entries.items() if entry.module is module] for address in addresses: entry = self.entries.pop(address) @@ -148,7 +147,7 @@ def dynamic(self, module: RuntimeModule, reference: str | int) -> ApiEntry: if arena is None or offset + API_SLOT_SIZE > arena.virtual_size: raise WindowsEmuError(f"dynamic API arena exhausted for {module.name}") slot = module.base + arena.virtual_address + offset - # Never overwrite guest patches or restore guest-changed page protections. + # Do not write over a guest patch, and do not restore a protection that the guest changed. perms = next( (p for start, end, p in self.emu.get_mem_regions() if start <= slot and slot + API_SLOT_SIZE <= end + 1), 0 ) @@ -182,7 +181,7 @@ def symbol(self, address: int) -> str | None: entry = self.entries.get(address) if entry is not None: return entry.symbol - # Interior lookup is for presentation only; it never controls dispatch. + # A name for an address inside a slot is only for display. Dispatch uses only the token. index = bisect_right(self._addresses, address) - 1 if index >= 0: start = self._addresses[index] diff --git a/speakeasy/windows/api_resolution.py b/speakeasy/windows/api_resolution.py index 57a7da325..dba650a56 100644 --- a/speakeasy/windows/api_resolution.py +++ b/speakeasy/windows/api_resolution.py @@ -20,8 +20,8 @@ class HandlerImplementation: """ The API has a handler. - For an import by ordinal, name is the handler name. Otherwise it is the name that - the guest called. + For an import by ordinal, ``name`` is the handler name. Otherwise it is the name + that the guest called. """ module: ApiHandler @@ -36,15 +36,15 @@ class UserHookImplementation: """ Speakeasy has no handler for the API, but the user added hooks for it. - The user adds hooks with add_api_hook. The module and API names of a hook can have wildcards, so one - hook can match many APIs. The hook takes the place of the handler: it gives - the argument count and the calling convention, and its return value goes to - the guest. Only the last hook that was added runs, because there is no - handler for the hooks to call, and the hooks can give different argument - counts. + The user adds hooks with ``add_api_hook``. The module and API names of a hook can + have wildcards, so one hook can match many APIs. The hook takes the place of the + handler: it gives the argument count and the calling convention, and the emulator + gives its return value to the guest. The emulator runs only the hook that the user + added last, because there is no handler for the hooks to call, and the hooks can + give different argument counts. - If the API has a handler, the result is HandlerImplementation, and each - hook for the API can call the handler. + If the API has a handler, the result is ``HandlerImplementation``, and each hook + for the API can call the handler. """ hooks: list[common.ApiHook] @@ -63,7 +63,7 @@ class SignatureImplementation: @dataclass(frozen=True) class StubImplementation: - """The API is not known, and functions_always_exist is set.""" + """The API is not known, and ``functions_always_exist`` is set.""" @dataclass(frozen=True) @@ -92,10 +92,8 @@ def get_handler_candidates(dll: str, name: str) -> Iterator[tuple[str, str]]: """ Yield other (dll, name) pairs whose handler can run for dll!name, best first. - For example: the name without the A or W suffix, the name without the CRT - "_o_" prefix, the Zw name for an Nt name, ntoskrnl and msvcrt for ntdll, the - modules that have the handlers of an API set or DLL, advapi32 for kernel32 - Reg functions, and psapi for kernel32 K32 functions. + For example: the name without the A or W suffix, the Zw name for an Nt name, or + the modules that implement an API set. """ seen = {(dll.lower(), name)} for cand_dll, cand_name in _get_aliases(dll, name): @@ -107,7 +105,7 @@ def get_handler_candidates(dll: str, name: str) -> Iterator[tuple[str, str]]: def _get_aliases(dll: str, name: str) -> Iterator[tuple[str, str]]: alias = "" - # Handle ANSI vs UNICODE functions + # One handler without the suffix can serve the A and the W variant if name.endswith(("A", "W")): alias = name[:-1] @@ -115,7 +113,7 @@ def _get_aliases(dll: str, name: str) -> Iterator[tuple[str, str]]: if name.startswith("_o_"): alias = name[3:] - # Bridge ntdll funcs to ntoskrnl if supported + # ntdll uses the ntoskrnl handler of an API, if one exists if dll.lower().startswith("ntdll"): yield "ntoskrnl", name nt_alias = _get_nt_zw_alias(name) or alias @@ -125,7 +123,7 @@ def _get_aliases(dll: str, name: str) -> Iterator[tuple[str, str]]: yield "msvcrt", name return - # Handle Zw*/Nt* function overlap + # One ntoskrnl handler serves both the Nt and the Zw name if dll.lower().startswith("ntoskrnl"): alias = _get_nt_zw_alias(name) or alias diff --git a/speakeasy/windows/com.py b/speakeasy/windows/com.py index b51fd18de..54d3c98a3 100644 --- a/speakeasy/windows/com.py +++ b/speakeasy/windows/com.py @@ -14,7 +14,7 @@ class COM: - """The Component Object Model (COM) manager for the emulator. This will manage COM interfaces.""" + """Build the COM interface vtables in guest memory.""" def __init__(self, config: SpeakeasyConfig): super().__init__() @@ -25,15 +25,20 @@ def get_method_callback(self, emu, iface_name, method_name): """ Get the address that a vtable slot of ``iface_name`` holds for ``method_name``. - A method without a handler gets an address too, so - that a call to it stops the run as an unsupported API. + A method without a handler gets an address too, so that a call to it goes through handler + resolution like an API without a handler. """ method = getattr(com_api.ComApi, f"{iface_name}_{method_name}", None) func_name = method.__apihook__[0] if method else f"{iface_name}.{method_name}" return emu.add_callback(com_api.ComApi.name, func_name) def get_interface(self, emu, ptr_size, name): - """Get COM interface.""" + """ + Map a vtable for a COM interface, with one method address in each slot. + + Raises: + Win32EmuError: if the interface is not known, or it has a field of an unsupported type. + """ iface = comdefs.IFACE_TYPES.get(name) if not iface: raise Win32EmuError(f"Invalid COM interface: {name}") @@ -46,12 +51,11 @@ def get_interface(self, emu, ptr_size, name): field_offset = 0 for field in fields: field_name, field_obj = field - # Determine if the field is an inherited interface (e.g., IUnknown) + # A Structure field is an inherited interface, such as IUnknown. if issubclass(field_obj, ctypes.Structure): if field_name not in comdefs.IFACE_TYPES: raise Win32EmuError(f"COM interface {name} inherits unsupported interface {field_name}") - # Iterate inherited interface fields for subfield in field_obj._fields_: subfield_name, subfield_type = subfield if issubclass(subfield_type, (ctypes.c_uint32, ctypes.c_ulong, ctypes.c_ulonglong)): diff --git a/speakeasy/windows/common.py b/speakeasy/windows/common.py index 7ac143857..743123b84 100644 --- a/speakeasy/windows/common.py +++ b/speakeasy/windows/common.py @@ -13,8 +13,7 @@ from speakeasy.struct import Enum from speakeasy.winenv.defs.nt import ddk -# GDT Constants needed to set our emulator into protected mode -# Access bits +# Access bits and flags of the GDT descriptors that the emulator builds for protected mode. GDT_ACCESS_BITS = Enum() GDT_ACCESS_BITS.ProtMode32 = 0x4 GDT_ACCESS_BITS.PresentBit = 0x80 @@ -111,9 +110,9 @@ def get_datetime_from_filetime(filetime: int) -> datetime.datetime: return FILETIME_EPOCH + datetime.timedelta(microseconds=filetime // 10) -# DLL and API set name prefixes, and the modules that have handlers for their -# functions. The first prefix that matches is used. When the functions of a set -# are in more than one module, the main module is first. +# DLL and API set name prefixes, and the modules that have handlers for their functions. +# ``get_dll_hosts`` uses the first prefix that matches. When the functions of a set are in +# more than one module, the main module is first. DLL_HOSTS: tuple[tuple[tuple[str, ...], tuple[str, ...]], ...] = ( ( ("api-ms-win-crt", "api-ms-win-core-crt", "vcruntime", "ucrtbased", "ucrtbase", "msvcr", "msvcp", "crtdll"), @@ -149,7 +148,7 @@ def get_dll_hosts(name: str) -> tuple[str, ...]: """ Get the modules that have handlers for the functions of a DLL or API set. - A DLL that is not in DLL_HOSTS has its own handlers. + A DLL that is not in ``DLL_HOSTS`` has its own handlers. """ lower = name.lower() for prefixes, hosts in DLL_HOSTS: @@ -164,7 +163,7 @@ def normalize_dll_name(name: str) -> str: class _PeParser(pefile.PE): - """Represents PE files loaded into the emulator.""" + """A pefile PE with the mapped image, the imports and the exports that the loaders read.""" def __init__(self, path=None, data=None, emu_path="", fast_load=False): super().__init__(name=path, data=data, fast_load=fast_load) @@ -238,7 +237,7 @@ def _get_pe_exports(self): return exports for exp in pe.DIRECTORY_ENTRY_EXPORT.symbols: - entry = namedtuple("export", ["name", "address", "forwarder", "ordinal"]) # type: ignore[name-match] # legacy: namedtuple name differs from var + entry = namedtuple("export", ["name", "address", "forwarder", "ordinal"]) # type: ignore[name-match] entry.name = exp.name entry.address = exp.address + pe.base entry.forwarder = exp.forwarder @@ -281,7 +280,7 @@ def is_driver(self): return True def is_dotnet(self): - """Is the current PE file a .NET assembly?""" + """Test whether the PE imports a .NET entry point or has a CLR header (data directory 14).""" for dll, func in self.imports.values(): if dll == "mscoree" and func in ["_CorExeMain", "_CorDllMain"]: return True @@ -292,8 +291,9 @@ def get_mapped_image(self) -> bytes: """ Return the image with each section at its virtual address. - pefile leaves out a section whose raw data goes past the end of the file. Windows maps the - bytes that the file has and fills the rest with zeros, so this keeps them. + pefile ``get_memory_mapped_image`` leaves out a section whose raw data goes past the end of the + file. Windows maps the bytes that the file has and fills the rest with zeros, so this method + keeps the bytes that the file has. """ data = self.__data__ alignment = self.OPTIONAL_HEADER.SectionAlignment @@ -319,7 +319,6 @@ def rebase(self, to): self.base = to self.ep = self.OPTIONAL_HEADER.AddressOfEntryPoint - # After relocation, generate a new memory mapped image self.mapped_image = self.get_mapped_image() self.imports = self._get_pe_imports() diff --git a/speakeasy/windows/cpu_exceptions.py b/speakeasy/windows/cpu_exceptions.py index 5ba3736a2..b85704134 100644 --- a/speakeasy/windows/cpu_exceptions.py +++ b/speakeasy/windows/cpu_exceptions.py @@ -1,12 +1,12 @@ """ CPU exceptions as Windows reports them to user mode. -Unicorn reports interrupts, invalid instructions and IN/OUT with its own -conventions for where the program counter points. This module turns each of -those into the exception code, ExceptionAddress, parameters and context -instruction pointer that a Windows exception handler would see, following -Wine's implementation, which reproduces the quirks that anti-debugging code -depends on (dlls/ntdll/unix/signal_x86_64.c). +Unicorn reports interrupts, invalid instructions and IN/OUT with its own rules +for where the program counter points. This module converts each of these events +to the exception code, ExceptionAddress, parameters and context instruction +pointer that a Windows exception handler sees. The conversion follows Wine +(dlls/ntdll/unix/signal_x86_64.c), which copies the quirks that anti-debugging +code uses. """ from dataclasses import dataclass @@ -17,19 +17,19 @@ STATUS_GUARD_PAGE_VIOLATION = 0x80000001 STATUS_ASSERTION_FAILURE = 0xC0000420 -# Interrupt gates user mode may use; any other `int n` is a general protection fault +# Interrupt gates that user mode can use. Any other `int n` is a general protection fault. INT_BREAKPOINT = 0x03 INT_ASSERTION = 0x2C INT_DEBUG_SERVICE = 0x2D -# Gates that are system calls rather than exceptions, which emulation does not implement: -# __fastfail and KiSystemService, and on x86 KiGetTickCount and KiCallbackReturn +# Gates for system calls, which the emulator does not send to exception handlers: 0x29 is +# __fastfail and 0x2e is KiSystemService, and on x86 0x2a is KiGetTickCount and 0x2b is KiCallbackReturn SYSTEM_GATES = {_arch.ARCH_AMD64: (0x29, 0x2E), _arch.ARCH_X86: (0x29, 0x2A, 0x2B, 0x2E)} # The first parameter of an access violation ACCESS_CODES = {"read": 0, "write": 1, "fetch": 8} -# With these debug service codes in rax, `int 2d` does nothing when no debugger is attached +# With these debug service codes in rax, `int 2d` raises no exception when no debugger is attached DEBUG_SERVICE_NO_EXCEPTION = (1, 3, 4, 5) @@ -45,12 +45,12 @@ def from_interrupt(intno: int, pc: int, code_bytes: bytes, rax: int, trap_flag: """ Get the exception for an interrupt that Unicorn reports. - pc is the program counter, which is past an ``int n`` instruction or at a faulting - instruction. code_bytes are the two bytes before it. trap_flag tells a single step + pc is the program counter, which is after an ``int n`` instruction or at a faulting + instruction. code_bytes are the two bytes before pc. trap_flag tells a single step from ``int 1``. - Returns None when Windows raises no exception: then execution continues at the - returned exception-less pc, see skip_debug_service. + Return None when Windows raises no exception. Then execution continues at + ``skip_debug_service(pc)``. """ if intno == 0: # divide error: pc is at the faulting instruction return CpuException(ddk.STATUS_INTEGER_DIVIDE_BY_ZERO, pc, pc) @@ -60,7 +60,7 @@ def from_interrupt(intno: int, pc: int, code_bytes: bytes, rax: int, trap_flag: # trap flag: pc is after the stepped instruction return CpuException(ddk.STATUS_SINGLE_STEP, pc, pc) if intno == INT_BREAKPOINT: - # The kernel backs up one byte: onto `int3`, or into the middle of `int 3` + # The kernel moves the address back one byte: to `int3`, or into the middle of `int 3` return CpuException(ddk.STATUS_BREAKPOINT, pc - 1, pc - 1, (0,)) if intno == INT_DEBUG_SERVICE: if rax in DEBUG_SERVICE_NO_EXCEPTION: @@ -76,7 +76,7 @@ def from_interrupt(intno: int, pc: int, code_bytes: bytes, rax: int, trap_flag: def skip_debug_service(pc: int) -> int: - """Where execution continues after an `int 2d` that raises no exception.""" + """Get the address where execution continues after an ``int 2d`` that raises no exception.""" return pc + 1 @@ -85,7 +85,7 @@ def general_protection(pc: int) -> CpuException: def from_invalid_instruction(pc: int, code_bytes: bytes) -> CpuException: - """The exception for an instruction at pc that Unicorn cannot execute.""" + """Get the exception for an instruction at pc that Unicorn cannot execute.""" if code_bytes[:1] == b"\xf1": # icebp: a debug trap after the instruction return CpuException(ddk.STATUS_SINGLE_STEP, pc + 1, pc + 1) return CpuException(ddk.STATUS_ILLEGAL_INSTRUCTION, pc, pc) @@ -98,5 +98,5 @@ def privileged_instruction(pc: int) -> CpuException: def memory_fault( pc: int, address: int, access_type: str | None, code: int = ddk.STATUS_ACCESS_VIOLATION ) -> CpuException: - """An access violation, or a guard page violation, at address by the instruction at pc.""" + """Get the access violation, or the guard page violation, of an access to address by the instruction at pc.""" return CpuException(code, pc, pc, (ACCESS_CODES.get(access_type or "", 0), address)) diff --git a/speakeasy/windows/cryptman.py b/speakeasy/windows/cryptman.py index 7e387ab58..7069da19f 100644 --- a/speakeasy/windows/cryptman.py +++ b/speakeasy/windows/cryptman.py @@ -27,7 +27,7 @@ def __init__(self, hasher): class CryptContext: - """Represents crypto context used by crypto functions.""" + """Represents a cryptographic provider context and its keys.""" def __init__(self, handles, cname, pname, ptype, flags): self.handles: HandleTable = handles @@ -55,7 +55,7 @@ def delete_key(self, hnd) -> bool: class CryptoManager: """ - Manages the emulation of crypto functions. + Keep the provider contexts, keys and hashes of the CryptoAPI handlers. Providers, keys and hashes are not kernel objects, so their handles are in a table of the manager and not in the process handle table, and CloseHandle does not close them. diff --git a/speakeasy/windows/driveman.py b/speakeasy/windows/driveman.py index 5e914a41b..70e1fc171 100644 --- a/speakeasy/windows/driveman.py +++ b/speakeasy/windows/driveman.py @@ -11,7 +11,7 @@ class DriveManager: - """Manages the emulation of Windows drives. Currently assumes one volume per drive.""" + """Keep the drives of the config. Each drive has one volume.""" def __init__(self, config: list[DriveConfig] | None = None): super().__init__() diff --git a/speakeasy/windows/export_manifest.py b/speakeasy/windows/export_manifest.py index 6cf795199..2bd442ce9 100644 --- a/speakeasy/windows/export_manifest.py +++ b/speakeasy/windows/export_manifest.py @@ -30,7 +30,7 @@ class ManifestExport(BaseModel): ) forwarder: str | None = Field( default=None, - description="Forwarder string in `DLL.name` or `DLL.#ordinal` form, when the export forwards elsewhere.", + description="Forwarder string in `DLL.name` or `DLL.#ordinal` form, if the export is a forwarder.", ) kind: Literal["function", "data"] = Field( default="function", @@ -61,7 +61,7 @@ class ExportManifest(BaseModel): size_of_image: int = Field(gt=0, description="SizeOfImage from the PE optional header.") sha256: str = Field(pattern=r"^[0-9a-f]{64}$", description="SHA-256 of the source binary, in lower-case hex.") exports: tuple[ManifestExport, ...] = Field( - description="Exports sorted by ordinal, then by name. Names are unique. RVA 0 entries are left out." + description="Exports sorted by ordinal, then by name. Names are unique. Exports with RVA 0 are not in the list." ) @model_validator(mode="after") diff --git a/speakeasy/windows/fileman.py b/speakeasy/windows/fileman.py index e5e6e1445..68f979bd0 100644 --- a/speakeasy/windows/fileman.py +++ b/speakeasy/windows/fileman.py @@ -67,7 +67,7 @@ def _get_speakeasy_root(): class MapView: - """Represents a shared memory view.""" + """Represents a view of a file mapping in guest memory.""" def __init__(self, base, offset, size, protect, process=None): self.base: int = base @@ -192,7 +192,7 @@ def remove_data(self): self.pos = 0 def handle_file_data(self): - """Get the data to return from a read request, based on the emulation config.""" + """Get the initial data of the file from its config entry: a host file, a byte fill, or no data.""" if not self.config: return io.BytesIO(b"") @@ -231,7 +231,7 @@ def write_data(self, data): class Pipe(File): - """Emulated named pipe objects.""" + """Represents a named pipe.""" def __init__(self, name, mode, num_instances, out_size, in_size, config=None): super().__init__(path=name, config=config) @@ -243,17 +243,14 @@ def __init__(self, name, mode, num_instances, out_size, in_size, config=None): class FileManager: - """Manages file system activity during emulation.""" + """Keep the emulated files, mappings and directories, and record the file events.""" def __init__(self, config, emu): super().__init__() - # A view keeps its mapping after the last handle to the mapping closes + # The file manager never removes a mapping, because a view keeps its mapping after the last handle closes. self.mappings: list[FileMap] = [] - # top level config self.config: SpeakeasyConfig = config - - # "files" key of config self.file_config: FilesystemConfig = self.config.filesystem self.emu: Any = emu @@ -261,7 +258,7 @@ def __init__(self, config, emu): self.emulated_binname: str = shlex.split(cmdline)[0] if cmdline else "" - # First file in this list seems to always be the module itself + # When the input is a PE, load_module adds it as the first file. self.files: list[File] = [] self.deleted_paths: set[str] = set() self.created_dirs: set[str] = set() @@ -403,7 +400,7 @@ def is_image_alias(self, path): return proc is not None and (proc.path or "").lower() == self.files[0].path.lower() def add_existing_file(self, path, data): - """Register an existing file that the emulation space already includes, with its data.""" + """Add a file with its data, for example the input module.""" f = File(path, data=data) self.files.append(f) return f @@ -520,8 +517,7 @@ def get_emu_file(self, path): if get_path_key(path) in self.deleted_paths: return None - # Does this file exist in our emulation environment - # See if we have a handler for this exact file + # A full_path entry matches first, then a configured module, then a by_ext entry, then the default entry. for f in self.file_config.files: if f.mode == "full_path": if fnmatch.fnmatch(path.lower(), f.emu_path.lower()): @@ -532,7 +528,6 @@ def get_emu_file(self, path): if any(m.path == path for m in modules.user_modules) or any(m.path == path for m in modules.system_modules): return FileEntryFullPath(mode="full_path", emu_path=path) - # If no full path handler exists, do we have an extension handler? for f in self.file_config.files: path_ext = ntpath.splitext(path)[-1:][0].strip(".") if path_ext: @@ -540,7 +535,6 @@ def get_emu_file(self, path): if path_ext.lower() == f.ext: return f - # Finally, do we have a catch-all default handler? for f in self.file_config.files: if f.mode == "default": return f @@ -666,8 +660,7 @@ def file_open(self, path, create=False, truncate=False, is_dir=False, dispositio else: f = self.get_file_from_path(path) if f: - # Deep-copy this file so we can have separate file - # offset pointers + # The new file object shares the data, but has its own file pointer. f = f.duplicate() else: f = self.get_file_by_path(path) diff --git a/speakeasy/windows/guest_callbacks.py b/speakeasy/windows/guest_callbacks.py index 61e2ed892..9deb5250b 100644 --- a/speakeasy/windows/guest_callbacks.py +++ b/speakeasy/windows/guest_callbacks.py @@ -1,30 +1,25 @@ """ -Guest callbacks: Python code that calls guest functions and acts on their results. - -"Guest callback" means the emulator calling back into guest code, as opposed to -the guest calling the emulator through an API. API handlers already do this -with a fixed list of calls decided up front (GuestCallbackFrame.pending), e.g. to -run a DllMain. This module adds code that decides what to call next from what -the previous call returned, like the x64 exception dispatcher, which runs -filters and handlers. That code runs for an API handler or, for a CPU -exception, with no API involved at all; the name says what it does rather than -who starts it. - -Such code is a generator handler: it yields a GuestCall for each guest function -it calls, is sent that call's return value, and finally returns either a return -value for the API that started it or a Resume to continue the guest at another -context: +Guest callbacks: Python code that calls guest functions and uses their results. + +A guest callback is a call from the emulator into guest code. An API handler can +give a fixed list of calls (``GuestCallbackFrame.pending``), for example to run a +DllMain. A generator handler selects the next call from the result of the +previous call, as the x64 exception dispatcher does when it runs filters and +handlers. A generator handler runs for an API handler, or for a CPU exception +without an API call. + +A generator handler yields a ``GuestCall`` for each guest function that it calls, +and gets the return value of that call. At the end, it returns the return value +of the API that started it, or a ``Resume`` that continues the guest at a +different context: def handler(emu, callback): result = yield GuestCall(callback, (1, 2)) return result + 1 -Emulation is never nested: each guest call runs in the normal run loop on the -stack of the handler's frame and returns to GUEST_CALLBACK_RETURN_ADDR, where the -handler continues. The emulator only calls into this module where an API -handler returns a generator (start_api_handler) or a Resume, such as NtContinue -(transfer), where a guest call returns (resume), and to unwind through handler -frames (unwind_frame). +Emulation is never nested. Each guest call runs in the normal run loop, on the +stack of the handler frame, and returns to ``GUEST_CALLBACK_RETURN_ADDR``. Then +the handler continues. """ from collections.abc import Generator @@ -49,7 +44,7 @@ class GuestCall: @dataclass(frozen=True) class Resume: - """Continue the guest at a CONTEXT, abandoning the stack below it.""" + """Continue the guest at a CONTEXT, and drop the guest callback frames at or below its stack pointer.""" context: Any @@ -58,7 +53,7 @@ class Resume: class CallbackError(Exception): - """A generator handler failure: the run ends with an error of type kind at address.""" + """A generator handler failed. The run ends with an error of type ``kind`` at ``address``.""" def __init__( self, kind: ErrorType, address: int, access_type: str | None = None, pc: int | None = None, **details: Any @@ -95,9 +90,9 @@ def get_handler_fault(emu, error: Exception, pc: int) -> tuple[ErrorType, int, s """ Get the error type, address, access type and message for an exception raised by an API handler. - A handler that reads or writes an invalid guest pointer fails like guest code does at the - first byte it cannot access. Unicorn reports the start of the whole access. - Any other exception is an internal_error, with the exception text as the message. + If a handler reads or writes an invalid guest pointer, the error address is the first byte + that the handler cannot access, as for guest code. Unicorn gives the start of the whole access. + Any other exception is an ``internal_error``, with the exception text as the message. """ access = HANDLER_FAULT_ACCESS.get(error.errno) if isinstance(error, uc.UcError) else None if access is None or not error.args: @@ -111,21 +106,26 @@ def get_handler_fault(emu, error: Exception, pc: int) -> tuple[ErrorType, int, s @dataclass class HandlerState: generator: GeneratorHandler - # Guest stack reserved below the frame for data the guest calls see + # Size of the guest stack below the frame that holds data for the guest calls scratch: int = 0 - # CONTEXT64 of the guest state the handler interrupted, if it has no caller to return to + # Address of the CONTEXT of the guest state that the handler interrupted, if the handler has no caller context: int = 0 def start(emu, frame: GuestCallbackFrame, generator: GeneratorHandler) -> None: - """Run a generator handler on frame up to its first guest call. A CallbackError of that first step propagates.""" + """ + Run a generator handler on ``frame`` up to its first guest call. + + Raises: + CallbackError: if the handler fails before its first guest call. + """ frame.handler = HandlerState(generator) emu.get_current_run().guest_callbacks.append(frame) _step(emu, frame, None) def start_api_handler(emu, frame: GuestCallbackFrame, generator: GeneratorHandler) -> int | None: - """Run the generator an API handler returned; return the API result if it already finished.""" + """Run the generator that an API handler returned. Return the API result if the generator finished.""" try: start(emu, frame, generator) except CallbackError as e: @@ -142,7 +142,7 @@ def resume(emu, frame: GuestCallbackFrame) -> None: def fail(emu, e: CallbackError) -> None: - """End the run with the error e describes.""" + """End the run with the error of ``e``.""" run = emu.get_current_run() if e.pc is not None: emu.set_pc(e.pc) @@ -187,10 +187,10 @@ def transfer(emu, context) -> None: def drop_abandoned(emu, sp: int) -> None: """ - Drop the frames at or below sp, where the guest runs. + Drop the guest callback frames at or below ``sp``, where the guest continues. - The guest left their guest calls without returning, as a handler that continues in an __except block leaves the - dispatch of its exception. + The guest left the guest calls of these frames without a return, for example when a handler continues in an + ``__except`` block and leaves the dispatch of its exception. """ callbacks = emu.get_current_run().guest_callbacks while callbacks and callbacks[-1].stack_pointer <= sp: @@ -198,7 +198,7 @@ def drop_abandoned(emu, sp: int) -> None: def current(emu) -> HandlerState: - """The state of the running generator handler.""" + """Get the state of the running generator handler.""" state: HandlerState = emu.get_current_run().guest_callbacks[-1].handler return state @@ -214,7 +214,7 @@ def alloca(emu, data: bytes) -> int: def caller_context(emu): """ - Get the context of the caller of the API being handled. + Get the context of the caller of the current API call. The context is valid in a generator handler before its first guest call, while the registers are still the caller's. """ @@ -227,9 +227,10 @@ def caller_context(emu): def unwind_frame(emu, ctx) -> None: - """Unwind ctx, returning to GUEST_CALLBACK_RETURN_ADDR, to the guest state its handler frame interrupted.""" + """Unwind ``ctx`` at ``GUEST_CALLBACK_RETURN_ADDR`` to the guest state that its frame interrupted.""" frames = [f for f in emu.get_current_run().guest_callbacks if f.stack_pointer >= ctx.Rsp] - if not frames: # the guest's own copy of the address, on a corrupt stack: the walk ends + # No frame has this return address, for example a guest copy on a corrupt stack. End the walk. + if not frames: ctx.Rip = 0 return frame = min(frames, key=lambda f: f.stack_pointer) diff --git a/speakeasy/windows/hammer.py b/speakeasy/windows/hammer.py index bec5948ff..859ad414d 100644 --- a/speakeasy/windows/hammer.py +++ b/speakeasy/windows/hammer.py @@ -15,7 +15,7 @@ logger = logging.getLogger(__name__) -# default list of APIs to always allow despite triggering API hammering detection +# The mitigation does not patch calls to these APIs, unless the config gives its own allow list. _default_api_hammer_allowlist = [ "kernel32.WriteProcessMemory", "kernel32.WriteFile", @@ -29,9 +29,9 @@ def _lowercase_set(tt): class ApiHammer: """ - Detect and attempt to mitigate API hammering. + Find API hammering and patch the call sites. - Malware samples use API hammering as part of anti-sandbox or anti-emulation. + Malware calls an API many times in a loop to use up the time of a sandbox or an emulator. """ def __init__(self, emu): @@ -59,21 +59,21 @@ def is_allowed_api(self, apiname): return apiname.lower() in self.allow_list def handle_import_func(self, imp_api, conv, argc): - """Identifies possible API hammering and attempts to patch in mitigations.""" + """ + Count the calls to an API from one return address. + + When the count gets to the threshold, patch the call site on x86, so that the call returns 0 + without a call to the API. + """ if not self.enabled: - # api hammering mitigation not enabled, so exit return if self.is_allowed_api(imp_api): - # this is an api that we always want to allow, don't bother trying to - # prevent api hammering return hammer_key = imp_api + f"{self.emu.get_ret_address():x}" self.api_stats[hammer_key] += 1 if self.api_stats[hammer_key] < self.api_threshold: return - # TODO: better parameterize the checking & dispatch of the types of calls/jmps to imports - # so we can more easily loop through them & clean up the the logic below - # TODO: track patches in the hammer_memregion & reuse when possible + # TODO: reuse the stubs in hammer_memregion. if self.emu.get_arch() == e_arch.ARCH_X86: try: eip = self.emu.get_ret_address() - 6 @@ -115,5 +115,5 @@ def handle_import_func(self, imp_api, conv, argc): return if self.emu.get_arch() == e_arch.ARCH_AMD64: - # TODO + # TODO: patch the call sites on x64. pass diff --git a/speakeasy/windows/handles.py b/speakeasy/windows/handles.py index 354d4bd9e..7a9f09079 100644 --- a/speakeasy/windows/handles.py +++ b/speakeasy/windows/handles.py @@ -1,4 +1,4 @@ -"""The handle table of a process: the objects that its handle values refer to.""" +"""Handle tables: the objects that handle values refer to.""" from __future__ import annotations @@ -20,7 +20,10 @@ class HandleTable: """ - Map the handle values of a process to their objects. + Map handle values to their objects. + + Each process has a table for its kernel object handles. Objects that are not kernel objects, such as + sockets and windows, have their handles in other tables. Each handle holds a reference: ``new_handle`` adds 1 to ``ref_cnt`` of the object, and ``close`` calls ``release`` with the object. ``pseudo`` gives the object of a pseudo handle, such as the diff --git a/speakeasy/windows/hardware_breakpoints.py b/speakeasy/windows/hardware_breakpoints.py index 1e9eca43f..001e7d6ee 100644 --- a/speakeasy/windows/hardware_breakpoints.py +++ b/speakeasy/windows/hardware_breakpoints.py @@ -1,13 +1,14 @@ """ Hardware execute breakpoints, from the debug registers of the current thread. -Unicorn has no debug registers. While the current thread's Dr7 enables an -execute breakpoint, a code hook at its address raises STATUS_SINGLE_STEP before -the instruction runs, as the CPU's debug exception does, and sets the -breakpoint's bit in Dr6. A handler that continues at the breakpoint with the -resume flag set runs the instruction once without breaking again. +Unicorn has no debug registers. When Dr7 of the current thread enables an +execute breakpoint, the emulator adds a code hook at the breakpoint address. +The hook raises STATUS_SINGLE_STEP before the instruction runs, as the debug +exception of the CPU does, and sets the bit of the breakpoint in Dr6. If a +handler continues at the breakpoint with the resume flag set, the instruction +runs one time and does not break again. -Data breakpoints (a Dr7 R/W field other than execute) are not implemented. +Data breakpoints (a Dr7 R/W field other than execute) have no effect. """ from __future__ import annotations @@ -27,7 +28,7 @@ def execute_breakpoints(debug_registers: dict[str, int]) -> dict[int, int]: - """Address to index of each execute breakpoint that Dr7 enables.""" + """Get the index of each execute breakpoint that Dr7 enables, by address.""" dr7 = debug_registers["Dr7"] breakpoints: dict[int, int] = {} for i in range(4): @@ -54,11 +55,11 @@ def update(self) -> None: self.emu.remove_hook(self.hooks.pop(address)) for address in wanted.keys() - self.hooks.keys(): self.hooks[address] = self.emu.add_code_hook(self._hit, begin=address, end=address) - # Code that already ran was translated without the hook + # Unicorn translated the code that already ran without the hook self.emu.emu_eng.invalidate_translations(address, address + 1) def resume_at(self, address: int) -> None: - """The current thread continues at address with the resume flag: run it without breaking.""" + """Let the current thread run the instruction at address one time without a break, as the resume flag does.""" self.resume = None if address in self.hooks: self.resume = (self.emu.get_current_thread(), address) diff --git a/speakeasy/windows/ioman.py b/speakeasy/windows/ioman.py index 6b695a941..6fb2b1c95 100644 --- a/speakeasy/windows/ioman.py +++ b/speakeasy/windows/ioman.py @@ -6,12 +6,7 @@ class IoManager: - """ - Direct IO requests to a module handler. - - For example, if a user mode application sends an ioctl to a device, this class can - handle it. - """ + """Send device I/O control requests to the emulated kernel module of the device driver.""" def __init__(self): super().__init__() @@ -21,11 +16,9 @@ def dev_ioctl(self, arch, dev, ioctl, inbuf): rv = ddk.STATUS_INVALID_DEVICE_REQUEST outbuf = b"" - # Get parent driver for you drv = dev.driver bn = drv.get_basename() - # Find the emulated kernel module (if any) mods = [m for m in self.emu_kmods if bn == m.get_mod_name()] if not mods: rv = ddk.STATUS_INVALID_DEVICE_REQUEST diff --git a/speakeasy/windows/kernel.py b/speakeasy/windows/kernel.py index 3b87353dc..68f3d855f 100644 --- a/speakeasy/windows/kernel.py +++ b/speakeasy/windows/kernel.py @@ -19,7 +19,7 @@ class WinKernelEmulator(WindowsEmulator): - """Class used to emulate Windows drivers.""" + """Emulate a Windows driver in kernel mode.""" ssdt_ptr: int @@ -100,7 +100,7 @@ def load_module(self, path=None, data=None, filename=None) -> RuntimeModule: return rtmod def pool_alloc(self, pooltype, size, tag="None"): - """Allocate memory in the emulated "pool".""" + """Map a pool allocation in the System process, and tag it with the pool type and ``tag``.""" if pooltype == ddk.POOL_TYPE.NonPagedPool: pt = "NonPagedPool" elif pooltype == ddk.POOL_TYPE.PagedPool: @@ -150,21 +150,20 @@ def init_processes(self, processes): p.path = proc.path p.image = ntpath.basename(p.path) - # Create an initial thread for each process t = objman.Thread(self) self.add_object(t) t.process = p p.threads.append(t) self.processes.append(p) - # The SYSTEM process should be the starting context + # The System process is the current process at start. sp = [p for p in self.processes if p.name.lower() == "system"] if sp: sp = sp[0] self.set_current_process(sp) def alloc_peb(self, proc): - """Allocate PEB and related substructures for a given process.""" + """Map the loader data of the PEB of a process if it is not mapped, and return the PEB.""" ldr = proc.peb_ldr_data if not ldr.address: size = ldr.sizeof() @@ -174,17 +173,17 @@ def alloc_peb(self, proc): return proc.peb def get_process_peb(self, process): - """Retrieve the PEB for the supplied process.""" + """Get the PEB of a process, with the loaded modules in its loader lists.""" self.alloc_peb(process) process.init_peb(self.get_peb_modules()) return process.peb def set_current_process(self, process): - """Set the current process context.""" + """Set the current process.""" self.curr_process = process def get_current_process(self): - """Get the current process context.""" + """Get the current process.""" self.get_object_manager("kernel current-process resolution") return self.curr_process @@ -202,19 +201,22 @@ def _get_process_state(self) -> WindowsEmulator: return self def run_module(self, module, *, all_entrypoints=False, entry_point=None) -> None: - """Begin emulation fo a previously loaded kernel module.""" + """ + Add a DriverEntry run for a loaded driver, and run the queue. + + If ``all_entrypoints`` is true, the emulator also runs the exports, the IRP handlers and the unload + routine of the driver. + """ self.all_entrypoints = all_entrypoints # Create the service key for the driver drv = self.create_driver_object(pe=module) svc_key = self.regman.create_key(drv.reg_path) - # Create the values for the service key svc_key.create_value("ImagePath", REG_VALUE_TYPE.REG_EXPAND_SZ, module.emu_path) svc_key.create_value("Type", REG_VALUE_TYPE.REG_DWORD, 0x1) # SERVICE_KERNEL_DRIVER svc_key.create_value("Start", REG_VALUE_TYPE.REG_DWORD, 0x3) # SERVICE_DEMAND_START svc_key.create_value("ErrorControl", REG_VALUE_TYPE.REG_DWORD, 0x1) # SERVICE_ERROR_NORMAL - # Create the parameters subkey self.regman.create_key(drv.reg_path + "\\Parameters") if entry_point is not None: @@ -231,8 +233,8 @@ def run_module(self, module, *, all_entrypoints=False, entry_point=None) -> None self.add_run(run) if self.all_entrypoints: - # Only emulate a subset of all the exported functions - # There are some modules (such as the windows kernel) with thousands of exports + # Limit the number of export runs, because some modules, such as the Windows kernel, have + # thousands of exports. exports = [k for k in module.get_exports()[:MAX_EXPORTS_TO_EMULATE]] if exports: @@ -242,16 +244,16 @@ def run_module(self, module, *, all_entrypoints=False, entry_point=None) -> None run.type = EntryPointType.EXPORT run.name = exp.name or None run.start_addr = exp.address - # Here we set dummy args to pass into the export function + # Each export gets the same placeholder arguments. run.args = args - # Store these runs and only queue them before the unload routine - # this is because some exports may not be ready to be called yet + # The export runs wait in delayed_runs until the run queue is empty after DriverEntry, + # because an export can depend on the state that DriverEntry sets up. self.delayed_runs.append(run) self.start() def create_device_object(self, name="", drv=0, ext_size=0, devtype=0, chars=0, tag=""): - """Create a device object to use for kernel emulation.""" + """Create a device object and add it to the device list of ``drv``.""" dev = objman.Device(self) alloc_size = ext_size + dev.sizeof() @@ -291,7 +293,6 @@ def create_device_object(self, name="", drv=0, ext_size=0, devtype=0, chars=0, t if next_dev.object.NextDevice: next_dev = self.get_object_from_addr(drv.object.NextDevice) else: - # This is the last in the list, add our new device next_dev.object.NextDevice = dev.address next_dev.write_back() break @@ -312,7 +313,7 @@ def add_symlink(self, symlink, devname): self.get_object_manager("symlink creation").add_symlink(symlink, devname) def _call_driver_dispatch(self, func, dev_addr, irp_addr): - """Call a WDM driver dispatch function with a supplied IRP.""" + """Set the arguments and the pc for a call to a dispatch function with an IRP.""" stk_ptr = self.get_stack_ptr() self.set_func_args(stk_ptr, self.return_hook, dev_addr, irp_addr, conv=_arch.CALL_CONV_STDCALL) self.set_pc(func) @@ -322,7 +323,6 @@ def new_irp(self): return objman.Irp(emu=self) def irp_mj_create(self, func, dev): - # Generate an IRP for the create request irp = self.new_irp() self._call_driver_dispatch(func, dev.address, irp.address) @@ -365,7 +365,7 @@ def irp_mj_cleanup(self, func, dev): return irp def driver_unload(self, drv): - """Call the unload routine for a driver.""" + """Add a run for the unload routine of a driver. If there is none, or it ran, end the emulation.""" if not drv.on_unload or drv.unload_called: self.on_emu_complete() return @@ -390,19 +390,18 @@ def next_driver_func(self, drv): if self.curr_run is not None and self.curr_run.type is not None: self.curr_run.return_value = self.get_return_val() - # Check if theres anything in the run queue if len(self.run_queue): return if not self.all_entrypoints: return - # TODO right now just use the first device object that was created + # TODO: send the IRPs to each device, not only the first device. dev = None if len(drv.devices): dev = drv.devices[0] - # Run any remaining IRP handlers + # Find the first IRP major function that has no run yet. for hdlr, major, irp_name in ( (self.irp_mj_create, ddk.IRP_MJ_CREATE, "IRP_MJ_CREATE"), (self.irp_mj_dev_io, ddk.IRP_MJ_DEVICE_CONTROL, "IRP_MJ_DEVICE_CONTROL"), @@ -411,7 +410,6 @@ def next_driver_func(self, drv): (self.irp_mj_close, ddk.IRP_MJ_CLOSE, "IRP_MJ_CLOSE"), (self.irp_mj_cleanup, ddk.IRP_MJ_CLEANUP, "IRP_MJ_CLEANUP"), ): - # Did we run this mj func yet? if not any(r.type == EntryPointType.IRP and r.name == irp_name for r in self.runs): func_handler = hdlr func_addr = int(drv.mj_funcs[major]) @@ -425,7 +423,7 @@ def next_driver_func(self, drv): self.delayed_runs = [] if not func_addr or not dev or func_handler is None: - # We are done here, call the unload routine + # No IRP handler is left, so run the unload routine. self.driver_unload(drv) return @@ -448,7 +446,6 @@ def on_run_complete(self): if drv.pe == self.curr_mod: self.next_driver_func(drv) - # Dispatch the next run return self._prepare_next_run() def on_emu_complete(self): @@ -466,7 +463,7 @@ def on_emu_complete(self): self.stop() def set_hooks(self): - """Set the emulator callbacks.""" + """Add the hooks of the kernel emulator, then the tracing, coverage and debug hooks.""" super().set_hooks() if not self.has_builtin_hooks: @@ -481,12 +478,16 @@ def set_hooks(self): def get_kernel_base(self): """Get the base address of the kernel image (ntoskrnl.exe).""" - # Get kernel base address kern = self.get_kernel_mod() return kern.base def get_kernel_mod(self): - """Get the kernel image module.""" + """ + Get the module of the kernel image. + + Raises: + KernelEmuError: if ``ntoskrnl`` is not loaded. + """ mod = self.get_mod_by_name("ntoskrnl") if mod: return mod @@ -499,10 +500,10 @@ def setup_kernel_mode(self): idt = objman.IDT(self) idt.init_descriptors() - # selector, base, limit, flags + # Unicorn takes the IDTR as (selector, base, limit, flags). self.reg_write(_arch.X86_REG_IDTR, (0, idt.object.Descriptors, idt.object.Limit, 0)) - # Setup the SSDT + # Set up the SSDT. ssdt = self.ktypes.SSDT(self.get_ptr_size()) size = self.get_ptr_size() * 256 @@ -517,7 +518,7 @@ def setup_kernel_mode(self): self.get_object_manager("symlink creation").add_symlink(sl.name, sl.target) def setup_msrs(self): - """Setup machine specific registers for kernel emulation.""" + """On x64, map a ``KiSystemCall64`` stub next to the kernel and point the LSTAR MSR to it.""" km = self.get_kernel_mod() if self.get_arch() == _arch.ARCH_AMD64 and km.image_size > 0: diff --git a/speakeasy/windows/kernel_mods/volmgr.py b/speakeasy/windows/kernel_mods/volmgr.py index a36eb323d..c7a10e9a3 100644 --- a/speakeasy/windows/kernel_mods/volmgr.py +++ b/speakeasy/windows/kernel_mods/volmgr.py @@ -21,7 +21,7 @@ class VOLUME_DISK_EXTENTS(EmuStruct): class DriverModule(km.KernelModule): - """Class for emulation of specific drivers (e.g. their ioctl handlers).""" + """The volmgr driver. Each IOCTL gets one disk extent, as for IOCTL_VOLUME_GET_VOLUME_DISK_EXTENTS.""" def __init__(self): super().__init__() diff --git a/speakeasy/windows/loaders.py b/speakeasy/windows/loaders.py index 7aa2229ca..89941fd8a 100644 --- a/speakeasy/windows/loaders.py +++ b/speakeasy/windows/loaders.py @@ -20,7 +20,12 @@ def validate_forwarder(value: str) -> None: - """Validate DLL.name / DLL.#ordinal syntax without resolving its target.""" + """ + Check that ``value`` is a forwarder string in ``DLL.name`` or ``DLL.#ordinal`` form. Do not resolve the target. + + Raises: + ValueError: if ``value`` is not a valid forwarder string. + """ if not value or "\0" in value: raise ValueError("Invalid forwarder") if not value.isascii(): @@ -113,7 +118,7 @@ class ExportEntry: @dataclass(frozen=True) class BaseRelocation: - """An entry of the base relocation table of a PE. The loader adds the base delta to the value at rva.""" + """An entry of the base relocation table of a PE. The loader adds the base delta to the value at ``rva``.""" rva: int type: int @@ -124,10 +129,10 @@ class LoadedImage: """ An image and its addresses. - A loader returns the image at its preferred base, image_base. The addresses are RVAs: + A loader returns the image at its preferred base, ``image_base``. The addresses are RVAs: the bases of the regions, the IAT slots, the exports, the entry points, the TLS callbacks - and the TLS and resource directories. relocate_image returns a copy at a base, with - absolute addresses. relocations is None if the image can load only at its preferred base. + and the TLS and resource directories. ``relocate_image`` returns a copy at a base, with + absolute addresses. ``relocations`` is None if the image can load only at its preferred base. """ arch: int @@ -153,7 +158,7 @@ class LoadedImage: def _write_image_base(image: LoadedImage, data: bytearray, base: int) -> None: - """Write base in the ImageBase field of the PE header, in the data of the region at RVA 0.""" + """Write ``base`` in the ImageBase field of the PE header, in the data of the region at RVA 0.""" if image.pe_metadata is None or len(data) < 0x40: return size = 8 if image.pe_metadata.magic == 0x20B else 4 @@ -165,14 +170,14 @@ def _write_image_base(image: LoadedImage, data: bytearray, base: int) -> None: def relocate_image(image: LoadedImage, base: int) -> LoadedImage: """ - Return a copy of the image at base, with absolute addresses. + Return a copy of the image at ``base``, with absolute addresses. - If base is not the preferred base, add the difference to the value at each base relocation - and write base in the ImageBase field of the PE header, as the Windows loader does. + If ``base`` is not the preferred base, add the difference to the value at each base relocation + and write ``base`` in the ImageBase field of the PE header, as the Windows loader does. Raises: - ValueError: if the image does not fit at base, the image has no relocation data and - base is not its preferred base, or a base relocation is not valid. + ValueError: if the image does not fit at ``base``, the image has no relocation data and + ``base`` is not its preferred base, or a base relocation is not valid. """ if type(base) is not int or base < 0 or base + image.image_size > 1 << image.arch: raise ValueError(f"Image does not fit in the address space at {base:#x}") @@ -304,10 +309,15 @@ def args(self) -> tuple[int, int, int]: def _delay_import_directory(pe: Any) -> tuple[int, bytes, list[int]]: """ - Check raw delay descriptors before trusting pefile's permissive parser. + Check the raw delay import descriptors, because pefile accepts malformed descriptors. - Legacy x86 descriptors contain VAs. pefile normalizes their struct fields - in place; retain the raw bytes so mapping does not rewrite the guest table. + Return the RVA and the raw bytes of the directory, and the thunk count of each + descriptor. Old x86 descriptors hold VAs, and pefile changes their fields to RVAs. + The caller writes the raw bytes back, so that the mapped image has the original table. + + Raises: + ValueError: if a descriptor or its tables are outside the image or not terminated, + or the attributes of a descriptor are not valid for the architecture. """ if len(pe.OPTIONAL_HEADER.DATA_DIRECTORY) <= 13: return 0, b"", [] @@ -356,10 +366,10 @@ def _delay_import_directory(pe: Any) -> tuple[int, bytes, list[int]]: class PeLoader: """ - Load guest PEs, tolerating malformed optional inventories. + Load a guest PE. - Malformed import and export metadata is skipped with a warning. Image - mapping and architecture validation are mandatory. + The loader logs a warning and skips malformed import, export and TLS callback data. + It rejects a PE with an unsupported machine type, or with headers or sections that do not fit the image. """ def __init__( @@ -381,7 +391,7 @@ def _get_image_problems(self, pe: Any) -> list[str]: ep_rva = pe.OPTIONAL_HEADER.AddressOfEntryPoint if ep_rva >= pe.image_size: problems.append(f"entry point RVA {ep_rva:#x} is outside the image (size {pe.image_size:#x})") - # Only raw data up to VirtualSize is mapped, and files often end + # The loader maps raw data only up to VirtualSize, and files often end # inside the FileAlignment padding of the last section. alignment = pe.OPTIONAL_HEADER.FileAlignment or 1 file_end = -(-pe.file_size // alignment) * alignment @@ -403,8 +413,8 @@ def _get_image_problems(self, pe: Any) -> list[str]: return problems def _import_dll_name(self, pe: Any, entry: Any, source: str) -> str: - # pefile replaces non-ASCII names with *invalid*. Read the original - # bounded string so accepting Latin-1 does not lose the name's bytes. + # pefile replaces a non-ASCII name with *invalid*. Read the raw string, so that + # the Latin-1 decode keeps all bytes of the name. rva = entry.struct.Name if source == "static" else entry.struct.szName if not 0 < rva < pe.image_size: raise ValueError("Import DLL name lies outside the image") @@ -416,8 +426,8 @@ def _import_dll_name(self, pe: Any, entry: Any, source: str) -> str: if terminator < 0: raise ValueError("Unterminated imported DLL name") raw = raw[:terminator] - # Retain pefile's filename rules for ASCII, allowing high bytes only - # through the explicit lossless decoding policy. + # Apply the file name rules of pefile, but accept bytes above 0x7F. The Latin-1 + # decode below keeps them. ascii_name = bytes(c if c < 128 else ord("x") for c in raw) if not raw or not pefile.is_valid_dos_filename(ascii_name): raise ValueError("Invalid imported DLL name") @@ -432,13 +442,16 @@ def _import_dll_name(self, pe: Any, entry: Any, source: str) -> str: return dll def _static_import_slots(self, pe: Any, entry: Any, parsed: Any, budget: list[int]): - """Retain raw thunk positions when pefile filters malformed names.""" + """ + Yield the IAT slot RVA and the symbol of each import of a static descriptor. + + Read the thunk table directly, so that each import keeps its slot when pefile drops a malformed name. + """ width = pe.arch // 8 table_rva = entry.struct.OriginalFirstThunk or entry.struct.FirstThunk if table_rva != entry.struct.FirstThunk: - # pefile falls back to the IAT when the preferred table is empty - # or unreadable. Static symbol addresses track relocation; validate - # their slot offsets before using thunk_rva as table-choice evidence. + # pefile reads the IAT if the ILT is empty or cannot be read. If the thunk_rva of a + # pefile symbol is its IAT slot, read the IAT too. Check the slot offset first. for symbol in parsed: address = getattr(symbol, "address", None) thunk_rva = getattr(symbol, "thunk_rva", None) @@ -469,8 +482,8 @@ def _static_import_slots(self, pe: Any, entry: Any, parsed: Any, budget: list[in return if not value: return - # Allow a terminator after exhaustion, but stop on the first - # further nonzero thunk: at most one lookahead per descriptor. + # When the limit is reached, a NUL thunk still ends the table, but the next + # nonzero thunk stops the read. if not budget[0]: self._optional_error("static import name table", ValueError("Import symbol limit exceeded")) return @@ -484,7 +497,7 @@ def _static_import_slots(self, pe: Any, entry: Any, parsed: Any, budget: list[in continue symbol = symbols.get(thunk_rva) if symbol is None: - # Recover metadata only; never rewrite guest thunks or IAT bytes. + # pefile dropped this symbol. Read its ordinal or name from the thunk. name = None if not ordinal: try: @@ -503,8 +516,8 @@ def _static_import_slots(self, pe: Any, entry: Any, parsed: Any, budget: list[in def make_image(self) -> LoadedImage: class InventoryParser(_PeParser): - # PeLoader builds these inventories from the raw pefile entries. - # Avoid legacy eager UTF-8 decoding before our policy can run. + # PeLoader reads the imports and exports from the pefile entries. The + # _PeParser versions decode names as UTF-8 and fail on other bytes. def _get_pe_imports(self): return {} @@ -521,8 +534,8 @@ def _get_pe_exports(self): if pe.OPTIONAL_HEADER.SizeOfHeaders > pe.image_size: raise ValueError("PE headers extend beyond the image") for section in pe.sections: - # Raw file-alignment padding is not a declared virtual allocation. - # Malware images may put that padding beyond SizeOfImage. + # Check only the virtual size. In malware, the FileAlignment padding of the + # raw data can go past SizeOfImage. if section.Misc_VirtualSize and section.VirtualAddress + section.Misc_VirtualSize > pe.image_size: raise ValueError("PE section extends beyond the image") @@ -551,8 +564,8 @@ def _get_pe_exports(self): except (ValueError, pefile.PEFormatError) as error: self._optional_error("delay import directory", error) parsed_delay = () - # Restore bounded raw descriptors even if their inventory is bad: - # pefile may have normalized legacy VA fields in place. + # Write back the raw descriptors also when they are not valid, because pefile + # can change old VA fields to RVAs. directory = pe.OPTIONAL_HEADER.DATA_DIRECTORY[13] delay_rva, size = directory.VirtualAddress, directory.Size delay_raw = b"" @@ -570,8 +583,8 @@ def _get_pe_exports(self): imports: list[ImportEntry] = [] ptr_size = pe.arch // 8 - # Eager delay binding uses exactly the same inventory as ordinary imports. - # pefile normalizes legacy x86 VA-based delay descriptors to RVAs. + # The loader binds delay imports at load time, with the same import list as static + # imports. pefile changes the VAs of old x86 delay descriptors to RVAs. parsed_static = { entry.struct.get_file_offset(): entry.imports for entry in getattr(pe, "DIRECTORY_ENTRY_IMPORT", ()) } @@ -579,18 +592,16 @@ def _get_pe_exports(self): if len(pe.OPTIONAL_HEADER.DATA_DIRECTORY) > 1: directory = pe.OPTIONAL_HEADER.DATA_DIRECTORY[1] if directory.VirtualAddress: - # The names-only pass retains descriptors whose entire symbol - # inventory was discarded by pefile's permissive parser. + # With dllnames_only, pefile keeps a descriptor also when it drops all of its symbols. try: static_entries = ( pe.parse_import_directory(directory.VirtualAddress, directory.Size, dllnames_only=True) or () ) except pefile.PEFormatError as error: self._optional_error("static import directory", error) - # pefile's > MAX_IMPORT_SYMBOLS guard can retain MAX + 1 symbols. - # Preserve that surface with a shared nonzero-slot cap, including - # malformed entries. Terminators do not consume it. This bounds our - # single-table walk without duplicating pefile's ILT/IAT read counter. + # pefile stops after more than MAX_IMPORT_SYMBOLS symbols, so it can keep + # MAX_IMPORT_SYMBOLS + 1. Use the same limit for the nonzero thunks of all static + # descriptors, also the malformed ones. NUL thunks do not count. static_budget = [pefile.MAX_IMPORT_SYMBOLS + 1] for entries, source in ((static_entries, "static"), (parsed_delay, "delay")): for entry in entries: @@ -627,8 +638,8 @@ def _get_pe_exports(self): parsed_exports = getattr(getattr(pe, "DIRECTORY_ENTRY_EXPORT", None), "symbols", ()) if pe.OPTIONAL_HEADER.DATA_DIRECTORY: directory = pe.OPTIONAL_HEADER.DATA_DIRECTORY[0] - # Windows classifies forwarders by the full declared range, even - # when Size runs past the image, so only the header must fit. + # Windows uses the full Size of the directory to find forwarders, also when Size + # goes past the image. So only the directory header must fit. directory_end = directory.VirtualAddress + directory.Size if (directory.VirtualAddress or directory.Size) and ( not directory.VirtualAddress or directory.Size < 40 or directory.VirtualAddress + 40 > pe.image_size @@ -651,7 +662,7 @@ def _get_pe_exports(self): if terminator < 0: raise ValueError("Unterminated forwarder within export directory") forwarder = directory_bytes[offset:terminator].decode("ascii") - # An empty forwarder is one the image writes at runtime. + # The image can write the forwarder string at run time, so an empty string is valid. if forwarder: validate_forwarder(forwarder) section = pe.get_section_by_rva(exp.address) @@ -688,7 +699,7 @@ def _get_pe_exports(self): def read_mapped(rva: int, size: int) -> bytes: return bytes(mapped_image[rva : rva + size]).ljust(size, b"\0") - # Windows reads the directory from the mapped image, so it may extend + # Windows reads the directory from the mapped image, so it can extend # into the zero-filled tail of a section that pefile does not parse. if tls_rva and tls_rva + 4 * ptr_size + 8 <= pe.image_size: tls_directory = tls_rva @@ -763,14 +774,13 @@ def read_mapped(rva: int, size: int) -> bytes: else: res_id = resource_id.struct.Id - # Regular resource entry if hasattr(resource_id, "directory"): for resource_lang in resource_id.directory.entries: if hasattr(resource_lang, "data"): data_rva = resource_lang.data.struct.OffsetToData size = resource_lang.data.struct.Size lang_id = resource_lang.id or 0 - # Calculate RVA of the entry structure for HRSRC compatibility + # FindResource returns the address of the IMAGE_RESOURCE_DATA_ENTRY as the HRSRC entry_offset = resource_lang.data.struct.get_file_offset() entry_rva = pe.get_rva_from_offset(entry_offset) diff --git a/speakeasy/windows/mapped_imports.py b/speakeasy/windows/mapped_imports.py index 6b40b55fc..19f382e31 100644 --- a/speakeasy/windows/mapped_imports.py +++ b/speakeasy/windows/mapped_imports.py @@ -97,8 +97,9 @@ def get_import_slots(read: Reader, base: int, ptr_size: int) -> Iterator[ImportS Yield nothing if the memory at base is not a PE image for this pointer size, or if the image has no imports. Check each RVA against SizeOfImage, and limit - the number of descriptors and thunks to read. Skip a descriptor or a thunk table - that cannot be read, and log a warning. + the number of descriptors and thunks to read. Stop at a descriptor that cannot + be read. Skip a descriptor without a valid DLL name, and the rest of a thunk + table that cannot be read. Log a warning for each of these. Raises: ValueError: if the headers cannot be read or the import directory is outside the image. @@ -156,7 +157,7 @@ def get_import_slots(read: Reader, base: int, ptr_size: int) -> Iterator[ImportS break current = int.from_bytes(image.read_rva(iat, ptr_size), "little") except _PARSE_ERRORS as error: - # The rest of this table cannot be read safely. The other descriptors do not depend on it. + # Skip the rest of this thunk table. The other descriptors do not depend on it. logger.warning("skipping injected PE import table %s at %#x: %s", dll_name, base, error) break yield ImportSlot(image, dll_name, base + iat, thunk, current, bool(ilt_rva), ptr_size) diff --git a/speakeasy/windows/module_loader.py b/speakeasy/windows/module_loader.py index 131fed728..700725c10 100644 --- a/speakeasy/windows/module_loader.py +++ b/speakeasy/windows/module_loader.py @@ -25,9 +25,9 @@ class ModuleLoader: its PEB by default, and the DLLs whose DllMain and TLS callbacks must run. The class records the result of each callback. - WindowsEmulator._load_image maps one image, and the emulator keeps the module - list. If a load fails, or a DllMain returns FALSE, this class removes what the - load added. + ``WindowsEmulator._load_image`` maps one image, and the emulator keeps the module + list. If a load fails, or an initializer fails in ``LoadLibrary``, this class + removes what the load added. """ def __init__(self, emu: WindowsEmulator) -> None: diff --git a/speakeasy/windows/netman.py b/speakeasy/windows/netman.py index b4811106b..3b74b4113 100644 --- a/speakeasy/windows/netman.py +++ b/speakeasy/windows/netman.py @@ -90,7 +90,7 @@ def get_recv_data(self, size, peek=False): class WininetComponent: - """Base class used for WinInet connections.""" + """Base class for the WinInet handle objects: instance, session and request.""" def __init__(self, handles): super().__init__() @@ -103,12 +103,12 @@ def get_handle(self): class WininetRequest(WininetComponent): - """WinInet request object.""" + """Represents a WinInet request.""" def __init__(self, handles, session, verb, objname, ver, ref, accepts, flags, ctx): super().__init__(handles) - # The WiniNet APIs default to a HTTP "GET" if no verb is specified + # WinInet uses GET when the caller gives no verb. self.verb: str if not verb: self.verb = "get" @@ -180,7 +180,15 @@ def get_response_size(self): return size def get_response(self): - """Check the configuration file for a handler for the current WinInet request.""" + """ + Get the response data from the HTTP config. + + A ``by_ext`` entry for the verb that matches the URL extension gives the data, else the + ``default`` entry. With no entry, the response has no data. + + Raises: + NetworkEmuError: if the config has no HTTP responses. + """ cfg = self.get_instance().config if self.response: @@ -264,7 +272,7 @@ def new_session(self, server, port, user, password, service, flags, ctx): class NetworkManager: """ - Class that manages network connections during emulation. + Keep the sockets and WinInet objects, and record the network events. Sockets and WinInet objects are not kernel objects, so their handles are in tables of the manager and not in the process handle table. Each WinInet level (instance, session and @@ -309,15 +317,13 @@ def get_address(self, domain): names = self.dns.names - # Do we have an IP for this name? if domain.lower() not in names.keys(): - # use the default IP (if any) return names.get("default") return names.get(domain.lower()) def get_dns_txt(self, domain): - """Return a configured DNS TXT record (if any).""" + """Get the data of the configured TXT record of a domain, or of the default record, or None.""" def _read_txt_data(txt): path = txt.path diff --git a/speakeasy/windows/objman.py b/speakeasy/windows/objman.py index 128d19f1f..180385200 100644 --- a/speakeasy/windows/objman.py +++ b/speakeasy/windows/objman.py @@ -70,7 +70,7 @@ def __init__(self, allocator: HandleAllocator): class KernelObject: - """Base class for Kernel objects managed by the object manager.""" + """Base class for the kernel objects of the object manager.""" def __init__(self, emu: WindowsEmulator): self.emu: WindowsEmulator = emu @@ -112,7 +112,7 @@ def get_mem_tag(self): class Driver(KernelObject): - """Class that represents DRIVER_OBJECTs created by the Windows kernel.""" + """Represents a DRIVER_OBJECT.""" def __init__(self, emu): super().__init__(emu=emu) @@ -129,7 +129,7 @@ def __init__(self, emu): self.object = self.nt_types.DRIVER_OBJECT(emu.get_ptr_size()) def create_reg_path(self, name): - """Create the service path in the registry for the created driver.""" + """Write the registry service path of the driver to guest memory as a UNICODE_STRING.""" self.reg_path = "\\Registry\\Machine\\System\\CurrentControlSet\\Services\\%s\x00" % (name) # noqa buf = self.reg_path.encode("utf-16le") self.reg_path = self.reg_path.strip("\x00") @@ -172,12 +172,10 @@ def init_driver_section(self): name_addr = ldte.address + ldte.sizeof() self.emu.mem_write(name_addr, dllname) - # Set the dll full name ldte.object.FullDllName.Length = len(dllname) ldte.object.FullDllName.MaximumLength = len(dllname) ldte.object.FullDllName.Buffer = name_addr - # Set the dll base name dllname = ntpath.basename(mod_name).encode("utf-16le") ldte.object.BaseDllName.Length = len(dllname) ldte.object.BaseDllName.MaximumLength = len(dllname) @@ -220,7 +218,6 @@ def init_driver_object(self, name=None, pe=None): if not pe and not self.name: name = "none" - # Allocate the driver object addr = self.emu.mem_map(drvobj.Size + len(us), tag=f"emu.object.{name}") drvobj.DriverName.Length = len(us) @@ -253,7 +250,7 @@ def read_back(self): class Device(KernelObject): - """Represents a DEVICE_OBJECT created by the windows kernel.""" + """Represents a DEVICE_OBJECT.""" def __init__(self, emu): super().__init__(emu=emu) @@ -269,7 +266,7 @@ def __init__(self, emu): class FileObject(KernelObject): - """Represents a FILE_OBJECT created by the windows kernel.""" + """Represents a FILE_OBJECT.""" def __init__(self, emu): super().__init__(emu=emu) @@ -288,12 +285,12 @@ def __init__(self, emu): super().__init__(emu=emu) self.object = self.nt_types.IO_STACK_LOCATION(emu.get_ptr_size()) - # Allocate two stack locations for now to handle IoGetNextIrpStackLocation calls + # Map two stack locations, so that IoGetNextIrpStackLocation gets mapped memory. self.address = emu.mem_map(self.sizeof() * 2, tag=self.get_mem_tag()) class Irp(KernelObject): - """I/O request packet used when performing device input/output.""" + """Represents an IRP, the I/O request packet that a driver gets for a device request.""" def __init__(self, emu): super().__init__(emu=emu) @@ -317,7 +314,7 @@ def get_curr_stack_loc(self): class Thread(KernelObject): - """Represents a Windows ETHREAD object that describes an OS level thread.""" + """Represents an ETHREAD.""" def __init__(self, emu, stack_base=0, stack_commit=0): super().__init__(emu=emu) @@ -408,13 +405,13 @@ def __init__(self, emu): class Process(KernelObject): - """An EPROCESS object used by the Windows kernel to represent a process.""" + """Represents an EPROCESS.""" def __init__( self, emu, pe=None, user_modules=None, name="", path="", cmdline="", base=0, session=0, fixed_peb=True ): super().__init__(emu=emu) - # TODO: For now just allocate a blank opaque struct for an EPROCESS + # Make ActiveProcessLinks an empty list. ntoskrnl.EPROCESS gives the offsets. self.object = self.nt_types.EPROCESS(emu.get_ptr_size()) self.address = emu.mem_map(self.sizeof(), tag=self.get_mem_tag(), perms=1, base=0xE0000000) if emu.get_arch() == _arch.ARCH_X86: @@ -444,7 +441,7 @@ def __init__( self.is_modified: bool = False self.veh_handlers: list[int] = [] self.unhandled_exception_filter: int = 0 - # Unwind data registered at runtime for code outside loaded images + # RtlAddFunctionTable and RtlInstallFunctionTableCallback add unwind data for code outside the images. self.function_tables: list[unwind.FunctionTable] = [] # As on Windows, a new process gets a copy of the current directory and environment of its parent. The # first process gets them from the config. @@ -456,7 +453,6 @@ def __init__( self.stdout: int = (0xF000) + 2 self.stderr: int = (0xF000) + 3 - # Initialize the process PEB self.peb: PEB = PEB(emu=emu, fixed=fixed_peb) self.peb_ldr_data: PebLdrData = PebLdrData(self.emu) self.is_peb_active: bool = False @@ -491,7 +487,7 @@ def set_process_parameters(self, emu): self.peb.write_back() def get_session_manager(self) -> SessionManager: - # only the user-mode emulator has sessions, and only it creates console processes and reads desktop names + # Only the user mode emulator has sessions. Only it creates console processes and reads desktop names. return cast("Win32Emulator", self.emu).get_session_manager() def alloc_console(self): @@ -536,8 +532,8 @@ def remove_all_modules_from_peb(self): self._peb_modules.clear() def add_module_to_peb(self, module): - # One loader entry per mapped image in this process. _peb_modules - # indexes membership by base; ldr_entries keeps load order. + # The process has one loader entry for each mapped image. _peb_modules finds a module by its base, and + # ldr_entries keeps the load order. if module.base in self._peb_modules: self._sync_peb_module_links() return @@ -550,12 +546,10 @@ def add_module_to_peb(self, module): name_addr = ldte.address + ldte.sizeof() self.emu.mem_write(name_addr, dllname) - # Set the dll full name ldte.object.FullDllName.Length = len(dllname) - 2 ldte.object.FullDllName.MaximumLength = len(dllname) ldte.object.FullDllName.Buffer = name_addr - # Set the dll base name dllname = (ntpath.basename(module.emu_path) + "\x00").encode("utf-16le") ldte.object.BaseDllName.Length = len(dllname) - 2 @@ -568,17 +562,16 @@ def add_module_to_peb(self, module): self._sync_peb_module_links() def remove_module_from_peb(self, module): - """Detach a mapped image during loader rollback without freeing memory.""" + """Remove the loader entry of a module when the loader undoes a load. The module memory is not freed.""" if module.base not in self._peb_modules: return - # Update the list in place so external references and surviving loader - # entry objects retain their identity and order. + # Change the list in place, because other code keeps a reference to it. self.ldr_entries[:] = [entry for entry in self.ldr_entries if entry.object.DllBase != module.base] del self._peb_modules[module.base] self._sync_peb_module_links() def _sync_peb_module_links(self): - """Link each loader ring through its own PEB_LDR_DATA sentinel.""" + """Link the entries of each loader list into a ring through the list head in PEB_LDR_DATA.""" pld = self.peb_ldr_data main_base = self.pe.base if self.pe is not None else self.base lists = ( @@ -594,7 +587,7 @@ def _sync_peb_module_links(self): link = getattr(entry.object, link_field) node = entry.address + getattr(entry.object.get_cstruct(), link_field).offset if link_field == "InInitializationOrderLinks" and module.base == main_base and module.is_exe(): - # The main executable is not a member of the initialization ring. + # Windows does not put the main executable in the initialization order list. link.Flink = link.Blink = node continue entries.append((node, link)) @@ -613,8 +606,8 @@ def _sync_peb_module_links(self): self.peb.write_back() def init_peb(self, modules): - # Repeated initialization extends the existing membership without - # reallocating entries or losing modules attached dynamically. + # A second call adds only the new modules. It keeps the existing loader entries, also the entries of the + # modules that the guest loaded. for mod in modules: self.add_module_to_peb(mod) self._sync_peb_module_links() @@ -688,8 +681,7 @@ class PEB(KernelObject): """ Represents the process environment block. - This structure contains a large amount of fields that are used internally by Windows. - Shellcode may parse this structure in order to resolve exported functions. + Shellcode often walks the loader lists of the PEB to find modules and their exports. """ def __init__(self, emu, address=None, fixed=True): @@ -705,11 +697,7 @@ def __init__(self, emu, address=None, fixed=True): class TEB(KernelObject): - """ - Represents the thread environment block. - - This structure contains a large amount of fields that are used internally by Windows. - """ + """Represents the thread environment block.""" def __init__(self, emu, address=0): super().__init__(emu=emu) @@ -744,10 +732,10 @@ def __init__(self, emu, dllname, tag=""): class IDT(KernelObject): """ - Represents the Interrupt descriptor table. + Represents the interrupt descriptor table. - This is currently a dummy structure and - only exists to detect if samples read or write to it. + The kernel emulator loads the table address into IDTR, so a sample can find and read the table. The + descriptors point into the kernel image. The emulator does not send interrupts through them. """ emu: WinKernelEmulator @@ -783,19 +771,19 @@ def init_descriptors(self): class Event(KernelObject): - """Describes event objects used by Windows for synchronization.""" + """Represents a KEVENT.""" def __init__(self, emu): super().__init__(emu=emu) self.object = self.nt_types.KEVENT(emu.get_ptr_size()) self.address = emu.mem_map(self.sizeof(), tag=self.get_mem_tag()) - # Events whose state no handler tracks stay signaled, so waits on them return. + # A new event is signaled, so a wait on an event that no handler sets or resets returns at once. self.signaled: bool = True self.manual_reset: bool = True class Mutant(KernelObject): - """Describes mutant objects used by Windows for synchronization.""" + """Represents a KMUTANT.""" def __init__(self, emu): super().__init__(emu=emu) @@ -804,7 +792,7 @@ def __init__(self, emu): class Semaphore(KernelObject): - """Describes semaphore objects used by Windows for synchronization.""" + """Represents a KSEMAPHORE.""" def __init__(self, emu): super().__init__(emu=emu) @@ -813,7 +801,7 @@ def __init__(self, emu): class ObjectManager: - """Class that manages kernel objects during emulation.""" + """Keep the kernel objects and the symbolic links of the emulator.""" def __init__(self, emu: WindowsEmulator): super().__init__() @@ -847,7 +835,7 @@ def remove_object(self, obj): self.objects.pop(addr) def dec_ref(self, obj): - """Dereferece an object and remove it from the object manager when its reference count is 0.""" + """Decrement the reference count of an object, and remove the object when the count is 0.""" if hasattr(obj, "ref_cnt"): obj.ref_cnt -= 1 if obj.ref_cnt <= 0: diff --git a/speakeasy/windows/regman.py b/speakeasy/windows/regman.py index 3e6951526..ffcf50ce7 100644 --- a/speakeasy/windows/regman.py +++ b/speakeasy/windows/regman.py @@ -80,7 +80,7 @@ def __init__(self, name, val_type, data): self.data: str | int | bytes | None = self.normalize_value(val_type, data) def normalize_value(self, val_type, data): - """Convert registry values to python types.""" + """Convert value data to the stored form. REG_BINARY data is stored as base64.""" if val_type in (REG_VALUE_TYPE.REG_EXPAND_SZ, REG_VALUE_TYPE.REG_MULTI_SZ, REG_VALUE_TYPE.REG_SZ): if not isinstance(data, str): raise RegistryEmuError("Invalid registry value expected string") @@ -157,7 +157,7 @@ def get_value(self, val_name): class RegistryManager: """ - Manages the emulation of the windows registry. This includes creating keys, subkeys and values. + Keep the keys and values of the emulated registry. The handles of open keys go into the table that ``get_handle_table`` returns, the table of the current process. The predefined keys, such as ``HKEY_LOCAL_MACHINE``, are fixed values and not @@ -237,7 +237,7 @@ def get_key_from_path(self, path: str) -> RegKey | None: return None def get_key_paths(self): - """Get the paths of the open keys and of the keys in the config.""" + """Get the paths of the known keys and of the config keys that the guest did not delete.""" paths = [key.get_path() for key in self.keys] if self.config: paths += [key.path for key in self.config.keys if key.path.lower() not in self.deleted_paths] @@ -248,9 +248,6 @@ def is_key_a_parent_key(self, path): return any(p.lower().startswith(prefix) for p in self.get_key_paths()) def get_subkeys(self, key): - # TODO: once we revamp the registry emulation, - # make this better - parent_path = key.get_path().rstrip("\\") + "\\" subkeys: list[str] = [] for test_path in self.get_key_paths(): @@ -269,7 +266,7 @@ def get_subkeys(self, key): return subkeys def get_key_from_config(self, path): - """See if the emulator config file contains a handler for the requested registry path.""" + """Get a new key from the config entry of a path, or None if no entry has the path or the key is deleted.""" if not self.config or path.lower() in self.deleted_paths: return None for key in self.config.keys: @@ -282,14 +279,12 @@ def get_key_from_config(self, path): return None def create_key(self, path): - """Create a registry key.""" + """Get the key of a path, and create the key if it does not exist.""" path = self.normalize_reg_path(path) - # Does this key already exist? key = self.get_key_from_path(path) if key: return key - # Does this key exist in our config key = self.get_key_from_config(path) if key: self.keys.append(key) @@ -329,18 +324,16 @@ def get_key_handle(self, path, create): """Open or optionally create a registry key with no event, and return a new handle or None.""" hnd = None path = self.normalize_reg_path(path) - # Does the key already exist? key = self.get_key_from_path(path) if key: return self.get_handle_table().new_handle(key) - # Does this key exist in our config key = self.get_key_from_config(path) if key: self.keys.append(key) return self.get_handle_table().new_handle(key) - # If we are instructed to create the key, do so + # A key that is the parent of a known key exists, also when no config entry names it. if create or self.is_key_a_parent_key(path): key = RegKey(path) hnd = self.get_handle_table().new_handle(key) @@ -352,7 +345,7 @@ def delete_key(self, handle): """ Delete the key that a handle names, and return False when the handle is not valid. - The handle stays open. After the delete, the configuration does not give the key. + The delete does not close the handle. After the delete, the configuration does not give the key. """ key = self.get_key(handle) if not key: diff --git a/speakeasy/windows/seh.py b/speakeasy/windows/seh.py index 430e2311b..3b9261bfc 100644 --- a/speakeasy/windows/seh.py +++ b/speakeasy/windows/seh.py @@ -1,11 +1,11 @@ """ Exception dispatch code that x64 (seh64.py) and x86 (seh32.py) share. -This is exception records, the vectored handlers that run before frame-based handlers, reporting, -and the entry points for CPU exceptions and RaiseException, which take the -architecture's dispatcher. +This module has the exception records, the vectored handlers that run before frame-based handlers, +the report of each exception, and the entry points for CPU exceptions and RaiseException. The entry +points get the dispatcher of the architecture as an argument. -The functions here are generator handlers (see guest_callbacks.py). +The dispatch functions are generator handlers (see guest_callbacks.py). """ from __future__ import annotations @@ -41,7 +41,7 @@ # Exceptions whose parameters are the access type and the address MEMORY_FAULTS = (ddk.STATUS_ACCESS_VIOLATION, STATUS_GUARD_PAGE_VIOLATION) -# Frames, registrations or scopes walked before a chain is taken as corrupt +# Maximum steps of a walk over frames, registrations or scopes, because a corrupt chain can loop MAX_FRAMES = 4096 ACCESS_TYPES = {code: name for name, code in ACCESS_CODES.items()} @@ -52,7 +52,7 @@ class UnhandledException(guest_callbacks.CallbackError): def __init__(self, code: int, address: int, information=()): if code == ddk.STATUS_ACCESS_VIOLATION and len(information) >= 2 and information[0] in ACCESS_TYPES: - # The error of the fault, as when no handler sees it + # Use the error of the fault, the same as when no handler gets the exception access_type = ACCESS_TYPES[information[0]] super().__init__(ErrorType(f"invalid_{access_type}"), information[1], access_type, pc=address) else: @@ -132,7 +132,12 @@ def vectored(self) -> GeneratorHandler: return False def resume(self) -> Resume: - """Continue at the context, which handlers may have changed.""" + """ + Continue at the context, which handlers can change. + + Raises: + UnhandledException: if the exception is not continuable. + """ if self.flags & EXCEPTION_NONCONTINUABLE: raise UnhandledException(STATUS_NONCONTINUABLE_EXCEPTION, self.address) return resume_at(self.emu, read_context(self.emu, self.context)) @@ -141,8 +146,11 @@ def unhandled(self) -> GeneratorHandler: """ Offer an exception that no frame takes to the top-level filter. - The filter is the one that SetUnhandledExceptionFilter sets. It Windows calls from the outermost frame of - the thread, and continue if it says so. Otherwise the exception is unhandled. + The sample sets the filter with SetUnhandledExceptionFilter. Windows calls the filter from the outermost frame + of the thread. If the filter returns EXCEPTION_CONTINUE_EXECUTION, execution continues at the context. + + Raises: + UnhandledException: if there is no filter, or the filter does not continue execution. """ top_level_filter = self.emu.get_unhandled_exception_handler() if top_level_filter: @@ -157,11 +165,11 @@ def dispatch_fault( emu, exception: CpuException, dispatch_exception: Dispatcher, error: guest_callbacks.CallbackError | None = None ) -> bool: """ - Dispatch a CPU exception like KiUserExceptionDispatcher, on the stack of the faulting thread. - - Return ``False`` when no handler takes it. + Dispatch a CPU exception as KiUserExceptionDispatcher does, on the stack of the faulting thread. - error is how the run ends when no handler takes it, by default an UnhandledException. + Return ``False`` if the dispatch fails before it calls a guest handler, for example when the thread has no + handlers. If the dispatch calls a handler and then no handler takes the exception, the run ends later with + ``error``, or with an UnhandledException when ``error`` is ``None``. """ ctx = emu.get_thread_context() thread_context.set_control(ctx, exception.pc, thread_context.stack_pointer(ctx)) @@ -193,10 +201,10 @@ def dispatch() -> GeneratorHandler: def raise_exception(emu, code: int, flags: int, params, dispatch_exception: Dispatcher) -> GeneratorHandler: """ - RaiseException: dispatch an exception raised by the caller of the API. + RaiseException: dispatch an exception at the caller of the API. - Windows ends the process when no handler takes the exception. Here RaiseException - returns instead, so that emulation goes on, unless the exception is not continuable. + Windows ends the process when no handler takes the exception. The emulator returns 0 from + RaiseException instead, so emulation continues. A non-continuable exception still ends the run. """ try: return (yield from raise_in_caller(emu, code, flags, params, dispatch_exception)) @@ -216,7 +224,7 @@ def raise_in_caller(emu, code: int, flags: int, params, dispatch_exception: Disp def rtl_raise_exception(emu, record: int, dispatch_exception: Dispatcher) -> GeneratorHandler: - """RtlRaiseException: dispatch the caller's record, raised at the caller.""" + """RtlRaiseException: set the address in the record of the caller to the caller, then dispatch the record.""" ctx = guest_callbacks.caller_context(emu) offset = windefs.EXCEPTION_RECORD(emu.get_ptr_size()).get_cstruct().ExceptionAddress.offset emu.write_ptr(record + offset, thread_context.instruction_pointer(ctx)) @@ -224,7 +232,7 @@ def rtl_raise_exception(emu, record: int, dispatch_exception: Dispatcher) -> Gen def capture_context(emu, address: int) -> None: - """RtlCaptureContext: the context of the caller, with CONTEXT_FULL (and segments on x64).""" + """RtlCaptureContext: write the context of the caller to address, with CONTEXT_FULL, and CONTEXT_SEGMENTS on x64.""" context = guest_callbacks.caller_context(emu) if thread_context.is_amd64(context): context.ContextFlags = thread_context.CONTEXT_FULL_AMD64 | thread_context.CONTEXT_SEGMENTS diff --git a/speakeasy/windows/seh32.py b/speakeasy/windows/seh32.py index 32f0e770d..3b4ed16a0 100644 --- a/speakeasy/windows/seh32.py +++ b/speakeasy/windows/seh32.py @@ -1,17 +1,16 @@ """ -x86 exception dispatch, after ntdll's RtlDispatchException and RtlUnwind. +x86 exception dispatch, modeled on RtlDispatchException and RtlUnwind of ntdll. -Handlers come from the chain of EXCEPTION_REGISTRATION_RECORDs at fs:[0]. +The emulator gets the handlers from the chain of EXCEPTION_REGISTRATION_RECORDs at fs:[0]. -The entry points are dispatch_fault, for CPU exceptions, and the API handlers -raise_exception, rtl_unwind, and the MSVC language handlers except_handler3 and -except_handler4_common. These are generator handlers (see guest_callbacks.py). -What x64 shares is in seh.py. +dispatch_fault dispatches CPU exceptions. The API handlers, such as raise_exception, rtl_unwind +and the MSVC language handlers, are generator handlers (see guest_callbacks.py). seh.py has the +code that x86 and x64 share. -MSVC frames (_except_handler3 and the SEH4 of _except_handler4_common) extend the -registration record with a scope table and a try level, and keep the frame's -saved esp and exception pointers next to it. Filters and __finally blocks run -with the frame's ebp; an __except block is entered with the frame's ebp and esp. +MSVC frames (_except_handler3, and the SEH4 frames of _except_handler4_common) add a scope table +and a try level to the registration record. The saved esp and the exception pointers of the frame +are just below the record. Filters and __finally blocks run with the ebp of the frame. An __except +block runs with the ebp and esp of the frame. See http://www.openrce.org/articles/full_view/21 """ @@ -36,15 +35,23 @@ END_OF_CHAIN = 0xFFFFFFFF -# An MSVC frame, by offset from its registration record of next, handler, scope -# table and try level, which is at ebp - 0x10 +# An MSVC frame. The offsets are from the registration record, which is at ebp - 0x10. +# +# struct CPPEH_RECORD { +# DWORD SavedEsp; // -0x08 +# PEXCEPTION_POINTERS ExceptionPointers; // -0x04 +# PVOID Next; // 0x00 +# PVOID ExceptionHandler; // 0x04 +# PVOID ScopeTable; // 0x08 +# DWORD TryLevel; // 0x0c +# }; SAVED_ESP = -0x08 EXCEPTION_POINTERS = -0x04 SCOPE_TABLE = 0x08 TRY_LEVEL = 0x0C FRAME_POINTER = 0x10 -# Try levels out of every scope +# Try levels outside of all scopes TRYLEVEL_NONE_SEH3 = 0xFFFFFFFF # -1 TRYLEVEL_NONE_SEH4 = 0xFFFFFFFE # -2 # SEH4 scope tables start with GS and EH cookie offsets @@ -70,7 +77,7 @@ def registrations(emu) -> Iterator[tuple[int, int, int]]: """ Yield the registration, next registration and handler of each record on the chain at fs:[0]. - A record that cannot be read ends the chain, as one outside the stack does on Windows. + The walk ends at a record that the emulator cannot read, as Windows ends it at a record outside the stack. """ registration = exception_list(emu) for _ in range(seh.MAX_FRAMES): @@ -106,10 +113,10 @@ def rtl_raise_exception(emu, record: int) -> GeneratorHandler: def call_handler(emu, handler: int, record: int, registration: int, context: int, dispatcher_context: int) -> GuestCall: """ - Build a call of an exception handler from a frame like the one ExecuteHandler2 leaves at EBP. + Build a call of an exception handler, with EBP at a frame like the one that ExecuteHandler2 leaves. - The frame has the saved EBP, a return address, then the arguments, which some - handlers, such as Borland C++'s, read through EBP. + The frame has the saved EBP, a return address, then the arguments. Some handlers, such as the + Borland C++ handlers, read the arguments through EBP. """ args = (record, registration, context, dispatcher_context) frame = guest_callbacks.alloca(emu, struct.pack("<6I", emu.reg_read(_arch.X86_REG_EBP), 0, *args)) @@ -144,7 +151,7 @@ def unwind_registrations(emu, target_frame: int, record: int = 0) -> Generator[G dispatcher_context = guest_callbacks.alloca(emu, bytes(4)) for registration, next_registration, handler in registrations(emu): - # Registrations are in stack order: one above the target means it is not on the chain + # Registrations are in stack order, so a registration above the target shows that the target is not on the chain if registration == target_frame or target_frame and registration > target_frame: break yield call_handler(emu, handler, record, registration, context, dispatcher_context) @@ -241,7 +248,11 @@ def except_handler3(emu, record: int, registration: int, context: int, dispatche def except_handler4_common( emu, cookie: int, check_cookie: int, record: int, registration: int, context: int, dispatcher: int ) -> GeneratorHandler: - """_except_handler4_common: SEH4, whose scope table is encoded with the security cookie. Cookies are not checked.""" + """ + _except_handler4_common: SEH4 XORs the scope table pointer with the security cookie. + + The emulator does not check the cookies. + """ scope_table = _u32(emu, registration + SCOPE_TABLE) ^ _u32(emu, cookie) frame = MsvcFrame(emu, registration, scope_table, SEH4_SCOPE_TABLE_HEADER, TRYLEVEL_NONE_SEH4) return (yield from msvc_handler(emu, frame, record, context)) diff --git a/speakeasy/windows/seh64.py b/speakeasy/windows/seh64.py index a786731e0..85fe78637 100644 --- a/speakeasy/windows/seh64.py +++ b/speakeasy/windows/seh64.py @@ -1,16 +1,16 @@ """ -x64 exception dispatch, after ntdll's RtlDispatchException and RtlUnwindEx. +x64 exception dispatch, modeled on RtlDispatchException and RtlUnwindEx of ntdll. -Handlers come from the unwind data of the frames on the stack. +The emulator gets the handlers from the unwind data of the frames on the stack. -The entry points are dispatch_fault, for CPU exceptions, and the API handlers -raise_exception, rtl_unwind_ex and c_specific_handler. These are generator -handlers (see guest_callbacks.py): they yield a GuestCall for each vectored -handler, language handler, filter and __finally block they run, and are sent -back its return value. What x86 shares is in seh.py. +dispatch_fault dispatches CPU exceptions. The API handlers, such as raise_exception, +rtl_unwind_ex and c_specific_handler, are generator handlers (see guest_callbacks.py). +A generator handler yields a GuestCall for each vectored handler, language handler, +filter and __finally block that it runs, and gets the return value of the call back. +seh.py has the code that x86 and x64 share. -Data that guest handlers see (EXCEPTION_RECORD, CONTEXT, DISPATCHER_CONTEXT) -lives on the guest stack below the handler's frame, as it does on Windows. +The emulator puts the data that guest handlers see (EXCEPTION_RECORD, CONTEXT, +DISPATCHER_CONTEXT) on the guest stack below the frame of the handler, as Windows does. See https://learn.microsoft.com/en-us/cpp/build/exception-handling-x64 """ @@ -57,8 +57,8 @@ def find_function(emu, pc: int) -> unwind.RuntimeFunction | None: """ Get the unwind data of the function at pc, like RtlLookupFunctionEntry. - The data comes from a loaded image, else from a table registered at runtime. Callback tables are not called: their - functions unwind as leaf functions. + The function gets the data from a loaded image, or else from a table that the sample added at runtime. The emulator + does not call the callbacks of callback tables, so their functions unwind as leaf functions. """ module = emu.get_module_from_addr(pc) fn = module and unwind.lookup_function_entry(emu.mem_read, module.base, pc) @@ -74,9 +74,9 @@ def walk_frames(emu, ctx, handler_type: int) -> Iterator[tuple[Any, unwind.Runti """ Unwind ctx in place, yielding (frame context, function, frame info) for each frame with unwind data. - Frames of the emulator itself (a generator handler's call into the guest) are unwound to - the guest state they interrupted, so a walk from a handler continues into the - frames that raised the exception, as it does through KiUserExceptionDispatcher. + At the return address of a guest callback, the walk restores the guest state that the guest callback + interrupted. So a walk from a handler continues into the frames that raised the exception, as a walk + through KiUserExceptionDispatcher does on Windows. """ for _ in range(seh.MAX_FRAMES): pc = ctx.Rip @@ -129,7 +129,7 @@ def rtl_raise_exception(emu, record: int) -> GeneratorHandler: def rtl_unwind_ex(emu, target_frame: int, target_ip: int, record: int, return_value: int) -> GeneratorHandler: - """RtlUnwindEx, called by the guest: unwind from its caller.""" + """RtlUnwindEx: unwind from the caller of the API.""" return ( yield from unwind_to(emu, guest_callbacks.caller_context(emu), target_frame, target_ip, record, return_value) ) @@ -153,15 +153,18 @@ def dispatch_exception(emu, record: int, context: int) -> GeneratorHandler: disposition = yield GuestCall(info.handler, (record, info.establisher_frame, context, dispatcher_context)) if disposition == windefs.ExceptionContinueExecution: return exception.resume() - # ExceptionContinueSearch; nested and collided unwinds are not distinguished + # Other dispositions, also ExceptionNestedException and ExceptionCollidedUnwind, continue the search return (yield from exception.unhandled()) def unwind_to(emu, ctx, target_frame: int, target_ip: int, record: int, return_value: int) -> GeneratorHandler: """ - Walk up from ctx and call termination handlers, like RtlUnwindEx. + Walk up from ctx and call the termination handlers, as RtlUnwindEx does. Then resume the target frame at target_ip with return_value in rax. + + Raises: + UnhandledException: if the walk does not find target_frame. """ fields = seh.read_record(emu, record) if record else seh.exception_record(emu, STATUS_UNWIND, 0, ctx.Rip) fields.ExceptionFlags |= EXCEPTION_UNWINDING if target_frame else UNWINDING @@ -219,7 +222,7 @@ def c_specific_handler( if flags & EXCEPTION_TARGET_UNWIND and any( b <= target_pc < e and h == handler and t == target for b, e, h, t in scopes ): - break # the scope also encloses the target: it is not being left + break # the scope also contains the target, so the unwind does not leave the scope if not target: # __finally emu.mem_write(dispatcher_context + SCOPE_INDEX_OFFSET, struct.pack(" None: def register_state(ctx) -> dict[str, int]: - """The general purpose registers of ctx, named like the emulator's register state.""" + """Get the general purpose registers of ctx, with the names of the register state of the emulator.""" if is_amd64(ctx): names = ("Rsp", "Rbp", "Rip", "Rsi", "Rdi", "Rax", "Rbx", "Rcx", "Rdx") + tuple(f"R{i}" for i in range(8, 16)) else: diff --git a/speakeasy/windows/unwind.py b/speakeasy/windows/unwind.py index 527c5ff29..88f9e1447 100644 --- a/speakeasy/windows/unwind.py +++ b/speakeasy/windows/unwind.py @@ -1,9 +1,9 @@ """ x64 table-based unwinding: RtlLookupFunctionEntry and RtlVirtualUnwind. -These operate on guest memory through a ``read(address, size) -> bytes`` -callable and on a CONTEXT64 struct that is unwound in place, so they serve -both the exception dispatcher and the API handlers of the same name. +The functions read guest memory through a ``read(address, size) -> bytes`` +callable and unwind a CONTEXT64 in place. The exception dispatcher and the API +handlers of the same name both use them. See https://learn.microsoft.com/en-us/cpp/build/exception-handling-x64 """ @@ -76,15 +76,14 @@ def read_runtime_function(read: Reader, image_base: int, address: int) -> Runtim @dataclass(frozen=True) class FunctionTable: """ - A table of RUNTIME_FUNCTIONs registered at runtime, for code outside loaded images. + A table of RUNTIME_FUNCTIONs that the sample adds at runtime, for code outside loaded images. - The table comes from RtlAddFunctionTable, or with a callback from - RtlInstallFunctionTableCallback. + RtlAddFunctionTable adds a table, and RtlInstallFunctionTableCallback adds a callback table. """ identifier: int # the table's address, or the callback table's identifier base: int - begin: int # the addresses it covers + begin: int # the address range of the table end: int count: int = 0 callback: int = 0 @@ -94,7 +93,8 @@ def read_function_table(read: Reader, address: int, count: int) -> Iterator[tupl """ Yield the (begin, end, unwind info) RVAs of a table of RUNTIME_FUNCTIONs. - The function reads a page at a time, so that a wrong count fails where the table runs out of memory. + The function reads about one page of entries at a time. So a count that is too large fails at the first unmapped + page, and the function does not try to read a very large range. """ per_page = 0x1000 // 12 for first in range(0, count, per_page): @@ -102,7 +102,7 @@ def read_function_table(read: Reader, address: int, count: int) -> Iterator[tupl def search_function_table(read: Reader, table: FunctionTable, pc: int) -> RuntimeFunction | None: - """Find pc in a registered table, whose entries need not be sorted.""" + """Find pc in a table that the sample added. The entries can be in any order, so the search is linear.""" for i, (begin, end, _) in enumerate(read_function_table(read, table.identifier, table.count)): if begin <= pc - table.base < end: return read_runtime_function(read, table.base, table.identifier + 12 * i) @@ -150,7 +150,7 @@ class UnwindInfo: def _slots(op: int, info: int) -> int: - """Number of 16-bit slots taken by an unwind code.""" + """Get the number of 16-bit slots that an unwind code uses.""" if op == UWOP_ALLOC_LARGE: return 3 if info else 2 return {UWOP_SAVE_NONVOL: 2, UWOP_SAVE_NONVOL_FAR: 3, UWOP_SAVE_XMM128: 2, UWOP_SAVE_XMM128_FAR: 3}.get(op, 1) @@ -174,12 +174,16 @@ class UnwindError(Exception): """Unwind data that cannot be followed.""" -# Chained unwind infos followed before the data is taken as corrupt +# Maximum length of a chain of unwind infos, because corrupt data can loop MAX_CHAIN = 32 def _prolog_offset(fn: RuntimeFunction, pc: int) -> int: - """The offset of pc into fn, unsigned: pc before fn, in a fragment of it, is past its prolog.""" + """ + Get the offset of pc into fn as an unsigned value. + + If pc is in a fragment of fn at a lower address than fn, the offset is very large, so pc is past the prolog. + """ return (pc - fn.image_base - fn.begin) & 0xFFFFFFFF @@ -199,15 +203,17 @@ def _establisher_frame(info: UnwindInfo, offset: int, ctx) -> int: def virtual_unwind(read: Reader, fn: RuntimeFunction, pc: int, ctx, handler_type: int = 0) -> FrameInfo: """ - Unwind one frame: ctx is updated in place to the caller's state. + Unwind one frame: update ctx in place to the state of the caller. + + Return the establisher frame, and the handler of ``handler_type`` (UNW_FLAG_EHANDLER or + UNW_FLAG_UHANDLER) if the frame has one. A fragment of a function uses the handler of the + primary entry at the end of its chain. - Returns the establisher frame and, when the frame has a handler of - ``handler_type`` (UNW_FLAG_EHANDLER or UNW_FLAG_UHANDLER), the handler, - which a fragment of a function shares with the function's primary entry - at the end of its chain. + A pc in an epilogue unwinds as a pc in the body. Windows simulates the rest of the epilogue + instead. - Not handled: a pc inside an epilogue, which Windows unwinds by simulating - the remaining epilogue instructions. + Raises: + UnwindError: if the chain of unwind infos is too long. """ base = fn.image_base info = read_unwind_info(read, base + fn.unwind_info) @@ -217,7 +223,7 @@ def virtual_unwind(read: Reader, fn: RuntimeFunction, pc: int, ctx, handler_type for _ in range(MAX_CHAIN): offset = _prolog_offset(fn, pc) for c in info.codes: - if c.prolog_offset > offset: # in the prolog, before this code's instruction ran + if c.prolog_offset > offset: # pc is in the prolog, before the instruction of this code continue if c.op == UWOP_PUSH_NONVOL: setattr(ctx, GPRS[c.info], _u64(read, ctx.Rsp)) @@ -257,6 +263,6 @@ def virtual_unwind(read: Reader, fn: RuntimeFunction, pc: int, ctx, handler_type def unwind_leaf(read: Reader, ctx) -> None: - """A function without unwind data is a leaf: its return address is at rsp.""" + """Unwind a leaf function, which has no unwind data and has its return address at rsp.""" ctx.Rip = _u64(read, ctx.Rsp) ctx.Rsp += 8 diff --git a/speakeasy/windows/win32.py b/speakeasy/windows/win32.py index d87c17e02..b1e88eb81 100644 --- a/speakeasy/windows/win32.py +++ b/speakeasy/windows/win32.py @@ -41,7 +41,7 @@ class Win32Emulator(WindowsEmulator): - """User Mode Windows Emulator Class.""" + """Emulate a user mode Windows process.""" file_name: str mod_name: str @@ -56,7 +56,7 @@ def __init__(self, config, argv=None, *, debug=False, exit_event=None, gdb_port= self.com: COM = COM(config) def get_argv(self): - """Get command line arguments (if any) that are being passed to the emulated process. (e.g. main(argv)).""" + """Get the ``argv`` of the emulated process, from the configured arguments or the command line.""" argv0 = "" out = [] @@ -105,15 +105,11 @@ def get_service_main_char_width(self, module, export_name): return 2 def get_session_manager(self) -> SessionManager: - """ - Get the session manager for the emulator. - - This will manage things like desktops, windows, and session isolation - """ + """Get the session manager, which holds the sessions, desktops and windows.""" return self.sessman def add_vectored_exception_handler(self, first, handler): - """Add a vectored exception handler that will be executed on an exception.""" + """Add a vectored exception handler. If ``first`` is true, the handler goes before the others.""" handlers = self.get_veh_handlers() if handler in handlers: return @@ -132,7 +128,7 @@ def init_processes(self, processes): """ Add the processes of the config to the process list. - Skip the entry with is_main_exe: the main process is the container process or the process of the input EXE. + Skip the entry with ``is_main_exe``: the main process is the container process or the process of the input EXE. """ for proc in processes: if proc.is_main_exe: @@ -189,7 +185,7 @@ def prepare_module_for_emulation(self, module, all_entrypoints, entry_point=None raise Win32EmuError("Module not found") startup = [] - # Check if any TLS callbacks exist, these run before the module's entry point + # TLS callbacks run before the entry point of the module. tls = module.get_tls_callbacks() for i, cb_addr in enumerate(tls): base = module.base @@ -223,9 +219,8 @@ def prepare_module_for_emulation(self, module, all_entrypoints, entry_point=None startup.append(run) if all_entrypoints: - # Only emulate a subset of all the exported functions - # There are some modules (such as the windows kernel) with - # thousands of exports + # Limit the number of export runs, because some modules, such as the Windows kernel, have + # thousands of exports. exports = [k for k in module.get_exports()[:MAX_EXPORTS_TO_EMULATE]] if exports: @@ -247,22 +242,21 @@ def prepare_module_for_emulation(self, module, all_entrypoints, entry_point=None argc, argv = self.build_service_main_args("IPRIP", char_width=char_width) run.args = [argc, argv] else: - # Here we set dummy args to pass into the export function + # Each export gets the same placeholder arguments. run.args = args if module.arch == _arch.ARCH_X86 and exp.name and THISCALL_MEMBER.match(exp.name): if this is None: this = self.mem_map(8, tag="emu.export_this", base=0x41420000) run.registers = {_arch.X86_REG_ECX: this} - # Store these runs and only queue them before the unload - # routine this is because some exports may not be ready to - # be called yet + # The export runs come after the DllMain run in the queue, because an export can depend + # on the state that DllMain sets up. self.add_run(run) return startup def run_module(self, module, *, all_entrypoints=False, emulate_children=False, entry_point=None) -> None: """ - Begin emulating a previously loaded module. + Add the runs of a loaded module to the run queue, and run the queue. Args: module: Module to emulate. @@ -272,8 +266,7 @@ def run_module(self, module, *, all_entrypoints=False, emulate_children=False, e """ startup = self.prepare_module_for_emulation(module, all_entrypoints, entry_point=entry_point) - # Create an empty process object for the module if none is - # supplied, only do this for the main module + # The main module gets a new process if there is no container process. if self.curr_process is None: if self.argv: self.command_line = f'"{module.emu_path}" {subprocess.list2cmdline(self.argv)}' @@ -298,16 +291,13 @@ def run_module(self, module, *, all_entrypoints=False, emulate_children=False, e peb = self.alloc_peb(self.curr_process) - # Set the TEB self.init_teb(t, peb) - # Begin emulation of main module self.start() if not emulate_children or len(self.child_processes) == 0: return - # Emulate any child processes finished = self.curr_process while len(self.child_processes) > 0: child = self.child_processes.pop(0) @@ -410,7 +400,12 @@ def load_shellcode(self, path, arch, data=None, filename=None) -> int: return sc_addr def run_shellcode(self, sc_addr, stack_commit=0x4000, offset=0) -> None: - """Begin emulating position independent code (i.e. shellcode) to prepare for emulation.""" + """ + Add a run for the loaded shellcode at ``sc_addr`` + ``offset``, and run the queue. + + Raises: + Win32EmuError: if no loaded module starts at ``sc_addr``. + """ target = None for mod in self.modules: if isinstance(mod, RuntimeModule) and mod.base == sc_addr: @@ -435,7 +430,7 @@ def run_shellcode(self, sc_addr, stack_commit=0x4000, offset=0) -> None: self.add_run(run) - # Create an empty process object for the shellcode if there is no container process + # The shellcode gets a new process if there is no container process. if self.curr_process is None: p = objman.Process(self) self.processes.append(p) @@ -453,13 +448,16 @@ def run_shellcode(self, sc_addr, stack_commit=0x4000, offset=0) -> None: peb = self.alloc_peb(self.curr_process) - # Set the TEB self.init_teb(t, peb) self.start() def alloc_peb(self, proc): - """Allocate memory for the Process Environment Block (PEB).""" + """ + Set up the PEB of a process and return it. + + Map the loader data, fill in the version fields and add the modules to the loader lists. + """ if proc.is_peb_active: return proc.peb size = proc.peb_ldr_data.sizeof() @@ -528,7 +526,7 @@ def _ensure_core_dlls_loaded(self): self.module_loader.shared_peb_modules.add(self.get_mod_by_name(dll).base) def set_unhandled_exception_handler(self, handler_addr): - """Establish a handler for unhandled exceptions that occur during emulation.""" + """Set the unhandled exception filter of the current process.""" self._get_process_state().unhandled_exception_filter = handler_addr def init_machine_structures(self): @@ -551,7 +549,7 @@ def init_machine_structures(self): self.init_processes(self.config.processes) def init_sys_modules(self, modules_config): - """Get the system modules (e.g. drivers) that are loaded in the emulator.""" + """Load the system modules of the config, hide them from the PEB and create their devices.""" rtmods = super().init_sys_modules(modules_config) for mod in rtmods: mod.visible_in_peb = False @@ -570,10 +568,11 @@ def load_container(self): """ Load the container image and make its process the current process. - A DLL or shellcode input runs in this process. Call this after init_machine and before the input image loads. + A DLL or shellcode input runs in this process. Call this method after ``init_machine`` and before the + input image loads. - The container image is the image of the entry in processes of the config that - has is_main_exe set. If the config has no such entry, do nothing. + The container image is the image of the entry in ``processes`` of the config that + has ``is_main_exe`` set. If the config has no such entry, do nothing. Raises: WindowsEmuError: if the machine is not initialized, or the container is already loaded. @@ -604,11 +603,7 @@ def load_container(self): return process def exit_process(self): - """ - An emulated binary is attempted to terminate its current process. - - Signal that the run has finished. - """ + """End the current run, because the guest ended its process.""" self.enable_deferred_work_hook() self.is_run_complete = True @@ -625,7 +620,7 @@ def _hook_mem_unmapped(self, emu, access, address, size, value): return super()._hook_mem_unmapped(emu, access, address, size, value) def set_hooks(self): - """Set the emulator callbacks.""" + """Add the hooks of the user mode emulator, then the tracing, coverage and debug hooks.""" super().set_hooks() if not self.has_builtin_hooks: @@ -633,7 +628,7 @@ def set_hooks(self): self.add_interrupt_hook(cb=self._hook_interrupt) self.add_invalid_instruction_hook(cb=self._hook_invalid_instruction) if self.config.exceptions.dispatch_handlers: - # User mode cannot execute these; Windows raises an exception instead + # IN and OUT are privileged in user mode, so Windows raises an exception. self.add_instruction_hook(cb=self._hook_in_instruction, insn=UC_X86_INS_IN) self.add_instruction_hook(cb=self._hook_out_instruction, insn=UC_X86_INS_OUT) self.has_builtin_hooks = True @@ -644,12 +639,10 @@ def set_hooks(self): def stop(self) -> None: self.is_run_complete = True - # self._map_reserved_range() - # self.unset_hooks() super().stop() def on_emu_complete(self): - """Called when all runs have completed emulation.""" + """Find the decoded strings if the config asks for them, then stop the emulation.""" if not self.is_emu_complete: self.is_emu_complete = True if self.config.analysis.strings and self.profiler: @@ -663,11 +656,7 @@ def on_emu_complete(self): self.stop() def on_run_complete(self): - """ - Clean up after a run completes. - - This function will pop the next run from the run queue and emulate it. - """ + """Record the result of the run, then prepare the next run in the queue.""" self.is_run_complete = True run = self.curr_run if run is not None: @@ -682,7 +671,7 @@ def on_run_complete(self): return self._prepare_next_run() def _capture_memory_layout(self, run: Run) -> None: - """Capture current memory layout and loaded modules for the run report.""" + """Record the memory layout and the loaded modules in the report of the run.""" EXCLUDED_TAG_PREFIXES = ("emu.stack", "api.heap", "emu.process_heap") prot_map = { diff --git a/speakeasy/windows/winemu.py b/speakeasy/windows/winemu.py index 98b1e566b..07b777c59 100644 --- a/speakeasy/windows/winemu.py +++ b/speakeasy/windows/winemu.py @@ -72,16 +72,16 @@ if TYPE_CHECKING: from speakeasy.speakeasy import Speakeasy -# The number of exports of a DLL that the emulator runs when it emulates all exports +# The maximum number of exports of a module that the emulator runs when it emulates all exports MAX_EXPORTS_TO_EMULATE = 10 -# GDB signal number reported for memory faults +# The signal number that the GDB server reports for memory faults GDB_SIGSEGV = 11 -# Upper bound on the static TLS data copied for each thread +# The maximum size of the static TLS data that the emulator copies for each thread TLS_TEMPLATE_MAX = 0x1000000 -# Registers a callee must preserve; guest callbacks of an API return them to its caller unchanged +# The registers that a callee must preserve. The emulator restores them after the guest callbacks of an API. NONVOLATILE_X86 = (_arch.X86_REG_EBX, _arch.X86_REG_EBP, _arch.X86_REG_ESI, _arch.X86_REG_EDI) NONVOLATILE_AMD64 = ( _arch.AMD64_REG_RBX, @@ -109,10 +109,10 @@ class _ExecutionBudget: @dataclass(frozen=True) class SuspendedApiCall: """ - The guest reached the trap of an API. The handler of the API runs next. + The guest called an API. The run loop calls the handler of the API next. - The stack pointer, the return address and the code of the API are the values at the trap. - If a debugger changes one of them, the guest runs the code of the API again, and the handler does not run. + The stack pointer, the return address and the code of the API are the values when the guest fetched the token. + If a debugger changes one of them, the emulator drops the exit, and the guest runs the code of the API again. """ entry: ApiEntry @@ -128,12 +128,12 @@ class RunReturn: @dataclass(frozen=True) class CallbackReturn: - """A guest callback returned to GUEST_CALLBACK_RETURN_ADDR. The API handler that called it continues.""" + """A guest callback returned to ``GUEST_CALLBACK_RETURN_ADDR``. The API handler that called it continues.""" @dataclass(frozen=True) class FaultReturn: - """A fault ended the run. The run completes after Unicorn stops, also if Unicorn then raises an error.""" + """A fault ended the run. The run loop completes the run after Unicorn stops, also if Unicorn raises an error.""" @dataclass(frozen=True) @@ -141,7 +141,7 @@ class ExecRecovery: """ The guest fetched code from a guest PE page without execute permission. - The run loop sets perms on the page and runs the code again. + The run loop sets the protection of the page to ``perms`` and runs the code again. """ page: int @@ -150,19 +150,20 @@ class ExecRecovery: @dataclass(frozen=True) class RaisedCpuException: - """A CPU exception that goes to the exception handlers of the guest.""" + """A CPU exception that the run loop sends to the exception handlers of the guest.""" exception: cpu_exceptions.CpuException - error_type: ErrorType # the run's error if no handler takes it + error_type: ErrorType # the error of the run if no handler takes the exception interrupt_num: int | None = None access_type: str | None = None - rax: int | None = None # restored before dispatch, since IN overwrites it - # (page, perms) of a guard page, whose protection the access restores + rax: int | None = None # the run loop restores RAX before dispatch, because the IN instruction writes it + # (page, perms) of a guard page. The run loop restores perms on the page before dispatch. guard_page: tuple[int, int] | None = None -# The reason why Unicorn stopped. A hook records it and stops Unicorn, and then the run loop handles it. -# If a second hook records an exit before Unicorn stops, the first exit stays. +# The reason why Unicorn stopped. A hook records the exit and stops Unicorn, or _get_exit_at_pc gets the exit from +# the PC after Unicorn stops. Then the run loop handles the exit. If two hooks record an exit, the emulator keeps the +# first. NativeExit = SuspendedApiCall | RunReturn | CallbackReturn | FaultReturn | ExecRecovery | RaisedCpuException @@ -190,7 +191,11 @@ def api_name(self) -> str: def execution_scope(func: Callable[Concatenate[S, P], R]) -> Callable[Concatenate[S, P], R]: - """Share one active-time cap across every start in a public invocation.""" + """ + Apply one ``max_total_time`` limit to all emulation in a call of the decorated method. + + A nested call of a decorated method uses the limit of the outer call. + """ @wraps(func) def wrapped(self: S, *args: P.args, **kwargs: P.kwargs) -> R: @@ -218,7 +223,7 @@ def _module_type_from_path(path: str, default: str = "dll") -> str: def get_page_protection_runs(page_perms: dict[int, int], page_size: int) -> list[tuple[int, int, int]]: - """Merge per-page permissions into (base, size, perms) runs of contiguous pages with equal permissions.""" + """Merge the permissions of single pages into (base, size, perms) ranges of adjacent pages with equal perms.""" runs: list[tuple[int, int, int]] = [] for page_base in sorted(page_perms): perms = page_perms[page_base] @@ -233,7 +238,7 @@ def get_page_protection_runs(page_perms: dict[int, int], page_size: int) -> list @dataclass class RunLimits: - """The time and instruction limits of one call of the run loop, and the instructions it counts.""" + """The time and instruction limits of one call of the run loop, and the instruction count of the call.""" # When the call started, from time.monotonic(). started: float @@ -251,12 +256,12 @@ class RunLimits: budget: int # The instructions that the run can still execute, or -1 for no limit. remaining: int - # The last instruction that the accounting hook saw start, when there is one. + # The address of the last instruction that the accounting hook counted, or None. last_executed: int | None = None @classmethod def from_run(cls, run: Run, timeout: float, budget: int, total_budget: _ExecutionBudget | None) -> "RunLimits": - """Get the limits that apply to run from now, with the nearer of the two time limits as the deadline.""" + """Get the limits that apply to ``run`` from now. The deadline is the nearer of the two time limits.""" started = time.monotonic() spent = run.execution_elapsed deadline = started + max(timeout - spent, 0) if timeout > 0 else None @@ -284,28 +289,26 @@ def is_total_time_reached(self) -> bool: ) def get_native_timeout(self) -> float: - """Get the timeout for the next native start: the time to the deadline, or 0 for none.""" + """Get the timeout for the next start of Unicorn: the time to the deadline, or 0 for no deadline.""" return max(self.deadline - time.monotonic(), 0.000001) if self.deadline is not None else 0 def get_native_count(self) -> int: """ - Get the instruction count for the next native start, or 0 for no limit. + Get the instruction count for the next start of Unicorn, or 0 for no limit. - The count stops native execution at the instruction limit, so that - tracing hooks do not see an instruction beyond it. The accounting hook - counts the instructions across the native starts of one call. + The count stops Unicorn at the instruction limit, so that the tracing + hooks do not see an instruction after the limit. The accounting hook + counts the instructions of all starts of Unicorn in one call. """ return self.remaining if self.budget > 0 else 0 class WindowsEmulator(BinaryEmulator): """ - Base class providing emulation of all Windows modules and shellcode. - - This class is meant to provide overlapping functionality for both user mode and kernel mode samples. + Base class of the user mode and kernel mode emulators. - Subclasses must define: - peb_addr: Address of the Process Environment Block + It adds processes, threads, modules, runs, API calls and SEH to BinaryEmulator. Win32Emulator sets ``peb_addr`` + to the address of the PEB pointer in the TEB (``fs:[0x30]`` or ``gs:[0x60]``). """ _X64_EXEC_ADDR_MASK = (1 << 52) - 1 @@ -316,12 +319,12 @@ class WindowsEmulator(BinaryEmulator): @abstractmethod def alloc_peb(self, proc: Any) -> None: - """Allocate memory for the Process Environment Block (PEB). Subclasses must implement.""" + """Allocate memory for the Process Environment Block (PEB).""" ... @abstractmethod def init_processes(self, processes: list[Any]) -> None: - """Initialize configured processes. Subclasses must implement.""" + """Create the processes of the config.""" ... def __init__(self, config, *, exit_event=None, debug=False, gdb_port=None, gdb_host=None): @@ -356,7 +359,7 @@ def __init__(self, config, *, exit_event=None, debug=False, gdb_port=None, gdb_h self.api_registry: ApiRegistry = ApiRegistry(self) self._native_exit: NativeExit | None = None self._api_call_site: tuple[int, int] | None = None - # Guard pages, mapped without access, with the protection their first access restores + # Maps each guard page to the protection that its first access restores. Guard pages have no access. self.guard_pages: dict[int, int] = {} self.hardware_breakpoints = hardware_breakpoints.HardwareBreakpoints(self) self.module_loader: ModuleLoader = ModuleLoader(self) @@ -368,20 +371,18 @@ def __init__(self, config, *, exit_event=None, debug=False, gdb_port=None, gdb_h self.virtual_mem_base: int = 0x50000 # Code hook for work that must wait until the next instruction, such as the end - # of a run. It is enabled only when there is such work. + # of a run. The emulator enables the hook only when there is such work. self.deferred_work_hook: Any | None = None self.is_run_complete: bool = False self.is_emu_complete: bool = False self.processes: list[Any] = [] - # Child processes created by calls to CreateProcess - # by any module. This is separate from self.processes in order - # to not mix up config processes with child processes + # The processes that the guest creates with CreateProcess. self.processes does not include them. self.child_processes: list[Any] = [] self.curr_thread: Any | None = None self.prev_pc: int = 0 - # The state of the kernel, and of a user-mode emulator before a process exists. Each process has its own. - # The environment is in BinaryEmulator.env. + # The state of the kernel, and of a user mode emulator before a process exists. Each process has its own + # copy of these fields. BinaryEmulator.env has the environment. self.cd: str = "" self.heap_allocs: list[tuple[int, int, str]] = [] self.veh_handlers: list[int] = [] @@ -402,7 +403,6 @@ def __init__(self, config, *, exit_event=None, debug=False, gdb_port=None, gdb_h self.handle_allocator: HandleAllocator = HandleAllocator() # The handles of the kernel, and of a user-mode emulator before a process exists self.handles: HandleTable = HandleTable(self.handle_allocator, self.dec_ref, self.get_pseudo_handle_object) - # OS resource managers self.regman: RegistryManager = RegistryManager( self.get_handle_table, self.config.registry, os_ver=self.config.os_ver, emu=self ) @@ -416,17 +416,16 @@ def __init__(self, config, *, exit_event=None, debug=False, gdb_port=None, gdb_h self._io_manager: Any | None = None def _parse_config(self, config): - """Parse the emulation config file.""" super()._parse_config(config) self.cd = self.config.current_dir self.command_line: str = self.config.command_line def init_machine(self, arch): """ - Create the CPU, the disassembler and the API layer for arch. + Create the Unicorn engine, the disassembler and the API layer for ``arch``. Then create the Windows structures and the modules of the machine. Load images after this. The container image - of a DLL or shellcode input is not part of the machine: Win32Emulator.load_container loads it. + of a DLL or shellcode input is not part of the machine. Win32Emulator.load_container loads it. Raises: WindowsEmuError: if the machine is already initialized, or arch is not x86 or amd64. @@ -474,27 +473,22 @@ def get_object_manager(self, reason: str) -> objman.ObjectManager: return self.om def on_run_complete(self): - """ - Clean up after a run completes (implemented in the child class) since this may mean different things depending. - - This function will pop the next run from the run queue and emulate it. - """ - # Implemented by a subclass (e.g. kernel/user mode emulators) + """Record the result of the current run and prepare the next run from the queue.""" raise NotImplementedError() def end_run_on_fault(self): """ End the current run after a handled memory fault. - With a debugger attached, stop the engine at the fault instead, and end the run after the debugger resumes. - Faults at a PC in the reserved emulator range, such as the return hook, are not sample faults, so they do not + With a debugger attached, stop Unicorn at the fault, and end the run after the debugger resumes. + A fault at a PC in the reserved range, such as the return hook, is not a fault of the guest, so it does not stop the debugger. """ pc = self.get_pc() in_reserved = winemu.EMU_RESERVED <= pc < winemu.EMU_RESERVED + winemu.EMU_RESERVE_SIZE if not self._should_stop_on_faults or in_reserved: - # The fault belongs to this run. Go to the next run only after Unicorn stops, - # also if Unicorn raises an error after emu_stop. Otherwise the fault can change the next run. + # The run loop goes to the next run only after Unicorn stops, also if Unicorn raises an error + # after the stop. Otherwise the rest of the fault can change the next run. self._record_exit(FaultReturn()) return self._pending_fault_stop = StopReason(signal=GDB_SIGSEGV, kind="exception", address=pc) @@ -559,7 +553,7 @@ def pipe_open(self, path, mode, num_instances, out_size, in_size): return self.fileman.pipe_open(path, mode, num_instances, out_size, in_size) def does_file_exist(self, path): - """Test if a file handler for a specified emulated file exists.""" + """Get whether the emulated file at path exists.""" return self.fileman.does_file_exist(path) def file_create_mapping(self, hfile, name, size, prot): @@ -607,7 +601,7 @@ def dev_ioctl(self, arch, dev, ioctl, inbuf): return self._io_manager.dev_ioctl(arch, dev, ioctl, inbuf) def reg_open_key(self, path, *, create=False): - """Open or create a registry key in the emulation space.""" + """Open a key of the emulated registry. If create is set, create the key if it does not exist.""" return self.regman.open_key(path, create) def reg_get_subkeys(self, hkey): @@ -727,7 +721,7 @@ def record_http_event(self, server, port, headers="", body=b"", secure=False): ) def _unmap_reserved_range(self): - """Unmap reserved memory space so we can handle events (e.g. import APIs, entry point returns, etc.).""" + """Unmap the reserved range, so that a return to the return hook or the exit hook faults.""" if not self.is_reserved_range_unmapped: self.mem_unmap(winemu.EMU_RETURN_ADDR, winemu.EMU_RESERVE_SIZE) self.is_reserved_range_unmapped = True @@ -744,9 +738,9 @@ def _map_reserved_range(self): def get_valid_ranges(self, size, addr=None): """ - Keep allocations out of the reserved emulator range. + Get a free range for an allocation outside the reserved range. - The range stays unmapped so that a return to the return hook faults. + The emulator keeps the reserved range unmapped, so that a return to the return hook faults. """ base, total = super().get_valid_ranges(size, addr=addr) if base < winemu.EMU_RESERVED_END and base + total > winemu.EMU_RESERVED: @@ -758,7 +752,11 @@ def add_run(self, run): self.run_queue.append(run) def _prepare_next_run(self): - """Execute the next run from the emulation queue.""" + """ + Finish the current run, and prepare the next run from the queue. + + Return the prepared run. If the queue is empty, complete the emulation and return None. + """ self._finish_run_initializers(self.curr_run) self._native_exit = None try: @@ -778,9 +776,9 @@ def _prepare_next_run(self): def _queue_startup_initializers(self): """ - Add a run for each initializer of the guest DLLs that the process loaded at start. + Add a run for each DllMain and TLS callback of the DLLs that the process loaded at start. - These runs go before the first run in the queue. + Put these runs before the first run in the queue. """ if self.is_kernel_mode or not self.run_queue: return @@ -832,7 +830,7 @@ def _finish_run_initializers(self, run): ] def call(self, addr, params=None) -> None: - """Start emulating at the specified address.""" + """Add a run that starts at addr. If the queue is empty, start the emulation.""" self.reset_stack(self.stack_base) self.reset_cpu_context() mm = self.get_address_map(self.stack_base - 1) @@ -849,7 +847,7 @@ def call(self, addr, params=None) -> None: self.add_run(run) def _prepare_run_context(self, run): - """Prepare CPU and memory state for the given run without starting emulation.""" + """Set the registers, the stack, the process and the thread for run. Do not start Unicorn.""" logger.info("* exec: %s", run.get_label()) self.curr_run = run @@ -877,9 +875,7 @@ def _prepare_run_context(self, run): self.curr_run.stack = MemAccess(base=stk_map.base, size=stk_map.size) - # Set the process context if possible if run.process_context: - # Init a new peb if the process context changed: if run.process_context != self.get_current_process(): self.alloc_peb(run.process_context) self.set_current_process(run.process_context) @@ -895,7 +891,6 @@ def _prepare_run_context(self, run): self.set_current_thread(thread) if not self.is_kernel_mode: - # Reset the TIB data thread = self.get_current_thread() if thread: thread.exit_code = None @@ -914,23 +909,24 @@ def _prepare_run_context(self, run): return run def mem_cast(self, obj, addr): - """Turn bytes from an emulated memory pointer into an object.""" + """Read a structure of the type of obj from guest memory at addr.""" size = obj.sizeof() struct_bytes = self.mem_read(addr, size) return self.cast(obj, struct_bytes) def setup_user_shared_data(self): - """Setup the shared user data section that is often used to share data between user mode and kernel mode.""" + """Map KUSER_SHARED_DATA at its user mode address and its kernel mode address, and write its data.""" aliases = [] if self.get_arch() == _arch.ARCH_X86: aliases.append(0xFFDF0000) elif self.get_arch() == _arch.ARCH_AMD64: aliases.append(0xFFFFF78000000000) + # On x64, Unicorn truncates the address of a guest memory access to 52 bits, so a read from the kernel + # address goes to the truncated address. Map the page there too. mirrors = {a & self._X64_EXEC_ADDR_MASK for a in aliases if a & self._X64_EXEC_ADDR_MASK != a} - # This is a read-only address for KUSER_SHARED_DATA, - # and this is the same address for 32-bit and 64-bit. + # The user mode address is the same on x86 and x64. bases = [0x7FFE0000] + aliases + list(mirrors) kuser_tag = "emu.struct.KUSER_SHARED_DATA" @@ -939,9 +935,8 @@ def setup_user_shared_data(self): for base in bases: mm = self.get_address_map(base) if mm and not (mm.tag or "").startswith(kuser_tag): - # Another mapping already owns this alias: leave it completely - # alone rather than colliding with it while mapping (which - # breaks every kernel-mode load) or stomping its contents here. + # Another mapping uses this address. A second map fails and stops the kernel mode load, and + # a write changes the data of the other mapping. continue if not mm: self.mem_map(self.page_size, base=base, tag=kuser_tag) @@ -1036,9 +1031,9 @@ def _start_unicorn( self, address: int, limits: RunLimits, origin_run: Run, debugger: GdbServer | None, stepping: bool ) -> bool: """ - Run guest code in Unicorn from address until it stops, and raise the CPU exception of a `hlt` that ran. + Run guest code in Unicorn from address until it stops, and raise the CPU exception of a ``hlt`` that ran. - Get False if a fault stops the debugger. + Return False if a fault stops the debugger. Raises: uc.UcError: if Unicorn fails and no native exit explains the failure. @@ -1051,15 +1046,14 @@ def _start_unicorn( registers = self.get_register_state() counted = (limits.remaining, origin_run.budget_instructions, origin_run.instr_cnt) if limits.budget <= 0: - # No accounting hook records the instruction of the step, - # which is the one at its start address. + # There is no accounting hook, so record the instruction of the step here. limits.last_executed = address self.emu_eng.start(address, timeout=native_timeout, count=native_count) if stepping and self._step_had_no_effect(registers, debugger): - # Unicorn translates a store into the page of its own code - # again and runs it again, which uses up a count of 1. The - # second start counts the instruction as a run without a - # debugger does, so the first start must not count it. + # When an instruction writes to the page of its own code, Unicorn + # translates the page again and runs the instruction again. This + # uses up the count of 1. Undo the counts of the first start, so + # that the second start counts the instruction once. limits.remaining, origin_run.budget_instructions, origin_run.instr_cnt = counted self.emu_eng.start(address, timeout=native_timeout, count=2) except uc.UcError as exc: @@ -1087,7 +1081,7 @@ def _exit_explains_error(self, exc: uc.UcError) -> bool: return bool(exc.errno == uc.UC_ERR_FETCH_UNMAPPED) def _record_exit(self, native_exit: NativeExit) -> None: - """Record native_exit and stop Unicorn. If an exit is already recorded, it stays.""" + """Record native_exit and stop Unicorn. If an exit is already recorded, keep that exit.""" if self._native_exit is None: self._native_exit = native_exit self.emu_eng.stop() @@ -1110,7 +1104,7 @@ def _get_exit_at_pc(self) -> NativeExit | None: return None def _add_accounting_hook(self, limits: RunLimits, origin_run: Run, debugger: GdbServer | None): - """Add a code hook that counts each instruction against the instruction limit and stops the engine at it.""" + """Add a code hook that counts each instruction and stops Unicorn at the instruction limit.""" def account_instruction(_emu, address, _size): if debugger is not None and debugger.has_pending_stop(): @@ -1154,8 +1148,8 @@ def _finish_current_run_execution(self, limits: RunLimits, origin_run: Run, debu if not self.is_run_complete and origin_run.error is None: origin_run.error = ErrorInfo(type=ErrorType.MAX_TOTAL_TIME, pc=self.get_pc()) elif prepared is not None: - # Completing the run already prepared the next one. It - # did not execute and must not appear as a started run. + # on_run_complete prepared the next run. If that run did not + # execute, remove it from the list of runs. self._cancel_execution_runs(prepared) if prepared.execution_elapsed == 0 and prepared.get_api_count() == 0: self.runs[:] = [run for run in self.runs if run is not prepared] @@ -1170,7 +1164,7 @@ def _handle_exit(self, limits: RunLimits, origin_run: Run, debugger: GdbServer | """ Handle the native exit. - Get False if the run loop must return, because the debugger stops, a limit is reached, or the run ended. + Return False if the run loop must return, because the debugger stops, a limit is reached, or the run ended. """ native_exit = self._native_exit match native_exit: @@ -1206,12 +1200,12 @@ def _dispatch_api_entry( self, entry: ApiEntry, limits: RunLimits, origin_run: Run, debugger: GdbServer | None ) -> bool: """ - Run the handler of the API whose trap the guest reached, and end the run if the handler completed it. + Call the handler of the API that the guest called, and end the run if the handler completed it. - Get False if the call of the run loop must return, because the debugger stops at an API with no handler, a - fault stops the debugger, or the total execution time ran out. + Return False if the call of the run loop must return, because the debugger stops at an API with no + implementation, a fault stops the debugger, or the total execution time ran out. """ - # A missing ABI remains a suspended call under the debugger. + # With a debugger, an API without an implementation stops the debugger at the API, and the run continues. if debugger is not None and isinstance( self.resolve_api_implementation(entry.dll, entry.name), api_resolution.UnsupportedImplementation ): @@ -1235,7 +1229,7 @@ def _stop_without_exit(self, limits: RunLimits, origin_run: Run, debugger: GdbSe """ Stop origin_run after Unicorn stopped with no native exit. - If a limit is reached, stop at the limit. Otherwise, end the run with an engine_stopped error, if the run is + If a limit is reached, stop at the limit. Otherwise, end the run with an ``engine_stopped`` error, if the run is not complete and no debugger is attached. """ reached = limits.get_reached_limit() @@ -1264,8 +1258,7 @@ def _stop_at_limit(self, kind: str, limits: RunLimits, origin_run: Run, debugger logger.error("* Total execution time of %d sec(s) reached.", limits.total_budget.limit) limits.total_budget.exhausted = True self._cancel_execution_runs() - # A limit that expires while a fault is completing keeps the fault as - # the run's error. + # If the run already has an error, such as a fault, keep that error. if self.curr_run is origin_run and origin_run.error is None: origin_run.error = ErrorInfo(type=ErrorType(kind), pc=self.get_pc()) else: @@ -1290,11 +1283,11 @@ def _drop_stale_exit(self, address: int) -> int: if isinstance(native_exit, SuspendedApiCall): entry = native_exit.entry if address not in (entry.address, entry.trap): - # A debugger register edit abandons the suspended dispatch. + # The debugger changed PC, so the guest does not call the API. self._native_exit = None elif self._get_suspended_api_call(entry) != native_exit: - # Patches and frame edits while stopped must execute the revised - # public bytes instead of bypassing them through a retained trap. + # The debugger changed the stack or the code of the API. Run the + # code of the API again, so that the changes take effect. self._native_exit = None address = entry.address self.set_pc(address) @@ -1312,7 +1305,11 @@ def _take_pending_fault_stop(self) -> StopReason | None: return stop def _cancel_execution_runs(self, prepared=None): - """Cancel unexecuted work without leaving DLL attachments initializing.""" + """ + Remove the runs in the queue, the suspended runs and the run ``prepared``. + + Clear the ``INITIALIZING`` state of the DllMain and TLS callbacks of these runs. + """ canceled = [*self.run_queue, *self.suspended_runs] if prepared is not None: canceled.append(prepared) @@ -1324,10 +1321,9 @@ def _cancel_execution_runs(self, prepared=None): def resume(self, addr, count=-1) -> None: """ - Resume emulation directly at an address. + Continue the current run at addr, without the GDB server. - This low-level API bypasses the GDB command loop; callers that enable - GDB should drive execution through :meth:`start` instead. + With the GDB server, use ``start``. """ if self.curr_run is None: self.curr_run = Run() @@ -1337,7 +1333,7 @@ def resume(self, addr, count=-1) -> None: @execution_scope def start(self, addr=None, size=None): - """Begin emulation executing each run in the specified run queue.""" + """Add the runs of the startup initializers, then run all runs in the queue.""" total_budget = self._execution_budget assert total_budget is not None if total_budget.exhausted: @@ -1354,8 +1350,8 @@ def start(self, addr=None, size=None): self.set_hooks() self._unmap_reserved_range() - # Initialize run context/register state before exposing the target to GDB, - # so the first stop reports a meaningful PC/SP/etc. + # Prepare the first run before the GDB server starts, so that the first + # stop shows the registers of the run. self._prepare_run_context(run) completed = True @@ -1382,7 +1378,11 @@ def _execute_runs( debugger: GdbServer | None = None, debug_action: ResumeAction = ResumeAction(), ) -> bool: - """Execute prepared runs, optionally under control of an active GDB session.""" + """ + Run the current run and the runs in the queue, with or without a GDB session. + + Return False if the user or the exit event stopped the emulation. + """ total_budget = self._execution_budget assert total_budget is not None detached_resume_addr: int | None = None @@ -1471,8 +1471,8 @@ def _execute_runs( terminal_signal = GDB_SIGSEGV if debugger is not None: - # Ensure a pending Ctrl-C cannot be lost, then report the - # target fault while its register state is still available. + # finish_run keeps a pending Ctrl-C. Then report the fault + # while the registers of the fault are available. debugger.finish_run(debug_action) debug_action = debugger.command_loop( StopReason(signal=terminal_signal, kind="exception", address=self.get_pc()) @@ -1494,8 +1494,8 @@ def _execute_runs( if terminal_signal: debugger.notify_signal(terminal_signal) else: - # Stop once more while memory is still mapped, so the client - # can inspect the final state before the exit reply. + # Stop once more while the memory is mapped, so that the client + # can read the final state before the exit reply. debug_action = debugger.command_loop(StopReason(kind="exit")) if debug_action.kill: return True @@ -1507,7 +1507,7 @@ def _execute_runs( def _end_at_total_budget(self, debugger: GdbServer | None = None) -> None: """ - Cancel the runs that are left and end the emulation, because the total budget is spent. + Cancel the runs that are left and end the emulation, because the total time limit is reached. Args: debugger: the GDB session to send the exit reply to, if the client still waits for one. @@ -1537,22 +1537,22 @@ def get_dropped_files(self) -> list[File]: return self.fileman.get_dropped_files() def set_hooks(self): - """Reserves memory that will be used to handle events that occur during emulation.""" + """Add the hooks in the hook lists to Unicorn.""" super().set_hooks() def get_processes(self): - """Get the processes that exist in the emulation space.""" + """Get the process list. It does not include the child processes.""" return self.processes def kill_process(self, proc): - """Terminate a process (i.e. remove it from the known process list).""" + """Remove proc from the process list.""" try: self.processes.remove(proc) except ValueError: pass def get_current_thread(self): - """Get the current thread that is emulating.""" + """Get the current thread.""" return self.curr_thread def set_last_error(self, code): @@ -1566,7 +1566,7 @@ def get_last_error(self): return self.curr_thread.last_error def heap_alloc(self, size, heap="None"): - """Allocate a memory chunk and add it to the "heap".""" + """Map a block for a heap, and record it in the heap allocations. Return 0 if there is no free memory.""" try: addr = self.mem_map(size, base=None, tag=f"api.heap.{heap}") except OutOfMemoryError: @@ -1593,11 +1593,11 @@ def exit_process(self) -> None: raise WindowsEmuError("process exit needs user mode") def get_current_process(self): - """Get the current process that is emulating.""" + """Get the current process.""" return self.curr_process def set_current_process(self, process): - """Set the current process that is emulating.""" + """Set the current process.""" self.curr_process = process def set_current_thread(self, thread): @@ -1613,9 +1613,9 @@ def set_debug_registers(self, thread, ctx): def _setup_gdt(self, arch): """ - Set up the GDT so we can access segment registers correctly. + Map the GDT and the TEB segment, and load the segment registers. - The setup differs depending on architecture. + The guest reads the TEB through FS on x86 and through GS on x64. """ GDT_SIZE = 0x1000 SEG_SIZE = 0x1000 @@ -1625,10 +1625,7 @@ def _setup_gdt(self, arch): gs_addr = 0 gdt_addr = None - # For a detailed explaination of whats happening here, see: - # https://wiki.osdev.org/Global_Descriptor_Table - # We need to init the GDT so that shellcode can accurately access - # segment registers which is needed for TEB access in user mode + # The format of a descriptor: https://wiki.osdev.org/Global_Descriptor_Table def _make_entry(index, base, access, limit=0xFFFFF000): access = access | (winemu.GDT_ACCESS_BITS.PresentBit | winemu.GDT_ACCESS_BITS.DirectionConformingBit) @@ -1647,7 +1644,7 @@ def _create_selector(index, flags): return flags | (index << 3) # x86 user mode uses the selectors of native 32-bit Windows: CS 0x1B, DS and ES 0x23, FS 0x3B. - # SS stays a ring 0 selector because Unicorn runs the guest at CPL 0 and rejects an RPL 3 SS. + # SS is a ring 0 selector, because Unicorn runs the guest at CPL 0 and rejects an RPL 3 SS. native_x86 = arch == _arch.ARCH_X86 and not self.is_kernel_mode data_index, code_index, fs_index = (4, 3, 7) if native_x86 else (16, 17, 19) @@ -1677,7 +1674,7 @@ def _create_selector(index, flags): teb_addr = None if self.is_kernel_mode else winemu.TEB_ADDRESS if arch == _arch.ARCH_X86: - # FS segment needed for PEB access at fs:[0x30] + # The guest reads the TEB through FS, for example the PEB pointer at fs:[0x30] fs_addr, fs_size = self.get_valid_ranges(SEG_SIZE, addr=teb_addr) self.mem_map(fs_size, base=fs_addr, tag="emu.segment.fs") @@ -1688,7 +1685,7 @@ def _create_selector(index, flags): self.reg_write(_arch.X86_REG_FS, selector) elif arch == _arch.ARCH_AMD64: - # GS Segment needed for PEB access at gs:[0x60] + # The guest reads the TEB through GS, for example the PEB pointer at gs:[0x60] gs_addr, gs_size = self.get_valid_ranges(SEG_SIZE, addr=teb_addr) self.mem_map(gs_size, base=gs_addr, tag="emu.segment.gs") @@ -1704,7 +1701,7 @@ def _create_selector(index, flags): return fs_addr, gs_addr def init_peb(self, user_mods, proc=None): - """Initialize the Process Environment Block.""" + """Initialize the PEB of proc or of the current process. For the current process, write the PEB pointer.""" p = proc if not p: p = self.curr_process @@ -1714,7 +1711,7 @@ def init_peb(self, user_mods, proc=None): return p.peb def init_teb(self, thread, peb): - """Initialize the Thread Information Block.""" + """Initialize the TEB of thread at the base of the FS or GS segment.""" if self.get_arch() == _arch.ARCH_X86: thread.init_teb(self.fs_addr, peb.address) elif self.get_arch() == _arch.ARCH_AMD64: @@ -1722,9 +1719,9 @@ def init_teb(self, thread, peb): def init_tls(self, thread): """ - Initialize implicit thread local storage. + Copy the static TLS data of the module of the current run to thread. - Meant to be called after init_teb. + Call this after init_teb. """ run = self.curr_run if run is None: @@ -1750,9 +1747,9 @@ def init_tls(self, thread): def read_tls_template(self, tls_dirp: int) -> bytes: """ - Return the initial TLS data of a module, from its IMAGE_TLS_DIRECTORY. + Get the static TLS data of a module from its IMAGE_TLS_DIRECTORY at tls_dirp. - Set the TLS index of the module to 0. A directory with an invalid range gives an empty template. + Set the TLS index of the module to 0. A directory with an invalid or too large range gives empty data. """ ptrsz = self.get_ptr_size() fmt = "<4QI" if ptrsz == 8 else "<5I" @@ -1799,14 +1796,11 @@ def get_mod_from_addr(self, addr): def ensure_pe_import_hooks(self, base_addr): """ - Bind imports of an injected mapped PE using the public API registry. + Bind the imports of a PE that the guest mapped itself, for example by process hollowing. - Bind each valid IAT slot independently and skip invalid slots. - A zero OriginalFirstThunk may reuse an already bound IAT; recorded - bindings preserve idempotence without interpreting code addresses as RVAs. - - Intended for PEs injected via WriteProcessMemory (process hollowing) - that bypass the normal module loader. + Bind each valid IAT slot, and skip the invalid slots. If OriginalFirstThunk is zero, the IAT can have + addresses that this method bound in an earlier call. The emulator records its bindings, so that it skips + these slots and does not read their addresses as RVAs. """ ptr_size = self.get_ptr_size() import_errors = (ValueError, struct.error, uc.UcError, WindowsEmuError) @@ -1824,8 +1818,8 @@ def read(address, size): if bindings.get(slot.iat) == slot.current: continue if slot.iat in bindings and (not slot.has_ilt or slot.current != slot.thunk): - # The guest changed an entry that we bound. Keep the guest's value, - # unless the guest wrote back the original value from the ILT. + # The guest changed an entry that the emulator bound. Keep the value of + # the guest, unless the guest wrote back the original value from the ILT. continue if slot.is_bound_externally: continue @@ -1887,8 +1881,8 @@ def load_image(self, image, base: int | None = None, *, is_input: bool = False) """ Load an image and its dependencies. - Set is_input for the input image. The input image is the first module in modules, and only the input image - allocates the stack. + Set is_input for the input image. The input image is the first module in modules, and the emulator + allocates the stack only for the input image. """ return self.module_loader.load_image(image, base, is_input=is_input) @@ -1899,7 +1893,8 @@ def _load_image(self, image, base=None, *, is_input=False): An image with preferred base 0 loads at any free address. Raises: - ValueError: if the image must move, but it has no relocation data. + ValueError: if the image cannot move to the free base, for example because it has no relocation data. + WindowsEmuError: if the machine is not initialized, or the image does not match the machine. """ if not self.arch: raise WindowsEmuError("the machine is not initialized: call init_machine before loading images") @@ -1970,13 +1965,13 @@ def _load_image(self, image, base=None, *, is_input=False): end = (mod.base + section.virtual_address + section.virtual_size + self.page_size - 1) & ~( self.page_size - 1 ) - # PE sections can be smaller than a page and multiple sections can share one page. - # Merge permissions per page so a later tiny read-only section does not clobber - # earlier writable/executable permissions already required on that same page. + # Sections can be smaller than a page, and two sections can share a page. Merge the + # permissions of each page, so that a small read-only section does not remove the + # write or execute permission of an earlier section on the same page. for page in range(start, end, self.page_size): page_perms[page] = page_perms.get(page, 0) | section.perms - # Each unicorn mem_protect call splits a region, and the cost grows with the region - # count, so protect runs of contiguous same-permission pages with one call each. + # Each mem_protect call of Unicorn splits a region, and the cost of a call grows with the + # number of regions. So protect each range of adjacent pages with equal perms in one call. for start, length, perms in get_page_protection_runs(page_perms, self.page_size): self.mem_protect(start, length, perms) @@ -2090,9 +2085,10 @@ def new_object(self, otype): def create_process(self, path=None, cmdline=None, image=None, *, child=False, cd=""): """ - Create a process object that will exist in the emulator. + Create a process with one thread for the program in path, or in cmdline if path is empty. - An empty cd gives the process the current directory of the current process. + Add the process to child_processes if child is set, else to processes. An empty cd gives the process the + current directory of the current process. Raises: IsADirectoryError: if the program path is a directory. @@ -2132,7 +2128,6 @@ def create_process(self, path=None, cmdline=None, image=None, *, child=False, cd if cd: p.cd = cd - # Create a thread object for the new process t = om.new_object(objman.Thread) t.process = p @@ -2167,7 +2162,12 @@ def create_process(self, path=None, cmdline=None, image=None, *, child=False, cd return p def create_thread(self, addr, ctx, proc_obj, name=None, *, is_suspended=False): - """Create a thread object that will exist in the emulator.""" + """ + Create a thread of proc_obj and a run that starts at addr. + + Add the run to the queue, or to the suspended runs if is_suspended is set. Return the handle and the thread, + or (0, None) if the queue has max_runs runs. + """ om = self.get_object_manager("thread creation") if len(self.run_queue) >= self.max_runs: @@ -2192,11 +2192,10 @@ def create_thread(self, addr, ctx, proc_obj, name=None, *, is_suspended=False): thread.suspend_count = 1 self.suspended_runs.append(run) - # Returns handle return hnd, thread def resume_thread(self, thread): - """Resume a previously suspended thread.""" + """Move the suspended run of thread to the queue. Return False if thread has no suspended run.""" for r in self.suspended_runs: if r.thread == thread: _run = self.suspended_runs.pop(self.suspended_runs.index(r)) @@ -2209,12 +2208,10 @@ def get_process_peb(self, process): def add_callback(self, mod_name, func_name): """ - Adds a callback to the emulation callback list. - - A "callback" in this context refers to a function that is not imported statically or dynamically. + Get the address of an entry for the handler mod_name!func_name in the speakeasy_callbacks module. - For example, a pointer that is set in a function table - (e.g. PsSetCreateProcessNotifyRoutine). + The guest calls such a function through a pointer that a handler wrote, for example in a function table or a + vtable, not through an import. """ module = self.get_mod_by_name("speakeasy_callbacks") if module is None: @@ -2262,10 +2259,10 @@ def resolve_export(self, module, reference, *, allow_dynamic=False, _seen=None, return 0 hooks = self.get_api_hooks(module.name, str(reference)) eligible = allow_dynamic or self.config.modules.functions_always_exist or hooks - # Declared functions that the physical manifest lacks still resolve, - # because other Windows builds export them. + # A function in the signature database resolves even if the module does not + # export it, because other Windows builds export it. eligible = eligible or (isinstance(reference, str) and self.has_api_signature(module.name, reference)) - # Empty placeholder modules deliberately offer dynamic-only functions. + # A synthetic module without exports gives a dynamic entry for each name. eligible = eligible or (module._image.source == "synthetic" and not module.get_exports()) if not eligible or module._image.source != "synthetic" or module is getattr(self, "container_image", None): return 0 @@ -2308,7 +2305,7 @@ def resolve_export(self, module, reference, *, allow_dynamic=False, _seen=None, return entry.address def _get_export_directory(self, module): - """Read the export directory (RVA, size) from the mapped header, or None when it cannot be read.""" + """Read the (RVA, size) of the export directory from the mapped header, or None if the read fails.""" try: nt = module.base + struct.unpack(" str: return hosts[0] def get_proc(self, mod_name, func_name): - """Resolve an explicit import request to stable mapped code or storage.""" + """Get the address of func_name in the module that implements mod_name. Load the module if it is not loaded.""" host = self.get_import_host(mod_name, func_name) module = self.get_mod_by_name(host) if module is None: @@ -2342,6 +2339,11 @@ def get_proc(self, mod_name, func_name): return self.resolve_export(module, func_name, allow_dynamic=True) def _intercept_api_trap(self, access, address, size): + """ + Record the native exit of a fetch from a return address or an API token, and fault other token accesses. + + Return True if the access is one of these, so that the other hooks for invalid memory do not run. + """ if self.emu_eng.mem_access.get(access) == common.INVALID_MEM_EXEC: if address in (self.return_hook, self.exit_hook) and self.curr_run: self._record_exit(RunReturn()) @@ -2371,7 +2373,7 @@ def _raise_memory_fault(self, address, access_type): return False def raise_cpu_exception(self, exception, error_type: ErrorType, **details): - """From a Unicorn hook: stop Unicorn. The run loop then sends the exception to the guest.""" + """From a Unicorn hook: record the exception as a native exit and stop Unicorn. Then the run loop sends it.""" self._record_exit(RaisedCpuException(exception, error_type, **details)) def _dispatch_cpu_exception(self, raised: RaisedCpuException) -> None: @@ -2403,11 +2405,11 @@ def _can_raise_api_fault(self): def _raise_api_fault(self, frame, access_type, address): """ - Raise an access violation in the caller of an API whose handler touched an invalid guest pointer. + Raise an access violation in the caller of an API, because the handler of the API accessed a bad pointer. - The access violation is the one the API's own code would raise. + On Windows, the code of the API raises this access violation. - Return False when no handler looks at it. + Return False if the dispatch fails before it calls guest code. """ params = (cpu_exceptions.ACCESS_CODES[access_type], address) try: @@ -2420,7 +2422,7 @@ def _raise_api_fault(self, frame, access_type, address): return True def set_guard_pages(self, address, size, perms): - """Map pages without access until their first access, which restores perms and raises an exception.""" + """Remove all access from the pages. The first access restores perms and raises a guard page exception.""" base = address & ~(self.page_size - 1) end = (address + size + self.page_size - 1) & ~(self.page_size - 1) for page in range(base, end, self.page_size): @@ -2449,21 +2451,22 @@ def _raise_protection_fault(self, address, access_type): return False def _step_had_no_effect(self, registers, debugger): - """Get whether a single step stopped before its instruction with nothing pending.""" + """Get whether a single step did not change the registers, and no stop or exit is pending.""" if debugger.has_pending_stop() or self._native_exit is not None: return False return self._pending_fault_stop is None and self.get_register_state() == registers def _stopped_at_hlt(self, deadline, last_executed, counted, debugger): """ - Unicorn ends emulation without an error after `hlt`, which user mode cannot execute. + Get whether Unicorn stopped because the guest ran ``hlt``, which user mode cannot run. - Unicorn also stops that way for other reasons, so only report a `hlt` that surely ran: the last instruction - the accounting hook saw or a debugger step ran, or with nothing else that stops the engine. + Unicorn stops without an error after ``hlt``, and also for other reasons. So report a ``hlt`` only if it ran: + it is the last instruction that the accounting hook counted or that a debugger step ran, or nothing else + stopped Unicorn. - Nothing better reports `hlt`: Unicorn hooks only `in`, `out`, `syscall`, `sysenter` - and `cpuid` among x86 instructions, `hlt` raises no interrupt, and a code hook on - every instruction would slow all emulation down. + Unicorn has no better way to report ``hlt``. Of the x86 instructions, Unicorn hooks only ``in``, ``out``, + ``syscall``, ``sysenter`` and ``cpuid``. ``hlt`` raises no interrupt, and a code hook on each instruction + makes all emulation slower. """ if self.is_kernel_mode or not self.config.exceptions.dispatch_handlers: return False @@ -2515,12 +2518,12 @@ def _get_suspended_api_call(self, entry: ApiEntry) -> SuspendedApiCall: return SuspendedApiCall(entry, sp, return_address, code) def _handle_invalid_fetch(self, emu, address, size, value): - """Called when an attempt to emulate an instruction from an invalid address.""" + """Handle a fetch from unmapped memory.""" if address == self.return_hook or address == self.exit_hook: self._map_reserved_range() return True - # The rest of an instruction that crosses into the next page; the run already ends at its start + # The fetch is the rest of an instruction that crosses into the next page. The fault at its start ends the run. if isinstance(self._native_exit, FaultReturn) or self._pending_fault_stop is not None: return False @@ -2551,7 +2554,7 @@ def _resolve_region_info(self, addr: int) -> RegionInfo | None: return None def _find_nearby_regions(self, addr: int, count: int = 2) -> list[RegionInfo]: - """Return up to `count` nearest memory regions to an unmapped address.""" + """Return up to ``count`` memory regions nearest to an unmapped address.""" maps = self.get_mem_maps() if not maps: return [] @@ -2580,10 +2583,9 @@ def _build_context_summary( nearby_regions: list[RegionInfo] | None, ) -> str: """ - Build a human-readable one-line triage summary. - - Examples:: + Build a one-line summary of an error for the report. + Examples: read of unmapped 0x12345678 from sample.exe+0x8d14; nearest: heap [0x12340000-0x12340fff] write of 0x401500 from sample.exe+0x2000; in .text [0x401000-0x405fff] fetch at 0xdeadbeef from pc=0xdeadbeef @@ -2616,11 +2618,11 @@ def get_error_info( message: str | None = None, ) -> ErrorInfo: """ - Collect emulator state information in the event of an error. + Get the state of the emulator for an error. - For an error at an API trap, give the API name and its return address - as the address, because the trap holds no instruction to show. - For an internal_error, give the text of the Python exception as message. + For an error at an API call, give the API name, and the return address + as address, because the token has no instruction to show. + For an ``internal_error``, give the text of the Python exception as message. """ run = self.get_current_run() pc = self.get_pc() @@ -2714,7 +2716,7 @@ def format_api_arg(arg: ApiArg) -> str: return f"{arg.name}: {text}" if arg.name is not None else text def log_api(self, pc: int, imp_api: str, rv: int | None, args: list[ApiArg], *, run=None) -> None: - """Log an API call and record it with the profiler.""" + """Log an API call and record it in the report.""" call_str = f"{imp_api}({', '.join(self.format_api_arg(arg) for arg in args)})" rv = self.get_machine_word(rv) @@ -2745,7 +2747,12 @@ def _get_signature_arch(self) -> str: return sigdb.ARCH_X86 if self.get_arch() == _arch.ARCH_X86 else sigdb.ARCH_X64 def _lookup_api_declaration(self, dll: str, name: str) -> sigdb.FuncSig | None: - """Find the authoritative declaration without choosing an execution ABI.""" + """ + Get the declaration of dll!name, also a declaration that lookup_api_signature rejects. + + Raises: + WindowsEmuError: if the signature database fails. + """ db = self.get_signature_db() arch = self._get_signature_arch() try: @@ -2761,7 +2768,7 @@ def _lookup_api_declaration(self, dll: str, name: str) -> sigdb.FuncSig | None: return sig def _lookup_api_set_declaration(self, dll: str, name: str) -> sigdb.FuncSig | None: - """Find a declaration that win32metadata files under an API set that maps to ``dll``.""" + """Get a declaration that the signature database has under an API set whose host is ``dll``.""" db = self.get_signature_db() arch = self._get_signature_arch() sig = db.lookup(dll, name, arch) @@ -2779,8 +2786,8 @@ def lookup_api_signature(self, dll: str, name: str) -> sigdb.FuncSig | None: return sig def _can_stub_unknown_api(self, dll: str, name: str) -> bool: - # A known unsupported declaration is not unknown and still requires a - # handler. + # A function with a declaration that the emulator cannot use is not + # unknown, so it does not get a stub. return self.config.modules.functions_always_exist and self._lookup_api_declaration(dll, name) is None def has_api_signature(self, dll: str, name: str) -> bool: @@ -2788,10 +2795,10 @@ def has_api_signature(self, dll: str, name: str) -> bool: def get_handler_signature(self, dll: str, name: str, argc: int) -> sigdb.FuncSig | None: """ - Find the signature that names the arguments of a call served by a handler. + Get the signature that names the arguments of a call to a handler. - Returns None when the function is unknown, variadic, or its declaration does not consume exactly the ``argc`` - slots the handler reads. + Return None if the function is unknown or variadic, or if its declaration does not use exactly the ``argc`` + slots that the handler reads. """ try: sig = self.get_signature_db().lookup(dll, name, self._get_signature_arch()) @@ -2804,7 +2811,7 @@ def get_handler_signature(self, dll: str, name: str, argc: int) -> sigdb.FuncSig def get_signature_formatter(self) -> sigfmt.ArgFormatter: """ - Get the formatter that renders arguments of signature-emulated calls for the API trace. + Get the formatter that renders API arguments from their signatures for the report. The formatter renders strings, enums, flags and struct contents. """ @@ -2829,7 +2836,7 @@ def _render_signature_args(self, sig: sigdb.FuncSig, argv: list[int]) -> list[si values = sig.values_from_slots(argv, self.get_ptr_size()) return [self._render_signature_arg(param, value, i) for i, (param, value) in enumerate(zip(sig.params, values))] - # Upper bound on how much memory a single Out parameter is zero-filled with + # The maximum number of bytes that the emulator zeroes for one Out parameter MAX_OUT_ZERO_FILL = 0x10000 def _read_uint_for_signature(self, addr: int, size: int) -> int | None: @@ -2840,10 +2847,10 @@ def _read_uint_for_signature(self, addr: int, size: int) -> int | None: def _zero_fill_out_params(self, sig: sigdb.FuncSig, values: list[int], ptr_size: int) -> None: """ - Give Out-only pointer parameters deterministic contents. + Zero the buffers of the Out parameters that are not also In parameters. - A call we only know the signature of reports success without producing any data, so the memory the caller - reads back is zeroed (empty strings, NULL handles, zero counts) rather than left as uninitialized stack. + A call from a signature returns success but writes no data. So the caller reads zeros, such as empty strings, + NULL handles and zero counts, and not old stack data. """ db = self.get_signature_db() for index, (param, value) in enumerate(zip(sig.params, values)): @@ -2861,7 +2868,7 @@ def _zero_fill_out_params(self, sig: sigdb.FuncSig, values: list[int], ptr_size: ) def _default_return_for_signature(self, sig: sigdb.FuncSig) -> int | None: - """Pick a plausible "success" return value for a call we only know the signature of.""" + """Get a success value for the return type of a call from a signature.""" kind = sig.ret_kind if kind == "v": return None @@ -2871,16 +2878,16 @@ def _default_return_for_signature(self, sig: sigdb.FuncSig) -> int | None: if self.om is not None: return self.om.allocate_handle() return 4 - # Status codes (HRESULT, NTSTATUS, WIN32_ERROR), counts and pointers all - # read as "nothing happened" at zero. + # For status codes (HRESULT, NTSTATUS, WIN32_ERROR), 0 is success. For + # counts and pointers, 0 means no data. return 0 def _handle_api_from_signature(self, dll: str, name: str, sig: sigdb.FuncSig, call_pc: int) -> int | None: """ - Emulate an import that has no handler using only its declared signature. + Run an API that has no handler from its signature. - Consume the right number of argument slots, log the call with decoded arguments, return a - type-appropriate success value and clean up the stack according to the calling convention. + Read the arguments, log the call, zero the Out buffers, and return a success value for the return type. Remove + the arguments from the stack as the calling convention requires. """ imp_api = f"{dll}.{name}" ptr_size = self.get_ptr_size() @@ -2976,7 +2983,7 @@ def dispatch_api_call(self, dll: str, name: str) -> None: logger.error("API handler %s.%s changed SP without returning", dll, name) self.end_run_on_fault() except uc.UcError as e: - # Reading the arguments or writing the results of the call touched unmapped memory + # The API code accessed unmapped memory when it read the arguments or wrote the results kind, address, access_type, _ = guest_callbacks.get_handler_fault(self, e, entry_pc) if access_type is None or self._should_stop_on_faults: raise @@ -3043,8 +3050,7 @@ def _handle_api_with_handler(self, call: _ApiCallState, impl: api_resolution.Han if hooks: hooked_func = MethodType(impl.func, impl.module) orig = lambda args: hooked_func(self, args, ctx) # noqa - # Hooks execute in FIFO order (first registered, first called). - # All hooks run; the last hook's return value is used. + # All hooks run in the order of get_api_hooks. The return value of the last hook is the result. prev_ctx, self.api_ctx = self.api_ctx, ctx try: for hook in hooks: @@ -3056,8 +3062,7 @@ def _handle_api_with_handler(self, call: _ApiCallState, impl: api_resolution.Han rv = self.api.call_api_func(impl.module, impl.func, argv, ctx=ctx) except Exception as e: if self._should_stop_on_faults: - # Let the outer GDB fault path expose the failed frame - # before run cleanup and report signal termination. + # _execute_runs reports the fault to the debugger before it ends the run. raise desc, address, access_type, message = guest_callbacks.get_handler_fault(self, e, self.get_pc()) if access_type and self._can_raise_api_fault(): @@ -3083,7 +3088,7 @@ def _handle_api_with_handler(self, call: _ApiCallState, impl: api_resolution.Han frame.result = rv mm = self.get_address_map(ret) - # Is this function being called from a dynamcially allocated memory segment? + # The return address is in memory that VirtualAlloc allocated. if mm and mm.tag and "virtualalloc" in mm.tag.lower(): self._fire_dyn_code_hooks(ret) @@ -3098,7 +3103,6 @@ def _handle_api_with_handler(self, call: _ApiCallState, impl: api_resolution.Han ): self.do_call_return(impl.argc, ret, rv, conv=impl.conv) - # Deferred lifecycle/SEH work is processed before guest execution resumes. if not self.is_run_complete: self.enable_deferred_work_hook() return True @@ -3107,8 +3111,8 @@ def _handle_api_hook(self, call: _ApiCallState, hooks: list[common.ApiHook]) -> """ Run the user hook of an API that has no handler. - The hook gives the argument count and the calling convention. Only the last hook that was added runs, because - the hooks can give different values. + The hook gives the argument count and the calling convention. The hooks can give different values, so only the + last hook from get_api_hooks runs. """ if len(hooks) > 1: logger.warning( @@ -3155,7 +3159,7 @@ def _handle_api_stub(self, call: _ApiCallState) -> None: self.do_call_return(argc, call.return_address, 1, conv=conv) def _handle_unsupported_api(self, call: _ApiCallState) -> None: - """End the run with an unsupported_api error.""" + """End the run with an ``unsupported_api`` error.""" error = self.get_error_info(ErrorType.UNSUPPORTED_API, call.return_address, api_name=call.api_name) logger.error("Unsupported API: %s (ret: 0x%x)", call.api_name, call.return_address) self.log_api(call.call_pc, call.api_name, None, [], run=call.run) @@ -3176,7 +3180,7 @@ def _check_api_limit(self, origin_run, imp_api): self.on_run_complete() def start_guest_callback(self, frame, function, args, sp=None): - """Enter a guest callback on the stack of its frame, or below sp.""" + """Call function with args as a guest callback of frame, on the stack of frame or below sp.""" if frame.saved_registers is None: registers = NONVOLATILE_AMD64 if self.get_arch() == _arch.ARCH_AMD64 else NONVOLATILE_X86 frame.saved_registers = tuple((register, self.reg_read(register)) for register in registers) @@ -3184,7 +3188,7 @@ def start_guest_callback(self, frame, function, args, sp=None): sp = frame.stack_pointer if sp is None else sp if self.ptr_size == 8: # Win64 callees expect RSP + 8 to be 16-byte aligned at entry, and - # each argument after the fourth occupies one 8-byte stack slot. + # each argument after the fourth uses one 8-byte stack slot. sp &= ~0xF if max(len(args) - 4, 0) % 2: sp -= 8 @@ -3192,9 +3196,13 @@ def start_guest_callback(self, frame, function, args, sp=None): self.set_pc(function) def _continue_guest_callback(self): - """Resume a typed guest callback continuation outside Unicorn callbacks.""" + """ + Continue the API handler after one of its guest callbacks returned. + + Resume the handler generator, start the next pending callback of the frame, or return from the API. + """ run = self.get_current_run() - # Frames below the one returning were left without returning + # Drop the frames of nested callbacks that the guest left without a return while len(run.guest_callbacks) > 1 and run.guest_callbacks[-1].stack_pointer < self.get_stack_ptr(): run.guest_callbacks.pop() frame = run.guest_callbacks[-1] @@ -3229,12 +3237,12 @@ def _fail_library_initializer(self, frame, routine): self.module_loader.rollback_guest_load(frame) def restore_caller_registers(self, frame): - """Restore the nonvolatile registers the guest calls of frame may have changed, as Windows keeps them.""" + """Restore the nonvolatile registers that the guest callbacks of frame can change. Windows keeps them.""" for register, value in frame.saved_registers or (): self.reg_write(register, value) def _hook_mem_unmapped(self, emu, access, address, size, value): - """High level function used to catch all invalid memory accesses that occur during emulation.""" + """Handle an access to unmapped memory, or an access that the page protection denies.""" try: access = self.emu_eng.mem_access.get(access) logger.debug("mem_unmapped: access=%s addr=0x%x size=0x%x", access, address, size) @@ -3295,7 +3303,7 @@ def _unprotect_resource_write(self, address: int) -> bool: return True def get_symbol_from_address(self, address) -> str | None: - """If the supplied address is related to a known symbol, look it up here.""" + """Get the symbol at address as "dll.name", or None.""" symbol = self.api_registry.symbol(address) if symbol is not None: return symbol @@ -3306,10 +3314,10 @@ def get_symbol_from_address(self, address) -> str | None: def get_symbols(self) -> dict[int, tuple[str, str]]: """ - Snapshot public addresses as legacy (dll, name) tuples. + Get a copy of the symbols as {address: (dll, name)}. - Registry entries take precedence over auxiliary labels at the same address. - Forwarder strings and private dispatch tokens are not public symbols. + If two symbols have the same address, the API registry entry wins. + The result does not include forwarder strings or API tokens. """ symbols = { address: value @@ -3327,7 +3335,7 @@ def get_symbols(self) -> dict[int, tuple[str, str]]: return symbols def get_api_symbols(self) -> dict[int, str]: - """Snapshot public function/data and auxiliary symbols as string labels.""" + """Get a copy of the symbols of get_symbols as {address: "dll.name"}.""" return {address: "{}.{}".format(*value) for address, value in self.get_symbols().items()} def _set_run_error(self, error: ErrorInfo) -> None: @@ -3336,7 +3344,7 @@ def _set_run_error(self, error: ErrorInfo) -> None: self.curr_run.error = error def _end_run_on_hook_error(self, description: str, exception: Exception) -> bool: - """Log the current exception, save it as the run error, and stop emulation.""" + """Log the current exception, save it as the error of the run, and end the emulation.""" logger.exception(description) error = self.get_error_info( ErrorType.INTERNAL_ERROR, @@ -3388,11 +3396,7 @@ def _get_region_access(self, address: int) -> MemAccess | None: return run.mem_access.setdefault(mmap, MemAccess(base=mmap.base, size=mmap.size)) def _hook_mem_read(self, emu, access, address, size, value): - """ - Hook each memory read event that occurs. - - This hook is used to lookup symbols and modules that are read from during emulation. - """ + """Count the reads of each symbol, section and memory map.""" try: symbol = self.get_symbol_from_address(address) if symbol: @@ -3410,11 +3414,7 @@ def _hook_mem_read(self, emu, access, address, size, value): return self._end_run_on_hook_error("Exception during memory read", e) def _hook_mem_write(self, emu, access, address, size, value): - """ - Hook each memory write event that occurs. - - This hook is used to track memory modifications to interesting memory locations. - """ + """Count the writes of each symbol, section and memory map.""" try: symbol = self.get_symbol_from_address(address) if symbol: @@ -3432,7 +3432,7 @@ def _hook_mem_write(self, emu, access, address, size, value): return self._end_run_on_hook_error("Exception during memory write", e) def _handle_invalid_read(self, emu, address, size, value): - """Hook each invalid memory read event that occurs.""" + """Handle a read from unmapped memory.""" mod = self.get_mod_from_addr(address) if mod: return True @@ -3443,10 +3443,9 @@ def _handle_invalid_read(self, emu, address, size, value): return self._raise_memory_fault(address, "read") def _handle_prot_fetch(self, emu, address, size, value): - """Called when non-executable code is emulated.""" - # Ordinary analysis recovers execution in non-X guest sections. - # Synthetic API protections and debugger stops are always - # authoritative; a symbol never causes dispatch from this hook. + """Handle a fetch from memory without execute permission.""" + # Without a debugger, the guest can run code in its own memory without execute permission. + # A fetch from a synthetic module or the token range, or any fetch with a debugger, is a fault. module = self.get_mod_from_addr(address) if ( not self._should_stop_on_faults @@ -3454,8 +3453,8 @@ def _handle_prot_fetch(self, emu, address, size, value): and not self.api_registry.overlaps_traps(address) ): if module is not None and module._image.source == "guest_pe": - # A change of protection while Unicorn handles this fetch can - # make its current translation incorrect. Change it after Unicorn stops. + # A change of protection while Unicorn handles this fetch can make + # the current translation incorrect. Change the protection after Unicorn stops. native_perms = next(perms for start, end, perms in self.get_mem_regions() if start <= address <= end) perms = next(perms for perms, native in self.emu_eng.perms.items() if native == native_perms) page = address & ~(self.page_size - 1) @@ -3468,8 +3467,7 @@ def _handle_prot_fetch(self, emu, address, size, value): return False def _handle_invalid_write(self, emu, address, size, value): - """Called when non-writable address is written to.""" - # ignore patches to APIs + """Handle a write to unmapped memory.""" if winemu.EMU_RESERVED <= address < winemu.EMU_RESERVED_END: return self._complete_reserved_access(address, size, "write") @@ -3488,15 +3486,15 @@ def _hook_code_deferred_work(self, emu, addr, size): """ Code hook that runs at the next instruction after an API call, a fault or a request to end the run. - It ends the run if it is complete and unmaps the reserved range. It disables itself when it has no more work. + End the run if it is complete, and unmap the reserved range. Disable the hook when there is no more work. """ try: if addr == self.return_hook or self.is_run_complete: self.on_run_complete() return False - # Handler instructions are not progress at the original fault site. - # After continuation, retain the guard until the guest advances. + # A guest exception handler continued at the fault address. Clear the repeat count of the fault + # when PC leaves that address. if self._seh_resume_pc is not None and addr != self._seh_resume_pc: self._seh_last_fault = None self._seh_repeat_count = 0 @@ -3510,7 +3508,7 @@ def _hook_code_deferred_work(self, emu, addr, size): return self._end_run_on_hook_error("Exception during code hook (deferred work)", e) def _hook_code_coverage(self, emu, addr, size): - """Persistent code hook that records every executed address for coverage.""" + """Record the address of each instruction that runs, for coverage.""" try: if self.curr_run is not None: self.curr_run.coverage.add(addr) @@ -3519,11 +3517,7 @@ def _hook_code_coverage(self, emu, addr, size): return self._end_run_on_hook_error("Exception during code hook (coverage)", e) def _hook_code_tracing(self, emu, addr, size): - """ - Persistent code hook for memory tracing. - - The hook counts instructions and tracks symbol execution and per-region execution. - """ + """Count the instructions, and the executes of each symbol, section and memory map.""" try: if logger.isEnabledFor(logging.DEBUG): disasm = self.get_disasm(addr, size)[2] @@ -3550,11 +3544,7 @@ def _hook_code_tracing(self, emu, addr, size): return self._end_run_on_hook_error("Exception during code hook (tracing)", e) def _hook_code_debug(self, emu, addr, size): - """ - Log disassembly and register state for every instruction when debug mode is enabled. - - This is a persistent code hook. - """ + """Log the disassembly and the registers of each instruction in debug mode.""" x = self.get_disasm(addr, size)[2] if self.get_arch() == _arch.ARCH_AMD64: regs: tuple[str, ...] = ("rax", "rbx", "rcx", "rdx", "rsi", "rdi", "rbp", "rsp", "r8", "r9") @@ -3624,7 +3614,11 @@ def load_library(self, mod_name): return module.base def load_module_by_name(self, name, emu_path=None, base=None, native_path=None, data=None): - """Load one coherent native or catalog-generated module instance.""" + """ + Load the module name, or get it if it is loaded. + + Load the module from data, else from the module directory, else as a synthetic API module. + """ name = module_name(name) existing = self.get_mod_by_name(name) if existing is not None: @@ -3661,11 +3655,8 @@ def load_module_by_name(self, name, emu_path=None, base=None, native_path=None, self.module_loader.guest_dependencies.append(module) return module - # This will create a module from a file inside Speakeasy's - # object manager. file_path is expected to point to a valid PE - # file, like it would on a real Windows machine - # Returns: raw data that represents a PE file def get_module_data_from_emu_file(self, file_path): + """Get the contents of the file at file_path in the emulated file system, or None if there is no file.""" if not self.does_file_exist(file_path): return None @@ -3674,9 +3665,7 @@ def get_module_data_from_emu_file(self, file_path): if not mod_file: return None - # This file could have been read from, so don't mess - # with its file offset pointer. Just get the raw bytes - # from the BytesIO object + # get_contents keeps the file pointer of the guest. return mod_file.get_contents() def init_sys_modules(self, modules_config): @@ -3695,7 +3684,7 @@ def _init_module_group(self, modules_config, default_base=None): return rtmods def get_thread_context(self, thread=None): - """Get the current thread CPU context.""" + """Get the saved CONTEXT of thread, or the CONTEXT of the CPU if thread is None.""" if thread: return thread.get_context() if self.get_arch() == _arch.ARCH_AMD64: @@ -3714,7 +3703,7 @@ def get_thread_context(self, thread=None): return ctx def load_thread_context(self, ctx, thread=None): - """Set the current thread CPU context.""" + """Write ctx to the CPU registers, and its debug registers to the current thread.""" if self.get_arch() == _arch.ARCH_AMD64: registers = thread_context.CPU_REGISTERS_AMD64 else: @@ -3769,7 +3758,12 @@ def record_exception(self, code, address, handler, registers, faulting_address=N _SEH_MAX_REPEAT = 4 def dispatch_seh(self, faulting_address, exception, error): - """Dispatch exception to the guest handlers. Return whether a handler takes it.""" + """ + Send exception to the guest handlers with SEH dispatch. Return True if a handler takes it. + + Return False if the same fault occurs ``_SEH_MAX_REPEAT`` times in a row, so that a handler that does not fix + the fault cannot make an infinite loop. + """ self._seh_resume_pc = None fault_key = (self.get_pc(), faulting_address) if fault_key == self._seh_last_fault: @@ -3784,19 +3778,19 @@ def dispatch_seh(self, faulting_address, exception, error): @property def active_api_frame(self) -> GuestCallbackFrame | None: - """The frame of the API call being handled, if any.""" + """The frame of the API call that the emulator runs, or None.""" return self._active_api_frame def exception_dispatcher(self): - """The exception dispatch of the architecture.""" + """Get the SEH dispatch module of the architecture.""" return seh64 if self.get_arch() == _arch.ARCH_AMD64 else seh32 def note_seh_resume(self, pc): """ - A handler continues execution at pc. + Record that a guest exception handler continues execution at pc. - Resuming elsewhere than the fault is progress; at the fault, the guard against repeating it holds until - execution moves on. + If pc is not the address of the last fault, clear the repeat count. If pc is the address of the fault, keep the + count until the deferred work hook sees PC move away. """ if self._seh_last_fault is None: return @@ -3810,9 +3804,9 @@ def note_seh_resume(self, pc): def _new_object_handle(self, otype, name): """ - Create an object that only its handles keep. + Create an object of type otype with one handle. - The handle takes over the reference of the object manager, so the last close removes the object. + The handle takes the reference of the object manager, so the last close of a handle removes the object. """ om = self.get_object_manager("object creation") obj = om.new_object(otype) @@ -3826,7 +3820,7 @@ def create_event(self, name=""): return self._new_object_handle(objman.Event, name) def dec_ref(self, obj): - """Dereference an object.""" + """Decrement the reference count of obj.""" om = self.get_object_manager("object dereference") return om.dec_ref(obj) @@ -3841,7 +3835,7 @@ def create_semaphore(self, name=""): return self._new_object_handle(objman.Semaphore, name) def _hook_interrupt(self, emu, intnum): - """Called when software interrupts occur.""" + """Handle a software interrupt.""" self.emu_eng.clear_exception_state() if ( not self.is_kernel_mode @@ -3861,10 +3855,10 @@ def _hook_interrupt(self, emu, intnum): self.raise_cpu_exception(exception, ErrorType.UNHANDLED_INTERRUPT, interrupt_num=intnum) return True - # Handle __fastfail interrupt introduced in Windows 8 + # int 0x29 is __fastfail (Windows 8 and later) if intnum == 0x29: ecx = self.reg_read(_arch.X86_REG_ECX) - # Cookie security init failed, just return since we are in __security_init_cookie + # __security_init_cookie raises FAST_FAIL_GS_COOKIE_INIT (6). Return to its caller. if ecx == 6: def _tmp_hook(emu, addr, size): diff --git a/speakeasy/winenv/api/api.py b/speakeasy/winenv/api/api.py index 3afca9f61..ab93ea299 100644 --- a/speakeasy/winenv/api/api.py +++ b/speakeasy/winenv/api/api.py @@ -40,8 +40,9 @@ def get_format_slot_sizes(spec: re.Match, ptr_size: int) -> list[int]: """ - Get the size in bytes of each argument that one printf conversion reads, - in order: the width, the precision and then the value. + Get the size in bytes of each argument that one printf conversion reads. + + The order is the width, the precision, then the value. """ sizes = [ptr_size] * [spec["width"], spec["precision"]].count("*") conv, length = spec["conv"], spec["length"] @@ -57,8 +58,9 @@ def get_format_slot_sizes(spec: re.Match, ptr_size: int) -> list[int]: class HandlerArg: """ - One argument of a handled call, as recorded in the report. A handler sets - ``display`` to show a value it decoded, such as the path behind a handle. + One argument of an API call, as the report shows it. + + A handler sets ``display`` to show a value that it decoded, such as the path of a handle. """ def __init__(self, arg: ApiArg, param: sigdb.ParamSig | None = None) -> None: @@ -84,12 +86,13 @@ def display(self) -> str: @display.setter def display(self, text: object) -> None: """ - Show ``text`` for this argument. The type becomes ``str`` on a - parameter declared as a string, and ``text`` otherwise. A parameter - that the signature renders as an enum or flags keeps that rendering, - so all such values have one format. A value that is not a string, - such as ``None`` from a failed lookup or a NULL pointer the handler - did not decode, keeps the current display. + Show ``text`` for this argument. + + The type changes to ``str`` for a parameter that the signature declares as a + string, else to ``text``. A parameter that the signature shows as an enum or + flags keeps that display, so that all these values have one format. If ``text`` + is not a string, for example ``None`` from a failed lookup or a NULL pointer + that the handler did not decode, the display does not change. """ if not isinstance(text, str) or ( self._param is not None and self._arg.type in (ApiArgType.ENUM, ApiArgType.FLAGS) @@ -107,11 +110,13 @@ def _detached_arg() -> HandlerArg: class HandlerArgs: """ - The arguments of a handled call. With a usable signature there is one - entry per parameter, found by parameter name or index. Without one there - is one entry per argument slot, found by slot index. A name or index that - does not match an entry gives a detached argument whose changes are not - recorded, so a missing signature never stops emulation. + The arguments of an API call. + + If the emulator can use the signature of the API, there is one entry for each + parameter, and a handler gets an entry by parameter name or index. Else there is + one entry for each argument slot, and a handler gets an entry by slot index. A + name or index without an entry gives a detached argument. The report does not + record changes to a detached argument, so a missing signature does not stop emulation. """ def __init__(self, args: list[ApiArg], params: list[sigdb.ParamSig] | None = None, discard: bool = False) -> None: @@ -124,8 +129,10 @@ def from_signature( cls, sig: sigdb.FuncSig, ptr_size: int, rendered: list[sigfmt.RenderedArg], slots: list[int] ) -> HandlerArgs: """ + Make the arguments from the rendered values of the signature parameters. + Raises: - ValueError: if the slots do not match the slots the signature consumes. + ValueError: if the slots do not match the slots that the signature uses. """ return cls(sigfmt.get_call_args(sig, ptr_size, rendered, slots), list(sig.params)) @@ -176,9 +183,7 @@ class ApiContext: class ApiHandler: - """ - Base class for handling exported functions - """ + """Base class of the API handlers of one emulated DLL.""" name = "" @@ -297,20 +302,14 @@ def heap_alloc(self, size, heap): return self.emu.heap_alloc(size, heap) def heap_size(self, addr): - """ - Get the requested size of a block that heap_alloc returned, or None - for any other address. - """ + """Get the requested size of a block that ``heap_alloc`` returned, or None for any other address.""" mm = self.emu.get_address_map(addr) if not mm or mm.free or mm.base != addr or not (mm.tag or "").startswith("api.heap"): return None return next((n for a, n, _ in reversed(self.emu.get_heap_allocs()) if a == addr), mm.size) def heap_free(self, addr): - """ - Free a block that heap_alloc returned. Return False, and free nothing, - for any other address. - """ + """Free a block that ``heap_alloc`` returned. Return False, and free nothing, for any other address.""" if self.heap_size(addr) is None: return False self.emu.mem_free(addr) @@ -323,9 +322,7 @@ def mem_free(self, addr): return self.emu.mem_free(addr) def win_perms_to_emu_perms(self, win_perms): - """ - Get the emulator permissions for a Windows page protection. - """ + """Get the emulator permissions for a Windows page protection.""" new = 0 if win_perms & (PAGE_PROTECTION_FLAGS.PAGE_EXECUTE_READWRITE | PAGE_PROTECTION_FLAGS.PAGE_EXECUTE_WRITECOPY): new = common.PERM_MEM_RWX @@ -373,9 +370,7 @@ def normalize_res_identifier(self, emu, cw, val): return name def read_res_identifier(self, emu, ctx, arg, cw, val): - """ - Read a resource name or integer ID (MAKEINTRESOURCE) and show it as the display of ``arg``. - """ + """Read a resource name or integer ID (MAKEINTRESOURCE) and show it as the display of ``arg``.""" name = self.normalize_res_identifier(emu, cw, val) ctx.args[arg].display = name if isinstance(name, str) else hex(name) return name @@ -440,7 +435,6 @@ def record_process_event(self, proc, event_type: EventType, **kwargs): profiler.record_process_event(run, pos, proc, event_type, kwargs) def get_max_int(self): - # Byte order is irrelevant here return int.from_bytes(b"\xff" * self.get_ptr_size(), "little") def mem_read(self, addr, size): @@ -499,9 +493,8 @@ def decode_string(self, data: bytes, width: int) -> str: raise ApiEmuError(f"No encoding found for char width: {width}") def mem_write(self, addr, data): - # If the data being written to a shared memory mapping, update all mappings - # This will likely have to be made more robust to handle more complicated - # scenarios with varying file offsets + # Write the data to all views of a shared file mapping, at the same offset from the + # base of each view. This is wrong for views at different file offsets. mm = self.emu.get_address_map(addr) if mm and mm.shared: fm = self.emu.get_file_manager() @@ -522,9 +515,7 @@ def read_ptr(self, addr: int) -> int: return int.from_bytes(self.mem_read(addr, self.ptr_size), "little") def write_dword(self, addr: int, value: int) -> None: - """ - Write the low 32 bits of value. A negative value becomes its two's complement. - """ + """Write the low 32 bits of ``value``. A negative value becomes its two's complement.""" self.mem_write(addr, (value & 0xFFFFFFFF).to_bytes(4, "little")) def write_qword(self, addr: int, value: int) -> None: @@ -553,26 +544,22 @@ def exit_process(self): def get_char_width(self, ctx): """ - Based on the API name, determine the character width - being used by the function. An export without the A or W - suffix, such as lstrcpy, is ANSI. + Get the character width of the API from its name: 2 for a W suffix, else 1. + + An export without the A or W suffix, such as lstrcpy, is ANSI. """ if ctx.func_name.endswith("W"): return 2 return 1 def get_va_arg_count(self, fmt): - """ - Get the number of pointer-sized argument slots that a printf format reads - """ + """Get the number of pointer-sized argument slots that a printf format reads.""" ptr_size = self.get_ptr_size() sizes = (size for spec in FORMAT_SPEC.finditer(fmt) for size in get_format_slot_sizes(spec, ptr_size)) return sum(size // ptr_size for size in sizes) def va_args(self, va_list, num_args): - """ - Get the variable argument list - """ + """Read ``num_args`` pointer-sized values from ``va_list``.""" args = [] ptr = va_list ptrsize = self.get_ptr_size() @@ -585,8 +572,13 @@ def va_args(self, va_list, num_args): def setup_callback(self, func, args): """ - For APIs that call functions, we will setup the stack to make this flow - naturally. + Call the guest function ``func`` with ``args`` from the current API handler. + + If the API frame already has a guest call, add the call to the pending calls of + the frame. The API returns to its caller after the last call. + + Raises: + ApiEmuError: if no API call is active. """ run = self.emu.get_current_run() @@ -601,10 +593,11 @@ def setup_callback(self, func, args): def do_str_format(self, string, argv, wide=False): """ - Format a string similar to msvcrt.printf. With ``wide``, %s and %S - follow the wide-character functions: %s is a wide string and %S is - an ANSI string, unless an h or l prefix gives the width. Missing - arguments read as 0. + Format a string as msvcrt printf does. + + With ``wide``, %s and %S follow the wide-character functions: %s is a wide + string and %S is an ANSI string, unless an h or l prefix gives the width. + Missing arguments read as 0. """ ptr_size = self.get_ptr_size() slots = list(argv) diff --git a/speakeasy/winenv/api/compare.py b/speakeasy/winenv/api/compare.py index 55229cece..ee194995d 100644 --- a/speakeasy/winenv/api/compare.py +++ b/speakeasy/winenv/api/compare.py @@ -22,14 +22,14 @@ class Fold(enum.Enum): def fold_upper(c: str) -> str: - # Windows folds each UTF-16 code unit to one code unit, so "ß" and the characters - # outside the BMP stay the same. + # Windows folds each UTF-16 code unit to one code unit, so it does not change "ß" or a character + # outside the BMP. upper = c.upper() return upper if len(upper) == 1 and ord(c) < 0x10000 else c -# RtlUpperChar folds a byte in the ANSI code page. A byte whose upper case is not in the code page -# stays the same. +# RtlUpperChar folds a byte in the ANSI code page. If the upper case of a byte is not in the code page, +# RtlUpperChar does not change the byte. ANSI_UPPER_TABLE = bytes( DECODING_TABLE.index(u) if (u := fold_upper(c)) in DECODING_TABLE else b for b, c in enumerate(DECODING_TABLE) ) diff --git a/speakeasy/winenv/api/exception_apis.py b/speakeasy/winenv/api/exception_apis.py index 94637ed07..d83363e95 100644 --- a/speakeasy/winenv/api/exception_apis.py +++ b/speakeasy/winenv/api/exception_apis.py @@ -1,8 +1,8 @@ """ -Exception and unwind APIs that ntdll implements and kernel32 forwards to it. +Exception and unwind APIs that ntdll implements and kernel32 forwards to ntdll. -ExceptionApis is a mixin of the kernel32 and ntdll API handlers. Most of these -APIs exist only on x64, where exceptions are dispatched from unwind data. +The kernel32 and ntdll API handlers both inherit ExceptionApis. Most of these +APIs exist only on x64, where the exception dispatch uses the unwind data. """ from __future__ import annotations diff --git a/speakeasy/winenv/api/sigdb.py b/speakeasy/winenv/api/sigdb.py index 02e3823e6..62e8447ff 100644 --- a/speakeasy/winenv/api/sigdb.py +++ b/speakeasy/winenv/api/sigdb.py @@ -1,25 +1,20 @@ """ -Win32 API signature database. - -Speakeasy implements a curated subset of the Windows API with hand-written -handlers (see ``speakeasy.winenv.api``). For everything else, this module -supplies just enough information about an import to keep emulation coherent: -how many argument slots the call consumed, which calling convention cleans -them up, the name and basic type of each parameter, and the return type. - -Signatures come from pluggable :class:`SignatureSource` backends. Two are -bundled, both reading the same compact JSON table format: - -* :class:`Win32MetadataSource` reads ``resources/win32/win32json_signatures.json.gz``, - generated from Microsoft's win32metadata (via the ``deps/win32json`` - submodule) by ``scripts/gen_win32_signatures.py``: the documented Win32 API. -* :class:`PhntSource` reads ``resources/win32/phnt_signatures.json.gz``, - generated from the phnt native API headers (``deps/phnt``) by - ``scripts/gen_phnt_signatures.py``: ``Nt*``/``Zw*``/``Rtl*``/``Ldr*`` and - other ntdll exports win32metadata leaves undocumented. - -Both files are build artifacts and may be absent from a source checkout; a -source with no file degrades to "no signatures" and logs a single warning. +The Win32 API signature database. + +Speakeasy has handlers for a part of the Windows API (see ``speakeasy.winenv.api``). +For the other APIs, this module gives the declaration of an import: the number of +argument slots, the calling convention, the name and type of each parameter, and the +return type. + +A ``SignatureSource`` gives the signatures. Two sources come with Speakeasy, and both +read the same compact JSON table format: + +* ``Win32MetadataSource``: the documented Win32 API, from Microsoft win32metadata. +* ``PhntSource``: the ntdll exports that win32metadata does not document, such as + the ``Nt`` and ``Rtl`` functions, from the phnt native API headers. + +The build generates the signature files, so a source checkout can be without them. +A source without its file gives no signatures and logs one warning. """ from __future__ import annotations @@ -51,10 +46,9 @@ ARCH_X86 = "x86" ARCH_X64 = "x64" -# Byte size of each type code kind; None means pointer-sized. Kinds that carry -# a qualifier (``ps:NAME``, ``st:NAME:SIZE``) are handled in ParamSig.size(). -# Pointer kinds may qualify their target: ``p:u32`` (DWORD*), ``a:u16`` -# (WCHAR buffer), ``ps:NAME`` (struct pointer). +# Byte size of each type code kind. None means pointer-sized. ParamSig.size gives the +# size of the ``st`` and ``arr`` kinds. A pointer kind can name its target type: +# ``p:u32`` (DWORD*), ``a:u16`` (WCHAR buffer), ``ps:NAME`` (struct pointer). TYPE_SIZES: dict[str, int | None] = { "v": 0, "i8": 1, @@ -97,14 +91,14 @@ def normalize_dll(name: str) -> str: @dataclass(frozen=True) class EnumDef: - """A named set of integer constants, optionally combinable as flags.""" + """A named set of integer constants. With ``flags``, a value can combine more than one constant.""" name: str values: tuple[tuple[str, int], ...] flags: bool = False def decode(self, value: int) -> str: - """Render ``value`` with the rules of :func:`speakeasy.winenv.defs.lookup.get_value_names`.""" + """Render ``value`` with the rules of ``speakeasy.winenv.defs.lookup.get_value_names``.""" return "|".join(get_value_names(self.values, value, self.flags)) @@ -131,8 +125,9 @@ def size(self, ptr_size: int) -> int: @dataclass(frozen=True) class StructDef: """ - Layout of a struct or union. ``fields`` is empty when the generator could - not resolve every field; the size is still reliable then. + Layout of a struct or union. + + ``fields`` is empty if the generator could not resolve all fields. The size is correct also then. """ name: str @@ -150,10 +145,10 @@ class ParamSig: """ One parameter of an API signature. - ``buffer_len`` describes how long the buffer a pointer parameter refers to - is, when the declaration says: ``("n", i)`` parameter *i* holds the byte - count, ``("c", i)`` parameter *i* holds the element count, ``("k", n)`` - a fixed count of *n* elements. + ``buffer_len`` gives the length of the buffer of a pointer parameter, if the + declaration gives it. With ``("n", i)``, parameter ``i`` has the byte count. With + ``("c", i)``, parameter ``i`` has the element count. With ``("k", n)``, the + buffer has ``n`` elements. """ name: str @@ -172,14 +167,14 @@ def qualifier(self) -> str | None: @property def enum(self) -> str | None: - """Name of the enum an integer parameter draws its value from, if any.""" + """Name of the enum of an integer parameter, if any.""" if self.kind in INT_KINDS: return self.qualifier return None @property def pointee(self) -> str | None: - """Type code of what a pointer/buffer parameter points at, if known.""" + """Type code of the target of a pointer or buffer parameter, if known.""" kind = self.kind if kind in ("p", "a"): return self.qualifier @@ -190,7 +185,7 @@ def pointee(self) -> str | None: return None def elem_size(self, ptr_size: int) -> int | None: - """Size in bytes of one pointed-at element, or None when opaque (void*).""" + """Size in bytes of one target element, or None for an opaque pointer (void*).""" pointee = self.pointee if pointee is None: return None @@ -198,7 +193,7 @@ def elem_size(self, ptr_size: int) -> int | None: @property def is_in(self) -> bool: - # Parameters without any direction annotation are treated as inputs. + # A parameter without a direction is an input return "i" in self.flags or "o" not in self.flags @property @@ -210,7 +205,7 @@ def is_optional(self) -> bool: return "?" in self.flags def size(self, ptr_size: int) -> int: - """Size in bytes of the parameter's value for the given pointer size.""" + """Size in bytes of the value of the parameter for ``ptr_size``.""" kind = self.kind if kind == "st": # st:NAME:SIZE or st:NAME:SIZE32/SIZE64 @@ -231,14 +226,14 @@ def size(self, ptr_size: int) -> int: def slots(self, ptr_size: int) -> int: """Number of pointer-sized argument slots this parameter occupies.""" if ptr_size == 8 and self.kind in AGGREGATE_KINDS: - # Win64 passes aggregates that are not 1, 2, 4 or 8 bytes by - # reference, so they always take exactly one slot. + # Win64 passes an aggregate of 1, 2, 4 or 8 bytes in one slot, and other + # aggregates by reference in one slot. return 1 size = self.size(ptr_size) return max(1, -(-size // ptr_size)) def value_mask(self, ptr_size: int) -> int | None: - """Mask to apply to a slot value to recover a narrower-than-slot parameter.""" + """Mask that gets a parameter narrower than a slot from the slot value, or None.""" if self.kind in AGGREGATE_KINDS: return None size = self.size(ptr_size) @@ -272,14 +267,15 @@ def supports_arch(self, arch: str) -> bool: def supports_emulation(self, ptr_size: int) -> bool: """ - Whether the current generic integer-slot executor can use this ABI. - - This conservative capability check does not affect catalog membership - or handler/hook availability. Floating-point transport, aggregate - returns, variadic calls, and non-cdecl/stdcall conventions need - explicit support. On x86, wide integer and aggregate arguments occupy - multiple stack slots; on x64, an aggregate argument takes one slot. Wide - returns require EDX:EAX and cannot use the generic return path. + Return True if the emulator can call this function from the signature, without a handler. + + The emulator can call a cdecl or stdcall function whose arguments and return + value are integers, pointers or structs passed by value. It cannot call a + variadic function, a declaration with a skip marker, or a function with + floating-point arguments, a struct return value or a return value wider than a + pointer (EDX:EAX on x86). On x86, a 64-bit integer or a struct argument uses + more than one stack slot. On x64, a struct argument uses one slot. The result + does not change the exports of a module or the handlers and hooks that can run. """ arch = ARCH_X86 if ptr_size == 4 else ARCH_X64 if not self.supports_arch(arch) or self.skip is not None or self.variadic: @@ -297,14 +293,16 @@ def slot_layout(self, ptr_size: int) -> list[int]: return [p.slots(ptr_size) for p in self.params] def slot_count(self, ptr_size: int) -> int: - """Total argument slots consumed by a call (what speakeasy calls ``argc``).""" + """Total number of argument slots of a call. This is the ``argc`` of a handler.""" return sum(self.slot_layout(ptr_size)) def values_from_slots(self, slots: list[int], ptr_size: int) -> list[int]: """ - Fold raw argument slots back into one integer per parameter. Multi-slot - parameters (64-bit scalars and by-value structs on x86) are joined - little-endian; narrower parameters are masked to their size. + Join the raw argument slots into one integer for each parameter. + + A parameter with more than one slot, such as a 64-bit integer or a struct passed + by value on x86, gets its slots in little-endian order. A parameter narrower than + a slot gets only the bits of its size. """ values = [] pos = 0 @@ -323,16 +321,17 @@ def values_from_slots(self, slots: list[int], ptr_size: int) -> list[int]: # ``read_uint(addr, size)`` reads a little-endian unsigned integer from emulated -# memory, None when the address is unmapped. +# memory, or returns None if the address is unmapped. ReadUInt = Callable[[int, int], "int | None"] LookupStruct = Callable[[str], "StructDef | None"] def buffer_count(sig: FuncSig, index: int, values: list[int], ptr_size: int, read_uint: ReadUInt) -> int | None: """ - Value of the count/size parameter ``index`` of a call with argument - ``values``. Counts passed by pointer (``PDWORD pcbSize``) are read through - ``read_uint(addr, size) -> int | None``; unknown shapes give None. + Get the value of the count or size parameter ``index`` of a call with the argument ``values``. + + Read a count passed by pointer (``PDWORD pcbSize``) with ``read_uint``. Return None + for a NULL or output pointer, an unmapped pointer or other parameter types. """ param = sig.params[index] value = values[index] @@ -340,7 +339,7 @@ def buffer_count(sig: FuncSig, index: int, values: list[int], ptr_size: int, rea if kind in INT_KINDS or (kind == "p" and param.qualifier is None): return value if kind == "p" and param.qualifier is not None: - # count lives behind a pointer; only meaningful if it is an input + # Only an input pointer has a count before the call if not value or not param.is_in: return None size = param.elem_size(ptr_size) or ptr_size @@ -352,9 +351,11 @@ def out_buffer_size( sig: FuncSig, index: int, values: list[int], ptr_size: int, lookup_struct: LookupStruct, read_uint: ReadUInt ) -> int | None: """ - How many bytes an ``Out`` pointer parameter is declared to receive, or None - when the signature does not say (``void*`` with no size, arrays with no - count). Strings without a declared size count only their terminator. + Get the size in bytes of the buffer of an out pointer parameter, as the declaration gives it. + + Return None if the signature does not give the size, for example for a ``void*`` + without a size or an array without a count. A string without a declared size + counts only its terminator. """ param = sig.params[index] if param.buffer_len is not None: @@ -385,7 +386,7 @@ class SignatureSource(ABC): @property @abstractmethod def available(self) -> bool: - """True when this source has any signatures to offer.""" + """True if this source has signatures.""" @abstractmethod def lookup(self, dll: str, func: str, arch: str) -> FuncSig | None: @@ -393,14 +394,14 @@ def lookup(self, dll: str, func: str, arch: str) -> FuncSig | None: def iter_functions(self, dll: str, arch: str) -> Iterator[FuncSig]: """ - Enumerate exact, architecture-eligible module declarations, independently of ABI support. + Yield the declarations of ``dll`` for ``arch``, also those that the emulator cannot call. - Lookup-only custom sources contribute no catalog entries by default. + The default yields nothing, so a source that has only ``lookup`` adds no exports. """ return iter(()) def lookup_enum(self, name: str) -> EnumDef | None: - """Return the enum definition a type code qualifier refers to, if this source has it.""" + """Return the enum that a type code qualifier names, if this source has it.""" return None def lookup_struct(self, name: str) -> StructDef | None: @@ -412,10 +413,10 @@ class Win32MetadataSource(SignatureSource): """ Signatures generated from win32metadata via win32json. - Entries are indexed by function name first: win32metadata records the DLL - that implements a function (``KERNEL32.dll!K32EnumProcesses``) rather than - the forwarder a sample imports from (``psapi.dll!EnumProcesses``), so the - DLL is used to disambiguate and to select name prefixes, not as a hard key. + The index key is the function name. win32metadata records the DLL that implements + a function (``KERNEL32.dll!K32EnumProcesses``), not the DLL that a sample imports it + from (``psapi.dll!EnumProcesses``). So ``lookup`` uses the DLL only to select one of + the declarations of a name and to add name prefixes. """ name = "win32metadata" @@ -475,8 +476,8 @@ def _load(self) -> None: ) return self._functions = doc.get("functions", {}) - # Keep raw declarations and their original indices so catalogs share - # the lookup cache without eagerly constructing any FuncSig objects. + # Keep the raw declarations and their indices, so that iter_functions and lookup + # share the FuncSig cache, and make a FuncSig only when a caller asks for it. for name, entries in self._functions.items(): for idx, entry in enumerate(entries): dll = normalize_dll(entry["dll"]) @@ -509,11 +510,11 @@ def canonical_dll(self, dll: str) -> str: def iter_functions(self, dll: str, arch: str) -> Iterator[FuncSig]: """ - Yield exact DLL declarations in name order, converting them lazily. + Yield the declarations of ``dll`` for ``arch``, in name order. - Unlike lookup, this never reuses another DLL's declarations or applies - DLL aliases/name prefixes. The first architecture-eligible declaration - for each exact name wins; skip markers do not affect membership. + Unlike ``lookup``, use only the declarations of this DLL, without DLL aliases or + name prefixes. For each name, yield the first declaration for ``arch``, also if + it has a skip marker. """ self._load() dll = normalize_dll(dll) @@ -543,8 +544,8 @@ def lookup(self, dll: str, func: str, arch: str) -> FuncSig | None: if not candidates: return None - # Prefer the declaration whose implementing DLL matches the import; - # otherwise trust the (almost always unique) function name. + # Use the declaration of the DLL of the import. Else use the first declaration + # of the name, because most names have only one. matching = [c for c in candidates if c[2]["dll"] == dll] name, idx, entry = (matching or candidates)[0] if not matching: @@ -612,10 +613,10 @@ class PhntSource(Win32MetadataSource): """ Signatures generated from the phnt native API headers. - Same table format as :class:`Win32MetadataSource`. Everything it declares - is exported by ``ntdll`` (and, for ``Nt*``/``Zw*`` services, ``ntoskrnl``); - struct layouts are not included, so ``ps:`` codes resolve against the - other sources in the database. + It uses the table format of ``Win32MetadataSource``. ntdll exports all of its + functions, and ntoskrnl also exports the ``Nt`` and ``Zw`` services. It has no + struct layouts, so the other sources of the database give the structs of ``ps:`` + codes. """ name = "phnt" @@ -626,7 +627,7 @@ class PhntSource(Win32MetadataSource): class SignatureDatabase: - """Ordered collection of signature sources; the first source with an answer wins.""" + """An ordered list of signature sources. The first source with an answer wins.""" def __init__(self, sources: list[SignatureSource] | None = None) -> None: if sources is None: @@ -654,11 +655,10 @@ def lookup(self, dll: str, func: str, arch: str) -> FuncSig | None: def get_functions(self, dll: str, arch: str) -> dict[str, FuncSig]: """ - Map the exact declarations of one DLL to their signatures. + Get the declarations of one DLL, by name. - Membership comes from each source's catalog, never from a permissive - lookup or an ABI capability check. The first source that declares a - name wins. + The names come from ``iter_functions`` of each source, not from ``lookup`` or + ``FuncSig.supports_emulation``. The first source that declares a name wins. """ key = (normalize_dll(dll), arch) functions = self._functions.get(key) @@ -690,7 +690,7 @@ def lookup_struct(self, name: str) -> StructDef | None: def get_default_database() -> SignatureDatabase: - """Process-wide database backed by the bundled win32metadata and phnt signatures.""" + """Get the shared database of the bundled win32metadata and phnt signatures.""" global _default_db if _default_db is None: with _default_lock: diff --git a/speakeasy/winenv/api/sigfmt.py b/speakeasy/winenv/api/sigfmt.py index b8b9d00fb..91d140998 100644 --- a/speakeasy/winenv/api/sigfmt.py +++ b/speakeasy/winenv/api/sigfmt.py @@ -1,18 +1,16 @@ """ -Human readable rendering of the arguments of API calls with a known signature. - -:class:`ArgFormatter` turns raw argument values into the text that appears in -the API trace, using the type information in a -:class:`~speakeasy.winenv.api.sigdb.SignatureDatabase`: strings are read from -memory, BOOLs become ``TRUE``/``FALSE``, enum and flag values get their -symbolic names, and pointers to known structs are expanded into a JSON-like -``{field: value, ...}`` rendering, following nested pointers a bounded number -of levels. Each top-level argument also gets a kind that tells a consumer how -to read its text; strings inside a struct rendering are quoted, top-level -strings are not. - -Memory access goes through callbacks so the formatter is independent of the -emulator and testable against a plain ``bytes`` buffer. +Text of the arguments of API calls with a known signature. + +``ArgFormatter`` changes raw argument values into the text of the API trace. It +uses the types of a ``SignatureDatabase``: it reads strings from memory, shows +BOOLs as ``TRUE`` or ``FALSE``, shows the names of enum and flag values, and +shows a pointer to a known struct as ``{field: value, ...}``. It follows nested +pointers to a fixed depth. Each top-level argument also gets a kind that tells a +consumer how to read its text. Strings inside a struct have quotes, top-level +strings do not. + +``ArgFormatter`` reads memory through callbacks, so it does not need the +emulator, and a test can give it a ``bytes`` buffer. """ from __future__ import annotations @@ -29,7 +27,7 @@ ReadMem = Callable[[int, int], bytes] ReadXmm = Callable[[int], int] -# Structs with a more useful rendering than their raw fields +# Structs that show as one value, not as their fields _COUNTED_STRING_STRUCTS = {"UNICODE_STRING": 2, "STRING": 1, "ANSI_STRING": 1, "CUNICODE_STRING": 2} _INTEGER_STRUCTS = ("LARGE_INTEGER", "ULARGE_INTEGER") @@ -67,7 +65,7 @@ def __init__( # -- parameters -------------------------------------------------------- def render_param(self, param: sigdb.ParamSig, value: int, index: int) -> RenderedArg: - """Render parameter ``index`` of a call whose folded argument value is ``value``.""" + """Render parameter ``index`` of a call. ``value`` is the parameter value, made from its argument slots.""" kind = param.kind if kind in sigdb.FLOAT_KINDS: if self.ptr_size == 8 and index < 4 and self.read_xmm is not None: @@ -80,13 +78,13 @@ def render_param(self, param: sigdb.ParamSig, value: int, index: int) -> Rendere if kind == "g" and self.ptr_size == 4: return RenderedArg(self._format_guid(value.to_bytes(16, "little")), ApiArgType.GUID) if kind == "st" and not (self.ptr_size == 8 and param.size(8) not in (1, 2, 4, 8)): - # by value in the argument slots themselves + # A struct passed by value is in the argument slots return self._render_struct_bytes(_struct_key(param), value.to_bytes(param.size(self.ptr_size), "little")) if kind in ("ps", "st", "g", *sigdb.STRING_KINDS) and (not param.is_in or not value): - # Out-only pointers hold nothing meaningful before the call + # Show NULL and out-only pointers as addresses. An out-only buffer has no data before the call. return RenderedArg(hex(value), ApiArgType.PTR) if kind == "g": - # aggregate passed by reference on Win64 + # Win64 passes a GUID by value as a pointer to a copy try: return RenderedArg(self._format_guid(self.read_mem(value, 16)), ApiArgType.GUID) except Exception: @@ -235,7 +233,7 @@ def _format_struct_bytes( return "{" + ", ".join(parts) + "}" def _render_special_struct(self, struct: sigdb.StructDef, data: bytes) -> RenderedArg | None: - """Counted strings render as their (unquoted) text, LARGE_INTEGERs as one number.""" + """Render a counted string as its text without quotes, and a LARGE_INTEGER as one number, else None.""" name = struct.name if name in _INTEGER_STRUCTS and len(data) >= 8: return RenderedArg(hex(int.from_bytes(data[:8], "little")), ApiArgType.INT) @@ -308,12 +306,12 @@ def get_call_args( slots: list[int], ) -> list[ApiArg]: """ - Assemble the named arguments of a call from the rendering of its - parameters. ``slots`` holds the raw argument slots as read from the call - and gives each parameter's value. + Make the named arguments of a call from its rendered parameters. + + ``slots`` has the raw argument slots of the call, and gives the value of each parameter. Raises: - ValueError: if the slots do not match the slots the signature consumes. + ValueError: if the slots or the rendered parameters do not match the signature. """ if len(slots) != sig.slot_count(ptr_size) or len(rendered) != len(sig.params): raise ValueError(f"{sig.name}: argument slots do not match the signature") @@ -325,12 +323,12 @@ def get_call_args( def get_slot_args(slots: list[int]) -> list[ApiArg]: - """Arguments of a call without a usable signature, one entry per argument slot.""" + """Get the arguments of a call without a usable signature, one for each argument slot.""" return [ApiArg(type=ApiArgType.INT, value=slot, display=hex(slot)) for slot in slots] def _struct_key(param: sigdb.ParamSig) -> str | None: - """Struct table key named by a ps:NAME or st:NAME:SIZE code.""" + """Get the struct name of a ps:NAME or st:NAME:SIZE parameter, or None.""" qualifier = param.qualifier or "" if param.kind == "st": return qualifier.rsplit(":", 1)[0] or None diff --git a/speakeasy/winenv/api/winapi.py b/speakeasy/winenv/api/winapi.py index cb4957079..d498a9962 100644 --- a/speakeasy/winenv/api/winapi.py +++ b/speakeasy/winenv/api/winapi.py @@ -63,13 +63,12 @@ def get_export_func_handler(self, mod_name, exp_name): return (mod, mod.get_func_handler(exp_name)) def call_api_func(self, mod, func, argv, ctx): - """ - Call the handler to implement the imported API - """ return func(mod, self.emu, argv, ctx) def call_data_func(self, mod, func, ptr): """ - Call the handler to initialize and return imported data variables + Call the data handler of an exported variable. + + The handler writes the initial value at ``ptr`` and returns ``ptr``. """ return func(mod, ptr) diff --git a/speakeasy/winenv/arch.py b/speakeasy/winenv/arch.py index 5ff589e09..fdab1d039 100644 --- a/speakeasy/winenv/arch.py +++ b/speakeasy/winenv/arch.py @@ -101,7 +101,7 @@ CALL_CONV_FLOAT = 3 VAR_ARGS = -1 -# Register string constants +# Register names to register IDs REG_LOOKUP = { # x86 registers "eax": X86_REG_EAX, diff --git a/speakeasy/winenv/codepage.py b/speakeasy/winenv/codepage.py index a286daf17..6c0b72007 100644 --- a/speakeasy/winenv/codepage.py +++ b/speakeasy/winenv/codepage.py @@ -3,8 +3,8 @@ ``GetACP`` returns 1252, so ANSI strings use Windows-1252. Python's ``cp1252`` codec has no mapping for 0x81, 0x8D, 0x8F, 0x90 and 0x9D. Windows maps them to the C1 controls with the same value, so -the table here does too. Then each byte decodes to one character, and a decode then an encode gives -the same bytes. +``DECODING_TABLE`` does too. With this table, each byte decodes to one character, and an encode of the +decoded text gives the same bytes. """ import codecs @@ -37,7 +37,7 @@ def encode_ansi(text: str) -> bytes: def get_codec(code_page: int) -> str | None: """ - Get the Python codec for a code page that the conversion functions take. + Get the Python codec for a code page argument of ``MultiByteToWideChar`` or ``WideCharToMultiByte``. Return None for the ANSI code page, which uses ``decode_ansi`` and ``encode_ansi``.