architecture: simplify Data Machine around one canonical execution model

[chubes4]

Goal

Simplify Data Machine substantially without reducing its execution, extensibility, security, portability, or curated product interfaces.

The canonical product model is:

agent -> pipeline -> flow -> job -> packets/artifacts

Abilities are the reusable business-operation boundary, handlers extend steps, and Action Scheduler supplies durable execution.

REST controllers, WP-CLI commands, and chat tools are intentional adapters that create Data Machine's preferred, stable namespace for each consumer. They are architecture, not accidental duplication.

Principle

Reduce duplicate domain implementations and competing internal representations, not execution semantics or purpose-built adapter namespaces.

Adapter boundary

Preserve these curated surfaces:

• datamachine/v1/... as the stable REST product namespace.
• wp datamachine ... as the coherent operator namespace.
• Data Machine chat tools as the constrained agent-facing namespace.

Adapters may and should own transport/product concerns including route shape, HTTP semantics, CLI presentation and exit behavior, model-facing names and schemas, context injection, approvals, and composition. Cleanup applies only where an adapter independently reimplements validation, authorization, persistence, or execution already owned by an ability.

Accidental complexity to remove

• Multiple competing internal result representations across abilities, steps, jobs, tools, and bundle runs.
• Duplicated domain logic behind REST, CLI, or chat adapters rather than delegation to abilities.
• Legacy bundle arrays, directory value objects, and package projections used as parallel internal forms.
• Overlapping run/job/task/workflow/bundle-run vocabulary and status normalization.
• Manual registration blocks and zero-behavior ability facades.
• Tool compatibility paths that duplicate ability execution or require contradictory policy passes.
• Domain integrations retained in core despite having clear extension ownership.

Preserve explicitly

• The preferred REST, CLI, and chat namespaces and their purpose-built adapters.
• Pipeline versus flow lifecycle distinction.
• Per-item child jobs, bounded fan-out, retries, and observability.
• CAS updates, leases, generation fences, cancellation, and idempotency.
• Meaningful terminal states including empty, skipped, cancelled, waiting, and failed.
• Agent/user/network memory scopes.
• Tool authorization and adjacent-handler boundaries.
• Pending-action and delegated-operation security semantics.
• Bundle artifact hashes, local modification detection, and conflict-safe upgrades.
• Protocol-specific HTTP behavior such as webhooks, callbacks, downloads, and streaming.

Workstreams

1. Inventory public and persisted contracts: abilities, adapter namespaces, hooks, tools, bundle schemas, statuses, and engine-data keys.
2. Remove proven zero-behavior indirection and centralize registration.
3. Standardize abilities on typed success values and WP_Error failures while preserving adapter output contracts.
4. Establish one canonical step/job result contract, with presentation and compatibility translation at adapter edges.
5. Make the directory bundle model canonical internally and thin bundle execution around normal workflow/job execution.
6. Ensure chat tools delegate to abilities where appropriate while retaining their curated names, schemas, policy, context, approvals, and composed behavior.
7. Audit REST and CLI adapters for duplicated domain logic without removing their product namespaces.
8. Declare canonical CLI spellings and compatibility aliases explicitly within the preferred wp datamachine namespace.
9. Treat run as a projection of job unless evidence requires a separate persisted entity.
10. Update current architecture documentation and clearly mark historical migration documents.

Existing tracked work

• Thin Data Machine bootstrap into service providers #2416
• Migrate ability callbacks from legacy success arrays to WP_Error failures #2522
• Unify AgentBundler with bundle value objects and workflow spec compilation #1501
• Remove legacy ability-tool projection compatibility paths #2479
• Move domain-specific SEO, media, and email abilities out of core #2223
• Final pre-1.0 cleanup: remove migrations and backward-compatibility paths #2418
• Centralize dependency and capability checks #2417
• conversation-loop.php is a 2,724-line god-file with a 15-param/6-by-ref closure factory #2803
• Converge native fan-out (mode => fanout) onto the agents-api parallel-branch primitive #2829
• fix: stop runtime ability execution from re-registering abilities #3054
• audit: Abilities in data-machine (6) #2388
• audit: Api in data-machine (18) #2389
• audit: Tools in data-machine (3) #2401
• audit: duplication in data-machine (8) #2403
• audit: intra method duplication in data-machine (141) #2405
• audit: structural in data-machine (16) #2407

Sequencing constraints

• Inventory extension and external usage before deleting compatibility surfaces.
• Do not remove or bypass preferred Data Machine adapter namespaces in favor of raw ability exposure.
• Do not combine public contract changes with underlying ability behavior changes in one release.
• Preserve persisted-data migration paths until usage and upgrade evidence proves removal safe.
• Prefer existing WordPress Abilities and Agents API primitives; do not create a parallel container or wrapper substrate.
• Each child issue should state the canonical representation it converges on and the compatibility edge or duplicated domain implementation it removes.

Definition of done

• One documented canonical model for definitions, configured flows, executions, packets, artifacts, and results.
• Abilities own reusable business logic; REST, CLI, and chat retain curated Data Machine namespaces and own consumer-specific semantics.
• Bundle internals use one representation.
• Registration and bootstrap are explicit but materially smaller.
• Public and persisted contracts have migration tests.
• No loss of durable execution, extension, security, observability, portability, or product-interface behavior.