Adaptive backend routing - #670
Draft
qti-dflocos wants to merge 4 commits into
Draft
Conversation
Introduces `"auto"` as a new `backend_type` that defers
`QnnBackendManager` creation from the `QnnEp` constructor (where it is
normally created based on the passed `backend_type`) to
`GetCapabilityImpl`, where the ONNX graph is available. A single-pass
classifier (`qnn_model_classifier.{h,cc}`) returns `GenAI`, `NonGenAI`,
or `Unknown`; the EP then loads `QnnGpu.dll` for `GenAI` and
`QnnHtp.dll` otherwise.
Classifier signals (early-exit on definitive GenAI markers):
- heavy/light transformer contrib ops, `MatMulNBits`
- KV-cache input names, decomposed-attention pattern
- `LayerNorm`/`Conv`/`BatchNorm` density
- classification-head pattern, 4D image-shaped inputs
Two new `QnnEp` members carry state across the deferred path:
`auto_select_backend_` and `deferred_backend_config_`, used by
`GetCapabilityImpl` before `SetupBackend()`.
`"auto"` must be passed explicitly; an absent `backend_type` does not
trigger classification.
…en ready Introduces the `hot_migration` session option. When enabled with an explicit GPU backend, the session composes both GPU and HTP graphs during `CompileImpl` while `OrtGraph*` pointers are still valid, finalizes GPU synchronously, and defers HTP finalization to a background thread. Once HTP is ready, the next `ComputeImpl` call atomically swaps every subgraph from GPU to HTP, transparent to ORT. `hot_migration` requires GPU to be selected explicitly. Migration state (two side-by-side `QnnBackendManager`s, per-subgraph atomic model slots, background HTP finalization thread) lives in `QnnEp` and is cleared in its destructor. `GetSupportedNodes` now has a `QnnBackendManager*` overload; hot migration calls it twice (GPU then HTP) and claims only the intersection. Nodes in one set but not the other fall to CPU EP permanently for the session. `InitQnnHtpGraphConfigs` no longer guards internally on backend type - the name already implies the caller knows the backend is HTP. Each call site now guards on its own manager so the hot path applies HTP configs to the correct backend.
|
|
qti-dflocos
changed the title
Capture the graph structure during compilation so HTP compose can run on a background thread after the original graph pointers are freed. Graph reads are routed through an injected API layer that transparently resolves to either live ORT objects or the snapshot, letting the entire compose pipeline work unchanged. Also migrates attribute helpers and node-group utilities to use the same injected API path. Node validation for both backends now happens concurrently.
…odes When some nodes are supported only by GPU, split the snapshot into segments by backend affinity and compile each independently. At execution time, run segments in sequence with intermediate buffers wiring outputs to inputs across segment boundaries. Nodes that both backends support still take the full migration path as before.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Auto backend selection (
backend_type|auto)Defers backend selection to
GetCapabilityImplwhere the ONNX graph is available. A single-pass model classifier categorizes the model as GenAI, NonGenAI, or Unknown based on graph structure. GenAI routes to GPU and the others route to NPU.Hot migration (
hot_migration|1)Composes both GPU and NPU graphs synchronously in
CompileImpl(ORT frees graph pointers after Compile returns, so both must be composed before that point), finalizes GPU in-place, then launches a background thread for NPU finalization. Once NPU is ready,TryMigrateIfReadyatomically swaps all subgraphs from GPU to NPU, drains in-flight GPU work, and promotes the NPU backend manager. On background failure, the session continues on GPU permanently.NPU compose on the critical path adds overhead to session creation - GPU inference only begins after both graphs are composed. Only the NPU finalization step runs in the background.
Partitioning claims only the intersection of GPU and NPU support sets to prevent post-migration failures. Requires
backend_type|gputo be set explicitly.Motivation and Context
Auto backend selection:
Applications currently must know which QNN backend suits a given model. The classifier lets the EP make this decision based on model architecture, removing that burden.
Hot migration:
NPU graph finalization adds seconds to session creation before the first token can be generated. GPU compiles fast but has lower sustained throughput. The goal of hot migration is to reduce the time-to-first-token by deferring NPU finalization to a background thread, serving initial inference on GPU, then transparently switching to NPU for higher throughput once finalization completes.