Miscellanous TRT EP changes for DLA - #15
Open
hrishikeshm wants to merge 6 commits into
Open
Conversation
hrishikeshm
commented
Aug 5, 2026
hrishikeshm
commented
Contributor
- Add support for DLA transforms
- Support for arena allocator on DLA
- Compilation fixes for ARM64 cross-compilation
- DLA specific unittests
When cross-compiling (e.g. x64 host to ARM64 target), FetchContent builds protoc for the target arch which cannot execute on the host. The shim creates an IMPORTED protobuf::protoc target from a pre-built host-arch binary when -DPROTOC_EXECUTABLE is passed, with -Dprotobuf_BUILD_PROTOC_BINARIES=OFF to skip building the target-arch protoc. No-op for normal native builds.
… logger suppression - BindContextInput/Output: unregister old DLA address before rebinding if pointer changed - Add trt_dla_static_io_buffers option to defer post-inference unregistration - Suppress WARNING-level TRT log messages in TensorrtLogger
Replace per-run cudaMalloc/cudaFree with a BFC arena allocator that retains freed GPU blocks in size-class bins and reuses them without calling into the CUDA runtime. Implements Shrink() and matches canonical DeallocateChunk stream-clear behaviour.
Adds trt_dla_transform_enable provider option that can be used to apply DLA transforms before TRT capability checking and engine compilation. - Capability check: transforms applied to check_buf; on full TRT support, parser_nodes_list is rebuilt with N-space indices since supportsModelV2 returns M-space subgraph indices for the transformed model. Partial support raises an error. Partially supported graphs will be supported via a followup change - Engine compile: ApplyDlaTransforms mutates string_buf in-place before the TRT parser sees it. - Cache key gains _dlatransform suffix to avoid loading engines built without transforms. - DLA_TRANSFORMS_ROOT / onnxruntime_ep_tensorrt_DLA_TRANSFORMS CMake option controls optional linkage against dla_transforms.lib.
…transforms scope - Wrap both GetSupportedList call sites in GetCapabilityImpl (noexcept) with try/catch(std::exception) converting throws to OrtStatus* returns; previously an uncaught std::runtime_error from THROW() would call std::terminate() - Extract duplicated DLA tensor address rebind pattern from BindContextInput and BindContextOutput into a single SetTensorAddressDla() helper - Restore original TensorrtLogger::log() behaviour: remove the hardcoded kERROR cap, reinstate full severity label string and correct ORT log-level mapping (WARNING -> ORT_LOGGING_LEVEL_WARNING, ERROR -> ORT_LOGGING_LEVEL_ERROR) - Restrict onnxruntime_ep_tensorrt_DLA_TRANSFORMS to Windows: option is now declared on all platforms but emits a CMake FATAL_ERROR on non-Windows builds; removes the unused Linux .so link path
Python: - test_basic_inference.py: FP16 Conv smoke test - test_dla_options.py: DLA option validation (Cat 1) and functional tests (Cat 3-6): mem_pool_limit, static_io_buffers, uint8_asymmetric_quantization, adjust_for_dla - test_dla_transforms.py: A16W8 MatMul with DLA transforms - conftest.py: plugin EP registration, has_dla / has_dla_transforms fixtures - ort_helpers.py: create_session, run_in_large_stack, run_session_once C++ (GTest): - test_main.cc / test_config.h: shared env, CLI flags (--target=gpu|dla), EP lib auto-discovery adjacent to binary - tensorrt_basic_test.cc / cuda_graph_test.cc: refactored to shared globals - tensorrt_ep_dla_transforms_test.cc: mirrors test_dla_transforms.py - tensorrt_ep_dla_options_test.cc: mirrors test_dla_options.py (Cat 1, 3-6) - CMakeLists.txt: BUILD_TESTS wires all sources, copies EP + DLA DLLs post-build
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.