A Comprehensive Platform for Context Management in Scientific Computing
Overview ·
Installation ·
Components ·
Quickstart ·
Documentation ·
Contributing
CLIO Core is a unified framework that integrates multiple high-performance components for context management, data transfer, and scientific computing. CLIO Core enables developers to build efficient data processing pipelines for HPC, storage systems, and near-data computing applications.
It provides:
- High-performance context management for computational contexts and data transformations.
- Heterogeneous-aware I/O with multi-tiered, dynamic buffering.
- A modular runtime with dynamically loadable processing modules.
- Shared-memory data structures that work across host, CUDA, and ROCm.
- Distributed-by-construction scaling from single node to clusters.
┌──────────────────────────────────────────────────────────────┐
│ Applications │
│ (Scientific Workflows, HPC, Storage Systems) │
└──────────────────────────────────────────────────────────────┘
│
┌─────────────────────┼─────────────────────┐
│ │ │
┌───────────────┐ ┌──────────────────┐ ┌────────────────┐
│ Context │ │ Context │ │ Context │
│ Exploration │ │ Assimilation │ │ Transfer │
│ Engine │ │ Engine │ │ Engine │
└───────────────┘ └──────────────────┘ └────────────────┘
│ │ │
└─────────────────────┼─────────────────────┘
│
┌─────────────────┐
│ CLIO Runtime │
│ (Module System)│
└─────────────────┘
│
┌─────────────────────────┐
│ Context Transport │
│ Primitives │
│ (Shared Memory & IPC) │
└─────────────────────────┘
The pip wheel ships a portable, self-contained build with all
dependencies statically linked. No system installs are required beyond
glibc 2.28+ (manylinux_2_28: RHEL/Rocky/AlmaLinux 8 and newer, Ubuntu
20.04+, Debian 11+) and Python 3.10+.
pip install iowarp-coreThe wheel includes the CLIO runtime, the clio_run CLI, the CTE, CAE, and
CEE engines, and the clio_cee Python bindings. A default configuration is
seeded at ~/.clio/clio.yaml on first import.
The pip wheel covers the common case. Build from source if you need any of:
- NVIDIA GPU (CUDA) or AMD GPU (ROCm) acceleration
- MPI for distributed multi-node deployment
- HDF5 / ADIOS2 adapters
- FUSE adapter
- Compression backends (LibPressio, Blosc, etc.)
- Custom ChiMods built against the C++ headers
- Sanitizer or debug builds
git clone --recurse-submodules https://github.com/iowarp/clio-core.git
cd clio-core
# Common presets: release, debug, cuda-release, rocm-release
cmake --preset=release
cmake --build build -j"$(nproc)"
sudo cmake --install buildFor the per-feature apt / dnf dependency list and the complete
CLIO_*_ENABLE_* flag checklist, see INSTALL.md. Other install
methods (Conda, Docker, Spack) are documented in
docs/getting-started/installation.
| Component | Location | Purpose |
|---|---|---|
| Context Transport Primitives | context-transport-primitives/ |
Shared-memory containers, allocators, sync primitives, networking (ZMQ / libfabric / Thallium). GPU-aware (CUDA, ROCm). |
| CLIO Runtime | context-runtime/ |
Coroutine-based modular runtime (< 10 µs task latency). Hosts ChiMods and provides admin + bdev modules. |
| Context Transfer Engine | context-transfer-engine/ |
Multi-tiered, heterogeneous-aware I/O buffering. Tag + blob storage with adapters for POSIX, HDF5 (VFD/VOL), ADIOS2, MPI-IO, FUSE3, GDS. |
| Context Assimilation Engine | context-assimilation-engine/ |
OMNI-YAML-driven data ingestion (binary, HDF5, Globus) into CTE. |
| Context Exploration Engine | context-exploration-engine/ |
High-level C++ and Python (clio_cee) API for bundling, querying, and retrieving data. Includes an MCP server for AI agents. |
Installation seeds a default configuration at ~/.clio/clio.yaml, so the
runtime works out of the box:
clio_run start # foreground
clio_run start & # backgroundTo override the configuration, point CLIO_SERVER_CONF at your own YAML file:
export CLIO_SERVER_CONF=/path/to/my_config.yaml
clio_run startclio_cte_fuse exposes the Context Transfer Engine as a POSIX filesystem, so
unmodified applications read and write IOWarp-managed data through ordinary
file I/O. The wheel ships this binary on Linux and Windows; macOS wheels do not
(macOS has no FUSE3 API).
1. Install the FUSE runtime. libfuse3 is a system dependency and is
deliberately not bundled in the wheel, so install your distribution's package:
sudo apt install fuse3 libfuse3-3 # Debian / Ubuntu
sudo dnf install fuse3 fuse3-libs # Fedora / RHELOn Windows, install WinFsp instead. The clio_cte_fuse
console script prepends WinFsp's bin directory to PATH itself, so
winfsp-x64.dll resolves without a system-wide PATH edit.
2. Mount. Start the runtime, then point the daemon at a mountpoint. The
daemon create-or-binds the filesystem pool and the CTE pool underneath it, so
no separate compose step is required:
clio_run start &
mkdir -p ~/clio-mnt
CLIO_WITH_RUNTIME=0 clio_cte_fuse ~/clio-mnt -fCLIO_WITH_RUNTIME=0 attaches to the runtime you just started rather than
spawning an embedded one — without it the daemon brings up its own runtime and
the data will not be visible to other clients. -f keeps the daemon in the
foreground, which is the configuration CI exercises. Every remaining argument
is handed to fuse_main, so standard libfuse options apply (-o allow_other,
-d for protocol tracing). On Windows the mountpoint is a drive letter:
clio_cte_fuse Z:.
The mount then behaves like any other filesystem:
cp big_input.h5 ~/clio-mnt/
python analysis.py ~/clio-mnt/big_input.h53. Unmount:
fusermount3 -u ~/clio-mnt # or: fusermount -u ~/clio-mntTo size the storage tiers explicitly instead of taking the defaults, compose a
CTE pool before mounting — see
context-transfer-engine/test/integration/fuse-manual/cte_compose.yaml
for a working single-tier example:
clio_run compose start my_cte.yamlTwo semantics worth knowing: writes are write-through, so each write()
reaches the chimod synchronously and the logical file size is always exact;
and the kernel attribute and entry caches are disabled, so metadata that
another client changed is never served stale.
import clio_cee as cee
ctx_interface = cee.ContextInterface()
# Assimilate inline strings into IOWarp storage.
# src="string::<blob_name>" names the blob; src_data carries the payload.
docs = [
("climate_report", "Arctic sea ice extent fell to a record low in 2023."),
("ocean_temps", "Ocean surface temperatures rose 0.3°C above the 20-year mean."),
("co2_levels", "Atmospheric CO₂ reached 421 ppm, the highest in 800,000 years."),
]
bundle = [
cee.AssimilationCtx(
src=f"string::{name}",
dst="iowarp::climate_docs",
format="string",
src_data=text,
)
for name, text in docs
]
ctx_interface.context_bundle(bundle)
# Query for blob names matching a regex.
blobs = ctx_interface.context_query("climate_docs", ".*")
# Query the top-2 most relevant blob names via BM25 semantic search.
blobs = ctx_interface.context_query("climate_docs", ".*",
max_results=2,
prompt="temperature anomaly over Arctic")
# Retrieve blob payloads (regex).
data = ctx_interface.context_retrieve("climate_docs", ".*")
# Retrieve the top-2 most relevant blobs via BM25 semantic search.
data = ctx_interface.context_retrieve("climate_docs", ".*",
max_results=2,
prompt="temperature anomaly over Arctic")
# Clean up.
ctx_interface.context_destroy(["climate_docs"])For direct CTE put/get from C++, see the canonical example and operation reference in the Context Transfer Engine README.
cd build/release
ctest -VV # all unit tests
ctest -R context_transport # CTP tests
ctest -R runtime # runtime tests
ctest -R cte # CTE tests
ctest -R omni # CAE tests
ctest -R context # CEE testsCLIO Core ships two main benchmarks; pass --help to either for the full
parameter list.
clio_run_thrpt_bench— runtime task throughput and latency (bdev_io,bdev_allocation,bdev_task_alloc,latencytest cases).clio_cte_bench— CTE Put / Get / PutGet throughput across threads, async depth, I/O size, and key-space cardinality.
Example:
clio_run_thrpt_bench --test-case bdev_io --threads 8 --duration 30
clio_cte_bench --op PutGet --threads 8 --depth 16 --io-size 1m --io-count 200- AGENTS.md — unified development guide and coding standards.
- INSTALL.md — bare-metal source-build instructions.
- Context Transport Primitives
- CLIO Runtime
- Context Transfer Engine — canonical C++ CTE API reference.
- Context Assimilation Engine
- Context Exploration Engine
- Full documentation site: https://grc.iit.edu/docs/category/iowarp
Scientific computing: data processing pipelines, near-data computing, custom storage engines, workflows with context management.
Storage systems: distributed file system backends, object storage, multi-tiered caches, high-throughput I/O buffering.
HPC and data-intensive workloads: accelerated I/O, ingestion and transformation pipelines, heterogeneous computing with GPU support, real-time streaming analytics.
- Task latency: < 10 µs for local task execution.
- Memory bandwidth: up to 50 GB/s with the RAM bdev backend.
- Scalability: single node to multi-node clusters.
- Concurrency: thousands of concurrent coroutine-based tasks.
- Fork the repository.
- Create a feature branch:
git checkout -b feature/amazing-feature. - Follow the standards in AGENTS.md.
ctest --test-dir build/releasebefore opening a PR.- Submit a pull request against
iowarp/clio-core.
CLIO Core is licensed under the BSD 3-Clause License. See LICENSE for the full text.
Copyright (c) 2024, Gnosis Research Center, Illinois Institute of Technology
CLIO Core is developed at the GRC lab at Illinois Institute of Technology as part of the IOWarp project, supported by the National Science Foundation (NSF) to advance next-generation scientific computing infrastructure.
- IOWarp project: https://grc.iit.edu/research/projects/iowarp
- IOWarp organization: https://github.com/iowarp
- Documentation hub: https://grc.iit.edu/docs/category/iowarp