Skip to content

knowledge-layer.md

Latest commit

 

History

History
598 lines (461 loc) · 27.6 KB

knowledge-layer.md

File metadata and controls

598 lines (461 loc) · 27.6 KB
Raw

Knowledge Layer

A pluggable abstraction for document ingestion and retrieval. Swap backends without changing application code.

Looking to build a custom backend adapter? Refer to the SDK Reference for data schemas, interfaces, and implementation examples.

Key Features

  • Rich Output Schema - Chunk model with 12 fields: content types, citations, images, structured data
  • Full Ingestion Pipeline - BaseIngestor with async job tracking and status polling
  • Collection Management - create/delete/list collections per session or use case
  • File Management - upload/delete/list files with status tracking (UPLOADING -> INGESTING -> SUCCESS/FAILED)
  • Content Typing - TEXT, TABLE, CHART, IMAGE enums for frontend rendering
  • Backend Agnostic - Swap among LlamaIndex, hosted RAG Blueprint, Azure AI Search, and OpenSearch without core agent code changes

Table of Contents

Available Backends

Backend Config Name Mode Vector Store Best For
llamaindex "llamaindex" Local Library ChromaDB Dev, prototyping, macOS/Linux
foundational_rag "foundational_rag" Hosted Service Remote Milvus Production, multi-user
azure_ai_search "azure_ai_search" Managed Service Azure AI Search Managed hybrid retrieval
opensearch "opensearch" External Service OpenSearch k-NN index Self-hosted OpenSearch, Amazon OpenSearch Service, or Serverless

Local Library Mode - Everything runs in your Python process. No external services needed.

  • llamaindex - LlamaIndex + ChromaDB. Lightweight, great for development. Works on macOS and Linux.

External Service Modes - Connect to deployed services. They require infrastructure but support shared, durable stores.

  • foundational_rag - Connects to NVIDIA RAG Blueprint through HTTP.
    • Tested with: NVIDIA RAG Blueprint v2.4.0 (Helm chart nvidia-blueprint-rag)
    • Deployment Guide
    • Backend-specific documentation: sources/knowledge_layer/src/foundational_rag/README.md
  • azure_ai_search - Stores client-generated embeddings in namespaced Azure AI Search indexes and supports vector, hybrid, and semantic-ranked retrieval.
  • opensearch - Uses one vector index per AI-Q collection with none, basic, or SigV4 authentication.
    • Supports self-hosted OpenSearch, Amazon OpenSearch Service (es), and Amazon OpenSearch Serverless (aoss).
    • Can ingest in the local process or dispatch ingestion to Dask workers.
    • Refer to Amazon OpenSearch Serverless for the AOSS/EKS deployment path.

Quick Start

Before you begin documentation ingestion and retrieval, run the following commands to install the backend knowledge layer.

Prerequisites: Complete the main setup first (refer to the project README.md): clone repo, run ./scripts/setup.sh, obtain API keys.

Tip: Instead of exporting env vars each time, add them to deploy/.env and use dotenv -f deploy/.env run <command> to run any command with those vars loaded automatically.

# 1. Set up environment variables (add to deploy/.env to avoid exporting each time)
export NVIDIA_API_KEY=nvapi-your-key-here

# 2. Install backend (choose one)
uv pip install -e "sources/knowledge_layer[llamaindex]"        # Recommended for local dev - works on macOS/Linux
uv pip install -e "sources/knowledge_layer[foundational_rag]"  # Requires deployed server
uv pip install -e "sources/knowledge_layer[azure_ai_search]"   # Requires an Azure AI Search service
uv pip install -e "sources/knowledge_layer[opensearch]"        # Requires an OpenSearch endpoint

New to Knowledge Layer? Start with llamaindex - it requires no external services and works on macOS and Linux.

# 3. Verify
python -c "from aiq_agent.knowledge import get_retriever; print('OK')"

Usage

To use the knowledge layer, you can change the variables in the YAML config file.

With NeMo Agent Toolkit (YAML Config) - Recommended

The knowledge_retrieval function is registered as a NeMo Agent Toolkit function type. YAML config is the recommended single source of truth for workflow configuration:

# Example knowledge_retrieval function configuration
functions:
  knowledge_search:
    _type: knowledge_retrieval      # NeMo Agent Toolkit function type
    backend: llamaindex             # Required: which adapter to use
    collection_name: my_docs        # Retrieval fallback when no session context is present
    top_k: 5                        # Results to return

    # Summarization options (optional, all backends):
    # generate_summary: true                  # Generate one-sentence summary per document
    # summary_model: summary_llm                    # LLM reference from llms: section (required if generate_summary is true)
    # summary_db: sqlite+aiosqlite:///./summaries.db  # Summary storage (SQLite or PostgreSQL)

    # Backend-specific options (each backend uses different fields):
    chroma_dir: /tmp/chroma_data              # llamaindex only
    rag_url: http://localhost:8081/v1         # foundational_rag only
    ingest_url: http://localhost:8082/v1      # foundational_rag only
    timeout: 120                              # foundational_rag only
    # verify_ssl: true                        # foundational_rag only (set false for self-signed certs)

    # opensearch_url: http://localhost:9200   # opensearch only
    # opensearch_auth_type: none              # none, basic, or sigv4
    # opensearch_index_prefix: aiq
    # opensearch_ingestion_mode: local        # local, dask, or auto
    # embed_model: nvidia/nemotron-3-embed-1b

You can also use environment variable substitution in YAML for deployment-specific values:

functions:
  knowledge_search:
    _type: knowledge_retrieval
    backend: foundational_rag
    rag_url: ${RAG_SERVER_URL:-http://localhost:8081/v1}
    collection_name: ${COLLECTION_NAME:-default}

Note: Each backend has different config options. Only the options matching your backend value are used - others are ignored (a warning will be logged). To add new config fields, edit KnowledgeRetrievalConfig in sources/knowledge_layer/src/register.py.

Collection Routing

AI-Q selects ingestion and retrieval collections independently. This routing policy applies consistently to all shipped knowledge backends: LlamaIndex, Foundational RAG, Azure AI Search, and OpenSearch.

The storage mapping is backend-specific: LlamaIndex and Foundational RAG use named collections, OpenSearch maps each collection to a physical index, and Azure AI Search isolates logical collections with collection_id filters inside one AI-Q-owned physical index.

Usage Ingestion target Retrieval target
Web UI UI-created session collection (s_<uuid>) Active UI session collection
API with a conversation-id header Collection named in /v1/collections/{collection_name}/documents conversation-id header value
API without conversation context Collection named explicitly by the ingestion operation Configured collection_name fallback

collection_name controls only the retrieval fallback. It does not choose an API ingestion destination, and it does not override an active UI session. Shipped profiles commonly populate it with ${COLLECTION_NAME:-test_collection}; the environment value is resolved when the workflow configuration is loaded.

Use COLLECTION_NAME for a deployment-wide retrieval default when API or CLI requests do not carry conversation context. To select a collection for an individual HTTP request, pass that collection name in the conversation-id header:

curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "conversation-id: research-papers" \
  -d '{"messages": [{"role": "user", "content": "Summarize the uploaded documents."}], "stream": false}'

For /v1/chat/completions, a conversation_id field in the JSON body is not used for collection routing. Use the conversation-id header instead.

Switching Backends

To switch backends, change the backend field and its corresponding options. Here are complete examples for each backend:

LlamaIndex (ChromaDB) - macOS/Linux

functions:
  knowledge_search:
    _type: knowledge_retrieval
    backend: llamaindex
    collection_name: my_docs
    top_k: 5
    chroma_dir: /tmp/chroma_data    # ChromaDB persistence directory

Foundational RAG (Hosted Server)

functions:
  knowledge_search:
    _type: knowledge_retrieval
    backend: foundational_rag
    collection_name: my_docs
    top_k: 5
    rag_url: http://your-server:8081/v1      # Rag server
    ingest_url: http://your-server:8082/v1   # Ingestion server
    timeout: 120

Azure AI Search (Managed Service)

functions:
  knowledge_search:
    _type: knowledge_retrieval
    backend: azure_ai_search
    collection_name: my_docs

Set AZURE_SEARCH_ENDPOINT and NVIDIA_API_KEY in the environment. Setting AZURE_SEARCH_API_KEY selects key authentication; otherwise Azure DefaultAzureCredential is used. The workload identity needs Search Service Contributor for index management and Search Index Data Contributor for document ingestion and retrieval. Embedding defaults can be shared with the LlamaIndex backend through AIQ_EMBED_BASE_URL and AIQ_EMBED_MODEL; set AIQ_EMBED_DIM when changing the model dimensions. Set a deployment-unique AIQ_AZURE_SEARCH_INDEX_PREFIX when multiple AI-Q deployments share a search service.

Azure stores all logical collections in one physical index selected by the prefix, schema version, embedding model, and dimension. Collection, file, and chunk manifests enforce logical isolation. Retrieval is always hybrid, and chunking is fixed at 1024 tokens with 128-token overlap.

Upload responses return canonical UUID file IDs. Same-name uploads coexist as independent files. Collection cleanup uses AIQ_COLLECTION_TTL_HOURS (24 hours by default) and AIQ_TTL_CLEANUP_INTERVAL_SECONDS (one hour by default), matching the other knowledge backends.

OpenSearch (Self-Hosted or AWS)

functions:
  knowledge_search:
    _type: knowledge_retrieval
    backend: opensearch
    collection_name: my_docs
    top_k: 5
    opensearch_url: ${OPENSEARCH_URL:-http://localhost:9200}
    opensearch_auth_type: ${OPENSEARCH_AUTH_TYPE:-none}
    opensearch_aws_region: ${AWS_REGION:-us-east-1}
    opensearch_aws_service: ${OPENSEARCH_AWS_SERVICE:-aoss}
    opensearch_index_prefix: ${OPENSEARCH_INDEX_PREFIX:-aiq}
    opensearch_embedding_dim: ${OPENSEARCH_EMBEDDING_DIM:-2048}
    opensearch_ingestion_mode: ${OPENSEARCH_INGESTION_MODE:-auto}
    opensearch_dask_scheduler_address: ${NAT_DASK_SCHEDULER_ADDRESS:-}
    embed_model: ${AIQ_EMBED_MODEL:-nvidia/nemotron-3-embed-1b}
    embed_base_url: ${AIQ_EMBED_BASE_URL:-https://integrate.api.nvidia.com/v1}

Use opensearch_auth_type: none only with a protected local development endpoint. Configure basic or sigv4 authentication for every remote, shared, or production OpenSearch deployment. For basic authentication, set OPENSEARCH_USERNAME and OPENSEARCH_PASSWORD. For AWS, use sigv4 and set opensearch_aws_service to es or aoss.

The embedding model's output dimension must match opensearch_embedding_dim (environment variable OPENSEARCH_EMBEDDING_DIM, default 2048) before the collection index is created. For example, if a test embedding response contains 2,048 values, keep the default; if it contains 1,024 values, set opensearch_embedding_dim: 1024 or OPENSEARCH_EMBEDDING_DIM=1024 before creating the collection. Use a new collection/index after changing dimensions because an existing knn_vector mapping cannot change its dimension. The full shipped profile is configs/config_web_opensearch.yml.

Changing the embedding model

Persisted vector stores are tied to both the embedding model and its output dimension. Changing only AIQ_EMBED_MODEL is not a compatible in-place update:

  • Chroma: delete only the affected logical collection through the Knowledge API or UI, then re-ingest its documents. Configuring a new AIQ_CHROMA_DIR also creates an isolated store. Deleting the existing shared AIQ_CHROMA_DIR removes every named collection in that store and can destroy unrelated data.
  • OpenSearch: set OPENSEARCH_EMBEDDING_DIM to the new model's exact output length, delete the existing AI-Q collection/index, and re-ingest every document. AI-Q rejects unmarked or incompatible indexes before ingestion or retrieval.
  • Azure AI Search: model and dimension are part of the physical index identity; changing either creates an isolated index that must be populated by re-ingestion.

OpenSearch ingestion is text-only: it extracts text from PDF, DOCX, PPTX, and supported plain-text formats, but does not perform LlamaIndex table/image/chart extraction. Distributed Dask ingestion also disables document-summary generation because the configured summary LLM is not serialized to workers; use local ingestion when summaries are required.

Multimodal Extraction (LlamaIndex Only)

By default, LlamaIndex ingests text only and uses the NVIDIA hosted embedding models. When AIQ_EXTRACT_IMAGES or AIQ_EXTRACT_CHARTS is enabled, a Vision Language Model (VLM) is used during ingestion to caption embedded images and extract structured data from charts (axis labels, data points, chart type). This makes visual content in PDFs searchable and retrievable alongside text. The VLM is only invoked at ingestion time, not at query time.

All options below can be overridden via environment variables:

Variable Default Description
Embedding
AIQ_EMBED_MODEL nvidia/nemotron-3-embed-1b NVIDIA embedding model
AIQ_EMBED_BASE_URL https://integrate.api.nvidia.com/v1 Embedding API base URL — override for local NIM
OPENSEARCH_EMBEDDING_DIM 2048 OpenSearch vector dimension; must equal the selected embedding model's output length before index creation
Extraction Flags
AIQ_EXTRACT_TABLES false Extract tables from PDFs as markdown
AIQ_EXTRACT_IMAGES false Extract and caption images with VLM
AIQ_EXTRACT_CHARTS false Classify images as charts and extract structured data
Vision Model
AIQ_VLM_MODEL nvidia/nemotron-3-nano-omni-30b-a3b-reasoning VLM for image captioning
AIQ_VLM_BASE_URL https://integrate.api.nvidia.com/v1 VLM API base URL — override for local NIM

When enabled, the startup log shows the active mode:

LlamaIndexIngestor initialized: persist_dir=/app/data/chroma_data, mode=text + tables + images

Note: AIQ_EXTRACT_IMAGES and AIQ_EXTRACT_CHARTS work together. If both are enabled, each image is classified by the VLM as either a chart or a regular image. Foundational RAG handles multimodal extraction server-side. OpenSearch performs text extraction only, so these flags apply only to the LlamaIndex backend.

Document Summaries

Document summaries help research agents understand what files are available before making tool calls. When enabled, the knowledge layer generates a one-sentence summary during ingestion and injects it into agent system prompts.

llms:
  summary_llm:
    _type: nim
    model_name: google/gemma-4-31b-it
    base_url: "https://integrate.api.nvidia.com/v1"
    temperature: 0.3
    max_tokens: 150

functions:
  knowledge_search:
    _type: knowledge_retrieval
    generate_summary: true
    summary_model: summary_llm     # Required: LLM reference from llms: section
    summary_db: ${AIQ_SUMMARY_DB:-sqlite+aiosqlite:///./summaries.db}

When generate_summary: true, you must configure summary_model to reference an LLM from the llms: section. For production deployments, use PostgreSQL for summary_db instead of SQLite.

For details on how summaries are stored, how agents consume them, and how to implement summaries in custom backends, refer to the SDK Reference - Document Summaries.

Supported File Types

File type support depends on the configured backend:

Backend Supported Types
LlamaIndex PDF, DOCX, TXT, MD, HTML, JSON, CSV
Foundational RAG PDF, DOCX, PPTX, TXT, MD, HTML, images (PNG, JPG)
OpenSearch PDF, DOCX, PPTX, TXT, MD, CSV, JSON, YAML, YML, LOG
Azure AI Search PDF, DOCX, TXT, MD

For custom backends, supported types are determined by the backend implementation.

Note: The backends support more types than the default upload allowlist. The frontend and backend API default to .pdf,.docx,.txt,.md (the common subset across all backends). Types like HTML, JSON, CSV, and images are supported by some backends but must be explicitly enabled and supported by the selected backend.

The frontend and backend API use the same upload controls:

Variable Effect
FILE_UPLOAD_ACCEPTED_TYPES Comma-separated extension allowlist; the API also validates declared and actual content
FILE_UPLOAD_MAX_SIZE_MB Maximum size of each file and of all files combined in one request
FILE_UPLOAD_MAX_FILE_COUNT Maximum number of files in one request

Set identical values for both application components:

Deployment Where to set
CLI (start_e2e.sh) deploy/.env
Docker Compose deploy/.env (passed to the frontend and backend containers)
Helm deploy/helm/deployment-k8s/values.yaml under both the backend and frontend apps' env sections

For Foundational RAG, add .pptx to include PowerPoint support: FILE_UPLOAD_ACCEPTED_TYPES=.pdf,.docx,.pptx,.txt,.md

Programmatic Usage

# Import the adapter module to trigger registration
from knowledge_layer.llamaindex import LlamaIndexRetriever, LlamaIndexIngestor

# Use the factory to get instances
from aiq_agent.knowledge import get_retriever, get_ingestor

# Ingest documents
ingestor = get_ingestor("llamaindex", config={"persist_dir": "/tmp/chroma"})
ingestor.create_collection("my_docs")
file_info = ingestor.upload_file("doc.pdf", "my_docs")

# Check ingestion status
status = ingestor.get_file_status(file_info.file_id, "my_docs")
print(f"Status: {status.status}")  # UPLOADING, INGESTING, SUCCESS, FAILED

# Retrieve
retriever = get_retriever("llamaindex", config={"persist_dir": "/tmp/chroma"})
result = await retriever.retrieve("query", "my_docs", top_k=5)
for chunk in result.chunks:
    print(f"{chunk.display_citation}: {chunk.content[:100]}")

Web UI Mode

Run the backend API server and frontend UI together for document upload, collection management, and chat.

Start Backend

# Foundational RAG example (requires deployed FRAG server)
# dotenv loads API keys (NVIDIA_API_KEY, etc.) from deploy/.env
# Additional env vars needed: RAG_SERVER_URL, RAG_INGEST_URL
dotenv -f deploy/.env run nat serve --config_file configs/config_web_frag.yml --host 0.0.0.0 --port 8000

Start Frontend

cd frontends/ui
npm run dev

Open http://localhost:3000 in your browser.

API Endpoints

Method Endpoint Description
POST /v1/collections Create collection
GET /v1/collections List collections
GET /v1/collections/{name} Get collection details
DELETE /v1/collections/{name} Delete collection
POST /v1/collections/{name}/documents Upload files
GET /v1/collections/{name}/documents List documents in collection
DELETE /v1/collections/{name}/documents Delete files
GET /v1/documents/{job_id}/status Poll ingestion status
GET /v1/knowledge/health Check knowledge backend health

Session Collections

All four shipped knowledge backends support session-based collections (s_<uuid>) created by the UI. Each UI conversation gets its own isolated logical collection; the physical storage mapping differs by backend as described in Collection Routing.

The active session collection is used for both UI ingestion and retrieval and takes precedence over the configured collection_name fallback.

TTL Cleanup

Collections inactive for 24 hours are auto-deleted based on updated_at timestamp. Background thread runs hourly.

COLLECTION_TTL_HOURS = 24
TTL_CLEANUP_INTERVAL_SECONDS = 3600

Architecture

Core Library (src/aiq_agent/knowledge/)

src/aiq_agent/knowledge/
    __init__.py        # Exports: Chunk, get_retriever, get_ingestor, etc.
    base.py            # Abstract classes: BaseRetriever, BaseIngestor
    schema.py          # Data models: Chunk, RetrievalResult, FileInfo, CollectionInfo
    factory.py         # Registry + factory: register_retriever(), get_retriever()
    summary_store.py   # SQLAlchemy-backed document summary persistence
File Purpose
base.py Defines the interface all backends must implement
schema.py Universal data models - backends convert native formats to these
factory.py Registration decorators + factory functions for instantiation
summary_store.py Persistent storage for document summaries (SQLite/PostgreSQL)

Backend Adapters (sources/knowledge_layer/src/)

sources/knowledge_layer/src/
    <backend_name>/
        __init__.py      # Imports adapter to trigger registration
        adapter.py       # @register_retriever/@register_ingestor decorated classes
        README.md        # Backend-specific documentation
        pyproject.toml   # Optional: isolated dependencies

How Registration Works

Backends register themselves using decorators when their module is imported:

# In adapter.py
from aiq_agent.knowledge.factory import register_retriever, register_ingestor

@register_retriever("my_backend")  # Registration name used in config
class MyRetriever(BaseRetriever):
    ...

@register_ingestor("my_backend")
class MyIngestor(BaseIngestor):
    ...

The registration name (for example, "my_backend") is what you use in:

  • YAML config: backend: my_backend
  • Factory calls: get_retriever("my_backend")

Important: The adapter module must be imported for registration to happen. This is why:

  1. __init__.py imports the adapter classes
  2. The NeMo Agent Toolkit function imports from knowledge_layer.<backend>.adapter

NeMo Agent Toolkit Integration

sources/knowledge_layer/src/
    register.py      # @register_function exposes retrieval to agents

The register.py defines KnowledgeRetrievalConfig which maps YAML config to backend instantiation.

Configuration

Configuration Precedence

Configuration values are resolved in the following order (highest to lowest priority):

  1. Explicit parameter - Values passed directly to factory functions (get_retriever("llamaindex"))
  2. YAML config file - The backend: field and other options in your workflow config (recommended)
  3. Environment variables - KNOWLEDGE_RETRIEVER_BACKEND, RAG_SERVER_URL, etc.
  4. Hardcoded defaults - Built-in fallback values

Recommendation: Use YAML config as your single source of truth for workflow configuration. Environment variables are useful for:

  • Container deployments (12-factor app pattern)
  • CI/CD overrides
  • Secrets management (API keys)

Environment Variables

Variable Backend Description
NVIDIA_API_KEY All Required for embeddings/VLM
KNOWLEDGE_RETRIEVER_BACKEND All Default retriever backend (fallback if not in YAML)
KNOWLEDGE_INGESTOR_BACKEND All Default ingestor backend (fallback if not in YAML)
AIQ_CHROMA_DIR llamaindex ChromaDB persistence path
AIQ_COLLECTION_TTL_HOURS all local/managed backends Hours before stale collections are deleted (default: 24)
AIQ_TTL_CLEANUP_INTERVAL_SECONDS all local/managed backends Collection cleanup interval (default: 3600)
RAG_SERVER_URL foundational_rag Query server URL (port 8081)
RAG_INGEST_URL foundational_rag Ingestion server URL (port 8082)
OPENSEARCH_URL opensearch OpenSearch endpoint URL
OPENSEARCH_AUTH_TYPE opensearch none, basic, or sigv4
OPENSEARCH_USERNAME, OPENSEARCH_PASSWORD opensearch Credentials for basic authentication
AWS_REGION, OPENSEARCH_AWS_SERVICE opensearch SigV4 region and service (es or aoss)
OPENSEARCH_INDEX_PREFIX opensearch Prefix for AI-Q-managed indexes
OPENSEARCH_INGESTION_MODE opensearch local, dask, or auto
OPENSEARCH_DASK_SCHEDULER_ADDRESS opensearch Optional Dask scheduler for distributed ingestion
AIQ_EMBED_MODEL, AIQ_EMBED_BASE_URL llamaindex, opensearch, azure_ai_search Embedding model and endpoint
COLLECTION_NAME All Default retrieval collection when no conversation or session context is present

Troubleshooting

Issue Cause Fix
Unknown backend: my_backend Adapter not imported/registered Import the adapter module before calling factory
ormsgpack attribute error Version conflict with LangGraph uv pip install "ormsgpack>=1.5.0"
Empty retrieval results Collection empty Run ingestion first, verify collection name matches
Job status 404 Different process/instance Factory uses singletons - ensure same process
milvus-lite required Missing dependency uv pip install "pymilvus[milvus_lite]"
opensearchpy import error OpenSearch extra not installed uv pip install -e "sources/knowledge_layer[opensearch]"
OpenSearch 401 or 403 Auth mode, credentials, IAM, or AOSS data-access policy mismatch Verify opensearch_auth_type; for AOSS follow the IAM and data-access steps in the deployment guide
Backend registered twice Module imported multiple times Normal - factory logs warning but works fine

Debug Registration

# Check what's registered
from aiq_agent.knowledge.factory import list_retrievers, list_ingestors, get_knowledge_layer_config

print("Retrievers:", list_retrievers())
print("Ingestors:", list_ingestors())
print("Full config:", get_knowledge_layer_config())

Related Documentation

Document Description
SDK Reference Build custom backend adapters - data schemas, interfaces, full implementation example
Foundational RAG Setup (sources/knowledge_layer/src/foundational_rag/README.md) Production deployment with NVIDIA RAG Blueprint
Amazon OpenSearch Serverless Deploy the OpenSearch backend on EKS with AOSS and SigV4