Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 9 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,11 @@ keywords = ["codebase", "graph", "mcp", "cli", "analysis"]
categories = ["command-line-utilities", "development-tools"]
include = [
"src/**",
"!src/**/.DS_Store",
"tests/**",
"assets/*.json",
"assets/ladybug-extensions/0.19.0/**/*.xz",
"vendor/tree-sitter-wat/**",
"build.rs",
"README.md",
"LICENSE",
Expand Down Expand Up @@ -47,10 +49,17 @@ tree-sitter-cpp = "0.23.4"
tree-sitter-css = "0.25.0"
tree-sitter-fortran = "0.6.0"
tree-sitter-go = "0.25.0"
tree-sitter-html = "0.23.2"
tree-sitter-javascript = "0.25.0"
tree-sitter-language = "0.1"
tree-sitter-python = "0.25.0"
tree-sitter-rust = "0.24.2"
tree-sitter-typescript = "0.23.2"
walkdir = "2"

[build-dependencies]
cc = "1.2"

[target.'cfg(windows)'.dependencies]
windows-sys = { version = "0.61", features = ["Win32_Foundation", "Win32_System_ProcessStatus", "Win32_System_Threading"] }

Expand Down
7 changes: 4 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
# codebaseGraph

`codebaseGraph` turns a local source repository into a searchable code graph for
AI coding agents. It indexes Python, Rust, Go, C, C++, Fortran, CSS, Markdown,
and MDX, then exposes compact context, schema information, query helpers, and
bounded read-only graph queries through a native CLI and MCP server.
AI coding agents. It indexes Python, Rust, Go, C, C++, Fortran, CSS, HTML,
JavaScript, JSX, TypeScript, TSX, WebAssembly Text, Markdown, and MDX, then
exposes compact context, schema information, query helpers, and bounded
read-only graph queries through a native CLI and MCP server.

This workspace also ships `k-wiki`, an optional subsystem for curated repository
knowledge. The graph and wiki have separate source and generated state.
Expand Down
27 changes: 27 additions & 0 deletions build.rs
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
fn main() {
build_tree_sitter_wat();

if !cfg!(windows) && std::env::var_os("LBUG_SHARED").is_none() {
println!("cargo:rustc-link-arg=-rdynamic");
}
Expand Down Expand Up @@ -32,6 +34,31 @@ fn main() {
}
}

fn build_tree_sitter_wat() {
let source_dir = std::path::Path::new("vendor/tree-sitter-wat/src");
let parser = source_dir.join("parser.c");
let scanner = source_dir.join("scanner.c");

let mut build = cc::Build::new();
build
.std("c11")
.include(source_dir)
.flag_if_supported("-Wno-unused-parameter")
.file(&parser)
.file(&scanner);
if std::env::var("CARGO_CFG_TARGET_ENV").as_deref() == Ok("msvc") {
build.flag("-utf-8");
}
build.compile("tree-sitter-wat");

println!("cargo:rerun-if-changed={}", parser.display());
println!("cargo:rerun-if-changed={}", scanner.display());
println!(
"cargo:rerun-if-changed={}",
source_dir.join("tree_sitter/parser.h").display()
);
}

fn link_macos_x86_compiler_runtime() {
println!("cargo:rerun-if-env-changed=DEVELOPER_DIR");
let output = std::process::Command::new("xcrun")
Expand Down
27 changes: 27 additions & 0 deletions knowledge/architecture/language-html.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
---
description: Built-in HTML discovery, grammar, syntax-catalog, and conservative graph-normalization contract.
resource: repository-architecture
tags:
- architecture
- html
- language-support
- parser
- tree-sitter
timestamp: 2026-08-25
title: HTML Parsing Contract
type: architecture
---
# HTML Parsing Contract

The Graph Runtime indexes HTML with a built-in Tree-sitter profile while preserving the distinction between raw markup syntax and higher-level application components.

## Public contract

- `html` recognizes `.html` and `.htm` using `tree_sitter_html@0.23.2`.
- The key is advertised by the CLI syntax catalog and MCP `graph_syntax` schema.
- Ordered raw syntax includes documents, doctypes, elements, tags, scripts, styles, attributes, and text.
- HTML elements are not promoted to ontology `Component` nodes because markup alone does not establish framework or component identity.

Embedded JavaScript and CSS injection parsing is outside this contract; standalone files are handled by their own language profiles.

Related: [Graph Runtime](./graph-runtime.md) and [Repository Ownership Map](./repository-map.md).
28 changes: 28 additions & 0 deletions knowledge/architecture/language-javascript.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
---
description: Built-in JavaScript and JSX discovery, grammar, catalog, and semantic-normalization contract.
resource: repository-architecture
tags:
- architecture
- javascript
- jsx
- language-support
- parser
- tree-sitter
timestamp: 2026-08-25
title: JavaScript and JSX Parsing Contract
type: architecture
---
# JavaScript and JSX Parsing Contract

The Graph Runtime exposes JavaScript as one built-in syntax language backed by the Tree-sitter JavaScript grammar, which includes JSX syntax.

## Public contract

- `javascript` recognizes `.js`, `.jsx`, `.mjs`, and `.cjs` using `tree_sitter_javascript@0.25.0`.
- The key is advertised by the CLI syntax catalog and MCP `graph_syntax` schema.
- Classes, functions, generator functions, methods, imports, and calls map to established ontology nodes.
- JSX elements remain available through ordered raw syntax captures; they are not promoted to framework-specific component nodes.

Framework inference and embedded-language extraction are outside this contract.

Related: [Graph Runtime](./graph-runtime.md) and [Repository Ownership Map](./repository-map.md).
28 changes: 28 additions & 0 deletions knowledge/architecture/language-typescript-tsx.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
---
description: Built-in TypeScript and TSX discovery, grammar, catalog, and semantic-normalization contract.
resource: repository-architecture
tags:
- architecture
- language-support
- parser
- tree-sitter
- typescript
- tsx
timestamp: 2026-08-25
title: TypeScript and TSX Parsing Contract
type: architecture
---
# TypeScript and TSX Parsing Contract

The Graph Runtime treats TypeScript and TSX as sibling built-in syntax languages so each file selects the correct Tree-sitter grammar and node catalog without changing the one-grammar-per-language profile contract.

## Public contract

- `typescript` recognizes `.ts`, `.mts`, and `.cts` using `tree_sitter_typescript@0.23.2`.
- `tsx` recognizes `.tsx` using the TSX grammar and node catalog from the same package version.
- Both keys are advertised by the CLI syntax catalog and MCP `graph_syntax` schema.
- TypeScript declarations, interfaces, enums, type aliases, functions, methods, imports, and calls map to established ontology nodes. JSX/TSX elements remain available through ordered raw syntax captures.

Framework-specific template semantics and embedded-language extraction are outside this contract.

Related: [Graph Runtime](./graph-runtime.md) and [Repository Ownership Map](./repository-map.md).
35 changes: 35 additions & 0 deletions knowledge/architecture/language-webassembly.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
---
description: Built-in WebAssembly Text discovery, vendored grammar, syntax-catalog, and semantic-normalization contract.
resource: repository-architecture
tags:
- architecture
- language-support
- parser
- tree-sitter
- webassembly
- wat
timestamp: 2026-08-25
title: WebAssembly Text Parsing Contract
type: architecture
---
# WebAssembly Text Parsing Contract

The Graph Runtime indexes WebAssembly Text files through a built-in profile backed by a pinned, vendored Tree-sitter WAT grammar.

## Public contract

- `webassembly` recognizes `.wat` files using `tree_sitter_wat@0.1.0+e3769473`.
- The language key is advertised by the CLI syntax catalog and MCP `graph_syntax` schema.
- Named modules, functions, and type definitions map to `Module`, `Function`, and `TypeAlias` ontology nodes.
- Imports normalize to `module.item`; exports use their declared string name; direct `call` and `return_call` instructions use the referenced index or identifier.
- Anonymous modules, functions, and type definitions remain available as ordered raw syntax without fabricated semantic identities.

Binary `.wasm` decoding, `.wast` scripts, validation, execution, and disassembly are outside this contract.

## Vendored grammar

The generated native parser, external scanner, headers, node catalog, license, and exact upstream provenance live under `vendor/tree-sitter-wat`. The grammar is pinned to upstream commit `e3769473b2d90643d8af500b5cfc2f25a674888a` and compiled by the package build script so crates.io source packages build offline.

The upstream payload-free scanner callbacks are locally corrected to return zero explicitly. Upstream and patched SHA-256 values are retained in the vendor provenance document. Grammar updates must replace the generated set atomically, preserve license and provenance, verify ABI compatibility, and pass parser, native, and crates.io package checks.

Related: [Graph Runtime](./graph-runtime.md), [Repository Ownership Map](./repository-map.md), and [Native Release Verification](./release-verification.md).
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
---
agent_memory:
version: 1
kind: procedural
scope: repository
status: candidate
owner: codex
created_at: 2026-08-25T12:39:22+09:30
last_verified_at: null
verified_by: null
review_after: null
supersedes: []
superseded_by: null
sources:
- kind: source
reference: Cargo.toml package include = ["src/**", ...]
content_hash: null
- kind: runtime-observation
reference: 2026-08-25 cargo publish dry-run for PR branches 101-104
content_hash: null
- kind: pull-request
reference: https://github.com/rabii-chaarani/codebaseGraph/pull/101
content_hash: null
history: []
description: Ignored Finder metadata under src/** can make cargo publish dirty and contaminate exact package measurements even when git status is clean.
tags:
- cargo
- packaging
- release-verification
- clean-worktree
timestamp: 2026-08-25T12:39:22+09:30
title: Verify Cargo packages from a clean worktree when include globs are broad
type: agent-memory
---
When `Cargo.toml` uses a broad package allowlist such as `src/**`, ignored untracked files beneath that tree can still make `cargo publish --dry-run` report a dirty package. During the web-language PRs, local `src/.DS_Store` and `src/adapters/.DS_Store` triggered this condition even though ordinary `git status` was clean.

For release-representative verification, preserve user metadata and run `cargo publish --dry-run --locked` from a clean detached worktree at the exact commit. Then pass the produced `.crate` to the xtask size gate. When verifying a version already present on crates.io, Cargo may retain the archive under `target/package/tmp-crate/`; use the path Cargo actually produced rather than assuming the top-level package path.
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
---
agent_memory:
version: 1
kind: procedural
scope: repository
status: active
owner: codex
created_at: 2026-08-25T13:21:26+09:30
last_verified_at: 2026-08-25T13:21:40+09:30
verified_by: codex
review_after: null
supersedes: []
superseded_by: null
sources:
- kind: source
reference: vendor/tree-sitter-wat/src/scanner.c
content_hash: null
- kind: documentation
reference: vendor/tree-sitter-wat/UPSTREAM.md
content_hash: null
- kind: source
reference: build.rs and Cargo.toml
content_hash: null
- kind: test
reference: cargo test --workspace --locked and cargo publish --dry-run --locked --allow-dirty
content_hash: null
history:
- from: candidate
to: active
actor: codex
at: 2026-08-25T13:21:40+09:30
reason: Reviewed the callback claim against the pinned upstream scanner, the documented upstream and patched hashes, the local C patch, a warning-clean build, the passing full workspace suite, and the verified extracted source package.
description: Vendored Tree-sitter external scanners must return valid payload and serialization values even when upstream builds only warn.
tags:
- c
- ffi
- parser
- supply-chain
- tree-sitter
- vendoring
timestamp: 2026-08-25T13:21:26+09:30
title: Verify external-scanner callback returns when vendoring Tree-sitter grammars
type: agent-memory
---
When vendoring generated Tree-sitter grammar sources, compile the external scanner with warnings enabled and inspect every callback against Tree-sitter's scanner ABI. The pinned `g-plane/tree-sitter-wat` scanner had two payload-free callbacks that compiled but violated their C return contracts: `tree_sitter_wat_external_scanner_create` fell off a non-void function, and `serialize` returned `1` without writing a byte. This can propagate an indeterminate payload pointer or nondeterministic serialized state.

For a stateless scanner, make `create` return null/zero and `serialize` return zero. Preserve the upstream source hash and the patched vendored hash in provenance rather than presenting the local file as byte-for-byte upstream. Then verify the focused parser tests, full native workspace suite, extracted crates.io package build, and package-size gate. Recheck the callbacks and provenance whenever the grammar commit changes.
2 changes: 1 addition & 1 deletion src/adapters/cli/format/help.rs
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ pub(in crate::adapters::cli) fn graph_schema_help() -> &'static str {
}

pub(in crate::adapters::cli) fn graph_syntax_help() -> &'static str {
"codebase-graph syntax\n\nUSAGE:\n codebase-graph syntax <language> [--format json|block] [--json] [--pretty]\n\nOPTIONS:\n <language> One of c, cpp, css, fortran, go, markdown, python, or rust\n --format <format> block or json; defaults to block\n --json Emit compact JSON output\n --pretty Pretty-print JSON output"
"codebase-graph syntax\n\nUSAGE:\n codebase-graph syntax <language> [--format json|block] [--json] [--pretty]\n\nOPTIONS:\n <language> One of c, cpp, css, fortran, go, html, javascript, markdown, python, rust, tsx, typescript, or webassembly\n --format <format> block or json; defaults to block\n --json Emit compact JSON output\n --pretty Pretty-print JSON output"
}

pub(in crate::adapters::cli) fn graph_query_helpers_help() -> &'static str {
Expand Down
Loading