Skip to content

feat(extract): tree-sitter-based Dart extraction with source_location line anchors#2114

Open
cornwe19 wants to merge 1 commit into
Graphify-Labs:v8from
cornwe19:dart-tree-sitter
Open

feat(extract): tree-sitter-based Dart extraction with source_location line anchors#2114
cornwe19 wants to merge 1 commit into
Graphify-Labs:v8from
cornwe19:dart-tree-sitter

Conversation

@cornwe19

Copy link
Copy Markdown
Contributor

What

Dart is currently the one mainstream language whose extractor emits source_location: None on every node and edge — extract_dart is a ~520-line regex extractor predating the generic tree-sitter framework, so Dart symbols can't be jumped to or read by line range downstream, unlike Java/Kotlin/Swift/etc.

This PR makes tree-sitter the primary Dart path, using the community tree-sitter-dart grammar as a new optional [dart] extra (prebuilt wheels for win/macOS/Linux, x86+arm):

  • AST-driven with line anchors: declarations (classes, mixins, enums, extension types, extensions, typedefs), imports/exports, part of redirection, annotations, and functions/factory constructors all come from the AST with L{n} anchors on nodes and edges.
  • Proven heuristics retained: the Flutter-framework sweeps (Bloc events/emissions, Riverpod providers, navigation routes) and Dart 3 pattern-destructured variables — which the 0.1.x grammar still parses with ERROR nodes — stay on the existing regexes, now line-anchored via match offsets and scoped by the AST.
  • Regex fallback preserved verbatim: without the extra installed, extraction falls back to the previous implementation unchanged (_extract_dart_regex) — the same optional-grammar pattern as extract_pascal (feat(extract): add Pascal/Delphi and Lazarus IDE support #781). No new required dependency.

Fixes along the way

The AST also closes two regex false-positive classes:

  • Declarations inside string literals (e.g. lint-test fixtures with code in '''...''') no longer produce phantom class/function nodes.
  • Multi-line named/optional parameter lists (required String id, sits at the 0–2-space indent the variable regex scans) no longer leak parameters as variable nodes, including a case that produced labels containing newlines.

Validation

  • All existing tests/test_dart.py expectations pass on both paths (grammar installed and absent); new tests cover anchor stamping, part of redirection with anchors, the extension family (named/anonymous/generic), named/optional parameters, factory vs named constructors, and generators.
  • Ran against a large production Flutter codebase (2,780 .dart files): 100% of locally-defined nodes and all edges anchored, 100% anchor spot-check (2,547 sampled symbols found on their anchored line), node counts within 0.5% of the regex baseline (the delta is the string-literal phantoms the AST correctly drops), and the tree-sitter path ran slightly faster than the regex path (~5s vs ~5s for the whole repo, same order).
  • Error recovery verified: files the grammar can't fully parse still extract (localized ERROR nodes), and any walker exception falls back to the regex extractor per file, so a grammar gap can never cost a file entirely.

uv.lock regenerated with uv lock (adds tree-sitter-dart to the all/dart extras; the surrounding marker churn is current uv re-normalizing environment markers).

🤖 Generated with Claude Code

… line anchors

Dart was the one mainstream language whose extractor emitted
source_location: None on every node and edge — a ~520-line regex
extractor predating the generic tree-sitter framework — so Dart symbols
couldn't be jumped to or read by line range downstream.

extract_dart now parses with the community tree-sitter-dart grammar
(optional [dart] extra, prebuilt wheels for all platforms):
declarations (classes, mixins, enums, extension types, extensions,
typedefs), imports/exports, `part of` redirection, annotations, and
functions come from the AST with L{n} anchors on nodes and edges. The
Flutter-framework heuristics (Bloc/Riverpod/navigation) and Dart 3
pattern-destructured variables — which the young grammar still parses
with ERROR nodes — stay on the proven regexes, now line-anchored via
match offsets and scoped by the AST (declarations inside string
literals and multi-line named-parameter lists no longer produce
phantom nodes).

Without the extra installed, extraction falls back to the previous
regex extractor, preserved verbatim as _extract_dart_regex — the same
optional-grammar pattern as extract_pascal (Graphify-Labs#781).

Validated on a large production Flutter codebase (2,780 files):
100% of locally-defined nodes and all edges anchored, anchor spot-check
100% (2,547 sampled symbols on their anchored line), node counts within
0.5% of the regex baseline, and extraction slightly faster than the
regex path. All existing dart tests pass on both paths.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant