feat(app): DQX Studio dqlake integration

[berrybluecode]

Summary

This branch brings DQX Studio from the thinner Studio shell on main to the full dqlake-parity product (~84 non-merge commits, ~757 files). Relative to main, it adds the governed rule lifecycle, monitored-table / collection workflows, equal-rule-weight scoring, Results/Genie, RBAC, Marketplace, demo seed, and a complete Studio docs section.

Platform & governance

• Rules Registry — reusable rules with draft → pending → approved lifecycle, versioning, revoke, import/export, and ODCS/contract import
• Monitored tables & Collections (Table Spaces) — apply rules, pin versions, schedules, approval queues, steward ownership
• RBAC & object permissions — roles, entitlements, object-level grants, steward grant UX
• Lakebase / BYO warehouse — Postgres OLTP path updates, external SQL warehouse support, deploy/ops docs

Quality results

• Equal-rule-weight DQ score — row-level pass rates vs table-level binary verdicts; score cache + UC metric views
• Results UI — table/collection/rule/global breakdowns, quarantine samples, SQL-pushdown failed-row filters
• Ask Genie — Genie space provisioning + in-app chat over results

Authoring & AI

• Built-in / visual builder / custom SQL (including cross-table and row vs table-level)
• AI rule suggestion, AI SQL write/improve (with granularity context), native rule tests
• Apply-on-tag (governed tags), pass thresholds, Labels/tags fixes
• Rules Marketplace — curated reusable packs with import + already-added UX

Docs, demo & polish

• New Studio docs (start-here, authoring, monitoring, running, governance, what’s new) including quality-score and cross-table guides
• Built-in demo seed content and many Studio UX/perf/bug-bash fixes

Resolves

#1296

Test plan

• Fresh deploy from this branch (Lakebase + warehouse); app starts, migrations apply
• Rules Registry: create (built-in / visual / SQL), submit, approve, revoke, edit-in-place revision, import/export
• Cross-table authoring (FK, visual join, SQL row-level + table-level); apply and run on a monitored table
• AI suggest + AI SQL Write/Improve with Applies-to toggle;
• Monitored table + Collection: apply rules, schedule, Run now, Drafts & Review approve/reject/revoke
• Results: DQ score for mixed row/table-level rules; breakdowns; failed-record drill-down; Ask Genie
• Permissions: role gating + object grants / steward; Marketplace pack import
• Smoke Studio docs locally (docs/dqx) — especially quality-score, cross-table, permissions
• Rebase/merge main and resolve conflicts (branch is ~88 ahead / ~30 behind)

[ghanse]

This looks good overall, a few findings below:

SQL Parameterization and Escaping

Parameterized SQL statements are built using Python f-strings and called through _sql.execute(...) and _sql.query(...)`. Any future safety mechanisms we introduce will require updates across every f-string.

We could add a small query-builder to handle the common CRUD statements. For example:

# current:
self._sql.execute(
  f"UPDATE {self._table} SET pinned_version = {value} WHERE id = '{e}'"
)

# suggested:
self._sql.update(  # builds and executes the UPDATE statement
  self._table, set={"pinned_version": value}, where={"id": rule_id}
)

Code Quality

A few minor code quality nits. Verify if these need fixing:

• There are several very large files with >1,200 lines of code
• There are several instances of linter silencing
• There are a lot of except Exception blocks that only log

[vb-dbrks]

I built a golden set eval for the rule suggester and ran it live on field-eng against real endpoints. Numbers below. Short version is that I think the default model choice is a problem, and there is a bug behind it that means some users get nothing at all.

This replaces my earlier review ... everything from that one was addressed, thanks for the quick turnaround.

The default is broken on wide tables

AI_ENDPOINT_NAME_DEFAULT is databricks-gpt-5-4-nano, and get_ai_enabled returns True when unset, so a fresh deploy has AI on and pre-pointed at nano.

On a 90 column table nano returns zero suggestions. available comes back False with "AI judge returned an unparsable response". Reproduced three times out of three.

The cause is that the judge's output size scales with column count while max_tokens stays at a fixed 2048. I truncated a synthetic table to different widths to find where it breaks:

columns	10	20	30	45	64	90
completion_tokens	1174	1599	1344	1831	2048 capped	2048 capped
usable	yes	yes	yes	yes	NO	NO

So anything past roughly fifty columns loses everything. With thousands of tables in a real estate I don't think 100 column event and feature tables are an edge case.

It is also mis-reported. AIGateway only raises its budget specific error when the content is empty and finish_reason says length. This endpoint returns non empty truncated JSON and leaves finish_reason unset, so we fall through to the generic parse error and the user is told the response was unparsable rather than that the answer did not fit. That sends whoever debugs it in the wrong direction.

And it is not just nano

I scored six candidate judges against 48 labelled answers, embeddings on gte-large-en throughout:

judge	precision	recall	f1	suggestions	decoys taken	90 col table	secs
gpt-5-4-nano (current default)	0.456	0.854	0.594	90	3 of 7	FAILS	45
gpt-5-4-mini						FAILS
gpt-5-4	0.478	0.896	0.623	90	0	ok	55
claude-haiku-4-5	0.618	0.875	0.724	68	2 of 7	ok	45
claude-sonnet-4-5	0.595	0.917	0.721	74	1 of 7	ok	81
claude-opus-4-7	0.469	0.938	0.625	96	0	ok	87

gpt-5-4-mini fails the wide table the same way nano does, so stepping up one size in that family does not fix it. Only the full gpt-5-4 survives there.

Two things I did not expect. A more capable model makes over suggestion worse, not better ... opus returned 96 suggestions for 48 correct answers, the most of any model, because it follows the "apply universal checks broadly" instruction more thoroughly. So over suggestion is a prompt problem and we cannot buy our way out of it with a bigger model. And haiku beats opus on the number a reviewer actually feels, best precision and fewest suggestions to wade through.

The decoys column is the one I care about most. Those are seven plausible but wrong column bindings I planted, things like the email format rule on a free text note field, or the uniqueness rule on a foreign key that legitimately repeats. Every one of them passes all our structural checks, because each is a real column of the right family with all slots filled, so post process cannot catch them and a reviewer might well approve them.

What I am asking for

1. Fix the output budget so it scales with column count, and fix the error path so truncation does not report as unparsable.
2. Raise the default. I would go with claude-sonnet-4-5 ... recall 0.917, only one wrong column binding, survives wide tables. Haiku has better precision and half the latency, but it took two of the seven decoys, and I would rather trade some speed than ship a default that is twice as likely to produce a wrong binding a reviewer approves.
3. MAX_RETRIEVAL_COLUMNS, see the inline comment. It costs 25% recall and does not bound the thing that is actually breaking.

Raising the default alone is not enough, because the endpoint is an admin dropdown ... anyone who picks nano reinstates a silently broken deploy. Both need doing.

Caveats

One run per model. From repeat runs, recall and the wide table pass or fail are stable, but precision drifts about 0.02 to 0.04, so please do not pick between haiku and sonnet on precision alone. gpt-5-4-mini failing is a single observation where nano's is three.

Read the precision column as a floor rather than a score. My answer key labels the rules a table must have, while the prompt tells the judge to apply universal checks to every column a reasonable analyst would consider required, so defensible suggestions get counted against it. Recall is the number to trust.

The eval is in this PR

I pushed it to this branch. tests/test_rule_suggester_eval.py is the deterministic half, 24 tests, runs inside make app-test in about three seconds with no workspace and no tokens, and it holds a committed baseline so a retrieval or post process change shows up as a number. tests/ai_eval is the live half behind make app-ai-eval, gated on an env var so it cannot fire by accident. Everything above is reproducible from it, and the model comparison is one command.

Happy to walk anyone through it, and happy to be told the fixture is wrong somewhere ... the answer key is a judgement call in places and I would rather argue about it with the harness in front of us.

[vb-dbrks]

LGTM. There needs to be a follow up PR in the future to fail the harness tests on regression with the quality of AI suggestions.

[laurencewells]

LGTM