Skip to content

Composite merchant keys for same-name multi-category rules [5/6] - #102

Open
terryaney wants to merge 10 commits into
davidfowl:mainfrom
terryaney:feature/merchant-composite-keys
Open

Composite merchant keys for same-name multi-category rules [5/6]#102
terryaney wants to merge 10 commits into
davidfowl:mainfrom
terryaney:feature/merchant-composite-keys

Conversation

@terryaney

@terryaney terryaney commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

[5/6] Based on feature/charts-reimagined (#101).
Merging this PR into main also lands #98, #99, #100, and #101 if not already merged.
This layer only: feature/charts-reimagined...feature/merchant-composite-keys

Fixes #88 — same-named merchant rules with different categories now work:

[Costco]
match: contains("COSTCO") and amount <= 200
category: Food
subcategory: Grocery

[Costco]
match: contains("COSTCO") and amount > 200
category: Shopping
subcategory: Wholesale

Changes

  • analyzer.pyby_merchant uses a composite (merchant, category, subcategory) key so same-named merchants with different categories appear as separate report rows
  • spending_report.jsfilteredViewTotals and chartAggregations now use per-transaction txn.tags instead of the merchant-level tag union, fixing misclassification when one transaction's tags don't apply to all transactions at that merchant
  • commands/explain.py — follows by_merchant to composite keys, so a name that now resolves to several entries can return all of them

Behavior change — tally explain --format json output shape

Intentional, and required by the fix: one merchant name can now legitimately resolve to more than one entry, which a single object cannot express. Merchant queries now emit a wrapper:

{
  "query": "Costco",
  "match_mode": "exact",
  "matched_names": ["Costco"],
  "merchants": [ { "name": "Costco", "category": "Food", ... } ]
}

where it previously emitted a bare merchant object. match_mode is exact, case_insensitive, or partial.

This also repairs output that was already invalid: the pre-existing partial-match path printed a human-readable Merchants matching 'x': header followed by several concatenated JSON objects, even under --format json, which no parser could consume. That path now emits one well-formed document.

Nothing persisted changes — explain computes from config and data files at run time and reads no output of tally up, so there is no mixed-version concern. The only thing that breaks is a script parsing the old single-object shape from a previous release, and the wrapper is deliberately uniform rather than switching shape on match count, so such a script has one form to move to rather than two.

Stack

# Branch PR Description Diff
1 feature/globbing-documentation #98 Glob pattern docs and CLI tests from main
2 feature/report-json-determinism #99 Deterministic report JSON from PR98 · from main
3 feature/ui-tweaks #100 Date filter, transaction details, per-txn tags from PR99 · from main
4 feature/charts-reimagined #101 Reimagined charts and KPIs from PR100 · from main
5 feature/merchant-composite-keys #102 Composite merchant keys from PR101 · from main
6 feature/categorization #103 Categorization review file from PR102 · from main

Independent: feature/ci-repair#97 — fixes fork PR builds.

- Add glob pattern examples in settings.yaml.example, clarify formats docs for file globs.
- Add CLI tests covering multi-file matching, no-match behavior, diag visibility, and sorted processing order.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Introduces composite merchant identities so same-name merchants remain distinct across categories, alongside stacked report enhancements.

Changes:

  • Groups merchants by name, category, and subcategory with stable report IDs.
  • Adds per-transaction tag classification, recurrence analysis, new charts, and date filtering.
  • Adds report_fields, deterministic output, glob tests/docs, and chart documentation.

Reviewed changes

Copilot reviewed 28 out of 41 changed files in this pull request and generated 9 comments.

Show a summary per file
File Description
src/tally/analyzer.py Adds composite keys and recurrence inference.
src/tally/report.py Generates stable merchant IDs and report data.
src/tally/spending_report.html Adds report controls, charts, and details UI.
src/tally/spending_report.css Styles the redesigned report.
src/tally/spending_report.js Implements filtering, charts, and UI state.
src/tally/parsers.py Surfaces configured report fields.
src/tally/format_parser.py Adds report_fields metadata.
src/tally/config_loader.py Resolves global and per-source report fields.
src/tally/merchant_engine.py Suppresses empty computed fields.
src/tally/merchant_utils.py Adds rule provenance metadata.
src/tally/templates.py Updates starter configuration.
src/tally/cli_utils.py Updates starter title generation.
src/tally/commands/run.py Deduplicates report source names.
src/tally/commands/explain.py Supports composite merchants in explanations.
src/tally/commands/diag.py Documents additional special tags.
src/tally/commands/reference.py Documents fields and recurrence tags.
src/tally/commands/workflow.py Shows recurrence tags in workflow help.
tests/test_report.py Tests stable, unambiguous merchant IDs.
tests/test_cli.py Tests globs and composite explain output.
tests/test_analyzer.py Tests determinism, fields, keys, and recurrence.
config/settings.yaml.example Documents globs, fields, and recurrence tags.
docs/charts.html Adds the chart guide.
docs/formats.html Documents globs and report fields.
docs/reference.html Documents report-field behavior.
docs/index.html Updates assets and chart navigation.
docs/guide.html Adds chart navigation.
docs/quickstart.html Links the chart guide.
docs/sitemap.xml Adds chart documentation and image URLs.
Suppressed comments (3)

src/tally/spending_report.html:90

  • Each month is implemented as a click-only <div>, making month selection unavailable from the keyboard. Render these cells as buttons with an accessible selected state, or provide equivalent focus and Enter/Space handling.
    src/tally/spending_report.html:151
  • This element declares role="button" but has neither a tab stop nor keyboard activation; only a JavaScript click listener clears peek mode. Add tabindex="0" plus Enter/Space handling, or use a real button.
    src/tally/spending_report.html:260
  • The Transaction Details collapse header is a clickable <div> with no focus or keyboard handling. Use a semantic button, or make it focusable and support Enter/Space while exposing the expanded state.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/tally/report.py Outdated
Comment thread src/tally/config_loader.py Outdated
Comment thread src/tally/commands/explain.py
Comment thread src/tally/spending_report.js
Comment thread src/tally/spending_report.html Outdated
Comment thread src/tally/spending_report.html Outdated
Comment thread docs/reference.html Outdated
Comment thread docs/charts.html Outdated
Comment thread src/tally/spending_report.html Outdated
@terryaney terryaney mentioned this pull request Aug 4, 2026
terryaney and others added 9 commits August 4, 2026 16:18
The data-source table's file row used markdown backticks, which render
literally in an HTML page. Every other cell in that table already uses
<code> elements.
`tally up` writes <output>.json beside the HTML and diffs the next run
against it, so the same config and data must produce the same bytes. It
did not: pattern.tags serialized in arbitrary order.

- analyzer.py: sort pattern.tags unconditionally. The isinstance(set)
  guard was dead - merchant_utils.normalize_merchant already does
  list(result.tags), so the value arrives as a list carrying the set's
  arbitrary iteration order. Mirrors the top-level merchant tags field,
  which was already sorted.
- run.py: extract collect_source_names() and dedupe the report subtitle
  sources while preserving settings.yaml declaration order.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…le Info

1. BIGGEST UI GAIN: Revamped date filter to be more functional than simply just a month list
2. IMPORTANT DATA FIX: Two data-correctness bugs - in both, the JS classified transactions using
   merchant.tags — the union of every tag across all of a merchant's transactions — instead of the
   transaction's own txn.tags (see Bug Fix Details below))
3. Removed year from title to support multi-year data/reporting and made page title match content title
4. Data from... subtitle is distinct list of source
5. `Include Negatives` has badge count and hides when 0
6. Hide all app components (and added progress bar) while VUE was mounting/initializing
7. Charts UI Polish/Changes
   - Renamed: "Monthly Trend" -> "Cash Flow Trend", "By Category" -> "Spending
     by Category", "Category Trends by Month" -> "Spending by Category Trend".
   - Cash Flow Trend now includes Credits to match cash-flow definition.
   - Removed the synthetic Income/Investment datasets from the category trend chart;
     it is now purely spending by category.
   - Legends are visibility toggles on all three charts. The category trend
     legend previously added a category filter on click.
   - Removed the data-click filter handlers from both category charts. The
     Cash Flow Trend bar click (month filter) is retained.
   - Both category charts share one ranking: top 10 categories plus an "Other" rollup.
   - Series with no data are no longer emitted, so an all-zero Investment
     series no longer leaves a phantom legend entry.
   - Months with nothing to plot are dropped per chart (a transfer-only month
     drew an empty column).
   - Cash Flow Trend bar clicks index into monthlyChartMonths — the months
     actually plotted — instead of availableMonths. Under a date filter the two
     diverged and a bar click filed the wrong month.
8. Transaction Details - container + collapser polish
   - Introduce **containment** — wrap the button row + all category collapsers in one **Transaction Details**
   - Left-align the view buttons; add collapse/expand controls on the right and polish each collapser
   - Fix view-toggle animation cascade in Transaction Details
     - Unify Merchant/Subcategory category views into one keyed v-for
	 - Vue patches <section> nodes in place instead of remounting on a groupByMode toggle
   - Wrap the toggle on mobile (<=640px) — view modes on line 1, actions on line 2; Collapse becomes icon-only
   - Reworked merchant row action presentation: explicit info and filter links
   - Refined expanded transaction row structure for better presentation
   - Fixed 'report fields' badge in transaction detail rendering
   - Adaptive column sizing architecture
     - Introduced dynamic transaction column sizing via CSS variables for
       date, account, and amount tracks.
     - Added startup-time width profiling and debounced resize recalculation
       for desktop-first responsiveness.
9. Rule Info Fixes
   - Added explicit rule provenance into popup data (ruleName, pattern, source)
     so "Why This Matched" reflects the actual classification path.
   - Improved fallback behavior: when no categorization rule matches, promote
     the first matching tag rule so popup rule details remain meaningful.
   - Switched popup tag display to merchant-level union tags (matching the main grid)
   - Cleaned up tag-source reporting to avoid redundant/misleading "from [rule]"
     lines when the displayed rule already explains the tag.
10. Save Layout Settings to Local Storage
   - Transaction Trends collapsed state
   - Transaction Details collapsed state
   - Details view mode (Merchant, Subcategory, View)
   - Include Negatives toggle
   - Per-section collapse state
   - Per-item expansion state (merchant/subcategory rows)
   - Per-section sort settings
11. Fix merchant filter regression and stale tests from row-UX/date polish
- getFilterDescriptor() used item.id (category-specific) instead of
  item.displayName for merchant-type filters, breaking cross-category
  merchant filtering when the same merchant appears in multiple categories.
- Test suite is in line with two intentional behavior changes
  already on this branch that its own tests hadn't caught:
  - Category sections now default to collapsed on a fresh load
    (regardless of "Save Layout Settings to Local Storage")
  - Transaction dates always show the year now

Bug Fix Details:

Both bugs are the same root cause: the JS classified transactions using
merchant.tags — the union of every tag across all of a merchant's
transactions (analyzer.py:122 builds it that way) — instead of the
transaction's own txn.tags, which report.py already emits per transaction.
categorizeAmount() is a strict income -> investment -> transfer -> sign
chain, so one tagged transaction re-bucketed every other transaction at
that merchant. This changed reported dollar amounts; it is not cosmetic.

Bug 1 — charts and the Filtered View tile (filteredViewTotals,
chartAggregations). Every number in all three charts and in the Filtered
View KPI tile was wrong. A merchant whose union is
[monthly-bill, refund, transfer] had all of its ordinary purchases counted
as transfers and vanish from spending. Now classified per transaction, the
JS reconciles with the Python KPIs to the dollar.

Both KPI tiles now agree line-for-line on unfiltered data.

Bug 2 — Section View percentages (grossSpending). grossSpending was
grandTotal + creditsTotal, and both inputs drop a whole merchant when its
tag union is excluded from spending. It feeds the (X%) badge beside each
section header, which renders whenever typeTotals is absent — that is,
Section View always (Category/Subcategory pass server-computed typeTotals
and were never affected). The denominator ran 15% low: $215,322 against a
true $253,781, hiding $6,910 of Amazon spending, $2,448 of Minnesota
Department of Revenue, and more. grossSpending now derives from the same
per-transaction pass and equals Python's spendingTotal exactly. Monthly
Bills reads 23.8% (was 28.0%), Food & Dining 20.0%.

Documented, not fixed: creditMerchants has the same merchant-union bug. It
is unrendered — the "Credits Applied" section was dropped from the template
in ad9477e while docs/reference.html still documents it — so a comment on
the computed records the bug and the two ways to fix it when the section is
restored.
Report title (report.py, spending_report.js)
- Escape the title before it is interpolated into the loading shell's
  <title> and <h1>; a title carrying markup was executable output.
- Escape '</' in the embedded spendingData JSON, so a value containing
  '</script>' cannot close the script element it is embedded in.
- Coerce a non-string title: 'title: 2025' arrives from YAML as an int
  and raised TypeError inside str.replace().
- Resolve the title once and store it back on spendingData, so the
  static shell and the mounted app cannot disagree. The two fallbacks
  differed, so an untitled report was renamed on mount.

Date handling (spending_report.js)
- parseTypedDate now rejects dates a calendar does not have. 2/31/2026,
  month 13 and 99/99/2026 passed the shape regexes and became chips that
  could never match a transaction.
- expandMonthRange validates both halves are YYYY-MM before iterating.
  A hand-edited '#+dr:garbage..garbage' walked to 'NaN-NaN', which
  compares less than 'garba' forever - the report hung with the array
  growing without bound.
- The date popover now edits include-mode filters only. An excluded
  month or range rehydrated into the pending set and came back out of
  Apply as an inclusion, silently inverting the filter.

Percentages (spending_report.js, spending_report.html)
- Section percentages divided a raw total, which includes income,
  investment and transfers, by grossSpending, which excludes them.
  Carry a spending-only subtotal on the same per-transaction basis and
  use it as the numerator.

Performance (spending_report.js)
- Cache transaction column measurements by string. Every transaction
  forced three synchronous layouts, repeated across all three profiles;
  large reports stalled. Cleared per recompute so a resize re-measures.

Accessibility (spending_report.html, spending_report.css)
- Year tabs and month cells are buttons rather than click-only divs, so
  they are keyboard-reachable and announce their selected state.
- Name the reset control with aria-label rather than leaving the glyph
  as its only accessible name.

Config and docs
- Validate report_fields. A bare 'report_fields: memo' was iterated as
  four capture names and an empty value raised TypeError; accept the
  bare string, treat empty as none, and reject other types by name.
- Correct the reference: a field: directive's output is not an input to
  other rules, but the captured column report_fields surfaces is
  available to every rule as field.<name>.
Reimagines the KPI and chart experience in the HTML spending report while preserving the same core budgeting and analysis intent. The functionality is still there, but presented with a different interaction model.

- New KPI dashboard sparkline and consistency layout
- New charts: Spending Seasonality, Expense Volatility, Fixed Spending Audit, and Fixed vs Variable Spending
- Chart Changes
  - By Category becomes Spending By Category column chart (for consistency)
  - Monthly Trend becomes Cash Flow
- All column charts have 'group by' buttons, compare-year toggles, and 'focused' line chart mode
- Improve responsive label behavior, resize rerendering, and compare-year paging
  - Debounced chart rerender on viewport/layout changes via resize + ResizeObserver
  - Disable re-animation for resize-triggered rerenders to avoid redraw jitter
  - Make x-axis labels adaptive (horizontal/angled/vertical) with autoskip under tight widths
  - Expand Playwright chart coverage for compare-year windowing/paging and updated chart controls
- KPI Sparklines are based on last month containing data
- Changed KPI to be month based instead of all time
- Reconcile KPI detail math with trend baseline and clarify secondary labels
  - align 12 Month Avg detail row with trend prior-12 baseline window
  - keep trend on prior-window comparison while preserving anchor-month behavior
  - include credits in Income detail primary values; keep gray secondary informational
  - add a tooltip spelling out what the gray secondary value contributes
  - update KPI styling and report_html coverage for new semantics
- Make Spending by Category legend chips filter the report
  - chips are tri-state (regular / selected / not-selected) rather than show/hide;
    selection is stored as ordinary category filters, so chips and the filter bar
    are the same state. All-on and all-off both collapse back to no filter.
  - clicking a bar segment drills into that category plus that bucket's date range
  - the "Other" chip renders inert; it can never be filtered on
  - the chart's own bars source from a category-exempt aggregation, so selecting a
    category dims its peers instead of dropping them off the canvas
  - a chip toggle flips dataset visibility on the live chart instead of rebuilding
    it, so the stack does not reanimate on every click
- Treat chart-driven filters as transient "peek" state
  - tagged source: 'chart' and deliberately not persisted to the URL hash
  - a Peek Mode badge on the chart title and a Clear Filters button expose and
    clear them; the badge stays visible while the panel is collapsed
  - adding any filter by hand (search, table filter button, date apply) ends peek
- Exclude transfer-only merchants from fixed outputs
  - Build recurring merchant lists from spending-classified transactions only;
    transfer/income/investment-only merchants do not appear in fixed calculations.
  - Update Recurring vs Variable footnote to Top 10 Fixed: ..., + N more, ranked
    by monthly cost so the "+ N more" tail is the cheap end.
  - Update charts documentation wording to match fixed terminology.
- Stabilize chart behavior and default-hide empty legend series
  - Remove split-row and single-expanded reactive layout logic
  - Rerender when a collapsed panel or the whole Transaction Trends section
    expands: a chart measured at zero width bakes rotated, skipped x-axis ticks
    into its config that a plain resize never recomputes
  - Otherwise rerender only when chart-section width changes, or chart toggles change
  - Make chip legends hide zero-data items by default when values are provided
  - Fix cash-flow tooltip Net math to exclude Investments
- Fix subcategory filter chips picking up the "N merchants" summary as their text
Recurrence inference (analyzer.py)
- months_active counts the months a merchant appears in, which says nothing
  about how far apart they are. Require the charges to be dense over the
  merchant's own lifespan as well as covering enough of the reporting period.
  Three January charges span twenty-five months but read as three active
  months, so in a January-only export they cleared the existing test and an
  annual premium was booked as a monthly cost - twelve times its real value.
  The classification also no longer depends on unrelated merchants' data;
  the existing tests needed a dummy monthly merchant present purely to hold
  num_months high enough.

Accessibility (spending_report.html, spending_report.css, spending_report.js)
- Chart panel collapse controls are buttons inside their headings, carrying
  aria-expanded and aria-controls, rather than click handlers on the <h3>.
  Applies to all six panels: category, seasonality, cash, volatility, fixed
  and audit. The heading stays a heading, so document navigation is unchanged.
- The peek badge is a real button rather than a span with role="button", so
  it is reachable and operable from the keyboard.
- A disabled legend chip is now natively disabled instead of only carrying a
  disabled class. The class dropped the click listener but left the control
  focusable and announcing as enabled.

Documentation (docs/charts.html)
- Fixed vs Variable described a mapping of categories. It is a per-merchant
  split driven by inferred recurrence, with fixed/variable rule tags as
  overrides. Document what it actually does, including how the cadence is
  inferred and how to override it.
Allows easier auditing of spending to a merchant.
Of course, if you prefer unique merchant names, that is still possible.

Instead of requiring:

```
[Costco Grocery]
match: contains("COSTCO") and amount <= 200
category: Food
subcategory: Grocery

[Costco Bulk]
match: contains("COSTCO") and amount > 200
category: Shopping
subcategory: Wholesale
```

Can instead be:

```
[Costco]
match: contains("COSTCO") and amount <= 200
category: Food
subcategory: Grocery

[Costco]
match: contains("COSTCO") and amount > 200
category: Shopping
subcategory: Wholesale
```
@terryaney
terryaney force-pushed the feature/merchant-composite-keys branch from af84b33 to 69176f8 Compare August 4, 2026 22:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Same merchant name with different categories causes incorrect category display in report

2 participants