Proposal & intend to implement: Extended rule engine for categorization (categories, AFK, multi-device, multi-watchers)

[petrroll]

Following up on #1169, I tried to sketch a more concrete design before starting implementation, mainly to check whether this is a direction that could realistically get merged.

I'm happy to contribute the change, but I understand such a big change might not be in line with the vision so discussing here first. If it can't be merged then I'll just fork and slop it 😅.

Motivation:

What I would like to express is stuff like:

• Visual Studio is Work, but Visual Studio on virtual desktop Personal is Personal.
• A meeting watcher can make the period active even when the laptop is AFK, and can also provide the activity which gets categorized.
• Rules can depend on a specific host (for better multihost control)
• Things on Personal virtual desktop are always personal even if they accidentally match deep rule in Work category subtree.

The rough idea is to make both the active-time/AFK rule and category rules expression trees:

How:

• AND / OR groups, including nesting/parentheses.
  • (I'm also open to a bit more simplified design, but I think builders for these nowadays have decent UX to be usable
• Basic condition = source/bucket + field + regex, optionally negative, host-specific, and with a match weight.
  • Defaults to using the inbuild buckets / fields to mimic current behavior.
  • If multiple categories match, the highest score wins. AND adds the matched condition weights, while OR only uses the winning matched branch (or the highest). Existing rules would all have weight 0, so the current deepest-category behavior stays as the fallback.
• Matching is based on time overlap. If a window event overlaps the Personal virtual-desktop event for four minutes, the combined rule matches only those four minutes.
• A child category can optionally require selected parent rules to match as well.

I don't want to move responsibilities around unnecessarily. Canonical-query composition—selecting buckets, building the active mask, source precedence, multi-device merge, etc.—would stay in  aw-webui  / the existing client query builders where it is now. The Python/Rust transform layer would only gain rule-expression evaluation (or maybe no support at all? and keep it front-end only? but it might be useful for others querying the serer) and the overlap-enrichment primitive needed for consistent behavior.

Raw events and buckets would not change. Existing settings would migrate without rewriting data, and the UI should remain as simple as it is now by default: name + regex, with the more complex editor hidden under an Advanced option (or some other progressive exploration, not sure about the UX yet).

wdyt?

---

Cc @ErikBjare

[petrroll]

I've started implementing it just to proof of concept it and some examples (the UX can be cleaned up). By default everything defaults to settings which match current state. But it's way more powerful.

New editor of categorization allowing matching arbitrary watcher / source:

You can build arbitrary trees with and/or (I don't have good examples r.n). Rule can also require parent to match, to alllow distinguishing e.g.
Visual studio on Private virtual destkop from VS on work virtual destkop.

AFK rules also allow arbitrary expression, want to have AFK based on e..g your slack presence watcher -> it's possible:

You can rename sources to be nicer on timeline view. You can set which sources can induce an activity (app window is no longer privilidged (but it's the default!)).

I have reworked the timeline a bit to show category separately (as now it's less tied to the window) + it's clearer in showing what's afk irrespective of what category it'd be if active.

I'm happy to have a call and show it in motion and explain the datamodel more.

[petrroll]

The code is 100 % sol and I didn't even look at it yet, before merging I'd do many manual passes - of course: https://github.com/petrroll/activitywatch/tree/feat/flexible-rules-v2

[TMA-2]

I'm not smart enough to make any contributions to AW, but this is an awesome idea. I'm a lowly PowerShell guy, not a python or rust guy.

one that I had a long time ago was to be able to specify which watcher (for starters) your rules applied to, as well as granular control over which fields it matched, as perhaps one rule should only match against the chrome URL, and another should match the page title. same for the window watcher, which I feel desperately needs the path name in addition to the executable and window title.

I love your additions to the rule editor UI as well, perhaps even giving options to match property type (string, int, bool), or even date/time. but yeah, picking which host(s) and watcher(s) each rule applies to is great.

[ErikBjare]

Been thinking along these lines before, but wasn't high-priority and the effort didn't seem worth it pre-LLMs (at least not for me personally). I'm also a bit hesitant of adding too much configuration complexity over decent defaults.

I haven't thought through your proposal fully, but if the code and UX for it ends up clean enough I'd consider merging it. Ideally these things would be part of the canonicalQuery abstraction (in aw-client-py/rs/js) and stored in server settings so that we don't have to update every caller, maybe with just a transform or two for the custom rules (haven't looked at the code you linked yet).

I like the "added row for categories in timeline", worth a separate PR.

[petrroll]

Is there any way that'd allow you to think about the UX more? Video? Quick call with demonstration? Pre-build WIP version (with not yet cleaned up code)?

[petrroll]

I think it's enough of a difference that the clients would mostly need to be updated. Watchers would remain as is, and old clients could fallback to compatibility shims, but the model is inherently more complicated (if nothing adjusting rules is) & the current model has come assumptions the broader model breaks; so in any case extending the boundary would be needed.

[petrroll]

@ErikBjare just friendly bump, is there any UX demonstration that'd be preferrable, or jumping directly to good enough PR for potential merge would be best course of action?