Using ActivityWatch to map out work processes to agentic workflows

[curiousjbird]

I'm curious is anyone has used ActivityWatch to map out not just what they are looking at, but "how" they do something. E.g. I receive an email from a client that asks a question, I need to review another file before I respond, then I write a response. There are numerous micro-flows that we perform when working that are critical to understand as we try to automate or make agentic. Could ActivityWatch be a good platform for this mapping?

[pcuci]

I fed my AW week's activity into chatgpt o3 and asked it to organize my attention better

ended up with an updated schedule for next week

not sure what you mean by how/what

I would start with why, as per https://en.wikipedia.org/wiki/Start_with_Why

then how, then what

[Lopesnextgen]

Hey jbird,

ActivityWatch could be a strong data source, but it is not a complete process-mining system by itself

ActivityWatch is good at answering:

• which application was active
• which window, file, project, or website was visible
• when context switches occurred
• how long each state lasted
• whether the user was active or AFK

Your example asks a deeper question:

Which sequence of actions belonged to the same client request, and why did the user perform them?

ActivityWatch does not currently know that automatically.

The missing concept is a case or task ID

Traditional process mining usually expects events resembling:

• case: client-request-482
• activity: read-email
• timestamp: 10:02
• object: customer-a
• activity: review-contract
• timestamp: 10:06
• object: contract-17
• activity: write-response
• timestamp: 10:14

ActivityWatch records timestamps and contextual metadata, but normally does not know that those three activities belong to the same work item.

Without a case ID, an analyzer has to infer task boundaries from signals such as:

• email subject or conversation ID
• customer/project name
• document path
• repository and branch
• browser domain or URL
• application transitions
• time between events
• calendar meetings
• issue or ticket identifiers

That inference can be useful, but it will always have some ambiguity.

A practical architecture

I would use ActivityWatch as the local event collection layer and build the process analysis separately.

1. Collect multiple signals

The normal window, browser, AFK, and editor watchers provide a good foundation. Custom watchers could add:

• email conversation metadata
• calendar events
• ticket or issue identifiers
• document identifiers
• repository and Git branch
• terminal working directory
• communication application/channel
• clipboard activity, with considerable privacy restrictions

I would avoid recording complete email bodies, document contents, commands, or clipboard values by default. Process metadata is often enough and carries much less risk.

2. Normalize events into activities

Raw applications are not necessarily meaningful workflow steps.

For example:

• Outlook window becomes review-client-email
• PDF viewer becomes review-reference-document
• Word or Outlook compose becomes draft-response
• browser on a CRM URL becomes check-customer-record

This translation could use rules first, with a local model only where the signal is ambiguous.

3. Segment the timeline into work sessions

Possible boundaries include:

• long AFK periods
• calendar transitions
• explicit task/project changes
• switching to an unrelated client
• a large gap between related events
• completion signals such as sending an email or closing a ticket

A small optional hotkey such as “mark current task” would dramatically improve accuracy without becoming full manual time tracking.

4. Correlate events into cases

The analyzer could assign events to inferred cases using shared metadata.

For example:

• email contains ticket ABC-123
• browser opens ABC-123
• editor uses branch ABC-123-fix
• terminal operates in the same repository
• response returns to the original email conversation

That is much stronger evidence of a workflow than application names alone.

5. Discover recurring patterns

Once events are grouped into cases, sequence analysis could reveal patterns such as:

• email → CRM → document → response
• ticket → repository → terminal → pull request
• alert → dashboard → logs → incident channel
• meeting → notes → task creation → follow-up email

From there, an LLM could explain the patterns, but I would calculate the event sequences and durations deterministically before asking the model to summarize them.

Where ActivityWatch fits well

ActivityWatch is particularly suitable because it is:

• local-first
• timestamp-oriented
• extensible through custom watchers
• able to preserve raw events
• queryable through its API and Python client
• capable of correlating overlapping time periods

I would work from raw or lightly processed events rather than only the default Activity view. The canonical query is designed for time summaries, so it may remove or merge context that is useful for workflow discovery.

Important limitations

Passive observation cannot reliably capture:

• why an action was taken
• whether two similar windows belong to different clients
• whether a transition was intentional or a distraction
• work performed on a phone or another untracked device
• decisions made without visible computer interaction
• exceptions that happen too rarely to form a recognizable pattern

It can reconstruct probable behavior, not ground-truth intent.

There is also a danger in converting observed behavior directly into an agent workflow. A human process may contain unnecessary steps caused by poor interfaces, missing integrations, organizational policy, or habit. The goal should be to understand the process first, then decide which steps should be automated, assisted, redesigned, or intentionally kept human.

A good first experiment

I would start with one narrowly defined workflow, such as responding to a support request.

Collect:

• active application/window
• browser URL or domain
• email conversation ID
• ticket ID
• files opened
• repository/branch, where relevant
• AFK periods

Then manually label perhaps 20 completed cases and compare those labels against automatic segmentation. That will reveal which additional signals are actually required before building a larger system.

So my answer is: yes, ActivityWatch could be an excellent foundation for task mining and agentic-workflow discovery, but it would need a correlation and process-analysis layer above it. The watchers observe activity; the additional layer must infer tasks, boundaries, relationships, and workflow meaning.

References:

• ActivityWatch data model: https://docs.activitywatch.net/en/latest/buckets-and-events.html
• Custom watchers: https://docs.activitywatch.net/en/latest/examples/writing-watchers.html
• Working with ActivityWatch data: https://docs.activitywatch.net/en/latest/examples/working-with-data.html
• Watchers and custom visualizations: https://docs.activitywatch.net/en/latest/watchers.html