AutoDetect

A self-writing, self-tuning Splunk detection agent.

AutoDetect decides what to detect, writes the SPL detection itself, deploys it, then measures and tunes away its own false positives — a SOC that improves itself with no human writing a single line of SPL.

Built for the Splunk Agentic Ops Hackathon (Security track).

---

What it does

Press one button and watch the agent run the full loop live:

decide → author → validate → (self-correct) → deploy → measure → self-tune

• Decides what to detect — surveys your live Splunk data and existing detections, then proposes the highest-value missing detections (ranked, MITRE-mapped).
• Writes the SPL — an LLM authors the detection, grounded on real sampled events so it doesn't hallucinate field names.
• Self-corrects — if the SPL is invalid or misses the threat, it reads the error and rewrites its own rule.
• Deploys — registers the detection as a scheduled Splunk saved search.
• Self-tunes — watches which entities fire, spots false positives on benign users, and retunes the threshold until it cleanly separates the attacker from the noise.

See architecture_diagram.md for diagrams and data flow.

How it talks to Splunk

• Hands — Splunk REST API (saved/searches, search/jobs): create / tune / retire detections + run searches. All writes go here.
• Eyes — Splunk MCP Server (read/execute): agent-native discovery + search. (MCP is read-only, so writes use REST.)

How the AI is integrated

An OpenAI-compatible LLM powers the Profiler, Author, and Tuner — swappable via one .env line (we run Ollama Cloud gpt-oss:120b; Groq, OpenAI, local Ollama, or a Splunk-hosted model all work). The Author is grounded two ways so it doesn't make things up: a per-technique signature playbook (so it writes behavioural detections, not raw volume counts) and live calibration — it measures the real distribution of the exact metric it thresholds on, through the detection's own pipeline, so thresholds are never guessed or borrowed from the wrong quantity. The Validator is deterministic (real dry-runs) so the agent reasons over ground truth.

---

Quick start

1. Splunk

1. Install Splunk Enterprise (60-day free trial); apply a Developer License.
2. (Optional) install the MCP Server for Splunk app from Splunkbase.
3. (For the real-data demo) load the BOTSv3 dataset — see Dataset below.

2. Backend (Python agent + API)

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env        # fill in SPLUNK_USERNAME/PASSWORD + an LLM key

Set your LLM in .env (LLM_PROVIDER, LLM_MODEL, LLM_API_KEY). Free options: Ollama Cloud (gpt-oss:120b, https://ollama.com/settings/keys) or Groq (llama-3.3-70b-versatile, https://console.groq.com/keys).

3. Verify your instance

python scripts/day1_connection_test.py   # GO/NO-GO: can the agent run + deploy detections?
python scripts/test_llm.py               # is the LLM reachable?

4. Run it

Web UI (the demo):

./scripts/run_dev.sh        # backend :8000 + Next.js UI :3000
# open http://localhost:3000
#   "Run AutoDetect"               → synthetic brute-force: author → self-tune
#   "Run on real attack data"      → BOTSv3: profile → calibrate → self-correct → deploy
#   "Discover coverage gaps"       → the agent proposes what to detect next

Or headless, in the terminal:

python scripts/vertical_slice.py   # one detection: author → validate → deploy
python scripts/agent_loop.py       # full self-tuning loop (false positives → retune)
python scripts/profile_gaps.py     # the agent proposes what to detect
python scripts/botsv3_demo.py      # author + calibrate a detection on real BOTSv3 data

---

Dataset (BOTSv3)

The real-data demo runs against Splunk's Boss of the SOC v3 dataset — ~2M real events from a simulated enterprise breach (AWS, Windows, endpoint, network telemetry).

1. Download the botsv3_data_set from the BOTSv3 repo.
2. Extract it into $SPLUNK_HOME/etc/apps/ and restart Splunk.
3. It loads as index=botsv3. BOTSv3 is historical (2018–2019), so all searches over it use earliest=0 (all-time) — the agent does this automatically.

The synthetic brute-force demo needs no dataset — it generates its own attack + benign-noise events via the Splunk REST API, so you can run that one immediately.

---

Project layout

src/autodetect/
  config.py        # typed config from .env
  llm.py           # OpenAI-compatible LLM client (Groq/Ollama/OpenAI)
  splunk_rest.py   # REST client — run searches + create/tune/retire detections (hands)
  mcp_client.py    # MCP client — discover + run searches (eyes)
  attack.py        # deterministic attack replay + benign noise
  profiler.py      # surveys data → proposes missing detections
  author.py        # LLM → SPL detection (+ self-correction)
  playbook.py      # per-technique signature guidance (behaviour, not raw volume)
  validator.py     # dry-runs SPL against real data
  tuner.py         # measures false positives → retunes
  agent.py         # AutoDetectAgent — closed loop: explore → calibrate → self-correct
  flows.py         # reusable demo flow (shared by CLI + server)
  server.py        # FastAPI + WebSocket backend
web/               # Next.js dashboard (live agent feed, detections, coverage gaps)
scripts/           # connection test, LLM test, vertical slice, agent loop, profiler, run_dev.sh

---

License

MIT