kigeon (like the Pigeon, but with 'k') is a Kubernetes event exporter which allows to send these events to a long term storage.
Kubernetes events are short-lived by default (1 hour TTL). kigeon watches for events cluster-wide, persists them in an internal queue, and forwards them to configurable backends. It is designed to survive restarts without re-sending events that have already been delivered.
- Persistent queue — events are buffered in an embedded NATS JetStream store so no events are lost on restart
- Deduplication — events are deduplicated both in-flight and across restarts, so each event is sent exactly once
- Namespace filtering — senders can be scoped to namespaces matching a label selector, which is kept up-to-date dynamically
- Per-event hooks — a Starlark script can modify the sender config on a per-event basis (e.g. routing to different tenants, adding labels)
- Multiple senders — multiple sender instances can run in parallel, each with their own filter and configuration
| Type | Description |
|---|---|
loki |
Grafana Loki |
helm repo add kigeon https://ninech.github.io/kigeon
helm repo update
# Example installation with a Loki sender and persistent storage:
helm install kigeon kigeon/kigeon \
--namespace kigeon --create-namespace \
--set persistence.enabled=true \
--set config.eventsenders[0].name=loki \
--set config.eventsenders[0].type=loki \
--set config.eventsenders[0].config.url=http://loki:3100Hook scripts can be passed with hooks, they are mounted to
/etc/kigeon/hooks/<name>:
helm install kigeon kigeon/kigeon \
--set-file hooks.my-hook\\.star=./my-hook.star \
--set config.eventsenders[0].config.hook.script=/etc/kigeon/hooks/my-hook.star \
...For a full list of configuration options see deploy/helm/kigeon/values.yaml.
Container images are published to ghcr.io/ninech/kigeon for each release.
See the releases page for available versions.
docker pull ghcr.io/ninech/kigeon:0.0.6kigeon is configured via a YAML file passed with --config.
global:
# Directory where kigeon stores its queue data.
# Default: /var/lib/kigeon/data
dataDir: /var/lib/kigeon/data
# Log level: debug, info, warn, error. Default: info
logLevel: info
# Log format: json, text. Default: json
logFormat: json
# Address of the metrics and health endpoints. Default: :8080
metricsBindAddress: ":8080"
queue:
# How long undelivered events are kept in the queue. Default: 24h
eventsMaxAge: 24h
# Maximum queue size on disk. The data directory needs to provide at least
# this amount plus 32Mi for the deduplication store. Default: 500Mi
eventsMaxBytes: 500Mi
# How long processed event UIDs are remembered (prevents re-sending after
# restart). Default: 1h
kubernetesEventsMaxLifetime: 1h
pusher:
# Timeout for pushing a single event into the queue. Default: 5s
pushTimeout: 5s
pods:
# Emit synthetic "OOMKilled" events for containers which were killed due to
# running out of memory. Default: false
oomKilledEvents: false
# How long deleted pods remain available for pod enrichment. Default: 5m
deletedPodRetention: 5m
eventsenders:
- name: my-loki
type: loki
# Optional: only forward events from namespaces matching these labels.
filter:
namespaceSelector:
matchLabels:
environment: production
# Also forward events for cluster-scoped objects (Nodes, etc.).
includeNonNamespaced: false
# How often to force-refresh the namespace list. Default: 0 (disabled)
hardRefreshInterval: 4h
# Sender-specific configuration (see below).
config:
url: http://loki:3100
tenantID: my-tenant
streamLabels:
app: kigeon
cluster: my-clusterconfig:
# Required. Base URL of the Loki instance (e.g. http://loki:3100, without path).
url: http://loki:3100
# X-Scope-OrgID header for multi-tenant Loki.
tenantID: my-tenant
# Static labels added to every log stream. Labels of the event (see "Stream
# labels" below) take precedence.
streamLabels:
app: kigeon
# Basic authentication. Values can be set directly or via environment
# variable references (direct values take precedence).
basicAuth:
username: user
password: secret
# Alternative: read from environment variables at startup.
usernameEnvVar: LOKI_USER
passwordEnvVar: LOKI_PASS
# TLS configuration for the Loki HTTP client.
tlsConfig:
# Disable TLS certificate verification (e.g. for self-signed certs).
insecureSkipVerify: false
# Optional Starlark hook — see "Hooks" section below.
hook:
script: /etc/kigeon/hooks/my-hook.star
timeout: 100ms # Max execution time per event. Default: 100ms
onError: use-default # use-default | skip | fail
enrichPod: false # Fetch full Pod definition and expose it as "pod"
skipOnPodNotFound: true # Skip event if the involved Pod no longer existsHooks allow a Starlark script to dynamically modify the Loki sender config on a per-event basis — for example to route events to different tenants, add stream labels, or change credentials.
The script must define a transform(config, event) function that receives the
current config and the Kubernetes event as dicts, and returns either the
(potentially modified) config dict or None to skip the event entirely.
Returning None acknowledges the event without sending it — useful for
filtering out events that do not match your criteria:
def transform(config, event):
# Skip non-Pod events
if (event.get("involvedObject") or {}).get("kind") != "Pod":
return None
ns = (event.get("metadata") or {}).get("namespace", "")
if ns == "production":
config["tenantID"] = "prod"
return configOutput of print() is logged at info level, for example to find events which
a hook does not handle yet:
def transform(config, event):
if event.get("reason") not in ("Started", "BackOff"):
print("dropped event with reason", event.get("reason"))
return None
return configBy default, every event is sent with the configured streamLabels and these
labels of the event:
| Label | Value |
|---|---|
namespace |
Namespace of the event |
involved_object_kind |
Kind of the involved object |
involved_object_name |
Name of the involved object |
reason |
Reason of the event |
type |
Normal or Warning |
source_component |
Component which reported the event (if set) |
source_host |
Node which reported the event (if set) |
A hook receives these labels in config["streamLabels"] and the sender uses
exactly the labels the hook returns. Labels like involved_object_name or
source_host create many small streams, which Loki does not handle well. A
hook can replace the labels to keep their number low:
def transform(config, event):
config["streamLabels"] = {
"namespace": event["involvedObject"].get("namespace", ""),
"source": "event",
}
return configIf Loki rejects an event as invalid (HTTP 400, e.g. because of an invalid label), the event is dropped and a warning is logged. Other errors are retried.
| Field | Description | Default |
|---|---|---|
script |
Path to the .star script file |
required |
timeout |
Max execution time per event | 100ms |
onError |
Behaviour on script error: use-default, skip, fail |
use-default |
enrichPod |
Fetch the full Pod definition and expose it as pod on the event dict |
false |
skipOnPodNotFound |
Skip the event (without calling the hook) if the involved Pod no longer exists. Only meaningful with enrichPod: true. Transient API errors still fall through to the hook. |
false |
| Value | Description |
|---|---|
use-default |
Use the base config and send the event normally |
skip |
Acknowledge the event without sending it |
fail |
Return an error so the event is redelivered via NATS |
When enrichPod: true is set and the involved object is a Pod, kigeon makes the
full Pod definition available as pod on the event dict. Pods are served from
a cluster-wide pod informer, which is only started if a sender enables pod
enrichment or pods.oomKilledEvents is set.
Deleted pods remain available for pods.deletedPodRetention, as events are
often processed after the pod they refer to was deleted (e.g. short-lived job
pods or when catching up after a restart). Pods which the informer did not
observe yet are fetched from the Kubernetes API.
Always guard against the pod being unavailable:
def transform(config, event):
pod = event.get("pod") or {}
labels = pod.get("metadata", {}).get("labels") or {}
...See examples/hooks/ for ready-to-use scripts:
| Script | Description |
|---|---|
route-by-namespace.star |
Route events to different Loki tenants based on namespace |
add-pod-labels-as-stream-labels.star |
Forward selected pod labels as Loki stream labels |
route-oom-events.star |
Send OOMKilling events to a dedicated Loki tenant |
Kubernetes does not emit an event when a container gets killed due to running
out of memory. The kernel OOM kill is only reported as an OOMKilling event on
the Node, which can not be attributed to a pod or namespace.
With pods.oomKilledEvents: true, kigeon watches the container statuses of all
pods and emits a synthetic Warning event with reason OOMKilled for the pod
(source component kigeon), for example:
Container app was killed because it ran out of memory (memory limit 512Mi)
These events are queued like any other event, so namespace filters, hooks and
pod enrichment apply to them as well. OOM kills which happened more than
queue.kubernetesEventsMaxLifetime ago (e.g. while kigeon was not running)
are not reported, as they might have been sent already.
kigeon --config /etc/kigeon/config.yaml
# Use a local kubeconfig instead of in-cluster credentials
kigeon --config /etc/kigeon/config.yaml --kubeconfig ~/.kube/config
# Override log level
kigeon --config /etc/kigeon/config.yaml --log-level debugkigeon serves Prometheus metrics on /metrics, a liveness check on
/healthz and a readiness check on /readyz (ready once all senders are
subscribed and events are watched) on global.metricsBindAddress. The Helm
chart can create a ServiceMonitor with serviceMonitor.enabled.
| Metric | Labels | Description |
|---|---|---|
kigeon_events_watched_total |
source (kubernetes, synthetic) |
Events received from Kubernetes or created by kigeon (e.g. OOM killed events) |
kigeon_events_published_total |
result (published, duplicate, error) |
Results of publishing events to the queue |
kigeon_queue_bytes / kigeon_queue_messages |
Size and number of events in the queue | |
kigeon_queue_max_bytes |
Maximum size of the queue, the oldest events are dropped once it is reached | |
kigeon_sender_pending_events |
sender |
Events a sender did not process yet |
kigeon_sender_events_total |
sender, result (sent, filtered, skipped, rejected, failed) |
Results of processing events per sender. failed events are retried |
kigeon_sender_http_requests_total |
sender, code |
HTTP requests to the receiver. Requests without a response (e.g. connection errors) are only counted as failed events |
kigeon_sender_http_request_duration_seconds |
sender, code |
Duration of HTTP requests to the receiver |
kigeon_hook_duration_seconds |
sender |
Duration of hook executions, including the pod enrichment |
kigeon_hook_errors_total |
sender |
Failed hook executions |
kigeon_kubernetes_requests_total |
code, method |
Requests to the Kubernetes API |
kigeon_kubernetes_request_retries_total |
code, method |
Retried requests to the Kubernetes API |
kigeon_kubernetes_request_duration_seconds |
verb |
Duration of non-watch requests to the Kubernetes API |
Example alerts:
# a sender does not keep up or its receiver is unavailable
- alert: KigeonSenderBacklog
expr: kigeon_sender_pending_events > 1000
for: 15m
# the queue is about to drop events which were not sent yet
- alert: KigeonQueueAlmostFull
expr: kigeon_queue_bytes / kigeon_queue_max_bytes > 0.8
for: 15m
- alert: KigeonSenderFailing
expr: sum by (sender) (rate(kigeon_sender_events_total{result="failed"}[10m])) > 0
for: 15m
# e.g. missing RBAC permissions
- alert: KigeonKubernetesRequestErrors
expr: sum by (code) (rate(kigeon_kubernetes_requests_total{code=~"4..|5.."}[10m])) > 0
for: 15m
# no events received from the Kubernetes API
- alert: KigeonNoEventsWatched
expr: sum(rate(kigeon_events_watched_total{source="kubernetes"}[1h])) == 0
for: 1hKubernetes API
│ (informer)
▼
EventPusher ← watches all events cluster-wide
│ (deduplicate + publish)
▼
EventQueue ← embedded NATS JetStream (persistent)
│ (fan-out)
├──▶ Sender 1 (loki) ──▶ Loki instance A
└──▶ Sender 2 (loki) ──▶ Loki instance B
Each sender subscribes independently to the queue. An event is removed from
the queue once all senders acknowledged it, so the queue only grows while a
sender can not deliver events (bounded by eventsMaxAge and
eventsMaxBytes). Subscriptions of senders which were removed from the
configuration are deleted on startup. Namespace filters and hooks
are applied per sender, so different senders can have completely different
routing logic from the same event stream.
