Skip to content
ninechPublic

About

kigeon is a Kubernetes event exporter

Resources

Stars

3 stars

Watchers

0 watching

Forks

Repository files navigation

kigeon

kigeon (like the Pigeon, but with 'k') is a Kubernetes event exporter which allows to send these events to a long term storage.

Kubernetes events are short-lived by default (1 hour TTL). kigeon watches for events cluster-wide, persists them in an internal queue, and forwards them to configurable backends. It is designed to survive restarts without re-sending events that have already been delivered.

Features

  • Persistent queue — events are buffered in an embedded NATS JetStream store so no events are lost on restart
  • Deduplication — events are deduplicated both in-flight and across restarts, so each event is sent exactly once
  • Namespace filtering — senders can be scoped to namespaces matching a label selector, which is kept up-to-date dynamically
  • Per-event hooks — a Starlark script can modify the sender config on a per-event basis (e.g. routing to different tenants, adding labels)
  • Multiple senders — multiple sender instances can run in parallel, each with their own filter and configuration

Supported backends

Type Description
loki Grafana Loki

Installation

Helm

helm repo add kigeon https://ninech.github.io/kigeon
helm repo update

# Example installation with a Loki sender and persistent storage:
helm install kigeon kigeon/kigeon \
  --namespace kigeon --create-namespace \
  --set persistence.enabled=true \
  --set config.eventsenders[0].name=loki \
  --set config.eventsenders[0].type=loki \
  --set config.eventsenders[0].config.url=http://loki:3100

Hook scripts can be passed with hooks, they are mounted to /etc/kigeon/hooks/<name>:

helm install kigeon kigeon/kigeon \
  --set-file hooks.my-hook\\.star=./my-hook.star \
  --set config.eventsenders[0].config.hook.script=/etc/kigeon/hooks/my-hook.star \
  ...

For a full list of configuration options see deploy/helm/kigeon/values.yaml.

Container image

Container images are published to ghcr.io/ninech/kigeon for each release. See the releases page for available versions.

docker pull ghcr.io/ninech/kigeon:0.0.6

Configuration

kigeon is configured via a YAML file passed with --config.

global:
  # Directory where kigeon stores its queue data.
  # Default: /var/lib/kigeon/data
  dataDir: /var/lib/kigeon/data

  # Log level: debug, info, warn, error. Default: info
  logLevel: info

  # Log format: json, text. Default: json
  logFormat: json

  # Address of the metrics and health endpoints. Default: :8080
  metricsBindAddress: ":8080"

queue:
  # How long undelivered events are kept in the queue. Default: 24h
  eventsMaxAge: 24h

  # Maximum queue size on disk. The data directory needs to provide at least
  # this amount plus 32Mi for the deduplication store. Default: 500Mi
  eventsMaxBytes: 500Mi

  # How long processed event UIDs are remembered (prevents re-sending after
  # restart). Default: 1h
  kubernetesEventsMaxLifetime: 1h

pusher:
  # Timeout for pushing a single event into the queue. Default: 5s
  pushTimeout: 5s

pods:
  # Emit synthetic "OOMKilled" events for containers which were killed due to
  # running out of memory. Default: false
  oomKilledEvents: false

  # How long deleted pods remain available for pod enrichment. Default: 5m
  deletedPodRetention: 5m

eventsenders:
  - name: my-loki
    type: loki

    # Optional: only forward events from namespaces matching these labels.
    filter:
      namespaceSelector:
        matchLabels:
          environment: production
      # Also forward events for cluster-scoped objects (Nodes, etc.).
      includeNonNamespaced: false
      # How often to force-refresh the namespace list. Default: 0 (disabled)
      hardRefreshInterval: 4h

    # Sender-specific configuration (see below).
    config:
      url: http://loki:3100
      tenantID: my-tenant
      streamLabels:
        app: kigeon
        cluster: my-cluster

Loki sender

config:
  # Required. Base URL of the Loki instance (e.g. http://loki:3100, without path).
  url: http://loki:3100

  # X-Scope-OrgID header for multi-tenant Loki.
  tenantID: my-tenant

  # Static labels added to every log stream. Labels of the event (see "Stream
  # labels" below) take precedence.
  streamLabels:
    app: kigeon

  # Basic authentication. Values can be set directly or via environment
  # variable references (direct values take precedence).
  basicAuth:
    username: user
    password: secret
    # Alternative: read from environment variables at startup.
    usernameEnvVar: LOKI_USER
    passwordEnvVar: LOKI_PASS

  # TLS configuration for the Loki HTTP client.
  tlsConfig:
    # Disable TLS certificate verification (e.g. for self-signed certs).
    insecureSkipVerify: false

  # Optional Starlark hook — see "Hooks" section below.
  hook:
    script: /etc/kigeon/hooks/my-hook.star
    timeout: 100ms       # Max execution time per event. Default: 100ms
    onError: use-default # use-default | skip | fail
    enrichPod: false     # Fetch full Pod definition and expose it as "pod"
    skipOnPodNotFound: true  # Skip event if the involved Pod no longer exists

Hooks

Hooks allow a Starlark script to dynamically modify the Loki sender config on a per-event basis — for example to route events to different tenants, add stream labels, or change credentials.

The script must define a transform(config, event) function that receives the current config and the Kubernetes event as dicts, and returns either the (potentially modified) config dict or None to skip the event entirely.

Returning None acknowledges the event without sending it — useful for filtering out events that do not match your criteria:

def transform(config, event):
    # Skip non-Pod events
    if (event.get("involvedObject") or {}).get("kind") != "Pod":
        return None

    ns = (event.get("metadata") or {}).get("namespace", "")
    if ns == "production":
        config["tenantID"] = "prod"
    return config

Output of print() is logged at info level, for example to find events which a hook does not handle yet:

def transform(config, event):
    if event.get("reason") not in ("Started", "BackOff"):
        print("dropped event with reason", event.get("reason"))
        return None
    return config

Stream labels

By default, every event is sent with the configured streamLabels and these labels of the event:

Label Value
namespace Namespace of the event
involved_object_kind Kind of the involved object
involved_object_name Name of the involved object
reason Reason of the event
type Normal or Warning
source_component Component which reported the event (if set)
source_host Node which reported the event (if set)

A hook receives these labels in config["streamLabels"] and the sender uses exactly the labels the hook returns. Labels like involved_object_name or source_host create many small streams, which Loki does not handle well. A hook can replace the labels to keep their number low:

def transform(config, event):
    config["streamLabels"] = {
        "namespace": event["involvedObject"].get("namespace", ""),
        "source": "event",
    }
    return config

If Loki rejects an event as invalid (HTTP 400, e.g. because of an invalid label), the event is dropped and a warning is logged. Other errors are retried.

Hook config

Field Description Default
script Path to the .star script file required
timeout Max execution time per event 100ms
onError Behaviour on script error: use-default, skip, fail use-default
enrichPod Fetch the full Pod definition and expose it as pod on the event dict false
skipOnPodNotFound Skip the event (without calling the hook) if the involved Pod no longer exists. Only meaningful with enrichPod: true. Transient API errors still fall through to the hook. false

onError behaviour

Value Description
use-default Use the base config and send the event normally
skip Acknowledge the event without sending it
fail Return an error so the event is redelivered via NATS

Pod enrichment

When enrichPod: true is set and the involved object is a Pod, kigeon makes the full Pod definition available as pod on the event dict. Pods are served from a cluster-wide pod informer, which is only started if a sender enables pod enrichment or pods.oomKilledEvents is set.

Deleted pods remain available for pods.deletedPodRetention, as events are often processed after the pod they refer to was deleted (e.g. short-lived job pods or when catching up after a restart). Pods which the informer did not observe yet are fetched from the Kubernetes API.

Always guard against the pod being unavailable:

def transform(config, event):
    pod = event.get("pod") or {}
    labels = pod.get("metadata", {}).get("labels") or {}
    ...

Examples

See examples/hooks/ for ready-to-use scripts:

Script Description
route-by-namespace.star Route events to different Loki tenants based on namespace
add-pod-labels-as-stream-labels.star Forward selected pod labels as Loki stream labels
route-oom-events.star Send OOMKilling events to a dedicated Loki tenant

OOM killed containers

Kubernetes does not emit an event when a container gets killed due to running out of memory. The kernel OOM kill is only reported as an OOMKilling event on the Node, which can not be attributed to a pod or namespace.

With pods.oomKilledEvents: true, kigeon watches the container statuses of all pods and emits a synthetic Warning event with reason OOMKilled for the pod (source component kigeon), for example:

Container app was killed because it ran out of memory (memory limit 512Mi)

These events are queued like any other event, so namespace filters, hooks and pod enrichment apply to them as well. OOM kills which happened more than queue.kubernetesEventsMaxLifetime ago (e.g. while kigeon was not running) are not reported, as they might have been sent already.

Running

kigeon --config /etc/kigeon/config.yaml

# Use a local kubeconfig instead of in-cluster credentials
kigeon --config /etc/kigeon/config.yaml --kubeconfig ~/.kube/config

# Override log level
kigeon --config /etc/kigeon/config.yaml --log-level debug

Metrics

kigeon serves Prometheus metrics on /metrics, a liveness check on /healthz and a readiness check on /readyz (ready once all senders are subscribed and events are watched) on global.metricsBindAddress. The Helm chart can create a ServiceMonitor with serviceMonitor.enabled.

Metric Labels Description
kigeon_events_watched_total source (kubernetes, synthetic) Events received from Kubernetes or created by kigeon (e.g. OOM killed events)
kigeon_events_published_total result (published, duplicate, error) Results of publishing events to the queue
kigeon_queue_bytes / kigeon_queue_messages Size and number of events in the queue
kigeon_queue_max_bytes Maximum size of the queue, the oldest events are dropped once it is reached
kigeon_sender_pending_events sender Events a sender did not process yet
kigeon_sender_events_total sender, result (sent, filtered, skipped, rejected, failed) Results of processing events per sender. failed events are retried
kigeon_sender_http_requests_total sender, code HTTP requests to the receiver. Requests without a response (e.g. connection errors) are only counted as failed events
kigeon_sender_http_request_duration_seconds sender, code Duration of HTTP requests to the receiver
kigeon_hook_duration_seconds sender Duration of hook executions, including the pod enrichment
kigeon_hook_errors_total sender Failed hook executions
kigeon_kubernetes_requests_total code, method Requests to the Kubernetes API
kigeon_kubernetes_request_retries_total code, method Retried requests to the Kubernetes API
kigeon_kubernetes_request_duration_seconds verb Duration of non-watch requests to the Kubernetes API

Example alerts:

# a sender does not keep up or its receiver is unavailable
- alert: KigeonSenderBacklog
  expr: kigeon_sender_pending_events > 1000
  for: 15m
# the queue is about to drop events which were not sent yet
- alert: KigeonQueueAlmostFull
  expr: kigeon_queue_bytes / kigeon_queue_max_bytes > 0.8
  for: 15m
- alert: KigeonSenderFailing
  expr: sum by (sender) (rate(kigeon_sender_events_total{result="failed"}[10m])) > 0
  for: 15m
# e.g. missing RBAC permissions
- alert: KigeonKubernetesRequestErrors
  expr: sum by (code) (rate(kigeon_kubernetes_requests_total{code=~"4..|5.."}[10m])) > 0
  for: 15m
# no events received from the Kubernetes API
- alert: KigeonNoEventsWatched
  expr: sum(rate(kigeon_events_watched_total{source="kubernetes"}[1h])) == 0
  for: 1h

How it works

Kubernetes API
     │  (informer)
     ▼
EventPusher       ← watches all events cluster-wide
     │  (deduplicate + publish)
     ▼
EventQueue        ← embedded NATS JetStream (persistent)
     │  (fan-out)
     ├──▶ Sender 1 (loki) ──▶ Loki instance A
     └──▶ Sender 2 (loki) ──▶ Loki instance B

Each sender subscribes independently to the queue. An event is removed from the queue once all senders acknowledged it, so the queue only grows while a sender can not deliver events (bounded by eventsMaxAge and eventsMaxBytes). Subscriptions of senders which were removed from the configuration are deleted on startup. Namespace filters and hooks are applied per sender, so different senders can have completely different routing logic from the same event stream.

About

kigeon is a Kubernetes event exporter

Resources

Stars

3 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages