Formicary

Formicary is a distributed, cloud-native orchestration engine for executing complex workflows, data pipelines, and CI/CD jobs.

Formicary uses a declarative YAML-based approach to define jobs as a Directed Acyclic Graph (DAG) of tasks. It's built on a robust leader-follower architecture (Queen and Ants) that efficiently distributes work to executors running on Docker, Kubernetes, or as simple shell commands.

It is designed for use cases that require complex dependency management, parallel execution, and sophisticated error handling.

Key Features

• Declarative Workflows: Define complex jobs with multiple tasks and dependencies in simple YAML files.
• Extensible Executors: Natively supports Kubernetes, Docker, Shell, HTTP/REST, and Websockets.
• Advanced Flow Control: Sophisticated retry logic, conditional execution (on_exit_code), error handling, and optional tasks.
• Parallel & Concurrent Execution: Fork jobs to run in parallel and join the results, with concurrency limits.
• Powerful Templating: Use Go templates in your job definitions for dynamic workflows.
• Built-in Caching: Speed up jobs with caching for dependencies (e.g., node_modules, vendor, .m2).
• Robust Artifacts Management: Persist task outputs to an S3-compatible object store for use in later stages or for download.
• Security First: RBAC, encrypted secrets, and JWT-based API authentication.
• Real-time & Observability: Stream logs in real-time, get Prometheus metrics, and view detailed job statistics.

Getting Started

The quickest way to run Formicary is with a single Docker command — no clone required.

Auth is enabled by default. Supply your JWT secret and OAuth credentials, or pass COMMON_AUTH_ENABLED=false to disable auth for local testing.

Option A — Google OAuth (default, auth enabled):

mkdir -p ~/formicary-data
# Patch kubeconfig: replace 127.0.0.1 with host.docker.internal and skip TLS verify
# (Docker Desktop's k8s cert is valid for localhost, not host.docker.internal)
python3 -c "
import sys, yaml
with open(sys.argv[1]) as f: kc = yaml.safe_load(f)
for c in kc.get('clusters', []):
    cl = c.get('cluster', {})
    cl['server'] = cl.get('server','').replace('https://127.0.0.1','https://host.docker.internal')
    cl.pop('certificate-authority-data', None)
    cl['insecure-skip-tls-verify'] = True
with open(sys.argv[2], 'w') as f: yaml.dump(kc, f)
" ~/.kube/config ~/formicary-data/kubeconfig
docker run --rm -p 7777:7777 -p 19000:19000 \
  -e COMMON_AUTH_JWT_SECRET=<your-jwt-secret> \
  -e COMMON_AUTH_GOOGLE_CLIENT_ID=<your-google-client-id> \
  -e COMMON_AUTH_GOOGLE_CLIENT_SECRET=<your-google-client-secret> \
  -e COMMON_AUTH_GOOGLE_CALLBACK_HOST=localhost \
  -v ~/formicary-data:/data \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v ~/formicary-data/kubeconfig:/home/formicary-user/.kube/config:ro \
  plexobject/formicary:latest

Option B — No auth (local testing only):

mkdir -p ~/formicary-data
python3 -c "
import sys, yaml
with open(sys.argv[1]) as f: kc = yaml.safe_load(f)
for c in kc.get('clusters', []):
    cl = c.get('cluster', {})
    cl['server'] = cl.get('server','').replace('https://127.0.0.1','https://host.docker.internal')
    cl.pop('certificate-authority-data', None)
    cl['insecure-skip-tls-verify'] = True
with open(sys.argv[2], 'w') as f: yaml.dump(kc, f)
" ~/.kube/config ~/formicary-data/kubeconfig
docker run --rm -p 7777:7777 -p 19000:19000 \
  -e COMMON_AUTH_ENABLED=false \
  -e COMMON_AUTH_JWT_SECRET=<your-jwt-secret> \
  -v ~/formicary-data:/data \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v ~/formicary-data/kubeconfig:/home/formicary-user/.kube/config:ro \
  plexobject/formicary:latest

Data (SQLite DB + artifacts) is stored in ~/formicary-data/ — inspect with ls ~/formicary-data/ after first run.

The kubeconfig patch replaces 127.0.0.1 with host.docker.internal — Docker Desktop's special hostname that lets containers reach the host's Kubernetes API. make docker-run does this automatically.

Both options start the Queen server, an embedded Ant worker, embedded SeaweedFS (artifact storage), and SQLite — no Redis, MinIO, or separate services needed.

To use a locally-built image instead of the published one:

DOCKER_IMAGE=formicary:latest make docker-run

Or use docker-compose (persistent data, restart policy):

curl -fsSL https://raw.githubusercontent.com/bhatti/formicary/main/docker-compose.yaml -o docker-compose.yaml
COMMON_AUTH_JWT_SECRET=<your-jwt-secret> \
COMMON_AUTH_GOOGLE_CLIENT_ID=<your-google-client-id> \
COMMON_AUTH_GOOGLE_CLIENT_SECRET=<your-google-client-secret> \
docker compose up

Explore!

• Open the Formicary Dashboard at http://localhost:7777.
• Follow the Quick Start Guide to run your first job.

Documentation

• Introduction
  • Introduction to Formicary
  • Core Concepts
  • Architecture Deep Dive
• Getting Started
  • Installation Guide
  • Quick Start Tutorial
• Guides & Tutorials
  • Job & Task Definitions
  • Executors
  • Scheduling & Triggers
  • Artifacts & Caching
  • CI/CD Pipelines
  • Data Pipelines
  • Advanced Workflows
  • Security Guide
• Reference
  • Configuration Reference
  • CLI Reference
  • API Reference
  • Troubleshooting & FAQ
• Community
  • Contributing Guide
  • Code of Conduct
  • License

Comparison with Other Tools

• See how Formicary compares to tools like Airflow, Jenkins, GitLab CI, and GitHub Actions.

Blogs/Articles

• Building a distributed orchestration and graph processing system
• Building Resilient, Interactive Playbooks with an Orchestration Engine
• Task Scheduling Algorithms in Distributed Orchestration Systems

License

Formicary is licensed under the GNU AGPLv3 License.