Self-Hosting

Candela is designed to run on your infrastructure. This guide covers deploying the server with Docker Compose for local development, Cloud Run for production, and the full configuration reference.

Deployment Architectures

Candela supports two deployment topologies:

Self-Hosted / Zero-Cloud:
- Span Storage: Embedded DuckDB (candela.duckdb) or SQLite (candela.db)
- User & Governance Store: Embedded SQLite (candela.db)
- Model Providers: Local runtimes (Ollama, vLLM) for air-gapped deployments, or direct API keys (OpenAI, Anthropic, Mistral, DeepSeek, Qwen) when outbound network access is available
- Dependencies: No GCP, Firebase, or external database dependency; hosted model providers still require outbound network access.
Cloud-Connected / Enterprise (GCP):
- Span Storage: BigQuery (partitioned analytics dataset)
- User & Governance Store: Cloud Firestore
- Model Providers: Vertex AI, direct API keys, or hybrid
- Dependencies: GCP Project, Firebase, IAM service account

Prerequisites (Cloud Mode Only)

If deploying in Cloud-Connected mode with GCP services, ensure you have:

GCP project with billing enabled
Firebase project linked to your GCP project (for Firestore-backed user management)
APIs enabled in your GCP project:
- Vertex AI API (aiplatform.googleapis.com)
- Cloud Firestore API (firestore.googleapis.com)
- BigQuery API (bigquery.googleapis.com) — if using BigQuery storage
Docker installed locally (for building images)
gcloud CLI authenticated (gcloud auth login)

Required IAM Roles (Cloud Mode)

The service account running Candela in GCP needs these roles:

Role	Purpose
`roles/aiplatform.user`	Proxy requests to Vertex AI models
`roles/datastore.user`	Read/write Firestore (users, budgets, catalog)
`roles/bigquery.dataEditor`	Write spans to BigQuery (production storage)
`roles/bigquery.jobUser`	Run BigQuery queries (dashboard)

Quick Start with Docker Compose

The fastest way to run Candela locally:

# From the candela repo root
docker compose -f deploy/docker-compose.yml up

This starts the Candela server with:

DuckDB storage (local file, no external dependencies)
Dev mode auth (no Firebase token validation)
All proxy providers enabled
UI on http://localhost:3000, API on http://localhost:8181

Health check endpoint: http://localhost:8181/readyz

Docker Image

Candela provides a multi-stage Dockerfile that builds the Go backend with DuckDB support:

# Build the server image
docker build -f deploy/Dockerfile -t candela-server .

# Run with DuckDB storage
docker run -p 8181:8181 \
  -e CANDELA_STORAGE_BACKEND=duckdb \
  candela-server

The image runs as a non-root candela user (CIS Docker Benchmark 4.1) and uses Debian Bookworm slim for glibc compatibility with DuckDB.

Cloud Run Deployment

For production, deploy to Cloud Run with BigQuery storage:

1. Build and Push

# Build and push to Artifact Registry
gcloud builds submit \
  --tag us-docker.pkg.dev/YOUR_PROJECT/candela/candela-server:latest \
  --dockerfile deploy/Dockerfile

2. Deploy

gcloud run deploy candela-server \
  --image us-docker.pkg.dev/YOUR_PROJECT/candela/candela-server:latest \
  --region us-central1 \
  --port 8181 \
  --memory 1Gi \
  --cpu 2 \
  --min-instances 1 \
  --set-env-vars "\
CANDELA_STORAGE_BACKEND=bigquery,\
CANDELA_BQ_PROJECT=YOUR_PROJECT,\
CANDELA_BQ_DATASET=candela,\
CANDELA_BQ_LOCATION=US,\
CANDELA_PROXY_ENABLED=true,\
CANDELA_VERTEX_PROJECT=YOUR_PROJECT,\
CANDELA_VERTEX_REGION=us-east5,\
CANDELA_FIRESTORE_PROJECT=YOUR_PROJECT,\
CANDELA_FIRESTORE_DATABASE=candela,\
CANDELA_DEV_MODE=false,\
CANDELA_DEFAULT_DAILY_BUDGET=5.00"

3. Verify

curl https://candela-server-HASH-uc.a.run.app/readyz
# {"status":"ok"}

Environment Variables

The entrypoint script generates config.yaml from environment variables at startup, so you never need to bake secrets into the image.

Variable	Default	Description
`CANDELA_STORAGE_BACKEND`	`duckdb`	Span storage backend: `duckdb`, `sqlite`, `bigquery`
`CANDELA_USER_BACKEND`	`sqlite` (or `firestore` if configured)	User & budget storage backend: `sqlite`, `firestore`, `none`
`CANDELA_SQLITE_USER_PATH`	`candela.db`	SQLite database file for user records, budgets, grants, and rate limits
`CANDELA_BQ_PROJECT`	(empty)	BigQuery GCP project ID
`CANDELA_BQ_DATASET`	`candela`	BigQuery dataset name
`CANDELA_BQ_LOCATION`	`US`	BigQuery dataset location
`CANDELA_PROXY_ENABLED`	`false`	Enable the LLM proxy
`CANDELA_VERTEX_PROJECT`	(empty)	Vertex AI GCP project ID
`CANDELA_VERTEX_REGION`	`us-east5`	Vertex AI region
`CANDELA_CACHING_MODE`	`auto`	Prompt caching: `off`, `auto`, `system-only`
`CANDELA_CACHE_TTL`	(empty)	Vertex AI cache TTL (`5m` or `1h`)
`CANDELA_DEV_MODE`	`false`	Skip Firebase auth validation
`CANDELA_FIRESTORE_PROJECT`	(empty)	Firestore GCP project ID
`CANDELA_FIRESTORE_DATABASE`	`candela`	Firestore database ID
`CANDELA_DEFAULT_DAILY_BUDGET`	`5.00`	Default daily budget for new users (USD)
`CANDELA_SERVICE_ACCOUNT`	(empty)	Allowed service account email
`CANDELA_MISTRAL_REGION`	`us-central1`	Vertex AI region for Mistral models

Storage Backends

DuckDB (Analytics & Local Development)

DuckDB is an embedded OLAP database — no external services, high-performance columnar analytics, single-file storage.

storage:
  backend: "duckdb"
  duckdb:
    path: "candela.duckdb"

Best for: local development, single-user setups, high-performance local span queries.

SQLite (Lightweight & Testing)

SQLite can be used for span storage in low-concurrency or ephemeral environments:

storage:
  backend: "sqlite"
  sqlite:
    path: "candela.db"  # Use ":memory:" for ephemeral

BigQuery (Production Cloud)

BigQuery provides unlimited scale, partitioned span tables, and deep cost analytics across your enterprise.

storage:
  backend: "bigquery"
  bigquery:
    project_id: "your-gcp-project-id"
    dataset: "candela"
    table: "spans"
    location: "US"

Best for: enterprise production, multi-tenant teams, long-term analytics.

User Management & Governance Backends

Candela manages user identities, access tiers, budgets, task spend, and token-bucket rate limits using a pluggable UserStore.

SQLite UserStore (Zero-Cloud / Air-Gapped)

When running self-hosted without Firebase or GCP, Candela defaults to an embedded SQLite UserStore. This provides full governance capabilities locally:

Users & Status: Active, suspended, and soft-deleted states
Budgets & Grants: Daily/monthly spend tracking, automatic overdraft grace, and time-bounded grant waterfalls
Model Limits: Request and token caps per user and model
Task Budgets: Scoped budget caps tied to specific agent tasks (task_id)
Rate Limiting: Sliding token-bucket rate limiters per user
Audit Logging: Structured user and global governance audit trails

user_store:
  backend: "sqlite"
  path: "candela.db"

Firestore UserStore (Enterprise Cloud)

For multi-region, distributed deployments backed by Google Cloud:

firestore:
  enabled: true
  project_id: "your-gcp-project-id"
  database_id: "candela"

Full Configuration Reference

# Candela Server Configuration
server:
  host: "0.0.0.0"
  port: 8181

storage:
  backend: "duckdb"              # duckdb | sqlite | bigquery
  duckdb:
    path: "candela.duckdb"
  sqlite:
    path: "candela.db"
  bigquery:
    project_id: ""
    dataset: "candela"
    table: "spans"
    location: "US"

proxy:
  enabled: true
  project_id: "default"
  vertex_ai:
    project_id: ""               # GCP project for Vertex AI
    region: "us-east5"           # Vertex AI region
    prompt_caching: true         # Enable Anthropic prompt caching
    cache_ttl: "5m"              # Cache TTL: "5m" or "1h"
    provider_overrides:          # Per-provider region overrides
      mistral:
        region: "us-central1"
  providers:                     # Enabled proxy providers
    - openai
    - google
    - anthropic
    - anthropic-vertex
    - anthropic-direct
    - anthropic-bedrock
    - gemini-oai
    - gemini-vertex
    - mistral
    - deepseek
    - qwen

  # Safety controls
  # max_request_cost_usd: 5.00  # Reject single requests above this cost
  # daily_limits:                # Per-model daily spend caps
  #   - model: "claude-opus-4"
  #     max_daily_usd: 50.00

  # Model allowlist
  # policy:
  #   allowed_models:
  #     - provider: openai
  #       models: ["gpt-4o", "gpt-4o-mini"]

cors:
  allowed_origins:
    - "http://localhost:3000"
    - "http://localhost:8181"

auth:
  dev_mode: false
  # allowed_service_accounts:
  #   - "candela-ci@project.iam.gserviceaccount.com"

catalog:
  backend: "config"              # config | firestore
  # firestore:
  #   collection: "model_catalog"
  #   project_id: ""

pricing:
  discount_percent: 0.0
  # models:
  #   - provider: openai
  #     model: gpt-4o
  #     input_per_million: 2.00
  #     output_per_million: 8.00

# User store configuration (self-hosted / zero-cloud)
user_store:
  backend: "sqlite"              # sqlite | firestore | none
  path: "candela.db"

firestore:
  enabled: false                 # set true to use GCP Firestore for users & budgets
  project_id: ""
  database_id: "candela"

worker:
  batch_size: 100
  flush_interval: "2s"

users:
  default_daily_budget_usd: 5.00

# budget:
#   timezone: "America/New_York"

# Export sinks
# sinks:
#   otlp:
#     enabled: true
#     required: false
#     endpoint: "http://localhost:4318"
#     protocol: "http"
#     compression: "gzip"
#     timeout_sec: 30
#     headers:
#       Authorization: "Bearer <token>"

HTTPS and Custom Domains

Cloud Run

Cloud Run provides HTTPS automatically. To use a custom domain:

gcloud run domain-mappings create \
  --service candela-server \
  --domain candela.example.com \
  --region us-central1

Behind a Load Balancer

If running on GKE or Compute Engine, terminate TLS at the load balancer and proxy to Candela on port 8181.

Monitoring and Health Checks

Health Endpoint

Candela exposes /readyz for container orchestrator health checks:

curl http://localhost:8181/readyz
# {"status":"ok"}

The Docker Compose configuration includes a built-in health check:

healthcheck:
  test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://localhost:8181/readyz"]
  interval: 10s
  timeout: 5s
  retries: 5

Structured Logging

Candela emits structured JSON logs compatible with Cloud Logging. Key log fields:

level — INFO, WARN, ERROR
msg — human-readable message
provider, model — for proxy request logs
user_id — for budget and auth events
cost_usd, tokens — for cost tracking

OTLP Export

Export spans to any OpenTelemetry-compatible backend (Datadog, Grafana Tempo, Jaeger, Honeycomb):

sinks:
  otlp:
    enabled: true
    required: false                    # fail-open if collector is down
    endpoint: "http://localhost:4318"  # OTel Collector OTLP/HTTP endpoint
    protocol: "http"                   # "http" or "grpc"
    compression: "gzip"
    timeout_sec: 30
    headers:
      Authorization: "Bearer <token>"