Bilis
Features Pricing Docs Blog
Documentation

Ingestion

View raw

Traces

Sending spans over OTLP/HTTP, why gRPC is not supported, per-SDK setup, one Collector config for both signals, and how traces link to logs.

Bilis accepts distributed traces at one endpoint, in the same two encodings the logs endpoint takes, with the same API key and the same never-blame-the-client contract.

Endpoint Payload Success
POST /api/v1/traces OTLP ExportTraceServiceRequest, JSON or protobuf 200

One row is stored per span. Spans that cannot be parsed are skipped and counted in an OTLP partialSuccess response; a storage failure answers 503 with Retry-After. Bilis never returns a 4xx for the contents of a payload — OTel clients treat 4xx as permanent and drop the batch. Bodies sent with Content-Encoding: gzip or deflate are inflated, which is what the SDKs and the Collector send by default.

gRPC is not supported

This is the thing that will look like an outage and is not one. The OpenTelemetry Collector and most SDKs default to OTLP over gRPC on port 4317. Bilis speaks OTLP over HTTP only. Point your exporter at the HTTP protocol explicitly, and give it Bilis' /api prefix as the base:

OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf   # or http/json
OTEL_EXPORTER_OTLP_ENDPOINT=https://your-bilis-host/api
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer%20bilis_your_key_here

With http/protobuf or http/json the SDK appends the signal path to the signal-agnostic endpoint itself — /v1/traces for spans, /v1/logs for logs — which is why the value above ends in /api and not in /api/v1/traces. Set to the bare host, an SDK would post to /v1/traces, which does not exist here.

The header value is url-encoded, except where it is not

OTEL_EXPORTER_OTLP_HEADERS is specified as url-encoded, which is why the space in Bearer bilis_... is written %20 above. The Collector and the Go SDK decode it. The PHP SDK does not — its MapParser splits on = and , and trims, and nothing else — so a PHP exporter sends the literal string Bearer%20bilis_..., which is not a bearer token, and every export comes back 401. It looks exactly like a wrong key.

The reliable way out is to stop needing an encoded character at all. Bilis accepts the key in X-Bilis-Key as well as in Authorization, and that value has no space in it:

OTEL_EXPORTER_OTLP_HEADERS=x-bilis-key=bilis_your_key_here

That line is correct in every SDK and in the Collector, with or without url-decoding, quoted or bare. Prefer it.

The per-signal variable is used verbatim instead, so that one names the full path:

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://your-bilis-host/api/v1/traces

Either form works; use the per-signal one when logs and traces go to different places.

Quickstart per SDK

Each of these is the auto-instrumentation route: no spans written by hand, the SDK instruments your HTTP server, client and database driver. Set OTEL_SERVICE_NAME — it colours the span's bar in the waterfall and is the service filter in the trace list. Metrics are switched off in each example because Bilis has nowhere to put them; an exporter left on would retry them forever.

Node

npm install @opentelemetry/api @opentelemetry/auto-instrumentations-node

OTEL_SERVICE_NAME=checkout
OTEL_TRACES_EXPORTER=otlp
OTEL_METRICS_EXPORTER=none
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_ENDPOINT=https://your-bilis-host/api
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer bilis_your_key_here"

node --require @opentelemetry/auto-instrumentations-node/register app.js

Python

pip install opentelemetry-distro opentelemetry-exporter-otlp-proto-http
opentelemetry-bootstrap -a install

OTEL_SERVICE_NAME=checkout
OTEL_TRACES_EXPORTER=otlp
OTEL_METRICS_EXPORTER=none
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_ENDPOINT=https://your-bilis-host/api
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer bilis_your_key_here"

opentelemetry-instrument python app.py

The Python distro defaults to gRPC, so OTEL_EXPORTER_OTLP_PROTOCOL is not optional here.

Go

Go has no auto-instrumentation switch; the exporter is wired in code and the libraries you use are wrapped (otelhttp, otelsql, and so on). The exporter is protobuf-only, which Bilis decodes:

exporter, err := otlptracehttp.New(ctx,
	otlptracehttp.WithEndpoint("your-bilis-host"),
	otlptracehttp.WithURLPath("/api/v1/traces"),
	otlptracehttp.WithHeaders(map[string]string{
		"Authorization": "Bearer " + os.Getenv("BILIS_API_KEY"),
	}),
	otlptracehttp.WithCompression(otlptracehttp.GzipCompression),
)
if err != nil {
	return err
}

provider := sdktrace.NewTracerProvider(
	sdktrace.WithBatcher(exporter),
	sdktrace.WithResource(resource.NewSchemaless(semconv.ServiceName("checkout"))),
)
otel.SetTracerProvider(provider)
defer provider.Shutdown(ctx)

http.ListenAndServe(":8080", otelhttp.NewHandler(mux, "server"))

The environment variables from the previous section work too — OTEL_EXPORTER_OTLP_TRACES_ENDPOINT replaces the WithEndpoint / WithURLPath pair. The Go guide covers logs from the same service.

PHP and Laravel

There are two ways to instrument a Laravel application, and the difference between them is whether you can install a PECL extension.

Without the extension — keepsuit/laravel-opentelemetry. This is what Bilis itself uses. It instruments through Laravel's own events and container rather than through userland hooks, so it needs no ext-opentelemetry, and it knows about Octane, Horizon and queue workers, which is what decides whether spans ever get flushed in a long-running process.

composer require keepsuit/laravel-opentelemetry
php artisan vendor:publish --tag=opentelemetry-config

OTEL_SERVICE_NAME=checkout
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_ENDPOINT=https://your-bilis-host/api
OTEL_EXPORTER_OTLP_HEADERS=x-bilis-key=bilis_your_key_here
OTEL_METRICS_EXPORTER=null
OTEL_LOGS_EXPORTER=null

Two of those lines are load-bearing and are not defaults. The package ships with the metrics and logs exporters set to otlp: left alone, it posts metrics to /v1/metrics, which Bilis does not serve — metrics are out of scope — and retries three times before printing a stack trace, once per request. Set both to null. Logs should leave a Laravel app through the Monolog channel instead, which is where the correlation between those lines and these spans is described.

Two more worth knowing before you go looking for missing spans:

  • Console instrumentation is a whitelist. ConsoleInstrumentation is enabled by default but its commands list is empty, and an empty list traces nothing. Name the commands you care about, or '*'-suffixed prefixes.
  • Exclude whatever path you export to, if you are exporting to an application that is itself instrumented. Tracing the receiving endpoint means each export produces spans, which are exported, which produce spans. Bilis ships with api/v1/traces in excluded_paths for exactly this reason.

With the extension — the official opentelemetry-auto-laravel. Vendor neutral, supports older Laravel versions, and requires ext-opentelemetry:

pecl install opentelemetry
composer require open-telemetry/sdk open-telemetry/exporter-otlp \
    open-telemetry/opentelemetry-auto-laravel

OTEL_PHP_AUTOLOAD_ENABLED=true
OTEL_SERVICE_NAME=checkout
OTEL_TRACES_EXPORTER=otlp
OTEL_METRICS_EXPORTER=none
OTEL_LOGS_EXPORTER=none
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf   # http/json if you skip the protobuf dependency
OTEL_EXPORTER_OTLP_ENDPOINT=https://your-bilis-host/api
OTEL_EXPORTER_OTLP_HEADERS=x-bilis-key=bilis_your_key_here

Either way, requests, queued jobs, Eloquent queries, cache calls and outbound HTTP become spans without a code change.

Collector configuration

If a Collector sits between your services and Bilis, one otlphttp exporter carries both signals. This is the complete, runnable file — receivers, extension and both pipelines — and it is the same one the Shippers and Go pages refer to:

receivers:
    otlp:
        protocols:
            # Your services may still speak gRPC to the Collector; only the
            # hop to Bilis has to be HTTP.
            grpc: { endpoint: 0.0.0.0:4317 }
            http: { endpoint: 0.0.0.0:4318 }

extensions:
    file_storage:
        directory: /var/lib/otelcol/storage

exporters:
    otlphttp/bilis:
        logs_endpoint: https://your-bilis-host/api/v1/logs
        traces_endpoint: https://your-bilis-host/api/v1/traces
        headers:
            Authorization: Bearer bilis_your_key_here
        # gzip is the exporter's default; Bilis inflates gzip and deflate.
        compression: gzip
        # Protobuf is the default encoding and is decoded in-process;
        # `encoding: json` works too.
        sending_queue:
            enabled: true
            storage: file_storage # survives a Collector restart
        retry_on_failure:
            enabled: true

service:
    extensions: [file_storage]
    pipelines:
        logs:
            receivers: [otlp]
            processors: [] # deliberately no batch processor
            exporters: [otlphttp/bilis]
        traces:
            receivers: [otlp]
            processors: []
            exporters: [otlphttp/bilis]

Use the exporter's own sending_queue rather than the standalone batch processor, and give the queue persistent storage if you care about surviving a Collector restart. There is no metrics pipeline on purpose: a metrics pipeline pointed at Bilis would fail every export. The reasoning behind each setting is on the Shippers page.

Sending a span by hand

curl -X POST https://your-bilis-host/api/v1/traces \
  -H "Authorization: Bearer bilis_your_key_here" \
  -H "Content-Type: application/json" \
  -d '{
    "resourceSpans": [{
      "resource": {"attributes": [
        {"key": "service.name", "value": {"stringValue": "checkout"}}
      ]},
      "scopeSpans": [{
        "spans": [{
          "traceId": "5b8efff798038103d269b633813fc60c",
          "spanId": "eee19b7ec3c1b174",
          "name": "POST /checkout",
          "kind": 2,
          "startTimeUnixNano": "1756550400000000000",
          "endTimeUnixNano": "1756550400250000000",
          "status": {"code": 2, "message": "checkout failed"}
        }]
      }]
    }]
  }'

traceId is 32 hex characters and spanId is 16. A span with no parentSpanId, or one whose parent is all zeroes, is the trace's root — its name and service are what the trace list shows, so a trace whose root never arrives is listed without an operation name.

kind and status.code accept the proto enum number (as above) or its name (SPAN_KIND_SERVER, STATUS_CODE_ERROR). Both are stored in the OpenTelemetry Collector's own spelling: Server, Error.

Traces and logs are linked both ways

The logs table already carries TraceId and SpanId. When your logger records them — most OTel-instrumented loggers do automatically — Bilis joins the two signals for you:

  • a log line showing a trace id gets a link straight to that trace's waterfall,
  • a span in the waterfall links back to the log viewer, filtered to that exact trace and span.

This is the main reason to keep both signals in one place, and it costs nothing beyond having the ids on the log record. For the simple JSON endpoint that means a top-level trace_id and span_id; the Laravel Monolog channel lifts them out of the log context for you — see Shippers. Exceptions that arrive through the Sentry-compatible endpoint link the same way when the SDK put a trace context on the event.

Retention

Spans are kept for 30 days. Trace summaries — the row behind each entry in the trace list, carrying the root operation, duration, span count and error count — are kept for 90 days.

So a trace can outlive its own detail: after 30 days it still appears in the list with its shape intact, but its waterfall can no longer be drawn. That is deliberate, and the interface says so rather than showing you an empty chart. The ALTER statements that change either window are in Limits and behavior.

Sizing

A span costs roughly 70 bytes on disk after compression, and applications typically emit far more spans than log lines — an order of magnitude is a reasonable planning assumption. At 1,000 spans per second, 30 days of retention is about 200 GB.

If that is too much, in order of what to try first: sample at the collector, shorten the span TTL, or drop span events. A full disk turns ClickHouse read-only and takes the whole install down, so watch it before it matters.