Fleet tracing#

Bounded, live distributed observability across Breeze services, without requiring Jaeger, Zipkin, Tempo, or an OpenTelemetry Collector. Services propagate the standard W3C traceparent header, export spans asynchronously, and the Fleet Aggregator assembles them into a topology and merged timelines consumed by the existing dashboard.

Quick start#

tracer := fleet.New(fleet.TracerConfig{
	Enabled: true, ServiceName: "orders", AggregatorURL: "http://fleet:9000/fleet",
})
router.Use(fleet.Middleware(tracer)) // before the dashboard middleware
defer tracer.Close(context.Background())

Propagate explicitly on every downstream call — this is opt-in per call by design, and if omitted the downstream service safely starts a new root trace:

req, _ := http.NewRequest(http.MethodPost, ordersURL, body)
fleet.PropagateFromHTTP(ctx, req)
resp, err := http.DefaultClient.Do(req)

Run the aggregator, then activate the Fleet page in an existing dashboard:

go run ./cmd/fleet-aggregator

cfg := dashboard.DefaultConfig()
cfg.FleetAggregatorURL = "http://127.0.0.1:9000/fleet"
dashboard.Install(app, router, cfg)

Migrating a single-service dashboard to Fleet#

The application-side change is deliberately three lines:

+ tracer := fleet.New(fleet.TracerConfig{Enabled: true, ServiceName: "orders", AggregatorURL: os.Getenv("FLEET_URL")})
+ router.Use(fleet.Middleware(tracer))
  router.Use(coll.Middleware())
+ defer tracer.Close(context.Background())

When dashboard.Config.FleetAggregatorURL is empty, the 15th dashboard page and its navigation entry are simply absent, preserving the old dashboard exactly.

Features#

• W3C Trace Context parsing and propagation; malformed headers safely become new roots and increment a counter.
• Trace-wide fixed-rate sampling plus lightweight always-on error spans.
• Custom fleet.Tag(ctx, key, value) attributes propagated as bounded baggage.
• Lock-bounded local buffering and asynchronous batch export with capped retry.
• Bounded aggregator storage, service heartbeats, trace assembly, orphan spans, skew flags, topology percentiles, deterministic root cause, and blast radius.
• Asynchronous contract checks against heartbeat-advertised OpenAPI schemas.
• A dashboard Fleet page with topology, trace filters, merged waterfalls, violations, catalog, and incident state.
• Trace-correlated log stitching — every log line inside a request is stamped with its trace id automatically, and the merged view fans out to each service's own log store instead of duplicating log storage in the aggregator.

Reading the topology graph#

The graph is a mesh, not a tree. Every observed caller-to-callee pair draws as two arcs — the request leaving the caller and the response coming back — bowed to opposite sides so they never overlap. One arrow per pair would tell you A calls B but never whether B answered, which is precisely what you need during an incident.

Each arc carries its timing inline: with no trace open, edges show the aggregate over the retention window (p50 5.0ms · p95 9.0ms); with a trace open, they switch to that trace's measured hop durations, and a hop called more than once shows its total and a ×n count. Timings print on the graph rather than hiding behind a hover, so a screenshot pasted into an incident channel still carries its data.

While stepping through a trace with the playback controls, arc styling tracks the cursor: the request leg lights up while a hop is in flight, and the response leg lights up once the cursor passes the hop's last span — red if that hop's worst status was a 5xx.

Why this differs from Jaeger, Zipkin, or Datadog#

Live contract validation against the schema the running service itself generates. A generic tracing backend sees spans but has no framework-owned, always-current OpenAPI model tied to each route — Fleet can report a real request missing a required field, or a response returning the wrong type, without a separately maintained Pact fixture.

Deterministic root-cause and blast-radius highlighting. Fleet walks the causal span tree to mark one earliest failure, labels later failures as derived effects, and traverses its incrementally maintained service graph to show which observed dependencies are impacted. This is graph math, not probabilistic inference — explainable, and available offline.

Configuration#

Tracer#

Aggregator#

Transport status#

Networked event delivery goes through the generic events.Backend seam (Publish/Subscribe), so the broker is pluggable rather than a fork of the transport. Memory (default) needs no external infrastructure. Kafka is shipped as the durable/replayable option, in its own nested Go module at fleet/transport/eventtransport/backends/kafka — the kafka-go client is only compiled by applications that opt in:

backend, err := kafka.New(kafka.Config{
	Brokers: []string{"kafka-1:9092", "kafka-2:9092"},
	GroupID: "fleet-aggregator",
})

The base module's go.mod/go.sum are unchanged by this feature — no broker, WebSocket, or gRPC dependency is reachable from a plain go get github.com/nelthaarion/breeze/v2.

Security#

Use both credential classes in production: IngestToken protects span/heartbeat ingestion writes (constant-time compared), and Username+Password protect human read APIs — both must be non-empty for Basic Auth to be enforced, matching the dashboard's own convention. Keep the aggregator on a private network and terminate TLS at the normal service edge. Spans contain header names, never header values; error text and captured JSON payloads are scrubbed at the originating service before export.

Architecture#

Performance and limits#

Disabled RecordSpan and well-formed traceparent parsing stay at zero allocations. Export never blocks a handler — overload drops the oldest local span rather than growing memory or waiting on the network.

Known v1 limits, stated as boundaries rather than hidden failure modes: no durable storage, no multi-aggregator coordination, no OTLP, and no completed gRPC transport.

Running the example#

The complete three-service example is cmd/fleet-example:

powershell -File cmd/fleet-example/build.ps1
docker compose -f cmd/fleet-example/docker-compose.yml up --build
curl http://localhost:3000/api/orders/123

Each service excludes its own health-check and introspection routes from tracing with a skipUntraced predicate wrapped around fleet.Middleware — worth copying for your own app, because the aggregator's own /openapi.json fetch carries no inbound traceparent, and tracing it would misrender the deepest hop in the fleet as the root of a new trace.

To exercise deterministic root-cause and blast-radius highlighting:

curl -X POST http://localhost:3002/chaos/fail
curl http://localhost:3000/api/orders/123
curl -X POST http://localhost:3002/chaos/recover