14-Tracing-Backend

Appearance

14.1 Requirements & Capacity Estimation ​

Functional Requirements ​

  • Ingest OpenTelemetry OTLP Protobuf and JSON span batches over HTTP/gRPC.
  • Execute tail-based sampling to retain 100% of error and latency outlier traces while downsampling nominal traces.
  • Reconstruct span DAGs on demand and compute the critical execution path.
  • Provide sub-second multi-dimensional search over billions of stored spans.
  • Run statistical anomaly detection and generate automated LLM incident postmortems.

Non-Functional Requirements ​

  • High Ingestion Availability: 99.99% ingestion uptime; ingestion failure must never crash instrumented upstream services.
  • Low Query Latency: p95 trace retrieval under 500ms for traces containing up to 5,000 spans.
  • Data Durability: Zero telemetry loss during graceful rolling collector restarts.

Capacity Estimation Arithmetic ​

Ingestion Volume ​

  • Target Scale: 10,000 requests/second across upstream microservices.
  • Spans Per Request: Average 15 spans generated per distributed transaction.
  • Gross Span Ingestion Rate: $10,000 \times 15 = 150,000\text{ spans/second}$.

Data Storage Volume ​

  • Average Uncompressed Span Size: ~500 bytes (service name, span name, tags, timestamps).
  • Gross Ingestion Bandwidth: $150,000 \times 500\text{ bytes} = 75\text{ MB/second} = 600\text{ Mbps}$.
  • Tail Sampling Reduction: 90% of healthy nominal traffic is discarded; 10% retained.
  • Net Ingestion Rate to Storage: $15,000\text{ spans/second} = 7.5\text{ MB/second}$.
  • ClickHouse Columnar Compression: 6x compression ratio using ZSTD on timestamps and dictionary encoding on strings.
  • Net Disk Write Rate: $\frac{7.5\text{ MB/s}}{6} = 1.25\text{ MB/second} \approx 108\text{ GB/day}$.
  • 30-Day Retention Disk Requirement: $108\text{ GB/day} \times 30\text{ days} \approx 3.24\text{ TB}$ of storage.

30-Second Spoken Interview Answer ​

"When designing capacity for this tracing engine, I modeled 10,000 user requests per second with an average fanout of 15 spans, generating 150,000 spans per second. Rather than storing every nominal span—which would require 6.5 terabytes per day—our tail sampling engine retains 100% of errors and p99 outliers while downsampling nominal traces to 10%. Combined with ClickHouse's 6x columnar compression, we write just 1.25 megabytes per second to disk, storing 30 days of high-fidelity incident history in under 3.3 terabytes."