#Design a logging and monitoring system

The most write-heavy design in the set, and the only one where the system must stay up precisely when everything else is failing. Datadog, Splunk, Prometheus, CloudWatch.

#1 · Scope (0–5)

IN SCOPE                          OUT OF SCOPE
- ingest logs, metrics, traces    - APM auto-instrumentation
- store with retention tiers      - log parsing DSL
- query and dashboard             - incident management workflow
- alerting rules                  - billing

NON-FUNCTIONAL
- 10,000 hosts, 5M events/s peak
- ingest must NEVER block the applications sending data
- query: dashboards < 2s, ad-hoc search < 30s
- retention: metrics 13 months, logs 30 days, traces 7 days
- the system must survive the outage it is reporting on   <- the hard one
- some loss is acceptable for logs; NOT for alerting metrics

The requirement that makes this design different: "it must survive the outage it is reporting on." A monitoring system that fails with the infrastructure it monitors is worse than useless — it fails silently at the exact moment you need it. That drives isolation, independent failure domains, and a bias toward dropping data rather than blocking.

#2 · Estimation (5–8)

METRICS   10,000 hosts x 500 series each = 5M active series
          scraped every 10s -> 500,000 points/s
          a point is ~16 bytes compressed (timestamp delta + value)
          -> 500k x 16 B x 86,400 = ~700 GB/day raw
          with delta-of-delta + XOR compression: ~1-2 bytes/point
          -> ~50 GB/day        <- compression is the whole story

LOGS      10,000 hosts x 100 lines/s = 1M lines/s
          x 500 bytes = 500 MB/s = ~43 TB/day
          compressed ~10:1 -> ~4 TB/day
          x 30 days = ~120 TB                <- the dominant cost

TRACES    5M requests/s, sampled at 1% = 50,000 spans/s
          -> manageable, and sampling is why

CONCLUSION
  - metrics are small but numerous -> a purpose-built time-series store
  - logs are enormous -> object storage + an index, tiered aggressively
  - the ingest path is the bottleneck, not the query path
  - COMPRESSION and SAMPLING are not optimisations here, they are the design

"Logs are 100× the volume of metrics for a fraction of the query value" is the observation that justifies treating them completely differently.

#3 · Three signals, three shapes

The central design decision: do not build one pipeline.

Metrics answer "is something wrong?", traces answer "where?", logs answer "why?" That workflow is why they need different stores: you query metrics constantly and cheaply, traces occasionally, and logs rarely but deeply.

#4 · Architecture

flowchart TD
    APP["Applications<br/>10,000 hosts"] --> AG["Agent (per host)<br/>buffer · batch · compress"]
    AG -->|"push, backpressure-aware"| GW["Ingest gateway<br/>auth, rate limit, validate"]
    GW --> K[["Kafka<br/>partitioned by source"]]

    K --> MP["Metrics consumer"]
    K --> LP["Logs consumer"]
    K --> TP["Traces consumer"]

    MP --> TSDB[("Time-series DB<br/>13 months, downsampled")]
    LP --> IDX[("Search index<br/>hot: 7 days")]
    LP --> OBJ[("Object storage<br/>cold: 30 days")]
    TP --> TS[("Trace store")]

    TSDB --> AL["Alerting engine<br/>rule evaluation"]
    AL --> NOT["Notification<br/>paging, chat"]

    TSDB --> Q["Query / dashboards"]
    IDX --> Q
    TS --> Q

    classDef crit fill:#fbeee0,stroke:#b15a08,stroke-width:2px
    class AL,NOT crit

Kafka in the middle is the load-bearing decision, for four reasons worth stating:

#5 · The agent — where correctness starts

The most important component, because it is inside your customers' processes.

"Drop rather than block" is the answer, and it should be stated as a deliberate trade-off: losing some log lines is acceptable; adding latency to every customer request is not. Emit a metric counting dropped events — a silent drop is the failure mode that destroys trust in the whole system.

#6 · Metrics storage and the cardinality problem

The single most important concept on this page.

A time series is identified by its name PLUS its full label set.

    http_requests{service="api", region="us-east", status="200"}

Every distinct combination is a SEPARATE series to store and index.

    service (50) x region (10) x status (20)     = 10,000 series.  Fine.

Now someone adds user_id as a label:

    x user_id (10,000,000)                        = 10^11 series.

The database falls over. This is CARDINALITY EXPLOSION, and it is the
most common way a metrics system is destroyed -- by a one-line code change.

The right answer to "how do you handle high cardinality?" is not "scale the database". It is: that data does not belong in metrics. Metrics are for aggregates over bounded dimensions; per-user or per-request detail belongs in traces, joined by an ID. Saying that shows you understand the model rather than just the plumbing.

Storage layout:

Time-partitioned blocks (e.g. 2-hour chunks), each holding:
  - a compressed column of timestamps  (delta-of-delta encoding)
  - a compressed column of values      (XOR encoding — Gorilla-style)
  - an inverted index from label -> series IDs

Compression gets ~16 bytes/point down to 1-2 bytes, because
consecutive timestamps are evenly spaced and consecutive values
usually differ only in the low bits.

DOWNSAMPLING by age:
  0-7 days      raw, 10s resolution
  7-30 days     1-minute rollups
  30 days-13mo  1-hour rollups

Downsampling is what makes 13-month retention affordable, and it is lossy by design: nobody queries second-level detail from eight months ago, but everybody wants the yearly trend.

#7 · Logs — hot and cold

flowchart LR
    K[["Kafka"]] --> P["Parse · enrich · redact"]
    P --> H[("HOT: search index<br/>last 7 days<br/>fast, expensive")]
    P --> C[("COLD: object storage<br/>compressed, partitioned<br/>by hour + service")]
    H -->|"age out"| C
    Q["Query"] --> H
    Q -.->|"older than 7d"| C

Partition cold storage by time and service so a query for one service on one day reads a handful of objects rather than scanning everything. Partition pruning is what makes cold queries feasible at all.

Redact at ingest, not at query. Passwords, tokens and card numbers must never be written to storage — once they are in cold storage across thirty days of files, removing them is an incident.

#8 · Alerting

The part that decides whether the system is loved or muted.

flowchart TD
    R["Rule evaluation<br/>every 30s"] --> C{"Condition met?"}
    C -->|"no"| R
    C -->|"yes"| P["PENDING<br/>wait for the 'for' duration"]
    P -->|"still true"| F["FIRING"]
    P -->|"resolved"| R
    F --> D["Deduplicate<br/>+ group related alerts"]
    D --> S{"Silenced or<br/>inhibited?"}
    S -->|"yes"| DROP["suppress"]
    S -->|"no"| N["Notify: page / chat / ticket"]

Alert on symptoms, not causes — and on error-budget burn rate rather than fixed thresholds. See observability.

Multi-window burn-rate alerting is the mature answer: page when the budget is being consumed fast enough to matter — say 14× normal over an hour — and open a ticket when it is a slow burn at 2× over six hours. It catches real incidents quickly without paging for a blip.

And the meta-point: the alerting path must not depend on the systems it monitors. If alert evaluation runs on the same cluster as everything else, the outage takes out the alerting too. Run it in an independent failure domain, and have an external dead-man's switch that fires when the monitoring system stops reporting.

#9 · Failure and wrap

"Summary: three signals with genuinely different shapes, so three pipelines behind one ingest path. Kafka decouples ingest from storage, which matters most during an incident when the log volume spikes exactly as the backends struggle.

Metrics go to a time-series store with delta-of-delta and XOR compression — that is what turns 700 GB a day into 50 — and downsampling by age is what makes thirteen months affordable. Logs are 100× the volume for a fraction of the query value, so they are tiered: seven days in an index, the rest in object storage partitioned by hour and service.

The failure mode I would design hardest against is cardinality explosion, because it is caused by a one-line code change adding a user ID as a label and it takes the database down. Per-tenant series caps enforced at ingest, and the honest answer to high-cardinality data is that it belongs in traces, not metrics.

And the agent never blocks the application — it drops and counts what it dropped. A silent drop is the thing that destroys trust in a monitoring system."

#10 · Follow-ups

#Stop condition

You can do this design when you can:

1. justify three pipelines from the three signals' different shapes,
2. explain cardinality explosion and give the right answer, not "scale it",
3. describe metric compression and age-based downsampling,
4. defend hot/cold log tiering with the volume-versus-value argument,
5. explain why the agent drops rather than blocks — and counts drops, and
6. explain the dead-man's switch and why silence is the dangerous failure.