Designing Petabyte Log Search: Index Everything vs Grep Smarter

Logs invert every assumption search engines are built on. A web search index is small relative to queries — millions of queries per second over a corpus built once. Logs are the opposite: petabytes written per day, and any given log line has a near-zero probability of ever being read. Designing log search is therefore an economics problem wearing an engineering costume: how much do you spend at write time (indexing) to accelerate a read that probably never comes? The industry's three answers — index everything, index almost nothing, and the bloom-filter middle — make this the cleanest cost-model interview in the observability family.

Answer 1: full inverted index (the Splunk/Elasticsearch shape)

Tokenize every line, build an inverted index (term → posting list), shard by time + index name. Queries are instant term lookups; analysts get ad-hoc full-text search over everything.

The bill: index size rivals the raw data (often 50-100%+ with positions), and write amplification means a petabyte/day of logs needs a massive hot cluster doing CPU-heavy tokenization on data nobody may read. High-cardinality tokens (UUIDs, request IDs — which is most of a log line) bloat posting lists that get consulted once or never. This shape is right when search is the product (security/SIEM, where analysts hunt through everything daily) and indefensible for "we keep logs in case something breaks."

Answer 2: index labels only, grep the rest (the Loki bet)

Loki's design bet: logs arrive already organized by their source — {service, pod, region}. Index only those labels (a tiny index, kilobytes per stream), store the log lines as compressed chunks in object storage, and at query time brute-force scan the chunks matching the label selector and time range, in parallel:

{service="checkout", region="eu"} |= "timeout" [last 1h]
  -> index: which streams match labels -> which chunks in range (tiny lookup)
  -> fan out: N workers decompress + grep chunks -> merge

The arithmetic that makes it sane: labels + time typically narrow a petabyte to gigabytes, and sequential decompress-and-scan runs at GB/s per core — a 100-worker scatter chews 500GB in seconds. Storage is just compressed blobs on S3 (10-20× cheaper than an indexed hot tier), ingestion is nearly free of CPU, and the system scales readers elastically only when someone actually searches. The constraint it imposes is cardinality discipline, same as the metrics post: user_id as a label would explode the stream count — identities belong inside the line, found by grep, not in the index.

Answer 3: the middle — skip indexes and bloom filters

Between the poles: store per-chunk skip metadata — bloom filters over tokens (or n-grams), min/max timestamps, per-field dictionaries — so brute force can skip most chunks without decompressing them. A needle query ("this trace ID, sometime last month") consults compact bloom filters and touches only the handful of chunks that might contain it; false positives cost a wasted scan, never a miss. ClickHouse's skip indexes and recent Loki bloom-filter tiers converge here, and columnar log storage (parse lines into typed columns at ingest) compounds the effect — filters on status=500 become column predicates, and compression triples because similar values cohabit. The write-time cost stays a small fraction of full inversion; the common queries stop paying full-scan price.

The shared chassis

Whichever indexing bet you make, the rest of the pipeline is fixed: agents ship with backpressure and disk buffers (never block the app; drop before the app hurts — observability must not take down the observed, third time this series has said it); a Kafka buffer absorbs the incident-correlated bursts (log volume triples exactly when everyone's querying); object storage is the truth tier with time-based retention doing the deleting (dropping a day = dropping objects — the immutability dividend); and the query tier is stateless scatter-gather with per-query resource budgets, because one |= "a" over 30 days is a self-inflicted DDoS without limits. Tail-based patterns from the tracing post apply to logs-as-events too: errors and slow-path logs can earn longer retention than the INFO firehose (tiered retention by level/sampling is the budget knob nobody uses enough).

Interview probe	Answer sketch
Which shape for a 200-eng company's app logs?	Label-indexed + bloom middle: workflows are "known service, known window, grep" — full inversion is paying for ad-hoc search nobody runs
Find one request ID across everything, fast?	That's what trace IDs + the tracing backend are for; in logs, bloom-filter tiers make it a seconds-scale scatter, and that's acceptable for a rare query
Structured vs unstructured?	Push structure to the edge (JSON logs, parsed at ingest to columns); grep is the fallback for the long tail, not the plan
Legal hold / audit logs?	Different system requirements wearing the same clothes: WORM object storage, verified integrity, retention locks — don't mix with debug logs

The transferable frame: write-time cost is paid always; read-time cost is paid rarely — so index exactly as much as your actual query distribution repays. Full inversion, label-only, and bloom-skips aren't competing philosophies; they're three points on one curve, and your query logs tell you where to sit.