World Cup Log Analytics Platform
A reproducible distributed system that routes traffic through one gateway, preserves request-level context across three services, and turns structured logs into deterministic analytics.
- 100,000
- validated requests
- 5
- MapReduce jobs
- 13
- verified artifacts
- 3
- FastAPI services
The problem was trust, not just throughput.
Problem
Requests crossed multiple services, but raw logs alone did not provide a reliable way to trace one request, reproduce a workload, or prove that the final aggregate matched the source data.
My individual role
I designed and implemented the gateway routing, three backend services, traffic generator, structured logging contract, Hadoop Streaming jobs, fixture tests, and final cross-checking workflow.
One observable path from request to result.
The public boundary stays at Nginx. Each request receives correlation context, flows to one of three private services, becomes structured JSONL, and is processed through five deterministic analytics jobs.
X-Request-ID, X-Client-Country, and X-Scenario travel with every request so gateway and service events can be reconciled later.
The difficult choices favored reproducibility.
Keep services private
Only Nginx is exposed. This makes the gateway the single place for routing and correlation context instead of letting clients bypass the observability boundary.
Make ties deterministic
Analytics use a single reducer where global ordering matters and lexical tie-breaking where values are equal, so repeated runs cannot silently choose different winners.
Guard destructive reruns
Pipeline scripts separate explicit cleanup from normal execution. Existing HDFS and output paths are checked before a rerun instead of being overwritten implicitly.
Verify more than job success
A fixture test simulates shuffle/sort against hand-calculated results, while final verification cross-checks the summary against intermediate CSV files.
The architecture and final verification in a real run.
A pipeline with evidence at every boundary.
- 100,000 requestsThe guarded final run generated and validated the required request volume.
- Five analytics jobsEach MapReduce stage has a defined input, output, and deterministic ordering rule.
- 13 materialized artifactsCSV and JSON deliverables are checked for presence, schema, and consistency.
- End-to-end reconciliationGateway counts, service counts, and final summaries are cross-checked rather than accepted independently.
Prometheus metrics and trace export would complement the durable JSONL audit trail with live operational visibility.