Design a Real-Time Anomaly Detection Service

Asked by:Google

streamingreal_timeevent_drivendata_modeling

Problem Overview & Requirements

Design a stateful streaming service that learns or maintains per-stream baselines, evaluates numeric observations in near real time, and publishes reproducible anomaly decisions.

Key Pillars: High Availability, Low Latency, Scalability, Fault Tolerance.

Topics & Components: streaming, real_time, event_driven, data_modeling.

System Design Breakdown

1. Requirements & Scope

Functional & non-functional requirements, DAU/MAU estimates, read/write throughput.

2. High Level Architecture

API Gateway, Load Balancers, Application Microservices, Caching Layer & Message Queue.

3. Data Model & Storage

SQL vs NoSQL selection, schema design, sharding keys, indexing, and replication strategy.

4. Bottlenecks & Trade-offs

CAP theorem trade-offs, consistency models, rate limiting, and cache invalidation strategies.