System Design Drill: A High-Throughput Rule Engine

Design a service that evaluates events against a changing set of rules and returns or publishes decisions. This problem is useful because it forces trade-offs between throughput, rule freshness, latency, and correctness.

1. Clarify the contract#

Start by separating the control plane from the data plane.

• The control plane lets authorized users create, validate, version, approve, and roll back rules.
• The data plane evaluates a high-volume event stream against an active rule snapshot.

Ask for peak event rate, event size, P99 decision latency, rule count and complexity, acceptable rule-propagation delay, ordering needs, and the required behavior during partial failure.

2. Establish a simple flow#

For an asynchronous design:

Producers → durable event log → partitioned evaluators → decision topic → consumers
                                  ↑
                            versioned rule cache

Partition on a key that preserves the ordering the product needs, such as account or device. Evaluators consume a local, immutable rule snapshot so the hot path does not call the rule database per event.

For synchronous decisions, place a thin API layer in front of the same evaluator model and define timeout and degradation behavior explicitly.

3. Version rule changes#

The control plane validates a new rule set, stores it as a version, and publishes an activation event. Evaluators load and verify the snapshot before switching atomically. Each decision records the event ID and rule-set version.

That makes decisions explainable and allows safe rollback. It also avoids a half-updated fleet evaluating the same event differently without traceability.

4. Handle duplicates and replay#

Assume at-least-once delivery. If downstream actions must happen once, consumers need idempotency based on the decision ID. Retain the event and decision streams long enough to replay after bugs or rule corrections, with access controls appropriate to the data.

5. Make operations part of the answer#

Track input rate, consumer lag and age, evaluation latency, rule-load failures, decision distribution, and resource saturation. Add canary activation for risky rule changes and a kill switch that can disable a rule without rebuilding the service.

Finally, name the largest risks: hot partitions, expensive rules, state growth, poison events, and correlated failures during rule rollout. A good design interview answer shows how the system behaves when assumptions break.