Smoke or Stack presents a critical choice for teams evaluating observability platforms. Understanding who dies in each scenario helps organizations balance reliability, cost, and insight when designing monitoring strategies.
Modern infrastructures demand clarity on failure modes, so teams can invest in the right telemetry approach. The following sections break down scenarios, tradeoffs, and practical guidance for selecting and operating these patterns.
| Pattern | Primary Failure Mode | Impact on Metrics | Typical Use Cases |
|---|---|---|---|
| Smoke | Exporter or agent crashes | Silent gaps until restart | Lightweight hosts, short-lived containers |
| Stack | Pipeline or collector overload | Backpressure, dropped batches, delays | High-cardinality environments, strict SLAs |
| Combined Defense | Agent + collector failures | Partial loss, buffered fallback | Regulated workloads, financial systems |
| Resilient Design | Resource exhaustion or network partition | Degraded fidelity, local retention | Multi-site, edge-to-cloud architectures |
Smoke Pattern Risks and Detection Gaps
The smoke pattern relies on lightweight agents that push metrics in near real time. When these processes die due to crashes, OOM kills, or configuration errors, observability stops without warning.
Because smoke agents often run with minimal health checks, silent gaps appear in dashboards and alerts. Teams may only discover the outage when incidents escalate or SLO breaches are detected downstream.
Root Causes in Smoke Deployments
Common triggers include unhandled exceptions in the agent code, file descriptor leaks, resource constraints on shared nodes, and abrupt container termination. Without persistent queues, buffered data is lost during these events.
Stack Pattern Failure Modes
The stack pattern centralizes processing before data reaches storage. Failures here usually involve backpressure, queue saturation, or pipeline bottlenecks rather than immediate silence.
Instead of dying completely, a stressed stack may drop low-priority metrics, delay critical signals, or throttle ingestion. This behavior introduces latency that can mask emerging incidents.
Bottlenecks and Degradation Paths
Memory pressure, thread contention, slow remote storage writes, and oversized label sets can choke a stack. Horizontal scaling helps, but without smart sampling and prioritization, important signals may be discarded first.
Reliability Tradeoffs by Design Choice
Architectural decisions directly influence who dies and how gracefully observability degrades. Buffering, replication, and batching change the risk profile compared to fire-and-forget shipping.
Evaluating tradeoffs between latency, durability, and overhead guides teams toward patterns that align with their tolerance for data loss and operational complexity.
Operational Safeguards and Controls
Robust deployments combine monitoring for the monitors with clear runbooks. Health dashboards, synthetic checks, and automated restarts reduce time-to-detection and time-to-repair.
Strategic use of local storage, retry loops, and rate limiting ensures that transient failures do not cascade into prolonged observability blackouts.
Recommendations for Choosing and Hardening Your Pattern
- Enable persistent local buffering on agents to survive short outages.
- Implement health dashboards and synthetic checks for telemetry pipelines.
- Use prioritized metrics and sampling to protect critical signals under load.
- Automate restarts and integrate with incident response for fast recovery.
- Test failure modes regularly with controlled chaos experiments.
FAQ
Reader questions
Which pattern loses data fastest when an agent crashes?
The smoke pattern loses data fastest because agents typically lack durable queues, so metrics in memory vanish on crash.
Does stacking always protect against data loss during backpressure?
No, a stack may drop low-value metrics or throttle pipelines under backpressure, causing selective data loss despite queuing.
Can small teams benefit from the stack pattern despite its complexity?
Yes, small teams can benefit by using managed pipelines that handle scaling and retention, reducing operational burden.
What signals should be monitored to detect smoke or stack failures early?
Monitor agent heartbeat metrics, queue depths, error rates, and end-to-end latency to spot incipient failures before data disappears.