Did the user succeed?

That question often gets neglected. Not because teams don’t care, but because most reliability management is organized around services, while the user experiences transactions.

Reliability Is Felt in Transactions, Not Services

Consider an end-user experience like placing an order in an application. From the user’s perspective, this is one thing they are trying to accomplish. They add an item, confirm, and expect the order to go through. For them, the outcome is binary: the order worked, or it didn’t.

From a system perspective, that same experience fans out across many services. Inventory and pricing services check availability, payment authorization happens synchronously, a machine-learning-based fraud detection system evaluates the order in real time, and downstream fulfillment, recommendation updates, and notification steps continue asynchronously.

Each service might look “fine” when viewed in isolation. Latency could be slightly elevated in one place; retries could be masking failures in another, and asynchronous steps could be lagging, without any single service showing a clear red flag.

This is not a theoretical edge case. It’s a natural consequence of composing many independently reliable components into an end-to-end experience.

Reliability is managed at the service level, but it is felt at the level of transactions and flows.

Why Transaction-Level Reliability Doesn’t Collapse Into One Simple Metric

You can try to model this by defining reliability objectives for individual transactions, and in some cases that works well. A critical HTTP path on a frontend service can and should have a clear objective.

But systems rarely stop there. Real products are full of flows that are asynchronous, multi-step, or partially decoupled. Password reset workflows, AI-assisted report generation, onboarding sequences, background processing pipelines; these don’t fit neatly into a single request-response boundary.

In these cases, the user-facing “success” signal isn’t always available at the moment a request returns. You often only know whether the user truly succeeded after downstream work completes and state converges—sometimes minutes later, sometimes after multiple systems reconcile.

This is also where many SLO programs end up skewing toward internal service-level contracts: useful for ownership and operations, but not always capturing whether the user intent completed successfully end-to-end.

So the challenge isn’t “measure reliability.” It’s: measure reliability at the level where success is actually determined.

Product Ownership Doesn’t Map Cleanly to Services

There is another dimension where this becomes visible: products and features are rarely owned by a single service. They span multiple microservices, owned by different teams, evolving at different speeds. Developers and product managers tend to think in terms of features and user journeys, not in terms of which service is emitting which metric.

In a DevOps model, that creates friction. The people responsible for improving a product’s reliability need to understand how changes affect end-to-end behavior. Service-level views are necessary for fixing issues, but they are not sufficient for reasoning how reliable a feature or product feels to users over time.

And this is where a lot of reliability programs stall: they’re excellent at service health and incident response, but weaker at answering:

• Which user-facing flows are degrading?
• Which products are accumulating reliability risk?
• What is reliability trending like from the user’s perspective?
• Where are we “green” locally but failing end-to-end?

Consider a subscription upgrade flow initiated through a support chatbot. A customer messages: “Upgrade me to Pro”. The bot confirms success immediately.

Behind the scenes, that single interaction fans out across a chain of services, agents and actions: billing is updated, an entitlements agent validates the new plan and pushes updated permissions, usage limits refresh, and internal automation agents orchestrate background jobs reconcile state across systems.

No single service is down. The agents are executing as designed. Everything looks acceptable in isolation.

But minutes or hours later, the customer hits a limit they shouldn’t have, or a feature remains locked. Support tickets spike. Revenue recognition is delayed. Trust erodes.

From a service view, everything looks acceptable. From a product view, the upgrade flow is unreliable.

Closing the Gap Requires Product- and Flow-Centric Views

Addressing that gap requires additional lenses. Reliability systems need ways to look at signals through transaction-centric and product-centric views. Views that reflect how users experience the system and how teams build products across services, not only how services behave in isolation.

Services remain the building blocks. Service-level discipline remains essential. But to understand what users are actually feeling, and how reliable a product truly is, reliability needs to be observable at the level where experience happens: across transactions, flows, and products composed from many services working together.

We’ve been working on addressing this gap in our own product. We are exploring how reliability systems can make transaction- and product-level views first-class, while strengthening the service foundations teams rely on today. We’ll share more about that soon.

In the meantime, we’re curious whether this resonates: Does this match what you’re seeing in your own systems, or do you think service level reliability already cover more of this than we’ve described?

Let us know! We love to hear and read what you think: reach out to us via email or discuss with us on LinkedIn!

That delay is the Page-to-Understanding Gap. It is the translation cost of turning a page into an explicit explanation of system behavior. This is why MTTR improvements plateau even in teams with mature alerting and observability. 

Engineers feel this cost most acutely during incidents today, but it is also what prevents alerts from being usable inputs to automated response workflows.

What comes next is closing that gap. That means grounding alerts directly in system behavior and causal context, so investigation starts with understanding instead of translation. 

The problem alerts create during investigation 

When an alert fires, the first requirement is not diagnosis. It is orientation. 

Engineers need to know what kind of behavior is happening, whether multiple alerts point to the same underlying issue, and where to focus first. Alerts do not answer those questions. They were never designed to. 

Most alerting systems encode thresholds, burn rates, or proxy signals that usually correlate with impact. They reflect past incidents and operational heuristics, not real-time system behavior. That is why the same alert can represent very different failure modes, and why many different alerts can describe the same underlying degradation. 

As a result, the first minutes of an incident are spent translating alert language into a mental model of the system. Engineers jump between dashboards, traces, and logs to figure out what the alert actually means right now. Slack fills with partial hypotheses. Context fragments before anyone agrees on what problem they are solving. 

This translation work is the gap. It is where time is lost and coordination breaks down. 

The missing bridge between alerts and understanding 

Alerts do carry some intent. They are not random noise. Each alert exists because someone believed a certain condition mattered. 

What has been missing is an explicit bridge between that alert intent and an explanation of system behavior that can drive consistent action. Without that bridge, alerts remain isolated signals. Engineers are left to do the mapping themselves under pressure. 

Closing the Page-to-Understanding Gap requires making that mapping explicit. An alert needs to land inside an explanation, not start a scavenger hunt for one. 

What changed in Causely 

Our recent release adds the missing bridge. 

Causely already ingests alerts from tools teams rely on, including Alertmanager, Prometheus or Mimir, incident.io, and Datadog. What’s new is that the relationship between those alerts and Causely’s symptom and causal model is now explicit and visible. 

Each ingested alert is mapped to the symptom it represents and shown directly in the context of the inferred root cause. 

Instead of treating alerts as separate problems, Causely treats them as evidence of system behavior. Multiple alerts that describe the same behavior collapse into a single symptom. Those symptoms are then connected through the causal model to the change, dependency, or resource actually driving the issue. 

Alerts are no longer just timestamps and labels. They become part of a coherent system story. 

You can also view alerts over time and see which symptoms and root causes they mapped to. This makes alert behavior visible and understandable, rather than noisy. 

How this changes the first minutes of an incident 

For the on-call engineer, the workflow changes immediately. 

When a page fires, the alert can be pasted directly into Ask Causely. Instead of starting with dashboards, the engineer sees which symptom the alert maps to and whether there is an active root cause. The first question shifts from what to check to whether action is required now. 

The same mapping is also available as structured context for automated workflows, so alerts no longer require manual interpretation to become actionable.

For teams dealing with alert noise, the change is cumulative. Over time, they can see which alerts consistently map to the same symptoms and causes. This builds trust that important behavior is covered and makes it easier to reason about which alerts are redundant versus genuinely distinct. 

For reliability leads, gaps become visible. Alerts that do not map cleanly to existing symptoms stand out. Those gaps are no longer discovered mid-incident. They become clear opportunities to extend coverage where it actually matters. 

From paging interface to investigation entry point 

Alerts still wake people up. That does not change. 

What changes is where investigation starts. Instead of beginning with translation and hypothesis building, teams start with an explanation that already accounts for how alerts relate to each other and to the system. 

Over time, this shifts behavior. Engineers stop debating which alert is primary. War rooms converge faster on a shared narrative. Investigation energy moves from interpretation to containment. More importantly, alerts become explicit, interpretable signals that can drive response consistently, whether the next step is taken by a human or a system.

Causely does not replace alerting. It turns alerts into inputs to understanding. 

Closing the gap 

Alerts are not the investigation. They never were. 

The cost teams pay today is not because they lack alerts or data, but because understanding arrives too late. By explicitly mapping alerts to symptoms and root causes, Causely closes the Page-to-Understanding Gap where most incidents stall, turning alerts from pages into grounded inputs for understanding and action.

Know what to chase when everything breaks. 

Instead of treating traces as a data stream we might analyze someday, we should be opinionated about what matters at the moment of decision. As we argued in The Signal in the Storm, raw telemetry only becomes useful when we extract meaningful patterns.

In this guide, you’ll build a repeatable workflow that turns OpenTelemetry database spans into span-derived metrics you can dashboard and alert on—so you can identify what’s slow, what matters most, and what just regressed.

We’ll make this concrete with slow SQL queries, serving two use cases:

• Optimization: Which queries yield the most value if made faster, weighted by traffic?
• Incident response: Which queries are behaving abnormally right now?

We’ll build a lab where your app emits OpenTelemetry traces, and we distill those into actionable metrics, starting with simple slow query detection, then adding traffic-weighted impact, and finally anomaly detection.

What Makes a Query Slow?

“Slow” isn’t a single problem. It’s a symptom with fundamentally different causes. A 50ms query might be fine for a reporting dashboard but catastrophic for checkout. As High Performance MySQL emphasizes, understanding why a query is slow determines how to fix it. Here are the most common problems that may cause slow queries:

Excessive Work

The database does more than necessary—typically full table scans due to missing or unusable indexes. Without an index on customer_id, a simple SELECT * FROM orders WHERE customer_id = $1 grows from 20ms at 10K rows to minutes at 10M rows. The query didn’t change; the data volume did. See Use The Index, Luke! for the fundamentals.

Aggregations and joins compound this. Even indexed queries can explode when the planner misjudges cardinality and chooses the wrong join strategy.

Resource Contention

Perfectly optimized queries can be slow when waiting for resources. Lock contention blocks queries until other transactions release rows. Connection pool exhaustion adds latency before the query even starts. A query spending 95% of its time waiting for locks won’t be fixed by query optimization—it needs transaction redesign.

Environmental Pressure

CPU saturation, I/O bottlenecks, and memory pressure can slow any query. The same SQL with the same plan performs completely differently under resource contention.

Plan Regressions

Performance degrades when execution plans change—even with identical queries and data. Parameter-sensitive plans optimize for one set of values but fail for others. Stale statistics after bulk loads cause the planner to choose terrible strategies. The PostgreSQL Performance Tips documentation covers how to catch these regressions.

Pathological Patterns

Some slowness doesn’t appear in slow query logs. The N+1 problem executes 100 fast queries (2ms each) sequentially, adding 200ms latency plus network overhead. No individual query is “slow,” but the pattern is catastrophic.

The classic workflow: DB-native tooling + manual triage

Databases ship with excellent diagnostic tools: slow query logs, query stores like PostgreSQL’s pg_stat_statements, and plan inspection with EXPLAIN. These tell you what’s expensive inside the database.

What they don’t provide is context. Which service triggered the slow query? Is it user-facing or background work? Does it correlate with the latency spike you’re investigating? You’re left with a list of slow queries and no signal about which ones matter most.

Typically, someone bridges this gap manually: a developer notices a slow endpoint, brings the query to a DBA, and they optimize it together. This works, but that manual linking is exactly what we can automate.

Bringing Context to Slow Queries

Database tools tell you what is slow, but not why it matters. When you find a slow query in your logs, you’re missing critical context: Which service triggered it? Is it user-facing or background work? Does it correlate with the latency spike you’re investigating?

Distributed traces provide this context. Each database span is embedded in a request context—it knows which service, endpoint, and user triggered it.

Instead of correlating database logs and traces after the fact, we analyze slow queries directly from traces with all the application context built in.

The Building Blocks

Now that we understand the philosophy and the value of context-rich traces, let’s look at the building blocks we’ll use to implement slow query analysis.

The Observability Stack

For our lab, we use the OpenTelemetry Collector paired with docker-otel-lgtm—a pre-packaged stack from Grafana that bundles Loki, Grafana, Tempo, and Mimir in a single container. This gives us a complete observability environment with minimal setup.

The Application

Our sample application is a simple Go-based “Album API” that serves music album data from PostgreSQL. It’s intentionally designed to produce the kind of intermittent slow queries that are common in production. They services use otelsql to instrument database calls, emitting spans with the stable OpenTelemetry database semantic conventions.

The Dashboards

We’ll build three dashboards, each adding a layer of insight:

1. A simple view of the queries by duration
2. Queries weighted by traffic to surface optimization opportunities
3. Anomaly detection to identify queries deviating from their normal behavior

Lab Setup

Let’s put the theory into practice. We’ll clone a sample application, start the observability stack, and explore three progressively more sophisticated approaches to slow query analysis. All you need is Docker installed.

Clone and Run

git clone https://github.com/causely-oss/slow-query-lab
cd slow-query-lab
docker-compose up -d

Once running, open Grafana at http://localhost:3001—that’s where we’ll explore our dashboards.

Queries by Duration

The first dashboard takes the most direct approach: query Tempo for database spans and aggregate them to find queries that take the longest time. This is what you’d naturally build when you first start exploring traces for slow query analysis.

What It Shows

The Slow SQL - By Duration dashboard queries traces directly using TraceQL:

{ span.db.system != "" } | select(span.db.query.text, span.db.statement)

This finds all spans with database attributes, then uses Grafana transformations to:

1. Group by root operation (API endpoint) and SQL statement
2. Aggregate duration into mean, max, and count
3. Sort by average duration (slowest first)

The result is a table showing your slowest queries, which endpoints triggered them, and how often they occur.

What’s Good About This

This approach gives you immediate visibility into queries with full application context:

• You can see exactly which SQL statements are taking the most time
• You know which API endpoints trigger them
• You have the count to understand frequency
• You can click through to individual traces for debugging

It’s a first improvement over raw database logs because you’re already seeing the application context that makes slow queries actionable.

The Limitation

Here’s the problem: sorting by average duration doesn’t tell you which queries matter most.

Consider two queries:

The complex report is “slower” by average duration, so it appears first. But the search query, despite being faster on average, runs 2,000 times more often. Its aggregate impact on your users is far greater.

This dashboard tells you what’s slow, but not what’s impactful. For that, we need to consider traffic volume.

Traffic-Weighted Impact Analysis

The second dashboard addresses this limitation by introducing an impact score: the product of average duration and call count.

What It Shows

The Slow SQL - Traffic Weighted dashboard uses the same TraceQL query but adds a calculated field:

Impact = Avg Duration × Count

This simple formula captures a key insight: a moderately slow query that runs thousands of times has more total impact than a very slow query that runs rarely. The dashboard sorts by impact score, surfacing the queries that matter most to your users.

The dashboard also adds:

• Service breakdown: See which service triggered each query
• Latency distribution: Visualize duration over time, not just averages
• Top queries by impact: A quick view of where to focus optimization efforts

What’s Good About This

Traffic-weighted impact gives you a much better prioritization signal for optimization work:

• High-volume, moderately-slow queries surface above rare-but-slow ones
• You can justify optimization work with concrete impact numbers
• The service and endpoint context helps you route issues to the right team

When someone asks “which slow queries should we optimize first?”, this dashboard gives you a defensible answer. It’s exactly what you need for planning performance improvements.

The Limitation

But this dashboard is for optimization, not incident response. Even with traffic-weighted impact, it can’t answer a critical question:

“What has changed?”

Suppose your search query has an impact score of 150,000. Is that normal? Is it higher than yesterday? Higher than last week? The dashboard shows you a snapshot of current state, but it has no concept of baseline.

This matters enormously during incidents. When latency spikes, you don’t just want to know “search queries are slow”—you want to know “search queries are slower than normal”. You need to distinguish between:

• A query that’s always been slow (known behavior, maybe acceptable)
• A query that just became slow (new problem, needs investigation)

Without a baseline, every slow query looks the same. You’re left manually comparing current values to your memory of what’s “normal,” or digging through historical data to establish context.

This is the gap that the third dashboard addresses.

Symptom Detection with Anomaly Baselines

Because of these limitations, the third dashboard changes our approach: instead of just querying traces, we distill metrics from spans and then apply anomaly detection to identify deviations from normal behavior.

The Setup

For this dashboard, we add the spanmetrics connector to the OpenTelemetry Collector. Here’s the relevant part of the collector configuration:

connectors:
  spanmetrics:
    dimensions:
      - name: db.system
        default: "unknown"
      - name: db.query.text
      - name: db.statement
      - name: db.name
        default: "unknown"
    exemplars:
      enabled: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [transform, batch]
      exporters: [spanmetrics, otlphttp/lgtm]
    
    metrics:
      receivers: [spanmetrics]
      processors: [batch]
      exporters: [otlphttp/lgtm]

The spanmetrics connector examines every database span and generates histogram metrics for query latency, labeled by:

• service_name: Which service made the query
• db_system: Database type (postgresql)
• db_query_text or db_statement: The SQL query
• db_name: Database name

These metrics are stored in Mimir (the Prometheus-compatible backend in docker-otel-lgtm), where we can apply PromQL-based anomaly detection.

Anomaly Detection with Adaptive Baselines

The sample app includes Prometheus recording rules from Grafana’s PromQL Anomaly Detection framework. These rules calculate:

• Baseline: A smoothed average of historical values (what’s “normal”)
• Upper band: Baseline + N standard deviations (upper threshold)
• Lower band: Baseline - N standard deviations (lower threshold)

When current values exceed the bands, we have an anomaly—a clear signal that something has changed.

What It Shows

The Slow SQL - Anomaly Detection dashboard displays:

1. Current latency plotted against the adaptive baseline bands
2. Anomaly indicators when latency exceeds normal bounds
3. Per-query breakdown so you can see which specific queries are anomalous

The key insight is the visual comparison: instead of just showing “p95 latency is 450ms”, it shows “p95 latency is 450ms, which is above the expected range of 200-350ms.”

Why This Is Better

This dashboard answers the question the previous one couldn’t: “What has changed?”

• A query that’s always slow (450ms baseline) won’t trigger anomalies when it runs at 450ms
• A query that’s normally fast (50ms baseline) will trigger anomalies if it suddenly runs at 200ms
• You get automatic context for what’s “normal” without maintaining manual thresholds

The anomaly detection acts as a symptom detector. It tells you: “This query is behaving differently than it usually does.” That’s a high-signal insight you can act on immediately.

From Metrics to Symptoms

Notice what we’ve achieved with this architecture:

1. Raw telemetry (traces) flows from the application
2. Distillation (spanmetrics connector) extracts metrics from those traces
3. Anomaly detection (Prometheus rules) identifies deviations from baseline
4. Symptoms (anomalous queries) surface for investigation

We went from thousands of trace spans to a handful of anomaly signals that tell you exactly where to look.

Taking This to Production

Metric Cardinality

Raw SQL in metric labels will explode your metrics backend—SELECT * FROM orders WHERE customer_id = 12345 becomes a separate series per customer. Use prepared statements (so instrumentation captures templates, not literals), normalize query text, or use aggregation_cardinality_limit in the spanmetrics connector.

Privacy

SQL may contain sensitive data. The Collector is the ideal place to redact: drop or transform sensitive attributes before shipping downstream. This aligns with distillation: sanitize at the edge, not centrally.

Anomaly Detection Baseline

Adaptive rules need 24-48 hours of data to establish baselines. Start with wider bands and tighten as confidence grows.

The Remaining Gap: From Symptoms to Root Causes

Even with anomaly detection, you’re still looking at symptoms. In real-world incident scenarios, especially in large environments, slow queries are just one of many symptoms that pop up at once. You’re not only trying to understand the cause of this one; you’re triaging a flood of alerts and correlating many symptoms to find the real root cause.

When the dashboard shows “search query latency spiked,” you know something changed. But you don’t know why it changed. The root cause might be:

• A missing index after a schema migration
• Query plan regression due to stale statistics
• Lock contention from a concurrent batch job
• Resource pressure from a noisy neighbor on the database host
• Upstream service degradation causing retry storms

Connecting the symptom (“search query is slow”) to the root cause (“index was dropped during last night’s migration”) requires causal reasoning—understanding the relationships between system components and tracing the chain of causation from effect back to cause.

You can absolutely do this reasoning yourself. Look at deployment timestamps, check for schema changes, investigate resource metrics, correlate with other symptoms. Good engineers do this every day.

But it’s manual, time-consuming, and doesn’t scale.

Going Beyond Symptoms with Causely

This is where Causely comes in: Causely extracts slow queries (and other symptoms) as distilled insights out of the box—the same pattern we implemented manually. But it goes further:

• Causal model: Slow queries are connected into a model of your system’s dependencies. You can see what they impact (which endpoints, which users) and what causes them (resource constraints, upstream failures, configuration changes).
• Root cause identification: Instead of showing you a list of symptoms to investigate, Causely traces causation chains to identify the underlying root cause. “Search queries are slow because the index was dropped.”
• Actionable recommendations: AskCausely helps you get to “what should we change?”—whether that’s adding an index, reverting a deployment, or addressing the upstream pressure that made the query slow in the first place.

The pattern we built in this post—distill, detect anomalies, surface symptoms—is the foundation. Causely is the natural next step: turning symptoms into root causes at scale.

Want to see how Causely connects your slow queries to their root causes? Try it yourself.

A queue starts to grow slowly. Nothing looks obviously broken at first. Publish calls are succeeding and consumers are still running, just not quite keeping up. Over time, messages begin to age out, and dead-letter queues start accumulating entries. Downstream services that depend on those messages begin to behave unpredictably. There are partial data, delayed processing and subtle customer-facing issues that are hard to tie back to a single event. By the time the impact is visible in latency or error rates elsewhere in the system, the original cause is buried several layers upstream and hours in the past. 

Teams do not miss these failures because they lack data. They miss them because the signals do not point clearly to the cause. 

Over the past several weeks, we’ve expanded Causely’s asynchronous and messaging queue capabilities to make these failures explicit, explainable, and actionable. This includes:  

• Expanding Causal Model for Amazon SNS and SQS and RabbitMQ 
• A new Producer Publish Rate Spike root cause   
• And adding Queue Size Growth and Dead-letter Queue as first class symptoms to our model  

Reliability Blind Spot in Messaging-Driven Architectures 

Asynchronous communication is foundational to how modern systems scale. The same advantages systems like Kafka and RabbitMQ provide, decoupling services and absorbing traffic spikes, also introduce new reliability challenges. 

The core issue is not that these systems fail quietly, but that cause and effect are separated. A producer can overload the system without returning errors. A broker can continue accepting traffic while consumers fall behind. By the time downstream symptoms appear, the triggering behavior has often already passed. 

For engineering managers and or those on the frontline of the on-call slack channel, this creates a familiar and frustrating dynamic. Reliability degrades without a clear trigger. Incident response turns into a debate about whether the producer or consumer is responsible. Teams chase anomalies across dashboards while backlogs continue to grow. By the time a decisive action is taken, the customer impact is already real. 

Why Traditional Observability Falls Short 

Metrics, logs, and traces are excellent at answering local questions. They tell you what a service is doing, how long an operation took, or how many messages are currently sitting in a queue. 

What they do not provide is causal understanding across asynchronous boundaries. 

In messaging-driven systems, cause and effect are separated in time and space. A spike in publish rate from one service may not create visible impact until hours later, in a different service, owned by a different team. A slow consumer may be the result of downstream backpressure rather than a defect in the consumer itself. Dead-letter queues tell you that messages failed, but not why the system reached that state. 

Without a causal model of how producers, exchanges, queues, and consumers interact, teams are forced to infer failures indirectly. That inference is slow, fragile, and heavily dependent on tribal knowledge. Under pressure, it leads to overcorrection, unnecessary rollbacks, and missed root causes. 

Expanding the Causal Model for Messaging Systems 

To close this gap, we have significantly expanded Causely’s causal model for asynchronous messaging systems. 

Rather than treating queues as opaque buffers, Causely now models messaging infrastructure the way it actually operates in production. Producers, exchanges, queues, and consumers are represented as distinct entities with explicit relationships and data flows. This applies across common technologies, including Amazon SQS, Amazon SNS, and RabbitMQ, whether used in simple queue mode or exchange-based pub/sub patterns. 

By modeling the topology directly, Causely can reason about how work enters the system, how it is routed, where it accumulates, and how pressure propagates across services. This makes it possible to explain failures that previously required intuition and guesswork. 

Making Queue Growth and Dead-Letter Failures First-Class Signals 

We have also expanded the causal model to treat queue size growth and dead-letter queue activity as first-class symptoms, not secondary indicators. 

This changes how asynchronous failures are diagnosed. Instead of surfacing queue metrics as passive signals, Causely reasons about them causally, linking backlog growth and dead-letter events directly to the producers, consumers, and operations involved. 

As a result, queue-related failures are no longer inferred indirectly from downstream latency or error spikes. The failure mode is explicit, explainable, and traceable to the point where intervention is most effective.  

A New Root Cause: Producer Publish Rate Spike 

One of the most common and least understood asynchronous failure modes is a sudden change in publish behavior. Causely now includes a dedicated root cause for this pattern: Producer Publish Rate Spike. 

This occurs when a service, HTTP path, or RPC method begins publishing messages at a significantly higher rate than normal. The increase may be triggered by a code change, a configuration update, or an unexpected shift in traffic patterns. Downstream queues absorb the initial surge, but consumers cannot keep up indefinitely. Queue depth grows, message age increases, and backpressure begins to affect the rest of the system. 

What makes this failure particularly dangerous is that the producer often looks healthy. Publish requests succeed, error rates remain low, and nothing appears obviously wrong at the source. Without causal reasoning, teams frequently blame consumers or infrastructure capacity, missing the true trigger entirely. 

Causely now detects this condition explicitly. It ties unexpected increases in publish rate to queue growth, consumer pressure, and downstream service degradation, making the failure both visible and explainable. 

What This Changes for Reliability Teams 

For teams responsible for revenue-critical services, these capabilities change how asynchronous failures are handled in practice. 

Instead of reacting after queues are saturated and customers are impacted, teams can see which producer initiated the failure, how pressure propagated through the messaging system, and where intervention will have the greatest effect. Slow consumers, misconfigured routing, and unexpected publish spikes are distinguished clearly rather than conflated into a single “queue issue.” 

This shortens incident response, reduces unnecessary mitigation, and eliminates the finger-pointing that often arises when failures span multiple teams. More importantly, it enables a proactive reliability posture in systems that are constantly changing. 

Asynchronous Reliability Without Guesswork 

Asynchronous architectures are essential for scale, but they demand a different approach to reliability than synchronous request paths. 

With its expanded messaging and asynchronous causal model, Causely provides deterministic, explainable reasoning over how data flows through your system. Teams do not need to stitch together dashboards to reconstruct timelines after the fact. They do not need to trust black-box AI summaries that cannot explain their conclusions. They no longer have to exhaustively eliminate possibilities to arrive at a root cause. 

Instead, they get clear answers to the questions that matter most: what is breaking, why it is breaking, and where to act first to protect reliability and revenue. 

If you’ve been on call, you’ve seen this movie.

This is also why plenty of “AI-powered observability” rollouts still don’t change the lived experience of on-call. Leadership expects response times to improve. On-call gets richer dashboards, smarter summaries, and more plausible explanations. To be fair, those do help with faster lookup and briefing in the room, but the social reality stays the same: alerts get silenced in PagerDuty, rules get tagged as “flappy,” and old pages keep firing long after anyone can justify what they were meant to protect. The problem isn’t effort. It’s that the page still doesn’t reliably collapse into shared understanding. Call it the Page-to-Understanding Gap: the time and coordination cost of turning a threshold into a system story.

Alerts are supposed to start an investigation. Too often, they start translation: what is the system doing right now? That translation slows containment, splinters context, and stretches customer impact. 

That decoding work is what incident response depends on, yet it’s rarely made explicit. It’s why MTTR gains often plateau even after teams invest heavily in monitoring and dashboards. 

Alerts are a paging interface, not a language of explanation 

Alerting is optimized for one job: interrupt a human at the right moment. 

So alerts are built from what’s easiest to express at scale: thresholds, proxy signals, and rules that “usually work.” They encode operational history, not system truth. 

That’s not a failure of alerting. It’s what alerting is for, but it also makes alerts a shaky foundation for understanding. They’re a simplified label over messy reality. When alerts aren’t grounded in clear definitions of “healthy” behavior, the signal loses meaning and teams stop trusting it. 

One alert name can mean multiple different realities 

Take a familiar page: “latency high,” or the modern equivalent: an SLO burn rate page. 

Burn rate fires on checkout. You assume checkout is slow, start in the service dashboard, see p99 up, then notice retries doubled. Meanwhile, Slack is already split: “DB” vs. “checkout.” Fifteen minutes later you realize a downstream dependency is brownouting and checkout is just drowning in retries. The giveaway is usually the shape: one hop shows rising timeouts and retry storms while upstream looks "healthy" until it saturates. The alert didn’t lie—it just didn’t tell you what you needed first. 

The page looks identical. The mechanism isn’t. 

So teams build muscle memory around “what usually causes this,” and it works—until the system changes just enough that it stops working. Scale and change are exactly what modern organizations optimize for, so the failure mode is guaranteed. When that happens, the alert doesn’t just wake you up. It points you in the wrong direction. 

Many alerts can describe the same underlying behavior 

The reverse problem happens just as often. A single degradation creates a cascade of pages across services: latency, errors, saturation, queueing, burn rate alarms. Each is technically “true,” but treating them as separate problems creates thrash. 

People split into parallel investigations, duplicate context gathering, and argue about which page is “the real one.” Context fragments across Slack threads and war rooms, ownership ping‑pongs, and escalations get noisy. By the time you agree which page is “primary,” you’ve already created a coordination incident. The outcome isn’t just wasted engineer time—it’s that nobody has one shared narrative everyone can repeat while impact is unfolding. The incident becomes less about understanding and more about sorting competing signals. This is how alert fatigue turns into incident fatigue. 

The silent gap: important behavior you don’t alert on 

The costly ones are the slow degradations that ship impact before they page. 

A dependency gets a little slower. Retries creep up. One critical route starts timing out for a slice of customers. Averages look fine. Thresholds don’t trip. You don’t notice until the page fires, or until customers do. 

It’s not that teams don’t care. It’s that these behaviors don’t fit neatly into alert rules: risk builds up, dependencies decay, partial impact hides in aggregates, and propagation only makes sense once you’ve traced it end to end. 

Most alerts only get defined after you’ve understood the behavior in the middle of an incident. The post-mortem produces a new rule and a brief feeling of closure. Then traffic shifts, dependencies evolve, and the next incident arrives with a different shape. You’re never done. 

So teams find these late, after impact is already underway, when time is most expensive. 

Why teams don’t switch investigation entry points 

When teams adopt a new investigation system, they often ask a simple question: “Does it match our alerts?” 

What they’re really asking is: “Can I trust this in the first 90 seconds?” Because in the first minute, the primary goal isn’t elegance, it’s not making it worse.

A system that generates more hypotheses doesn’t help if it can’t connect the page to what the system is doing in a way the on-call trusts. 

If the system describes an incident in a different language than the alert model engineers rely on, mismatches get interpreted as duplication, contradiction, or risk. The result is predictable: people consult it late, after they’ve already committed to a direction. 

Even correct insights arrive too late to change behavior. 

Why this problem is getting worse 

Systems are becoming more dynamic: more dependencies, faster deploys, and more integration points. Deploy frequency keeps climbing—and AI-assisted coding is only accelerating it—so the number of failure paths keeps growing. Meanwhile, alert fatigue is already high, and teams are hesitant to change workflows mid-incident. 

Better tooling can speed up lookup and correlation, but it can’t compensate for an alerting model that no longer maps cleanly to real system behavior. 

So the interpretation workload keeps rising. Every page demands more interpretation, more cross-checking, more manual stitching of symptoms into a coherent story. 

What’s actually broken 

Most organizations are operating with two different languages: the language of paging and the language of understanding. The persistent MTTR plateau is the Page-to-Understanding Gap between them.

Incidents start with the first, but the work happens in the second. 

A better way to think about alerts 

Alerts are not the investigation. They’re a notification that something is going sideways. 

The goal is not to tune thresholds until the noise feels tolerable. It’s to shorten the time from page to shared understanding: what behavior is emerging, what changed, what’s being impacted—and whether it matters to the business. 

Treating that translation as unwritten know‑how is not a workflow quirk. It’s a structural weakness. If your incident response starts with decoding alerts, you’re spending your best engineers on interpretation instead of containment. 

When these systems fail, the failure rarely shows up where it starts.

Teams often notice stale data, degraded behavior, or latency spikes elsewhere in the system. By the time those symptoms appear, the underlying problem has usually been present for some time.

In many real-world failures, two signals appear earlier: queue growth and dead-letter queues. They are widely monitored, but they are still widely misunderstood.

The common misunderstanding 

Queues are often treated as infrastructure components rather than behavioral signals. When a queue grows, it is attributed to load. When messages land in a DLQ, it is treated as a retry policy doing its job. Investigation tends to focus downstream, where symptoms are visible.

This framing obscures where asynchronous systems actually break. In many failures, message brokers continue to accept traffic normally. Producers succeed. Nothing looks obviously down. The problem is that consumers are no longer able to keep up reliably or consistently. That distinction matters.

Queue growth is not just volume 

Queue growth occurs when messages arrive faster than they can be processed successfully over time. This does not require a traffic spike. It can result from: 

• Consumers slowing due to code changes or resource pressure 
• Dependencies becoming latent or unreliable 
• Retry rates increasing 
• Backpressure failing to engage 
• Partition skew concentrating work unevenly 

In these cases, the broker behaves correctly. Messages are accepted. The queue grows quietly. What is accumulating is lag. A sustained backlog means work is no longer flowing through the system at the intended rate, even if no explicit failures are visible yet.  

Why this matters before anything looks broken 

Asynchronous systems are designed to absorb instability. Queues buffer mismatches. Retries smooth over failures. Backlogs delay visible impact. This is useful, but it also postpones feedback. As queues grow: 

• Processing time increases 
• Derived state falls behind 
• Downstream services operate on increasingly stale or incomplete data 

The transition from “degraded” to “broken” often appears sudden because the system has been accumulating lag for some time before any external threshold is crossed. 

Dead-letter queues signal a different failure 

Dead-letter queues exist to capture messages that cannot be processed successfully. Messages land there after repeated failures, timeouts, or deterministic errors. DLQs prevent infinite retries and protect the main pipeline. What they represent is not transient instability, but persistent processing failure under current system behavior. 

A non-empty DLQ means some class of messages cannot be handled as the system is currently operating. That incompatibility can come from: 

• Broken contracts between producers and consumers 
• Partial or skewed deployments 
• Schema drift 
• Unhandled edge cases 
• Dependencies that fail consistently rather than intermittently 

DLQs often grow alongside backlogs, but they can also appear independently. 

Why these problems are so common 

In real systems, producers and consumers evolve independently. Load shifts. Dependencies degrade. Retry behavior changes system dynamics in non-obvious ways. It is common for: 

• Brokers to continue accepting traffic 
• Queues to grow steadily 
• Consumers to fail intermittently or slow down 
• Processing failures to accumulate quietly 

Operationally, queues and DLQs sit between services. They rarely have clear ownership. They are easy to monitor superficially and hard to reason about in context. As a result, many teams only notice these issues once downstream behavior degrades.  

Queue growth and DLQs are related, but distinct 

Queue growth and DLQs are often discussed together, but they answer different questions. 

Queue growth asks: "Are messages flowing through the system fast enough?"

DLQs ask:  "Are some messages failing to be processed at all?"

In many incidents, sustained queue growth precedes DLQs. Consumers slow down, retries increase, and retry limits are eventually exceeded. In others, DLQs appear immediately due to deterministic processing failures, even while queue depth looks healthy. Treating one as a proxy for the other creates blind spots. 

The deeper diagnostic challenge 

Most teams diagnose asynchronous failures indirectly. They look at:

• Latency spikes 
• Error rates 
• Timeouts 
• User-visible symptoms 

Those signals matter, but they are downstream effects. Earlier and more precise signals exist inside the message pipeline itself: where messages are accepted, where they slow down, and where they fail to be processed reliably. When those signals are ignored or misinterpreted, teams spend time chasing symptoms rather than isolating where the workflow is actually breaking.  

A better way to think about queues 

Queues are not just buffers. Queue growth is not harmless backlog. Dead-letter queues are not operational exhaust. They are indicators of whether asynchronous workflows are functioning as intended or quietly degrading under real conditions. 

The goal is not to watch queue depth, instead, it’s to continuously understand flow: where work accumulates, why it accumulates, and which downstream interactions it impacts. 

Understanding them is not an optimization. It is foundational to operating reliable, event-driven systems. 

This limitation becomes especially visible in modern, hybrid architectures. Services span Kubernetes clusters, standalone EC2 instances, ECS tasks, and legacy infrastructure, all connected through real production traffic. Datadog can surface signals across these environments, but understanding how failures propagate across those boundaries remains a manual, error-prone exercise. The result is slower recovery, repeated incidents, and reduced confidence in change. 

With this expanded Datadog integration, Causely gives teams a unified, causal model of their entire application across Kubernetes and non-Kubernetes environments. This model that explains why services are impacted, not just where symptoms appear. 

From Observability Signals to System Understanding 

Many teams already rely on Datadog APM, infrastructure metrics, and monitors as the backbone of their observability stack. With Causely’s expanded support, those same Datadog signals can now be used to build a complete and accurate causal model of the system without changing existing instrumentation. 

Causely supports Datadog APM dual shipping, which allows trace data to be sent directly from the Datadog collector into Causely’s mediator. Teams continue using Datadog exactly as they do today, while Causely consumes the same traces for causal reasoning. This approach avoids additional agents, avoids data duplication, and does not introduce new egress costs. 

Just as importantly, Causely now supports services running outside Kubernetes and keeps causality intact across the hybrid boundary. By tagging Datadog APM traces with host identity metadata, Causely can stitch together services running on EC2 with those running inside Kubernetes clusters. What previously broke at environment boundaries becomes a single, end-to-end behavioral model of how the application actually runs in production. 

Datadog monitors can also be ingested directly into Causely and treated as symptoms rather than conclusions. Instead of reacting to alerts in isolation, Causely uses them as signals that inform its understanding of what is happening in the system and why. That’s how you get faster convergence, fewer false leads, and higher confidence in the fix. 

A Real-World Hybrid Application Scenario 

Consider a typical production application. Customer-facing APIs and frontend services run in a Kubernetes cluster. Background workers, billing services, or legacy processing jobs run on standalone EC2 instances. The application depends on shared infrastructure such as Postgres, Redis, and external APIs. Datadog is already deployed across all of it. 

Under normal conditions, everything appears healthy. Then, during a traffic spike, latency starts creeping up in one of the Kubernetes services. Shortly after, Datadog monitors begin firing for elevated error rates in downstream components. Engineers open dashboards, inspect traces, and try to correlate timelines across environments. The symptoms are visible, but the cause is not obvious. 

This is where Causely changes the workflow. 

Automatically Pinpointing the True Cause 

Using Datadog traces, infrastructure metadata, and monitor events, Causely continuously reconstructs end-to-end request paths and dependencies across Kubernetes and EC2. That continuity holds across environment boundaries, so you don’t have to manually stitch together “what talks to what” in the middle of an incident.  Instead of reacting to individual alerts, Causely continuously builds and maintains a behavioral model of the entire system. This model captures how services, infrastructure, and data flows interact, and how specific failure modes produce observable symptoms. 

Datadog APM traces provide the raw evidence of system behavior, including service interactions, request paths, and downstream dependencies. Datadog monitors are ingested and mapped as symptoms within Causely’s knowledge base. Together, these signals allow Causely to maintain an up-to-date causal model that explicitly links observed symptoms to the conditions and changes that produced them. 

Because this causal model is updated continuously, Causely can explain not just what is failing, but what changed first, how the impact propagated, and why specific services or endpoints are affected. The result is a precise, system-level explanation of performance degradation that teams can act on immediately. 

From Incident Response to Reliability Assurance 

This expanded Datadog integration is not just about faster root cause analysis during incidents. By continuously modeling system behavior, Causely enables teams to validate reliability before changes reach production, monitor how reliability evolves over time, and detect drift caused by infrastructure or configuration changes. 

Modern systems are hybrid by default, and reliability problems do not respect environment boundaries. To operate confidently at scale, teams need more than visibility. They need to understand how their systems behave and why failures occur. 

With expanded Datadog support across Kubernetes and EC2, Causely helps teams move from alert-driven firefighting to causal reliability engineering. The result is fewer war rooms, faster resolution, and the confidence to ship changes without fear. 

To learn more about using Causely with Datadog, explore the integration guide or reach out to see a unified service graph in action. 

In the first one, we said thanks to Grafana for donating Beyla and making it easier for teams to get to usable telemetry quickly. This time we want to zoom out to something that quietly runs under the hood at Causely every day: GitOps with FluxCD.

Causely is a member of the Cloud Native Computing Foundation (CNCF). That’s not just a logo on the website for us: our entire product and our own operations lean heavily on CNCF projects. We build on OpenTelemetry, we run on Kubernetes, and we dogfood our own reliability engine against that stack.

Another key piece in that puzzle is FluxCD. It is what takes “the desired state in git” and makes it true in our clusters, repeatedly. It’s the heartbeat behind our weekly releases.

Why Flux Helps

If you’re operating modern Kubernetes environments, you’ve probably felt this tension. On one hand, you want velocity: teams push changes constantly, services multiply, and configurations evolve all the time. On the other hand, every manual kubectl apply is a potential one-off change that no one can fully reconstruct later. Over time, clusters drift away from whatever was last written down as “how things should be,” and you are left relying on muscle memory and shell history.

Flux solves exactly that problem by turning git into the control surface and reacting to changes within seconds of a merge.

For us, that has very practical consequences.

• Releases are commits, not hand-crafted ceremonies.
• A new Causely version is rolled out by changing a tag or a value in git; Flux notices the change within seconds, reconciles the cluster, and either converges to the new state or loudly tells us why it couldn’t.
• Environments stay in sync because the same manifests back our test clusters, staging, and production. The differences between them are intentional and visible in overlays, not hidden in one-off fixes on a live production cluster.
• And drift becomes a signal, not a mystery: when the cluster does not match git, Flux shows it, which turns the familiar “what changed?” question into a quick investigation rather than a full-blown incident archaeology.

The net effect is that we can move quickly without giving up control. Our reliability engine depends on a stable substrate; Flux helps us keep it that way.

Best Practices We’ve Learned Using Flux

We didn’t get there on day one. It took a set of habits to turn FluxCD from a cool project into a core platform primitive.

1. Treat manifests like product code
2. Keep Kustomize overlays boring
3. Watch Flux like any other production controller
4. Make promotion a path, not an event

Treat manifests like product code

All of our Kubernetes manifests live in git. Not most of them, not “the important ones” – all of them. That sounds obvious, but it changes behavior in subtle ways.

• Reviews happen before things break, because a change to a HelmRelease or a Kustomize overlay goes through the same review process as a feature change.
• The commit history becomes an operational log: when we see a strange spike in errors, we can line it up against recent git changes, including configuration tweaks that would otherwise live only in someone’s bash history.
• Our issue tracker also stays connected to reality, because we reference issue numbers in commit messages, so the question “why did we change this setting?” always has a direct link back to the discussion that justified it.

In the early days, we still made the occasional manual change in production, usually in the name of speed. Those changes always came back to haunt us as confusing states that no one could fully explain. Once we committed to git as the only source of truth and forced ourselves to route every change through it, the platform became much more predictable.

Keep Kustomize overlays boring

We use Kustomize to manage environment-specific differences across test clusters, staging, production, and chaos environments. The rule we eventually settled on is simple: overlays describe differences, not alternative universes.

In practice, that means we maintain a clean base with shared resources such as namespaces, common HelmReleases, and shared configuration. On top of that base, we keep the environment overlays as thin as possible. They patch what truly needs to change, such as cluster names, resource limits, or a particular feature flag, rather than redefining whole stacks.

Whenever we tried to be clever with external references or overlays that diverged heavily from one another, troubleshooting became harder. Keeping overlays compact and predictable means we can scan a diff and understand at a glance what will change in a given cluster. Before committing, we render Kustomize configs locally as a quick sanity check that catches typos and misaligned paths before Flux has to complain about them.

Watch Flux like any other production controller

GitOps is not “set and forget.” Flux is a control loop running in production and, when it is unhappy, your platform will slowly drift.

We treat Flux like a critical controller. We watch reconciliation health and consider a stuck HelmRelease or Kustomization as important as any failing deployment. When Flux cannot talk to git, or when an apply keeps failing, that is something we alert on rather than something we notice days later in a dashboard. And when “nothing seems to be changing” in a cluster despite recent commits, Flux logs are one of the first places we look.

This mindset becomes even more important when GitOps extends beyond just core applications and starts to manage your observability stack, gateways, and even Causely itself.

Make promotion a path, not an event

Flux really shines when you treat deployments as a series of git-based promotions instead of isolated production pushes. A typical Causely release starts with a change landing in a test environment: we use clusters like test1 and test2 for this. We verify that the change behaves as expected there, including how it interacts with telemetry and Causely’s own reasoning about incidents. Once we are happy, we promote the same change to staging by updating the relevant overlay or values. Only after staging behaves as expected do we roll the change into production.

Alongside this path, we maintain dedicated chaos clusters, chaos1 and chaos2, where we deliberately break things to see how the system responds. Because everything flows through git, we can rehearse failure modes without fear of leaving behind strange manual fixes that only exist on one cluster. Keeping cluster-specific configuration isolated and well documented is what allows us to run realistic experiments in those chaos clusters without letting that complexity bleed into production.

Try Flux On Your Own

To really understand Flux, it helps to feel git driving your cluster. The smallest useful experiment is a git repository, a local kubernetes cluster, and Flux bootstrapped from that repository. The nice part is that flux bootstrap already does most of the heavy lifting: it creates the repository, installs the controllers, and wires everything together for you.

You can run the following guide on your laptop with kind.

Start by installing the Flux CLI. The easiest way is via the official install script; if you prefer Homebrew, apt, or other package managers, the Flux documentation lists those options as well in the Flux installation guide.

curl -s https://fluxcd.io/install.sh | sudo bash

Next, export your GitHub credentials so Flux can authenticate and create the repository for you. If you are logged in with the GitHub CLI (gh auth login), you can derive both the user name and token directly from it.

export GITHUB_USER="$(gh api user --jq '.login')"
export GITHUB_TOKEN="$(gh auth token)"

Now, create a local Kubernetes cluster and verify that Flux can run there.

kind create cluster --name flux-playground

When your cluster is ready, bootstrap Flux into it. This command will create a repository called flux-playground-gitops under your GitHub account, install Flux into the flux-system namespace, and configure it to track the ./clusters/flux-playground path in that repo.

flux bootstrap github \
  --owner=$GITHUB_USER \
  --repository=flux-playground-gitops \
  --branch=main \
  --path=./clusters/flux-playground \
  --personal

Clone the newly created repository to your machine and change into it.

gh repo clone flux-playground-gitops
cd flux-playground-gitops

You are now ready to define the OpenTelemetry demo as a git-managed workload by adding a manifest for the demo under clusters/flux-playground.

cat > clusters/flux-playground/oteldemo.yaml <<'EOF'
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: open-telemetry
  namespace: flux-system
spec:
  interval: 1m
  url: https://open-telemetry.github.io/opentelemetry-helm-charts---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: otel-demo
  namespace: flux-system
spec:
  interval: 1m
  chart:
    spec:
      chart: opentelemetry-demo
      sourceRef:
        kind: HelmRepository
        name: open-telemetry
        namespace: flux-system
  targetNamespace: otel-demo
  install:
    createNamespace: true
EOF

At this point your git repository fully describes both Flux itself and the OpenTelemetry demo that Flux will deploy.

Commit and push these changes so Flux can reconcile the new state.

git add .
git commit -m "Add OpenTelemetry demo via Flux"
git push origin main

Within seconds of the push, Flux will see the new revision, apply the changes, and start rolling out the OpenTelemetry demo. You can watch the pods come up.

watch kubectl get pods -n otel-demo

After a few minutes you will see the OpenTelemetry demo microservices starting in the otel-demo namespace.

You now have a real application being managed by Flux from git: the desired state lives in a repository, Flux reconciles it into the cluster within seconds of your merge, and you never had to use kubectl apply for the actual deployment.

Flux + Causely: GitOps All the Way Down 

If you followed the small lab above, you already have a local cluster, Flux installed, and the OpenTelemetry demo running under GitOps control. From there, adding Causely is just one more git-driven change. 

Conceptually, the flow is simple. You obtain a Causely access token, store it as a Kubernetes Secret, and add the Causely FluxCD manifests to the same git repository that Flux already manages. Git drives the rollout, Flux reconciles it into the cluster, the OpenTelemetry demo generates realistic behavior, and Causely explains what is going on. 

Our documentation has a full FluxCD installation guide with more background and variations. The example below is meant to be something you can copy and adapt directly from your existing flux-playground-gitops setup. 

First, retrieve your Causely access token from Causely and keep it handy. Then, create a namespace for Causely and a kubernetes secret with your token. The Causely FluxCD manifests expect a secret named causely-secrets and use Flux's native post-build substitution to inject CAUSELY_TOKEN into the HelmRelease, so the token never has to be committed to git: 

kubectl create namespace causely 
kubectl create secret generic causely-secrets \ 
  --from-literal=CAUSELY_TOKEN=your-actual-gateway-token-here \ 
  -n causely 

Next, from inside your GitOps repository, clone the public causely-deploy repository and copy the FluxCD manifests into your own cluster configuration. 

cd flux-playground-gitops 
git clone https://github.com/causely-oss/causely-deploy.git  
mkdir -p clusters/flux-playground/causely 
cp causely-deploy/kubernetes/fluxcd/causely/*.yaml clusters/flux-playground/causely/

With this setup, your git repository now contains everything Flux needs to deploy Causely, and you can commit and push these changes so Flux can reconcile the new state.

git add clusters/flux-playground 
git commit -m "Add Causely via Flux" 
git push origin main 

After the push, Flux notices the change, applies the new manifests, and starts deploying Causely into your cluster. You can watch it the same way you watched the OpenTelemetry demo roll out. 

flux get kustomizations -A 
kubectl get pods -n causely

 Once the Causely pods are healthy, you can return to the Causely portal, where the cluster you just configured will appear. Over time, topology fills in and, as issues arise, you will see root cause views associated with the services in your demo. 

At that point, you have a complete loop on a single laptop: Git drives change, Flux applies it, the OpenTelemetry demo generates behavior, and Causely explains what happens when things go wrong. If you want more variations, production-grade knobs, or to run this across multiple clusters, the FluxCD installation guide in our docs walks through additional options in detail. 

Closing: Thank You, Flux 

FluxCD is a great example of the kind of infrastructure we love in the CNCF ecosystem. It nudges teams toward good habits, turns the question “how did this get here?” into something with a clear, auditable answer, and helps keep complex Kubernetes estates boring and predictable.

As a CNCF member building on OpenTelemetry, Kubernetes, and the wider cloud-native stack, we are genuinely grateful for projects like Flux that quietly raise the floor for everyone.

So: thank you to the Flux maintainers and community for building and maintaining an engine that lets us practice what we preach about control, desired state, and autonomous reliability. If you are running kubernetes and still relying on manual deploys or one-off scripts, Flux is worth a serious look.  

And if you want to see what happens when you combine GitOps with causal reasoning, we are always happy to show you how Causely fits into that picture.

Book a demo today -> 

At the pace modern cloud-native systems move, reacting after symptoms appear is not enough. Teams need a reliability operating system that works continuously alongside their applications, maintains an up-to-date understanding of how everything fits together, and provides the context required for safe automation and proactive reliability. 

Why this recognition matters 

Modern cloud-native applications evolve constantly. Every deploy, configuration change, traffic surge, and infrastructure adjustment can reshape system behavior. The pace is so fast that even the best teams struggle to reason about what is happening and why. 

Traditional observability dashboards can show what happened after symptoms appear, and AI copilots can help speed up triage, but reacting after the fact isn’t good enough for business-critical applications. 

Teams need something that runs continuously alongside their systems, understands how everything fits together, and helps them see how changes will affect performance before they land in production. That’s the foundation for proactive reliability, not just faster incident response. 

What Gartner recognized about Causely 

Gartner recognized Causely for taking a fundamentally different approach to reliability in modern systems. Instead of reacting to symptoms after they spread, Causely maintains a live causality graph that reflects how services, dependencies, and performance constraints relate to one another as the environment evolves. By continuously analyzing telemetry, Causely identifies the underlying driver behind emerging changes in golden signals, even when failures cascade across multiple services, including the code change, configuration update, or operational event that first introduced risk. 

This continuous causal inference is what enables proactive reliability. Causely provides clear direction on where to focus and what action is most likely to reduce performance risk, long before issues escalate. The same causal model supports both pre-production and production, helping teams understand how behavior will shift during testing, rollout, and real-world load. 

Causely runs locally as a lightweight, containerized system and processes telemetry without exporting raw data. This eliminates the need for central pipelines, avoids sampling or data volume constraints, and gives teams high-fidelity insight that integrates directly into their engineering workflows through APIs, webhooks, and an MCP server. This structured context supports both human decisions and safe automation. 

For us, this recognition validates the direction we have been building toward. The future of reliability is proactive, predictive, and grounded in causal understanding. 

Who should care? 

Causely is designed for teams responsible for building and operating modern distributed systems. It gives engineering and operations organizations a clearer understanding of how system behavior changes over time and how those changes affect reliability. Whether preparing a release, managing growth, or diagnosing unexpected behavior, teams need deeper clarity to make confident decisions. Continuous causal inference provides that clarity. 

Looking ahead 

We’re grateful to Gartner for this recognition and excited about what it represents. Reliability is entering a new chapter. Systems are more dynamic, AI workloads are growing, and teams need deeper clarity to keep everything running smoothly. 

Causely’s mission is to provide that reliability. Continuous causal inference helps teams prevent issues before they escalate, support high-velocity engineering without sacrificing reliability, and give both humans and automation the context they need to act safely. 

We’re excited for what’s ahead and proud to help shape the future of reliable, intelligent, and resilient systems. 

Want to learn what this could look like for your organization? Get started with Causely today.  

Gartner subscribers can view the full report for more information. 

You take a breath. Maybe even consider sleeping. 

Then Slack lights up: 

“Did it work?” — CTO 

You stare at dashboards. Nothing’s red. But you still don’t actually know: 

• Did reliability improve, or quietly regress? 
• Did the change shift bottlenecks or introduce new stress points? 
• Are you now closer to the edge under peak load? 

In a 50 to 100+ microservice environment with dense service-to-service dependencies, even small regressions can cascade silently. And slowing down isn’t an option. Leadership needs faster delivery and fewer incidents. 

This is exactly why we built Reliability Delta. 

Reliability Delta: A Deterministic Answer to “Did This Change Make Things Better or Worse?”

Reliability Delta turns subjective guesswork (like manual diffing of dashboards, correlation hunts, “nobody is complaining” anecdotes) into clear, evidence-based reliability signals. 

It’s powered by Causely’s continuously updated understanding of your environment: 

Causality Mapping 

Causely builds a Bayesian network that models how issues propagate across services, enabling true cause-and-effect visibility. 

Attribute Dependency Graph 

A DAG of functional dependencies generated from live topology and Causely’s attribute models, highlighting how attributes influence one another. 

These models allow Causely to compare two snapshots—two releases, two load tests, or two moments in time—and determine whether system behavior: 

• Improved 
• Regressed 
• Or shifted in ways you need to investigate 

The result: deterministic signals engineers can trust.  

Use Cases for Reliability Delta 

1. Validate Every Release Instantly 

Know immediately whether your change introduced risk. 

Feature flags and canaries help, but they don’t guarantee safety. What matters is whether the system is behaving normally. 

Reliability Delta automatically surfaces: 

• Behavior changes isolated to a specific flag, tenant, or traffic segment 
• Downstream effects in pipelines, async jobs, and data flows 
• Subtle regressions that don’t trip alerts but violate known patterns of normal 

This isn’t “no alerts fired = good.” 

This is evidence-based release confidence.  

2. Understand Load Test Results Beyond Pass/Fail 

“With Causely’s Reliability Delta, we can quantify how each release behaves under identical load. It surfaces changes in bottlenecks, stress patterns, and causal relationships that traditional load tests miss. At our scale, having that level of confidence before shipping is critical.” 

Cade Moore, Performance Engineering Lead at Hard Rock Digital
 

Did this release push you closer to the breaking point? 

A load test passing doesn’t mean you’re safe. 

Reliability Delta shows: 

• How bottlenecks shifted compared to last time 
• Whether the same load now produces more stress 
• Early signs of fragility or shrinking performance margins 

It answers the question load tests never answer: 

“Are we drifting toward failure or away from it?” 

3. Detect Reliability Drift Over Time 

Systems naturally drift through config changes, dependency updates, scaling events, and organic load shifts. 

By capturing snapshots periodically, you can: 

• Spot slow-building risk 
• Track reliability trends 
• Validate that ongoing changes are improving SLO posture, not eroding it 

This moves teams from reactive firefighting to proactive reliability assurance. 

4. Validate Experiments with Confidence 

Know immediately whether your experiment improved or degraded system behavior. 

Teams frequently adjust timeouts, concurrency, sampling, queue behavior, or other system parameters, but these changes rarely trigger alerts, and standard dashboards make it hard to see their true impact. 

Reliability Delta lets you validate experiments with clear before-and-after evidence by automatically highlighting: 

• Shifts in bottlenecks or stress patterns across services 

• Degradations hidden behind “passing” performance metrics 

• Unexpected side effects in downstream dependencies 

• Whether the experiment made the system more resilient or more fragile 

This isn’t trial-and-error tuning. It is evidence-based experiment validation. 

Why Reliability Delta Matters for Modern Engineering Teams 

If you’re accountable for revenue-critical systems—measured by 99.9%+ SLOs, delivery pace, and incident reduction—you need more than observability dashboards. You need a deterministic framework for evaluating how change affects system behavior.  

Reliability Delta gives you: 

• Objective, repeatable comparisons between versions 
• Root-cause-aware analysis using causal models 
• Clear guardrails leadership can trust 
• Confidence to ship fast without risking SLOs or customer experience 

It transforms subjective judgment into trusted, actionable reliability signals—so every release, load test, and system change is safer, faster, and more predictable.  

Ship Faster with Confidence  

Reliability isn’t something you can eyeball anymore. With Reliability Delta, engineering leaders get the missing layer between observability and automation: clear causal evidence of how changes affect system behavior. It ensures your team can move fast, protect SLOs, and deliver with the confidence that every release is safer than the last. 

To learn more, see our docs: https://docs.causely.ai/in-action/reliability-delta/  
 

An Intellyx Brain Candy Brief

Causely monitors a real-time Bayesian network of semantically abstracted runtime operational telemetry data in order to observe and alert engineers to highly probable causes of issues and failure conditions, so they can ideally be resolved before they can emerge as customer-facing incidents.

Their “Causal Inference System” is a directed acyclic graph populated with tons of possible failure mode indicators. You can think of these indicators like micro-CVEs for observability, so that the system can know what to look for as it passively observes payloads within OTel logs and traces alongside golden signals such as latency or errors. It’s not another AI SRE, as the inferences it surfaces are deterministically based on live indicators that are semantically enriched with context.

When causes are observed, they can be captured as feedback so DevOps teams can flow through changes in the next CI/CD cycle, or reported into the enterprise’s incident management, ITSM and observability tools of choice, with direct links to contextual insights.

Sure, you could still do root cause hunting with any major observability platform worth its salt. However, to do that for a large distributed enterprise system, you would need to define thousands of policies that collect and tag telemetry data, and set up triggers and automation for each, such that the cognitive load and cloud data costs to keep it current might be prohibitively high. There’s always another way to do things!

Organizations collect far more telemetry than they use. The result? Exponential costs, PII risks, and still no root cause when incidents happen.

In this technical session, Yuri Oliveira (OllyGarden) and Severin Neumann (Causely) demonstrate:

• How to identify instrumentation problems at the source
• Why traces and semantic conventions enable purposeful telemetry
• How quality telemetry enables causal reasoning at scale

SRE teams are hitting a breaking point as Kubernetes environments scale faster than traditional workflows can keep up. Alerts pile up, incidents drag on, and teams lose hours in reactive firefighting instead of building reliability into their systems. At KubeCon + CloudNativeCon in Atlanta, Severin Neumann, Head of Community at Causely, offered a clear perspective on how AI can finally shift SRE practice from reactive to proactive. But this requires more than feeding logs into an LLM — it requires causal reasoning.

For years, the industry has talked about proactive SRE, but the reality rarely matches the aspiration. Neumann has spent more than a decade in observability and monitoring, and in his experience, teams always circle back to the same pattern: there’s an alert, teams react, things stabilize, and then the cycle starts again. The promise of AI offered a potential breakthrough, but early attempts fell into a familiar trap.

When companies say “AI SRE,” many immediately think of a large language model interpreting alerts and suggesting fixes. Neumann stressed that this approach often works only at the surface level. LLMs are powerful in pattern matching and summarization, but they don’t understand systems the way SREs need them to. They can connect correlations, but they can’t determine causation. And in high-pressure incident situations, an AI hallucination is more than an annoyance — it can waste hours and cost companies real money.

Causely takes a different approach. Their platform is built on a causal model that understands the structural relationships inside Kubernetes environments. Instead of giving an LLM raw observability data and hoping it finds meaning, Causely creates a deterministic model of how services interact and what components influence one another. The model knows which symptoms map to which possible root causes and can identify the single cause that explains them all.

This is a critical shift. Instead of chasing noisy correlations, SRE teams can rely on a reasoning engine that explains exactly why a problem is happening. LLMs do come into the picture, but only after the causal model has already identified the issue. At that point, a language model can help generate explanations, provide remediation guidance, or suggest kubectl or Helm commands to fix the problem. The heavy lifting — the understanding — remains deterministic.

Neumann also explained why this approach stands apart in the industry. Many teams tried the LLM-first path and burned their fingers. Some built internal solutions, others adopted competitors, but most eventually discovered the limitations. LLMs often fail when precision matters most, and in distributed systems, precision is everything. One misleading answer can waste hours in the middle of a firefight.

Causely’s causal reasoning avoids that by grounding all decisions in a model built around real system behavior. This also sets the stage for a more ambitious goal: shifting reliability left. Instead of waiting for symptoms, the model can analyze normal system behavior and identify bottlenecks or weak points before they cause downtime. This turns SRE work from reaction into prevention.

The conversation also explored the role of MCP (Model Context Protocol). Causely launched their MCP Server at KubeCon, enabling developers to pull causal insights directly into tools they already use. Through MCP, the remediation guidance — including detailed commands — can appear directly in an IDE or command line. The SRE no longer has to dig through logs or dashboards to figure out what’s going wrong. The causal model does that work and surfaces the fix.

Neumann outlined a vision where, over time, teams can set boundaries that allow AI to autonomously remediate certain classes of issues. If a service needs to scale up or memory needs to be increased, those actions could eventually be automated within limits defined by humans. This is where AI becomes a genuine partner — one that teams can trust to handle repetitive corrective tasks while they focus on system design, SLOs, and reliability architecture.

Trust is a key theme. Introducing AI into SRE workflows doesn’t eliminate human responsibility; it shifts it. Humans now become orchestrators who decide when to delegate and when to intervene. As Neumann put it, the more trust teams build in the system, the more they can remove themselves from situations where they’re no longer needed. This opens space for deeper reliability engineering, building guardrails, and designing better services.

A real-world example makes the value of causal reasoning clear. Imagine a financial services company running hundreds of microservices. Suddenly, everything turns red and user transactions fail. Traditional debugging would mean looking across alerts, logs, and traces to guess where the bottleneck is. Causely can cut straight through the noise, identify the one overloaded service, and surface the exact command needed to scale it up or adjust its memory. The time shaved off matters — for reliability, user experience, and cost.

Throughout the conversation, it became evident that AI isn’t replacing SREs. It’s allowing them to finally escape the cycle of constant firefighting. With causal reasoning, proactive reliability, and tools like MCP, SREs can shift to roles that emphasize guidance, architecture, and strategic improvements rather than crisis management.

Yotam Yemini, CEO of Causely, explains how the company integrates reliability engineering into product development, tackling challenges in cloud-native applications. The evolving role of Site Reliability Engineers (SREs) is discussed, highlighting the need to understand cause and effect in system performance. Causely provides tools for developers to enhance performance and reliability, promoting collaboration among engineering teams within a vibrant open source and cloud technology community.

Watch the interview here.

For a long time, observability has meant buying into a full stack you can’t really change: proprietary agents to collect the data, a proprietary protocol to move it, and a proprietary backend to look at it. Your telemetry lived inside a walled garden.

OpenTelemetry (OTel) is breaking that pattern. With community-standard instrumentation and the OTel Collector acting as a translation and routing engine, your metrics, logs, and traces are no longer trapped in that garden.

Observability isn’t a monolith 

There’s nothing inherently wrong with proprietary software; plenty of great systems are closed source. The problem is when your data becomes proprietary within these systems.

When collection, transport, and storage are tightly coupled to one vendor, your options shrink. Want support for a less common programming language? You might wait quarters for an agent. Want to change vendors? Prepare for weeks of reinstrumentation. Even simple ideas, like experimenting with a second backend in parallel, can become “projects.”

OTel changes the economics. Today, you can instrument almost everything consistently, and, yes, it has never been easier to swap your observability platform without touching application code. But, it’s not just about reducing vendor lock-in; when you own how your data moves, you can send it anywhere.

I promised not to rant about this again, but I need to come back to the “observability” debate once again: the way “observability” is used as a marketing term makes it seem like collecting, processing, and storing telemetry is one big monolith. It isn’t. Your pipeline is inherently composable, and the most leverage shows up at the tail: the “backend.” Treat that tail as a junction, not a culdesac.

What does SODA mean? 

That’s the idea behind SODA: Send Observability Data Anywhere.

SODA is simple on the surface: send a sensible combination or subset of your signals to the best tool for the job at hand. Under the hood, it means being deliberate about what you send and where you send it, keeping a durable copy you control, and refusing to recreate a new walled garden in the name of convenience. The OTel Collector makes this practical: you can enrich events, redact sensitive fields, apply sampling and routing policies, and fan out to multiple consumers, without touching application code.

We should understand that observability data is data, and often has value outside of just observability.

What does SODA look like in practice? 

By owning the transport of your data, your architecture becomes pluggable, and can adapt to how your needs change over time.

In the immediate-term, it means you can look at the most pressing concern most teams face: cost. You may not be able to move observability vendors overnight, but by instrumenting your app with OTel, you solve one of the biggest hurdles to getting started: testing a new observability stack is one configuration change away, and you can run multiple tools in parallel for comparison or during a migration.

For long-term retention, OTel makes it easy to store your complete historical data as a durable copy in inexpensive object storage in open formats. Doing so frees you from keeping all of your history in an expensive vendor bucket, and gives you replay: you can hydrate any alternative tool later without asking teams to reinstrument.

From there, you can send slices of telemetry to systems that act. Use signals inside the cluster to make decisions, like autoscaling with KEDA or the Horizontal Pod Autoscaler (HPA), gating progressive delivery so a canary only promotes when golden metrics and dependency health agree, or flipping feature flags when error budgets trend in the wrong direction. Security teams can mirror selected logs and traces to their SIEM, while the raw truth stays immutable in cold storage.

As AI enters the observability space, you are free to adopt new platforms that innovate in specific areas that matter to you, without being restricted to the capabilities of your core observability platform. You can use Causely to apply model-driven causal inference over your topology and recent changes to pinpoint likely causes for reliability issues and recommend or auto-apply safe actions. Or OllyGarden to help improve the quality of your OTel implementation.

Your telemetry is also product and business data in disguise, and can enable customer-facing teams to see product usage, user journeys, or customer impact as a result of an incident. Observability data is often silo’d and unavailable to internal analytics, but OTel allows you to bridge that gap whether you have separated data platforms, or a unified data stack. Platforms like ClickStack build on-top of flexible, open datastores like ClickHouse, and demonstrate how observability and business data can be co-located and correlated within one platform.

When the pipeline is open, you can experiment with, and adopt, new tools without rearchitecting your stack.

Composed, not siloed 

OpenTelemetry already breaks silos at the source by giving us a shared schema and transport for metrics, logs, and traces. SODA applies the same principle at the tail. Keep signals together as shared context on a durable stream and let specialized tools subscribe to that context, hand off seamlessly, and act without duplicating or fragmenting the truth. In a composed flow, a root cause identified in one place becomes the pivot to quantify impact and drive recovery in another—without losing the thread.

With freedom comes responsibility 

I’d like to stress an important note of caution: “send anywhere” is not the same as “send everything everywhere.”

Splitting your metrics, logs, and traces across disjointed backends that cannot be correlated is a fast path to longer MTTR and fingerpointing. If you want a thoughtful breakdown of why unified access matters for investigations, this article is a good primer.

The SODA posture helps to keep a coherent, durable source of truth under your control, then route purposeful subsets to the systems that extract additional value.

Drink some SODA and let us know what you think 

If you’ve read this far, you probably already have a mental list of places you wish your telemetry could go but currently doesn’t. That list is your SODA plan.

Maybe you start by dual-writing to object storage and turning on replay. Maybe you add a lightweight autoscaling signal for a spiky service. Maybe you route instrumentation health to a specialized tool while keeping unified access for investigations. The point isn’t to chase a shiny mesh of destinations; it’s to get more leverage from the data you already collect: safely, cheaply, and under your control.

We’d love to hear how you’re doing SODA today. Where are you sending telemetry beyond the one vendor you pay for? Which use cases are you covering: cost reduction, faster incident response, safer rollouts, richer product insights? Which ones do you want to see covered outside the “standard observability pipeline”?

It's back! This week is KubeCon, North America—one of the most beloved technology conferences in the world, centered around one of the most beloved enterprise technologies: Kubernetes. Born out of Google, Kubernetes has changed the very infrastructure on top of which modern applications are built. More than half of all enterprises have adopted Kubernetes, with millions of developers deploying their applications on top of Kubernetes every day.

But like any other technology, it comes with its trade-offs...mainly with the complexity of managing performance and debugging issues, which is why hundreds, if not thousands, of software vendors are touting their Kubernetes solutions this week. To help you separate the noise from the value, we want to highlight [] companies that truly help deliver value to enterprises using Kubernetes.

Gremlin

We're very excited to see this company ramp back up their public persona. Gremlin was founded by ex-Netflix and Amazon engineers in 2017 and made a lot of noise around the cutting-edge discipline of Chaos Engineering. For those of you who are unfamiliar, Gremlin has built a platform that empowers companies to run experiments on their systems in order to proactively identify where the weaknesses are.

Like getting people to the gym, it isn't always easy to convince people to change their habits—but the enterprises that build Chaos Engineering into their routines benefit from much healthier, reliable systems. At this KubeCon, Gremlin is announcing a strategic partnership with Dynatrace—a leader in observability and application performance monitoring—to help Kubernetes users keep their applications in a desired state.

Kubernetes services are automatically discovered within Gremlin, powered by Dynatrace's AI-driven observability and topology mapping. Health checks are then applied to Kubernetes' objects, allowing organizations to efficiently implement standardized reliability testing and gain deeper insights into their environments.

Mezmo

Mezmo is pioneering the concept of active telemetry in order to provide AI agents with better data and context. The company recently launched an RCA agent (root cause analysis) aimed at Kubernetes users that automatically identifies and fixes common issues such as deployment failures, resource issues, configuration errors, application-level failures, and more.

The proof is in the pudding: their AI agent consistently resolves issues in complex cloud environments faster and more accurately than other AI agents and models. According to the company's blog, "we're entering an era where incidents resolve themselves before engineers even know they exist." By leveraging agentic AI workflows, Mezmo rapidly analyzes telemetry data to pinpoint root causes, eliminate noise, and recommend actionable remediation steps.

Causely

Causely is pioneering the category of AI SRE—a category that will undoubtedly explode in popularity in 2026. Now more than ever, engineering teams are drowning in too much data and too many alerts. Having a system like Causely on the market—founded by a veteran of the industry with two prior startups in IT Operations—is a natural and critical response to the rise of AI code-generation tools that are shipping code faster than humans can reasonably understand it or manage it.

At this year's KubeCon, Causely is announcing an MCP Server that seamlessly integrates into any MCP-compatible IDE and enables developers to automatically diagnose, understand, and remediate complex issues within Kubernetes and application code using natural language prompts.

It works by analyzing the real-time state of the system, identifying whether the cause of an issue is in the infrastructure or application layer, recommending the precise code changes, configuration changes, or helm chart updates, and presenting these suggestions inline within the developer's IDE for review, refinement, or approval.

Komodor

It's hard to talk about Kubernetes troubleshooting and not mention Komodor. Founded by an ex-Google engineer, Komodor has been dedicated to the Kubernetes ecosystem since it launched out of stealth in 2021. Their management platform simplifies operations, provides automated troubleshooting, and helps teams manage complex environments. It tracks changes, analyzes their impact, and provides actionable context for issues, which reduces troubleshooting time and improves delivery velocity. Key features include automated drift detection, root cause analysis, and monitors for cluster health and resource optimization.

If you'll be in Atlanta, Georgia, this week for KubeCon 2025—stop by the booth of each of these companies. They are adding tremendous value to enterprises looking to maximize the performance and benefits of using Kubernetes.

Causely today unveiled a Model Context Protocol (MCP) server that enables developers to automatically diagnose, understand, and remediate complex issues within Kubernetes and application code using natural language prompts from within their integrated developer environment (IDE).

Severin Neumann, head of community for Causely, said the Causely MCP Server will make it easier for application developers to troubleshoot Kubernetes issues using a platform that already uses causal artificial intelligence (AI) models that were created to augment site reliability engineers (SREs).

The overall goal is to provide developers and SREs with insights needed to reduce downtime by analyzing the state of the system in real time to identify whether the cause of an issue is in the infrastructure or application layer. The Causely platform will then recommend precise code, configuration or Helm chart changes for developers to review, refine, or approve within their IDE.

Additionally, the Causely platform will also generate patches for Terraform, Helm, or application code to prevent issues from recurring.

It’s not clear at this point just how many software engineering teams are relying on AI to automate the management of Kubernetes workflows, but given the complexity of these environments, there is a significant opportunity to reduce stress and toil. The simple truth is that many organizations are limiting the number of Kubernetes clusters they might deploy simply because there isn’t enough SRE expertise available to manage them.

Many developers, meanwhile, have historically been intimidated by the complexity of Kubernetes clusters. The MCP server developed by Causely makes it simpler for developers to resolve issues on their own, both before and after cloud-native applications are deployed.

It’s not likely AI platforms such as Causely will replace the need for SREs any time soon, but they do present an opportunity for fewer SREs to successfully manage a larger number of Kubernetes clusters at scale at a time when the number of cloud-native applications being deployed continues to steadily increase. The issue, as always, is reducing mean time to remediation whenever all but inevitable incidents occur.

In fact, AI advances should enable SREs to spend more time on strategic issues such as ensuring availability versus constantly performing tactical incident management tasks, said Neumann.

Ultimately, multiple AI models that are being invoked by AI agents will soon be pervasively embedded across every DevOps workflow. The next major challenge will be finding a way to orchestrate the management of AI agents that will be accessing causal, predictive and generative AI models to optimize IT environments. Hopefully, those advances will significantly reduce the level of burnout that many software engineering teams experience as they are overwhelmed by repetitive manual tasks, especially when managing complex Kubernetes clusters that can be easily misconfigured.

In fact, it’s even conceivable that as AI starts to eliminate many of the bottlenecks that exist in software engineering workflows, much of the joy that attracted SREs and application developers to IT in the first place might soon be rediscovered.

If you haven't heard the term "AI SRE," you will soon. Following the boom of AI-generated code helping developers significantly increase their output, it was only a matter of time before more of the operations-related tasks around debugging, root cause analysis, and incident resolution were also AI-assisted.

One of the startups pioneering the AI SRE category is Causely—founded by IT Ops veteran Shmuel Kliger, who successfully sold two of his prior companies. Causely is unique in its ability to leverage causal inference to arrive at the root cause of issues accurately and more quickly than other solutions on the market. They are also shifting reliability left and identifying concerns before the code even gets shipped to production.

Today, the company announced the release of the new Causely MCP Server, a powerful tool designed to streamline and automate troubleshooting within Kubernetes. The company is making the announcement ahead of the popular conference KubeCon next week. This innovative solution integrates seamlessly with any MCP-compatible IDE, empowering developers to diagnose, understand, and resolve complex system issues using simple natural language prompts.

As Kubernetes' scalability and flexibility grow, so does the complexity of managing these systems. Issues like resource conflicts, unexpected pod evictions, and DNS delays often lead engineers to patch symptoms without uncovering root causes. Traditional monitoring tools offer valuable data but can make troubleshooting a manual, time-consuming process.

Causely's MCP Server aims to change that by embedding advanced causal reasoning directly into the developer workflow. Once integrated into popular IDEs such as Cursor or Claude, the system allows engineers to describe problems or desired outcomes conversationally, removing the need for manual searches or scripting.

Key features of the Causely MCP Server include:

• IDE-Centric Integration: Easy installation into MCP-compatible IDEs without significant infrastructure changes.
• Natural Language Prompts: Developers communicate issues and fixes naturally, streamlining problem reporting and resolution.
• Context-Aware Recommendations: The system uses real-time data and causal models to suggest effective fixes at the runtime, configuration, or code level.
• Upstream Fixes: Generates patches for Terraform, Helm, or application code to prevent similar issues in future deployments.
• Immediate Review & Refinement: Recommendations appear inline for iterative review before applying changes.

The MCP Server analyzes real-time system states to pinpoint whether an issue stems from infrastructure or application layers, then recommends precise modifications—be it code, configuration, or Helm chart adjustments. This approach simplifies maintaining systems in their desired state.

Karthik Ramakrishan, VP of Artificial General Intelligence at Amazon, praised the innovation: 

"Language models are powerful, but they require structured causal context to make the right decisions. Causely fills this gap, enabling real-time automation and reliable microservice operations."

By embedding intelligent, causal remediation directly into developers' workflows, Causely makes maintaining Kubernetes applications more straightforward and efficient than ever before. Certainly a company worth checking out if you'll be in Atlanta for KubeCon...or even if you won't be there, for that matter.

Kubernetes scalability and flexibility come with increased complexity. Services conflict for resources, pods are evicted unexpectedly, DNS queries lag, etc. When outages occur, engineers are often left patching symptoms without understanding the cause of these observed problems. Traditional monitoring and observability tools provide useful data, but troubleshooting remains a manual and time-consuming process.

“Causely’s MCP Server accelerates incident response by placing sophisticated causal reasoning directly in the hands of developers,” said Ben Yemini, Head of Product at Causely. “Once integrated into IDEs such as Cursor or Claude, the MCP Server allows engineers to describe problems or desired outcomes using simple natural language commands.”

Key Features of the Causely MCP Server include:

• IDE-Centric Integration: Installs seamlessly into any MCP-compatible IDE, requiring no significant infrastructure overhaul.
• Natural Language Prompts: Developers communicate problems or fixes conversationally, without needing to write scripts or manually search dashboards.
• Context-Aware Recommendations: The system uses real-time system data and causal models to propose specific, effective fixes at runtime, configuration, or code level.
• Upstream Fixes: Generates patches for Terraform, Helm, or application code to prevent issues from recurring in future deployments.
• Immediate Review & Refinement: Developers see recommendations inline, allowing iterative improvements before applying changes.

Causely’s new MCP server works by analyzing the real-time state of the system; identifying whether the cause of an issue is in the infrastructure or application layer; recommending the precise code changes, configuration changes, or helm chart updates; and presenting these suggestions inline within the developer’s IDE for review, refinement, or approval.

"If you’re serious about automating reliability in microservices you need what Causely is doing,” said Karthik Ramakrishan, VP of Artificial General Intelligence at Amazon. “Language models are powerful, but they can’t make the right calls without structured causal context. That’s the gap Causely fills, and it’s what makes real-time automation possible."

By embedding intelligent, causal remediation into the developer workflow, Causely makes it simpler than ever to maintain Kubernetes applications in their desired state. To learn more, read the announcement blog post or stop by their booth at KubeCon, Atlanta November 11-13.

About Causely

Causely is an AI startup dedicated to transforming Site Reliability Engineering through innovative automation, causal reasoning, and developer-centric tools. Their solutions help organizations manage complex distributed systems more efficiently and reliably.

Media Contact

Adam LaGreca

Founder, 10K Media

adam@10kmedia.co

The Problem: Complexity Hides the Real Issue 

Kubernetes gives teams the power to scale fast, but it also introduces new layers of complexity. Services contend for memory. Pods get evicted. DNS queries slow to a crawl. When something breaks, symptoms often show up far away from the real cause. 

A latency spike might surface at an API gateway, but the actual issue could be a congested message queue. Pod evictions might trace back to a misconfigured limit in a different service. Engineers end up chasing alerts, patching downstream effects, and firefighting without ever fully closing the loop. 

In systems like this, even well-intentioned remediation often lands in the wrong place. The action isn’t wrong; it’s just applied to the symptoms, not the cause.  

How Causely Approaches This 

Causely was built for distributed systems where problems propagate in non-obvious ways. It is designed to understand cause and effect across services and layers of the stack. 

At the core is a Causal Reasoning Engine (CRE) that applies domain-specific causal models to real-time telemetry. It maps causes to the symptoms they cause with high precision. It understands how services interact, how constraints emerge, and how changes ripple through the environment. This allows it to pinpoint the cause of service degradations, even in noisy environments where symptoms may be missing or spurious.  

Once the cause is pinpointed, Causely drives resolution at the appropriate layer, whether that’s a runtime adjustment, a configuration change, or a code fix. 

Closing the Loop with the MCP Server  

The new MCP Server connects this reasoning engine directly to the developer workflow. Through integrations with MCP-compatible editors like Cursor and Claude, Causely now: 

• Generates upstream patches to Terraform, Helm charts, or application code to prevent the same issue from recurring. 
• Remediates runtime issues in Kubernetes automatically, including CPU starvation, noisy neighbor interference, and memory exhaustion. 
• Delivers these remediations directly into the IDE for review, approval, or refinement, with full causal context. 

This isn't about writing scripts or building brittle rules. Causely analyzes the environment in real time and proposes the correct remediation for the right layer. 

Example: Slow Database Queries 

Let’s say your service slows down due to database query latency. Most tools might point you to the spike. Causely uses its CRE to map causes to the symptoms they cause, like a slow-running query may cause elevated latencies across multiple HTTP paths. It combines observed signals with domain-specific causal models to deterministically infer the cause of the observed symptoms. The MCP Server then surfaces the recommended fix directly in your IDE, with full causal context. 

Remediation, Where Engineers Work 

The MCP Server brings this reasoning and remediation into the tools engineers already use. There’s no jumping between dashboards, terminals, and editors to track an issue from signal to fix. Everything happens in place, with the right context. 

Whether you’re using Cursor, Claude, or any MCP-compatible editor, Causely now provides inline, explainable remediations. Developers can fix what’s broken—runtime, config, or code—without jumping between tools or digging through dashboards. 

Here’s another look at how this works in practice. Causely identifies a CPU resource contention issue that’s degrading multiple services, traces it to a misconfigured Helm value, and proposes the corrected setting—delivered directly into the IDE via the MCP Server. 

Built for Complex Systems; Kubernetes is Just the Start

We’ve focused first on Kubernetes because it concentrates so many reliability challenges in one environment: ephemeral workloads, misaligned configurations, and distributed dependencies. But the underlying problems we solve go beyond the cluster. 

Whether the root cause lives in your service mesh, Terraform plan, or application code, Causely’s causal model surfaces it, and delivers the fix where it matters. Kubernetes is one environment we operate in. The goal is broader: to make reliability engineered, not reactive, across every layer of modern software delivery.  

See It in Action 

If you’re attending KubeCon North America, come see it in action at Causely Booth #1661. We’ll show you how Causely detects what’s wrong, explains why it’s happening, and remediates it, right where the fix belongs.  

And if you’re ready to explore more on your own, start here: https://docs.causely.ai/ask-causely/mcp-server/  

“Bringing Causely to Google Cloud Marketplace will help customers quickly deploy, manage, and grow the Causal Reasoning Engine on Google Cloud's trusted, global infrastructure," said Dai Vu, Managing Director, Marketplace & ISV GTM Programs at Google Cloud.

Causely’s reasoning engine models how distributed systems behave, identifying the cause of reliability risks, their impacts, and the actions required to assure performance. With Google’s Gemini models, Causely automatically generates clear, context-rich explanations and remediation guidance that help teams act with speed and confidence.

“Our causal engine determines why services experience increased latency and error rates. Google’s Gemini models help communicate what to do next, translating our causal inference into actionable, automated guidance. We’re eliminating the need for incident war rooms and enabling proactive reliability,” said Yotam Yemini, CEO of Causely.

Two new features leverage Gemini models within Causely:

• Ask Causely: A chat interface for using Causely's inferencing engine to understand the cause of increased service latency and error rates and the corresponding blast radius.
• Enhanced Root Cause Descriptions: Causely prompts Gemini models with the metrics, symptoms, events, and logs associated with each root cause, allowing it to summarize the issue and expand Causely’s remediation with detailed, context-specific guidance.

Causely customers have reported up to 75 percent faster recovery and 25 percent fewer incidents, improving uptime and productivity. While optimized for Google Cloud, Causely remains multi-cloud and model-agnostic, operating across public clouds and on-prem environments and integrating with a range of large language models.

About Causely

Causely is an AI startup dedicated to transforming Site Reliability Engineering through innovative automation, causal reasoning, and developer-centric tools. Their solutions help organizations manage complex distributed systems more efficiently and reliably.

Media Contact

Adam LaGreca
Founder of 10KMedia
adam@10kmedia.co

Causely addresses this gap through its Causal Reasoning Engine, which models how dependencies interact in real time and accurately determines the cause of observed service latency and errors. By inferring the cause of performance degradation and understanding the affected dependencies, Causely enables automated actions to assure performance.  

Now, through a new collaboration with Google Gemini, engineering teams can act on those insights faster and more intuitively. 

Why We Started with Gemini  

Reliability engineering depends on both accuracy and trust. LLMs excel at interpreting vast, unstructured data, but without a principled understanding of cause and effect, their outputs are prone to hallucination.  

Causely provides that missing foundation. Its Causal Reasoning Engine models how services, dependencies, and resources interact. These causal models provide deterministic truth about the causes of performance anomalies, and their blast radius. LLMs build on this foundation by translating the results of this causal inference into natural language explanations and action plans that help teams act with confidence. The result is a real-time, closed loop between insight and action. 

We chose Gemini because of its contextual reasoning, enterprise-grade security, and deep integration with Google Cloud workloads. Gemini’s ability to interpret natural language, generate structured code, and summarize technical context complements Causely’s deterministic causal inference engine, turning complex telemetry into clear and reliable insights. 

How Causely Uses Gemini to Enhance Autonomous Reliability 

While Causely remains interoperable with any LLM that a customer wishes to use, we’ve integrated Gemini into Causely for two new features that make interacting with our Causal Reasoning Engine more intuitive and powerful. 

Ask Causely  

Ask Causely empowers users to ask complex questions about their environment, check service health, and identify both existing root causes and potential failure points. Ask Causely leverages Gemini’s natural language understanding and generation capabilities to deliver a conversational and seamless experience. It uses multiple Gemini models and takes advantage of Gemini’s generative features to provide a white-glove reliability experience. 

Gemini is integrated into several stages of the Ask Causely pipeline, offering a high degree of flexibility, control, and integration within Causely’s autonomous reliability framework. To ensure timely and accurate results, Causely uses low-latency Gemini models to support data-intensive operations such as log summarization, entity extraction, and contextual signal analysis across diverse telemetry sources.  

Key aspects of the integration include: 

• Adaptive Model Selection: Causely strategically deploys low-latency Gemini models to ensure quick responses while using higher-capability reasoning models to convert Causely’s causal diagnoses into clear, actionable remediations. 
• Grounded search for reliable knowledge: Ask Causely uses Gemini’s grounded search capability to deliver accurate, context-aware remediations based on trusted external sources such as vendor documentation, Stack Overflow, and GitHub. 
• Tool calling and code generation for live system intelligence: Ask Causely uses Gemini’s tool calling and code generation to query live services, interpret telemetry, and surface insights from Causely’s causal engine on identified symptoms and root causes. 
• Code generation for automation: Gemini’s code generation enables Causely’s Code Agents to analyze time series and topology data, generate diagnostic workflows, automate remediations, and perform dynamic analysis during active incidents. 
• Entity recognition: Gemini’s strong entity recognition helps Causely rapidly locate and correlate critical services, nodes, and components within complex environments. 
• Embeddings for enterprise grounding: Gemini embeddings enable Causely to integrate internal documentation and historical incidents to deliver organization-aware, contextually grounded insights. 
• Interpretable causal insights: Gemini translates Causely’s causal signal extraction from unstructured telemetry into clear, human-readable explanations and actionable remediations.  

Causal Explanation and Remediation 

Causely uses Gemini to generate SLO-aware, application-specific explanations and actionable remediations grounded in verified causal data and analysis. Causely infers the precise cause of observed anomalies and gathers the most relevant logs and events from across the environment to enable automated action. Gemini then contextualizes this evidence with Causely’s live causal graph to produce coherent, human/machine-readable descriptions and remediations that reflect the underlying issue, its operational impact, and suggested remediation steps 

Supporting features include: 

• Log and event contextualization: Gemini interprets logs and events selected by Causely’s reasoning engine, connecting raw telemetry to observed symptoms and their SLO implications. 
• Causal-grounded remediation actions: Recommendations are based on observed symptoms and tailored to the affected application or service context. 
• Automated post-incident summaries: Gemini compiles structured summaries that capture causal explanations, operational impact, and applied remediations, ensuring consistency and traceability across incidents.  

Open and Flexible 

We started with Gemini as an initial and foundational proof point. But our platform was built to be multi-cloud and model-agnostic. Causely runs across public clouds or on-prem environments, and we’ll continue to develop integrations with other large language models, giving customers flexibility without lock-in. If you have a particular model and use case in mind, please contact us! 

The Future: From Reactive to Autonomous Reliability 

Causely and Google Gemini together mark a step toward autonomous service reliability, where systems can understand, explain, and prevent issues before users are affected. This shift moves reliability from reactive firefighting to proactive, explainable prevention.  

See It in Action 

Explore the new Gemini-powered capabilities in Causely:

• View Causely on Google Cloud Marketplace ➜
• Request a Demo ➜ 

They cut through the hype and explore:

• Why AI SREs are gaining so much attention
• How to make AI benefits in operations tangible
• The limitations of language models in SRE work

Yotam shares insights from his unique journey from psychology to tech, and offers a fresh perspective on building reliable systems in the age of AI. Listen below or watch it on YouTube.

The goal: modernize without compromising performance. 

The constraint: no in-place upgrade path.

The outcome: a seamless migration, with zero regressions.

The Stakes: Migrating a Backbone Cluster Under Load

Quantum Metric is the customer-centered digital analytics platform for today’s leading organizations, enabling global brands to make smarter, faster decisions about their digital customer experiences. Their platform ingests and processes petabytes of data each day to help these brands avoid revenue loss and deliver more seamless digital journeys across their web and mobile applications.

Operating at this scale presents numerous challenges and Quantum Metric’s engineering team is constantly looking for ways to best serve their customers. The team wanted to upgrade to Google Cloud’s Dataplane V2 as a way to address operational challenges related to networking, but the team knew this wouldn’t be a simple migration to execute. The cluster in question hosted some of their highest-throughput services, including internal request handlers and real-time data processing pipelines. Several of these services alone handled over 30,000 requests per second.

With no in-place upgrade available, success meant spinning up a new cluster, migrating services in stages, and deprecating the old one without impacting production.

Stepwise Migration with No Room for Error

The team adopted a blue-green migration strategy. They started by shifting lower-risk, stateless services with minimal dependencies, which was ideal for early validation. From there, they moved to heavier components with deeper integrations and higher throughput.

Every step introduced risk. The services being moved were foundational; any missed dependency or regression could create a cascade of downstream failures.

Why Causely Was Essential

For Kevin Ard, Staff Platform Engineer at Quantum Metric and leading the migration, confidence throughout the process came from one source: Causely.

“Because of Causely, I didn’t need to do any custom telemetry work or worry about if something was off as changes were rolled out. The system made it simple to complete our migration with confidence.”

Causely continuously builds a live model of service and data flows using lightweight, eBPF-based tracing - no manual instrumentation, dashboard-building or query-writing is required. As Kevin shifted traffic, Causely proactively showed and analyzed real-time changes in service dependencies and system health. If something started to show signs of degradation, it showed exactly where and why so that changes could be applied ahead of any major problems. 

Example: Causely’s service and dataflow graphs show how requests and data move across services—and surface the true root cause directly within that flow.

Rather than stitching together dashboards or relying on intuition, Kevin was able to rely on Causely as a real-time copilot that analyzed cause-and-effect for him as changes were being made. 

The Result: Two Weeks Saved, No Fire Drills

Without Causely, Kevin estimates he would’ve spent two weeks building dashboards, coordinating instrumentation with at least three other engineers, and piecing together a view of the system. Instead, he had a single, always-on copilot keeping an eye on things as traffic shifted.

“With the sheer volume of telemetry our systems can generate, filtering the noise to focus on causality was a game-changer. Knowing that Causely would proactively spot degradations without needing to configure any rules or alerts was icing on the cake.”

The result:

✅ Zero regressions

✅ No performance degradations

✅ No fire drills

✅ A smooth cutover of a high-throughput cluster

Making High-Scale Reliability Practical

Even in environments with strong observability, Causely adds something critical: real-time causal reasoning. It understands not just what changed, but why – and it does this automatically without custom dashboards or complex rules. 

At Quantum Metric’s scale, reliability means preventing issues before they become incidents. During a high-risk migration, Causely gave Kevin and his team a new kind of clarity rooted in cause-and-effect across dynamic systems. This helped them improve how they think about managing complexity at scale and move fast without breaking things. 

Kubernetes has become the default backbone of cloud native architecture. But does it actually help you ship services more reliably, or is it just more moving parts?

Despite Betteridge’s law of headlines, the answer is yes, for the vast majority of companies. Kubernetes has proven essential for reliable service delivery in today’s cloud native world. But not because it magically sprinkles reliability over your stack. Rather, it gives you what you need to build reliability like an engineer, not a magician. Kubernetes is a means, never the end. But let me explain before you skeet this out of context…

Kubernetes Is Not the End, but a Means

Like any technology, Kubernetes is just a building block supporting your goals. Treat it that way. If you adopt it simply because nobody gets fired for choosing Kubernetes, you’re setting yourself up for pain. Kubernetes is powerful and unforgiving — there’s no shortage of real‑world lessons and failure stories from teams led by hype and not purpose.

Instead of chasing hype, start with the essentials. Cut to the core by asking a deceptively simple question: “Who cares?”

Who cares about the benefits —reliability, consistency and automation — that Kubernetes gives you? Teams running services on your platform expect those properties to hold under load and during change. Who cares about that? Your organization, offering those services to users. Users, who have their own goals. Those goals aren’t yours to reach, but you can make the path smooth and successful. When users succeed, they’re delighted — and that’s our end.

Why Delighted Users Matter to Engineers

Keeping that mindset, you might ask: “Why should I, as an engineer, care about delighted users?” You might wonder why you should care at all. Sure, you could go eat pretzels and scroll silly websites. But engineers don’t stop there. We look deeper — because by profession, you are a systematic thinker. You solve human needs with science and engineering.

So instead of just answering the initial question (“Do we even need Kubernetes for reliable service delivery?”), we approached it like engineers. We used a structured method called causal analytics — asking “Who cares?” — until the real goal emerges: Does Kubernetes deliver the level of service reliability that we need to delight our users?

All that remains is to break it down in your specific context: Who are your users? What is a delightful experience for them? What expectations for reliability do they have?

I am certain that after answering those questions, 95% of you will conclude that your needs for reliability are not exceptional, and that Kubernetes is the right choice to achieve those goals. This should not be surprising, because as with most requirements, there’s a bell curve; most teams need a reasonable amount of reliability, some need a little and a few need the extreme.

The teams needing extreme reliability are building systems where lives are literally at stake, if they fail — aviation, medicine and similar fields.

For the teams that need less than average reliability, here’s a story: Years ago, I asked the ops team of a world-class soccer club how much reliability mattered for their online store. Their answer surprised me — “Our users are fans! If they can’t get a jersey now, they’ll wait. Reliability isn’t our differentiator — we focus on a unique fan experience.“

Indeed, fans happily wait. Half the fun is in the waiting, which deepens their sense of belonging.

What I learned from that is simple — it’s perfectly fine to be average in most areas, as long as you know where not to compromise.

You’re an Engineer – Behave Like One!

As you might have guessed, the point of this text is not to settle the question of whether you need Kubernetes for reliable service delivery. It is to remind you that you should not only like building and tinkering like an engineer but also enjoy thinking and making decisions like one. It’s your job to know when it is perfectly fine to deliberately pick the boring, established standard and when you need to go against the grain.

When you make decisions that way, you gain clarity and calmness. You will read yet another blog post from teams ditching Kubernetes and understand that they might be one of the outliers that need something different. But because you made the choice to adopt it with your context in mind, you won’t regret it.

Even if you reach the point where you need to revisit your decision, as new information and context emerge, you’ll approach the problem with the right questions: Given our current context, is moving away from Kubernetes the right thing to do? Or is it preferable to continue using it, even if your enthusiasm has waned?

Finally, it’s not just about Kubernetes. Every technology you add is a means, not the end and should be evaluated in your context. This mindset is especially useful for approaching hype cycles. Right now, everyone is trying to solve every problem with LLMs. Next year it will be something else. The cycle never stops.

But as an engineer, your job isn’t to chase the latest shiny thing. It’s to understand your context, weigh the trade-offs and decide deliberately. Sometimes that means adopting the new; sometimes it means sticking with the boring. The strength lies in knowing the difference.

Causely already mediates to OpenTelemetry, Datadog, and Dynatrace to consume traces, metrics, alerts and logs. Today we’re adding IBM Instana Observability to that list.

This is yet another step in our journey to autonomous service reliability: Our Causal Reasoning Engine (CRE) takes the flood of data you already collect and turns it into action by automatically inferring and pinpointing the causes of service degradations, reliability issues, and emerging risks. Furthermore, Causely provides context-rich explanations and remediation guidance—so responders move faster and with confidence. 

Why This Matters 

Causely eliminates the need for war rooms. Causely’s reasoning engine analyzes telemetry, traces, logs and alerts from your observability stack—including IBM Instana Observability—minimizing MTTR, protecting SLOs, turning dashboards into decisions, and enabling reliability.

“Our goal is to collapse the war room—use the Instana data you already have to identify true causes and risks, then deliver clear, guided remediation,” says Ben Yemini, Head of Product at Causely. 

Causely’s users stop chasing alerts and act on clear, context-rich explanations and guided remediation. Causely maintains a continuously updated topology graph of your services, data flows and infrastructure and a causality graph that maps causes to symptoms. Using probabilistic inference, CRE pinpoints causes, evaluates blast radius, and prioritizes remediation—so teams converge faster with fewer escalations. Because the engine runs continuously, it also highlights emerging risks before they become incidents. 

As one customer put it:

“We don’t have a shortage of telemetry—we’ve been using Instana for years. We’re overwhelmed by data and dashboards. Dealing with performance issues is labor-intensive, and it’s been nearly impossible to prevent them.”

This integration is designed to change that. 

Key Features of the IBM Instana Observability Integration 

• Automatic root cause analysis pinpointing cause and risk in real time. Causely’s CRE contiguously ingests Instana signals and pinpoints the causes of service degradations and failures and surfacing emerging risks before they become incidents.  
• Automatic impact analysis and blast radius mapping. Causely’s CRE continuously infers the impacted services, endpoints, databases, and SLOs to drive the right prioritization.  
• Automatic remediation. Causely provides context-rich explanation of the root causes, but more importantly can automatically remediate the cause by providing the specific fix at runtime, configuration, or code level.  
• Fast, low-lift enablement. Connect Instana via API and start diagnosing in minutes without changes to your services. 

The result:  

• Continuously assuring service reliability 
• Minimizing MTTR 
• Improving operational efficiencies  
• Increasing productivity 

Learn More

To learn more about using Causely with IBM Instana Observability, see the integration guide. 

That matters to us because Causely turns your observability data into distilled insights to reason over cause and effect to put teams in control. The faster you get baseline, trustworthy signals, the faster we can do that work. 

Why Beyla Helps 

Teams come to us in two states. Some already have a healthy OpenTelemetry footprint; for them, we plug Causely in and get to causal reasoning quickly. Others are still early: traces are patchy, metrics are inconsistent, logs are everywhere. Beyla makes that second state less painful. You turn it on and a picture forms: service boundaries, dependency maps, request paths. Suddenly they’re no longer guessing. Our engine creates a consistent baseline of telemetry they can trust, and the engine can begin attributing symptoms to specific causes with confidence.  

The effect shows up in the first week: fewer blind spots, clearer causal analytics, and the ability to move from “seeing an error” to “knowing why it happened and what to change.” Beyla helps create the raw material; Causely turns it into decisions. 

What the OBI Donation Signals 

Donating Beyla to the OTel ecosystem signals a commitment to standards and longevity. It’s a move toward shared building blocks rather than one-off integrations. We value that because our promise — autonomous reliability without drama — depends on predictable inputs and open interfaces. The more OTel wins, the healthier the whole stack becomes. 

How This Looks With Causely 

Our agents roll out Beyla by default. Setup is minimal: deploy Beyla, establish a baseline of traces/metrics, and immediately begin causal reasoning. Beyla provides out-of-the-box visibility; Causely layers causal inference, risk scoring, and proposed actions.

You get: 

• A consistent baseline of traces/metrics powered by eBPF auto-instrumentation. 
• High-confidence root cause detection that doesn’t drown you in correlations. 
• Guardrails that respect zero-trust boundaries — you keep your data; Causely reasons over signals, not secrets. 

It adds up to a simple promise:

Faster time to control, not just faster dashboards.  

Try Beyla on Your Own 

If you’d like to see what we’re talking about, the easiest path is to stand up a tiny environment locally and watch Beyla fill in the picture. The steps below take you from a single service to a small conversation between services, and then into traces and metrics you can actually explore. 

Single Service + Beyla (Docker) 

Pick a simple HTTP service; the example below instruments a demo app running on port 8080 and prints spans to the console — no OTLP required to get started. 

# Terminal 1 — start your app 
docker run --rm --name demo -p 5678:5678 golang:1.23 go run github.com/hashicorp/http-echo@latest -text=hello 

# Terminal 2 — run Beyla next to it (console output only) 
docker run --rm \ 
  --name beyla \ 
  --privileged \ 
  --pid="container:demo" \ 
  -e BEYLA_OPEN_PORT=5678 \ 
  -e BEYLA_TRACE_PRINTER=text \ 
  grafana/beyla:latest 

Open the app in your browser (http://localhost:5678), click around to generate traffic, and watch spans print in your Beyla terminal. Save a short snippet of that output — we’ll use it below as an example of what “good” looks like. 

If you want to send data to an OpenTelemetry Collector, Tempo, or Jaeger, add the following to the Beyla container:  

  -e OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \ 
  -e OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 

A Small Web of Services (Docker Compose) 

To make this concrete, the snippet below starts two services that talk to each other, runs Beyla against the frontend, and wires up local backends (an OpenTelemetry Collector that feeds Jaeger for traces, and Prometheus scraping Beyla’s metrics). 

services: 
  frontend: 
    image: golang:1.23 
    command: ["go", "run", "github.com/hashicorp/http-echo@latest", "-listen=:5678", "-text=hello"] 
    ports: 
      - "5678:5678" 
    environment: 
      - BACKEND_URL=http://backend:9090 
 
  backend: 
    image: ealen/echo-server:latest 
    environment: 
      - PORT=9090 
    expose: 
      - "9090" 
 
  beyla: 
    image: grafana/beyla:latest 
    privileged: true 
    pid: "service:frontend" 
    environment: 
      - BEYLA_OPEN_PORT=5678 
      - OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf 
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4318 
      - BEYLA_PROMETHEUS_PORT=8999 
      - BEYLA_TRACE_PRINTER= 
    depends_on: 
      - frontend 
      - jaeger 
 
  jaeger: 
    image: cr.jaegertracing.io/jaegertracing/jaeger:2.10.0 
    ports: 
      - "16686:16686"  # Jaeger UI 
      - "4318:4318"    # OTLP/HTTP ingest (native in v2) 
 
  prometheus: 
    image: prom/prometheus:latest 
    volumes: 
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro 
    ports: 
      - "9090:9090" 

Create the Prometheus config next to your compose file: 

prometheus.yml 

global: 
  scrape_interval: 5s 
scrape_configs: 
  - job_name: "beyla" 
    static_configs: 
      - targets: ["beyla:8999"] 

Bring it up: 

docker compose up -d 

Hit the frontend to generate traffic: 

curl -s http://localhost:8080/ | head -n1 

When the stack is up, open the Jaeger UI at http://localhost:16686 and search for the frontend service to browse traces. For metrics, visit Prometheus at http://localhost:9090 and try queries like http_server_request_duration_seconds_count or http_client_request_duration_seconds_count to see call patterns emerge.

What to Look for 

Generate a little pressure (for example, hey -z 30s http://localhost:8080/). In Jaeger, follow a slow trace end-to-end and note where p95 shifts between the frontend and the backend call. In Prometheus, line up the client and server RED metrics to see where latency actually accumulates — it’s a simple way to separate symptoms from causes.

Where to Read More 

If you want the exact steps, the canonical source is the Beyla documentation and the OBI pages in OpenTelemetry.

Our own docs show how Beyla and the Causely agents fit together in a few minutes of setup. 

Closing the Loop 

Beyla is the right kind of infrastructure: minimal friction, maximal signal, and donated to the place where open standards live.

If you’re ready to move from “seeing it” to “controling it,” we’d be happy to show how Causely turns that signal into confident action. 

Ready when you are. 

The Stakes: Real-Time Bets, Real-Time Risk 

For this industry-leading platform, performance is crucial. During live events, hundreds of thousands of users rely on the app to place bets in real time. If a service slows down, bets fail. If latency spikes, revenue is lost. There’s no buffer for outages, and no time for guesswork. 

Many of the platform’s highest-traffic moments, such as marquee games, playoff weekends, or last-minute betting surges, occur on weekends or late at night. This adds strain on a small team that's already stretched thin, often pulling engineers into incident response during off hours, when the toll on focus, sleep, and morale is highest. 

The company had modern observability tools in place, including Grafana dashboards, Prometheus metrics, and alerting pipelines. But when something went wrong, their SREs still had to hunt for the cause across a vast array of charts and logs. They needed something that could connect the dots for them, quickly and accurately. 

The Challenge: Observability Without Clarity 

Like many teams operating large-scale microservices architectures, they have invested in and standardized on an observability platform that provides lots of visibility. What the team lacked was understanding to enable rapid decision-making. They could see when something was breaking, but not why, or what to fix first. 

The complexity came not just from the number of services, but from the nature of the system itself. Services communicated asynchronously over messaging queues, wrote to distributed databases, and scaled dynamically based on real-time load. These components operated independently yet were deeply interdependent with one another. During peak sporting events, traffic could spike dramatically in seconds. This created unpredictable cascades of latency, retries, and congestion. 

Under these conditions, their observability tools struggled to identify the real cause of issues. Dashboards showed metric anomalies. Alerts fired across disconnected tools. However, understanding what was actually happening required engineers to mentally reconstruct a constantly shifting topology and then guess at the most likely root cause based on limited data. 

Each incident became a manual, high-stakes investigation. Teams would dig through logs and traces, cross-reference alerts, and escalate issues across multiple teams to determine where to start. That meant long triage loops, missed SLOs, and far too much time reacting instead of preventing. 

What they needed was a system that could keep up with the complexity, something that could analyze the real-time behavior of the entire environment, understand how changes ripple across services, and surface what matters before user impact.

The Solution: Causal Reasoning Embedded in Grafana 

To close this gap, the team rolled out Causely’s Grafana plugin. It gave their SREs a new layer of intelligence, right inside the dashboards they already use. 

Causely continuously consumes telemetry from their existing observability tools and applies causal reasoning to identify what matters. It answers three essential questions: 

Where is the problem?
What is it?
And why did it happen? 

With that context, Causely enables remediation, either by guiding engineers to the right action or automating it through AI. 

In this case, when service latency began trending in the wrong direction, Causely didn’t just trigger an alert. It identified the actual cause, explained the potential blast radius, and gave the team the confidence to act quickly, without second-guessing or escalating the situation. 

Watch the demo:

The Outcome: Reliability Under Constant Change  

During a major live event, the platform began showing early signs of degradation. But instead of spinning up a war room, the SRE team used Causely to trace the issue to a misconfigured connection pool in an upstream service, before end users were ever impacted. 

They resolved the issue fast, stayed within SLOs, and avoided revenue loss during one of their busiest betting windows. There was no guesswork, no back-and-forth, and no missed opportunities.  

See It at ObservabilityCon 

Causely will be at Grafana’s ObservabilityCon, showing how causal reasoning is helping engineering teams stay ahead of incidents—and focus on building, not firefighting. 

Want a 1:1 walkthrough before the event? 

👉 Book time with our team.  

Causely turns observability into action. 

It brings clarity, speed, and confidence to every incident—right from your existing Grafana workflows. 

When a system fails, identifying and resolving the technical root cause is what brings services back to health. But the story doesn't end there. To fully recover, teams also need to understand the impact outside their systems: which customers were affected, how many transactions failed, and what actions to take next. Traditional observability tools stop at symptoms, leaving engineers without a clear view from failure to business impact.

Causely changes that. By continuously reasoning over your telemetry, Causely identifies the true root cause in real time and delivers it straight into Slack — no guesswork required. From there, engineers can drill into the Causely UI to validate technical details and confirm the non-technical context behind the failure.

Combined with ClickStack by ClickHouse, teams can move beyond the technical fix to quantify and remediate the wider impact. Starting from the root cause Causely highlights, ClickStack makes it easy to pivot directly to the affected users, measure the scope of the issue, and take recovery actions like targeted win-back campaigns. Together, they close the loop: Causely restores system health, while ClickStack ensures customer trust is recovered just as quickly.

Watch the video to see how Causely and ClickStack by ClickHouse help you fix failures confidently — and address their real-world impact.

Incidents create noise, pull in too many teams, and waste valuable engineering time chasing symptoms instead of causes.

That’s why we’re excited to share our new integration with incident.io. By combining Causely’s causal reasoning engine with incident.io’s powerful automation platform, engineering teams can identify the true root cause of incidents faster and respond with greater focus.

The result: fewer people distracted, faster MTTR, and incidents contained before they spread.

In this short demo, you’ll see how Causely analyzes incoming alerts in real time, maps them onto live dependency graphs, and pinpoints the single root cause behind cascading failures. incident.io then orchestrates the response, routing the incident to the right team with the right context from the start. Together, Causely and incident.io replace reactive firefighting with clarity, precision, and automation so your teams can spend less time triaging and more time building.

That’s why we’re excited to share how Causely now integrates with incident.io. By combining Causely’s causal reasoning engine with incident.io, engineering teams with complex microservices environments can go from incident to resolution much faster.

Why This Matters

When an incident begins, the first question everyone asks themselves is, “Why?” 

In modern microservices environments, answering that causality question is hard. Trying to do it in a programmatic way that enables automation feels to many as being near impossible. Dependencies and data flows shift unexpectedly, and latency and errors ripple outward in ways that are difficult to untangle. Without an understanding of causal relationships and emergent behaviors across services, multiple service owners often get pulled into noisy investigations and waste time chasing downstream effects instead of the actual cause.

We love incident.io as an automation platform because it has a great user experience and provides the structure and automation needed for automating incident response. For customers using incident.io in large and dynamic environments, Causely adds critical intelligence by applying causal reasoning directly to live infrastructure, services, and data flows to infer the single cause behind a flood of alerts. This ensures the right service owner is reached with the right context from the start.

The result: faster resolution, fewer people distracted, and incidents contained before they spread.

How It Works

1. Alerts flow into incident.io from multiple sources such as Alertmanager, cloud platforms, and observability tools.
2. Causely analyzes those alerts in real time, mapping them onto live dependency maps of services, infrastructure, and data flows it discovers. It applies causal inference to identify the cause of the anomalies. This requires no complex rules, policies, or prior training, and scales naturally to large, dynamic microservices environments.
3. incident.io orchestrates the response with this context: routing incidents to the right owners, engaging on-call engineers, opening Slack channels, automating where possible, and keeping stakeholders informed.

For large engineering organizations with hundreds or thousands of dependencies, the value is clear. Instead of multiple teams chasing scattered symptoms, the integration ensures the right team gets the right context immediately, and impacted teams get the fastest path to innocence.

The Joint Value

By combining causal inference with automated orchestration, teams get:

• Clarity without chaos: Causely distinguishes cause from effect across live dependency maps, so only the right team is engaged.
• Seamless execution: incident.io automates the response, removing friction from every operational step.
• Aligned productivity: The integration prevents cross-team thrash, reduces MTTR, and frees engineers to stay focused on building.

Together, Causely and incident.io replace reactive firefighting with precise, dependency-aware incident response and automation.

See It in Action

We’ll be showcasing this integration at incident.io’s Sev0 Conference in San Francisco on September 23rd. Stop by the Causely booth to see how our joint solution can work for you, or request a free consultation here.

These aren’t interchangeable hats. They’re disciplines with fundamentally different goals. Blurring them may help you survive at 20 engineers, but at 200 it will strangle velocity and reliability. Even Netflix and Spotify, the poster children for speed at scale, had to abandon blended roles once the stakes got high.  

This article cuts through the noise: what these roles really are, where they overlap, and why betting on “one-size-fits-all ops” is a costly mistake.   

The Three Disciplines 

DevOps: A Cultural Foundation, Not a Job Title 

DevOps is more than just a role: It is a mindset. Born to tear down the wall between dev and ops, DevOps emphasizes shared responsibility for software delivery. 

• Goal: Deliver software quickly, safely, and consistently 
• Focus Areas: Collaboration, automation, CI/CD 
• Common Activities: Pipelines, IaC, release orchestration 
• Metric: Deployment frequency and lead time for changes  

Where companies go wrong: hiring “DevOps engineers” as if DevOps were just another box on the org chart. Rather than existing as a role or even a team, DevOps should be considered the baseline culture for modern software delivery. 

Site Reliability Engineering (SRE): Reliability as a Feature 

SRE, created at Google, applies engineering discipline to operations. It treats reliability not as an afterthought but as a product feature with measurable outcomes. 

• Goal: Keep systems reliable, scalable, and performant 
• Focus Areas: SLOs, error budgets, capacity planning, incident response 
• Common Activities: Defining SLIs/SLOs, automating toil, postmortems 
• Metrics: Error rates, availability, latency 

Where DevOps creates shared accountability, SRE enforces it with data and rigor. Reliability becomes something you can track, budget for, and improve. 

Platform Engineering: Scaling Without Chaos 

Platform Engineering is the newest of the three, and it is rapidly becoming essential. Its job is to build internal products for developers, standardized pipelines, self-service infrastructure, and golden paths that reduce cognitive load. 

• Goal: Improve developer productivity and consistency 
• Focus Areas: Internal developer platforms (IDPs), golden paths, service catalogs 
• Common Activities: CI/CD frameworks, self-service infra, observability integrations 
• Metrics: Developer satisfaction, time-to-value for new features  

Where SRE enforces reliability, Platform Engineering makes speed sustainable by removing friction and standardizing how teams build and ship. 

How They Interrelate 

These disciplines do not compete, they reinforce one another: 

• DevOps creates the culture of shared responsibility. 
• SRE makes reliability measurable and actionable. 
• Platform Engineering productizes infrastructure so both can scale. 

Together, they form a virtuous cycle: culture drives collaboration, reliability ensures quality, and platforms remove friction. 

Why The Lines Blur 

In small companies, blending roles is inevitable. One engineer might build the pipeline, run on-call, and hack together infra automation. That works when survival depends on speed. 

But the trade-offs are real: constant context switching, shallow specialization, and bottlenecks that emerge as systems grow. What looks like efficiency at 20 engineers becomes fragility at 200. 

Common blends include: 

• DevOps + SRE: one ops-focused engineer juggling deployments and reliability. 
• SRE + Platform: reliability engineers building tooling to reduce toil. 
• DevOps + Platform: CI/CD pipelines evolving into internal platforms. 

These hybrids buy time early, but they do not scale. 

The Growth Path: From Blended Roles to Specialization 

As headcount and complexity rise, most organizations follow a predictable arc: 

• Early Stage (1–20 engineers): Generalists do everything. Speed over rigor. 
• Growth (20–100 engineers): “DevOps” teams emerge, basic SRE practices are introduced. 
• Scaling (100–500 engineers): Dedicated SRE and Platform teams form to manage reliability and developer experience. 
• Enterprise (500+ engineers): Clear ownership is established across disciplines, with DevOps principles embedded everywhere. 

Ignore this progression and you will pay the price, either in velocity lost to chaos or outages caused by brittle systems. 

Lessons from Netflix and Spotify 

Even the best had to evolve. 

• Netflix: Started with infra-savvy generalists. They embraced end-to-end DevOps culture, pioneered Chaos Monkey, and eventually built platforms like Spinnaker (CI/CD) and Titus (containers). Today, reliability is a shared mandate, supported by platform teams with a product mindset. 
• Spotify: Grew fast with generalist ops, then adopted squad-based DevOps. Over time, they created dedicated reliability teams and built Backstage (later open-sourced) to tame service sprawl. Platform and SRE teams now enable hundreds of squads to ship quickly without burning out.  

Both prove the same point: blended roles help in the sprint, but specialization wins the marathon. 

Conclusion: One-Size-Fits-All Ops is a Trap 

SRE, DevOps, and Platform Engineering are not buzzwords or interchangeable hats. They are complementary disciplines that companies must deliberately balance as they scale. 

The playbook is clear: 

1. Start with DevOps as cultural glue. 
2. Invest in SRE to treat reliability as a product feature. 
3. Build platforms once scale demands it.  

Done right, this progression does not just prevent outages. It transforms operational chaos into a lasting competitive advantage. 

Two topics stood out most in the series. The first is the ongoing wrestling match between the terms Application Performance Management (APM) and observability. The second is how AI will change this space over the coming years. Let’s take them one by one.  

APM vs. Observability — The Bikeshedding Debate 

When we talk about observability, it’s important to distinguish between the technical definition and the marketing definition. In control theory, observability means “a measure of how well internal states of a system can be inferred from knowledge of its external outputs” (Wikipedia). We borrowed that definition from control theory into the domain of software, where those “internal states” are the real things that are happening invisible to us: CPU cycles, memory consumption, packet flows, the actual bits moving through our systems. The “outputs” are what we measure: traces, metrics, logs, profiles, and ultimately what the end user sees. 

It’s a good definition because it clearly states how we can make a system objectively more observable: by providing increasingly better external outputs that describe the internal states as closely as possible, until in theory one could determine the entire system’s behavior from its outputs. This is how engineers think, which is why it resonates so well with us.  

Furthermore, we can see this approach in action, as open projects and standards like Prometheus and OpenTelemetry evolve, adding new signals, more domains, and even semantic conventions to steadily improve what can be inferred from telemetry. Observability in the technical sense is just about making more of the system’s inner state visible. 

But then there’s the marketing use of the term “observability,” particularly in how vendors brand and sell their offerings as “observability platforms.” Look under the hood and you’ll see evolution, not revolution. Many features touted as “new” under the observability banner — distributed tracing, combined app and infra visibility, end-user monitoring, business-centric insights — were already present in APM solutions long before.

APM, traditionally defined as the discipline of monitoring and managing the performance and availability of software applications, had long offered these capabilities — the key difference is that they were locked inside vendor walled gardens, fragmented across products, and often poorly implemented. Observability didn’t invent them; it generalized them and, crucially, opened them up.  

That’s the real accomplishment of observability, in its marketing sense: expanding a space once dominated by closed APM vendors, breaking down their walls, and making telemetry collection a commodity. 

Beyond that, does it really matter whether we call it APM or observability? The debate may have been useful to move the industry forward, but today it’s a distraction and a waste of energy; it’s bikeshedding. Use the terms however they suit you — what really matters is ensuring systems consistently deliver the service levels our users expect in the years ahead. And that’s where AI comes in. 

AI in Observability — Hype, Reality, and What’s Missing 

The APMdigest series also collected a wide range of AI predictions. Many of them, we at Causely agree with wholeheartedly. Machine learning is already reducing alert fatigue, clustering related events, and detecting anomalies faster than humans. LLMs are making telemetry more accessible in many languages, so engineers can query systems in their own tongue without having to learn domain-specific query languages. These are real wins. 

Where we diverge is in how far LLMs can take us. At their core, LLMs are pattern-matching machines. They can generate plausible explanations, but they don’t actually understand causality. That’s a critical limitation when the task is diagnosing why a system failed. Correlation is not causation — and if you’ve been on-call at 3 a.m., you know how costly the wrong guess can be. 

LLMs are also inherently reactive. If I have to ask the chatbot why something is broken, it’s already too late. They’re not designed to proactively detect precursors of failure, to flag the subtle changes that precede an outage. That requires something different: causal reasoning. 

Causal reasoning is what turns noise into signal. It builds an explicit model of how services depend on each other and how failures propagate. That model acts as a pre-processor layer: it distills raw telemetry into precise insights about what is really going on. Once you have that, LLMs can actually shine. They can take the output of causal analysis and do what they do best — generate natural-language explanations, propose remediations, even open pull requests to apply safe fixes. 

In other words, causal reasoning is the missing layer between raw observability data and generative AI. Without it, we risk drowning in correlations. With it, we unlock the path to truly autonomous reliability. We’ve written more about this in InfoQ, on our own blog, and in industry pieces like CIO Dive if you want to dive deeper. 

Closing Thoughts 

The APM vs. observability debate may make for lively discussion, but the more interesting question is how AI will actually change the way we run systems. At Causely, we agree with much of the optimism in the APMdigest series, but we believe the crucial next step is to go beyond correlation and embrace causal reasoning. That’s how we move from data to control, from dashboards to autonomous systems. 

It’s a privilege to be part of a community that debates these questions so openly. We’re grateful to APMdigest for hosting the series, and we look forward to future editions. In the meantime, we’d love to hear your perspective — drop us a note at community@causely.ai or join the conversation with us on social media. 

It’s a reassuring idea, but it’s a myth. Microservices don’t isolate failure; they multiply it. 

In real-world distributed systems operating at scale, failures do not stay politely in their lanes. They leak across queues, caches, retries, and shared state. They multiply through invisible dependencies. And the more services you run, the harder it gets to see where the blast radius actually stops and, critically, what the cause is. 

Microservices do not automatically deliver fault isolation by design. They replace one obvious forest fire with a sprawling network of subtle, cascading brush fires. 

The Promise of Microservices 

On paper, microservices look like a natural cure for fragility: 

• Each service is independent, so one crash should not cascade to others. 
• Circuit breakers, bulkheads, and fallbacks can contain failures. 
• Advanced designs like cell-based architectures further limit blast radius. 

This sounds good in theory. And in tightly disciplined environments with mature engineering practices, some of these promises hold. 

The Reality 

In practice, fault isolation is rarely automatic and microservices make it harder to understand and control the blast radius.  

• New failure modes emerge. Latency, coordination bugs, partial outages, and data drift become common. 
• Shared dependencies betray isolation. A database, queue, or cache hiccup can silently spread impact across dozens of services. 
• Partial degradation is worse than full failure. Services stuck in retry storms or serving stale data prolong incidents instead of containing them. 
• Operational burden grows. Effective isolation requires top-tier observability, disciplined retry policies, and carefully engineered degradation strategies. 

Without disciplined engineering focus, fault isolation remains more promise than reality. 

Our Perspective at Causely 

We do not believe resilience is a side effect of microservices. It is a design goal that must be deliberately engineered, monitored, and maintained. 

That is why our system focuses on causal reasoning: 

• Mapping dependencies. We continuously uncover how services and infrastructure actually connect, not just how architects think they do. 
• Analyzing blast radius. We model how failures propagate, not just where they originate. 
• Pinpointing cause and effect. We distinguish symptoms from true root causes, even when failures ripple through retries, caches, and queues. 

The challenge is not shrinking the blast radius. It is being able to understand it clearly and programmatically mitigate the damage. That is the gap our system closes. 

TL;DR? 

Resilience does not come from the architecture you choose. It comes from how well you understand causality inside it. 

For engineering teams, that means: 

• Continuously mapping dependencies and blast radius across services. 
• Designing isolation explicitly instead of assuming microservices will provide it. 
• Using systems that reason about cause and effect, so humans are not left guessing when it matters most. 

The myth is that microservices give you fault isolation for free. The reality is that they make causal reasoning non-optional. 

If your experience has confirmed (or contradicted) this reality, we would love to hear it. Engineers get stronger when we challenge the myths together. 

Introduction 

“AI SRE” – shorthand for applying artificial intelligence (AI) to Site Reliability Engineering (SRE) – has been bubbling up in conference talks, blog posts, and vendor marketing. SRE is a discipline that applies software engineering principles to ensure systems meet defined reliability targets. The idea of automating or augmenting this work with AI is compelling: fewer mistakes, less downtime, faster resolution, and lower cost. Not to mention, most engineering leaders would acknowledge a talent shortage in trying to fill these roles. 

But the conversation is often fuzzy due to: 

• A deafening level of marketing hype disconnected from reality. 
• Varied understanding and adoption of SRE as a job function. 
• Different interpretations and implementations of AI. 
• Technology being implemented that is a poor fit for the task at hand. 

In this article, I’ll strip away the buzzwords and define what I mean by AI SRE in precise, engineering-grounded terms. I’ll clarify how “AI” is being used today and what it should mean in the context of reliability engineering. From there, I’ll examine where it adds value, where it falls short, and outline a more effective starting point built on structured causal reasoning that can empower the safe and effective automation of reliability work. 

SRE: One Discipline, Different Meanings 

The SRE role traces its roots to Google in the 2000s, when Ben Treynor Sloss described it as “what happens when you ask a software engineer to design an operations function.” At its core, SRE is about taking accountability for reliability at a defined level that balances user and business needs while preserving engineering velocity. 

To support this, the discipline introduced a set of practices that gave teams a measurable way to manage reliability: Service Level Objectives (SLOs) to make goals explicit, error budgets to balance those goals with the pace of change, and blameless postmortems to learn from failure. Automation followed as a way to reduce toil and eliminate human error. Together, these practices positioned reliability as an engineering problem rather than an afterthought. 

However, in practice, many teams struggle to adopt these ideals. Reliability work often takes a backseat to feature delivery, and SLOs are often only partially implemented across sprawling microservices. What Google codified in its SRE handbook often reaches production environments as a patchwork. 

Organizations also implement the role itself in different ways: embedded with product teams, focused on platform infrastructure, parachuting in as consultants, or owning services end to end. Most end up with some hybrid. 

No matter the org structure or maturity, the discipline of reliability engineering is well understood: applying engineering principles to keep systems available, performant, and predictable. While the goals are clear, most engineering leaders would agree there is still plenty of room for improvement in practice. 

What “AI” Actually Means 

Before defining the ideal AI SRE, we need to clarify what “AI” means and how the term is used today.  

In its original sense, Artificial Intelligence refers to systems capable of completing tasks that normally require human intelligence: reasoning, learning, problem-solving, perception, and natural language understanding.  

Classical AI approached these challenges by decomposing complex tasks into structured representations of knowledge, explicit planning procedures, and inference engines that could compute outcomes from evidence. For example, building a diagnostic assistant for medicine required an ontology of diseases and symptoms, probabilistic rules for how symptoms map to likely conditions, and an inference engine to suggest tests or treatments based on the evolving patient state. Historically, AI has included: 

• Rule-based systems
• Search and planning algorithms
• Probabilistic reasoning
• Constraint satisfaction

For deeper study, I recommend: 

• Artificial Intelligence: A Guide for Thinking Humans – Melanie Mitchell
• Artificial Intelligence: A Modern Approach – Stuart Russell & Peter Norvig
• The Book of Why – Judea Pearl
• Judea Pearl on Cause and Effect (Sean Carroll Podcast) 

These systems performed well when the domain was well defined and the relationships between causes, observations, and interventions could be explicitly modeled. But they broke down when faced with ambiguity, missing context, or natural language. 

In common usage, “AI” today typically refers to large language models (LLMs), and in some cases, smaller-scale variants known as small language models (SLMs). These models generate language by predicting the most likely next word given the previous context, using patterns learned from massive training datasets. They have become popular because they handle a wide range of natural language tasks with impressive fluency, such as text generation, summarization, question answering, and code completion.  

But while language models are effective at producing language and code, they have no inherent understanding of system state, live telemetry, or causal dependencies.  

Why Language Models Are the Wrong Starting Point for Reliability Engineering 

Reliability engineering requires exactly this kind of causal reasoning, which is why language models alone are the wrong foundation. The irony is that in its foundational sense, AI is a natural fit for reliability engineering: diagnosing outages, predicting failures, and recommending fixes are structured decision problems that align well with decades-old AI techniques. But when “AI” is reduced to LLMs and SLMs, the fit becomes like shoving a square peg through a round hole. 

Surveying the market, most self-proclaimed AI SRE approaches fall into one or more of these categories: 

• Postmortem narratives – After-the-fact write-ups shaped as much by bias as data, crafted to explain what went wrong in hindsight. 
• Correlation engines – Systems that surface co-occurring anomalies and related events during incidents but conflate correlation with causation. 
• Data-fetching assistants – Interfaces that summarize telemetry and suggest plausible explanations without guarantees they are correct or verifiable. Some resemble traditional automation engines, requiring heavy setup in data formatting, rules, and conditions before delivering value. 

When language models are used as the foundation for reliability engineering, four weaknesses undermine their effectiveness: 

• Spurious causes – Coherent but incorrect diagnoses from hallucination, logical inconsistency, or lack of live-environment awareness. 
• Unprincipled reasoning – Mimicking the language of reasoning without performing structured inference. 
• Causal identification failures – Difficulty pinpointing causes in dynamic systems, especially when new evidence contradicts learned assumptions. 
• Runaway costs – Without precise prompting and context, LLMs consume large amounts of compute to generate answers that may still be inaccurate. 

For reliability engineering, starting with a language model is starting in the wrong place. These approaches assume you already know which signals are worth chasing, when in reality multiple noisy alerts often trace back to a single root cause. Without causal reasoning at the foundation, both humans and AI waste time chasing symptoms instead of causes. Whether you are building internally at enterprise scale or looking for an off-the-shelf capability, any viable AI SRE must begin with causal analysis as its core. 

What Effective AI for Reliability Engineering Looks Like 

An effective AI SRE isn’t just a chatbot or a rule engine. It’s a framework that combines structured causal knowledge, probabilistic inference, and agentic capabilities. This foundation enables advanced reasoning and supports automation of real reliability engineering work. 

Such a system needs at least three interdependent capabilities: 

1. A live causal representation of the environment

A domain-specific causal model that encodes how components in a distributed system can fail, how those failures propagate, and the symptoms they produce, paired with a continuously updated topology graph showing the real-time structure of services, infrastructure, and their interconnections. 

Why it matters: Replaces fuzzy LLM pattern-matching with a verifiable system map, enabling deterministic reasoning. 

Counter to LLMs: Addresses unprincipled reasoning by grounding analysis in a causal Bayesian network that models directionality as probabilities. 

Example: The model knows that a latency spike in a database layer can propagate through dependent APIs three layers away and can quantify that likelihood based on current conditions. 

2. Real-time probabilistic inference over live telemetry

Continuous ingestion of metrics, traces, and logs, mapped against the causal model to identify the most likely root cause at any given moment. This inference layer reflects both structural dependencies and observed patterns of failure propagation, updating conclusions the instant new evidence arrives. 

Why it matters: Dynamic systems change fast, with new code, new dependencies, and shifting load. LLMs cannot adapt without retraining, while a probabilistic inference engine adjusts instantly. 

Counter to LLMs: Addresses causal identification failures, especially in counterfactual scenarios where new observations contradict prior assumptions. 

Example: If a new deployment creates a previously nonexistent dependency, the model incorporates it into its reasoning immediately, without new rules or configs. 

3. Cross-attribute and cross-service dependency reasoning

Beyond linking causes to symptoms, the system maps how performance attributes such as latency, throughput, and utilization depend on one another across services and infrastructure layers. By modeling these relationships, it can trace how a change in one part of the system cascades elsewhere, identify emerging bottlenecks, and detect when operational constraints are at risk. An added benefit is a sharp reduction in alert fatigue, because by isolating the true point of failure, only the responsible service owner is notified rather than multiple teams chasing downstream symptoms. 

Why it matters: Many incidents stem from chains of interdependent changes, not a single fault. Modeling these relationships eliminates blind spots and false leads. 

Counter to LLMs: Addresses spurious causes by ruling out explanations that violate known constraints or dependencies. 

Example: If a queue length increase is due to a downstream service slowdown rather than local CPU saturation, the model identifies the real cause directly. 

Either way, the same foundation of causal reasoning, probabilistic inference, and dependency modeling is required. With that foundation in place, language models finally have a meaningful role to play. 

The Role of Language Models in an Effective AI SRE 

Earlier, I noted that large and small language models can add value to reliability workflows. When provided with the right context, they can help accelerate diagnosis, generate remediations, and improve resolution timelines. But on their own, they lack causal understanding. Without grounding in system state and structure, even the most advanced model tends to produce responses that are plausible but operationally useless. 

Modern AI SRE solutions must transcend summarizing symptoms or correlating metrics. To truly support autonomous site reliability, they must be able to understand complex IT environments, reason over live telemetry, and intervene intelligently to maintain SLOs and keep services healthy. AI for SRE should not be defined by language generation alone. It must include: 

• A continuously updated causal model of the environment
• The ability to perform probabilistic inference over live telemetry
• The capacity to reason across attributes, services, and dependencies
• An agentic interface that can propose or execute safe and effective actions

Taken together, these capabilities reframe AI SRE from buzzword to engineering discipline. 

Conclusion 

The conversation about AI SREs is too important to leave to buzzwords. “AI SRE” should not mean a chatbot guessing its way through your telemetry. Treating “AI” narrowly as LLMs or SLMs misses the chance to apply decades of proven AI techniques — causal reasoning, probabilistic inference, and constraint satisfaction — to one of the highest-impact opportunities in modern software engineering: autonomous service reliability. 

The future of AI SRE will not be built on pattern-matching language models alone, but on systems that can reason causally about live environments and adapt in real time. Language models then become a powerful augmentation layer. Without that foundation, AI SRE is just another buzzword. 

Whether you are an enterprise building internally and looking to integrate causal reasoning into your architecture, or a mid-market company that needs a turnkey solution, the same principle applies: causal reasoning must come first. At Causely, we provide the flexibility to address both situations - a platform that can serve as a foundation for your own build, or a ready-to-deploy solution that works out of the box. 

When an upstream API slows down or quietly rate-limits requests, your customers feel the pain long before your team finds the cause. Error rates spike, retries pile up, queues fill, and internal services appear unreliable. The hard part is proving it isn’t you. Traditional observability tools only show symptoms inside your stack — latency, 5xx errors, and timeouts across services — making it nearly impossible to tell whether the real problem is internal or hiding in an external provider.

Causely changes that. With eBPF auto-instrumentation, Causely sees client communication with external services and maps those dependencies directly into your topology and causal reasoning engine. When a provider slows down, Causely shows exactly how the impact ripples across your services and identifies the external API as the root cause. That means your team can stop chasing internal alerts and start taking action — escalating with proof in hand or triggering fallback strategies.

Watch the video to see how Causely helps you go from symptoms to root cause instantly, even when the root cause isn’t yours.

Key Takeaways

• Large language models (LLMs) in observability excel at turning high-volume telemetry such as logs, traces, and metrics into concise human-readable narratives, but they lack structural system knowledge and struggle to isolate root causes in complex distributed architectures.
• Current LLM and agentic AI approaches are prone to hallucinating plausible but incorrect explanations, mistaking symptoms for causes, and ignoring event ordering, which leads to misdiagnosis and incomplete remediation.
• Causal reasoning models service and resource dependencies explicitly, accounts for event temporality, and supports inference under partial or noisy observations, enabling more accurate root cause identification.
• Causal graphs and Bayesian inference allow for counterfactual and probabilistic reasoning, which lets engineers evaluate remediation options and their likely impact before taking action.
• Integrating LLM-based interfaces with continuously updated causal models and abductive inference engines provides a practical path to reliable, explainable, and eventually autonomous incident diagnosis and remediation in cloud native systems.

The central goal of IT operations and site reliability engineering (SRE) is to maintain the availability, reliability, and performance of services while enabling safe and rapid delivery of changes. Achieving this requires a deep understanding of how systems behave during incidents and under operational stress. Observability platforms provide the foundation for this understanding by exposing telemetry data (logs, metrics, traces) that support anomaly detection, performance analysis, and root cause investigations. However, modern applications are increasingly difficult to manage as cross-service calls, event-driven workflows, and distributed data stores introduce complex and dynamic interactions.

For instance, in July 2024, a faulty configuration update in CrowdStrike’s Falcon sensor caused widespread crashes on millions of Windows systems across industries worldwide. In another case, the 2016 removal of the tiny but widely used left-pad package from npm briefly broke thousands of builds and disrupted major websites until it was restored, revealing the fragility of transitive dependencies at scale. Whether the trigger is an external contingency or a rare emergent interaction within a highly coupled system, modern IT infrastructure can experience widespread service outages due to complex cross service dependencies. Implicit dependencies, asynchronous communication, and distributed state make it challenging to pinpoint the source of incidents or understand the chain of effects across the system.

A new class of AI-based observability solutions built on LLMs is gaining traction as they promise to simplify incident management, identify root causes, and automate remediation. These systems sift through high-volume telemetry, generate natural-language summaries based on their findings, and propose configuration or code-level changes. Additionally, with the advent of agentic AI, remediation workflows can be automated to advance the goal of self-healing environments. However, such tools remain fundamentally limited in their ability to perform root-cause analysis for modern applications. LLM-based solutions often hallucinate plausible but incorrect explanations, conflate symptoms with causes, and disregard event ordering, leading to misdiagnosis and superficial fixes.

Fundamentally, LLMs operating only on observed symptoms gleaned from telemetry are attempting to deduce root causes by traversing logs and shallow topologies. LLMs lack, however, an a priori understanding of the environment as a dynamic system with evolving interdependencies. As a result, underlying issues will persist even if symptoms are partially remediated in the short term.

Effective root-cause analysis in complex, distributed systems requires understanding the causal structure of events, services, and resources. Causal knowledge and reasoning remain critical missing components in modern AI-based observability solutions. Causal knowledge is codified into causal graphs, which model inter-service and resource dependencies explicitly. By supporting counterfactual inquiry, causal inference enables root-cause isolation and systematic remediation analysis. Augmenting LLMs and agentic AI with continuously updated causal models and an abductive inference engine (which identifies the best explanation for observed symptoms using causal reasoning), offers a path toward autonomous service reliability.

In this article, we begin by outlining the strengths of LLMs and agentic AI in observability and incident management. We then examine their limitations in performing accurate root cause analysis and driving effective remediation. Next, we introduce how causal knowledge and inference engines provide the missing context for precise incident diagnosis and response. Finally, we discuss how combining causal reasoning with AI agents enables proactive incident prevention, automated remediation, and the path toward autonomous service reliability.

The Strengths and Promise of LLMs and Agentic AI

The term "AI" has become increasingly overloaded with marketing hype and public fascination. AI now applies to everything from threshold-based alerting scripts to autonomous agents capable of planning and acting across complex workflows. For simplicity, AI solutions in the observability space can be categorized as rule-based systems, LLM-based tools, and agentic AI systems. Rule-based systems include hand-crafted logic and statistical models configured to monitor baseline deviations, detect known signal patterns, and apply threshold-based alerting across logs, metrics, and traces.

LLM-based solutions leverage the generative and language-understanding capabilities of language models to support natural-language interactions with observability data. LLMs can process unstructured telemetry such as logs, traces, and alert descriptions to generate summaries, interpret errors, and create remediation plans. Agentic AI allows LLMs to act in a managed environment by providing multi-step planning, tool-assisted execution, and direct code and configuration changes. Next, we examine the specific strengths of LLMs and agentic AI in the context of observability.

LLMs are neural architectures pretrained on large-scale corpora of natural language, code, and other text-based resources. Despite being fundamentally next-token predictors trained to model the conditional probability of text, when scaled to billions of parameters and exposed to terabytes of diverse data, LLMs become highly effective at producing coherent language and supporting a wide range of language-based tasks. Modern LLMs have been further fine-tuned for instruction-following, factual recall, code generation, and domain-specific question answering, used subsequently to explain errors, answer technical questions, and generate code, scripts, and configuration changes.

In observability contexts, LLMs can interpret complex logs and trace messages, summarize high-volume telemetry, translate natural-language queries into structured filters, and synthesize scripts or configuration changes to support remediation. Most LLM solutions rely on proprietary providers such as OpenAI and Anthropic, whose training data is opaque and often poorly aligned with specific codebases or deployment environments. More fundamentally, LLMs can only produce text. They cannot observe system state, execute commands, or take action. These limitations gave rise to agentic systems that extend LLMs with tool use, memory, and control.

Agentic AI comes closest to delivering on the speculative promise of AI. In practice, agentic systems commonly follow the ReAct framework introduced by Yao et al. (2022), which integrates reasoning and action in an interleaved loop. In this setup, the LLM generates intermediate reasoning steps, selects actions such as querying tools or retrieving information, and incorporates feedback from those actions to inform the next step. This cycle of thought, action, and observation allows the system to iteratively plan, update context, and progress toward a goal. With these capabilities, LLMs are able to write applications, solve multi-step reasoning problems, generate code based on system feedback, and interact with external services to complete goal-directed tasks.

Agentic AI shifts observability workflows from passive diagnostics to active response by predicting failure paths, initiating remediations, and executing tasks such as service restarts, configuration rollbacks, and state validation. However, current agentic systems lack a priori structural and causal models of the environment, which limits their ability to anticipate novel failure modes or explain observed behavior beyond surface-level associations. While these constraints remain, agentic AI represents a necessary step toward autonomous, tool-integrated systems capable of reasoning and acting within complex managed environments.

The ultimate promise of applying agentic AI to IT operations is autonomous service reliability. An ideal system continuously monitors telemetry, identifies potential failures, evaluates impact, and applies targeted interventions with limited human oversight. Integrated into the observability and operations stack, agentic AI should function as the control layer. It reasons over system state, coordinates diagnostics, and orchestrates remediations so that services operate reliably and in alignment with defined service level objectives (SLOs), which specify availability, latency, or other performance targets. Ultimately, autonomous service reliability reduces operational complexity, accelerates incident resolution, and improves service reliability across large-scale, dynamic environments.

On The Limitations of Modern AI and the Need for Causal Reasoning

Modern service architectures often rely on shared infrastructure and layered services, where dependencies are opaque and incident signals surface far from their origin. Imagine the simple service topology shown in Figure 1, which consists of two shared resources (R1 and R2) and a set of services (S1–S5) connected through overlapping dependencies. Now consider that we observe elevated latency and request timeouts at service S5. The underlying root cause is connection exhaustion on resource R2, which intermittently blocks new connections. This condition is not surfaced through direct telemetry because service S2, which depends on resource R2, reports only latency and timeouts without exposing the underlying resource-level failure. Tracing the issue upstream, service S3 shows increased request latency, and service S2 exhibits degraded performance. Meanwhile, service S1 also reports elevated CPU usage and latency, though its downstream service S6 remains unaffected.

An LLM-based agent begins with observable symptoms (see Figure 2). It queries telemetry, inspects logs, and follows trace spans starting at S5. Along the path through S3 and S2, it observes anomalies based on latency and request failures. It also notices performance degradation on S1 and considers it a potential contributor. The agent restarts S1 and S2 and observes that the latency and timeouts at S5 are mitigated, leading it to conclude the issue is resolved. However, the problem resurfaces once connection limits on resource R2 are hit again. This scenario illustrates two key challenges.

First, spurious signals can misdirect diagnosis by drawing attention to unrelated events. Second, some root causes cannot be directly observed through telemetry and must instead be inferred from incomplete or indirect symptoms. For instance, in this scenario, the connection exhaustion on R2 emits no direct signal. It must be inferred by reasoning over the observed symptoms across S2, S3, and S5 in combination with structural knowledge of the system. Therefore, resolving such incidents requires principled causal reasoning and structural causal knowledge.

Causal knowledge maps the various causal relationships typically found in modern service architectures. The architectural knowledge provides a structural understanding of the various resources, services, and data dependencies. Finally the causal graphs provide a probabilistic framework for inferring root causes given the observed symptoms in context of the architecture.

Causal knowledge represents the relationship between root causes and their observable symptoms. Such knowledge can be abstracted to support principled downstream reasoning over system behavior such as fault localization, impact analysis, and proactive mitigation. Causal graphs provide a formal structure for encoding this knowledge. Popularized by Judea Pearl, causal graphs are directed acyclic graphs that represent cause-effect relationships among variables. When applied to reliability engineering, causal graphs describe how specific failure conditions (e.g., memory exhaustion, resource saturation, lock contention, etc.) produce observable symptoms (e.g., latency, connection errors, service timeouts, etc.).

Unlike telemetry signals capturing only runtime observations or dependency graphs, which represent observed service call relationships, causal graphs provide a richer structural understanding of how faults propagate across services and resources. This knowledge can be utilized to support inferential reasoning and identify root causes from partially observed symptoms, even when the underlying issue is not directly visible. This can be accomplished by abductive causal reasoning.

Abductive causal reasoning provides a principled, logical, and inferential framework for identifying the most likely explanation for observed symptoms. Provided a set of plausible candidate root causes and a graphical model of their symptoms, abduction selects the cause that best accounts for the observed evidence. This approach offers several advantages for modern applications: it supports inference under partial observability, selects explanations based on causal sufficiency, and provides formal guarantees about the inferred root cause. When paired with causal Bayesian graphs extending causal graphs with probabilistic reasoning, abductive inference becomes tractable in large systems. These graphs encode prior knowledge about potential root causes and their associated symptoms, allowing the system to compute the most probable root cause without requiring prior training. Furthermore, likelihoods can be updated over time using posterior observations, enabling continuous refinement in dynamic environments.

Let’s revisit our scenario with the abductive reasoning framework (Figure 4). The framework begins with a defined set of all possible root causes. Each root cause is represented by a causal graph that maps the cause to its expected symptoms with associated prior probabilities. These priors capture the likelihood of each symptom occurring if the root cause is true, derived from historical incidents and domain expertise. The abductive process identifies which root cause best explains the observed symptoms while also accounting for those that are unobserved. It computes a likelihood score for each candidate by updating the prior probabilities of its associated symptoms with observational data.

In our scenario, the observed symptoms include timeouts at service S5 and latency at services S2 and S3, which align with the causal graph for connection exhaustion on resource R2. The graph also expects latency on S4; although it is not observed, that absence is included in the likelihood estimation. Competing explanations, such as CPU starvation on S1 or network congestion on S3, receive lower likelihoods because they either fail to explain all observed symptoms or have too few expected symptoms confirmed observationally. The key distinction between abductive causal reasoning and deductive reasoning is that abduction evaluates all candidate root causes against both the observed symptoms and the expected symptoms defined by the causal models.

The agent-based approach missed the actual root cause because it lacked structural causal knowledge and stopped at the first plausible symptom path. In contrast, the abductive process leverages causal models and Bayesian graphs to infer the most likely root cause even when observations are incomplete or include spurious symptoms. Abductive causal reasoning uses these structured probabilities to isolate the most coherent explanation despite partial observations and spurious symptoms.

Limitations of Causal Reasoning

While causal reasoning is a powerful approach, it does have limitations. Constructing causal models requires significant domain knowledge and ongoing effort. The graphs must accurately capture service and resource dependencies and need regular updates as architectures evolve. Coverage is another constraint. A reasoning engine can only work with root causes that are defined in the model. If a candidate cause is missing, the engine has no basis to infer it, which reduces its effectiveness in novel or poorly understood failure cases. There are also computational challenges. In large distributed environments, some root causes map to broad sets of symptoms. Computing conditional probabilities and running Bayesian inference across these sets can become costly, especially when multiple competing explanations must be evaluated in real time.

While these limitations do not diminish the value of causal reasoning, but they do highlight that such methods are more effective under specific circumstances. In service architectures with linear and well-understood dependencies, the root cause of an incident is often self-apparent, and a reasoning engine may not be required. The challenge arises in complex distributed environments where cross-service dependencies, asynchronous communication, and distributed state make it difficult to trace symptoms back to their causes. Humans can resolve these incidents, but the process is slow, resource-intensive, and often results in longer periods of downtime and service unavailability. Therefore, meeting this challenge requires developing more effective solutions that integrate causal reasoning with AI, moving beyond current limitations and enabling progress toward autonomous service reliability.

Towards the Promise of Autonomous Service Reliability

Causal knowledge is essential for diagnosing incidents in modern service architectures. LLM-based solutions and current agentic AI often miss the forest for the trees. These systems operate over logs, traces, and metrics, but lack the structural context required to reason comprehensively about system-level behavior. While LLMs can provide useful insights by aggregating and parsing telemetry, they are ultimately bounded by the inputs they receive. With incomplete observations, LLMs tend toward speculative explanations that often result in hallucinations. Without a causal understanding of how failures propagate through services and infrastructure, modern AI solutions cannot move beyond summarization of observed events to reliably identify underlying causes. Causal knowledge and abductive causal reasoning provide the missing keys that, when combined with LLMs and agentic AI, unlock effective identification of likely root causes from incomplete and partial observations in complex, distributed environments.

Therefore, it is necessary to augment modern LLMs with a causal reasoning engine to achieve effective incident analysis and autonomous reliability. A causal reasoning engine combines three key components: causal models that encode known root causes and their associated symptoms, Bayesian causal graphs that apply probabilistic reasoning over service topologies, and an abductive inference engine that selects the most likely root cause given partial or noisy observations. This engine can operate as an external reasoning layer, providing structured causal context that modern LLMs lack.

Drawing from the principles of neuro-symbolic reasoning, the LLM serves as the flexible language interface, while the causal reasoning engine verifies hypotheses, refines candidate root causes, and performs advanced reasoning that goes beyond the LLM’s predictive text generation. By integrating this engine, LLM-based agents can transition from surface-level incident triage to precise root-cause identification and actionable remediation, constructing a path toward proactive, autonomous service reliability.

Causal agents provide a meaningful step in the pursuit of autonomous service reliability. By integrating causal knowledge and abductive reasoning, these systems move beyond reactive response and manual triage. They enable proactive incident prevention by identifying emerging risks, support targeted remediation through structural awareness, and drive self-healing by isolating and addressing root causes without human intervention. The result is a system that not only detects symptoms but understands their context. This shift reduces downtime, accelerates resolution, and aids teams in managing reliability at scale with minimal manual intervention.

OTel’s Established Best Practices for Logging

The OTel project has laid a strong foundation for logs to work alongside metrics and traces: 

• Consistent Structure – OTel’s log data model ensures every log record has a predictable shape, making it machine-parseable and easier to route or query.  
• Context propagation – By embedding trace IDs, span IDs, and resource attributes in every log entry, OTel makes it possible to pivot seamlessly between logs, traces, and metrics in your analysis tools, without guesswork.  
• Unified Collection – The OTel Collector can ingest logs from multiple sources — applications, infrastructure, third-party services — and normalize them before routing to storage or analysis tools. 
• Enrichment and Filtering – Processors in the Collector allow you to enrich logs with metadata (region, environment, service version) and filter out noise or limit verbosity. 

If your team is following these practices, congratulations: yYou already have the building blocks for the next step. 

Why Logs Need Context to Be Useful 

Logs are essential to troubleshooting — they carry the fine-grained evidence of what went wrong. But in large, OTel-instrumented systems, collecting every log line from every service creates new challenges: 

• Mountains of data – terabytes of logs that make queries slow and expensive. 

• Information overload – too many log lines without guidance on which ones matter. 

• Slow incident response – engineers spend precious time searching instead of fixing. 

The goal isn’t to reduce the importance of logs — it’s to use them in context. By correlating logs with metrics and traces, you can elevate logs from raw text into causal signals: proof tied directly to the root cause of a problem. 
 
Instead of asking engineers to manually dig through a warehouse of logs, the right approach is to automatically pull the relevant 0.1% of logs that explain the incident. Logs don’t lose their role; they gain clarity by being connected to the surrounding signals. 

Tutorial: Implementing Contextual Logs in OTel 

Here’s a practical outline for how to set this up today using OTel: 

Step 1 – Ensure Rich Context in Logs 

Start with a simple log statement, for example: 

log.warn("Database connection pool exhausted – queries may fail");

On its own, this message is useful but limited. In a large distributed system, you need more context to understand which service emitted it, in what environment, and how it relates to the rest of a trace. 

That’s where OTel comes in. When the OTel SDK is properly configured, it automatically enriches log records with resource attributes (service name, version, environment, region, etc.) and correlation identifiers (trace_id, span_id). 

For example, the same log record might look like this once enriched: 

resource: 
  attributes: 
    service.name: "checkout-service" 
    service.version: "1.2.3" 
    deployment.environment: "production" 
    cloud.region: "us-east-1" 
trace_id: "91be8f92d45e471ea0bf1c25be8e3f1c" 
span_id: "2d4b1c73c9c3e7f0" 
severity_text: "WARN" 
body: "Database connection pool exhausted – queries may fail"

This enrichment happens automatically if you’ve instrumented with OTel, which means developers don’t have to manually add metadata. They just log as usual, and OTel ensures those logs are correlatable with traces and metrics. 

Step 2 – Collect and Route Logs Through the OTel Collector 

service: 
  pipelines: 
    logs: 
      receivers: [otlp, filelog] 
      processors: [attributes, filter] 
      exporters: [otlphttp]

Step 3 – Use Metrics/Traces to Narrow the Scope 

In your analysis tool, start with: 

• Error rate, latency, or saturation metrics to find affected services.
• Trace waterfalls to spot where requests slow down or fail. 

Step 4 – Pull Only Relevant Logs 

service.name = "checkout-service" 
AND timestamp >= "2025-02-10T14:03:00Z" 
AND timestamp <= "2025-02-10T14:08:00Z"

If using the OTel Collector with filtering, you can even forward only scoped logs to your log store during incidents. 

Step 5 – Confirm and Remediate 

Use the filtered logs to confirm the root cause hypothesis, capture the evidence you need for a fix, and move directly to remediation. 

How Causely Operationalizes This Workflow 

The tutorial above shows one way to approximate a more efficient log workflow by narrowing scope before querying logs. Causely builds on this approach by treating logs as integral to causal analysis — automatically surfacing the ones tied to the root cause. 

Here’s how we do it automatically: 

• Top-down causal reasoning – Causely continuously builds a model of your environment’s dependencies, behaviors, and known failure patterns.  
• Automatic root cause inference with contextual logs – Metrics, traces, logs, and topology context are analyzed together in real time to pinpoint the single most likely root cause. Relevant error logs are automatically surfaced in that context — tied to the specific service, time window, and failure mode.  
• Remediation specificity – Contextual logs and causal context are routed to an LLM, producing precise and actionable remediation paths. 
• Bring your own logs – Whether shipped via OTel or another pipeline, Causely ingests your logs and automatically pulls the right ~0.1% on demand. No pipeline reconfiguration or log warehouse required. 

The result: Logs remain first-class citizens. You see them in context of metrics and traces, as proof and evidence of the root cause and remediation path — not a warehouse to sift through, and not a second-class afterthought. 

Wrapping Up 

The OTel community has done the hard work of making logs consistent, enriched, and correlated with other telemetry. The next step is using them more effectively: elevating logs from raw records to causal signals. That shift makes troubleshooting faster, cheaper, and more reliable. 
 
If you’re following OTel logging best practices, you’ve laid the foundation. With Causely, you can take it to the next level: automatically inferring the root cause in real time, surfacing logs in context, and generating a clear remediation path. All without workflow changes, pipeline reconfiguration, or trade-offs between log richness and operational efficiency. 

This post explores four architecture patterns where standalone Docker is not only justified but recommended. We’ll dive into the technical reasons behind each pattern, explore the operational tradeoffs, and show how Causely reduces the complexity of managing performance across these hybrid systems.  

Architecture Pattern #1: Edge & Performance‑Critical Standalone Docker Services 

Use Case:

Real-time inference, inline data transformation, high-throughput gateways, protocol termination (e.g., gRPC or TLS offloading) 

Why Docker Makes Sense: 

• These services often sit at the ingress point of traffic or at the edge of the network, where ultra-low latency and deterministic behavior are required. 
• Standalone Docker provides tight control over CPU affinity, memory isolation, and scheduling—where Kubernetes can introduce variability through abstraction. 
• With no orchestrator overhead and direct access to the host network or hardware, teams can squeeze out performance optimizations critical to meeting service level expectations.  
• Example: A fraud scoring engine on a GPU-backed instance running outside the cluster for sub-10ms response time. 

Tradeoffs: 

• Requires homegrown automation for lifecycle management. 
• Correlating failures to upstream/downstream systems can be more difficult without orchestrated metadata. 

When to Consider This:

When you need deterministic performance, such as in ML inference, high-frequency trading, or streaming telemetry ingestion. 

Architecture Pattern #2: Specialized, Customer‑Specific Components  

Use Case: Multi-tenant SaaS platforms or platforms with dedicated microservices per enterprise customer or vertical 

Why Docker Makes Sense: 

• Standalone containers allow per-customer isolation without polluting the core K8s control plane. 
• Services can be versioned, tuned, and deployed independently, which is ideal when customers have differing SLAs, data residency needs, or workload profiles. 
• Enables progressive feature rollout, compliance isolation (e.g., GDPR), or regional data processing without touching the shared cluster. 
• Example: Deploying a custom analytics pipeline for an EU customer with stricter data locality rules. 

Tradeoffs: 

• More surface area to monitor and secure. 
• Requires tooling to manage container sprawl and avoid operational drift. 

When to Consider This:

When serving enterprise customers with tailored requirements or isolating sensitive data flows.  

Architecture Pattern #3: Asynchronous Messaging Backbone 

Use Case:

Internal event buses, Kafka sidecars, queue processors, or background workers handling fan-out/fan-in traffic patterns 

Why Docker Makes Sense: 

• These components often require tight control over retry logic, dead-letter queues, and message deduplication strategies. 
• Docker allows dedicated tuning of each message processor (e.g., CPU throttling, JVM heap tuning) without being subject to shared limits imposed by K8s resource quotas. 
• Services can scale independently of the producers and consumers managed in K8s. 
• Supports use cases like Command Query Responsibility Segregation (CQRS), or event sourcing, where replay behavior and ordering matter.  
• Example: Kafka sidecar processors that transform and route event data before entering the service mesh. 

Tradeoffs: 

• Failures can ripple across multiple services but be misattributed.  
• Requires deeper visibility into both the message layer and consuming systems. 

When to Consider This:

When messaging is your system’s backbone and needs dedicated, configurable components.  

Architecture Pattern #4: CI/CD, Build & Test Workloads 

Use Case:

Ephemeral build/test containers in CI pipelines, sandbox environments for pre-merge validation 

Why Docker Makes Sense: 

• CI systems like GitHub Actions, CircleCI, and GitLab are built around standalone container workloads. 
• Docker ensures parity between dev and test environments, reducing false positives from environment drift. 
• Local development with Docker Compose can mirror production service graphs with mocking or synthetic traffic. 
• Example: nightly regression tests that spin up mocked API services using Docker Compose, inject synthetic latency, and validate SLA adherence before promoting a build to staging. 

Tradeoffs: 

• Pre-prod performance issues may not emerge until you shift into the orchestrated production environment unless your test setup includes sufficient load or concurrency. 

When to Consider This:

If you need fast, reliable CI loops, or are troubleshooting performance regressions before services hit production.  

Why Performance Management Gets Hard in These Hybrid Architectures 

In a system composed of Kubernetes clusters, standalone Docker containers, and managed services like MongoDB or Kafka, performance issues don’t respect boundaries. 

Symptoms Emerge Far From Their Root Cause:  

• A degraded Kafka sidecar causes timeouts in upstream K8s services.  
• A CPU-hungry Docker-based customer module introduces latency that looks like a DB issue. 
• A slow edge container obscures the real issue in a downstream managed Redis instance. 

These Environments Break Traditional Monitoring Assumptions: 

• No single telemetry system captures the full execution path.  
• Logs, metrics, and traces must be stitched manually across platforms.  
• Minute-level delay in diagnosis equals dollars lost. 

You need to know not just what failed, but why. And you need it now. 

eBPF-Based Auto-Instrumentation: Zero-Overhead Insight 

eBPF provides kernel-level insight into what containers are doing, without needing in-app instrumentation, SDKs, or manual agents. 

In standalone Docker environments, Causely uses eBPF to:  

• Capture syscall activity, I/O latency, CPU/memory contention, and network issues directly from the host.  
• Observe containers even if they’re running outside Kubernetes or disconnected from the service mesh.  
• Collect minimal, high-value telemetry without generating a flood of low-signal noise. 

And because eBPF instrumentation is automatic,  

• There’s no overhead to developers 
• Visibility is always on 
• And you can trace performance across heterogeneous systems without writing a line of config. 

Causely also enables end-to-end tracing across Kubernetes, Docker, and managed environments—stitching together execution flows in real time and letting you see how a single failure propagates. 

Causely Makes Standalone Docker First-Class 

While eBPF gives us the raw signals, Causely’s causal reasoning engine makes them useful. 

• It applies a growing library of causal models which capture failure patterns across services, queues, and infrastructure. 
• It builds a real-time causal graph from observed behavior. 
• It reasons probabilistically to pinpoint the root cause—even across multiple failure domains. 

We Treat Docker Containers as Full Participants in the Production Graph

With Causely, you can:  

• Automatically discover Docker-hosted services and map their upstream/downstream relationships.  
• Trace symptoms across Kubernetes, Docker, and managed services without needing to standardize instrumentation.  
• Get clear, prioritized, explainable root causes in real time. 

You don’t have to fear hybrid complexity; you just need a system that understands it. 

Ready to Make Standalone Docker Part of Your Reliability Strategy, Not a Blind Spot? 

Causely installs in minutes. With our automated instrumentation and causal analysis, you’ll spend less time debugging and more time building. 

Learn more about how to add stand-alone Docker to your Causely deployment.  

When a message queue starts to back up, every second counts. One slow consumer can stall a Kafka topic, delay downstream services, and turn healthy SLOs into a wall of red.

The hard part? Your observability tools only show the symptoms: rising lag, service timeouts, and growing error rates. With dozens of potential culprits — from a code change on one service to a misconfigured broker or even a hidden infrastructure issue — finding the exact cause can feel like guesswork.

Causely takes the uncertainty out of the equation. By combining deep domain knowledge with your Kafka topology and real-time telemetry, Causely identifies the precise reason behind the lag — whether it’s inefficient garbage collection, resource contention, or broker congestion — and shows you exactly how it’s impacting your services. No endless alert-chasing. No service-by-service elimination. Just clear, evidence-backed answers so your team can act fast and keep the backlog from turning into a crisis.

Watch the video to see how Causely turns “Lag High” chaos into confident, informed action in seconds.

We'll explore five key techniques that power automatic instrumentation: monkey patching, bytecode instrumentation, compile-time instrumentation, eBPF, and language runtime APIs. Each technique leverages the unique characteristics of different programming languages and runtime environments to add observability without code changes.

What is Automatic Instrumentation?

According to the OpenTelemetry glossary, automatic instrumentation refers to “telemetry collection methods that do not require the end-user to modify application’s source code. Methods vary by programming language, and examples include bytecode injection or monkey patching.”

It’s worth noting that “automatic instrumentation” is often used to describe two related but distinct concepts. In the definition above and in this blog post, it refers to the specific techniques (like bytecode injection or monkey patching) that can be used to enable observability without code changes. However, when people use "automatic instrumentation" in conversations, they often mean complete zero-code solutions like the OpenTelemetry Java agent.

The distinction is important: there's actually a three-layer hierarchy here. At the bottom are the automatic instrumentation techniques (bytecode injection, monkey patching, etc.) that we explore in this blog post. These techniques are used by instrumentation libraries that target specific frameworks, for example, libraries that instrument Spring and Spring Boot, Express.js, Laravel, or other popular frameworks. Finally, complete solutions like the OpenTelemetry Java agent bundle these instrumentation libraries together and add all the boilerplate configuration for exporters, samplers, and other building blocks.

There are ongoing debates in the observability community about the right terminology, and this blog post won’t attempt to resolve those discussions.

Also note that what appears "automatic" to one person might be "manual" to another: if a library developer integrates the OpenTelemetry API into their code, the users of that library will get traces, logs, and metrics from that library “automatically” when they add the OpenTelemetry SDK to their application.

Want to try the techniques yourself?

This blog post contains small code snippets to illustrate the concepts. A full working examples can be found in a lab repository at https://github.com/causely-oss/automatic-instrumentation-lab where you can try them out yourself.

Before we explore these techniques, it’s important to note that you should not build your own automatic instrumentation from scratch, especially not using this blog post as a blueprint. The examples here are simplified for educational purposes and skip many complex details that you would encounter in real-world implementations. There are established tools and mechanisms available that handle much of the complexity and edge cases you would face when building instrumentation from the ground up. If you’re interested in diving deeper into this field, the best approach is to contribute to existing projects like OpenTelemetry, where you can learn from experienced maintainers and work with production-ready code.

Automatic Instrumentation Techniques

Now let’s explore how these techniques work under the hood.

Monkey Patching: Runtime Function Replacement

Monkey patching is perhaps the most straightforward automatic instrumentation technique, commonly used in dynamic languages like JavaScript, Python, and Ruby. The concept is simple: at runtime, we replace existing functions with instrumented versions that inject telemetry before and after calling the original function.

Here's how this works in Node.js:

const originalFunction = exports.functionName;

function instrumentedFunction(...args) {  
  const startTime = process.hrtime.bigint();  
  const result = originalFunction.apply(this, args); 
  const duration = process.hrtime.bigint() - startTime;   
  console.log(`functionName(${args[0]}) took ${duration} nanoseconds`);  
  return result;
}

exports.functionName = instrumentedFunction;

The require-in-the-middle library allows us to perform this replacement at module load time, intercepting the module loading process to modify the exported functions before they’re used by the application:

const hook = require("require-in-the-middle");
hook(["moduleName"], (exports, name, basedir) => {
  const functionName = exports.fibonacci; 
  ...
  exports.functionName = instrumentedFunction;
  return exports;
});

However, monkey patching has limitations. It can't instrument code that's already been compiled to machine code, and it may not work with functions that are called before the instrumentation is loaded. Additionally, the overhead of function wrapping can be significant for performance-critical applications. Monkey patching is also brittle when the implementation of the instrumented code changes significantly, as the instrumentation code needs to be updated to match the new interface.

To try this out yourself, take a look at the Node.js example from the lab.

If you’d like to see actively used implementations of monkey patching, you can take a look into the instrumentation libraries provided by OpenTelemetry for JavaScript or Python.

Bytecode Instrumentation: Modifying the Virtual Machine

For languages that run on virtual machines, bytecode instrumentation offers a powerful approach. This technique works by modifying the compiled bytecode as it’s loaded by the virtual machine, allowing us to inject code at the instruction level.

Java’s Instrumentation API provides the foundation for this approach. When a Java agent is specified with the -javaagent flag, the JVM calls the agent’s premain method before the main application starts. This gives us the opportunity to register a class transformer that can modify any class as it’s loaded.

public static void premain(String args, Instrumentation inst) {
    new AgentBuilder.Default()
        .type(ElementMatchers.nameStartsWith("com.example.TargetApp"))
        .transform((builder, typeDescription, classLoader, module, protectionDomain) ->
            builder.method(ElementMatchers.named("targetMethod"))
                   .intercept(MethodDelegation.to(MethodInterceptor.class))
        ).installOn(inst);
}

The interceptor then wraps the original method call with timing logic:

@RuntimeType
public static Object intercept(@Origin String methodName,
                            @AllArguments Object[] args,
                            @SuperCall Callable<?> callable) throws Exception {
    long startTime = System.nanoTime();
    Object result = callable.call();
    long duration = System.nanoTime() - startTime;
   
    System.out.printf("targetMethod(%s) took %d ns%n", args[0], duration);
    return result;
}

Bytecode instrumentation is particularly powerful because it works at the JVM level, making it language-agnostic within the JVM ecosystem. It can instrument Java, Kotlin, Scala, and other JVM languages without modification.

The main advantage of bytecode instrumentation is its comprehensive coverage—it can instrument any code that runs on the JVM, including code loaded dynamically or from external sources. However, it comes with some overhead due to the bytecode transformation process.

In real implementations, ByteBuddy is the go-to library for bytecode instrumentation in Java, providing a powerful and flexible API for creating Java agents. It abstracts away much of the complexity of bytecode manipulation and provides a clean, type-safe way to define instrumentation rules.

To try this out yourself, take a look at the Java example from the lab.

If you’d like to see actively used implementations of byte code instrumentation, you can take a look into the instrumentation libraries provided by OpenTelemetry for Java or .NET.

Compile-Time Instrumentation: Baking Observability into the Binary

For statically compiled languages like Go, compile-time instrumentation offers a different approach. Instead of modifying code at runtime, we transform the source code during the build process using Abstract Syntax Tree (AST) manipulation.

The process involves parsing the source code into an AST, modifying the tree to add instrumentation code, and then generating the modified source code before compilation. This approach ensures that the instrumentation is baked into the final binary, providing zero runtime overhead for the instrumentation mechanism itself.

func instrumentFunction() {
    fset := token.NewFileSet()
    file, err := parser.ParseFile(fset, "app/target.go", nil, parser.ParseComments)
   
    // Find the target function and add timing logic
    ast.Inspect(file, func(n ast.Node) bool {
        if fn, ok := n.(*ast.FuncDecl); ok && fn.Name.Name == "targetFunction" {
            // Add defer statement for timing
            deferStmt := &ast.DeferStmt{
                Call: &ast.CallExpr{
                    Fun: &ast.CallExpr{
                        Fun: &ast.Ident{Name: "trace_targetFunction"},
                    },
                },
            }
            fn.Body.List = append([]ast.Stmt{deferStmt}, fn.Body.List...)
        }
        return true
    })
   
    // Write the modified file back
    printer.Fprint(f, fset, file)
}

Compile-time instrumentation has several advantages. It provides zero runtime overhead for the instrumentation mechanism, and the resulting binary contains all the code it needs. This approach works well with compiled languages and can be integrated into existing build processes.

That said, it does come with trade-offs. It requires access to the source code and build system, which makes it impractical for instrumenting third-party applications or libraries. It also demands more sophisticated tooling to manipulate the abstract syntax tree (AST) correctly and consistently adding complexity to the build pipeline and potentially requiring changes to your CI/CD workflows.

To try this out yourself, take a look at the Go compile-time example from the lab.

If you’d like to see actively used implementations of compile-time instrumentation, you can take a look into the OpenTelemetry Go Compile Instrumentation project.

eBPF Instrumentation: Kernel-Level Observability

eBPF (Extended Berkeley Packet Filter) represents a fundamentally different approach to automatic instrumentation. Instead of modifying application code or bytecode, eBPF works at the kernel level, attaching probes to function entry and exit points in the running application.

eBPF programs are small, safe programs that run in the kernel and can observe system calls, function calls, and other events. For automatic instrumentation, we use uprobes (user-space probes) to attach to specific functions in our application.

#!/usr/bin/env bpftrace

uprobe:/app/fibonacci:main.fibonacci
{
    @start[tid] = nsecs;
}

uretprobe:/app/fibonacci:main.fibonacci /@start[tid]/
{
    $delta = nsecs - @start[tid];
    printf("fibonacci() duration: %d ns\n", $delta);
    delete(@start[tid]);
}

This bpftrace script attaches a probe to the function in our application. When the function is called, it records the start time. When the function returns, it calculates the duration and prints the result.

eBPF instrumentation is language-agnostic and works with any language running on Linux. It provides deep system-level observability without requiring any modifications to the application code or build process. The overhead is minimal since the instrumentation runs in the kernel.

However, eBPF instrumentation has some limitations. It requires Linux and root privileges to run, making it less suitable for containerized environments or applications that can’t run with elevated permissions.

For real-world use cases, bpftrace is just one of many eBPF tools available. While it’s excellent for learning and prototyping, production environments typically use more sophisticated frameworks like BCC (BPF Compiler Collection) or libbpf, which provide better performance, more features, and stronger safety guarantees.

To try this out yourself, take a look at the Go eBPF example from the lab.

If you’d like to see actively used implementations of compile-time instrumentation, you can take a look into the OpenTelemetry eBPF Instrumentation project (short “OBI”), which is the outcome of the donation of Belay by Grafana.

Language Runtime APIs: Native Instrumentation Support

Some languages provide built-in APIs for instrumentation, offering a more integrated approach. PHP’s Observer API, introduced in PHP 8.0, is a prime example of this approach.

The Observer API allows C extensions to hook into the PHP engine’s execution flow at the Zend engine level. This provides deep visibility into PHP application behavior without requiring code modifications.

static void observer_begin(zend_execute_data *execute_data) {
    if (execute_data->func && execute_data->func->common.function_name) {
        const char *function_name = ZSTR_VAL(execute_data->func->common.function_name);
        if (strcmp(function_name, "fib") == 0) {
            start_time = clock();
        }
    }
}

static void observer_end(zend_execute_data *execute_data, zval *retval) {
    if (execute_data->func && execute_data->func->common.function_name) {
        const char *function_name = ZSTR_VAL(execute_data->func->common.function_name);
        if (strcmp(function_name, "fib") == 0) {
            clock_t end_time = clock();
            double duration = (double)(end_time - start_time) / CLOCKS_PER_SEC * 1000;
            php_printf("Function %s() took %.2f ms\n", function_name, duration);
        }
    }
}

The Observer API provides a clean, supported way to add instrumentation to PHP applications. It operates at the language runtime level, similar to how other languages implement their instrumentation APIs. This approach is efficient and well-integrated with the language ecosystem.

However, it requires writing C extensions, which adds complexity and makes it less accessible to developers who aren’t familiar with C or PHP’s internal APIs. It’s also specific to PHP, so the knowledge doesn’t transfer to other languages.

To try this out yourself, take a look at the PHP Observer API example from the lab.

If you’d like to see actively used implementations of byte code instrumentation, you can take a look into the instrumentation libraries provided by OpenTelemetry for PHP.

A Note on Context Propagation

While we've covered the core techniques of automatic instrumentation, there's an important aspect we haven't discussed: context propagation. This involves injecting trace context information (trace IDs, span IDs) into HTTP headers, message metadata, and other communication channels to enable distributed tracing across service boundaries.

Unlike the purely observational techniques we've explored, context propagation actively modifies your application's behavior by altering data transmitted across service boundaries. This introduces additional complexity that deserves its own dedicated blog post.

Automatic Instrumentation as telemetry quality assurance

At Causely, we rely on automatic instrumentation to ensure consistent, high-quality telemetry, even when customers haven't instrumented their code manually. Our agent comes with automatic instrumentation powered by Grafana Beyla (now also known as OpenTelemetry eBPF Instrumentation), leveraging eBPF and OpenTelemetry out-of-the-box to ensure fast time-to-value and dependable insights from day one.

Conclusion

We've explored the core techniques behind automatic instrumentation, from monkey patching to bytecode instrumentation to eBPF probes. Each approach leverages the unique characteristics of different programming languages and runtime environments.

These techniques power production observability tools like OpenTelemetry, enabling developers to quickly add telemetry without modifying source code. The most successful observability strategies combine automatic and manual instrumentation: automatic instrumentation provides broad coverage for common patterns, while manual instrumentation captures business-specific metrics.

If you'd like to try out these techniques yourself, you can use the Automatic Instrumentation Lab.

If you're interested in contributing to these technologies, consider getting involved with OpenTelemetry's various Special Interest Groups (SIGs).

When dozens of teams are deploying code across services and environments, even a small change can quietly spiral into system-wide latency, retries, and customer pain.

And when things go sideways, your team is left scrambling: 

Was it the payments API?
A config change in the cluster?
A new version of Postgres? 

In most tools, you’re chasing alerts across dashboards trying to guess your way to the root cause. That’s why we built Causely — to surface the answer before the war room even begins.

In this short video, we show how Causely pinpoints the exact code change that triggered cascading performance issues — without requiring you to sift through logs or build custom dashboards. Our Causal Reasoning Engine continuously maps relationships between telemetry, deploys, and system behavior to deliver real-time root cause insight. So when your team asks “what changed?”, Causely already knows — and shows you what to fix, what was impacted, and who needs to take action.

In my recent talk at OpenTelemetry Community Day, “The Signal in the Storm: Practical Strategies for Managing Telemetry Overload,” I laid out a different path forward, one focused not on volume, but on meaning. 

More telemetry doesn’t guarantee more understanding. In many cases, it gives you the illusion of control while silently eroding your ability to reason about the system. That illusion becomes expensive, especially when telemetry pipelines are optimized for ingestion, not insight. 

Observability Needs a Better Model 

Traditional observability relies on emitting and aggregating raw signals (spans, logs, and metrics), then querying across that pile post-hoc. That model assumes the data will be useful after something goes wrong. But today’s distributed, dynamic, multi-tenant AI-driven systems don’t give you that luxury. The cost of collecting all raw signals without semantics and without context is too high. 

We need a shift: from streams of telemetry to structured, semantic representations of how systems behave. That starts by modeling the actual components of the system (entities) and the relationships between them. Not as metadata bolted onto spans, but as first-class signals. This is the work being advanced by the OpenTelemetry Entities SIG, and it's central to how we think about observability at Causely. 

What It Looks Like in Practice 

At Causely, we’ve been applying these ideas in production. We observe system behavior, and instead of centralizing all that raw output, we extract anomalies in the context of semantic entities and relationship graphs at the edge. This gives us fine-grained insight without overwhelming the pipeline. 

The impact is real: 

• Less data, more clarity: structured signals replace noisy aggregates 
• Better performance and cost-efficiency: telemetry becomes lean and targeted 
• Stronger privacy: raw, user-level data never needs to leave the cluster 
• Faster debugging: understanding is built-in, not reverse-engineered 

The goal isn’t to eliminate data; it’s to collect with intent. To make the system itself understandable, not just observable. 

Watch the Talk 

In the session, I go deep into what this looks like technically, including: 

• How teams are adapting their telemetry strategies to manage scale and cost 
• The role of entities, ontologies, and semantic modeling in modern observability 
• Why centralized data lakes aren’t a sustainable long-term answer 
• What we’ve learned building systems that can reason about their own behavior 

If you’ve felt the limits of traditional observability, and you're looking for a more scalable, reliable, and thoughtful path forward, I hope you’ll check it out. 

See the slides, or watch the recording:

You did the right things. 

• Your microservices are fully instrumented. 
• You’ve got distributed tracing and a modern observability stack.  
• You even built custom dashboards. 

But during a major production incident, it still took hours to figure out what was wrong. 

Sound familiar? 

In our recent webinar, 'Rethinking Reliability for Distributed Systems,' Causely co-founder Endre Sara shared a story we hear far too often: a large-scale customer, running mature microservices in Kubernetes with full observability coverage, still struggles to understand what’s broken during a high-stakes business event. 

Why microservices break differently 

Distributed systems aren't just complex, they're dynamic. Services spin up and down. Async data flows hide causal relationships. Teams own different pieces. Dashboards fill up with symptoms, not answers. 

That's what happened to a large enterprise team Endre worked with. They had: 

• Mature Kubernetes operations 
• Kafka for async comms 
• Comprehensive tracing + telemetry 
• A seasoned SRE team 

And still, they couldn't find the root cause of a high-stakes incident. 

The problem wasn't a lack of data. It was a lack of causal reasoning.  

Watch the recording: Rethinking Reliability for Distributed Systems 

In this session, Endre walks through: 

• Why microservices environments overwhelm traditional observability 
• How causal reasoning changes incident response 
• What teams can do to move from firefighting to foresight 

Whether you’re drowning in alerts or struggling to explain why something broke, this talk offers a clear new perspective and a path forward.  

When something breaks in production, most tools make you start with a question: What’s going on? But with Ask Causely, you start with the answer.

Powered by our real-time Causal Reasoning Engine, Causely continuously analyzes your telemetry and system topology to surface the root cause the moment an issue begins — no prompt required. Unlike other AI assistants that reactively dig through logs or metrics after you ask, Causely already knows what’s wrong, why it’s happening, and what’s likely to break next.

In this short demo, we show how Ask Causely shifts incident response from a fire drill to a focused, high-context workflow. Whether you're on-call, in a war room, or mid-debugging, Ask Causely can tell you who owns the failing service, what downstream systems are impacted, and what to do next — all from Slack, Teams, or your browser. It’s observability that works like a teammate, not just a tool.

Watch the video to see how your team can go from alerts to action in seconds.

Let’s Not Rename Observability — Let’s Make It Work 

Every few months, a new term pops up: understandability, explainability, controllability. And sure, we all want better systems. But do we really need new language? Or do we need better outcomes? 

As I said in the panel: six people will have twelve opinions on what observability means. But the real point is this: users don’t care what we call it! They want to catch root causes in production early and minimize the business impact during an incident. And that requires systems that deliver causal, actionable insights that allow teams to return their software into a healthy state quickly. 

Renaming observability and packaging it as something else is not conducive to getting to the true outcome everyone wants – they are just words without the proper action and change to back it up. We saw this before, that “monitoring” was replaced with “observability” without no actual change. 

Observability should support the full software lifecycle, from design to incident to fix. If it’s not helping you move faster and safer, then it’s just more dashboards. And no one wants that. Let’s not get stuck in terminology and instead get back to building systems that help us move. 

Value Over Volume 

One of the loudest themes from the roundtable: cost. Observability spend is skyrocketing, while the value people get is… questionable. 

We are stuck in a cycle of collecting everything, just in case! But volume does not equal value. More logs, more traces, more storage doesn’t solve problems, it mostly just adds noise. 

Any vendor claiming to innovate in observability has to answer this question: how do you shift the focus from collecting more data to instead only delivering useful insights? 

Causely’s answer: we do mediation at the cluster (i.e., edge-based processing), and only send distilled insights to the cloud . Our system doesn’t aim to collect all the data. It aims to understand what’s wrong, fast. That means we’re not just watching metrics, we’re diagnosing causal chains. That’s how we make observability affordable, and more importantly useful. 

From Optional to Invisible: The Future of OpenTelemetry 

During the roundtable I said that OpenTelemetry will have won when people use it without realizing it. When the libraries, frameworks and programming languages that we use every day come with an out of the box integration. Developers will use it like any other core feature of their language, like if-statements, variables and comments. 

That said, this vision is more of a stretch goal than a milestone on the horizon. In many ways, OpenTelemetry has already “won.” Making code observable, whether through OpenTelemetry or another system, is no longer optional, it is expected. Observability has become a baseline capability, and OpenTelemetry helped set that standard. 

Smart Automation, Not AI Hype 

A few years ago, AIOps promised to make sense of our systems with artificial intelligence and mostly delivered confusion. The hype faded, the promises didn’t hold up, and most teams were left with “smart” alert suppressors or dashboards that looked impressive but didn’t help when it counted. 

Today’s AI wave is louder: LLMs that summarize incidents, tools that promise auto-remediation with natural language, anomaly detectors wrapped in glossy UI. But most of them suffer from the same core flaw: LLMs are general-purpose tools, they excel at pattern recognition, but lack deep real-time causal reasoning needed to adopt to novel, dynamic environments. 

At Causely, we’re not building a chatbot. We’re not doing log clustering or time-series correlation. We’re building a causal reasoning system. 

Our system encodes known failure modes as structured models. That means it can infer root causes even when the triggering signal isn’t directly observable. It doesn’t just suppress noise; it explains it. It doesn’t just summarize symptoms; it traces causality. 

This isn’t black-box “AI.” It’s smart, explainable automation rooted in mathematics — built to help engineers understand why things happen and what to do next. 

And the goal isn’t to replace humans! It’s to get them out of the loop for the boring stuff: the CPU spike, the misconfigured downstream service, the memory leak that shows up every Tuesday. These aren’t mysteries. They’re patterns. And they can be handled automatically.  

That way engineers can spend time on doing what they love, building.   

Closing Thoughts 

The roundtable showed there’s still strong alignment across the industry: observability remains essential, but it needs to deliver more than just data. We need clearer value, smarter automation, and systems that help us move faster, not just monitor more. 

Thanks to Adam, Charity, Paige, Shahar, and Avi for a thoughtful and honest discussion. It’s good to see real progress, and even better to debate what is coming next. 

Grafana gives teams the power to visualize everything - but on Day 0, when your dashboards are live and alerts start firing, what your team really needs is clarity. That’s why we built the new Causely plugin for Grafana. In just minutes, Causely connects to your telemetry sources and begins surfacing the root cause of performance degradations - right inside your existing dashboards. No code changes. No sidecars. Just answers.

In this video, you’ll see how Causely helps teams cut through the noise of modern microservice environments by pinpointing why systems are breaking, not just where. Our Causal Reasoning Engine builds a real-time cause-and-effect map of your services - so instead of drowning in symptoms, your team is focused on what matters most. Watch the video to see how Causely helps you deliver real insight from Day 0 - and drive better reliability from Day 1.

But here is the problem: none of these are actually solving the Root Cause Analysis problem. Alarm suppression is NOT Root Cause Analysis.  

For teams operating modern, distributed systems - microservices, data pipelines, container orchestration, multi-cloud dependencies - these limitations aren’t minor. They make it impossible to reason clearly about why performance degradations and failures happen, and how to prevent them. 
 
For example, at a financial services company, a cascading Kafka issue was misdiagnosed for hours as a frontend memory spike. The result? Missed SLOs and three teams paged on a Saturday. 

What These Tools Miss 

Today's “RCA” tools suffer from several critical limitations: 

• They can’t infer the root cause unless it’s already present in the observed alarms – i.e., the output must be in the input. 
• They don't explain why the symptoms occurred – they just hide redundant signals. 
• They require perfect, complete signals to function reliability.  
• They often produce misleading or spurious outputs. 

At Causely, we’re not building a log aggregator, a correlation engine, alarm suppression, nor a chatbot. We are building a causal reasoning system. To understand how this approach is different and why it matters, we first need to re-establish what “root cause analysis” is meant to solve.  

What is the Root Cause Analysis Problem? 

 In managed cloud environments, the RCA problem requires the identification of the most likely cause of observed symptoms, based on a structured understanding of the environment and causal interdependencies between services. Put simply: it’s about using what you know about your system to explain what you see—not just matching alerts to patterns, but reasoning through cause and effect. 

What Most RCA Tools Actually Do 

Most tools that claim to perform RCA fall into one of three categories: 

• Postmortem Narratives: These are after-the-fact writeups, often shaped more by human bias than data, constructed to provide a retrospective of what went wrong.  
• Correlation Engines: These systems surface anomalies and related signals during incidents but confuse correlation with causation. They provide visibility into what things happened around the same time, but they don’t know why.   
• LLM-Powered Assistants: These interfaces can produce plausible-sounding explanations by summarizing the data they have access to, but they often generate spurious or unverifiable answers.  

While all three of these approaches can be useful in some ways, none of them utilize structured causal knowledge and reasoning to solve the RCA problem. That’s the missing piece. 

What Actual Causal Analysis Looks Like 

To be clear, causal analysis is not about finding "what’s weird." It’s about inferring what is the root cause. At Causely, our reasoning platform is built on three foundational principles: 

1. Root Causes Are Explicitly Defined 

A root cause is an underlying issue that results in multiple degradations and disruptions in the managed environment. Formally, a root cause is defined by causes and effects, where the cause is the underlying issue (e.g., a DB experiencing inefficient locking), and the effects are the disruptions it creates in the environment (e.g., degraded service response times).  

Our system monitors your environment and identifies which anomalous behaviors (such as high error rates, slow responses, or service crashes) are symptoms, using metrics gathered from telemetry and observability tools. 

Causely represents each root cause as a closure - a signature uniquely defined by a specific set of expected symptoms. Each root cause and its closure are automatically generated from causal knowledge and the discovered topology. Utilizing Causal Bayesian Networks, Causely can effectively and accurately infer root causes by reasoning over the relationships between the root causes and symptoms, rather than relying on simple mapping or correlations.  

Causely constructs precise causal graphs that show not just what broke, but why. This allows issues to be resolved faster and more efficiently. 

2. Causal Reasoning Is Bayesian 

Causely infers the root cause based on observed symptoms, even when the observations are incomplete or noisy using causal Bayesian networks. Why Bayesian networks? Because you can’t assume perfect signals and should always assume that the observed symptoms will be noisy: 

• Symptoms aren’t always fully observable. 
• Real systems behave unpredictably. 
• And we need to reason probabilistically under uncertainty. 

Causely uses Bayesian causal graphs to represent possible root causes and their effects, assigning probabilities to capture uncertainty in real-world systems. The prior probabilities in our models are defined by experts with decades of experience in distributed systems, microservices, data pipelines, and container orchestration. During active incidents, Causely uses observed symptoms to calculate posterior probabilities over possible root causes. These probabilities are then used to identify the most likely root cause, enabling teams to respond quickly and decisively.  

3. Causal Graphs Are Customer-Specific 

Our causal models are not static or generalized templates. Causely automatically constructs topologically grounded causal graphs that are specific to the customer’s environment and dynamically adapt as environment services and dependencies change. The causal graphs map environment-specific causal dependencies and represent how root causes propagate and manifest themselves across services.  

Benefits of the Causely Approach 

Causely conducts root cause analysis in a principled way to ensure: 

High Precision 

• Root causes are curated and relevant to managed cloud environments. 
• Causal structures mirror actual deployment topologies. 
• False positives are mitigated by never guessing outside the defined causal space. 

Generalizability & Rapid Deployment 

• Bayesian methods identify root causes even with sparse observations. 
• Topologies are grounded in causal graphs to ensure root cause accuracy. 
• Dynamic topology updates with real-time telemetry data let us adapt to each customer’s specific patterns. 

Predictive Power 

• Causal graphs are constructed a priori to ensure the reasoning engine can predict which downstream symptoms and disruptions may emerge when monitoring active root causes.  
• Causely’s causal graphs enable corrective interventions before all root cause symptoms manifest. 
• In addition to being used for real-time operations, the causal reasoning system can also be used to anticipate future failures and help prevent them. Causely identifies critical services and their failure risks based on causal pathways.  

What About LLMs? 

Strengths of LLM-Based RCA 

LLM-based approaches to RCA are gaining traction. They offer unique strengths: 

• Natural Language Interface: You can ask follow-up questions in plain English and get responses without needing to write a query or scan dashboards. 
• Trained on Broad Knowledge: LLMs draw from a vast, pre-trained corpus. Most LLM’s knowledge spans Stack Overflow, GitHub issues, and decades of online technical discourse. This breadth allows them to generate plausible explanations across diverse technologies. 
• Rapid Response: Most LLMs will respond within seconds and can complete tedious tasks quickly.  

But these benefits come with real tradeoffs. 

Limitations of LLM-Based RCA 

• Spurious Causes: LLMs often draw conclusions that appear coherent but are factually incorrect and contextually invalid - due to hallucinations, logical inconsistencies, and insufficient understanding of the managed environment  
• Unprincipled Reasoning: LLMs mimic the language of reasoning without performing structured inference. Research shows that LLMs suffer from content effects, where prior biases interfere with logical reasoning and result in reasoning fallacies.  
• Causal Identification Failures: Research shows LLMs systematically struggle with causal prediction, especially in dynamic settings due to the causal sufficiency problem. 

While LLMs are quickly gaining traction, they remain limited in accurately identifying root causes and reasoning in complex environments. Causely combines the best of both worlds by using LLMs responsibly to support natural language conversations, while grounding root cause analysis in structured causal models to ensure precision and accuracy. 

RCA Isn’t a Buzzword. It’s a well-defined problem. 

If you are calling something a “root cause,” you should be able to show how it caused the observed effects - not just that it co-occurred or provide an explanation that sounds plausible. 

At Causely we solve the RCA problem by being structured, explainable, and rooted in decades of engineering expertise. We don’t guess. We infer. We don’t react. We reason. 

Let’s stop diluting RCA into dashboards and chatbots. Let’s build systems that actually understand why things break and how to prevent future failures. 
 
If you’re tired of dashboards that guess and chatbots that bluff, it’s time to reason instead. Start your journey with Causely today. 

👉Access our sandbox and free trial environment 

👉Reach out for a customized demo 

When it comes to discussing how AI will impact the future of software development and IT management, most vendors hold on to the objective that it’s important to keep the human in the loop. They are afraid to publicly acknowledge what’s always been true — that machines are better than humans at many things. Meanwhile, that list of things continues to grow.

When it comes to observability and IT operations, our goal should be to get humans out of the loop as much as possible. In a world run by software, where SLAs guarantee 99.999% uptime, even one incident in a month — which requires human intervention — is enough downtime to violate your commitments to your customers.

With the rise of distributed, complex applications exacerbated by more and more AI-generated code, it is difficult for traditional IT operations to keep up across troubleshooting, provisioning infrastructure, performance monitoring, security, etc. Developers now deploy code 70% faster using AI, according to the latest DORA report — you can’t have enough SREs and security personnel to keep up with this kind of activity. The companies that succeed will leverage technology to get humans out of the way as quickly as possible. 

Failed Promises of AIOps 

It is worth stating the obvious that there’s a difference between where things are today and where they are heading in the future. As someone who has been in IT management for decades, I have seen all the buzzwords as well as both marketing nonsense and genuine progress.  

The term ‘AIOps’ was coined by Gartner in 2016 to address the increasing complexity and data volume in IT environments, aiming to automate processes such as event correlation, anomaly detection and causality determination. But as it turned out, many of the vendors who claimed to offer AIOps were nothing more than empty shells when you looked under the hood.  

It was essentially the same AI/ML that had been used for a decade beforehand, branded in a new way and making outsized claims that didn’t map to reality. We saw tech giants making acquisitions of point solutions, and then bundling them under the ‘AIOps’ category because it was trendy to do so and they had nowhere else better to put them. 

But I would argue that many of the companies that continued to trumpet these claims without actually delivering on the promises eventually suffered a hit to their reputation. We still aren’t at that stage where machines are acting autonomously to solve complex problems across performance, reliability and security. But they are doing more than ever before, and there’s every reason to believe we will reach that goal in the future. 

The Future of Autonomous Reliability 

Autonomous reliability platforms of the future will not only surface actionable insights, but they will also be competent enough to make autonomous decisions without human intervention. And why should this be impossible? If planes can mostly fly themselves, why can’t IT management become autonomous? 

The decades-long trend of collecting more and more data in the name of observability isn’t rendering autonomous service reliability. Nor is it feeding that data into a machine and magically hoping for answers. Machines trained on yesterday’s patterns might explain what went wrong in the past, but they can’t make the real-time decisions required to keep systems running, especially in dynamic, cloud-native environments. 

What we need is a paradigm shift from data collection to causal understanding. By capturing causal knowledge as part of an ontology, we can reason about cause-and-effect in complex, ever-changing systems — the key is to move beyond reactive alerting into autonomous reliability. Unlike the growing industry trend of offloading alerts to LLMs, causal reasoning gives us the context and clarity needed to take real control.

But even the clearest dashboard doesn’t give you what you need when something breaks. 

Observability allows you to see what is happening, but what about understanding why something is happening? With Causely, you can see the why behind what’s happening without having to leave your Grafana interface. 

That’s where Causely comes in. Our new plugin embeds root cause intelligence directly into your Grafana dashboards — so you can seamlessly shift from awareness to understanding and from incident response to proactive improvement, all in a single view. 

When Observability Stops at What 

Modern engineering teams operate in high-stakes, high-complexity environments:   

• Dozens (or hundreds) of microservices   
• CI/CD pipelines constantly push new code   
• SLOs that leave no margin for delay   

The result? Even with powerful observability tools like Grafana, most teams still end up in incident management mode: war rooms, escalations, and Slack threads at 2 a.m.   

Instead of building, you’re firefighting. Instead of improving systems, you're guessing at symptoms. 

Grafana tells you what happened.   

Causely shows you why — and what to do next. 

 Why Grafana + Causely Is Different 

This integration was built to help teams detect and truly understand problems—in context and real time. 

By embedding Causely’s reasoning engine inside Grafana, teams can now: 

• See the root causes for all active anomalies across your environment, not just alerts 
• Identify the highest-priority root cause impacting your services right now 
• Spot the root causes putting your SLOs at risk — before they’re breached 
• Reduce mean time to detect (MTTD) and resolve (MTTR) by surfacing not just data but decisions 

Causely continuously analyzes the relationships between services, symptoms, and anomalies — and uses causal reasoning to infer what’s driving those symptoms, all showcased natively in Grafana with the plugin.  

Why We Chose Beyla 

What’s the superpower behind this integration? It’s what we chose to run under the hood. 

Causely connects to your Kubernetes environment using Grafana Beyla, an open-source, eBPF-based instrumentation agent. It gives us deep visibility into your workloads — without sidecars, code changes, or custom configs. 

We chose Beyla because it enables our customers scaling on Kubernetes to gain value immediately and understand the root causes of issues impacting their applications.  Beyla enables quick time-to-value, minimal friction, and zero disruption to developer workflows. 

An important distinction here: Beyla isn’t part of the plugin — Beyla isn’t part of the plugin itself — it’s the component that connects to your environment and continuously monitors your services. The plugin then brings that intelligence into Grafana, making it actionable for your team." 

What You Get from Combining Causely and Grafana 

Bringing Causely into Grafana means you're gaining critical visibility when you need it most(no longer flying blind!). Together, they create a powerful loop: observability meets root cause analysis - inside your team's workflows. 

With Causely + Grafana, you get: 

• Instant visibility into root causes for all active anomalies and performance issues across your environment 
• The most urgent root cause of degrading your services surfaced right inside your Grafana dashboards 
• Proactive identification of issues putting your SLOs at risk - before violations occur 
• Context-rich visualizations and alert annotations that go beyond detection to show you what's broken, why it's happening, and the steps to take to resolve the issue 

Integrated with Grafana Alertmanager 

Causely's root cause alerts can be pushed directly into your existing Alertmanager workflows - so your teams don't just get notified that something's wrong; they get a head start on fixing it. That means fewer escalations, faster triage, and better SLO performance. 

Causely's reasoning engine powers all of this. It continuously maps how symptoms propagate through your services and precisely identifies the underlying causes. 

As mentioned, this is all enabled in minutes, thanks to Grafana Beyla. This open-source eBPF-based instrumentation layer lets Causely connect to your Kubernetes workloads without code changes, sidecars, or complex configurations. Beyla, let Causely start understanding your environment immediately - so you get value quickly without interrupting your team's flow. 

How to Use It 

Getting started is straightforward: 

• Install the Causely Plugin:   

   Please find it in the Grafana data source plugin catalog or install it via GitHub. 

• Link to Causely: 

Connect your Causely deployment to your Grafana instance. No changes to your instrumentation are needed, as Causely uses Beyla to automatically ingest telemetry from your Kubernetes workloads. 

• Visualize, Understand, Resolve:   

Add the Causely panel to your dashboards. See the root causes, understand the impact, and get ahead of issues before they become escalations. 

For complete setup steps, check out the Causely plugin docs.  

Start Seeing Root Causes, Not Just Symptoms  

The Causely Plugin for Grafana has officially launched and is ready for use. If you rely on Grafana to monitor your services, this is your next step in making observability genuinely actionable. 

Start Free Trial

Explore Documentation 

Request a Demo 

Modern observability tools help ensure engineers have all the data they could hope for at their disposal. But these tools cause a certain amount of “observability overload”; these tools do not have an understanding of whether an observed anomaly is a real problem and, more importantly, they do not on their own accurately infer the why (i.e. root cause) behind the what (i.e. observed symptoms).  

This was the focus of a recent conversation hosted by 10KMedia’s Adam LaGreca with guests Viktor Farcic (DevOps Toolkit) and Shmuel Kliger (founder of Causely). They explored why traditional observability approaches fall short on their own and what it will take to move beyond visibility toward real operational control. 

The Signal Problem No One's Solving 

The current state of reliability is defined by having too much data and not enough clarity. Alerts fire constantly and are often ignored. Dashboards contain all the data you could wish to have if you had all the time in the world to explore them and try to draw meaning from the data. What’s missing isn’t more data; it’s understanding. Without an understanding of cause and effect, teams are left guessing and leaning heavily on whichever domain experts they can grab at the moment.  

Causal Reasoning as a Path Forward 

The conversation centered on a new approach, causal reasoning, which leans away from bottom-up analysis of mountains of metrics, logs, and traces. By understanding service dependencies and identifying the cascading effect that load and code changes have on distributed systems, causal reasoning offers a way to eliminate the noise and focus on what truly matters for assuring reliable application performance. This helps reduce alert fatigue, speeds up troubleshooting, and frees engineers from the toil of reliability. 

Redefining the Role of AI 

While it makes logical sense to try to apply GenAI to the mountains of observability data we accumulate, leveraging LLMs without proper context will simply generate more noise at scale. By using causality to pinpoint the root cause of observed anomalies, a structured understanding of how systems behave can be used to get more efficient and effective outcomes from leveraging LLMs to automate operational work.  

The Future of Reliability 

Looking ahead, the vision is clear: fewer dashboards, fewer rabbit holes, and fewer hours lost to manual debugging. It’s a shift from observability to assurance. And for teams operating at the speed and scale of today’s distributed architectures, it’s a shift that can’t come soon enough. 

🔍 Want to see this approach in action? 
Check out the Causely sandbox or start a conversation with us: www.causely.ai 

If you’ve learned about how Causely works, you already know that our Causal Reasoning Platform includes a built-in causal knowledge base. This knowledge base guides system behavior by capturing the potential root causes in your environment and the symptoms they may cause. We are constantly exploring ways to expand that knowledge base, and one key source of inspiration for this is the work we do in the real-world with real users of our platform.

One of the most rewarding parts of my job is collaborating with awesome engineers who understand the value of our system and share ideas with me about how to make it better. Every customer scenario we encounter strengthens this knowledge base as we work through a four-step process:

Learn: Understand the root causes and observable symptoms of the scenario

Generalize: Capture the root causes and symptoms in the causality knowledge base

Implement: Develop the mediation to discover and monitor the required information

Deploy: Apply broadly and help all users benefit from this added knowledge

Whether it’s a metadata lock cascade in MySQL 8 or a Kubernetes resource noisy neighbor, this collaborative approach ensures that when one team faces a problem, the entire community can benefit from the expansion of our knowledge base. The following article is one such example of how this all played out in the real world with a real customer.

Modern Database Upgrades Can Be Painful

Last month I had a discussion with one of the engineering leaders at Yext, Peter Rimshnick. We explored a challenge many teams face: unexpected database locking in MySQL 8. Peter shared an incident where a routine schema change caused a bit of unexpected downtime. It turns out this scenario was tied to MySQL 8’s nuanced metadata locking behavior.

Metadata Locking in MySQL 8: What Changed?

MySQL 8 introduced critical improvements, but one underappreciated in their locking mechanism impacts teams daily. Before MySQL 8, A DDL statement (e.g., ALTER TABLE) locked only the target table. In MySQL 8, DDL operations now extend metadata locks to tables linked by foreign keys.

 Imagine this scenario:

A Migration Runs: A DROP COLUMN on the users table requests an exclusive MDL.

Dependencies Ignited: MySQL 8 locks the profiles table (linked via foreign key).

Queries Back Up: Reads/writes on both tables time out after lock_wait_timeout.

Symptoms Spread: APIs fail, dashboards freeze, teams chase false leads.

The Hidden Cost: Engineers manually trace foreign keys; customers see unrelated errors.

How Can Causely Help

Causely discovers dependencies such as users ↔ profiles, and observes how clients interact with the tables. Based on the symptoms it detects, it infers that the root cause is the DDL Locking during the database migration. 

Our platform can now infer root causes like DDL locking based on observed symptoms:

With Causely’s automated “DDL Excessive Lock” detection, engineers instantly pinpoint stalled schema changes—no more manual foreign-key tracing. This MySQL metadata-locking is just one of the many root causes we deliver out of the box. Explore the full library of insights to see how Causely can help you build reliable, resilient data pipelines.

The Network Effect of Shared Learning

This is how modern observability evolves: real problems solved once, scaled to many. Every collaboration like Yext’s isn’t just a fix; it’s a force multiplier that makes the platform more knowledgeable and eliminates manual troubleshooting. Every new root cause we learn from strengthens the entire platform. Join a community of engineers building a smarter, more resilient future.

 For engineers tired of playing whack-a-mole with outages:

• Learn more about Causely
• See Causely in action: Request a demo

But there are no free lunches -this adoption comes with a cost. As organizations adopt microservices, they encounter new operational challenges. Microservices architectures are dynamic, ever evolving and loosely coupled with intricate dependencies and interactions between services. Although built with standardized building blocks and common patterns, the emergent behavior of each system is unique and constantly changing as components are rewritten, upgraded, or replaced. The simplicity of small, loosely coupled services can quickly become overshadowed by the complexity of managing hundreds (or thousands) of interdependent components. 

Continuing the decades long trend in Observability of collecting more and more data won’t get us to the desired state of assuring service reliability. Furthermore, feeding this data to a machine and hoping the machine will magically generate the answers required to continuously assure service reliability is a false hope. Machines trained on yesterday's data may be able to understand the past but cannot make the real-time decisions necessary to continuously assure service levels, especially in cloud-native environments given the dynamic nature of such environments. 

At Causely, we believe that the key to overcoming these challenges lies in causal knowledge captured as part of an ontology, to enable reasoning about cause-and-effect relationships in complex systems. This stands in contrast to the industry's growing reliance on simply sending alerts to Large Language Models (LLMs), and we believe causal reasoning is the critical missing piece for autonomous service reliability. 

Why Traditional Approaches Fall Short 

Some organizations attempt to address reliability by dumping vast amounts of raw, unstructured telemetry into ML algorithms in the hope of surfacing meaningful patterns. When anomalies are detected, these observations are sometimes passed to LLMs to generate plausible explanations. While this can help contextualize events, it often falls short where it matters most. LLMs, by design, are general-purpose tools trained on broad, historical data. They excel at pattern recognition and language generation but lack the deep, real-time causal reasoning needed to adapt to novel, dynamic environments. They may describe "what" happened, but they struggle to uncover "why" it happened and "what" to do about it in a new, unseen context. Even when LLMs are fine-tuned on telemetry patterns, they generalize across environments. But reliability failures are context specific. What broke in one deployment doesn’t explain a novel failure in another deployment. 

Example 1 – Misleading Latency Correlations Across Services: 

In a microservices-based e-commerce platform, a simultaneous latency spike was observed in the checkout, inventory, and payment services. Observability tools showed strong correlations between these services, leading engineers to suspect the inventory service as the root cause. However, the real issue was a slow database query in the product-catalog service, which affected the three components above. The correlation misled the team into focusing on the wrong area, demonstrating the limitations of correlation without causal context. 

Example 2 – Misinterpreting Memory Spikes as Service Defects: 

An alert for repeated pod restarts in the recommendation engine, accompanied by memory spikes across backend services, led engineers to suspect a memory leak. Observability tools flagged backend services as anomalous based on correlated metrics. However, the actual root cause was a recent frontend change that dramatically increased the frequency and size of incoming requests. The backend services were merely reacting to an upstream trigger. This highlights how correlation can obscure the true origin of a problem, especially when external factors are involved. 

Relying on statistical correlation alone can be dangerous. Correlations can be misleading without causal grounding, leading teams to chase false positives, miss root causes, or implement ineffective remediations. 

Beyond Monitoring: Towards Autonomous Operations 

Traditional observability focuses on "what" happened. Causal Reasoning focuses on “why”. Causal Reasoning captures, represents, understands and analyzes cause-and-effect relationships and uses these, among other inferences, to automatically infer root cause based on observed anomalies. By embracing Causal Reasoning, organizations can move beyond the reactive model of monitoring and alerting to a world of autonomous operations, where systems can diagnose and heal themselves with minimal human intervention. This is essential for achieving the promise of resilient, scalable, always-on cloud-native applications. 

Causal Reasoning is driven by ontology. An ontology is a formal model that defines: 

• The types of entities, attributes and relationships in a domain, including root causes and symptoms 
• The relationships that can exist between entities, including the causality relationships between root causes and symptoms, and attribute dependencies  
• The behaviors or constraints (e.g., "a pod can be scheduled on one node ", "a pod can have multiple containers") 

It’s like the grammar and vocabulary for talking about a subject. 

Causal Reasoning uses a knowledge graph to organize the real-world information based on the ontology 

• Nodes are specific instances of entities (e.g., checkout-service, inventory-service, payment-service, production-database) 
• Edges are actual relationships between the instances (e.g., checkout-service -> depends_on -> production-database) 
• Edges can also be relationship between attribute dependencies (e.g. “calls to a user facing API invokes the backend GRPC method”, “backend GRPC method invocation produces async messages on a specific topic”) 
• Metadata: e.g. CPU usage, error logs, deployment time, configs 

The knowledge graph is the filled-out version of the ontology, populated with facts. The discovered topology of a microservices application is a knowledge graph, which describes the application components and their relationships using an ontology. It describes what is like a semantic network, but it doesn’t say anything about why or what will happen if something changes. 

Using the ontology and the knowledge graph Causal Reasoning automatically generates a causal graph. A causal graph is a directed acyclic graph (DAG) with focus on why things happen: 

• Nodes are specific causes and observations 
• Directed edges that represent causal links, not just association 
• Example: DatabaseMalfunction -> causes -> ClientServiceErrors 
• Allows you to ask "what if" questions: 
• What happens to service errors if the database is recovered? 

In short, a knowledge graph describes what is connected to what, while a causal graph, describes what causes what. 

Causal Reasoning: Engineering Intelligence into Service Operations 

The causal knowledge in the ontology captures essential system behaviors and relationships without getting lost in the weeds. Driven by the causal knowledge, causal reasoning enables engineering teams focus on what matters. Instead of reacting to every blip on a dashboard, casual reasoning drives a top-down focus on the critical causes that impact service reliability. 

Using causal reasoning, we can: 

• Understand why a service's performance degraded, not just that it did. 
• Infer the root causes instead of guessing based on symptoms. 
• Develop proactive, preventive strategies instead of reactive firefighting. 

Causal reasoning empowers teams to diagnose, remediate, and even predict and prevent risks to service-level objectives (SLOs) with clarity and confidence.   

Causely: A Purpose-built  Autonomous Reliability System 

At Causely, we’ve developed a purpose-built Causal Reasoning Platform for service reliability. The key tenets of the platform are: 

• Causal model: an ontology describing cloud-native application environments, including the causal knowledge of root causes, symptoms and causality between them 
• Topology graph: a knowledge graph of the specific managed environment. The graph is generated automatically by discovering the managed environment. 
• Abductive inference engine: an engine that automatically generates a causality graph from the ontology and the topology graph and used it in real time to infer root causes based on observed symptoms/anomalies   

Causely solves a problem that no other vendor is solving by delivering 

• Clarity in Complexity: Our models scale with your systems, maintaining meaningful insights even as your architecture evolves and grows more intricate. 
• Actionable Insights: We don't just flag anomalies—we infer root causes and deliver clear, prioritized paths to resolution. 
• Proactive Prevention: With an ontology and causal reasoning, we spot risks before they become incidents, shifting organizations from reactive to proactive. 
• Seamless Integration: Our platform integrates with your workflows and CI/CD pipelines, delivering instant value without requiring manual retraining or rule-writing. 

Conclusion 

We do see LLMs as a powerful technology that can benefit from being used in tandem with Causal reasoning, stay tuned for more on that.

Causal Reasoning is the foundation of Causely’s solution. It empowers organizations to cut through complexity, deliver precise insights, prevent downtime, and free engineers to focus on what matters most: innovation. 

Let us show you the power of causal reasoning and cross-organizational collaboration in cloud-native environments. See Causely for yourself. Book a meeting with the Causely team or start your free trial now. 

#cloudnative #servicereliability #causalreasoning #abstraction #siteReliabilityEngineering #sre #DevOps #observability #AI #ML 

Recently, we encountered an issue that perfectly illustrates the value of Causely’s Causal Reasoning Platform in action.  

A Crash That Didn’t Want to Be Found 

While reviewing our staging environment— which uses our existing OpenTelemetry instrumentation to identify root causes in our production deployment of Causely — I noticed something disturbing: a frequent crash failure had been quietly recurring over six days in one of our analytics microservices. 

This type of issue is notoriously difficult to detect. The logs were noisy. The failures were intermittent. The symptoms appeared scattered—some in one analysis module, others in a different component entirely. This issue would’ve gone unnoticed in most environments or been chalked up to transient behavior. But Causely identified the cause. Not as an alert or a spike in a dashboard – but as an active root cause that is impacting our system. It wasn’t just showing us individual failures. It was showing us causality - the propagation behind the symptoms. 

Different subcomponents—analysis-3, analysis-5—were failing in seemingly unrelated ways. One had high error rates; another had frequent panics. Without Causely, we would’ve been chasing these symptoms in isolation. Logging into pods, checking dashboards, comparing timelines. A manual, time-consuming wild goose chase. 

What Causely Did Differently 

 Causely automatically inferred these separate symptoms across services and time into a single root cause: a concurrency issue in a shared in-memory map used across multiple analysis workers. 

The issue? A concurrent write-read condition that would only manifest under higher-volume conditions – something nearly impossible to catch in development or test environments. But Causely connected the dots through symptom propagation and causal relationships, allowing us to zero in on the problem fast. 

Technical Deep Dive: The Bug Behind the Crash 

The crash was caused by concurrent access to a Go map, which is unsafe for concurrent reads and writes without explicit synchronization. 

We were using a plain map [string]SomeStruct to store intermediate analysis results across worker goroutines. This map was being: 

• Written to by one thread collecting event evidence in real time 
• Read from simultaneously by another thread responsible for emitting outputs 

Under low volume, this worked fine. But under sustained load in production, the race condition emerged—resulting in panics like: 

fatal error: concurrent map read and map write 

 This is a classic Go gotcha. The fix was straightforward: we wrapped the map with a sync.RWMutex to protect both reads and writes, ensuring thread safety. Alternatively, we could have used a sync.Map, but since our access patterns were relatively simple and performance-sensitive, the mutex approach was more appropriate. 

This was one of those bugs where: 

• It was impossible to reproduce locally 
• The crash symptoms varied depending on which thread accessed the map first 
• And the logs, while technically correct, gave no useful hint unless you already suspected the concurrency issue 

 Without Causely surfacing the underlying causality across services and across time, this would have remained an intermittent ghost bug. 

From Root Cause to Resolution – In Hours, Not Days 

Thanks to the inference Causely provided, we identified the concurrency bug, coded a fix, and shipped it – all within a couple of hours. 

This wasn’t just about speed. It was about precision. Without Causely, our team might have: 

• Spent days troubleshooting unrelated alerts and errors 
• Overlooked the root cause due to insufficient signal 
• Delayed resolving an issue that was actively degrading reliability 

Why This Matters 

Crashes like these don’t always surface in ways traditional monitoring or modern observability systems can catch. They emerge over time, appear intermittent, and produce symptoms that span components. 

Causely gave us what observability tools couldn’t: contextual understanding. Observability tools show you context. Causely helps you understand it. It didn’t just tell us something was broken. It told us what was breaking, why, and what needed to change. 

This is the power of causal reasoning over correlation. 

We built Causely to operate in the most demanding environments – and we hold ourselves to that same standard. This incident was a clear example of how our platform enables us to move faster, with more confidence, and resolve issues before they impact users. 

And that’s exactly the kind of reliability we aim to bring to every engineering team. 

Avoiding common observability pitfalls

We already know – based on decades of experience working in and with operations teams in the most challenging environments – that bridging the gap between the vast ocean of observability data and actionable insights has and continues to be a major pain point. This is especially true in the complex world of cloud-native applications.

Missing application insights

Application observability remains an elusive beast for many, especially in complex microservices architectures. While infrastructure monitoring has become readily available, neglecting application data paints an incomplete picture, hindering effective troubleshooting and operations.

Siloed solutions

Traditional observability solutions have relied on siloed, proprietary agents and data sources, leading to fragmented visibility across teams and technologies. This makes it difficult to understand the complete picture of service composition and dependencies.

To me, this is like trying to solve a puzzle with missing pieces – that’s essentially a problem that many DevOps teams face today – piecing together a picture of how microservices, serverless functions, databases, and other elements interact with one another, and underlying infrastructure and cloud services they run on. This makes it hard to collaborate and troubleshoot; it's a struggle to pinpoint the root cause of performance issues or outages.

Vendor lock-in

Many vendors’ products also lock customers’ data into their cloud services. This can result in customers paying through the nose, because licensing costs are predicated on the volume of data that is being collected and stored in the service providers’ backend SaaS. It can also be very hard to exit these services once locked in.

These are all pitfalls we wanted to avoid at Causely as we set out to build our our Causal Reasoning Platform.

The pillars of our observability architecture pointed us to OpenTelemetry

OpenTelemetry provides us with a path to break free from these limitations, establishing a common framework that transcends programming languages and platforms that we are using to build our services, and satisfying the requirements laid out in the pillars of our observability architecture:

Precise instrumentation

OpenTelemetry offers automatic instrumentation options that minimize the amount of work we need to do on manual code modifications and streamline the integration of our internal observability capabilities into our chosen backend applications.

Unified picture

By providing a standardized data model powered by semantic conventions, OpenTelemetry enables us to paint an end-to-end picture of how all of our services are composed, including application and infrastructure dependencies. We can also gain access to critical telemetry information, utilizing this semantically consistent data across multiple backend microservices even when written in different languages.

Vendor-neutral data management

OpenTelemetry allows us to avoid locking our application data into 3rd party vendors’ services by decoupling it from proprietary vendor formats. This gives us the freedom to choose the best tools on an ongoing basis based on the value they provide. If something new comes along that we want to exploit, we can easily plug it into our architecture.

Resource-optimized observability

With OpenTelemetry, we can take a top down approach to data collection, starting with the problems we are looking to solve and eliminating unnecessary information. In doing so, this minimizes our storage costs and optimizes compute resources we need to support our observability pipeline.

We believe that following these pillars and building our Causal Reasoning Platform on top of OpenTelemetry propels our product’s performance, enables rock-solid reliability, and ensures consistent service experiences for our customers as we scale our business. We also minimize ongoing operational costs, creating a win-win for us and our customers.

OpenTelemetry + causal analysis: scaling for performance and cost efficiency

Ultimately, observability aims to illuminate the behavior of distributed systems, enabling proactive maintenance and swift troubleshooting. Yet isolated failures manifest as cascading symptoms across interconnected services.

While OpenTelemetry enables back-end applications to use this data to provide a unified picture in maps, graphs and dashboards, the job of figuring out the cause and effect in the correlated data still requires highly skilled resources. This process can also be very time consuming, tying up personnel across multiple teams, with ownership for different elements of overall services.

There is a lot of noise in the industry right now about how AI and LLMs are going to magically come to the rescue, but reality paints a different picture. All of the solutions available in the market today focus on correlating data versus uncovering a direct understanding of causal relationships between problems and the symptoms they cause, leaving DevOps teams with noise, not answers.

Traditional AI and LLMs also require massive amounts of data as input for training and learning behaviors on a continuous basis. This is data that ultimately ends up being transferred and stored in some form of SaaS. Processing these large datasets is very computationally intensive. This all translates into significant cost overheads for the SaaS providers as customer datasets grow overtime – costs that ultimately result in ever increasing bills for customers.

At Causely, we're taking a different approach

Our causal reasoning software provides operations and engineering teams with an understanding of the “why”, which is crucial for effective and timely troubleshooting and decision-making.

Our Causal Reasoning Platform uses predefined models of how problems behave and propagate. When combined with real-time information about a system’s specific structure, Causely computes a map linking all potential problems to their observable symptoms.

This map acts as a reference guide, eliminating the need to analyze massive datasets every time the platform encounters an issue. Think of it as checking a dictionary instead of reading an entire encyclopedia.

The bottom line is, in contrast to traditional AI, Causely operates on a much smaller dataset, requires far less resources for computation and provides more meaningful actionable insights, all of which translate into lower ongoing operational costs and profitable growth.

Summing it up

There’s massive potential for causal analysis and OpenTelemetry to come together to tackle the limitations of traditional AI to get to the “why.” This is what we’re building at Causely – check out our recent integration news. Doing so delivers numerous benefits:

• Less time on Ops, more time on Dev: OpenTelemetry provides standardized data while Causely analyzes it to automate the root cause analysis (RCA) process, which will significantly reduce the time our DevOps teams have to spend on troubleshooting.
• Instant gratification, no training lag: We can eliminate AI’s slow learning curve, because Causely leverages OpenTelemetry’s semantic language and the Causal Reasoning Platform’s domain knowledge of cause and effect to deliver actionable results, right out of the box without massive amounts of data and with no training lag!
• Small data, lean computation, big impact: Unlike traditional AI’s data gluttony and significant computational overheads, Causely thrives on targeted data streams. OpenTelemetry’s smart filtering keeps the information flow lean, allowing Causely to identify the root causes with a significantly smaller dataset and compute footprint.
• Fast root cause identification: Traditional AI might tell us “ice cream sales and shark attacks rise together,” but causal reasoning reveals the truth – it’s the summer heat and not the sharks, driving both! By understanding cause-and-effect relationships, Causely cuts through the noise and identifies the root causes behind performance degradation and service malfunctions.

Having these capabilities is critical if we want to move beyond the labor intensive processes associated with how RCA is performed in DevOps today, and eventually achieve autonomous service reliability. This is why we are eating our own dog food and using Causely as part of our tech stack to manage the services we provide to customers.

Want to learn more about our integration with OpenTelemetry or see if Causely can help you build better, more reliable cloud-native applications? Book a meeting with the Causely team. We'd love to chat!

Our team at Causely has adopted OpenTelemetry within our own platform, and we recently announced a formal integration with OpenTelemetry, which prompted us to share a production-focused guide. Our goal is to help developers, DevOps engineers, software engineers, and SREs understand what OpenTelemetry is, its core components, and a detailed look at the OpenTelemetry Collector (OTel Collector). This background will help you use OTel and the OTel Collector as part of a comprehensive strategy to monitor and observe applications.

What Data Does OpenTelemetry Collect?

There are 3 types of data that are gathered by OpenTelemetry using the OTel Collector: logs, metrics, and traces.

Logs

Logs are records of events that occur within an application. They provide a detailed account of what happened, when it happened, and any relevant data associated with the event. Logs are helpful for debugging and understanding the behavior of applications.

OpenTelemetry collects and exports logs, providing insights into events and errors that occur within the system. For example, if a user reports a slow response time in a specific feature of the application, engineers can use OpenTelemetry logs to trace back the events leading up to the reported issue.

Metrics

Metrics are quantitative data that measure the performance and health of an application. Metrics help in tracking system behavior and identifying trends over time. OpenTelemetry collects metrics data, which helps in tracking resource usage, system performance, and identifying anomalies.

For instance, if a spike in CPU usage is detected using OpenTelemetry metrics, engineers can investigate the potential issue using the OTel data collected and make necessary adjustments to optimize performance.

Developers use OpenTelemetry metrics to see granular resource utilization data, which helps understand how the application is functioning under different conditions.

Traces

Traces provide a detailed view of request flows within a distributed system. Traces help understand the execution path, diagnose application behaviors, and see the interactions between different services.

For example, if a user reports slow response times on a website, developers can use trace data to help better identify which service is experiencing issues. Traces can also help in debugging issues such as failed requests or errors by providing a step-by-step view of how requests are processed through the system.

Introduction to OTel Collector

You can deploy the OTel Collector as a standalone agent or as a sidecar alongside your application. The OTel Collector also includes some helpful features for sampling, filtering, and transforming data before sending it to a monitoring backend.

How it Works

The OTel Collector works by receiving telemetry data from many different sources, processing it based on configured pipelines, and exporting it to chosen backends. This modular architecture allows for customization and scalability.

The OTel Collector acts as a central data pipeline for collecting, processing, and exporting telemetry data (metrics, logs, traces) within an observability stack.

Here’s a technical breakdown:

Data Ingestion:

• Leverages pluggable receivers for specific data sources (e.g., Redis receiver, MySQL receiver).
• Receivers can be configured for specific endpoints, authentication, and data collection parameters.
• Supports various data formats (e.g., native application instrumentation libraries, vendor-specific formats) through receiver implementations.

Data Processing:

• Processors can be chained to manipulate the collected data before export.
• Common processing functions include:
  • Batching: Improves efficiency by sending data in aggregates.
  • Filtering: Selects specific data based on criteria.
  • Sampling: Reduces data volume by statistically sampling telemetry.
  • Enrichment: Adds contextual information to the data.

Data Export:

• Utilizes exporters to send the processed data to backend systems.
• Exporters are available for various observability backends (e.g., Jaeger, Zipkin, Prometheus).
• Exporter configurations specify the destination endpoint and data format for the backend system.

Internal Representation:

• Leverages OpenTelemetry’s internal Protobuf data format (pdata) for efficient data handling.
• Receivers translate source-specific data formats into pdata format for processing.
• Exporters translate pdata format into the backend system’s expected data format.

Scalability and Configurability:

• Designed for horizontal scaling by deploying multiple collector instances.
• Configuration files written in YAML allow for dynamic configuration of receivers, processors, and exporters.
• Supports running as an agent on individual hosts or as a standalone service.

The OTel Collector is format-agnostic and flexible, built to work with various backend observability systems.

Setting up the OpenTelemetry (OTel) Collector

Starting with OpenTelemetry for your new system is a straightforward process that takes only a few steps:

1. Download the OTel Collector: Obtain the latest version from the official OpenTelemetry website or your preferred package manager.
2. Configure the OTel Collector: Edit the configuration file to define data sources and export destinations.
3. Run the OTel Collector: Start the Collector to begin collecting and processing telemetry data.

Keep in mind that the example we will show here is relatively simple. A large scale production implementation will require fine-tuning to ensure optimal results. Make sure to follow your OS-specific instructions to deploy and run the OTel collector.

Next, we need to configure some exporters for your application stack.

Integration with Popular Tools and Platforms

Let’s use an example system running a multi-tier web application using NGINX, MySQL, and Redis. Each source platform will have some application-specific configuration parameters.

Configuring Receivers

redisreceiver:

• Replace receiver_name with redisreceiver
• Set endpoint to the port where your Redis server is listening (default: 6379)
• You can configure additional options like authentication and collection intervals in the receiver configuration. Refer to the official documentation for details.

mysqlreceiver:

• Replace receiver_name with mysqlreceiver
• Set endpoint to the connection string for your MySQL server (e.g., mysql://user:password@localhost:3306/database)
• Similar to Redis receiver, you can configure authentication and collection intervals. Refer to the documentation for details.

nginxreceiver:

• Replace receiver_name with nginxreceiver
• No endpoint configuration needed as it scrapes metrics from the NGINX process.
• You can configure what metrics to collect and scraping intervals in the receiver configuration. Refer to the documentation for details.

The OpenTelemetry Collector can export data to multiple providers including Prometheus, Jaeger, Zipkin, and, of course, Causely. This flexibility allows users to leverage their existing tools while adopting OpenTelemetry.

Configuring Exporters

Replace exporter_name with the actual exporter type for your external system. Here are some common options:

• jaeger for Jaeger backend
• zipkin for Zipkin backend
• otlp/causely for Causely backend
• There are exporters for many other systems as well. Refer to the documentation for a complete list.

Set endpoint to the URL of your external system where you want to send the collected telemetry data. You might need to configure additional options specific to the chosen exporter (e.g., authentication for Jaeger).

There is also a growing list of supporting vendors who consume OpenTelemetry data.

Conclusion

OpenTelemetry provides a standardized approach to collecting and exporting logs, metrics, and traces. Implementing OpenTelemetry and the OTel Collector offer a scalable and flexible solution for managing telemetry data, making it a popular and effective tool for modern applications.

You can use OpenTelemetry as part of your monitoring and observability practice in order to gather data that can help drive better understanding of the state of your applications. The most valuable part of OpenTelemetry is the ability to ingest the data for deeper analysis.

How Causely Works with OpenTelemetry

At Causely, we use OpenTelemetry as one of many data sources to assure autonomous service reliability for our clients. OpenTelemetry data is ingested by our Causal Reasoning Platform, which is a model-driven AI system that automatically instruments your environment, defines SLOs, and pinpoints risk to reliability - without manual setup. By cutting through observability overload, Causely eliminates complex troubleshooting and ensures services meet reliability expectations. Engineering teams instantly improve service reliability, reduce downtime, and lower operational costs – freeing them to focus on innovation.

Click below to watch the interview, or check it out on Techstrong.tv here.

View the full article by Carl Williams on Tech Times.

Causely is announcing its integration with OpenTelemetry, bringing a fresh approach to observability that cuts through the noise and surfaces only what matters.

In today’s world of modern cloud applications, observability is both a necessity and a challenge. With countless services interacting dynamically, engineering teams are often flooded with logs, traces, and metrics, making it difficult to extract meaningful insights when something goes wrong. Traditional observability approaches rely on collecting everything—but the sheer volume of data can be overwhelming, turning troubleshooting into a time-consuming and frustrating process.

Causely is here to change that. Today, the company is announcing its integration with OpenTelemetry, bringing a fresh approach to observability that cuts through the noise and surfaces only what matters.

Yotam Yemini, CEO of Causely, highlights that OpenTelemetry adoption often requires significant effort, as teams must determine which data to collect, how to sample it, and how to extract useful insights. “Causely is focused on cutting through that noise to pinpoint what matters,” Yemini says. Causely does this by automatically filtering out unnecessary data and providing a structured understanding of system dependencies and potential failure points. The integration makes OpenTelemetry more practical and efficient, ensuring engineers receive only the most relevant insights.

The Challenge of Observability Today

Modern applications generate an immense amount of telemetry data, whether it’s from microservices, distributed systems, or cloud infrastructure. While OpenTelemetry has become the de facto standard for collecting and transmitting this data, its adoption often comes with significant hurdles. Engineers frequently struggle with:

• Too much data, too little insight – Logs and traces pile up, making it difficult to identify real issues.
• Slow root cause analysis – Teams spend too much time manually sifting through observability data.
• Uncertain data collection strategies – Debates about sampling vs. full collection leave engineers guessing about what’s really needed.

The result? An overload of information but very few actionable insights.

A Smarter Approach to Observability

Causely’s Causal Reasoning Platform takes a fundamentally different approach. Instead of just presenting raw telemetry data, it analyzes dependencies and causal relationships between different components of an application. This means that instead of looking at an overwhelming sea of logs, engineers can quickly pinpoint what is causing an issue and how it propagates through the system.

With the new OpenTelemetry integration, Causely makes it even easier for teams to:

• Seamlessly ingest OpenTelemetry traces and metrics without any complicated setup.
• Automatically filter out irrelevant data and highlight the most critical insights.
• Predict potential failures before they happen, allowing teams to be proactive instead of reactive.

By integrating with OpenTelemetry, Causely aims to bridge the gap between raw telemetry collection and intelligent observability. Teams can proactively manage service reliability, prevent latency issues, and optimize performance, making observability more effective and manageable.

How it Works

Causely provides two easy ways to integrate with OpenTelemetry. If your team is already using OpenTelemetry, you can simply point your existing collector at Causely. Within seconds, the system begins processing the telemetry data and identifying key dependencies, bottlenecks, and potential risks.

For teams that haven’t yet adopted OpenTelemetry, Causely offers an alternative: auto-instrumentation using eBPF. This allows teams to collect the necessary telemetry data without having to manually modify their applications. The best part? It all runs in your own cloud environment, ensuring data privacy and security while keeping infrastructure overhead minimal.

Beyond OpenTelemetry: Observability without Limits

While this new integration enhances how teams use OpenTelemetry, Causely isn’t limited to a single observability standard. The platform is designed to work with any telemetry source, whether proprietary or open-source, ensuring flexibility for engineering teams regardless of their stack.

By decoupling the intelligence layer from data collection, Causely provides a unified way to analyze and act on observability data, no matter where it comes from. This approach helps reduce the complexity caused by vendor sprawl, allowing teams to focus on application performance rather than managing observability infrastructure.

Who Benefits from Causely?

Causely is built for any engineering team dealing with complex, microservices-driven applications. Whether you’re operating a large-scale SaaS platform, managing a fintech infrastructure, or ensuring the reliability of a healthcare application, Causely helps keep things running smoothly by automating root cause detection and providing predictive insights.

Companies already leveraging Causely include publicly traded SaaS providers, financial institutions, telecom giants, and healthcare tech firms—all benefiting from the ability to reduce incident resolution times and improve service reliability.

Get Started with Causely

Observability doesn’t have to be overwhelming. With Causely’s OpenTelemetry integration, teams can move beyond raw data collection and gain instant, actionable insights that make troubleshooting faster and easier.

For those interested in trying it out, Causely offers a free trial where you can explore a demo environment or set up a real-world deployment in minutes. To get started, visit causely.ai and see how causal reasoning can transform your observability strategy.

Instead of spending hours digging through logs, let Causely do the heavy lifting—so you can focus on building great applications.

Guest: Yotam Yemini
Company: Causely
Show: Let’s Talk

Summary is written by Emily Nicholls

I’ve spent over three decades in IT Operations. Despite all the talk of transformation, many of the fundamental challenges remain unchanged, or have even worsened. The rise of modern DevOps and observability promised to revolutionize how we monitor and maintain systems, but in reality, we’ve simply scaled up the same old problems. More data, more dashboards, and more alerts haven’t led to better outcomes.  
 
The core issue? Our approach to observability has been misguided. We need a paradigm shift from a bottom-up approach to a top-down approach. Instead of bottoms up collecting everything hoping we find insights in the data, we need to start with the purpose or the insights we are looking for and collect only the data that will help us to infer these insights. 

Part 1: Looking Back at the Days of ITOps 

From the early days of IT Operations, we have been focusing on keeping the lights on. At the early days, systems were monolithic, monitoring was rudimentary, and troubleshooting often involved sifting through log files for hours on end. A major incident meant a war room filled with engineers manually correlating data, trying to pinpoint the root cause of an outage. 

As infrastructure grew more complex, the industry responded by layering on more tools, with each promising to make troubleshooting easier. But in practice, these tools often just created more dashboards, more logs, and more alerts, leading to information overload. IT Operations needed to evolve, which gave rise to modern DevOps, but we are still in the early innings of the game.  

Part 2: The Problem Today with Observability 

Observability was supposed to solve these challenges by giving teams a deeper understanding of their systems. The idea was that by collecting and analyzing vast amounts of telemetry – metrics, events, logs, and traces – organizations would gain better insights and respond to issues faster. 

Instead, we’ve seen an explosion of complexity. Today’s observability landscape is defined by: 

• Too much noise: The sheer volume of logs and alerts makes it nearly impossible for engineers to separate meaningful signals from the flood of data. 
• Too many tools: Companies rely on a fragmented ecosystem of monitoring, logging, and tracing solutions, leading to silos and inefficiencies. 
• Too much manual troubleshooting: Despite all this data, engineers still spend most of their time manually diagnosing incidents, correlating logs, and responding to false positives. 

The promise of observability has not been fully realized because organizations are still focusing on data collection instead of shifting their focus to intelligent data processing. 

Part 3: Causely’s Paradigm Shift 

At Causely, we believe that Observability must be disrupted and go through a paradigm shift from a bottom-up data collection to top-down purpose driven analytics that saves engineers from needing to spend hours sifting through data to figure out “the why behind the what.” We reject the notion that engineers will always need to spend time drilling through dashboards and making sense of the data coming from their tools. Instead, we believe that implementing the proper abstraction layer would allow systems to self-manage and take humans out of the troubleshooting loop. Instead of using tools that provide information for the engineers to analyze, the engineers should deploy systems that can actually make autonomous decisions. 

This journey to autonomous service reliability is built on a number of core principles that I outlined in a recent blog. This shift from passively collecting data to relying on a system that actively interprets and acts on these insights is the future of observability. 

Part 4: The Rise of AI and Agentic AI 

We are on the cusp of a new era where Agentic AI can fundamentally reshape IT Operations. Instead of engineers reacting to alerts, AI-driven systems will proactively maintain service reliability, predicting and preventing failures before they occur. 

I’m not talking about simple alerting rules or anomaly detection; I’m talking about true agentic AI that continuously analyzes system behavior and adapts in real-time to prevent service degradation.  

Causely is positioned to lead this transformation, helping organizations shift from reactive troubleshooting to proactive, AI-driven service reliability. This isn’t just a step forward in observability—it’s a paradigm shift in how we think about IT Operations. 

Conclusion 

For 30+ years, we’ve been fighting the same battles in IT Operations, just at a larger scale. It’s time to rethink observability, move beyond endless data collection, and embrace a future where AI-driven automation ensures service reliability with minimal human intervention. Organizations that adopt this new paradigm won’t just reduce downtime and incident costs—they’ll free their engineers to focus on what truly matters: building the future. 

Check out the interview below or view it on Spotify.

“For years, the IT industry has struggled to make sense of the overwhelming amounts of data coming from dozens of observability platforms and monitoring tools,” said Yotam Yemini, CEO of Causely. “OpenTelemetry is an important paradigm shift that gives teams a more flexible and vendor-neutral way to collect and use telemetry data, but it also exacerbates the big data problem of modern DevOps: deriving actionable insights from overwhelming amounts of telemetry data.”

Causely installs in cloud-native environments in under a minute and immediately bridges the gap between the overwhelming amount of observability data and the actionable insights that can be found within that data, maximizing the potential benefits of OpenTelemetry. Teams who have yet to adopt OpenTelemetry can leverage Causely’s eBPF-based auto-instrumentation to start collecting traces instantly. 

The Causely system works by automatically mapping an application’s topology and service dependencies, then applying a finite set of likely root causes to this data. This novel approach to observability is counter to traditional tooling and methods that encourage businesses to collect as much data as possible – a situation that hasn’t fundamentally changed in decades – which then requires human troubleshooting to respond to alerts, make sense of patterns, identify root cause, and ultimately determine the best action for remediation. 

“Traditionally an alert fires and we have to throw a bunch of subject matter experts at the problem to comb through all of the telemetry and troubleshoot why these things are going on,” said Matt Titmus, Engineering Manager at Yext. “I’m enthusiastic about Causely because it gets us to the root cause much faster and also helps us be more proactive."

Causely was founded by Shmuel Kliger who has been developing systems for IT Operations for over two decades. He was also the founder of Turbonomic (acquired by IBM) and the CTO of SMARTS (acquired by EMC), bringing together technical experience with a track record of successfully scaling companies.

About Causely

Causely leverages causal reasoning to cut through the observability noise and pinpoint what matters. Engineers are overwhelmed by too many tools, alerts, and data coming from their existing observability solutions. Causely automatically surfaces only the most critical risks to service reliability, enabling businesses to minimize operational overhead and maintain reliability without manual troubleshooting.

Contact

Adam LaGreca
Founder of 10KMedia
adam@10kmedia.co

Observability has become a cornerstone of application reliability and performance. As systems grow more complex—spanning microservices, third-party APIs, and asynchronous messaging patterns—the ability to monitor and debug these systems is both a necessity and a challenge. 

OpenTelemetry (OTEL) has emerged as a powerful, open source framework that standardizes the collection of telemetry data across distributed systems. It promises unprecedented visibility into logs, metrics, and traces, empowering engineers to identify issues and optimize performance across multiple languages, technologies and cloud environments. 

But with great visibility comes a hidden cost. While OTEL democratizes observability, it also exacerbates the “big data problem” of modern DevOps. 

This is where Causely comes in—today, we announced a new integration with OTEL that bridges the gap between OTEL's data deluge and actionable insights. In this post, we’ll explore the strengths and limitations of OpenTelemetry, the challenges it introduces, and how Causely transforms raw telemetry into precise, cost-effective analytics. 

The OpenTelemetry opportunity 

Microservices are a tangled web of interdependencies that communicate over REST or gRPC. Asynchronous systems like Kafka shuttle messages between loosely coupled services. Infrastructure dynamically scales resources to meet demand. Observability has become the glue that holds these systems together, enabling engineers to monitor performance, troubleshoot issues, and ensure reliability. 

At the heart of the observability revolution is OpenTelemetry (OTEL), an open-source standard that unifies the instrumentation and collection of telemetry data across logs, metrics, and traces. Its modular architecture, community-driven development, and broad compatibility with existing observability tools have made OTEL the de facto choice for modern DevOps teams. 

What does OpenTelemetry do? 

OpenTelemetry provides APIs, SDKs, and tools to capture three primary types of telemetry data: 

1. Logs: Detailed, timestamped records of system events (e.g., errors, warnings, and custom events). 
2. Metrics: Quantitative measurements of system health and performance (e.g., CPU usage, request latency, error rates). 
3. Traces: End-to-end views of requests flowing through distributed systems, mapping dependencies and execution paths. 

With OTEL, engineers can instrument their code to emit these telemetry signals, use an OpenTelemetry Collector to aggregate and process the data, and export it to observability backends like Prometheus, Tempo, or Elasticsearch. 

Why OpenTelemetry is a game-changer 

OpenTelemetry addresses a critical pain point in observability: fragmentation. Historically, different tools and platforms required unique instrumentation libraries, making it difficult to standardize observability across an organization. OTEL simplifies this by providing: 

• Vendor-Agnostic Instrumentation: A single API to instrument applications regardless of the backend. 
• Centralized Data Collection: The OpenTelemetry Collector serves as a pluggable data pipeline, consolidating telemetry from various sources. 
• Interoperability: Native support for popular backends like Prometheus, Tempo, and other vendors, allowing teams to integrate OTEL into their existing observability stack. 

Technical example: Debugging latency issues 

Consider a microservices-based e-commerce application experiencing high latency during checkout. With OTEL traces, you can get a lot of information about the performance of this service but it is hard to find out what is responsible for the latency. For example: https://github.com/esara/robot-shop/blob/instrumentation/dispatch/main.go#L172 

``` 

func processOrder(headers map[string]interface{}, order []byte) { 
    start := time.Now() 
    log.Printf("processing order %s\n", order) 
    tracer := otel.Tracer("dispatch") 
 
    // headers is map[string]interface{} 
    // carrier is map[string]string 
    carrier := make(propagation.MapCarrier) 
    // convert by copying k, v 
    for k, v := range headers { 
       carrier[k] = v.(string) 
    } 
 
    ctx := otel.GetTextMapPropagator().Extract(context.Background(), carrier) 
 
    opts := []oteltrace.SpanStartOption{ 
       oteltrace.WithSpanKind(oteltrace.SpanKindConsumer), 
    } 
    ctx, span := tracer.Start(ctx, "processOrder", opts...) 
    defer span.End() 
 
    span.SetAttributes( 
       semconv.MessagingOperationReceive, 
       semconv.MessagingDestinationName("orders"), 
       semconv.MessagingRabbitmqDestinationRoutingKey("orders"), 
       semconv.MessagingSystem("rabbitmq"), 
       semconv.NetAppProtocolName("AMQP"), 
    ) 

``` 

By exporting these traces to a backend like Tempo, engineers can visualize the request flow and identify bottlenecks, such as consuming messages from RabbitMQ in the dispatch service and inserting an order in a MongoDB database.  

The Big Data problem of observability 

OpenTelemetry’s ability to capture detailed telemetry data is a double-edged sword. While it empowers engineers with unprecedented visibility into their systems, it also introduces challenges that can hinder the very goals observability aims to achieve. The sheer volume of data collected—logs, metrics, and traces from thousands of microservices—can overwhelm infrastructure, slow down workflows, inflate costs, and most importantly drown engineers with data. This “big data problem” of observability is a natural consequence of OpenTelemetry’s strengths but must be addressed to make the most of its potential. 

OpenTelemetry collects a lot of data 

At its core, OpenTelemetry is designed to be exhaustive. This design ensures engineers can instrument their systems to capture every possible detail. For example: 

• A high-traffic e-commerce site might generate logs for every HTTP request, metrics for CPU and memory usage, and traces for each request spanning multiple services. 
• OpenTelemetry auto instrumentation libraries are an easy way to instrument HTTP, GRPC, messaging, database and caching libraries in all languages, but they generate metrics and traces for every call between every microservice, managed service, database and third-party API. 

Consider a production environment running thousands of microservices, each processing hundreds of requests per second. Using OpenTelemetry: 

• Logs: A single request might generate dozens of log entries, resulting in millions of logs per minute. 
• Metrics: Resource utilization metrics are emitted periodically, adding continuous streams of quantitative data. 
• Traces: Distributed traces can contain hundreds of spans, each adding its own metadata. 

While this level of detail is invaluable for debugging and optimization, it quickly scales beyond what many teams are prepared to manage. The amount of data makes it difficult to troubleshoot problems, manage escalations, be proactive about deploying new code, and plan for future investments. 

The cost of data 

The problem with this massive volume of telemetry data isn’t just about storage; it’s also about processing and time-to-insight. Let’s break it down: 

• Networking Costs: Transmitting telemetry data from distributed systems, microservices, or edge devices to central storage or processing locations incurs significant bandwidth usage. This can result in substantial networking costs, especially for real-time telemetry pipelines or when dealing with geographically dispersed infrastructure. 
• Storage Costs: Logs, metrics, and traces consume vast amounts of storage, often requiring specialized solutions like Elasticsearch, Amazon S3, or Prometheus’s TSDB. These systems must scale horizontally, adding significant operational overhead. 
• Compute Costs: Telemetry data needs to be parsed, indexed, queried, and analyzed. Complex queries, such as joining multiple traces to identify bottlenecks, can place a heavy burden on compute resources. 
• Time Costs: During a high-severity incident, every second counts. Pinpointing the root cause is like looking for a needle in a haystack. With OpenTelemetry, the haystack is much bigger, making the task harder and longer.  

Time-to-insight delays 

Imagine a scenario where an outage occurs in a distributed system. An engineer might start by querying logs for errors, then switch to metrics to identify anomalies, and finally inspect traces to pinpoint the failing service. Each query takes time, and engineers often waste effort chasing irrelevant leads. This delay increases Mean Time to Detect (MTTD) and Mean Time to Resolve (MTTR), directly impacting uptime and user satisfaction. 

Noise vs. signal 

Another challenge is separating the signal (useful insights) from the noise (redundant or irrelevant data). With OTEL: 

• Logs can be overly verbose, capturing routine events that clutter debugging efforts. 
• Metrics might lack the context needed to tie resource anomalies back to specific root causes. 
• Traces can become overwhelming in high-traffic systems, with thousands of spans providing more detail than is actionable. 

While OTEL excels at capturing data, it doesn’t inherently prioritize it. This creates a bottleneck for engineers who need actionable insights quickly. 

The need for top-down analytics 

Along with the benefits of modern observability tooling come challenges that need to be addressed. OpenTelemetry (OTEL) may unify telemetry data collection, but its bottom-up approach leaves teams drowning in redundant metrics, irrelevant logs, and sprawling traces. Without a clear purpose, teams end up collecting everything “just in case,” overwhelming engineers with noise and diluting the actionable insights needed to keep systems running. 

A top-down approach to observability flips the script. Instead of starting with what data is available, it begins with defining the goals: root cause analysis, SLO compliance, or performance optimization. By focusing on purpose, teams can build the analytics required to achieve those goals and then collect only the data necessary to power those insights.  

For example: 

• If the goal is root cause analysis, focus on traces that map dependencies across microservices, rather than capturing every granular log. 
• If the goal is performance optimization, prioritize metrics that highlight latency bottlenecks over exhaustive resource utilization data. 

This shift reduces noise, minimizes data storage and processing costs, and accelerates time-to-insight. 

The cost of ignoring purpose 

The current approach to observability is plagued by fragmentation. Point tools like APMs, native Kubernetes instrumentation, and cloud-specific monitors operate in silos, each with its own data model and semantics. This forces engineers to manually correlate information across dashboards, increasing time to resolution and undermining efficiency. Over time, the storage, compute, and human costs of managing fragmented data become unsustainable. 

Ask yourself: 

How much of your telemetry data is redundant or irrelevant? 

Are your engineers spending more time troubleshooting tools than resolving incidents? 

Is your observability stack delivering insights or merely adding complexity? 

Without a unified purpose and targeted analytics, observability becomes another “big data problem,” and your total cost of ownership (TCO) spirals out of control. 

Causely can help 

Causely transforms OpenTelemetry’s raw telemetry data into actionable insights by applying a top-down, purpose-driven approach. Instead of drowning in logs, metrics, and traces, Causely’s platform leverages built-in causal models and advanced analytics to automatically pinpoint root causes, prioritize issues based on service impact, and predict potential failures before they occur. This turns observability from a reactive big data challenge into a system that continuously assure application reliability and performance. 

How Causely brings focus 

Causely’s platform addresses these challenges head-on. Its causal reasoning starts with defining what matters: actionable insights to keep systems performing reliably and efficiently. Using built-in causal models and top-down analytics, Causely automatically pinpoints root causes and eliminates noise. By integrating with OTEL and other telemetry sources, Causely ensures that only the most critical data is collected, processed, and presented in real time. 

For example: 

• In a microservices architecture, Causely maps dependencies and pinpoints the root cause of cascading failures, reducing MTTR. 
• Similarly, with async messaging systems like Kafka, Causely pinpoints the bottlenecks that cause the consumer lag or delivery failures with actionable context, ensuring faster resolution. 
• In cases where a third-party software is a root cause of issues, Causely pinpoints the root cause   by analyzing services impact. 

This approach not only reduces the TCO of observability but also ensures teams can focus on delivering value rather than managing data. 

How Causely works with OpenTelemetry 

Causely Reasoning Platform is a model-driven, purpose-built Agentic AI system delivering multiple AI workers built on a common data model.   

Causely integrates seamlessly with OpenTelemetry, using its telemetry streams as input while applying context and intelligence to deliver precise, actionable outputs. Here’s how Causely solves common observability challenges: 

• Automated topology discovery: Causely automatically builds a dependency map of your entire environment, identifying how applications, services, and infrastructure components interact. OpenTelemetry’s traces provide raw data, but Causely’s topology discovery transforms it into a visual graph that highlights critical paths and dependencies.  
• Root cause analysis in real time: Using causal models, Causely automatically maps all potential root causes to the observable symptoms they may cause. Causely uses this mapping in real time to automatically pinpoint the root causes based on the observed symptoms, prioritizing those that directly impact SLOs. For instance, when  request latency spikes  are  detected across multiple services, Causely pinpoints whether the spikes stem from a database query (and which database), a messaging queue (and which queue), or an external API (and which one), reducing MTTD and MTTR. 
• Proactive prevention: Beyond solving problems, Causely helps prevent them. Its analytics can simulate “what-if” scenarios to predict the impact of configuration changes, workload spikes, or infrastructure upgrades. For example, Causely can warn you if scaling down a Kubernetes node pool might lead to resource contention under expected load. 

Example 1: Causely, OTEL, and microservices 

In a distributed e-commerce platform, a checkout service experiences intermittent failures. OpenTelemetry traces capture the flow of requests, but the data alone doesn’t explain the root cause. Causely’s causal models analyze the traces and identify that a dependent payment service is timing out due to a slow database query. This insight allows the team to address the issue without wasting time on manual debugging. 

Example 2: Causely, OTEL, and third-party software 

A team - using a third-party CRM API - notices degraded response times during peak hours. OpenTelemetry provides metrics showing increased latency, but engineers are left guessing whether the issue lies with their application or the external service. Causely reasons about the API latency and third-party requests and identifies that the CRM is rate-limiting requests, prompting the team to implement retry logic. 

Example 3: Causely, OTEL, and async messaging with Kafka 

A Kafka-based event pipeline shows sporadic delays in message processing. While OpenTelemetry traces highlight lagging consumers, it doesn’t explain why. Causely, reasoning about the behavior of the consumer microservices, identifies the root cause in the application’s mutex locking which is causing the slow consumption. The engineering team can focus on improving the locking of the data structure without the messaging infrastructure team having to scale up resources and waste time debugging Kafka. 

Reducing the big data burden 

Causely’s approach minimizes the data burden by focusing on relevance. Unlike traditional observability stacks that collect and store massive volumes of telemetry data, Causely processes raw metrics and traces locally, pushing only relevant context (e.g., topology and symptoms) to its backend analytics. This reduces storage and compute costs while ensuring engineers get the insights, they need, without delay. 

Conclusion: Transforming observability with Causely 

OpenTelemetry has redefined observability by standardizing how telemetry data is collected and processed, but its bottom-up approach leaves teams overwhelmed by the sheer volume of logs, metrics, and traces. Observability shouldn’t be about how much data you collect—it’s about how much insight you can gain to keep your systems running efficiently. Without clear prioritization and contextual insights, the observability stack can quickly become a costly burden—both in terms of infrastructure and engineering time. 

Causely integrates seamlessly with OpenTelemetry and helps bring order to the chaos, empowering teams to make smarter, faster decisions that directly impact reliability and user experience. Causely uses causal models, automated topology discovery and real-time analytics to pinpoint root causes, prevent incidents, and optimize performance. This reduces noise, eliminates unnecessary data collection, and allows teams to focus on delivering reliable systems rather than managing observability overhead. 

Ready to move beyond data overload and transform your observability strategy? Book a demo or start your free trial to see how Causely can help you take control of your telemetry data and build more reliable cloud-native applications. 

As an industry, we have made progress in many areas. But when it comes to technology operations and application management fundamentals, we are far from the desired state. No matter what new terms and buzz words people use, the industry still has a way to go. The reality is that no magic black box or new AI trend will get us there on its own. 

Autonomous Service Reliability requires a system that autonomically keeps all applications performing and meeting their objectives while satisfying their operational constraints (i.e. the “Desired State”). To continuously maintain the Desired State, the system needs to:  

• Assess whether all the applications are in their Desired State 
• Pinpoint the root cause(s) and identify the actions that will get the applications that are not in their Desired State back to their Desired State  
• Determine what actions will prevent applications from getting out of their Desired State in the first place 
• Continuously assess environment trends to identify what actions should be taken to prevent deviation from the Desired State 

These are the goals of our Causal Reasoning Platform, which is a model-driven, purpose-built Agentic AI system that includes multiple AI Workers built on a common data model. These AI Workers collaborate seamlessly to continuously assure application reliability and performance. Each of these workers utilizes specific analytics and they all share the common components of the Causal Reasoning Platform.  

There are seven AI Workers delivered by our Causal Reasoning Platform, sharing six common components. In this post, we’ll first introduce the six common components followed by a description of the seven workers, explaining what each worker is doing and how the platform works.  

Causal Models

The Causal Reasoning Platform is driven by Causal Models. Causely is delivered with built-in Causal Models that capture the root causes that can degrade application performance. These Causal Models enable Causely to automatically pinpoint root causes as soon as it is deployed in an environment with zero configuration.

There are at least a few important details to highlight about these Causal Models:  

• They capture potential root causes in a broad range of entities including applications, databases, caches, messaging, load balancers, DNS, compute, storage, and more. 
• They describe how the root causes will propagate across the entire environment and what symptoms may be observed when each of the root causes occurs.  
• They are completely independent from any specific environment and are applicable to any modern application environment.  

Attribute Dependency Models

Causely is delivered with built-in Attribute Dependency Models that extend the Causal Models to capture the dependencies between attributes across entities and the constraints attributes must satisfy. These Attribute Dependency Models enable Causely to automatically correlate performance trends across the entire environment, figure out the Desired State (as described earlier) and the actions to keep the environment in that state. 

There are at least a few important details to highlight about these Attribute Dependency Models:  

• They can capture attribute dependencies in a broad range of entities including services, applications, databases, caches, messaging, load balancers, DNS, compute, storage, and more. 
• They describe the functions between the attributes, but more importantly the functions can be learned.  
• They describe the desired state in terms of the applications' goals and the constraints they should operate within.  
• They are completely independent from any specific environment and are applicable to any modern application environment.

Automatic Topology Discovery

Cloud-native environments are a tangled web of applications and services layered over complex and dynamic infrastructure. Causely automatically discovers all the entities in the environment including the applications, services, databases, caches, messaging, load balancers, compute, storage, etc., as well as how they all relate to each other.

For each discovered entity, Causely automatically discovers its:  

• Connectivity - the entities it is connected to and the entities it is communicating with horizontally  
• Layering - the entities it is vertically layered over or underlying 
• Composition - what the entity itself is composed of  

Causely automatically stitches all of these relationships together to generate a Topology Graph, which is a clear dependency map of the entire environment. This Topology Graph updates continuously in real time, accurately representing the current state of the environment at all times. 

Automatic Causality Mapping Generation

Using the out-of-the-box Causal Models and the Topology Graph as described above, Causely automatically generates a causal mapping between all the possible root causes and the symptoms each of them may cause, along with the probability that each symptom would be observed when the root cause occurs. 

Causely automatically generates two data structures to capture the causality mapping:

• A Causality Graph is a directed acyclic graph (DAG), where the nodes are root causes and symptoms and the edges represent the causality, i.e., an edge from node A to node B means that A may cause B. The edges are labeled with the probability of the causality.
• A Codebook is a table where the columns represent the root causes and the rows represent the symptoms. Each column is a vector of probabilities defining a unique signature of the root cause. A cell in the vector represents the probability that the root cause may cause the symptom.

Automatic Attribute Dependency Graph Generation

Using the out-of-the-box Attributes Dependency Model and the Topology Graph as described above, Causely automatically generates an Attribute Dependency Graph. 

The Attribute Dependency Graph is a directed acyclic graph (DAG) where:

• The  nodes are attributes.
• The edges represent a dependency between the attributes. For example, an edge from attribute A to attribute B means that the value of B is a function of attribute A. 
• The edges are labeled with the functions. The functions can be defined in the Attributes Dependency Model or can be learned if they are not defined in the Model. 
• The nodes representing attributes that must satisfy a constraint will be decorated with the constraint the attribute must satisfy. 

Contextual Presentation

We believe explainability is a critical capability for AI-driven systems to demonstrate. For this purpose, the system presents its work intuitively in the Causely UI. This enables users to see the root causes, related symptoms, the service impacts and initiate actions. These insights can also be sent to external systems to initiate incident response workflows as well as to notify teams who are responsible for taking action and/or those whose services are impacted.

The Models, the automated topology discovery, and the automatic generation of the Causality Mapping and the Attribute Dependency Graph empower multiple AI workers that together deliver an autonomous application reliability system that continuously assures application performance.

Root Cause Analysis (RCA) Worker

The RCA Worker uses the Codebook described above to automatically pinpoint root causes based on observed symptoms in real time. No configuration is required for the worker to immediately pinpoint a broad set of root causes (100+), ranging from applications malfunctioning to services congestion to infrastructure bottlenecks.  

In any given environment, there can be tens of thousands of different root causes that may cause hundreds of thousands of symptoms. Causely prevents SLO violations by detangling this mess and pinpointing the root cause putting your SLOs at risk and driving remediation actions before SLOs are violated. For example, Causely proactively pinpoints if a software update changes performance behaviors for dependent services before those services are impacted. 

Performance Analysis Worker

The Performance Analysis Worker uses the Attribute Dependency Graph and Causality Graph to analyze microservices performance bottleneck propagation by automatically learning, based on your data:  

• The correlation between the loads on services, i.e., how a change in load of one cascades and impacts the loads on other services; 
• The correlation between services latencies, i.e., how latency of one cascades and impacts the latencies of other services; and  
• The likelihood a service or resource bottleneck may cause performance degradations on dependent services. 

Constraint Analysis Worker

The Constraint Analysis Worker uses the Attribute Dependency Graph decorated with performance goals like throughput and latency, as well as capacity and/or cost constraints, to automatically compute the Desired State of the environment and to figure out what actions need to be taken to assure the goals are accomplished while satisfying the defined constraints.  

Prevention Analysis Worker

The Prevention Analysis Worker uses the Causality Graph and the Attribute Dependency Graph to enable prevention analysis. Teams are empowered to analyze the potential impacts or problems of changes.

Teams can ask "what if'' questions to:

• Understand the services that may be degraded if a potential problem were to occur
• Understand the impact a planned change may have on services

In doing so, teams can support planning of service/architecture changes, maintenance activities, and service resiliency improvements, and assure that none of these cause unexpected outages that may dramatically impact the business.

Predictive Analysis Worker

The Predictive Analysis Worker uses machine learning (ML) together with the Causality Graph and the Attribute Dependency Graph for predictive analysis. Causely uses:

• ML to analyze the performance behavior of a small subset of attributes, e.g. some services loads, to predict their trends.
• The Attribute Dependency Graph and the predicted trends to predict the state of the environment, i.e., the state of all the attributes.
• The Causality Graph and the predicted future state to pinpoint potential bottlenecks and suggest actions that may prevent bottlenecks.

In doing so, Causely pinpoints the actions required to prevent future degradations, SLO violations, or constraint violations.

Service Impact Analysis Worker

The Service Impact Analysis Worker uses the Causality Graph to automatically analyze the impact of the root causes on SLOs, prioritizing the root causes based on the violated SLOs and those that are at risk. Causely automatically defines standard SLOs (based on latency and error rate) and uses machine learning to improve its anomaly detection over time. However, environments that already have SLO definitions in another system can easily be incorporated in place of Causely’s default settings. 

Postmortem Analysis Worker

The Postmortem Analysis Worker uses the Causality Graph to save the relevant context of prior incidents to enable postmortem analysis. Causely saves the root cause, the Causality Graph of the root cause, the symptoms in the Causality Graph and the relevant attribute trends. Teams can review prior incidents and see clear explanations of why these occurred and what the effect was, simplifying the process of postmortems and enabling actions to be taken to avoid re-occurrences.  

See Causely for Yourself!

Book a meeting with the Causely team and let us show you how to transform the state of escalations and cross-organizational collaboration in cloud-native environments, or start your free trial now. 

Collecting “more data” has been the defining characteristic of observability practices and tools for the last few decades. The allure of multiple tools that instrument virtually every layer of the stack, from applications to infrastructure and beyond, traps us into believing that more data equals more insights. But the reality is often the opposite: over-collection creates inefficiencies, noise, and cost without adding meaningful value. As I said during a recent podcast interview, it may make you less blind, but not necessarily smarter.   

Instead of clarifying what’s happening in your systems, excessive data leads to confusion. Teams are inundated with redundant metrics, irrelevant logs, and false-positive alerts. Each new piece of data requires engineers to interpret, process, and correlate, pulling focus away from actionable insights. As a result, observability becomes a “big data” problem, where the volume of information collected outweighs its usefulness. 

We always collected too much data, but microservices architectures and the new emerging instrumentation, eBPF and OpenTelemetry, exacerbate the problem. With tens of thousands of microservices, each collecting even a few metrics every few seconds, adds up very quickly to millions of data points requiring overloaded processing and terabytes of storage. The realization that we need application instrumentation to get insights about the higher layer on the stack and the interactions between services led to the adoption of eBPF and OpenTelemetry, but with these came a tsunami of traces that overwhelms observability tools and makes them practically useless. 

This trajectory must and can be changed. 

When collecting data offers diminishing returns 

Collecting data turns observability into a “big data problem” instead of its intended purpose: to keep systems up and running. As a result, you don’t simply apply a band-aid to a bullet hole, you end up applying band-aids upon band-aids upon band-aids. Each new big data problem results in more big data solutions, which result in more big data problems, and the cycle continues. 

Lack of purpose creates noise 

Observability data is often collected without a clear sense of purpose. Teams instrument everything they can, thinking it might be useful someday, but this creates a flood of irrelevant information. The result is alert fatigue, where low-priority or false-positive alerts drown out the critical signals engineers need to act on. In Kubernetes environments, for example, native instrumentation combined with third-party tools often generates more metrics than teams can effectively use, adding complexity instead of clarity. 

Earlier this week, we wrote about the difficulties of troubleshooting problems in production. Excessive data only makes the haystack larger; it doesn’t help you find the needle any faster. 

Ask yourself: what percentage of the collected and stored data is ever being looked at? What is the cost of processing and storing this data? 

Fragmentation of observability tools 

Observability is highly fragmented, with different tools collecting and processing data in silos. Each technology or layer of the application stack is monitored by a point tool that monitors that specific technology or layer. The applications may be monitored by an APM tool like Datadog, New Relic, or Dynatrace. The services layer may be monitored by eBPF and OpenTelemetry. The Kubernetes infrastructure may be monitored by the native Kubernetes instrumentation. Cloud services may be monitored by cloud provider’s tools like CloudWatch.  And the list goes on. 

Some of these tools collect metrics, some collect traces, and others ingest logs or events. Regardless of whether these point tools come from different vendors or are provided by a single vendor, each one of them has its own data model. They lack not only a unified data model, but a common semantic model to interpret the data to analyze the data across the entire stack and to provide actionable insights. Engineers are left to manually correlate information across dashboards, increasing time to resolution. This fragmentation creates inefficiencies that undermine the very purpose of observability: delivering actionable insights. 

Ask yourself:

How many different point tools are you using?
Which one do you use when and why? 
How much of the data is redundant? 

Overhead of storing and processing data 

Storing and analyzing vast amounts of observability data comes at a cost—both financial and operational. Many tools charge by data volume, making it cost-prohibitive to retain all collected data. Even for teams with generous budgets, processing and querying such large datasets consumes time and resources, often delivering diminishing returns. 

Ask yourself:

Are you in the business of collecting observability data or are you in the business of keeping your systems running reliably and efficiently? 

A top-down approach to observability 

 The solution isn’t more data—it’s the right data. A top-down approach flips the traditional model of observability on its head. Instead of starting with what data is available, start with the purpose: what are you trying to achieve?

Define the goals first—whether it’s root cause analysis, SLO compliance, or performance optimization—and then work down to identify the data needed to support those goals. 

Building the right analytics is critical. Analytics should be purpose-built to deliver insights that directly address the goals you’ve defined. For example, if the goal is to identify bottlenecks, focus on analytics that analyzes service latencies, instead of drowning in utilization metrics.   

Finally, only collect the data you need. With the purpose and analytics in place, you can focus on gathering metrics and logs that directly contribute to actionable insights. This reduces noise, eliminates unnecessary alerts, and lowers the overhead of storing and processing data. 

Causely helps you focus on what matters 

Our Causal Reasoning Platform is a model-driven, purpose-built AI system delivering multiple analytics built on a common data model. It is designed to minimize data collection while maximizing insight. (You can learn more about how it works here).

As a result, Causely is laser focused on collecting only the metrics, traces, and logs required as input for the above analytics. This dramatically reduces the amount of data collected by Causely. Furthermore, all the raw metrics and traces are processed locally in real time, and none if it is pushed out to the cloud. The only information pushed to the back-end analytics in the cloud is the topology and symptoms. The only data stored in the cloud are the pinpointed root causes and the contextual information associated with the root causes. This minimizes Causely's TCO! 

Conclusion 

Collecting more data doesn’t guarantee better observability. In fact, it often creates more problems than it solves. A top-down approach—starting with purpose, building targeted analytics, and focusing on necessary data—streamlines observability and makes it actionable. Platforms like Causely enable this approach, helping teams move beyond the big data trap and focus on delivering real value. Observability isn’t about how much data you collect—it’s about how much insight you can gain. 

This week, we covered numerous ways teams can shift their posture from reactive infrastructure fixes to more proactive observability. Organizations that are more proactive will enjoy several benefits, including: 

• Spending less time troubleshooting 
• Preventing escalations from devolving into finger-pointing and blame 
• Predicting the impact of changes before they are deployed 
• Planning for future environment and infrastructure changes 

Book a meeting with the Causely team and let us show you how you can bridge the gap between development and observability to build better, more reliable cloud-native applications. 

Yesterday, we talked about being more proactive about the code our developers write, anticipating how code changes will impact our systems and infrastructure more broadly. Now, we turn our attention to planning ahead for external factors that could affect our systems.  

Environmental changes are inevitable in any complex system. Whether it’s a sudden traffic spike, an unexpected hardware failure, or gradual infrastructure degradation, external pressures can quickly destabilize even the most carefully designed environments. The challenge lies in anticipating these changes before they disrupt services. In distributed systems, where small disruptions can ripple across dependencies, waiting until issues surface is simply not an option. 

Proactive observability provides a necessary safeguard against these challenges. By identifying potential risks in real time, predicting future demand, and adapting resources dynamically, teams can maintain reliability even under extreme conditions. This isn’t about eliminating unpredictability; it’s about building systems that respond intelligently to it. 

Challenges to being more proactive about environment changes 

Three core challenges make proactive management of environmental changes difficult: predicting load trajectories, gaining forward-looking visibility into the broader system, and scaling resources efficiently. While various tools exist to address parts of these problems, they often fall short in providing the level of predictability, visibility, and adaptability that modern systems require. 

Predicting future load is difficult 

Predicting how traffic patterns will evolve is one of the most complex aspects of managing modern systems. Seasonal events, marketing campaigns, or even unforeseen surges can create spikes in demand that exceed system capacity.  

When managing an increased number of users or handling heavier loads, developers must evaluate how the system handles scaling demands. This involves monitoring how many API calls are made to backend services for each user request and modeling the anticipated growth in API usage based on projected user activity. Monitoring tools can provide real-time data to understand current usage patterns, but they cannot predict the behavior of dependent services. Load-testing tools like Grafana Labs' k6 and Apache JMeter are commonly used during development to simulate expected traffic patterns. These tools are useful for pre-deployment stress testing but are static by design—they can’t account for real-time changes in traffic behavior or external variables like regional events or user behavior shifts. As a result, engineers often find themselves underprepared, scrambling to adjust capacity once demand has already outpaced infrastructure. 

Forward-looking visibility into evolving systems is limited 

Proactively managing environmental changes requires visibility not just into what’s happening now but into how the system is likely to evolve. For instance, if a service is handling a steadily increasing workload, engineers need to predict how soon it will hit a performance bottleneck and which dependent services may be affected.  

For systems using messaging architectures, such as event-driven designs, it’s crucial to evaluate the rate at which messages are published and consumed. Developers need to assess whether consumers can process messages fast enough to keep up with the increased production rate. This may involve checking the throughput of message queues (e.g., Kafka, RabbitMQ) and ensuring sufficient worker threads or instances are available to process the messages without creating backlogs. 

Database performance is another critical factor. As the number of user requests grows, so does the frequency of database queries. Developers must analyze the types of queries executed, the tables affected, and whether the database schema or indexes need optimization. Load testing can simulate increased query volume to predict the impact on database performance and resource usage. This analysis helps in determining whether the existing database can handle the load with vertical or horizontal scaling or if architectural changes, such as database sharding or read replicas, are necessary. 

Current tools often provide snapshots of real-time metrics but lack the ability to project those metrics forward into the future. 

Datadog and New Relic offer dashboards that display system dependencies and performance under current conditions, but they focus on reactive analysis. Similarly, Prometheus provides excellent real-time monitoring for resource metrics like CPU or memory utilization but does not include predictive capabilities for identifying future stress points.  

This lack of forward-looking insight forces engineers into a reactive posture, leaving them vulnerable to issues that could have been anticipated and avoided. 

Scaling resource efficiency is complicated 

Scaling infrastructure to meet demand is both a reliability and cost challenge. Reactive scaling, where systems respond only after resource thresholds are exceeded, risks under-provisioning during critical moments, leading to downtime. On the other hand, over-provisioning wastes resources and drives up operational costs.  

Tools like Kubernetes Horizontal Pod Autoscaler (HPA) and AWS Auto Scaling automate scaling based on resource metrics like CPU or memory utilization. While these tools are helpful, they are fundamentally reactive—they adjust only after usage patterns have shifted. They lack the predictive capabilities to anticipate future demand and scale resources proactively, leaving teams vulnerable to sudden surges or inefficiencies. 

What we require to be more proactive 

Proactively managing environmental changes requires observability systems that go beyond traditional monitoring and alerting. These systems must not only detect anomalies in real time but also provide predictive insights and adaptive workflows that empower teams to anticipate and mitigate risks before they impact users. Here are the critical capabilities needed to achieve this: 

• Real-time anomaly detection at scale. Identifying traffic surges, resource bottlenecks, or unexpected failures as they occur. Systems should process vast amounts of data with minimal latency, identifying deviations from normal patterns. Early detection prevents small issues from spiraling into widespread outages. 
• Predictive analytics for load and capacity needs. Beyond real-time data, teams also need analytics that use historical and live data to forecast future system behavior. These systems should forecast where and when additional capacity will be needed, including across long-term patterns such as seasonal demand spikes. Predictive insights allow teams to allocate resources proactively. 
• Integrated resource management. Armed with data about what to do, systems should be deeply integrated so that teams can act quickly on their predictive knowledge and act. 
• Comprehensive system modeling. Teams need the ability to test the impact of environmental changes before they happen. This includes simulating traffic surges, dependency failures, or configuration changes in a controlled environment to identify potential risks and validate resilience. 
• Clear, actionable insight. Systems must provide insights that are not only detailed but also actionable. This includes translating complex system behavior into clear recommendations that engineers and decision-makers can act on quickly. Think context-rich alerts and dashboards that highlight the root cause of anomalies, the potential impact of changes, and the recommended actions to mitigate it all. 

Achieving true proactive observability requires more than just reacting to anomalies. It demands systems that predict, adapt, and empower teams to address environmental changes before they become critical. With these capabilities in place, organizations can ensure their systems remain stable and reliable, no matter what external pressures arise. 

Causely delivers a proactive observability system 

Our Causal Reasoning Platform is a model-driven, purpose-built AI system delivering multiple analytics built on a common data model. You can learn more about its capabilities that uniquely help teams proactively anticipate and plan for environment changes here.

Conclusion 

Environmental changes like traffic spikes or hardware degradation are inevitable, but their impact doesn’t have to mean downtime. Reacting to problems after they occur leaves teams in firefighting mode, scrambling to restore stability. The key to reliability isn’t just responding quickly—it’s anticipating risks and adapting before issues arise. 

Proactive observability makes this possible. With real-time anomaly detection, predictive analytics, and adaptive workflows, teams can prevent disruptions and maintain stability under changing conditions. By adopting tools and practices that prioritize foresight over reaction, engineers can shift from firefighting to proactive system management, ensuring their systems remain reliable no matter what comes next. 

So far this week, we’ve talked about the difficulties involved with troubleshooting and the complexity of dealing with escalations. What if we could head off these problems before they even happen? 

Making changes to production environments is one of the riskiest parts of managing complex systems. Even a small, seemingly harmless tweak—a configuration update, a database schema adjustment, or a scaling decision—can have unintended consequences. These changes ripple across interconnected services, and without the right tools, it’s nearly impossible to predict their impact. 

Business needs speed and agility in introducing new capabilities and features but without scarifying reliability, performance and predictability. Hence, engineers need to know how a new feature, or even a minor change, will affect performance, reliability, and SLO compliance before it goes live. But existing observability tools lack the required capabilities to provide useful insights that enable engineers to safely deploy new features. The result? Reduced productivity, slowdown in feature development, and reactive firefighting when changes go wrong -- all leading to downtime, stress, and diminished user trust. 

Recently, we worked with a customer who sought to shift their reactive posture to one that more aggressively seeks out problems before they happen.  With this customer, a bug within one of their microservices (the data producer) caused the producer to stop updating the Kafka topic with new events.  This created a backlog of events for all the consumers of this topic.  As a result, their customers were looking at stale data.  Problems like this lead to poor customer experience and revenue loss, so many customers need to adopt a preventative mode of operations.

This post explores how this trend can be disrupted by providing the analytics and the reasoning capabilities to transform how changes are made, empowering teams to anticipate risks, validate decisions, and protect system stability—all before the first line of code is deployed. 

Being proactive is easier said than done 

Change management is a process designed to ensure changes are effective, resolve existing issues, and maintain system stability without introducing new problems. At its core, this process requires a deep understanding of both the system’s dependencies and its state before and after a change.  

Production changes are risky because engineers typically lack sufficient visibility into how changes will impact the behavior of entire systems. What seems like an innocuous change to an environment file or an API endpoint could have far-reaching ramifications that aren’t always obvious to the developer. 

While observability tools have come a long way in helping teams monitor systems, they lack the analytics required to understand, analyze, and predict the reliability and performance behavior of cloud-native systems. As a result, engineers are left to “deploy and hope for the best” ... and get a 3AM call when things didn’t work as expected. And, while we live in a veritable renaissance of developer tooling, most of these tools focus on developer productivity, not developer understanding, of whole systems and the consequences of changes made to one component or service. 

It’s hard to predict the impact of code changes 

When planning a change, the priority besides adding new functionality is to confirm whether it addresses the specific service degradations or issues it was designed to resolve. Equally important is ensuring that the change does not introduce new regressions or service degradations. 

Achieving these goals requires a comprehensive understanding of the system’s architecture, particularly the north-south (layering) dependencies and the east-west (service-to-service) interactions. Beyond mapping the topology, it is crucial to understand the data flow within the system—how data is processed, transmitted, and consumed—because these flows often reveal hidden interdependencies and potential impact areas.  

Even minor configuration changes can create cascading failures in distributed systems. For instance, adjusting the scaling parameters of an application might inadvertently overload a backend database, causing performance degradation across services. Engineers often rely on experience, intuition, or manual testing, but these methods can’t account for the full complexity of modern environments. 

Unpredictable performance behavior of microservices  

As we discussed in our post yesterday, loosely coupled microservices communicate with each other and share resources. But which services depend on which? And what resources are shared by which services? These dependencies are continuously changing and, in many cases, unpredictable.  

A congested database may cause performance degradations of some services that are accessing the database. But which one will be degraded? Hard to know. Depends. Which services are accessing which tables through what APIs? Are all tables or APIs impacted by the bottleneck? Which other services depend on the services that are degraded? Are all of them going to be degraded?  

These are very difficult questions to answer. As a result, analyzing, predicting, and even just understanding the performance behavior of each service is very difficult. Furthermore, using existing brittle observability tools to diagnose how a bottleneck cascades across services is practically impossible.  

There’s a lack of “what-if” analysis tools for testing resilience 

Even though it’s important to simulate and test the impact of changes before deployment, the tools currently available are sorely lacking. Chaos engineering tools like Gremlin and Chaos Monkey simulate failures, but don’t evaluate the impact of configuration changes. Tools like Honeycomb provide event-driven observability, but don’t help much with simulating what will happen with new builds. Inherently, if the tools can’t analyze the performance behavior of the services, they can’t support any “what-if” analysis. 

Developer tools are inherently focused on the “build and deploy” phases of the software lifecycle, meaning they prioritize pre-deployment validation over predictive insights. They don’t provide answers to critical questions like: “How will this change impact my service’s reliability or my system’s SLOs?” or “Will this deployment create new bottlenecks?” 

Predictive insights require correlating historical data, real-time metrics, dependency graphs, and most importantly deep understanding of the microservices performance behaviors. Developer tools simply aren’t built to ingest or analyze this kind of data at the system level. 

Developer and operations tools today are both insufficient 

Developer tools are essential for building functional, secure, and deployable code, but they are fundamentally designed for a different domain than observability. Developer tools focus on ensuring “what” is built and deployed correctly, while observability tools aim to identify “when” something is happening in production. The two domains overlap but rarely address the full picture. 

 Bridging this gap often involves integrating developer workflows—such as CI/CD pipelines—with observability systems. While this integration can surface useful metrics and automate parts of the release process, it still leaves a critical blind spot: understanding “why” something is happening. Neither traditional developer tools nor current observability platforms are designed to address the complexity of dynamic, real-world systems. 

To answer the “why,” you need a purpose-built system to unravel the interactions, dependencies, and behaviors that drive modern production environments. 

Building for reliability 

Building reliable performing applications was never easy, but it has become much harder. As David Shergilashvili correctly states in his recent post Microservices Bottlenecks, “In modern distributed systems, microservices architecture introduces complex performance dynamics that require deep understanding. Due to their distributed nature, service independence, and complex interaction patterns, microservices systems' performance characteristics differ fundamentally from monolithic applications.” 

Continuing to collect data and presenting it to developers in pretty dashboards with very little or no built-in analytics to provide meaningful insights won’t get us to build reliable distributed microservices applications. 

To accurately assess the impact of a change, the state of the system must be assessed both before and after the change is implemented. This involves monitoring key indicators such as system health, performance trends, anomaly patterns, threshold violations, and service-level degradations. These metrics provide a baseline for evaluating whether the change resolves known issues and whether it introduces new ones. However, the ultimate goal goes beyond metrics; it is to confirm that the known root causes of issues are addressed and that no new root causes emerge post-change.  

We need to build systems that enable engineers to introduce new features quickly, efficiently, and most importantly safely, i.e., without risking the reliability and performance of their applications. Reasoning platforms with built-in analytics need to provide actionable insights that anticipate implications and prevent issues. These systems must have the following capabilities: 

• Services dependencies. Be able to capture, represent and auto-discover service dependencies. Without this knowledge a developer can’t know and anticipate which services may be impacted by a change. 
• Causality knowledge. Be aware of all potential failures and the symptoms they may cause. Without these insights, a developer can’t assess the risk of introducing a change. Is the application code I am about to change a single point of failure? What will happen if for some reason my new code won’t perform as expected? 
• Performance analysis. Understanding the blast radius of performance bottlenecks. Without having a deep understanding of the interactions between the microservices as well as their shared resources, developers can’t predict all possible situations that may occur in production. Even the best tester can’t cover all possible scenarios. How can you know how a noisy neighbor will impact your application in production?  
• What-if. The ability to assess load fluctuations before they happen or potential impacts of code changes to be introduced. Is your service ready for the predicted Black Friday traffic? What about the dependent upstream service? 

Causely can help you build better cloud-native applications 

Our Causal Reasoning Platform is a model-driven, purpose-built AI system delivering multiple analytics built on a common data model. You can learn more about what makes it unique here.

With Causely, you can go from reactive troubleshooting to proactive design, development, and deployment of reliable cloud-native applications. 

Conclusion 

Making changes to production environments shouldn’t be a guessing game. Without the right systems, even the smallest change can introduce risks that compromise reliability, breach SLOs, and erode user trust. Reactive approaches to troubleshooting only address problems after they occur, leaving engineers stuck firefighting rather than innovating. 

A purpose-built reasoning platform can flip this narrative. By incorporating predictive analytics, “what-if” modeling, and automated risk detection, teams can anticipate issues before they arise and deploy with confidence. Platforms like Causely empower engineers with the insights they need to write reliable code, validate changes, and ensure system stability at every step. 

Book a meeting with the Causely team and let us show you how you can bridge the gap between development and observability to build better, more reliable cloud-native applications. 

Microservices architectures introduce complex, dynamic dependencies between loosely coupled components. In turn, these dependencies lead to complex, hard to predict interactions. In these environments, any resource bottleneck, or any service bottleneck or malfunction, will cascade and affect multiple services, crossing team boundaries. As a result, the response often spirals into a chaotic mix of war rooms, heated Slack threads, and finger-pointing. The problem isn’t just technical—it’s structural. Without a clear understanding of dependencies and ownership, every team spends more time defending their work than solving the issue. It’s a waste of effort that undermines collaboration and prolongs downtime. 

Yesterday, we resolved to spend less time troubleshooting in 2025. 

Troubleshooting and escalation are closely intertwined. A single unresolved bottleneck can ripple outward, forcing multiple teams into reactive mode as they struggle to isolate the true root cause. This dynamic creates inefficiencies and delays, with teams often focusing on band-aiding symptoms instead of remediating and solving the root causes. To eliminate this friction, we need systems that do more than detect anomalies—they must provide a seamless view of dependencies, understand and analyze the performance behaviors of the microservices, assign ownership intelligently, and guide engineers toward resolution with precision and context. 

Take, for example, an application developer who notices high request duration for users who are trying to interact with their application. This application communicates with many different services, and it happens to run within a container environment on public cloud infrastructure.  There are more than 50 possible root causes that might be causing the high request duration issue.  That developer would need to investigate garbage collection issues, disk congestion, app-locking problems, and node congestion among many other potential root causes until accurately determining that a congested database is the source of their problem.  The only proper way to determine root cause is by considering all the cause-and-effect relationships between all the possible root causes and the symptoms they may cause. This process can often take hours or days before the correct root cause is pinpointed, resulting in a variety of business consequences (unhappy users, missed SLOs, SLA violations, etc.). 

In this post, we’ll explore the challenges of multi-team escalations, and the capabilities needed to address them. From automated dependency mapping to explainable triage workflows, we’ll show how observability can be transformed from chaos into clarity, making escalations less contentious and far more productive. 

Escalations can cripple teams 

Escalations create inefficiencies that extend downtime, frustrate teams, and waste resources. These inefficiencies stem from a combination of structural and technical gaps in how dependencies are understood, root causes are isolated, and ownership is assigned. Here are some of the key challenges that make escalations so painful today: 

• There is a lack of cross-team visibility into dependencies 
• It can be hard to predict or analyze the performance behaviors of loosely coupled dependent microservices  
• It can be difficult to isolate the root cause among all affected services 
• Legacy observability tools must be stitched together to provide even partial visibility into issues 

Lack of cross-team visibility 

Microservices architectures are complex and full of deeply interconnected components. An issue in one can cascade into others. Without clear visibility into these dependencies, teams are left guessing which components are impacted and which team should take ownership. 

Your favorite observability tools help you visualize dependencies, but they lack real-time accuracy. These maps can quickly become outdated in environments with frequent changes. Some of them are great for aggregating logs, but don’t offer much insight into service relationships. Engineers are often left to piece together dependencies manually. 

Unpredictable performance behavior of microservices  

Loosely coupled microservices communicate with each other and share resources. But which services depend on which? And what resources are shared by which services? These dependencies are continuously changing and, in many cases, unpredictable.  

A congested database may cause performance degradations of some services that are accessing the database. But which one will be degraded? Hard to know. Depends. Which services are accessing which tables through what APIs? Are all tables or APIs impacted by the bottleneck? Which other services depend on the services that are degraded? Are all of them going to be degraded? These are very difficult questions to answer.  

As a result, predicting, understanding and analyzing the performance behavior of each service is very difficult. Using existing brittle observability tools to diagnose how a bottleneck cascades across services is practically impossible.  

Difficulty identifying root causes among all affected services 

Determining what’s a cause and what’s a symptom can be an incredibly time-consuming aspect of troubleshooting and escalations. Further, the person or team identifying a problem may well be looking at only their local maxima: the part of the system they work on or are directly affected by. They often don’t see the full picture of all intertwined systems. Identifying the root cause among all affected services can be inordinately difficult. 

Even if you have tools that are excellent for visualizing time-series data, you must still rely on engineers to manually correlate metrics. APM tools can help you examine application performance but require significant manual effort to link symptoms to underlying causes, especially in microservices-based, cloud-native applications. 

Legacy observability tooling only gives you partial functionality 

While both established and up-and-coming tools offer valuable capabilities, they often address only one part of the problem, leaving critical gaps. Dependency visibility, performance analysis and root cause isolation need to be integrated seamlessly to reduce the chaos of escalations. Today’s tools, however, are fragmented, requiring engineers to bridge the gaps manually, costing valuable time and effort during incidents. Solving these problems demands a holistic approach that ties all these elements together in real time. 

How escalations should be handled 

Escalations have negative consequences for organizations of all sizes. Let’s work together to build systems that render escalations less about blame and more about opportunities to foster trust and collaboration. These systems will have the following capabilities: 

• End-to-end discovery of service dependencies. Automatically discover and maintain, in real time, a complete view of how systems interact. 
• Workflow integration that directs root cause resolution to the correct team. Use the tools your organization has already invested in to turn root causes into actions for the correct team, reducing delays caused by miscommunication. 
• Performance analysis of bottlenecks propagations. Provide insights on how bottlenecks cascade across services. 
• Detailed identification of root causes across an entire system. Empower engineers to act confidently without over-reliance on senior team members. 

With these new systems, escalations can result in positive business outcomes: 

• Deep, real-time understanding of complete microservices systems 
• Better collaboration between teams to remediate issues as they arise 
• More innovation, less time in war rooms 
• Empowered engineering teams that solve problems instead of pointing fingers 

Causely helps you handle escalations quickly and confidently 

Our Causal Reasoning Platform is a model-driven, purpose-built AI system delivering multiple analytics built on a common data model. It includes several features to help you understand issues and handle escalations efficiently: 

• Out-of-the-box Causal Models. Causely is delivered with built-in causality knowledge capturing the common root causes that can occur in cloud-native environments. This causality knowledge enables Causely to automatically pinpoint root causes out-of-the-box as soon as it is deployed in an environment. There are at least a few important details to share about this causality knowledge:  
  • It captures potential root causes in a broad range of entities including applications, databases, caches, messaging, load balancers, DNS, compute, storage, and more. 
  • It describes how the root causes will propagate across the entire environment and what symptoms may be observed when each of the root causes occurs.  
  • It is completely independent from any specific environment and is applicable to any cloud-native application environment.  
• Automatic topology discovery. Cloud-native environments are a tangled web of applications and services layered over complex and dynamic infrastructure. Causely automatically discovers all the entities in the environment including the applications, services, databases, caches, messaging, load balancers, compute, storage, etc., as well as how they all relate to each other. For each discovered entity, Causely automatically discovers its:  
  • Connectivity - the entities it is connected to and the entities it is communicating with horizontally  
  • Layering - the entities it is vertically layered over or underlying 
  • Composition - what the entity itself is composed of  
    
    Causely automatically stitches all of these relationships together to generate a Topology Graph, which is a clear dependency map of the entire environment. This Topology Graph updates continuously in real time, accurately representing the current state of the environment at all times. 
• Root cause analysis. Using the out-of-the-box Causal Models and the Topology Graph as described above, Causely automatically generates a causal mapping between all the possible root causes and the symptoms each of them may cause, along with the probability that each symptom would be observed when the root cause occurs. Causely uses this causal mapping to automatically pinpoint root causes based on observed symptoms in real time. No configuration is required for Causely to immediately pinpoint a broad set of root causes (100+), ranging from applications malfunctioning to services congestion to infrastructure bottlenecks.  
  
  In any given environment, there can be tens of thousands of different root causes that may cause hundreds of thousands of symptoms. Causely prevents SLO violations by detangling this mess and pinpointing the root cause putting your SLOs at risk and driving remediation actions before SLOs are violated. For example, Causely proactively pinpoints if a software update changes performance behaviors for dependent services before those services are impacted. 
• Performance analysis. Causely analyzes microservices performance bottleneck propagation by automatically learning, based on your data:  
  • the correlation between the loads on services, i.e., how a change in load of one cascades and impacts the loads on other services; 
  • the correlation between services latencies, i.e., how latency of one cascades and impacts the latencies of other services; and  
  • the likelihood a service or resource bottleneck may cause performance degradations on dependent services. 
• Constraints analysis. Causely uses performance goals like throughput and latency, and capacity or cost constraints, and automatically figures out what actions need to be taken to assure the goals are accomplished while satisfying the constraints.  
• Prevention analysis. Teams can also ask "what if'' questions to understand the impact that potential problems might have if they were to occur to support the planning of service/architecture changes, maintenance activities, and service resiliency improvements.
• Predictive analysis. Causely automatically analyzes performance trends and pinpoints the actions required to prevent future degradations, SLO violations, or constraints.  
• Service impact analysis. Causely automatically analyzes the impact of the root causes on SLOs, prioritizing the root causes based on the violated SLOs and those that are at risk. Causely automatically defines standard SLOs (based on latency and error rate) and uses machine learning to improve its anomaly detection over time. However, environments that already have SLO definitions in another system can easily be incorporated in place of Causely’s default settings. 
• Contextual presentation. Results are intuitively presented in the Causely UI, enabling users to see the root causes, related symptoms, the service impacts and initiate remedial actions. The results can also be sent to external systems to alert teams who are responsible for remediating root cause problems, to notify teams whose services are impacted, and to initiate incident response workflows. 
• Postmortem analysis. Teams can also review prior incidents and see clear explanations of why these occurred and what the effect was, simplifying the process of postmortems, enabling actions to be taken to avoid re-occurrences.  

Conclusion 

Escalations don’t need to devolve into chaos, finger pointing, and frayed relationships. They can be opportunities for teams to solve real problems together. The key is having dependable, real-time information on service dependencies and root causes of problems. Armed with the right information, teams can work efficiently and collaboratively to maintain system reliability. 

Book a meeting with the Causely team and let us show you how to transform the state of escalations and cross-organizational collaboration in cloud-native environments. 

Troubleshooting is an unavoidable part of life for SREs and developers alike, and it often feels like an endless grind. The moment a failure occurs, the clock starts ticking. If the failure impacts a mission critical application, every second counts. Outages can cost hours of wasted productivity, to say nothing of lost revenue and pricing concessions when you’ve violated an SLO. Pinpointing the root cause requires sifting through piles of logs, metrics that blur together, and false positives. Troubleshooting becomes a search for a needle in a haystack, and to make things even more complex, the needle may not even be in the haystack. Furthermore, when the failure originates in your scope of control, the pressure intensifies—you’re expected to resolve it quickly, minimize downtime, and restore service without disrupting the rest of your work. It’s a reactive process, and it’s draining.  

But it doesn’t have to be this way. By adopting systems that solve the root cause analysis problem and automate troubleshooting, you can shift troubleshooting from a time-consuming, heavy-lifting chore to a streamlined task. Automated root cause analysis cuts through the noise and pinpoints the issue in no time.   

With the right approach, troubleshooting becomes a quick, manageable part of your day, freeing you to focus on building systems that don’t just react better but fail less often. 

What do we mean by troubleshooting? 

In a distributed microservices environment, troubleshooting often begins with an alert from the monitoring system or user feedback about degraded performance, such as increased latency or error rates. Typically, these issues are first observed in the service exposed to end users, such as an API gateway or frontend service. However, the root cause often lies deeper within the service architecture, making initial diagnosis challenging. The development team must begin by confirming the scope of the issue, correlating the alert with specific user-reported problems to identify whether it is isolated or systemic. 

The next step involves tracing the source of the alert within the service ecosystem. Using distributed tracing tools like OpenTelemetry, the team tracks requests as they propagate through various microservices, identifying where bottlenecks or failures occur. Concurrently, a service dependency map, often visualized through monitoring platforms, provides a bird’s-eye view of interactions between services, databases, caches, and other dependencies, helping to pinpoint potential hotspots in the architecture. 

Example service dependency map. Source: Grafana 

Once the potential hotspots are identified, developers turn to metrics and logs for further insights. Resource utilization metrics, such as CPU, memory, and disk I/O, are analyzed to detect bottlenecks, while logs reveal specific errors or anomalies like timeouts or failed database queries. This analysis attempts to correlate symptoms with the timeline of the issue, offering clues to its origin. Often, the team experiments with quick fixes, such as scaling up CPU, memory, or storage for the affected services or infrastructure. While these adjustments might temporarily relieve symptoms, they rarely address the root cause and must be rolled back if ineffective.  

When resource adjustments fail, a deeper dive into the affected components is necessary. Distributed traces provide detailed insights into slow transactions or failures, highlighting which services or calls are problematic. Developers then use continuous profiling tools to examine runtime data for each service, identifying resource-intensive methods, excessive memory allocations, or inefficient call paths. This granular analysis helps uncover inefficiencies or regressions in code performance. 

If the issue involves a database, further investigation focuses on query performance. Database profiling tools are used to analyze query execution times, frequency, and data volume. Developers assess whether queries are taking longer than usual, retrieving excessive data, or being executed too frequently. This step often reveals issues such as missing indexes, inefficient joins, or unoptimized queries, which could be contributing to overall service degradation. By iteratively analyzing and addressing these factors, the root cause of the problem is eventually resolved, restoring system stability and performance. 

Troubleshooting is reactive, time-consuming, and exhausting.  Developers should be focusing their time and energy (and their company’s investment) on innovation, yet troubleshooting forces them to turn their attention elsewhere. 

Troubleshooting doesn’t have to dominate your role; with the right systems, it can become efficient and manageable. 

Troubleshooting is hard 

When trying to find the root cause of service or application outages or degradations, developers face numerous challenges: 

• It’s hard to pinpoint which service is the source of the degradations amid a flood of alerts 
• It’s hard to diagnose and remediate the root cause 
• It’s hard to see the forest from the trees 

It’s hard to pinpoint which service is the source of the degradations amid a flood of alerts 

Failures propagate and amplify through the environment. A congested database or a congested resource will cause application starvation and service degradation that cascades throughout the system. Even if you deploy an observability tool to monitor the database or the resource, you may observe nothing on the database or the resource. And if you deploy an observability tool to monitor the applications and services, you will be flooded with alerts about application starvation and service degradations. Given the flood of alerts, pinpointing the bottleneck is very complex. As described above, it entails a time-consuming, heavy lifting, manual process under pressure.   

The more observability tools you deploy, the more data you collect and the harder the problem gets. More alerts, more noise, more data you need to sift through. This is a journey to nowhere, a trajectory you want to reverse. 

Diagnosing and remediating the root cause of a problem is hard 

Pinpointing the congested service, database or resource is hard and inefficient. Even if you know where the root cause is, you may not know what the root cause is. Without knowing what the root cause is, you can’t know what to remediate nor how to remediate.  

Whether pinpointing where the bottleneck is or pinpointing what the root cause is, engineers rely on manual workflows to sift through logs, metrics, and traces. While new observability tools have emerged over the past decade focusing on cloud-native application infrastructure, and the traditional old guards have expanded their coverage to monitor the new technology landscape, neither has solved the problem. Some may do a better job than others in correlating anomalies or slicing and dicing the information for you, but they leave it to you to diagnose and pinpoint the root cause, leaving the hardest part unsolved. Furthermore, most of them require time consuming setup and configuration, deep expertise to operate, and deep domain knowledge to realize their benefits. 

In practice, this means engineers are still performing most of the diagnostic work manually. The tools may be powerful, even elegant, but they don’t address the core challenge: diagnosing and remediating root causes remains a slow, resource-intensive process, particularly when time is of the essence during an incident. These gaps prolong resolution times, increase stress, and reduce time for proactive system improvements. 

It’s hard to see the forest from the trees 

Once the engineers turn to dashboards to investigate further, they stare at dashboards created by bottom-up tools. These tools collect a lot of data (often at great cost) and present this data in their dashboards without regard to the purpose of the information and the problem that needs to be solved. Engineers sift through metrics, logs, and time-series data, trying to understand context, composition, and dependencies so they can manually piece together patterns and correlations. This is highly labor-intensive and drives the engineer to get lost in the weeds without understanding the big picture of how the business or the service is impacted.  Are service level objectives (SLOs) being violated? Are SLOs at risk? 

Take your favorite observability tool. It probably excels at visualizing time-series data. However, it requires engineers to manually connect trends across dashboards and services, which can be especially challenging in distributed systems. Similarly, application performance management (APM) tools provide rich metrics and infrastructure insights, but the sheer volume of data presented in their dashboards can overwhelm users, making it difficult to focus on the most relevant information.  

These tools, while powerful, often fall short in helping engineers see the forest from the trees. Instead of guiding engineers toward the right priorities and actionable insights about the broader system or the root cause, or even better, automatically pinpointing the root cause and remediating, they frequently amplify the noise. Irrelevant data, ambiguous relationships, and false positives force engineers to wade through excessive details, wasting time and delaying resolution. The lack of a top-down perspective makes it harder to understand how symptoms connect to underlying problems, leaving engineers stuck in the weeds. 

The negative consequences of troubleshooting today 

The way troubleshooting is done today has serious ramifications for organizations, teams, and individuals. It affects business outcomes and quality of life. 

Failing to meet the SLAs 

Whether the goal is 5-nines, 4-nines, or even only 3-nines, if we continue to manually troubleshoot, we will never meet these SLAs. The table below illustrates how many minutes in a month the given SLA allows for downtime.  

Source: High Availability, Wikipedia

3-nines means 99.9% uptime—in other words, all services are performing reliably at least 99.9% of the time. So, if any of the services is degraded for more than 43.2 minutes in a month, the 3-nines SLA is not met. Because of the length of time manual troubleshooting entails, a single incident in the month will cause us to miss delivering on a 3-nines SLA. And 3-nines is not even so great! 

High Mean Time to Detect and Resolve (MTTD/MTTR) 

The longer it takes to detect and resolve an issue, the greater the impact on customers and the business. Traditional troubleshooting workflows, which often rely on reactive and manual processes, are inherently slow. Engineers are forced to navigate through an overwhelming volume of alerts, sift through logs, and correlate metrics without clear guidance. This delay can lead to: 

• Prolonged outages that damage user trust and satisfaction. 
• Breaches of service level objectives (SLOs), which can result in financial penalties for organizations with stringent service level agreements (SLAs) 
• Snowballing effects, where unresolved issues trigger secondary failures, compounding the problem and making resolution even more challenging. 

Individual stress and burnout from constant reactive tasks 

The reactive nature of troubleshooting takes a significant toll on individual engineers. When every incident feels like a race against the clock, the pressure to resolve issues quickly can become overwhelming. Engineers often work under constant stress, juggling: 

• Interruptions to their regular work, leading to disrupted schedules and decreased productivity. 
• Escalations where they are expected to step in as subject matter experts, often during nights or weekends. 
• Repeated exposure to alert noise, which can cause decision fatigue and desensitization to critical alerts. 

This relentless pace contributes to burnout. All it takes is a few hours of perusing the /r/sre subreddit to see that burnout is a very common issue among SREs and developers tasked with maintaining system reliability. Burnout not only affects individuals but also leads to higher attrition rates, disrupting team continuity and increasing hiring and training costs. 

Reduced time for proactive reliability engineering 

Troubleshooting dominates the time and energy of engineering teams, leaving little room for proactive reliability initiatives. As we will see later this week, proactive reliability engineering has extraordinary promise for the entire company: product/engineering, operations, business leaders. But instead of focusing on preventing incidents, engineers are stuck in a reactive loop. This trade-off results in: 

• Delayed implementation of improvements that could enhance system stability and scalability. 
• Accumulation of technical debt. 
• A vicious cycle where the lack of proactive work increases the likelihood of future incidents, perpetuating the troubleshooting burden. 

By constantly reacting to problems rather than proactively addressing underlying issues, teams lose the ability to innovate and build resilient systems. This dynamic not only affects engineering morale but also has broader implications for an organization’s ability to compete and adapt in fast-paced markets. 

How troubleshooting should look 

If we all recognize that the state of the art of troubleshooting is awful today, let’s work together to imagine a future where troubleshooting is routine and fast: 

• Systems automatically pinpoint root causes within your domain quickly and accurately. Modern troubleshooting workflows must prioritize speed and precision. Systems should go beyond flagging symptoms and directly pinpoint the underlying cause within your domain. 
• Actionable information provides necessary context upfront. Systems need to focus on identifying the actions and ideally automating the automatable. 
• Troubleshooting workflows are streamlined. Workflows should be intuitive and efficient, designed to minimize context switching and maximize focus with unified dashboards that integrate with your operational workflows. 

 These systems must have certain capabilities to be effective: 

• Causality. The ability to capture, represent, understand and analyze cause and effect relations. 
• Reasoning. Generic analytics that can reason about causality and automatically pinpoint root causes based on observed symptoms. 
• Automatic topology discovery. The ability to automatically discover the environment, the entities and the relationships between them. 

With these systems, proper troubleshooting can drive positive business outcomes, such as: 

• Delivering on SLOs and meeting SLAs. Reduce the number of incidents. 
• Faster issue resolution, minimizing downtime. Reduce mean time to detect (MTTD) and mean time to resolve or recover (MTTR), keeping systems operational and minimizing the impact on users. 
• Improved productivity by reducing time spent on reactive tasks. Enable engineers to focus on high-value innovation. 

Causely automates troubleshooting 

Our Causal Reasoning Platform is a model-driven, purpose-built AI system delivering multiple analytics built on a common data model. It is designed to make troubleshooting much simpler and more effective by providing:  

• Out-of-the-box Causal Models. Causely is delivered with built-in causality knowledge capturing the common root causes that can occur in cloud-native environments. This causality knowledge enables Causely to automatically pinpoint root causes out-of-the-box as soon as it is deployed in an environment. There are at least a few important details to share about this causality knowledge:  
  • It captures potential root causes in a broad range of entities including applications, databases, caches, messaging, load balancers, DNS compute, storage, and more. 
  • It describes how the root causes will propagate across the entire environment and what symptoms may be observed when each of the root causes occurs.  
  • It is completely independent from any specific environment and is applicable to any cloud-native application environment.  
• Automatic topology discovery. Cloud-native environments are a tangled web of applications and services layered over complex and dynamic infrastructure. Causely automatically discovers all the entities in the environment including the applications, services, databases, caches, messaging, load balancers, compute, storage, etc., as well as how they all relate to each other. For each discovered entity, Causely automatically discovers its:  
  • Connectivity - the entities it is connected to and the entities it is communicating with horizontally  
  • Layering - the entities it is vertically layered over or underlying 
  • Composition - what the entity itself is composed of 

Causely automatically stitches all of these relationships together to generate a Topology Graph, which is a clear dependency map of the entire environment. This Topology Graph updates continuously in real time, accurately representing the current state of the environment at all times. 

• Root cause analysis. Using the out-of-the-box Causal Models and the Topology Graph as described above, Causely automatically generates a causal mapping between all the possible root causes and the symptoms each of them may cause, along with the probability that each symptom would be observed when the root cause occurs. Causely uses this causal mapping to automatically pinpoint root causes based on observed symptoms in real time. No configuration is required for Causely to immediately pinpoint a broad set of root causes (100+), ranging from applications malfunctioning to services congestion to infrastructure bottlenecks.  

In any given environment, there can be tens of thousands of different root causes that may cause hundreds of thousands of symptoms. Causely prevents SLO violations by detangling this mess, pinpointing the root cause that’s putting your SLOs at risk, and driving remediation actions before SLOs are violated. For example, Causely proactively pinpoints if a software update changes performance behaviors for dependent services before those services are impacted. 

• Service impact analysis. Causely automatically analyzes the impact of the root causes on SLOs, prioritizing the root causes based on the violated SLOs and the ones that are at risk. Causely automatically defines standard SLOs (based on latency and error rate) and uses machine learning to improve its anomaly detection over time. However, environments that already have SLO definitions in another system can easily be incorporated in place of Causely’s default settings. 
• Contextual presentation. The results are intuitively presented in the Causely UI, enabling users to see the root causes, related symptoms, the service impacts and initiate remedial actions. The results can also be sent to external systems to alert teams who are responsible for remediating root cause problems, to notify teams whose services are impacted, and to initiate incident response workflows. 
• Prevention analysis. Teams can also ask "what if'' questions to understand the impact that potential problems might have if they were to occur to support the planning of service/architecture changes, maintenance activities and improving the resilience of services.  
• Postmortem analysis. Teams can also review prior incidents and see clear explanations of why these occurred and what the effect was, simplifying the process of postmortems, enabling actions to be taken to avoid re-occurrences.  

Conclusion 

Troubleshooting doesn’t have to dominate a developer’s or SRE’s nightmare when the right systems are in place. Empower yourself with the only system that solves the root cause analysis problem to make troubleshooting a small, manageable part of your job. 

Book a meeting with the Causely team and let us show you how to stop troubleshooting and consistently meet your reliability expectations in cloud-native environments. 

Best Use of AI

Winner: Causely

Many observability systems now claim to support Root Cause Analysis. At the same time though, most of these systems use algorithms – admittedly, advanced algorithms, which are based, fundamentally, on correlation rather than causation. For us Causely stands out as a system which truly embeds Causal AI in its reasoning and can therefore genuinely go beyond correlation and make more intelligent analysis of system behaviour.

Their full recap is available on the OpenTelemetry blog, and you can watch the full recording below.

What is CPU Throttling?

CPU throttling in containers occurs due to resource constraints set by control groups (cgroups). Kubernetes and other container orchestrators rely on cgroups to enforce resource limits. When a container attempts to use more CPU than its assigned quota, it gets throttled, delaying execution of tasks. (When containers have CPU limits defined, they will be converted to a cgroup CPU quota.)

Working on the vendor side for over a decade, I have seen the impact CPU throttling can have on different services across many industries.  Here are three top-of-mind examples, both from my days at Turbonomic and from recent conversations with customers at Causely:

Financial Systems

• • Example: A stock trading platform uses containers to handle real-time market data feeds and execute trades. Throttling during peak trading hours delays data processing, potentially causing missed opportunities or incorrect order placements.
  • Impact: Missed deadlines for transaction processing.

Gaming Servers

• • Example: Online multiplayer games hosted in containers experience throttling, leading to delayed responses (lag) during gameplay. Players may experience slow rendering of in-game actions or disconnections during high traffic.
  • Impact: Latency and poor user experience.

Video Streaming Platforms

• Example: A video-on-demand service runs encoding jobs in containers to transcode videos. Throttling increases encoding times, leading to delayed content availability or poor streaming quality for users.
• Impact: Degraded video quality and buffering issues.

How to Identify CPU Throttling

It’s often difficult to catch CPU throttling because it can happen even when the host CPU usage is low. It’s critical to have the right level of monitoring set up in order to see CPU throttling when it happens, or even better, before it becomes a problem.

Monitor Your cgroup Metrics

Linux cgroups provide detailed metrics about CPU usage and throttling. Look for the cpu.stat file within the container’s cgroup directory (usually under /sys/fs/cgroup):
Within the cpu.stat file there are three key metrics:

• nr_throttled: Number of times the container was throttled.
• throttled_time: Total time spent throttled – which I believe is in nanoseconds
• nr_periods: Total CPU allocation periods.

Example:
cat /sys/fs/cgroup/cpu/cpu.stat

Output:
nr_periods 12345
nr_throttled 543
throttled_time 987654321

If nr_throttled or throttled_time is high relative to nr_periods, then you have CPU throttling on your container.

Monitor Container Orchestration Metrics

If you’re running Kubernetes, you can use the kubectl top pod command to get metric data on the highest utilized pods.  Try the command below to get metrics for a pod and all the associated containers:
kubectl top pod --containers
This is a very manual process, and you will need to compare the CPU usage against the defined limits in the pod’s resource config.  If you run a describe command on the pod it will show you this information.  This also means you will need to know which pod is having an issue on it.  Usually when issues arise on an application it’s going to take some time to drill down to a component that might be performing poorly.  You will need the Kubernetes metric server to be installed in order to run commands like kubectl top but being able to access metrics like “container_cpu_cfs_throttled_periods_total” and “container_cpu_cfs_periods_total” offer valuable insights into CPU usage and throttling.

Application Performance Metrics

Although they can be expensive, application performance monitoring (APM) tools provide invaluable insights into CPU throttling, offering detailed visibility that can help uncover the issue. These tools can often track throttling over time, identify exactly when it first occurred, and, in some cases, even predict future throttling trends based on usage patterns. Many organizations use a combination of monitoring tools to get a comprehensive view of their systems. APM tools also highlight the symptoms of CPU throttling, which may manifest as:

• Prolonged request durations, leading to slower application response times.
• Decreased throughput, resulting in fewer transactions or tasks processed within a given timeframe.
• Irregular CPU usage patterns, which can signal performance instability or inefficiencies.

By combining the capabilities of APM tools with metrics collected from Kubernetes, teams can proactively address CPU throttling and ensure optimal application performance.

Best Practices to Manage CPU Throttling in Kubernetes

There are many ways to fix CPU throttling and even a few ways you can prevent it. Most root causes of CPU throttling are overcommitted nodes, or misconfigured CPU limits.  Below are some ways to fix CPU throttling when it occurs and some best practices to avoid it in the future.

Adjust CPU Limits

Update resource limits in your container or pod configuration. Kubernetes resource specs can be updated like in the example below.  Usually what I see from the customers I have worked with is they will set the limit just above the peak usage in the last 30,60, or even 90 days.  For non-critical workloads I have seen a few companies set this limit to 80% of max usage, and a few companies use more advanced techniques like calculating percentiles:

resources:
requests:
cpu: "500m"
limits:
cpu: "1000m"

• Increase the limits.cpu value to reduce throttling frequency.
• Set requests.cpu to ensure better performance during contention.  Note that if you do not set the request, Kubernetes will automatically set the request to the limit.

Use Autoscaling like Horizontal Pod Autoscaler

The Horizontal Pod Autoscaler (HPA) in Kubernetes helps address CPU throttling by dynamically adjusting the number of pods in a deployment based on real-time resource usage.  Resources like CPU and memory are monitored, and when certain thresholds are met, HPA kicks in to provision more pods.  In more idle periods it will also scale down the number of pods to help you run more efficiently.  By distributing the workload across more pods, HPA reduces the CPU demands on individual pods, thereby mitigating CPU throttling.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
targetAverageUtilization: 80

In this example, if average CPU utilization across all pods exceeds 80% then it will add more pods as necessary within the bounds of 2-10 (Min and Max Replicas).

Analyze Node Resource Allocation

Check overall CPU availability on nodes using a describe command:
kubectl describe node <node-name>
Ensure nodes aren’t overcommitted. Use taints and tolerations to control scheduling and ensure high-priority workloads run on dedicated nodes.  Nodes that are overcommitted run the risk of not having available CPUs.  If containers’ requests are higher than the node CPU availability, you are going to run into scheduling problems.  Or if the limits are set too high relative to CPU availability on the node and the workload suddenly increases, then you are going to have contention.

Tweak CPU CFS Settings

Containers use the Completely Fair Scheduler (CFS) by default. The CFS in Kubernetes is a mechanism inherited from the Linux kernel that enforces CPU usage limits on containers. It works by using two key parameters from Linux cgroups: cpu.cfs_quota_us and cpu.cfs_period_us. These parameters allow Kubernetes to control the amount of CPU time a container can use over a specific period:

• cpu.cfs_quota_us: Maximum microseconds of CPU time allowed per period.
• cpu.cfs_period_us: Length of a scheduling period in microseconds.

To prevent CPU throttling: Increase cpu.cfs_quota_us to provide more CPU time:
echo 200000 > /sys/fs/cgroup/cpu/cpu.cfs_quota_us
I have seen this create issues before though so be careful with this adjustment as it can lead to overcommitment.  In other words, if too many tasks are scheduled and you increase the amount of time a container can use the CPU, then it will create delays and throttling.  Start by playing around with this in Dev or Test clusters before you make any changes to prod… duh.

Use CPU Pinning

This is more of an edge case, but instead of using CPU shares and limits, pin containers to specific CPUs for predictable performance.  The Kubernetes CPU Manager controls how CPUs are allocated to containers.  The enable CPU pinning, the static CPU manager policy must be used, which provides exclusive CPU allocation.  Just enable the policy in the Kubelet configuration file:
cpuManagerPolicy: static
With the flag set to “static,” containers are allocated exclusive CPUs on the nodes in a cluster.  Kubernetes assigns the container specific CPUs and it runs only on those cores.  The big challenges with CPU pinning are overhead and scalability.  When you manage pinned workloads it requires detailed planning to avoid fragmentation and underutilization.  CPU pinning is good for workloads that are sensitive to CPU throttling but not ideal for volatile and dynamic workloads.

CPU Throttling is a Double-Edged Sword in Kubernetes

While CPU throttling plays a crucial role in resource management and stability, it can also hinder application performance if not managed correctly. By understanding how CPU throttling works and implementing best practices, you can optimize your Kubernetes environment, ensuring efficient resource use and enhanced application performance. As Kubernetes continues to grow and evolve, keeping a close eye on resource management will be key to maintaining robust and responsive applications.

In this post, we’ll explain what it means to be a Service Owner, outline key responsibilities associated with the role, and offer advice for companies looking to build a culture of service ownership.

How the Service Owner came to be

When the DevOps movement took off around 2010, it promised to fix the issues with fragmented teams and inefficient software lifecycle management that had been hindering application performance and reliability for years.  This new wave of IT fostered cross-team collaboration, communication, and transparency as a way to accelerate software delivery of software and deliver higher quality of service (QoS).

But there was something still missing: accountability across the end- to-end application lifecycle, from development to roadmap design to customer satisfaction. Hence the emergence of the Service Owner.

So, what is a Service Owner?

Many IT teams today, especially those using microservice architectures, employ multiple Service Owners.  The exact definition varies slightly from company to company, but most would define a Service Owner (SO) as the person who is responsible for making sure the application or service meets its designated service level objectives (SLOs). Implicit in this definition: Service Owners are accountable for the end-to-end lifecycle of applications, from development through operational performance production monitoring, to ensure uptime and customer satisfaction.

For a little more context, here’s how ITIL defines the role:

• The Service Owner is responsible for delivering a particular service within the agreed service levels.
• Typically, the Service Owner acts as the counterpart of the Service Level Manager when negotiating Operational Level Agreements (OLAs).
• Often, this role will lead a team of technical specialists or an internal support unit.

Usually, Service Owners are aligned to a specific product line for the business. Their formal title may be Product Owner or Engineering Manager. For example, imagine a healthcare tech company that sells solutions to businesses and hospitals.  They have 5 different products within their offering suite:

• AI platform
• Online therapy product
• Analytics product
• Telehealth service
• Financial management product

A Service Owner would be assigned to each of these product lines and assume full responsibility for its application lifecycle management and reliability engineering.  A Service Owner role can also be broken down by customer, by services within the products, or even by mobile vs desktop applications.

Responsibilities of the Service Owner

Service Owners’ responsibilities tend to vary slightly from company to company but they often include:

1. Design and Roadmap: Writing the code and overseeing the design and implementation of the service, ensuring it functions properly and is scalable. This includes managing the ongoing balance and prioritization of releasing new features vs. improving the reliability of existing functionality.
2. Maintenance and Support: Ensuring that the service is properly maintained, updated, and supported over its lifecycle.  This is where creating and adhering to SLOs comes into play.
3. Performance Monitoring: Monitoring the performance and reliability of the service, by implementing metrics and logging to track its health and flag when things break.  Performance Monitoring also includes implementing proactive monitoring to prevent downtime.
4. Collaboration: Working closely with other teams, such as Product Management, Sales, Platform Engineering, etc. to align the service with the business goals.
5. Documentation: Creating and maintaining comprehensive documentation for the service, including things like APIs, user guides, and architecture diagrams.
6. Governance and Compliance: Ensuring that the service adheres to relevant policies, standards, and regulatory requirements.
7. Stakeholder Communication: Acting as a point of contact for stakeholders, addressing their needs and concerns regarding the service.

Service Owners work closely with technical experts such as SREs, DevOps, or Developers to maintain service reliability, though they each have distinct roles and tool preferences:

• Service Owners typically use collaboration and project management tools like Jira or Asana and monitor high-level metrics on observability dashboards like Grafana.
• Technical Experts handle incident resolution and service reliability, relying primarily on observability and incident management tools like PagerDuty.

Tools like ServiceNow, Atlassian’s Service Management (OpsGenie, Jira), and PagerDuty’s workflow orchestration attempt to bridge the gap between these two roles by providing a unified space for planning, alerting, diagnosis, and response. This enables Service Owners and technical experts to operate more effectively together, allowing engineering teams to enhance alignment, transparency, and accountability.

Code it, ship it, own it

A service owner’s job goes beyond writing and compiling code and bug fixes.  They are responsible for their applications and services after they’ve been shipped to production.  When Service Owners own the lifecycle, organizations see improved QoS and faster MTTR.

If they wrote the code, they know how to fix it.  If something breaks, they are the first responders and take accountability for failures.  They have a deeper knowledge of issues within their service and application so they can fix and develop the fastest, but they must be held accountable for downtime.  No more finger pointing!

One thing service owners MUST do is align themselves with their customers.  They must understand what the customers’ needs and expectations are and then design the code, roadmap, and establish SLOs accordingly.

The benefits of this approach? Products directly solve customer pain, and in most cases, services are delivered faster to customers.  It reminds me of a funny meme I saw years ago — it couldn’t be more accurate.  This situation is exactly what Service Owners are preventing!

Source: Reddit

Building a culture of service ownership

Since service ownership is a culture and not a tool, it needs to grow over time; it can’t happen overnight, no matter how much pressure the business puts on IT.  There are ways to actively foster a shift in mentality, so teams thrive in an environment where they have more responsibility and where better services are being delivered to the customer.

Based on my conversations with leaders in IT, these are some common best practices for building a culture of service ownership.

1. Promote collaboration: Encourage open communication between teams—development, operations, and business units. Regular cross-functional meetings and collaborative tools can help break down silos faster and foster a shared understanding of service objectives.
2. Establish a customer-first mentality: As mentioned before, service ownership will not thrive if everyone has different ideas and goals.  Establishing a common goal like focusing on customers’ needs can align teams. If customer satisfaction is the north star, companies will have more satisfied customers, which means bigger checks 😉.  Defining customer specific SLOs is an excellent way to keep everyone aligned on the mission.  SLOs on latency, number of customer tickets/escalations, and even uptime are some standard ones I see most Service Owners using.
3. Embrace failure: Taking accountability and responsibility for something that directly impacts a business can be scary. That’s probably the single biggest reason why most software engineers are hesitant to adopt a service ownership role.  If leadership fosters a culture where failure is seen as progress and not regression, then it becomes more appetizing to developers.  No one wants to lose their job over a silly mistake, but they need to learn from these slips and drive towards a more reliable application architecture.

Building a culture of service ownership in IT requires a deliberate and consistent approach. By defining roles, fostering collaboration, and empowering teams, IT can create an environment where service ownership is continuously improved. This culture not only enhances QoS but also drives innovation and responsiveness, ultimately benefiting the health of any business.

FAQs

• What is application lifecycle management?
  Application lifecycle management is the end-to-end process of developing, building, deploying, and managing software applications over time to ensure consistent and ongoing quality, reliability, and resilience.
• What is reliability engineering?
  Reliability engineering is the practice of ensuring that applications, products, or systems function without failure. Reliability engineers focus on proactively identifying potential failures to determine their root cause and mitigation strategies before they happen.
• What is a Service Owner?
  A Service Owner is someone who is responsible for meeting agreed service levels. They ultimately usually own the overall engineering, management and governance of the lifecycle of a service.

And if you pay attention, you might start to notice some buzz building around one event in particular. Word on the KubeStreet is this will be one of those IYKYK type things, so I’m here to help you get in the know (cue Jaws music 🦈 🎶).

First, we need to set a bit of a foundation about the evolution of DevOps and what’s missing from the Observability market.

The beginning of DevOps

Patrick Debois, a tech consultant based in Belgium, is often cited with popularizing the term DevOps. This is because, in 2009, he organized the first DevOpsDays conference. However, the truth is the roots of DevOps extend further back.

Earlier that same year, John Allspaw and Paul Hammond’s influential talk at the Velocity Conference, 10+ Deploys per Day: Dev and Ops Cooperation at Flickr highlighted the importance of collaboration between development and operations to enable frequent, reliable deployments.

We can go back even further though, as the groundwork for DevOps was already being laid in 2001. Back then, The Agile Manifesto championed collaboration and iterative development, which are principles that would later shape the DevOps ethos. Meanwhile, tools like Puppet, Chef, and Jenkins were gaining traction in the oughts, helping bridge the gap between development and operations teams.

These influences helped shape what is commonly known as DevOps, which is a movement that champions collaboration, cultural transformation, and tooling integration to achieve high-frequency, reliable software delivery.

Then came Observability

As part of this movement, engineering teams started to adopt microservices architecture as a software development method of breaking applications down into small, independent services that interact with each other to perform the job of the application. This method became popular because of the speed and ease that it adds to the software development process.

On the other hand, a microservices approach makes it exponentially more difficult to diagnose, remediate, and prevent application performance problems. The growing popularity of DevOps practices and microservices architecture paved the way for advancements in monitoring and its evolution into what many know today as observability.

As DevOps led teams to release faster and decompose applications into microservices, traditional monitoring – which relies on static alerts based on isolated metrics – could no longer keep up with the resulting complexity. Observability tries to address this gap by offering a more holistic view to help engineers pinpoint issues, understand dependencies, and continuously improve system performance.

Companies large and small appreciate how observability tools provide their engineering teams with a single pane of glass, nifty dashboards, and helpful data correlation features, but the more discerning engineers have matured to view this market of tools as being more like a single glass of pain.

Cool, so what does this have to do with sharks?

Suppose I told you that ice cream consumption and shark attacks were correlated. From this you might infer that sharks love sugar. Of course, you’ve probably heard this example used before to help make the point that correlation does not equal causation. It’s certainly useful to understand that two metrics are temporally correlated, but it’s misleading that observability vendors claim this to be root cause analysis, and in some scenarios it can hurt more than it helps.

At Causely, we value the observability market and believe it’s provided a good start for engineers who want to reduce MTTD and MTTR. The more discerning engineers are coming to realize they will only reach the next frontier of autonomous service reliability if they start to look at the problem from a different, top-down perspective.

We’ll be happy to tell you more about this if you’re lucky enough to find the shark at the show next week. Here’s a hint: she’ll be holding something sweet. 🍨 🥞

Additional information about our presence at KubeCon is here.

Cloud-native and open source technologies are foundational to everything we’re building at Causely, and I’m looking forward to diving into the latest developments with observability tools that will shape the reliability of countless modern applications. We’re excited about how observability tools, such as Prometheus, OpenTelemetry, Grafana Beyla, and Odigos, are improving how systems are monitored and understood. These tools will underpin the reliability engineering strategy of countless modern application environments.

In this post, I’ll unpack some of the exciting innovations happening in the open source observability space, and highlight specific talks at KubeCon 2024 I’m looking forward to attending. Add them to your schedule if this is an area you’re following too, and I’ll see you there!

What’s New in the Observability Landscape

The observability space is evolving rapidly. Several key developments with OpenTelemetry and eBPF are reshaping how we approach monitoring and tracing in distributed systems. This helps make metric collection and auto-instrumentation easier and more flexible than ever.

Prometheus and OpenTelemetry are even better together

The collaboration between Prometheus and OpenTelemetry is delivering a more cohesive experience for capturing system metrics. Here’s what’s new:

• Using the OTel Collector’s Prometheus Receiver: OpenTelemetry now allows for streamlined ingestion of Prometheus metrics, creating a more unified data pipeline.
• Exploring OTel-Native Metric Collection Options: For those deeply involved in Kubernetes monitoring, tools like Kubernetes Cluster Receiver and the Kubeletstats Receiver provide robust options for flexible metric collection.

These integrations create a powerful foundation for system insights that can fuel everything from troubleshooting to strategic infrastructure optimizations.

Odigos makes migration to OpenTelemetry easy

Transitioning from proprietary observability tools can be challenging.  If you’re looking to transition from proprietary observability tools to OpenTelemetry, Odigos is a great enabler. Here’s why:

• Simplified Migration: Odigos takes the complexity out of moving to OpenTelemetry, helping reduce the friction and downtime that can accompany large-scale tooling changes.
• Access to Enhanced Capabilities: With OpenTelemetry, you gain access to a broader ecosystem, opening doors to new integrations and data visualizations that enrich your observability setup.

Streamline eBPF-based auto-instrumentation

Combining Grafana Beyla with Grafana Alloy makes eBPF-powered observability easy and minimally intrusive. Whether you’re working with standalone systems or Kubernetes clusters, this integration provides high-precision monitoring capabilities without heavy overhead.

KubeCon 2024 Deep Dives and Hands-On Sessions

The schedule at KubeCon North America 2024 will feature several sessions dedicated to observability. Here are some highlights I’m looking forward to:

1. OpenTelemetry: The OpenTelemetry Hero’s Journey – Working with Open Source Observability
   Takeaway: Dive into the current capabilities of OpenTelemetry, including correlated metrics, traces, and logs for a complete observability picture. The talk also addresses gaps in today’s open source observability tools.
2. Inspektor Gadget: eBPF for Observability, Made Easy and Approachable
   Takeaway: This project lightning talk will explain how Inspektor Gadget simplifies eBPF distribution and deployment, allowing you to build fast, efficient data collection pipelines that plug into popular observability tools.
3. Optimizing LLM Performance in Kubernetes with OpenTelemetry
   Takeaway: Speakers Ashok Chandrasekar (Google) and Liudmila Molkova (Microsoft) will help you gain practical insights into observing and optimizing Large Language Model (LLM) deployments on Kubernetes. This session covers everything from client and server tracing with OpenTelemetry to advanced autoscaling strategies.
4. Unifying Observability: Correlating Metrics, Traces, and Logs with Exemplars and OpenTelemetry
   Takeaway: Speakers Kruthika Prasanna Simha and Charlie Le (Apple) will help attendees learn how to correlate data across metrics, traces, and logs, with exemplars. This session demonstrates practical visualization techniques in Grafana, making it easy to move from high-level metrics down to detailed traces.
5. Now You See Me: Tame MTTR with Real-Time Anomaly Detection
   Takeaway: Speakers Kruthika Prasanna Simha and Raj Bhensadadia (Apple) will dive into the latest in real-time anomaly detection, with insights into applying machine learning to time series data in cloud-native environments.

Find Causely at KubeCon 2024!

If the above topics are also interesting to you, we’d love to meet up. At Causely, we’re passionate about helping organizations unlock the full potential of their observability data to continuously assure service reliability. We believe that flying with overwhelming volumes of observability data is just as bad as flying blind.

Here’s where you can find us at KubeCon:

• 🥂 Come to our happy hour! We’re co-hosting a happy hour with friends from NVIDIA, Alma Security, Edera, and 645 Ventures on November 12th.
• 🦈 Find the shark! Look for a shark at the show on the afternoon of November 13th. Here’s a hint: she’ll be holding something sweet.

Takeaways from eBPF Summit 2024

How are organizations applying eBPF to solve real problems in observability, security, profiling, and networking? It’s a question I’ve found myself asking as I work in and around the observability space – and I was pleasantly surprised when Isovalent’s recent eBPF Summit provided some answers.

For those new to eBPF, it’s an open source technology that empowers observability practices. Many organizations and vendors have adopted it as a data source (including Causely, where we use it to enhance our instrumentation for Kubernetes).

Many of the eBPF sessions highlighted real challenges companies faced and how they used eBPF to overcome them. In the spirit of helping others, my cliff notes and key takeaways from eBPF Summit are below.

Organizations like Netflix and Datadog are using eBPF in new, creative ways

The use of eBPF in Netflix

One of the Keynote presentations was delivered by Shweta Saraf who described specific problems Netflix overcame using eBPF, such as noisy neighbors. This is a common problem faced by many companies with cloud-native environments.

Shweta Saraf described Netflix’s use cases for eBPF

Netflix uses eBPF to measure how long processes spend in the CPU scheduled state.  When processes are taking too long, it usually indicates a performance bottleneck on CPU resources — like CPU throttling or over-allocation.  (Netflix’s compute and performance team released a blog recently with much more detail on the subject.)  In solving the noisy neighbor problem, the Netflix team also created a tool called bpftop which is designed to measure the CPU usage of the eBPF code they instrumented.

The Netflix team released bpftop for the community to use, and it will ultimately help organizations implement efficient eBPF programs.  This is especially useful if an eBPF program is hung, allowing teams to quickly identify any overhead that an eBPF program has.  We have come full circle: monitoring our monitoring programs 😁.

The use of eBPF in Datadog

Another use case for eBPF – and one that can be easily overlooked – is in chaos engineering.  Scott Gerring, a technical advocate at Datadog, shared his experience on the matter.  This quote resonated with me: “with eBPF… we have this universal language of destruction” – controlled destruction that is.

Scott Gerring discussed eBPF’s use in Datadog

The benefit of eBPF is that we can inject failures into cloud-native systems without having to re-write the code of an application.  Interestingly, there are open source projects out there for chaos engineering that already use eBPF, such as ChaosMesh.

Scott listed a few examples like Kernel Probes that are attached to the openat system that will cause access denied errors for 50% of calls made by system processes that a user can select or define.  Or, using the traffic control subsystem to drop packets for sockets on process you want to mark for failure.

eBPF will underpin AI development

Isovalent Co-founder and CTO Thomas Graf presented the eBPF roadmap and what he is most excited about.  Notably: eBPF will deliver value in enabling the GPU and DPU infrastructure wave fueled by AI.  AI is undoubtably one of the hottest topics in tech right now.  Many companies are using GPUs and DPUs to accelerate AI and ML (Machine Learning) tasks, because CPUs cannot deliver the processing power demanded by today’s AI models.

Thomas Graf talked about the value of eBPF in enabling GPU and DPU infrastructures

As Tom mentioned, whether the AI wave produces anything meaningful is up for debate, but companies will undoubtedly try, and they will make significant investments in GPUs and DPUs along the way.  The capabilities of eBPF will be applied to this new wave of infrastructure in the same manner they did for CPUs.

GPUs and DPUs are expensive, so companies do not want to waste processing power on programs that will drive up utilization. The efficiency of eBPF programs can help maximize the performance of costly GPUs. For example, eBPF can be used for GPU profiling by hooking into GPU events such as memory, sync, and kernel launches.  Unlocking this type of data can be used to understand which kernels are used most frequently, improving efficiencies of AI development.

eBPF support for Windows is growing

Another interesting milestone in eBPF’s journey is the support for Windows.  In fact, there is a growing Git Repository for eBPF programs on Windows that exists today: https://github.com/microsoft/ebpf-for-windows

The project supports Windows 10 or later and Windows Server 2019 or later, and while there is not feature parity yet to Linux, there is a lot of development in this space.  The community is hard at work porting over the same tooling for eBPF on Linux, but it is a challenging endeavor as the hook points for Linux eBPF components (like Just-In-Time compilation or eBPF bytecode signatures) will differ on Windows.

It will be exciting to watch the same networking, security, and observability eBPF capabilities on Linux become available for Windows.

The need for better observability is fueling eBPF ecosystem growth

eBPF tools have been created by the community for both applications and infrastructure use cases.  There a 9 major projects for applications and over 30 exciting emerging projects for applications.  Notably, while there are a few production-ready runtimes and tools within the infrastructure ecosystem (like Linux and LLVM Compiler), there are many emerging projects such as eBPF for Windows.

With a user base across Meta, Apple, Capital One, LinkedIn, and Walmart (just to name a few), we can expect the number of eBPF projects to grow considerably in the coming years.  The overall number of projects is actually forecasted in the triple digits by the end of 2025.

One of the top catalysts for growth? The urgent need for better observability.  Of all the topics at last year’s KubeCon in Chicago, observability ranked the highest, beating competing topics like cost and automation.  As with any other tool, eBPF can allow organizations gather a lot of data, but the “why” is important. Are you using that data to create more noise and more alerts, or can you apply it to get to the root cause of problems that surface, or for other applications?

It is exciting to watch the eBPF community develop and implement creative new ways to use eBPF and the 2024 eBPF summit was (and still is) an excellent source of real-world eBPF use cases and community-generated tooling.

There are so many ways to measure application reliability today, with hundreds of key performance indicators (KPIs) to measure availability, error rates, user experiences, and quality of service (QoS). Yet every organization I speak with struggles to effectively use these metrics.  Some applications and services require custom metrics around reliability while others can be measured with just uptime vs. downtime.

In my role at Causely, I work with companies every day who are trying to improve the reliability, resiliency, and agility of their applications. One method of measuring reliability that I keep a close eye on is MTT(XYZ).  Yes, I made that up, but it’s meant to capture all the different variations of mean time to “X” out there.  We have MTTR, MTTI, MTTF, MTTA, MTBF, MTTD, and the list keeps going.  In fact, some of these acronyms have multiple definitions.  The one whose meaning I want to discuss today is MTTR.

So, what’s the meaning of MTTR anyway?

Before cloud-native applications, MTTR meant one thing – Mean Time to Repair. It’s a metric focused on how quickly an organization can respond to and fix problems that cause downtime or performance degradation.  It’s simple to calculate too:

Total time spent on repairs is the length of time IT spends fixing issues, and number of repairs is the number of times a fix has been implemented.  Some organizations look at this over a week or a month in production. It’s a great metric to understand how resilient your system is and how quickly the team can fix a known issue.  Unfortunately, data suggests that most IT organizations’ MTTR is increasing every year, despite massive investments in the observability stack.

For monolithic applications, MTTR has historically been an excellent measurement; as soon as a fix is applied, the entire application is usually back online and performing well.  Now that IT is moving toward serverless and cloud-native applications, it is a much different story.  When a failure occurs in Kubernetes – where there are many different containers, services, applications, and more  all communicating in real time – the entire system can take much longer to recover.

The new MTTR: Mean Time to Recover

I am seeing more and more organizations redefine the meaning of MTTR from “mean time to repair” to “mean time to recover.”  Recover means that not only is everything back online, but the system is performing well and satisfying any QoS or SLAs AND a preventative approach has been implemented.

For example, take a common problem within Kubernetes: a pod enters a CrashLoopBackoff state.  There are many reasons why a pod might continuously restart including deployment errors, resourcing constraints, DNS resolution errors, missing K8s dependencies, etc.  But let’s say you completed your investigation and found out that your pod did not have sufficient memory and therefore was crashing/restarting.  So you increased the limit on the container or the deployment and the pod(s) seems to be running fine for a bit…. but wait, it just got evicted.

The node now has increased memory usage and pods are being evicted.  Or, what if now we created noisy neighbors, and that pod is “stealing” resources like memory from others on the same node?  This is why organizations are moving away from repair because sometimes when the applied fix brings everything online, it doesn’t mean the system is healthy. “Repaired” can be a subjective term. Furthermore, sometimes the fix is merely a band-aid, and the problem returns hours, days, or weeks later.

Waiting for the entire application system to become healthy and applying a preventative measure will get us better insight into reliability.  It is a more accurate way to measure how long it takes from a failure event to a healthy environment.  After all, just because something is online does not mean it is performing well.  The tricky issue here is: How do you measure “healthy”?   In other words, how do we know the entire system is healthy and our preventative patch is truly preventing problems?  There are some good QoS benchmarks like response time or transactions per second, but there is usually some difficulty in defining these thresholds.  An improvement in MTBF (mean time between failures) is another good benchmark to test to see if your preventative approach is working.

How can we improve Mean Time to Recover?

There are many ways to improve system recovery, and ultimately the best way to improve MTTR is to improve all the MTT(XYZ) that come before it on incident management timelines.

• Automation: Automating tasks like ticket creation, assigning incidents to appropriate teams, and probably most importantly, automating the fix can all help reduce the time from problem identification to recovery.  But, the more an organization scrutinizes every single change and configuration, the longer it takes to implement a fix.  Becoming less strict drives faster results.
• Well-defined Performance Benchmarks: Lots of customers I speak with have a couple KPIs they track, but the more specific the better.  For example, instead of making a blanket statement that every application needs to have 200ms of response time or less,  set these metrics on an app by app basis.
• Chaos Engineering: This is an often-overlooked methodology to improve recovery rate.  Practicing and simulating failures helps improve how quickly we can react, troubleshoot, and apply a fix.  It does take a lot of time though, so it is not an easy strategy to adhere to.
• Faster Alerting Mechanisms: This is simple: The faster we get notified of a problem, the quicker we can fix it.  We need to not just identify the symptoms but also quickly find the root cause.  I see many companies try to set up proactive alerts, but they often get more smoke than fire.
• Knowledge Base: This was so helpful for me in a previous role. Building a KB in a system like Atlassian, SharePoint, or JIRA can help immensely in the troubleshooting process.  The KB needs to be searchable and always changing as the environment evolves.  Being able to search for a specific string in an error message within a KB can immediately highlight not just a root cause but also a fix.

To summarize, MTTR is a metric that needs to capture the state of a system from the moment of failure until the entire system is healthy again.  This is a much more accurate representation of how fast we recover from a problem, and how resilient the application architecture is.  MTTR is a principle that extends beyond the world of IT; its applications exist in security, mechanics, even healthcare.  Just remember, a good surgeon is not only measured by how fast he can repair a broken bone, but by how fast the patient can recover.

Improving application resilience and reliability is something we spend a lot of time thinking about at Causely. We’d love to hear how you’re handling this today, and what metric you’ve found most useful toward this goal. Comment here or contact us with your thoughts!

In an article that I published nearly two years ago titled Are Humans Actually Underrated, I talked about how technology can be used to augment human intelligence to empower humans to work better, smarter and faster.

The notion that technology can enhance human capabilities is far from novel. Often termed Intelligence Augmentation, Intelligence Amplification, Cognitive Augmentation, or Machine Augmented Intelligence, this concept revolves around leveraging information technology to bolster human intellect. Its roots trace back to the 1950s and 60s, a testament to its enduring relevance.

From the humble mouse and graphical user interface to the ubiquitous iPhone and the cutting-edge advancements in Artificial Intelligence like ChatGPT, Intelligence Augmentation has been steadily evolving. These tools and platforms serve as tangible examples of how technology can be harnessed to augment our cognitive abilities and propel us towards greater efficiency and innovation.

An area of scientific development closely aligned with Intelligence Augmentation is the field of Causal Reasoning. Before diving into this, it’s essential to underscore the fundamental importance of causality. Understanding why things happen, not just what happened, is the cornerstone of effective problem-solving, decision-making, and innovation.

Humans Crave Causality

Our innate curiosity drives us to seek explanations for the world around us. This deep-rooted desire to understand cause-and-effect relationships is fundamental to human cognition. Here’s why:

Survival: At the most basic level it all boils down to survival. By understanding cause-and-effect, we can learn what actions lead to positive outcomes (food, shelter, safety) and avoid negative ones (danger, illness, death).

Learning: Understanding cause-and-effect is fundamental to learning and acquiring knowledge. We learn by observing and making connections between events, forming a mental model of how the world works.

Prediction: Being able to predict what will happen allows us to plan for the future and make informed choices. We can anticipate the consequences of our actions and prepare for them.

Problem-solving: Cause-and-effect is crucial for solving problems efficiently. By identifying the cause of an issue, we can develop solutions that address the root cause rather than just treating the symptoms.

Scientific Discovery: This innate desire to understand causality drives scientific inquiry. By seeking cause-and-effect relationships, we can unravel the mysteries of the universe and develop new technologies.

Technological Advancement: Technology thrives on our ability to understand cause-and-effect. From inventing tools to building machines, understanding how things work allows us to manipulate the world around us.

Societal Progress: When we understand the causes of social issues, we can develop solutions to address them. Understanding cause-and-effect fosters cooperation and allows us to build a better future for ourselves and future generations.

Understanding Cause & Effect In The Digital World

In the complex digital age, this craving for causality remains as potent as ever. Nowhere is this more evident than in the world of cloud native applications. These intricate systems, composed of interconnected microservices and distributed components, can be challenging to manage and troubleshoot. When things go wrong, pinpointing the root cause can be akin to searching for a needle in a haystack.

This is increasingly important today because so many businesses rely on modern applications and real time data to conduct their daily business. Delays, missing data and malfunctions can have a crippling effect on business processes and customer experiences which in turn can have significant financial consequences.

Understanding causality in this context is paramount. It’s the difference between reacting to symptoms and addressing the underlying issue. For instance, a sudden spike in error rates might be attributed to increased traffic. While this might be a contributing factor, the root cause could lie in a misconfigured database, a network latency issue, or a bug in a specific microservice. Without a clear understanding of the causal relationships between these components, resolving the problem becomes a matter of trial and error.

Today site reliability engineers (SREs) and developers, tasked with ensuring the reliability and performance of cloud native systems, rely heavily on causal reasoning. They do this by constructing mental models of how different system components interact, they attempt to anticipate potential failure points and develop strategies to mitigate risks.

When incidents occur, SREs and developers work together, employing a systematic approach to identify the causal chain, from the initial trigger to the eventual impact on users. We rely heavily on their  knowledge to implement effective remediation steps and prevent future occurrences.

In the intricate world of cloud native applications, where complexity reigns, this innate ability to connect cause and effect is essential for building resilient, high-performing systems.

The Crucial Role of Observability in Understanding Causality in Cloud Native Systems

OpenTelemetry and its ecosystem of observability tools provide a window into the complex world of cloud native systems. By collecting and analyzing vast amounts of data, engineers can gain valuable insights into system behavior. However, understanding why something happened – establishing causality – still remains a significant challenge.

The inability to rapidly pinpoint the root cause of issues is a costly affair. A recent PagerDuty customer survey revealed that the average time to resolve digital incidents is a staggering 175 minutes. This delay impacts service reliability, erodes customer satisfaction, revenue and consumes lots of engineering cycles in the process. A time-consuming process often leaves engineering teams overwhelmed and firefighting.

To drive substantial improvements in system reliability and performance, organizations must accelerate their ability to understand causality. This requires a fundamental shift in how we approach observability. By investing in advanced analytics that can reason about causality, we can empower engineers to quickly identify root causes and their effects so they can prioritize what is important and implement effective solutions.

Augmenting Human Ingenuity with Causal Reasoning

In this regard, causal reasoning software like Causely represent a quantum leap forward in the evolution of human-machine collaboration. By combining this capability with OpenTelemetry, the arduous task of causal reasoning can be automated, liberating SREs and developers from the firefighting cycle. Instead of being perpetually mired in troubleshooting, they can dedicate more cognitive resources to innovation and strategic problem-solving.

Imagine these professionals equipped with the ability to process vast quantities of observability data in mere seconds, unveiling intricate causal relationships that would otherwise remain hidden. This is the power of causal reasoning software that is built to amplify the processes associated with Reliability Engineering. They amplify human intelligence, transforming SREs and developers from reactive problem solvers into proactive architects of system reliability.

By accelerating incident resolution from today’s averages (175 minutes as documented in the PagerDuty’s customer survey) to mere minutes, these platforms not only enhance customer satisfaction but also unlock significant potential for business growth. With freed-up time, teams can focus on developing new features, improving system performance, and preventing future issues. Moreover, the insights derived from causal reasoning software can be leveraged to proactively identify vulnerabilities and optimize system performance, elevating the overall reliability and resilience of cloud native architectures.

The convergence of human ingenuity and machine intelligence, embodied in causal reasoning software, is ushering in a new era of problem-solving. This powerful combination enables us to tackle unprecedented challenges with unparalleled speed, accuracy, and innovation.

In the context of reliability engineering, the combination of OpenTelemetry and causal reasoning software offers a significant opportunity to accelerate progress towards continuous application reliability.

Related resources

• Read the blog: Explainability: The Black Box Dilemma in the Real World
• Watch the video: See how Causely leverages OpenTelemetry
• Take the interactive tour: Experience Causely first-hand

In this post, we’ll explore the reasons that OOM kills can occur and provide tactics to combat and prevent them.

Before diving in, it’s worth noting that OOM kills represent one symptom that can have a variety of root causes. It’s important for organizations to implement a system that solves the root cause analysis problem with speed and accuracy, allowing reliability engineering teams to respond rapidly, and to potentially prevent these occurrences in the first place.

Deep dive into an OOM kill

An Out-Of-Memory (OOM) kill in Kubernetes occurs when a container exceeds its memory limit, causing the Kubernetes kernel’s OOM killer to terminate the container. This impacts application stability and requires immediate attention.

Several factors can trigger OOM kills in your Kubernetes environment, including:

• Memory limits exceeded: This is the most common culprit. If a container consistently pushes past its designated memory ceiling, the OOM killer steps in to prevent a system-wide meltdown.
• Memory leaks: Applications can develop memory leaks over time, where they allocate memory but fail to release it properly. This hidden, unexpected growth eventually leads to OOM kills.
• Resource overcommitment: Co-locating too many resource-hungry pods onto a single node can deplete available memory. When the combined memory usage exceeds capacity, the OOM killer springs into action.
• Bursting workloads: Applications with spiky workloads can experience sudden memory surges that breach their limits, triggering OOM kills.

As an example, a web server that experiences a memory leak code bug may gradually consume more and more memory until the OOM killer intervenes to prevent a crash.

Another case could be when a Kubernetes cluster over-commits resources by scheduling too many pods on a single node. The OOM killer may need to step in to free up memory and ensure system stability.

The devastating effects of OOM kills: Why they matter

OOM kills aren’t normally occurring events. They can trigger a cascade of negative consequences for your applications and the overall health of the cluster, such as:

• Application downtime: When a container is OOM-killed, it abruptly terminates, causing immediate application downtime. Users may experience service disruptions and outages.
• Data loss: Applications that rely on in-memory data or stateful sessions risk losing critical information during an OOM kill.
• Performance degradation: Frequent OOM kills force containers to restart repeatedly. This constant churn degrades overall application performance and user experience.
• Service disruption: Applications often interact with each other. An OOM kill in one container can disrupt inter-service communication, causing cascading failures and broader service outages.

If a container running a critical database service experiences an OOM kill, it could result in data loss and corruption. This leads to service disruptions for other containers that rely on the database for information, causing cascading failures across the entire application ecosystem.

Combating OOM kills

There are a few different tactics to combat OOM kills in attempt to operate a memory-efficient Kubernetes environment.

Set appropriate resource requests and limits

For example, you can set a memory request of 200Mi and a memory limit of 300Mi for a particular container in your Kubernetes deployment. Requests ensure the container gets at least 200Mi of memory, while limits cap it at 300Mi to prevent excessive consumption.

resources:
  requests:
    memory: "200Mi"
  limits:
    memory: "300Mi"

While this may mitigate potential memory use issues, it is a very manual process and does not deal at all with the dynamic nature of what we can achieve with Kubernetes. It also doesn’t solve the source issue, which may be a code-level problem triggering memory leaks or failed GC processes.

Transition to autoscaling

Leveraging autoscaling capabilities is a core dynamic option for resource allocation. There are two autoscaling methods:

• Vertical Pod Autoscaling (VPA): VPA dynamically adjusts resource limits based on real-time memory usage patterns. This ensures containers have enough memory to function but avoids over-provisioning.
• Horizontal Pod Autoscaling (HPA): HPA scales the number of pods running your application up or down based on memory utilization. This distributes memory usage across multiple pods, preventing any single pod from exceeding its limit. The following HPA configuration shows an example of scaling based on memory usage:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: my-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80

Monitor memory usage

Proactive monitoring is key. For instance, you can configure Prometheus to scrape memory metrics from your Kubernetes pods every 15 seconds and set up Grafana dashboards to visualize memory usage trends over time. Additionally, you can create alerts in Prometheus to trigger notifications when memory usage exceeds a certain threshold.

Optimize application memory usage

Don’t underestimate the power of code optimization. Address memory leaks within your applications and implement memory-efficient data structures to minimize memory consumption.

Pool disruption budgets (PDB)

When deploying updates, PDBs ensure a minimum number of pods remain available, even during rollouts. This mitigates the risk of widespread OOM kills during deployments. Here is a PDB configuration example that helps ensure minimum pod availability.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
spec:
  minAvailable: 80%
  selector:
    matchLabels:
      app: my-app

Manage node resources

You can apply a node selector to ensure that a memory-intensive pod is only scheduled on nodes with a minimum of 8GB of memory. Additionally, you can use taints and tolerations to dedicate specific nodes with high memory capacity for memory-hungry applications, preventing OOM kills due to resource constraints.

nodeSelector:
  disktype: ssd
tolerations:
  - key: "key"
    operator: "Equal"
    value: "value"
    effect: "NoSchedule"

Use QoS classes

Kubernetes offers Quality of Service (QoS) classes that prioritize resource allocation for critical applications. Assign the highest QoS class to applications that can least tolerate OOM kills. Here is a sample resource configuration with QoS parameters:

resources:
  requests:
    memory: "1Gi"
    cpu: "500m"
  limits:
    memory: "1Gi"
    cpu: "500m"

These are a few potential strategies to help prevent OOM kills. The challenge comes with the frequency with which they can occur, and the risk to your applications when they happen.

As you can imagine, it’s not possible to manually manage resource utilization, and guarantee the stability and performance of your containerized applications within your Kubernetes environment.

Manual thresholds = Rigidity and risk

These techniques can help reduce the risk of OOM kills. The issue is not entirely solved though. By setting manual thresholds and limits, you’re removing many of the dynamic advantages of Kubernetes.

A more ideal way to solve the OOM kill problem is to use adaptive, dynamic resource allocation. Even if you get resource allocation right on initial deployment, there are many factors that change that affect how your application consumes resources. There is also a risk because application and resource issues don’t just affect one pod, or one container. Resource issues can reach every part of the cluster and degrade the other running applications and services.

Which strategy works best to prevent OOM kills?

Vertical Pod Autoscaling (VPA) and Horizontal Pod Autoscaling (HPA) are common strategies used to manage resource limits in Kubernetes containers. VPA adjusts resource limits based on real-time memory usage patterns, while HPA scales pods based on memory utilization.

Monitoring with tools like Prometheus may help with the troubleshooting of memory usage trends. Optimizing application memory usage is no easy feat because it’s especially challenging to identify whether it is infrastructure or code causing the problem.

Pod Disruption Budgets (PDB) may help ensure a minimum number of pods remain available during deployments, while node resources can be managed using node selectors and taints. Quality of Service (QoS) classes prioritize resource allocation for critical applications.

One thing is certain: OOM kills are a common and costly challenge to manage using traditional monitoring tools and methods.

At Causely, we’re focused on applying causal reasoning software to help organizations keep applications healthy and resilient. By automating root cause analysis, issues like OOM kills can be resolved in seconds, and unintended consequences of new releases or application changes can be avoided.

Related resources

• Read the blog: Understanding the Kubernetes Readiness Probe: A Tool for Application Health
• Read the blog: Bridging the gap between observability and automation with causal reasoning

Watch our webinar on Causal AI and why DevOps teams need it:

Thursday, August 22, 2024 – Today, Causely is excited to welcome Yotam Yemini as the company’s Chief Executive Officer. In this role, Yotam will be instrumental in helping Causely fulfill its mission to enable continuous application reliability and streamline the software development lifecycle for modern applications.

“We are excited to welcome Yotam to the team,” said Causely Founder Shmuel Kliger. “Yotam brings an exceptional track record of building and scaling go-to-market strategy. His leadership will be pivotal as we deliver our Causal Reasoning Platform to Engineering and DevOps teams.”

The news comes on the heels of strong market signals from its early access program and the hiring of Francis Cordón as Chief Customer Officer in July. Francis is a domain expert and seasoned technology leader. His experience includes customer success leadership at Quantum Metric and IBM Turbonomic, resiliency and performance architecture at BNY Mellon, and sales engineering at Dynatrace.

“Causely’s causal reasoning software is a well-timed and much needed innovation for the tech industry,” said Alex Sukennik, CIO, Semrush. “The team behind Causely is uniquely suited to help engineering teams assure service levels for business-critical applications.”

In addition to today’s news, the company shares its deepest gratitude to Causely Founder Ellen Rubin for her work building the company, early product and team over the past few years. We wish Ellen the best as she moves on to her next adventures as board member, investor, advisor, and builder in the Boston startup ecosystem.

See Causely first-hand through a self-guided tour at causely.io/resources/experience-causely or sign up today at causely.io/trial.

About Causely

Causely is the leading provider of causal reasoning software, which enables continuous application reliability and streamlines the software development lifecycle for modern applications. Whereas reliability engineering today tends to be overly complex and labor-intensive, Causely amplifies engineering productivity through its patent-pending Causal Reasoning Platform. The platform identifies cause and effect relationships in runtime to automate the process of root cause and impact analysis. This drastically shortens mean-time-to-repair (MTTR), reduces the number of incidents that occur, and empowers engineering teams to build more resilient applications and business services. Causely, Inc. is a remote-first company headquartered in New York City. Visit causely.io to learn more.

Media Contact

Karina Babcock

kbabcock@causely.io

PagerDuty recently published a study¹ that shines a light on how broken our existing incident response systems and practices are. The recent Crowdstrike debacle is further evidence of this. Even with all the investment in observability, AI Ops, automation, and playbooks, things aren’t improving. In some ways, they’re actually worse; we’re collecting more and more data and we’re overloaded with tooling, creating confusion between users and teams who struggle to understand the holistic environment and all of its interdependencies.

With a mean resolution time of 175 minutes, each customer-impacting digital incident costs both time and money. The industry needs to reset and revisit current processes so we can evolve and change the trajectory.

The impact of outages and application downtime

Outages erode customer trust. 90% of IT leaders report that disruptions have reduced customer confidence. Protecting sensitive data, ensuring swift service restoration, and providing real-time customer updates are essential for maintaining trust when digital incidents happen. Thorough, action-oriented postmortems are critical post-incident to prevent recurrences. And – at risk of reinforcing the obvious – IT organizations need to put operational practices in place to minimize outages from happening in the first place.

Yet even though IT leaders understand the implications on customer confidence, incident frequency continues to rise. 59% of IT leaders report an increase in customer-impacting incidents, and it’s not going to get better unless we change the way we observe and mitigate problems in our applications.

Automation can help, but adoption is slow

Despite the growing threat, many organizations are lagging behind in incident response automation:

• Over 70% of IT leaders report that key incident response tasks are not yet fully automated.
• 38% of responders’ time is spent dealing with manual incident response processes.
• Organizations with manual processes take on average 3 hours 58 minutes to resolve customer-impacting incidents, compared to 2 hours 40 minutes for those with automated processes.

It doesn’t take an IT expert to know that spending nearly half their time in manual processes is a waste of resources. And those that have automated operations are still taking almost 3 hours to resolve incidents. Why is incident response still so slow?

It’s not just about process automation. We also need to accelerate decision automation, driven by a deep understanding of the state of applications and infrastructure.

Causal reasoning: The missing link

Causal reasoning technology promises a bridge between observability and automated incident response. We're referring to causal reasoning software that applies machine learning to automatically capture cause and effect relationships. This has the potential to help Dev and Ops teams better plan for changes to code, configurations or load patterns, so they can stay focused on achieving service-level and business objectives instead of firefighting.

With causal reasoning technology, many of the incident response tasks that are currently manual can be automated:

• When service entities are degraded or failing and affecting other entities that make up business services, causal reasoning software surfaces the relationship between the problem and the symptoms it’s causing.
• The team with responsibility for the failing or degraded service is immediately notified so they can get to work resolving the problem. Some problems can be remediated automatically.
• Notifications can be sent to end users and other stakeholders, letting them know that their services are affected along with an explanation for why this occurred and when things will be back to normal.
• Postmortem documentation is automatically generated.
• There’s no more complex triage processes that would otherwise involve multiple teams and managers to orchestrate. Incidents and outages are reduced and root cause analysis is automated, so DevOps teams spend less time troubleshooting and more time shipping code.

Introducing Causely

This potential to transform the way DevOps teams work is why we built Causely. Our Causal Reasoning Platform automatically pinpoints the root cause of observed symptoms based on real-time, dynamic data across the entire application environment. Causely transforms incident response and improves mean time to resolution (MTTR), so DevOps teams focus on building new services and innovations that propel the business forward.

By automatically understanding cause-and-effect relationships in application environments, Causely also enables predictive maintenance and better overall operational resilience. It can help to prevent outages and identify the root cause of potential issues before they escalate.

Here’s how it works, at a high level:

1. Our Causal Reasoning Platform is shipped with out-of-the-box Causal Models that drive the platform’s behavior.
2. Once deployed, Causely automatically discovers the application environment and generates a Topology Graph of it.
3. A Causality Graph is generated by instantiating the Causal Models with the Topology Graph to reflect cause and effect relationships between the root causes and their symptoms, specific to that environment at that point in time.
4. A Codebook is generated from the Causality Graph.
5. Using the Codebook, our Causal Reasoning Platform automatically and continuously pinpoints the root cause of issues.

Users can dig into incidents, understand their root causes, take remediation steps, and proactively plan for new releases and application changes – all within Causely.

This decreases downtime, enhances operational efficiency, and improves customer trust long-term.

It’s time for a new approach

It’s time to shift from manual to automated incident response. Causely can help teams prevent outages, reduce risk, cut costs, and build sustainable customer trust.

Don’t hesitate to contact us about how to bring automation into your organization, or you can see Causely for yourself.

Related resources

• Read the blog: DevOps may have cheated death, but do we all need to work for the king of the underworld?
• Learn about our Causal Reasoning Platform
• See Causely for yourself

What Is The Black Box Dilemma?

Imagine a masterful illusionist, their acts so breathtakingly deceptive that the secrets behind them remain utterly concealed. Today’s software is much the same. We marvel at its abilities to converse, diagnose, drive, and defend, yet the inner workings often remain shrouded in mystery. This is often referred to as the “black box” problem.

The recent CrowdStrike incident is also a stark reminder of the risks of this opacity. A simple software update, intended to enhance security, inadvertently caused widespread system crashes. It’s as if the magician’s assistant accidentally dropped the secret prop, revealing the illusion for what it truly was – an error prone process with no resilience. Had organizations understood the intricacies of CrowdStrike’s software release process, they might have been better equipped to mitigate risks and prevent the disruptions that followed.

This incident, coupled with the rapid advancements in AI, underscores the critical importance of explainability. Understanding the entire lifecycle of software – from conception to operation – is no longer optional but imperative. It is the cornerstone of trust, a shield against catastrophic failures, and an important foundation for accountability.

As our world becomes increasingly reliant on complex systems, understanding their inner workings is no longer a luxury but a necessity. Explainability acts as a key to unlocking the black box, revealing the logic and reasoning behind complex outputs. By shedding light on the decision-making processes of software, AI, and other sophisticated systems, we foster trust, accountability, and a deeper comprehension of their impact.

The Path Forward: Cultivating Explainability in Software

Achieving explainability demands a comprehensive approach that addresses several critical dimensions.

• Software Centric Reasoning and Ethical Considerations: Can the system’s decision-making process be transparently articulated, justified, and aligned with ethical principles? Explainability is essential for building trust and ensuring that systems used to support decision making operate fairly and responsibly.
• Effective Communication and User Experience: Is the system able to communicate its reasoning clearly and understandably to both technical and non-technical audiences? Effective communication enhances collaboration, knowledge sharing, and user satisfaction by empowering users to make informed decisions.
• Robust Data Privacy and Security: How can sensitive information be protected while preserving the transparency necessary for explainability? Rigorous data handling and protection are crucial for safeguarding privacy and maintaining trust in the system.
• Regulatory Compliance and Continuous Improvement: Can the system effectively demonstrate adherence to relevant regulations and industry standards for explainability? Explainability is a dynamic process requiring ongoing evaluation, refinement, and adaptation to stay aligned with the evolving regulatory landscape.

By prioritizing these interconnected elements, software vendors and engineering teams can create solutions where explainability is not merely a feature, but a cornerstone of trust, reliability, and competitive advantage.

An Example of Explainability in Action with Causely

Causely is a pioneer in applying causal reasoning to revolutionize cloud-native application reliability. Our platform empowers operations teams to rapidly identify and resolve the root causes of service disruptions, preventing issues before they impact business processes and customers. The enables dramatic reductions in Mean Time to Repair (MTTR), minimizing business disruptions and safeguarding customer experiences.

Causely also uses its Causal Reasoning Platform to manage its SaaS offering, detecting and responding to service disruptions, and ensuring swift resolution with minimal impact. You can learn more about this in Endre Sara‘s article “Eating Our Own Dogfood: Causely’s Journey with OpenTelemetry & Causal AI”.

Causal Reasoning, often referred to as Causal AI, is a specialized field in computer science dedicated to uncovering the underlying cause-and-effect relationships within complex data. As the foundation of explainable AI, it surpasses the limitations of traditional statistical methods, which frequently produce misleading correlations.

Unlike opaque black-box models, causal reasoning illuminates the precise mechanisms driving outcomes, providing transparent and actionable insights. When understanding the why behind an event is equally important as predicting the what, causal reasoning offers superior clarity and reliability.

By understanding causation, we transcend mere observation to gain predictive and interventional power over complex systems.

Causely is built on a deep-rooted foundation of causal reasoning. The founding team, pioneers in applying this science to IT operations, led the charge at System Management Arts (SMARTS). SMARTS revolutionized root cause analysis for distributed systems and networks, empowering more than 1500 global enterprises and service providers to ensure the reliability of mission-critical IT services. Their groundbreaking work earned industry accolades and solidified SMARTS as a leader in the field.

Explainability is a cornerstone of the Causal Reasoning Platform from Causely. The company is committed to transparently communicating how its software arrives at its conclusions, encompassing both the underlying mechanisms used in Causal Reasoning and the practical application within organizations operational workflows.

Explainable Operations: Enhancing Workflow Efficiency for Continuous Application Reliability

Causely converts raw observability data into actionable insights by pinpointing root causes and their cascading effects within complex application and infrastructure environments.

Today incident response is a complex, resource-intensive process of triage and troubleshooting that often diverts critical teams from strategic initiatives. This reactive approach hampers innovation, erodes efficiency, and can lead to substantial financial losses and reputational damage, when application reliability is not continuously assured.

The complex interconnected nature of cloud-native environments magnifies the effect/impact, leading to cascading disruptions, which can propagate across services when Root Cause problems occur.

By automating the identification and explanation of cause-and-effect relationships, Causely accelerates incident response. Relevant teams responsible for root cause problems receive immediate alerts, complete with detailed explanations of the cause & effect, empowering them to prioritize remediation based on impact. Simultaneously, teams whose services are impacted gain insights into the root causes and who is responsible for resolution, enabling proactive risk mitigation without the need for extensive troubleshooting.

For certain types of Root Cause problems it may also be possible to automate the remediation when they occur without human intervention.

By maintaining historical records of the cause and effect of past Root Cause problems and identifying recurring patterns, Causely enables reliability engineering teams to anticipate future potential problems and implement targeted mitigation strategies.

Causely’s ability to explain the effect of potential degradations and failures, before they even happen through “what if” analysis, also empowers reliability engineering teams to identify single points of failure, changes in load patterns and assess the impact of planned changes on related applications, business processes, and customers.

The result? Through explainability organizations can dramatically reduce MTTR, improve business continuity, and increase cycles for development and innovation. Causely turns reactive troubleshooting into proactive prevention, ensuring application reliability can be continuously assured. This short video tells the story.

Unveiling Causely: How Our Platform Delivers Actionable Insights

Given the historical challenges and elusive nature of automating root cause analysis in IT operations, customer skepticism is warranted in this field. This problem has long been considered the “holy grail” of the industry, with countless vendors falling short of delivering a viable solution.

As a consequence Causely has found that it is important to prioritize transparency and explainability around how our Causal Reasoning Platform works and produces the results described earlier.

Much has been written about this — learn more here.

This is an approach which is grounded in sound scientific principles, making it both effective and comprehensible.

Beyond understanding how the platform works, customers also value transparency around data handling. In this regard our approach to data management offers unique benefits in terms of data privacy and data management cost savings. You can learn more about this here.

In Summary

Explainability is the cornerstone of Causely’s mission. As they advance their technology, their dedication to transparency and understanding will only grow stronger. Don’t hesitate to visit the website or reach out to me or members of the Causely team to learn more about the approach and to experience Causely firsthand.

What is a Kubernetes Readiness Probe?

A readiness probe is essentially a check that Kubernetes performs on a container to ensure that it is ready to serve traffic. This check is needed to prevent traffic from being directed to containers that aren’t fully operational or are still in the process of starting up.

By using readiness probes, Kubernetes can manage the flow of traffic to only those containers that are fully prepared to handle requests, thereby improving the overall stability and performance of the application.

Readiness probes also help in preventing unnecessary disruptions and downtime by only including healthy containers in the load balancing process. This is an essential part of a comprehensive SRE operational practice for maintaining the health and efficiency of your Kubernetes clusters.

How Readiness Probes Work

Readiness probes are configured in the pod specification and can be of three types:

1. HTTP Probes: These probes send an HTTP request to a specified endpoint. If the response is successful, the container is considered ready.
2. TCP Probes: These probes attempt to open a TCP connection to a specified port. If the connection is successful, the container is considered ready.
3. Command Probes: These probes execute a command inside the container. If the command returns a zero exit status, the container is considered ready.

Below is an example demonstrating how to configure a readiness probe in a Kubernetes deployment:

apiVersion: v1

kind: Pod

metadata:

  name: readiness-example

spec:

  containers:

  - name: readiness-container

    image: your-image

    readinessProbe:

      httpGet:

        path: /healthz

        port: 8080

      initialDelaySeconds: 5

      periodSeconds: 10

This YAML file defines the Kubernetes pod with a readiness probe configured based on the following parameters:

1. apiVersion: v1 – Specifies the API version used for the configuration.
2. kind: Pod – Indicates that this configuration is for a Pod.
3. metadata:
   • name: readiness-example – Sets the name of the Pod to “readiness-example.”
4. spec – Describes the desired state of the Pod.
   • containers:
     • name: readiness-container – Names the container within the Pod as “readiness-container.”
     • image: your-image – Specifies the container image to use, named “your-image.”
     • readinessProbe – Configures a readiness probe to check if the container is ready to receive traffic.
       • httpGet:
         • path: /healthz – Sends an HTTP GET request to the /healthz path.
         • port: 8080 – Targets port 8080 for the HTTP GET request.
       • initialDelaySeconds: 5 – Waits 5 seconds before performing the first probe after the container starts.
       • periodSeconds: 10 – Repeats the probe every 10 seconds.

This relatively simple configuration creates a Pod named “readiness-example” with a single container running “your-image.” It includes a readiness probe that checks the

endpoint on port 8080, starting 5 seconds after the container launches and repeating every 10 seconds to determine if the container is ready to accept traffic.

Importance of Readiness Probes

The goal is to make sure you can prevent traffic from being directed to a container that is still starting up or experiencing issues. This helps maintain the overall stability and reliability of your application by only sending traffic to containers that are ready to handle it.

Readiness probes can be used in conjunction with liveness probes to further enhance the health checking capabilities of your containers.

Readiness probes are important for a few reasons:

• Prevent traffic to unready pods: They ensure that only ready pods receive traffic, preventing downtime and errors.
• Facilitate smooth rolling updates: By making sure new pods are ready before sending traffic to them.
• Enhanced application stability: They can help with the overall stability and reliability of your application by managing traffic flow based on pod readiness.

Remember that your readiness probes only check for availability, and don’t understand why a container is not available. Readiness probe failure is a symptom that can manifest from many root causes. It’s important to know the purpose, and limitations before you rely too heavily on them for overall application health.

Best Practices for Configuring Readiness Probes

To make the most of Kubernetes readiness probes, consider the following practices:

1. Define Clear Health Endpoints: Ensure your application exposes a clear and reliable health endpoint.
2. Set Appropriate Timing: Configure initialDelaySeconds and periodSeconds based on your application’s startup and response time.
3. Monitor and Adjust: Continuously monitor the performance and adjust the probe configurations as needed.

For example, if your application requires a database connection to be fully established before it can serve requests, you can set up a readiness probe that checks for the availability of the database connection.

By configuring the initialDelaySeconds and periodSeconds appropriately, you can ensure that your application is only considered ready once the database connection is fully established. This will help prevent any potential issues or errors that may occur if the application is not fully prepared to handle incoming requests.

Limitations of Readiness Probes

Readiness probes are handy, but they only check for the availability of a specific resource and do not take into account the overall health of the application. This means that even if the database connection is established, there could still be other issues within the application that may prevent it from properly serving requests.

Additionally, readiness probes do not automatically restart the application if it fails the check, so it is important to monitor the results and take appropriate action if necessary. Readiness probes are still a valuable tool for ensuring the stability and reliability of your application in a Kubernetes environment, even with these limitations.

Troubleshooting Kubernetes Readiness Probes: Common Issues and Solutions

Slow Container Start-up

Problem: If your container’s initialization tasks exceed the initialDelaySeconds of the readiness probe, the probe may fail.

Solution: Increase the initialDelaySeconds to give the container enough time to start and complete its initialization. Additionally, optimize the startup process of your container to reduce the time required to become ready.

Unready Services or Endpoints

Problem: If your container relies on external services or dependencies (e.g., a database) that aren’t ready when the readiness probe runs, it can fail. Race conditions may also occur if your application’s initialization depends on external factors.

Solution: Ensure that external services or dependencies are ready before the container starts. Use tools like Helm Hooks or init containers to coordinate the readiness of these components with your application. Implement synchronization mechanisms in your application to handle race conditions, such as using locks, retry mechanisms, or coordination with external components.

Misconfiguration of the Readiness Probe

Problem: Misconfigured readiness probes, such as incorrect paths or ports, can cause probe failures.

Solution: Double-check the readiness probe configuration in your Pod’s YAML file. Ensure the path, port, and other parameters are correctly specified.

Application Errors or Bugs

Problem: Application bugs or issues, such as unhandled exceptions, misconfigurations, or problems with external dependencies, can prevent it from becoming ready, leading to probe failures.

Solution: Debug and resolve application issues. Review application logs and error messages to identify the problems preventing the application from becoming ready. Fix any bugs or misconfigurations in your application code or deployment.

Insufficient Resources

Problem: If your container is running with resource constraints (CPU or memory limits), it might not have the resources it needs to become ready, especially under heavy loads.

Solution: Adjust the resource limits to provide the container with the necessary resources. You may also need to optimize your application to use resources more efficiently.

Conflicts Between Probes

Problem: Misconfigured liveness and readiness probes might interfere with each other, causing unexpected behavior.

Solution: Ensure that your probes are configured correctly and serve their intended purposes. Make sure that the settings of both probes do not conflict with each other.

Cluster-Level Problems

Problem: Kubernetes cluster issues, such as kubelet or networking problems, can result in probe failures.

Solution: Monitor your cluster for any issues or anomalies and address them according to Kubernetes best practices. Ensure that the kubelet and other components are running smoothly.

These are common issues to keep an eye out for. Watch for problems that the readiness probes are not surfacing or that might be preventing them from acting as expected.

Summary

Ensuring that your applications are healthy and ready to serve traffic is necessary for maximizing uptime. The Kubernetes readiness probe is one helpful tool for managing Kubernetes clusters; it should be a part of a comprehensive Kubernetes operations plan.

Readiness probes can be configured in pod specifications and can be HTTP, TCP, or command probes. They help prevent disruptions and downtime by ensuring only healthy containers are included in the load-balancing process.

They also use the prevention of sending traffic to unready pods for smooth rolling updates and enhancing application stability. It’s good practice that your readiness probes include defining clear health endpoints, setting appropriate timing, and monitoring and adjusting configurations.

Don’t forget that readiness probes have clear limitations, as they only check for the availability of a specific resource and do not automatically restart the application if it fails the check. A Kubernetes readiness probe failure is merely a symptom that can be attributed to many root causes. To automate root cause analysis across your entire Kubernetes environment, check out Causely for Cloud-Native Applications.

Related resources

• Webinar: What is Causal AI and why do DevOps teams need it?
• Blog: Bridging the gap between observability and automation with causal reasoning
• Product Overview: Causely for Cloud-Native Applications

The Domino Effect: From Certificate Expiry to User Frustration

Imagine a scenario where a microservice’s certificate expires. This seemingly trivial issue prevents it from communicating with others. This disruption creates a ripple effect:

• Microservice Certificate Expiry: The seemingly minor issue is a certificate going past its expiration date.
• Communication Breakdown: This expired certificate throws a wrench into the works, preventing the microservice from securely communicating with other dependent services. It’s like the microservice is suddenly speaking a different language that the others can’t understand.
• Dependent Service Unavailability: Since the communication fails, dependent services can no longer access the data or functionality provided by the failing microservice. Imagine a domino not receiving the push because the first one didn’t fall.
• Errors and Outages: This lack of access leads to errors within dependent services. They might malfunction or crash entirely, causing outages – the domino effect starts picking up speed.
• User Frustration (500 Errors): Ultimately, these outages translate to error messages for the end users. They might see cryptic “500 errors” or experience the dreaded “service unavailable” message – the domino effect reaches the end user, who experiences the frustration.

The Challenge: Untangling the Web of Issues

Cascading failures pose a significant challenge due to the following reasons:

• Network Effect: The root cause gets obscured by the chain reaction of failures, making it difficult to pinpoint the source.
• Escalation Frenzy: Customer complaints trigger incident tickets, leading to a flurry of investigations across multiple teams (DevOps Teams, Service Desk, customer support, etc.).
• Resource Drain: Troubleshooting consumes valuable time from developers, SREs, and support personnel, diverting them from core tasks.
• Hidden Costs: The financial impact of lost productivity and customer dissatisfaction often goes unquantified.

Beyond Certificate Expiry: The Blast Radius of Microservice Issues

Certificate expiry is just one example. Other issues with similar cascading effects include:

• Noisy Neighbors: A resource-intensive microservice can degrade performance for others sharing the same resources (databases, applications) which in turn impact other services that depend on them.
• Code Bugs: Code errors within a microservice can lead to unexpected behavior and downstream impacts.
• Communication Bottlenecks: Congestion or malfunctioning in inter-service communication channels disrupts data flow and service availability.
• Third-Party Woes: Outages or performance issues in third-party SaaS services integrated with your microservices can create a ripple effect.

Platform Pain Points: When Infrastructure Falters

The impact can extend beyond individual microservices. Platform-level issues can also trigger cascading effects:

• Load Balancer Misconfigurations: Incorrectly configured load balancers can disrupt service delivery to clients and dependent services.
• Container Cluster Chaos: Problems within Kubernetes pods, nodes, can lead to application failures and service disruptions.

Blast Radius and Asynchronous Communication: The Data Lag Challenge

Synchronous communication provides immediate feedback, allowing the sender to know if the message was received successfully. In contrast, asynchronous communication introduces a layer of complexity:

• Unpredictable Delivery: Messages may experience varying delays or, in extreme cases, be lost entirely. This lack of real-time confirmation makes it difficult to track the message flow and pinpoint the exact location of a breakdown.
• Limited Visibility: Unlike synchronous communication where a response is readily available, troubleshooting asynchronous issues requires additional effort. You may only have user complaints as a starting point, which can be a delayed and incomplete indicator of the problem.

The root cause of problems could be because of several factors that result delays or lost messages in asynchronous communication:

Microservice Issues:

• Congestion: A microservice overloaded with tasks may struggle to process or send messages promptly, leading to delays.
• Failures: A malfunctioning microservice may be entirely unable to process or send messages, disrupting the flow of data.

Messaging Layer Issues:

Problems within the messaging layer itself can also cause disruptions:

• Congestion: Congestion in message brokers, clusters, or cache instances can lead to delays in message delivery.
• Malfunctions: Malfunctions within the messaging layer can cause messages to be lost entirely.

The Cause & Effect Engine: Unveiling the Root of Microservice Disruptions in Real Time

So what can we do to tame this chaos?

Imagine a system that acts like a detective for your application services. It understands all of the cause-and-effect relationships within your complex architecture. It does this by automatically discovering and analyzing your environment to maintain an up-to-date picture of services, infrastructure and dependencies and from this computes a dynamic knowledge base of root causes and the effects they will have.

This knowledge is automatically computed in a Causality Graph that depicts all of the relationships between the potential root causes that could occur and the symptoms they may cause. In an environment with thousands of entities, it might represent hundreds of thousands of problems and the set of symptoms each one will cause.

A separate data structure is derived from this called a “Codebook“. This table is like a giant symptom checker, mapping all the potential root causes (problems) to the symptoms (errors) they might trigger.

Hence, each root cause in the Codebook has a unique signature, a vector of m probabilities, that uniquely identifies the root cause. Using the Codebook, the system quickly searches and pinpoints the root causes based on the observed symptoms.

The Causality Graph and Codebook are constantly updated as application services and infrastructure evolve. This ensures the knowledge in the Causality Graph and Codebook stays relevant and adapts to changes.

These powerful capabilities enable:

• Machine Speed Root Cause Identification: Unlike traditional troubleshooting, the engine can pinpoint the culprit in real time, saving valuable time and resources.
• Prioritization Based on Business Impact: By revealing the effects of specific root causes on related services, problem resolution can be prioritized.
• Reduced Costs: Faster resolution minimizes downtime and associated costs.
• Improved Collaboration: Teams responsible for failing services receive immediate notifications and a visualize a Causality Graph explaining the issue’s origin and impact. This streamlines communication and prioritizes remediation efforts based on the effect the root cause problem has.
• Automated Actions: In specific cases, the engine can even trigger automated fixes based on the root cause type.
• Empowered Teams: Teams affected by the problem are notified but relieved of troubleshooting burdens. They can focus on workarounds or mitigating downstream effects, enhancing overall system resilience.

The system represents a significant leap forward in managing cloud native applications. By facilitating real-time root cause analysis and intelligent automation, it empowers teams to proactively address disruptions and ensure the smooth operation of their applications.

The knowledge in the system is not just relevant to optimize the incident response process. It is also valuable for performing “what if” analysis to understand what the impact of future changes and planned maintenance will have so that steps can be taken to proactively understand and mitigate the risks of these activities.

Through its understanding of cause and effect, it can also play a role in business continuity planning, enabling teams to identify single points of failure in complex services to improve service resilience.

The system can also be used to streamline the process of incident postmortems because it contains the prior history of previous root cause problems, why they occurred and what the effect was — their causality. This avoids the complexity and time involved in reconstructing what happened and enables mitigating steps to be taken to avoid recurrences.

The Types of Root Cause Problems & Their Effects

The system computes its causal knowledge based on Causal Models. These describe the behaviours of how root cause problems will propagate symptoms along relationships to dependent entities independently of a given environment. This knowledge is instantiated through service and infrastructure auto discovery to create the Causal Graph and Codebook.

Examples of these types of root cause problems that are modeled in the system include:

Science Fiction or Reality

The inventions behind the system go back to the 90’s, and was at the time and still is groundbreaking. It was successfully deployed, at scale, by some of the largest telcos, system integrators and Fortune 500 companies in the early 2000’s. You can read about the original inventions here.

Today the problems that these inventions set out to address have not changed and the adoption of cloud-native technologies has only heightened the need for a solution. As real-time data has become pervasive in today’s application architectures, every second of service disruption is a lost business opportunity.

These inventions have been taken and engineered in a modern, commercially available platform by Causely to address the challenges of assuring continuous application reliability in the cloud-native world. The founding engineering team at Causely were the creators of the tech behind two high-growth companies: SMARTS and Turbonomic.

If you would like to learn more about this, book a meeting with the Causely team. We'd love to chat!

Watch the video to see Causely in action, or take the product for a self-guided tour.

A Brief Round-Up

The recent advances in AI and Machine Learning open up enormous possibilities for the observability sector. Observability backends ingest vast amounts of telemetry and therefore have an ocean of raw data to mine for rich analytics and diagnostics. In addition to this, LLM's themselves have become a first class IT citizen in many organisations and are therefore also a subject for observability. Indeed, OpenTelemetry have now released a set of semantic conventions for working with Generative AI.

Observability is one of the most dynamic and competitive sectors in the IT industry and it is no surprise that vendors have begun to incorporate AI features into their platforms. It goes without saying that AI is a broad term and is used in different ways - it is generally used to cover both machine learning as well as generative AI. In this article we will briefly survey a number of leading platforms and products in the observability market and look at some of the ways they have incorporated AI capabilities into their products.

IBM Instana - Intelligent Remediation

IBM has a long and illustrious track record in the fields of AI and Machine Learning - including the 1997 victory of Deep Blue over chess grandmaster Gary Kasparov and IBM Watson’s winning turn on the US TV quiz Jeopardy in 2011. Watson has now been superseded by WatsonX, and this is the engine which powers the Intelligent Remediation feature in IBM’s Instana observability platform. Intelligent Remediation is a preview technology which continuously monitors a system for faults and anomalies. As well as drawing upon system telemetry, it also uses expert knowledge for causal analysis and then suggests remediations. The remediations can be implemented using pre-built actions selected from a catalogue. As well as the Remediation feature, Instana also has AI-driven capabilities for summarising, diagnostics and making recommendations.

Logz.Io - Anomaly Detection

Logz.Io is a popular full-stack observability platform built on top of open source technologies such as OpenSearch, Prometheus and Jaeger. Whilst their platform has been equipped with AI tooling for some time, the company are circumspect in not overplaying the current capabilities of AI. Whilst they recognise that AI can assist in areas such as reducing noise and summarising incidents, they do not make any claims in terms of causal analysis or remediation. You can learn more on the company’s AI posture from this really illuminating webinar.

The Logz.Io platform ships with an Observability IQ Assistant, which harnesses AI to support natural language querying and chat-based analytics on your telemetry data. The most powerful AI feature in the Logz.Io platform though, is probably the Anomaly Detection tooling that is integrated into the App 360 module. One problem with anomaly detection is that it is not business-aware, and, if not applied carefully, it may end up creating yet more alert fatigue. To combat this, Logz.io Anomaly Detection allows users to target critical services and take a more SLO-driven approach.

Elastic - Supercharging Search

The ELK Stack has been at the forefront of the log aggregation and analytics space for many years. Despite controversies over licensing, Elastic is still a hugely popular and influential product. Highly powerful search capabilities are at the core of its product offering and it is no surprise that this is a domain where Elastic seeks to differentiate itself from other platforms in terms of its AI tooling. It seems though, that Elastic’s ambitions extend far beyond log searching and it is positioning itself as a first-choice platform for advanced corporate data analytics.

The centrepiece of this vision is the Search AI Lake, which incorporates RAG, search and security functions and is built on a cloud-native architecture. The company claims that this enables search over vast volumes of data at high speed and low cost. A quick glance at Elastic's latest financial report really highlights the strategic importance of AI to the company. Pretty much every item listed in the Product Innovations and Updates section is AI-related. Search is obviously an area with great potential for AI and other vendors such as AWS have also incorporated AI into their search functionality. At the moment, this technology is still at the experimental stage, but when it matures natural language search over telemetry data will be a huge win for making observability systems accessible and of value across the enterprise.

New Relic - Observability for LLM's'

The AI revolution poses a two-fold challenge for observability vendors. As well as harnessing AI to create more powerful systems, they also need to extend their functional scope to provide insights into the LLM functionality that customers are building into their systems. New Relic were the first major vendor to add LLM monitoring to their stack - although Datadog and Elastic have now followed suit. The New Relic AI monitoring product will check for “bias, toxicity, and hallucinations“ as well as identifying processing bottlenecks and scanning for potential vulnerabilities. As well as the usual APM signals, the tool also captures AI-specific metrics such as response quality and token counts. Sending data to LLM's can obviously represent a potential security issue, so the system also includes safeguards for protecting sensitive data.

Grafana

Grafana's initial application of AI to their stack concentrated on reducing toil and providing 'delight' for the user. This entailed functionality such as generating incident summaries or providing automated suggestions for names and titles of panels and other objects. Recently though, they have started to ramp up their AI features. One of the most notable of these is AI-powered insights for continuous profiling. Flame graphs are a great tool, they can, however, be visually very dense and it can take some time to unpack all the data and identify root causes and bottlenecks. The Grafana Cloud Profiles tool now supports an AI-powered flame graph reader to speed up and simplify diagnostics and analysis.

Causely

Whilst most of the systems in this review harness AI to complement their existing stack, Causely is built on AI from the ground up. As the name suggests, it uses Causal AI - built on expert systems knowledge - to carry out root cause analysis as well as predictive diagnostics. This contrasts with most other systems whose root cause analysis is actually powered by correlation and inference - which are less reliable approaches. Causely is not a full stack system, instead it plugs in to your existing stack. If you are interested in digging deeper into Causely and causal AI then take a look at our recent feature article.

Open Source

It is not only the large vendors who are harnessing AI capabilities to build new products and features. K8sGPT is an open source project aiming to ease the burden for K8S admins by tapping into AI backends for assistance with diagnostics. Like much AI-based tooling, it works as a co-pilot rather than an autonomous operator. The tool is built on a set of Analysers which map to K8S resources such as pods, nodes, services etc and continually scan your cluster, looking for errors. It then sends a digest of the error context to the backend AI (it doesn’t have to be OpenAI) and presents the potential fixes to the user.

Langtrace AI is an open source tool offering observability for LLM Apps. It can be self-hosted, but there is also a SAAS version of the product. It provides full OpenTelemetry tracing support and also provides metrics around costs, accuracy and latency. It offers support for the Pinecone and ChromaDB vector databases and integrates with OpenAI and Anthropic LLM’s. There is also an integration for viewing your traces in SigNoz. There is an ambitious list of new features on the project’s backlog and it is likely to evolve quickly.

Conclusion

It is still early days in terms of the incorporation of AI capabilities in observability systems. However, there are a few trends we can see emerging and AI features seem to be coalescing around:

• reducing toil
• causal analysis/anomaly detection
• natural language search
• LLM observability

Some vendors, such as Cleric, have made some very bold claims about creating an AI Site Reliability Engineer and others have spoken about "closing the loop" but, in reality, this is a long way off. The changes we are seeing are incremental rather than transformational. Innovations such as natural language search will make tasks such as querying easier and more accessible, but these are assistive technologies. As the author of a recent article on the Incident.io blog put it, AI is best envisaged as an exoskeleton rather than a robot.

But this power comes at a cost. The intricate architecture behind real-time services can make troubleshooting issues a nightmare. Organizations that rely on real-time data to deliver products and services face a critical challenge: ensuring data is delivered fresh and on time. Missing data or delays can cripple the user experience and demand resolutions within minutes, if not seconds.

This article delves into the world of real-time data challenges. We’ll explore the business settings where real-time data is king, highlighting the potential consequences of issues. Then I will introduce a novel approach that injects automation into the troubleshooting process, saving valuable time and resources, but most importantly mitigating the business impact when problems arise.

Lags & Missing Data: The Hidden Disruptors Across Industries

Lags and missing data can be silent assassins, causing unseen disruptions that ripple through various industries. Let’s dig into the specific ways these issues can impact different business sectors.

Financial markets

• Trading: In high-frequency trading, even milliseconds of delay can mean the difference between a profitable and losing trade. Real-time data on market movements is crucial for making informed trading decisions.
• Fraud detection: Real-time monitoring of transactions allows financial institutions to identify and prevent fraudulent activity as it happens. Delays in data can give fraudsters a window of opportunity.
• Risk management: Real-time data on market volatility, creditworthiness, and other factors helps businesses assess and manage risk effectively. Delays can lead to inaccurate risk assessments and potentially large losses.

Supply chain management

• Inventory management: Real-time data on inventory levels helps businesses avoid stockouts and optimize inventory costs. Delays can lead to overstocking or understocking, impacting customer satisfaction and profitability.
• Logistics and transportation: Real-time tracking of shipments allows companies to optimize delivery routes, improve efficiency, and provide accurate delivery estimates to customers. Delays can disrupt logistics and lead to dissatisfied customers.
• Demand forecasting: Real-time data on customer behavior and sales trends allows businesses to forecast demand accurately. Delays can lead to inaccurate forecasts and production issues.

Customer service

• Live chat and phone support: Real-time access to customer data allows support agents to personalize interactions and resolve issues quickly. Delays can lead to frustration and longer resolution times.
• Social media monitoring: Real-time tracking of customer sentiment on social media allows businesses to address concerns and build brand reputation. Delays can lead to negative feedback spreading before it’s addressed.
• Personalization: Real-time data on customer preferences allows businesses to personalize website experiences, product recommendations, and marketing campaigns. Delays can limit the effectiveness of these efforts.

Manufacturing

• Machine monitoring: Real-time monitoring of machine performance allows for predictive maintenance, preventing costly downtime. Delays can lead to unexpected breakdowns and production delays.
• Quality control: Real-time data on product quality allows for immediate identification and correction of defects. Delays can lead to defective products reaching customers.
• Process optimization: Real-time data on production processes allows for continuous improvement and optimization. Delays can limit the ability to identify and address inefficiencies.

Other examples

• Online gaming: Real-time data is crucial for smooth gameplay and a fair playing field. Delays can lead to lag, disconnects, and frustration for players.
• Healthcare: Real-time monitoring of vital signs and patient data allows for faster diagnosis and treatment. Delays can have serious consequences for patient care.
• Energy management: Real-time data on energy consumption allows businesses and utilities to optimize energy use and reduce costs. Delays can lead to inefficient energy usage and higher costs.
• Cybersecurity: Real-time data is the backbone of modern cybersecurity, enabling rapid threat detection, effective incident response, and accurate security analytics. However, delays in the ability to see and understand this data can create critical gaps in your defenses. From attackers having more time to exploit vulnerabilities to outdated security controls and hindered automated responses, data lags can significantly compromise your ability to effectively combat cyber threats.

As we’ve seen, the consequences of lags and missing data can be far-reaching. From lost profits in financial markets to frustrated customers and operational inefficiencies, these issues pose a significant threat to business success. Having the capability to identify the root cause, impact and remediate issues with precision and speed is an imperative to mitigate the business impact.

Request a demo

The Delicate Dance: A Web of Services and Hidden Culprits

Modern user experiences that leverage real-time data rely on complex chains of interdependent services – a delicate dance of microservices, databases, messaging platforms, and virtualized compute infrastructure. A malfunction in any one element can create a ripple effect, impacting the freshness and availability of data for users. This translates to frustrating delays, lags, or even complete UX failures.

Let’s delve into the hidden culprits behind these issues and see how seemingly minor bottlenecks can snowball into major UX problems:

Slowdown Domino with Degraded Microservice

• Scenario: A microservice responsible for product recommendations experiences high latency due to increased user traffic and internal performance degradation (e.g., memory leak, code inefficiency).
• Impact 1: The overloaded and degraded microservice takes significantly longer to process requests and respond to the database.
• Impact 2: The database, waiting for the slow microservice response, experiences delays in retrieving product information.
• Impact 3: Due to the degradation, the microservice might also have issues sending messages efficiently to the message queue. These messages contain updates on product availability, user preferences, or other relevant data for generating recommendations.
• Impact 4: Messages pile up in the queue due to slow processing by the microservice, causing delays in delivering updates to other microservices responsible for presenting information to the user.
• Impact 5: The cache, not receiving timely updates from the slow microservice and the message queue, relies on potentially outdated data.
• User Impact: Users experience significant delays in seeing product recommendations. The recommendations themselves might be inaccurate or irrelevant due to outdated data in the cache, hindering the user experience and potentially leading to missed sales opportunities. Additionally, users might see inconsistencies between product information displayed on different pages (due to some parts relying on the cache and others waiting for updates from the slow microservice).

Message Queue Backup

• Scenario: A sudden spike in user activity overwhelms the message queue handling communication between microservices.
• Impact 1: Messages pile up in the queue, causing delays in communication between microservices.
• Impact 2: Downstream microservices waiting for messages experience delays in processing user actions.
• Impact 3: The cache, not receiving updates from slow microservices, might provide outdated information.
• User Impact: Users experience lags in various functionalities – for example, slow loading times for product pages, delayed updates in shopping carts, or sluggish responsiveness when performing actions.

Cache Miss Cascade

• Scenario: A cache experiences a high rate of cache misses due to frequently changing data (e.g., real-time stock availability).
• Impact 1: The microservice needs to constantly retrieve data from the database, increasing the load on the database server.
• Impact 2: The database, overloaded with requests from the cache, experiences performance degradation.
• Impact 3: The slow database response times further contribute to cache misses, creating a feedback loop.
• User Impact: Users experience frequent delays as the system struggles to retrieve data for every request, leading to a sluggish and unresponsive user experience.

Kubernetes Lag

• Scenario: A resource bottleneck occurs within the Kubernetes cluster, limiting the processing power available to microservices.
• Impact 1: Microservices experience slow response times due to limited resources.
• Impact 2: Delays in microservice communication and processing cascade throughout the service chain.
• Impact 3: The cache might become stale due to slow updates, and message queues could experience delays.
• User Impact: Users experience lags across various functionalities, from slow page loads and unresponsive buttons to delayed updates in real-time data like stock levels or live chat messages.

Even with advanced monitoring tools, pinpointing the root cause of these and other issues can be a time-consuming detective hunt. The triage & troubleshooting process often requires a team effort, bringing together experts from various disciplines. Together, they sift through massive amounts of observability data – traces, metrics, logs, and the results of diagnostic tests – to piece together the evidence and draw the right conclusions so they can accurately determine the cause and effect. The speed and accuracy of the process is very much determined by the skills of the available resources when issues arise

Only when the root cause is understood can the responsible team make informed decisions to resolve the problem and restore reliable service.

Transforming Incident Response: Automation of the Triage & Troubleshooting Process

Traditional methods of incident response, often relying on manual triage and troubleshooting, can be slow, inefficient, and prone to human error. This is where automation comes in, particularly with the advancements in Artificial Intelligence (AI). Specifically, a subfield of AI called Causal AI presents a revolutionary approach to transforming incident response.

Causal AI goes beyond correlation, directly revealing cause-and-effect relationships between incidents and their root causes. In an environment where services rely on real-time data and fast resolution is critical, Causal AI offers significant benefits:

• Automated Triage: Causal AI analyzes alerts and events to prioritize incidents based on severity and impact. It can also pinpoint the responsible teams, freeing resources from chasing false positives.
• Machine Speed Root Cause Identification: By analyzing causal relationships, Causal AI quickly identifies the root cause, enabling quicker remediation and minimizing damage.
• Smarter Decisions: A clear understanding of the causal chain empowers teams to make informed decisions for efficient incident resolution.

Causely is leading the way in applying Causal AI to incident response for modern cloud-native applications. Causely’s technology utilizes causal reasoning to automate triage and troubleshooting, significantly reducing resolution times and mitigating business impact. Additionally, Causal AI streamlines post-incident analysis by automatically documenting the causal chain.

Beyond reactive incident response, Causal AI offers proactive capabilities that focus on measures to reduce the probability of future incidents and service disruptions, through improved hygiene, predictions and “what if” analysis.

The solution is built for the modern world that incorporates real-time data, applications that communicate synchronously and asynchronously, and leverage modern cloud building blocks (databases, caching, messaging & streaming platforms and Kubernetes).

This is just the beginning of the transformative impact Causal AI is having on incident response. As the technology evolves, we can expect even more advancements that will further streamline and strengthen organizations’ ability to continuously assure the reliability of applications.

If you would like to learn more about Causal AI and its applications in the world of real-time data and cloud-native applications, don’t hesitate to reach out.

You may also want to check out an article by Endre Sara which explains how Causely is using Causely to manage its own SaaS service, which is built around a real-time data architecture.

Related Resources

• Watch the on-demand webinar: What is Causal AI and why do DevOps teams need it?
• Read the blog: Bridging the gap between observability and automation with causal reasoning
• See causal AI in action: Request a demo of Causely

Graphic from “Crossing the Chasm” showing the gap between early adopters and early majority. Image source: Patel, Neeral & Patlas, Michael & Mafeld, Sebastian. (2020).

Recently while hosting the Causely team at their beautiful new offices for our quarterly meetup, our investors at 645 Ventures gave everyone a copy of the latest edition of Crossing the Chasm. It was an opportunity for me to review the basic concepts. Re-reading it brought back years of memories of startups past and made me think about the book in a new context: how have Moore’s fundamental arguments withstood the decades of technology trends I’ve experienced personally? Specifically, what does “crossing the chasm” actually mean when new product adoption can be so different from one technology shift to another?

A Quick Refresher

One of Moore’s key insights is that innovators and early adopters are willing to try to a new product and work with a new company because it meets some specific needs – innovators love being first to try cool things, and early adopters see new technology as a way to solve problems not currently being met by existing providers. These innovators/early adopters then share their experiences with others in their organizations and industries, who trust and respect their knowledge. This allows the company to reach a broader market over time, cross the chasm and begin adoption by the early majority. Many years can go by during this process, much venture funding will be spent, and still the company may only have penetrated a small percent of the market. Only years later (and with many twists and turns) will the company reach the late majority and finally the laggards.

Photo by David Lusvardi on Unsplash

The Chasm Looks Different Over Time

Netezza and Data Warehousing

I started to think about this in terms of technology shifts that I’ve lived through. Earlier in my career I had the good fortune to be part of a company that crossed the chasm: Netezza. We built a data warehousing system that was 100x the performance of existing solutions from IBM and Oracle, at half the cost. While this was clearly a breakthrough product, the data warehousing industry had not changed in any meaningful way for over a decade and the database admins who ran the existing solutions were in no rush to try something new or different, for all the usual reasons. Within 10 years we created a new category, the “data warehouse appliance.” We gained traction first with some true innovators and then with early adopters who brought the product into their data centers, proved the value and then used it more widely as a platform. However, “crossing the chasm” took many more years – we had a couple of hundred customers at the time of IPO – and only once the company was acquired by IBM did more mainstream adopters become ready to buy (since no one ever gets fired for buying IBM, etc). The product was so good that it remained in the market for over 20 years until the cloud revolution changed things, but it’s hard to argue that it ever gained broad market adoption compared with more traditional data warehouses.

Datadog and Cloud Observability

A second example, which is closer to the market my current company operates in, is Datadog in the observability space. Fueled by the cloud computing revolution (which itself was in the process of crossing the chasm when Datadog was founded in 2010), Datadog rode the new technology wave and solved old problems of IT operations management for new cloud-based applications. While this is not necessarily creating a new category, the company moved very quickly from early cloud innovators and adopters to mainstream customers, rocketing to around $1B in revenues in 10 years. What’s more impressive is that Datadog has become the de facto standard for cloud application observability; today the company has almost 30,000 customers and is still growing quickly in the “early majority” part of the observability market. Depending on which market size numbers you use, Datadog has already crossed the chasm or is well underway, with plenty of room to expand with “late majority” customers.

OpenAI and GenAI

Finally, think about the market adoption in the current GenAI revolution. 100 million users adopted ChatGPT within two months of its initial release in late 2022 and OpenAI claims that over 80% of F500 companies are already using it. No need for me to add more statistics here (e.g., comparisons vs adoptions of internet and social technologies) – it’s clear that this is one of the fastest chasm-crossings in history, although it’s not yet clear how companies plan to use the new AI products even as they adopt them. The speed and market confusion make it hard to envision what crossing the chasm will mean for mainstream adopters and how the technology will fully solve a specific set of problems.

Defining Success as You Cross

Thinking through these examples made me realize some things I hadn’t understood earlier in my career:

• It’s easy to confuse a large financial outcome (through IPO or acquisition) with “crossing the chasm”, since the assumption is often that you’ve had enough market success for the outcome. In fact, these are not necessarily related issues. It’s possible to have a large $ acquisition or even a successful IPO (as Netezza did) without having yet crossed to mainstream adoption.
• The market and technology trends that surround and support a new company and product can lead to very different experiences in crossing the chasm: You can have a breakthrough and exciting product in a slow-moving market without major technology tailwinds (e.g., data warehousing in the early 2000s) but you can also have a huge tailwind like cloud computing that drives a new product to more mainstream adoption within 10 years (e.g., Datadog’s cloud-based observability). Or you can have a hyper-growth technology shift like GenAI that shrinks the entire process into a few years, leaving the early and mainstream adopters jumbled together and trying to determine how to turn the new products into something truly useful.
• It can be hard to tell if you’ve really crossed the chasm since people think of many metrics that indicate adoption: % of customers in the total addressable market (Moore defines a bell curve with percentages for each stage, but I’ve rarely seen people use these strictly), number of monthly active users, revenue market share, penetration within enterprise accounts, etc. Also at the early majority phase, the company can see so much excitement from early customers and analysts (“We’re a leader in the Gartner Magic Quadrant!”) that founders can confuse market awareness and marketing “noise” with true adoption by customers that are waiting for more proof points and additional product capabilities that weren’t as critical for the early adopters. It’s important to keep your eye on these requirements to avoid stalling out once you’ve reached the other side of the chasm.

I would love to hear from other founders who have made this journey! Please share your thoughts on lessons learned and how you’re thinking about the chasm in the new AI-centric world.

Related Resources

• Don’t Forget These 3 Things When Starting a Cloud Venture
• Are You Ready to Eat Your Own Dogfood?
• Building Startup Culture Isn’t Like It Used To Be

Causal reasoning solves a problem that observability can’t

Combining observability with causal reasoning can revolutionize automated troubleshooting and boost application health. By pinpointing the “why” behind issues, causal reasoning reduces human error and labor.

This triggers a lot of questions from application owners and developers, including:

• What is observability?
• What is the difference between causal reasoning and observability?
• How does knowing causality increase performance and efficiency?

Let’s explore these questions to see how observability pairs with causal reasoning for automated troubleshooting and more resilient application health.

What is Observability?

Observability can be described as observing the state of a system based on its outputs. The three common sources for observability data are logs, metrics, and traces.

• Logs provide detailed records of ordered events.
• Metrics offer quantitative but unordered data on performance.
• Traces show the journey of specific requests through the system.

The goal of observability is to provide insight into system behavior and performance to help identify and resolve any issues that are happening. However, traditional monitoring tools are “observing” and reporting in silos.

“Observability is not control. Not being blind doesn’t make you smarter.” – Shmuel Kliger, Causely founder in our recent podcast interview

Unfortunately, this falls short of the goal above and requires tremendous human effort to connect alerts, logs, and anecdotal application knowledge with possible root cause issues.

For example, if a website experiences a sudden spike in traffic and starts to slow down, observability tools can show logs of specific requests and provide metrics on server response times. Furthermore, engineers digging around inside these tools may be able to piece together the flow of traffic through different components of the system to identify candidate bottlenecks.

The detailed information can help engineers identify and address the root cause of the performance degradation. But we are forced to rely on human and anecdotal knowledge to augment observability. This human touch may provide guiding information and understanding that machines alone are not able to match today, but that comes at the cost of increased labor, staff burnout, and lost productivity.

Data is not knowledge

Observability tools collect and analyze large amounts of data. This has created a new wave of challenges among IT operations teams and SREs, who are now left trying to solve a costly and complex big data problem.

The tool sprawl you experience, where each observability tool offers a unique piece of the puzzle, makes this situation worse and promotes inefficiency. For example, if an organization invests in multiple observability tools that each offer different data insights, it can create a fragmented and overwhelming system that hinders rather than enhances understanding of the system’s performance holistically.

This results in wasted resources spent managing multiple tools and an increased likelihood of errors due to the complexity of integrating and analyzing data from various sources. The resulting situation ultimately undermines the original goal of improving observability.

Data is not action

Even with a comprehensive observability practice, the fundamental issue remains: how do you utilize observability data to enhance the overall system? The problem is not about having some perceived wealth of information at your fingertips. The problem is relying on people and processes to interpret, correlate, and then decide what to do based on this data.

You need to be able to analyze and make informed decisions in order to effectively troubleshoot and assure continuous application performance. Once again, we find ourselves leaving the decisions and action plans to the team members, which is a cost and a risk to the business.

Causal reasoning: cause and effect

Analysis is essential to understanding the root cause of issues and making informed decisions to improve the overall system. By diving deep into the data and identifying patterns, trends, and correlations, organizations can proactively address potential issues before they escalate into major problems.

Causal reasoning uses available data to determine the cause of events, identifying whether code, resources, or infrastructure are the root cause of an issue. This deep analysis helps proactively and preventatively address potential problems before they escalate.

For example, a software development team may have been alerted about transaction slowness in their application. Is this a database availability problem? Have there been infrastructure issues happening that could be affecting database performance?

When you make changes based on observed behavior, it’s extremely important to consider how these changes will affect other applications and systems. Changes made without the full context are risky.

Figure 1: A PostgreSQL-based application experiencing database congestion

Using causal reasoning based on the observed environment shows that a recent update to the application code is causing crashes for users during specific transactions. A code update may have introduced inefficient database calls, which is affecting the performance of the application. That change can also go far beyond just the individual application.

If a company decides to update their software without fully understanding how it interacts with other systems, it could result in technical issues that disrupt operations and lead to costly downtime. This is especially challenging in shared infrastructure where noisy neighbors can affect every adjacent application.

Figure 2: Symptoms, causes, and impact determination

This is an illustration showing how causal AI software can connect the problem to active symptoms, while understanding the likelihood of each potential cause. This is causal reasoning in action that also understands the effect on the rest of the environment as we evaluate potential resolutions.

Now that we have causal reasoning for the true root cause, we can go even further by introducing remediation steps.

Automated remediation and system reliability

Automated remediation involves the real-time detection and resolution of issues without the need for human intervention. Automated remediation plays an indispensable role in reducing downtime, enhancing system reliability, and resolving issues before they affect users.

Yet, implementing automated remediation presents challenges, including the potential for unintended consequences like incorrect fixes that could worsen issues. Causal reasoning takes more information into account to drive the decision about root cause, impact, remediation options, and the effect of initiating those remediation options.

This is why a whole environment view combined with real-time causal analysis is required to be able to safely troubleshoot and take remedial actions without risk while also reducing the labor and effort required by operations teams.

Prioritizing action over visibility

Observability is a component of how we monitor and observe modern systems. Extending beyond observability with causal reasoning, impact determination, and automated remediation is the missing key to reducing human error and labor.

In order to move toward automation, you need trustworthy, data-driven decisions that are based on a real-time understanding of the impact of behavioral changes in your systems. Those decisions can be used to trigger automation and the orchestration of actions, ultimately leading to increased efficiency and productivity in operations.

Automated remediation can resolve issues before they escalate, and potentially before they occur at all. The path to automated remediation requires an in-depth understanding of the components of the system and how they behave as an overall system.

Integrating observability with automated remediation empowers organizations to boost their application performance and reliability. It’s important to assess your observability practices and incorporate causal reasoning to boost reliability and efficiency. The result is increased customer satisfaction, IT team satisfaction, and risk reduction.

Related resources

• What is causal AI and why do DevOps teams need it? Watch the webinar.
• Moving beyond traditional RCA in DevOps: Read the blog.
• Assure application reliability with Causely: See the product.

The relationships between application and infrastructure components are complex and constantly evolving, which means relationships and related entities are dynamically changing too. It’s important not to conflate correlation with causation, or to assume that all application issues stem from infrastructure limitations.

In this webinar, Endre Sara defines Causal AI, explains what it means for IT, and talks through specific use cases where it can help IT and DevOps practitioners be more efficient.

We’ll dive into practical implementations, best practices, and lessons learned when applying Causal AI to IT. Viewers will leave with tangible ideas about how Causal AI can help them improve productivity and concrete next steps for getting started.

Tight on time? Check out these highlights

• What is root cause and what is it not? Endre defines what we mean by “root cause” and how to know you’ve correctly identified it.
• How do you install Causely? What resources does it demand? Endre shows how easy it is.

Startup culture used to happen organically

Back in the day, we took a small office space, gathered the initial team and started designing and building. Our first few months were often in the incubator space at one of our early investors. This was a fun and formative experience, at least until we got big enough to be kicked out (“you’re funded, now get your own space!”). Sitting with a small group crowded around a table and sharing ideas with each other on all topics may not have been very comfortable or even efficient. But it did create a foundational culture based on jokes, stories and decisions we would refer back to for years to come. Also, it established the extremely open and non-hierarchical cultural norms we wanted to encourage as we added people.

Once we hit initial critical mass and needed more space for breakouts or private discussions, it was off to the Boston real estate market to see what could possibly be both affordable and reasonable for commutes. The more basic the space, the better in many ways, since it emphasized the need to run lean and spend money only on the things that mattered – hiring, engineering tools, early sales and marketing investments, etc. But most important was to spend on things that would encourage the team to get to know each other and build trust. Lunches, dinners, parties, local activities were all important, as was having the right snacks, drinks and games in the kitchen area to encourage people to hang out together (it’s amazing how much the snacks matter).

The new normal

Fast forward to now, post-Covid and all the changes that have occurred in office space and working remotely. Causely is a remote-from-birth company, with people scattered around the US and a couple internationally. I would never have considered building a company this way before Covid, but when Shmuel and I decided to start the company, it just didn’t seem that big an issue anymore. We committed ourselves to making the extra effort required to build culture remotely, and talked about it frequently with the early team and investors.

Check out our open roles.

In my experiences hanging out with the local Boston tech community and hearing stories from other entrepreneurs, I’ve noticed some of the following trends (which I believe are typical for the startup world):

• Most companies have one or more offices that people come to weekly, but not daily; attendance varies by team and is tied to days of the week that the team is gathering for meetings or planning. Peak days are Tues-Thurs but even then, attendance may vary widely.
• Senior managers show up more frequently to encourage their teams to come in, but they don’t typically enforce scheduling.
• The office has become more a place to build social and mentoring relationships and less about getting work done, which may honestly be more efficient from home.
• Employees like to come in, and more junior staff in particular benefit from in-person interaction with peers and managers, as well as having a separate workspace from their living space. But the flexibility of working remotely is very hard to give up and is something people value.
• Gathering the entire company together regularly (and smaller groups in local offices/meetups) is much more important than it used to be for creating a company-wide culture and helping people build relationships with others in different teams and functional areas.

Given this new normal, I’ve been wondering where this takes us for the next generation of startup companies. It matters to me that people have a shared sense of the company’s vision and feel bound to each other on a company mission. Without this, joining a startup loses a big element of its appeal and it becomes harder to do the challenging, creative, exhausting and sometimes nutty things it takes to launch and scale. There are only so many hours anyone can spend on Zoom before fatigue sets in. And it’s harder to have the casual and serendipitous exchanges that used to generate new ideas and energize long-running brainstorming discussions.

Know where you want to go before you start

Building culture in the current startup world requires intention. Here are some things I think are critical to doing this well. I would love to hear about things that are working for other entrepreneurs!

1. Founders: spend more time sharing your vision on team calls and 1:1 with new hires – this is the “glue” that holds the company together.
2. Managers: schedule more frequent open-ended, 1:1 calls to chat about what’s on people’s minds and hear ideas on any topic. Leave open blocks of time on your weekly calendar so people can “drop by” for a “visit.”
3. Encourage local meetups as often as practical – make it easy for local teams to get together where and when they want.
4. Invest in your all-team meetups, and make these as fun and engaging as possible. (We’ve tried packed agendas with all-day presentations and realized that this was too much scheduling). Leave time for casual hangouts and open discussions while people are working or catching up on email/Slack.
5. Do even more sharing of information about the company updates and priorities – there’s no way for people to hear these informally, so more communications are needed and repetition is good 🙂
6. Encourage newer/younger employees to share their work and ideas with the rest of the team – it’s too easy for them to lack feedback or mentoring and to lose engagement.
7. Consider what you will do in an urgent situation that requires team coordination: simulations and reviews of processes are much more important than in the past.

There’s no silver bullet to building great company culture, but instead a wide range of approaches that need to be tried and tested iteratively. These approaches also change as the company grows – building cross-functional knowledge and creativity requires all the above but even more leadership by the founders and management team (and a commitment to traveling regularly between locations to share knowledge). Recruiting, already such a critical element of building culture, now has an added dimension: will the person being hired succeed in this particular culture without many of the supporting structures they used to have? Will they thrive and help build bridges between roles and teams?

It’s easy to lose sight of the overall picture and trends amidst the day-to-day urgency, so it’s important to take a moment when you’re starting the company to actually write down what you want your company culture to be. Then check it as you grow and make updates as you see what’s working and where there are gaps. The founding team still sets the direction, but today more explicit and creative efforts are needed to stay on track and create a cultural “mesh” that scales.

Related reading

• Don’t forget these 3 things when starting a cloud venture
• Why do this startup thing all over again? Our reasons for creating Causely
• Are you ready to eat your own dogfood?

In a modern production microservices environment, the number of alerts from observability tooling can quickly amount to hundreds or even thousands, and it’s extremely difficult to understand how all these alerts relate to each other and to the actual root cause. At Causely, we believe these overwhelming alerts should be consumed by software, and root cause analysis should be conducted at machine speed.

Our Causal Reasoning Platform automatically associates active alerts with their root cause, drives remedial actions, and enables review of historical problems as well. This information streamlines post-mortem analysis, frees DevOps time from complex, manual processes, and helps IT teams plan for upcoming changes that will impact their environment.

Causely installs in minutes and is SOC 2 compliant. Share your troubleshooting stories below or request a live demo – we’d love to see how Causely can help!

Tune in to hear Host Eric Kavanagh interview Ellen Rubin of Causely, as they explore how this fascinating new technology works.

By the end, you’ll have a clearer understanding of what a robust RCA solution powered by causal AI can offer and how it can empower your IT team to better navigate the complexities of your cloud-native environment and most importantly dramatically reduce MTTx.

The Rise (and Fall) of the Automated Root Cause Analysis Holy Grail

Modern organizations are tethered to technology. IT systems, once monolithic and predictable, have fractured into a dynamic web of cloud-native applications. This shift towards agility and scalability has come at a cost: unprecedented complexity.

Troubleshooting these intricate ecosystems is a constant struggle for DevOps teams. Pinpointing the root cause of performance issues and malfunctions can feel like navigating a labyrinth – a seemingly endless path of interconnected components, each with the potential to be the culprit.

For years, automating Root Cause Analysis (RCA) has been the elusive “Holy Grail” for service assurance, as the business consequences of poorly performing systems are undeniable, especially as organizations become increasingly reliant on digital platforms.

Despite its importance, commercially available solutions for automated RCA remain scarce. While some hyperscalers and large enterprises have the resources and capital to attempt to develop in-house solutions to address the challenge (like Capital One’s example), these capabilities are out of reach for most organizations.

Beyond Service Status: Unraveling the Cause-and-Effect Relations in Cloud Native Applications

Highly distributed systems, regardless of technology, are vulnerable to failures that cascade and impact interconnected components. Cloud-native environments, due to their complex web of dependencies, are especially prone to this domino effect. Imagine a single malfunction in a microservice, triggering chain reaction, disrupting related microservices. Similarly, a database issue can ripple outwards, affecting its clients and in turn everything that relies on them.

The same applies to infrastructure services like Kubernetes, Kafka, and RabbitMQ. Problems in these platforms might not always be immediately obvious because of the symptoms they cause within their domain. Furthermore symptoms manifest themselves within applications they support. The problem can then propagate further to related applications, creating a situation where the root cause problem and the symptoms they cause are separated by several layers.

Although many observability tools offer maps and graphs to visualize infrastructure and application health, these can become overwhelming during service disruptions and outages. While a sea of red icons in a topology map might highlight one or more issues, they fail to illuminate cause-and-effect relationships. Users are then left to decipher the complex interplay of problems and symptoms to work out the root cause. This is even harder to decipher when multiple root causes are present that have overlapping symptoms.

In addition to topology based correlation, DevOps team may also have experience of other types of correlation including event deduplication, time based correlation and path based analysis all of which attempt to reduce the noise in observability data. Don’t loose sight of the fact that this is not root cause analysis, just correlation, and correlation does not equal causation. This subject is covered further in a previous article I published Unveiling The Causal Revolution in Observability.

The Holy Grail of troubleshooting lies in understanding causality. Moving beyond topology maps and graphs, we need solutions that represent causality depicting the complex chains of cause-and-effect relationships, with clear lines of responsibility. Precise root cause identification that clearly explains the relationship between root causes and the symptoms they cause, spanning the technology domains that support application service composition, empowers DevOps teams to:

• Accelerate Resolution: By pinpointing the exact source of the issue and the symptoms that are caused by this, responsible teams are notified instantly and can prioritize fixes based on a clear understanding of the magnitude of the problem. This laser focus translates to faster resolution times.
• Minimize Triage: Teams managing impacted services are spared the burden of extensive troubleshooting. They can receive immediate notification of the issue’s origin, impact, and ownership, eliminating unnecessary investigation and streamlining recovery.
• Enhance Collaboration: With a clear understanding of complex chains of cause-and-effect relationships, teams can collaborate more effectively. The root cause owner can concentrate on fixing the issue, while impacted service teams can implement mitigating measures to minimize downstream effects.
• Automate Responses: Understanding cause and effect is also an enabler for automated workflows. This might include automatically notifying relevant teams through collaboration tools, notification systems and the service desk, as well as triggering remedial actions based on the identified problem.

Bringing This to Life with Real World Examples

The following examples will showcase the concept of causality relations, illustrating the precise relationships between root cause problems and the symptoms they trigger in interrelated components that make up application services.

This knowledge is crucial for several reasons. First, it allows for targeted notifications. By understanding the cause-and-effect sequences, the right teams can be swiftly alerted when issues arise, enabling faster resolution. Second, service owners impacted by problems can pinpoint the responsible parties. This clarity empowers them to take mitigating actions within their own services whenever possible and not waste time troubleshooting issues that fall outside of their area of responsibility.

Infra Problem Impacting Multiple Services

In this example, a CPU congestion in a Kubernetes Pod is the root cause and this causes symptoms – high latency – in application services that it is hosting. In turn, this results in high latency on other applications services. In this situation the causal relationships are clearly explained.

A Microservice Hiccup Leads to Consumer Lag

Imagine you’re relying on a real-time data feed, but the information you see is outdated. In this scenario, a bug within a microservice (the data producer) disrupts its ability to send updates. This creates a backlog of events, causing downstream consumers (the services that use the data) to fall behind. As a result, users/customers end up seeing stale data, impacting the overall user experience and potentially leading to inaccurate decisions. Very often the first time DevOps find out about these types of issues is when end users and customers complain about the service experience.

Database Problems

In this example the clients of a database are experiencing performance issues because one of the clients is issuing queries that are particularly resource-intensive. Symptoms of this include:

• Slow query response times: Other queries submitted to the database take a significantly longer time to execute.
• Increased wait times for resources: Applications using the database experience high error rate as they wait for resources like CPU or disk access that are being heavily utilized by the resource-intensive queries.
• Database connection timeouts: If the database becomes overloaded due to the resource-intensive queries, applications might experience timeouts when trying to connect.

Summing Up

Cloud-native systems bring agility and scalability, but troubleshooting can be a nightmare. Here’s what you need to conquer Root Cause Analysis (RCA) in this complex world:

• Automated Analysis: Move beyond time-consuming manual RCA. Effective solutions automate data collection and analysis to pinpoint cause-and-effect relationships swiftly.
• Causal Reasoning: Don’t settle for mere correlations. True RCA tools understand causal chains, clearly and accurately explaining “why” things happen and the impact that they have.
• Dynamic Learning: Cloud-native environments are living ecosystems. RCA solutions must continuously learn and adapt to maintain accuracy as the landscape changes.
• Abstraction: Cut through the complexity. Effective RCA tools provide a clear view, hiding unnecessary details and highlighting crucial troubleshooting information.
• Time Travel:Post incident analysis requires clear explanations. Go back in time to understand “why” problems and understand the impact they had.
• Hypothesis: Understand the impact that degradation or failures in application services and infrastructure will have before they happen.

These capabilities unlock significant benefits:

• Faster Mean Time to Resolution (MTTR): Get back to business quickly.
• More Efficient Use Of Resources: Eliminate wasted time chasing the symptoms of problems and get to the root cause immediately.
• Free Up Expert Resources From Troubleshooting: Empower less specialized teams to take ownership of the work.
• Improved Collaboration: Foster teamwork because everyone understands the cause-and-effect chain.
• Reduced Costs & Disruptions: Save money and minimize business interruptions.
• Enhanced Innovation & Employee Satisfaction: Free up resources for innovation and create a smoother work environment.
• Improved Resilience: Take action now to prevent problems that could impact application performance and availability in the future

If you would like to get to avoid the glitter of “Fools Gold” and get to the Holy Grail of service assurance with automated Root Cause Analysis don’t hesitate to reach out to me directly, or contact the team at Causely today to discuss your challenges and discover how they can help you.