When teams get ready to add agents to their on-call workflows, the hard part is deciding when to let those agents act on production versus just advise. A root cause claim only earns that trust if you can verify what the agent ruled out. Causely's causal model now exposes that reasoning directly to agents: which alternative diagnoses were considered for a symptom, what evidence ruled out those alternatives, and how far the failure would have spread if left unaddressed.

Acting vs. Advising

A recent r/kubernetes thread separated a remediation agent into two parts: an LLM proposes actions such as scaling, rolling back, cordoning, or draining, and a separate deterministic layer vetoes anything unsafe against the live cluster state before it executes. The design assumes a proposal and the decision to act on it are two different things.

The top reply doesn't argue with the architecture. It points to the actual gap: confidence in the cause and the safety to act are two different things, and they often get conflated. The suggested fix is to force the agent to explain why this diagnosis, specifically, before it hands over a proposed action. That way a human or a policy engine has something to review, not just a recommendation to accept or reject. Another engineer in the same thread pushes further: a veto needs memory. Blast-radius budgets per namespace, cooldowns, and a rule to halt and page a human if the expected effect doesn't show up. And the veto must check real cluster state itself, not trust what the agent reports about that state.

Platform teams everywhere are running into the same challenge: when is an agent's diagnostic capability good enough to let it act in production?

Why isn't "the agent is confident" the same as "the agent is right"?

An agent proposing an action and an agent explaining its diagnosis are two different things. Only one of them is checkable before anyone touches production. A confidence score is an assertion. A list of possible explanations for the symptom, with the causal chain behind each, is something a reviewer — human or automated — can inspect.

That's the same problem the thread's second engineer described: a veto that trusts the agent's self-report is a second opinion, not a check. A veto that can see what the model considered and rejected, and how large the blast radius of the winning diagnosis is, has something real to validate with.

What does it mean to ask an agent "why this and not that"?

When it comes to understanding cause and effect, most people familiar with the domain don't usually ask, "why did X happen?" They tend to ask "why X, rather than Y," a finding from explainable-AI research on contrastive explanation: an explanation is only satisfying relative to some alternative that didn't occur, even when that alternative is never stated out loud. A 2025 formalization of the problem frames it directly as "why P but not Q" and treats computing the difference between the two as the actual explanation task, not as an afterthought.

Causely's MCP server now exposes two tools designed specifically for this question. get_potential_diagnoses tells you what could explain a given symptom, not just the one the causal model selected, and shows the causal chain behind each. get_signal_potential_diagnoses runs the same question in the other direction: given an observed signal, what could explain it, and what's the causal chain connecting each candidate to that signal? Both draw on the same causal model that produces the primary diagnosis: the same reasoning, made visible instead of collapsed into a single answer.

Here is an example

Take a familiar incident: elevated latency and timeouts on a checkout service, several layers downstream from a shared database connection pool. An LLM-only agent traces this from the symptom alone. It queries telemetry, traces back from the checkout service, and finds a CPU spike on an adjacent service along the way. It diagnoses the CPU spike as the cause and restarts that service. The dashboard improves. The agent treats the incident as resolved.

The real cause, connection exhaustion on the shared database connection pool upstream, provides no direct signal at that layer, so the agent never sees it. The restart bought time by clearing the CPU spike, not by fixing the underlying problem. The incident resurfaces when the connection limit is reached again, and the agent repeats this loop.

Calling get_signal_potential_diagnoses on the original symptom surfaces the connection-exhaustion diagnosis alongside the CPU-spike alternative, each with the causal chain showing what it explains and what it fails to account for. The CPU spike doesn't explain the downstream timeout pattern; the connection exhaustion does. That comparison is what makes the diagnosis reviewable before a remediation agent acts on it, not just the final answer.

How far would this failure have spread if it wasn't caught?

A diagnosis with a small, well-understood blast radius and a diagnosis with a large, uncertain one are different risk decisions, even if a model is equally confident in both. rank_entities and get_diagnosis_observable_signals surface which entities the failure would affect and which signals to expect downstream before any action is taken.

This is the concrete version of the veto idea from the above Kubernetes thread: a blast-radius budget is only useful if something can compute the blast radius first. Ranking entities by dependency exposure is what gives a policy engine, or a human running the on-call rotation, a number to set that budget against instead of a guess.

Where can causal explanations go further?

Today's tools answer "why this and not that" and "how far would it spread" against the current state of the causal model. The next layer is forward simulation: "what would break if this component degraded," asked before any symptom appears, not after. That requires modeling failure modes as first-class entities with their own preconditions and blast radius, rather than only reasoning from observed anomalies backward. It's a natural extension of the same causal model: the same questions, asked earlier instead of after the fact.

FAQ

What is a contrastive explanation in root cause analysis? Contrastive explanation answers "why this cause, not that one" instead of just "why did this happen." Research on explainable AI has found that people evaluate explanations relative to an alternative that didn't occur, even if that alternative is unstated. In root cause analysis, this means showing which diagnoses were considered and ruled out, not only the one that was selected.

How is this different from a deterministic safety veto layer? A safety veto checks whether a proposed action is safe to execute against live system state. Contrastive diagnosis explanation checks whether the reasoning behind the diagnosis holds up, and which alternatives were ruled out and on what evidence. The two are complementary: a veto layer is more useful with a diagnosis it can inspect, rather than a bare confidence score.

How do I use the new MCP tools to check an agent's diagnosis before it acts? Call get_potential_diagnoses or get_signal_potential_diagnoses against the symptom an agent is reasoning to see what could explain it and the causal chain behind each candidate. Pair this with rank_entities or get_diagnosis_observable_signals to see the blast radius of the leading diagnosis before approving any remediation action.

Doesn't surfacing alternative diagnoses just mean the model is less confident? No. Showing what could explain a symptom and the causal chain behind each candidate makes a diagnosis checkable, not uncertain. A model that only reports its top answer is asking to be trusted. A model that shows what else it considered and why each was ruled out is being reviewed. The second is the stronger claim.

What to do next

• Read How Causal Reasoning Addresses the Limitations of LLMs in Observability for the causal model this builds on.
• Read Beyond the Blast Radius for more on dependency-aware impact analysis.
• Learn more about Causely MCP tools

Operating Agents Is Becoming a Layered Discipline 

You shipped an agent to production. When its output quality slips, or latency climbs, or it starts failing tasks it used to complete, the first question is always the same: why. Answering it has become its own discipline, and teams running agents at scale are assembling a proper operational stack to do it, the same way teams running microservices did a decade ago. 

Here is how that stack is taking shape, and where a causal reasoning system fits into it. 

Layer One: Instrumentation 

Everything starts with capturing what the agent actually did. OpenTelemetry's GenAI semantic conventions give you a common vocabulary for LLM calls, tool calls, token usage, and, increasingly, whole agent and MCP call trees as first-class spans. The conventions are still under active development as of mid-2026, so attribute names can shift and you should budget for the occasional rename. But they have consolidated enough to standardize on, and real backends already emit and ingest them. Instrumenting against them gives you a structured, correlated record of every interaction, the foundation the other layers build on. 

Layer Two: Evaluation and Agent Observability 

Once you have the signals, you need to know whether the agent is doing a good job. This is where the eval and LLM observability platforms live: Comet, Langfuse, Braintrust, Galileo (now part of Cisco), Arize (now part of Dynatrace), and others. They each have a different emphasis. Some lean toward the offline eval and experimentation loop, others toward production monitoring and tracing, but they share a purpose: scoring outputs, catching regressions, and tracking agent behavior against datasets and guardrails.

Output quality is only part of the story. This layer is also about understanding the agent's operational health: latency and error rates, along with agent-specific signals like repeated or redundant tool calls, cache efficiency, and token consumption, much of which falls out of the same OTel instrumentation from Layer One.

If you are operating agents in production, you should be using one of these, or a comparable solution. They answer two important questions: what did the agent do, and is it doing its job well? Every team needs to be able to answer those questions. 

Layer Three: Reasoning Across the Stack 

The first two layers answer what and whether. The third answers why, and it is the hardest of the three.

An agent in production sits on top of everything else you run. The LLM call is one hop in a chain that includes the services assembling context, the vector stores and databases serving retrieval, the APIs exposed as tools, the queues moving work between steps, and the compute, network, and storage underneath all of it. When output quality drops or latency climbs, the origin is often somewhere in that chain rather than in the agent itself:

• A tool starts returning empty results because a backing service degraded, and the agent compensates by retrying or looping around the gap.
• Retrieval quality slips because an embedding pipeline stalled and the index went stale, so answers get quietly worse while traces look normal.
• p95 latency doubles because a database serving context assembly is under memory pressure, not because the model slowed down.
• A routine deployment to a shared dependency changes a default timeout, and the agent starts hitting intermittent tool failures it never saw before, with nothing in the agent itself having changed.

Connecting a change in agent behavior to its origin somewhere below the agent is a distinct problem, and it is genuinely hard, because it means reasoning across the causal relationships between the agent and every layer of the stack it depends on. This is not a new goal. Cross-layer causal analysis has been the direction of travel in observability for years. What is still open is how you actually do it in software, rather than leaning on the tribal knowledge of a few senior engineers and an ever-growing pile of runbooks.

What Is the Best Approach for Managing AI Agents in Production? 

The industry consensus is clear, and the recent moves make it clearer. Cisco acquiring Galileo and Dynatrace acquiring Arize, at a reported $915M, are both bets on bringing agent telemetry and infrastructure observability under one roof. Datadog has been building toward the same goal organically. The Dynatrace announcement said the premise plainly: when an AI application's output quality slips, the cause can sit anywhere from the prompt to the model to the infrastructure underneath it. We agree that reasoning across the layers of the stack is the right goal, and it is the premise we started Causely on years ago.

Where approaches diverge is on how you go from observable symptoms to the specific issue that needs to be addressed. The dominant path has been correlation and machine learning over telemetry, and the recent acquisitions extend it by putting more of that telemetry, agent and infrastructure, in the same place. That is a step forward, and co-located data makes correlation easier. But it also has a ceiling: correlation tells you an eval score dropped around the time a database schema changed, which sometimes points at the real issue and sometimes at a coincidence.

In an environment with many components changing constantly, separating signal from noise takes a model of how system behaviors propagate across layers: which services feed context into which prompts, which tools depend on which backends, how a degradation in one place surfaces as a symptom somewhere else.

That model-based approach is the bet Causely makes. Instead of correlating signals after the fact, the system maintains a live causal model of the environment and reasons over it deterministically. Because the reasoning follows the model rather than pattern-matching over history, it does not surface hallucinated root causes, and it points at the specific issue to fix rather than a ranked list of correlated suspects. Across production deployments, customers have seen 63% faster incident resolution and 57% lower operating cost, with zero hallucinated root causes. 

How Does Causely Pinpoint the Cause of Agent Degradations?

We made a decision from day one to build a system that ingests data from any combination of sources, whether that includes OpenTelemetry, your existing observability platforms, or GenAI instrumentation, and produces a semantically consistent representation of your environment. From it, our system instantiates thousands to tens of thousands of causal models that encode how an issue in one part of your stack causes symptoms in others. All of this causal knowledge is continuously maintained as the environment changes. 

There are two primary patterns for how customers are putting this knowledge to use, which help illustrate how it fits alongside the rest of your stack: 

1. On Demand. An alert fires, Causely answers why. Route existing alerts into Causely, or let your agent query Causely's MCP server as part of an investigation. One call returns signal vs. noise, causal analysis, blast radius, and supporting evidence. 
2. Proactive. Causely surfaces emerging system degradation with causal analysis attached and triggers a background agent to act. The same payload can be sent to any destination you choose before an SLO is violated or an incident is declared. 

One customer described the proactive pattern in concrete terms. Causely detected emerging degradations and pushed the causal diagnosis to their background agents, which resulted in twelve agent-initiated pull requests that the team merged after human review. By their account, each change saved their five-person team two to three hours over the month, and that is before counting the incidents they prevented. They framed this benefit as the equivalent of adding a full-time engineer dedicated to keeping performance healthy and preventing incidents.

The eval layer keeps doing what it does best: output quality, regression testing, guardrails, the development loop. Causely sits underneath and across it, connecting what the agent did to why the surrounding system shaped that behavior. If you are building agents and automated workflows on top of this data, we have a lot to talk about. 

The Whole Stack, Working Together 

No single layer is the whole answer, and none of them replaces the others. Instrumentation tells you what the agent did. Evaluation tells you whether the output was good. Causal reasoning tells you why the behavior changed, across the services and infrastructure the agent runs on. The teams operating agents with the most confidence are the ones running all three.

If you want to see for yourself, try it free. Or reach me directly via email (yyemini@causely.ai) or on LinkedIn.

FAQ 

Does Causely replace my eval platform? No. Eval platforms own output quality: scoring, datasets, regression testing, guardrails. Causely pinpoints the cause of a degradation by reasoning across the agent layer together with the various services and infrastructure components the agent depends on. They are complementary. 

I use Arize or Galileo, what do the acquisitions mean for me? Those products now live inside larger observability portfolios. What the acquisitions signal is that agent and infrastructure telemetry are converging. Causely is interoperable with all of these products and will stay that way. We already consume data from Dynatrace, Datadog, OpenTelemetry, and others, and reason across all of it. 

How does Causely know about my agent workflows? Causely builds a topology of your environment from the data sources you already have, including OTel GenAI instrumentation where present. Your agent workflows appear in the causal model alongside the services, data stores, and infrastructure they depend on, so an issue anywhere in the stack can be mapped directly back to the workflows it affects. 

What does "proactive" mean here concretely? The system does not wait for you to query it. The causal model is evaluated continuously as telemetry flows in, so emerging degradations are identified and attributed to their cause as they develop. Causely can trigger a background agent, send a message, or initiate a remediation workflow, often before the issue rises to the level of an incident. 

Causely is a member of the AAIF, and we want to share why we think this news matters for anyone building agents for autonomous service reliability.  

The Protocol Layer Is Maturing 

If you think back to the early internet, almost nothing interesting got built until TCP/IP, DNS, and HTTP existed as neutral protocols. The same evolution is happening in the agent ecosystem, just compressed into a much shorter period. 

MCP quickly became the standard for how agents connect to tools and context. In parallel, A2A became the standard for how agents talk to each other about capability negotiation, task handoff, and replanning across agent boundaries. But while MCP became an AAIF project as part of the foundation being formed in December 2025, A2A stayed at the Linux Foundation level. 

With this announcement today, the two core protocols of the agent stack share one governance home and community. Shared neutral governance is what turns a promising spec into infrastructure people build on for a decade or more.  

Why This Matters for Autonomous Service Reliability 

Most teams we talk to are building their own agents today, and A2A makes it more practical to wire multi-agent systems together. With a standard protocol for capability negotiation, task handoff, and replanning, teams can compose systems from agents built by other teams, other vendors, or open source projects. We expect multi-agent architectures to become more common, and to increasingly include agents whose internals you never see. 

Each handoff is a message crossing an agent boundary, which is also where the risk compounds. If one agent is guessing, its errors stay local. But if that agent hands its conclusions to another agent over A2A, the guess travels. Like a childhood game of telephone, one agent’s guess gets treated as another agent’s fact. Audit trails help you reconstruct what went wrong, but only after the fact. At machine speed, the only real-time lever you have is grounding each agent with the right context at the right moment.  

This is a core problem Causely exists to solve. Our causal reasoning system gives your agents deterministic context about cause-and-effect across the entities that make up your cloud-native or GenAI application. When something degrades in a part of your stack, your agents don’t need to waste time building context from raw telemetry.  

With Causely, agents get the prioritized root cause, together with the diagnostic evidence that matters. Causely is also continuously monitoring SLOs and symptom activation to automatically detect emerging problems. That means it can trigger a background agent proactively with the specific issue and related context, rather than waiting for a human or scheduled run to initiate. 

If you are building SRE and ops agents or trying to achieve autonomous service reliability, you need both standardized transport between agents (A2A under the AAIF) and trustworthy context that grounds them (Causely). 

Why Causely Joined the AAIF 

We joined AAIF because our customers are building here. Causely ships as a containerized system with an MCP server. The teams we work with are wiring agents into their observability stack and related internal tooling, and they're starting to connect agents to each other. The standards governing that plumbing will shape what's possible in AI-driven operations for years to come. 

We're looking forward to helping shape those standards, especially interoperability ones that matter for operational context: how agents share context, how they keep each other grounded, and how they create feedback loops to check their work. 

What Comes Next 

Protocols are necessary but not sufficient. The next phase of this ecosystem is about the substance flowing over these wires. For our customers this means causal context that is deterministic and evidence-backed, so that ops agents they build can be trusted to act autonomously and at machine speed. 

Congratulations to the A2A maintainers and the AAIF governing board on today’s news. If you're building SRE or ops agents on MCP or A2A and thinking about the inherent trust and speed problems, we'd love to compare notes.

Open-Inspect gives a background agent somewhere to run: a full development environment, triggered by an event, with no human at the keyboard. Causely gives that agent a causal diagnosis instead of a pile of telemetry to interpret. The Cloud & Platform Operations team at Fountain wired both together for proactive production ops, and the pattern is now documented in a runnable example.

On June 12, 2026, we ran a live session with Cole Murray, creator of Open-Inspect, and Martin Roberts, Cloud & Platform Operations lead at Fountain. The topic was proactive remediation: agents that fix issues before anyone notices, not agents that wait for a page. Watch the recording for Martin's walkthrough of the production setup. This post provides more detail on the mechanics behind this setup and includes a link to a working example.

What does Open-Inspect run?

Open-Inspect is an open-source background coding agent, built by Cole Murray and modeled on Ramp's internal Inspect tool. It gives an agent a full development environment (Node.js, Python, git, browser automation, VS Code) and lets that agent pick up work from a webhook, a cron schedule, a Slack message, or a monitoring alert, then run unattended until the task is done.

Two important aspects of Open-Inspect are relevant to ops use cases. First, sessions persist: an agent can run for minutes or hours and be re-triggered later on the same thread, which fits an investigation that doesn't resolve in one shot. Second, the security model is single-tenant by design. Every user in a deployment shares the same GitHub App installation and repository access. That works for a trusted internal team where everyone already has access to the same repos. Everyone sharing that deployment shares the same GitHub App access. Organizations with multiple teams that need separate boundaries can run separate deployments, one per team, rather than pooling everyone into one.

Open-Inspect decides where an agent runs and what it's allowed to touch. It has no opinion on why and when to act. This is the context it needs to receive from another source.

What does Causely add to a background agent?

An agent with a working development environment and repository access still must be told what to act on. It's similar to the point we made in our blog about Claude Managed Agents: an agent that can reach every system in your environment still isn't one you'd trust to act on a guess.

The Causely MCP server gives your agent a specific, mechanistic diagnosis, such as Postgres idle-in-transaction accumulation or Redis cache miss storm, that is distinct from the symptoms it produces. A peer-reviewed study we published showed that agents working from Causely's causal context reached a diagnosis 2-3x faster than agents reasoning over raw telemetry.

How does Causely make agents faster?

The reason agents leveraging Causely diagnose problems faster is that they never need to rebuild the environment's dependency structure at query time. Not only dependencies between entities in the environment, but also the symptoms that may be observed and the potential causes that could produce those symptoms given the topology at that moment in time. Read the full benchmark study for the methodology here.

How does the integration work?

Open-Inspect answers where and how an agent can act. Causely answers when and what it should act on. An agent with both can efficiently move from an issue to a verified fix without a human bridging the gap.

Causely runs continuously over the environment, ingesting telemetry locally and maintaining a semantic understanding of the system. When it identifies an issue, a webhook fires carrying both the issue and the diagnosis. That notification routes to the specific background agent with repo access for that environment. The agent calls get_issue_details to retrieve the causal context behind it and other Causely MCP tools, like get_logs, if it needs more evidence. Then it opens a PR.

The agent never starts from a raw alert. The webhook already carries a diagnosis, not a symptom. get_issue_details and get_logs add depth to that diagnosis. They don't build it from scratch. The output is a PR, and a human reviews the fix to verify the change before it ships.

Fountain runs this exact sequence: webhook, get_issue_details, get_logs if needed, PR. An engineer reviews and merges from there. Martin described fixing issues before the traditional alert would have fired, let alone before any end-user escalation:

"An agent works right away. It receives the request and, within seconds, identifies the root cause with help from the Causely MCP tool. Within minutes it has the pull request ready. That is much, much faster than how any ops team can function while working on the problem".

— Martin Roberts, Cloud & Platform Operations lead at Fountain

What to do next

• Watch the full webinar for Martin's production walkthrough and the live Q&A.
• Read the causely-oss example and adapt the trigger pattern to your own alerts.
• If you're building a custom agent instead, start with Causely MCP Skills for the tool-selection logic Open-Inspect's example already uses.

FAQ

What's the difference between Open-Inspect and Causely? Open-Inspect is a background agent execution system: it decides where an agent runs and what it's allowed to touch. Causely is a causal reasoning system: it determines what issues need to be acted on and when. Open-Inspect provides the environment; Causely provides the diagnosis. Neither replaces the other.

Can a background agent remediate an issue without a causal model? It can work for specific events and runbooks that you have defined in advance, and it can work for localized, code-level problems. But in environments with many dependencies and complex topology, a background agent seeking to remediate an issue without a causal model is just as likely to make the situation worse as it is to fix the root cause. A causal model ensures the agent acts on the cause of the observed anomalies, rather than on the symptoms themselves.

Is Open-Inspect safe to deploy across multiple teams? Open-Inspect is single-tenant by design: everyone in a deployment shares the same GitHub App installation and repository scope. It's built for a trusted internal team with shared repository access, not for isolating multiple organizations or teams with different permissions within a single deployment.

How do I trigger a background agent from a production alert? Open-Inspect sessions can be triggered by webhooks, so a monitoring alert can start a session directly. The session then queries Causely for a diagnosis before taking any action, rather than starting from the raw alert payload. The causely-oss example documents this trigger pattern end-to-end.

Background agents run independently and can be triggered by events, making them a good fit for on-call work. Claude Managed Agents are one example, and MCP tunnels give them access to the environment they're making decisions over, reaching it, and authenticating. But access is just the starting point. An agent that can reach and log into every system still isn't automatically one you'd trust to run on-call. 

Why do background agents fit on-call work? 

Background agents differ from interactive assistants in one key way: they run on their own schedule, triggered by an event (e.g., a page, an alert) rather than by someone typing a prompt. This difference is what makes them an ideal fit for on-call ops work, where nobody has to be sitting at a keyboard when performance degrades. 

Background agents persist. They pick up an event, run for as long as the investigation takes, from seconds to hours, and can be triggered again later on the same thread. Cole Murray, the creator of Open-Inspect, walks through this distinction in detail, and it's worth watching: the architecture decisions that make sense for a chat assistant don't automatically transfer to something that runs unattended. 

Claude Managed Agents provide a similar instantiation: their sessions persist in the cloud, execute over time, and can be triggered by automated events.  Anthropic's own walkthrough of managed agents covers the mechanics and shows how to build an SRE incident response agent.

For on-call work, this approach is helpful in two ways. For reactive workflows, it ensures that whoever prompts the agent has the same setup and that the agent has the necessary access to assist with the investigation. More importantly, for proactive workflows, the agent can be triggered by events and start and end the investigation before an on-call engineer gets out of bed.  

Recently, at Code with Claude in London on May 19, 2026, Anthropic shipped two features that improve the performance of these agents when running them against production systems. Self-hosted sandboxes (public beta) and MCP tunnels (research preview). Together, they let a cloud-run agent reach a private Kubernetes cluster or an internal Prometheus instance without opening an inbound port, closing a gap that had been blocking some on-call use cases on security review alone. These welcomed improvements make this approach much more practical. It also surfaces the next key question: once the agent has access, does it have the context it needs to resolve the problem? 

Reaching your infrastructure is (mostly) a solved problem 

A managed agent runs in Anthropic's cloud; your cluster, your metrics, and your logs run in your environment. The agent never touches them directly. It calls MCP servers that do. Your cluster exposes a Kubernetes MCP server, your metrics stack exposes a Prometheus MCP server, and the agent calls both as tools. It never gets a raw kubeconfig or a database connection string.  

The first wall anyone hits is that a cloud agent can't reach localhost or a private API server. For local development or a kind cluster, a tunnel like cloudflared's quick-tunnel mode gets you unblocked in minutes. This works for a proof of concept, but not for running production traffic. The production answer is native MCP tunnels. A lightweight gateway inside your network opens a single outbound connection; there's no inbound port and no public MCP endpoint for a scanner to find. This is a general property of cloud-run agents, not something specific to Claude. Any agent platform running outside your perimeter has to solve the same reachability problem, and outbound-only tunneling is the shape the industry, including AWS and Google, has converged on this year. 

How do you authenticate an agent to systems it can now reach? 

The Managed Agents MCP connector sends one thing to your server: an Authorization: Bearer header. Credentials live in a vault matched by the MCP server's URL, not configured on the server entry itself. This means the hard part of auth isn't the agent side, it's getting your existing auth approach to work with that model. 

Three cases show up in practice, in roughly escalating difficulty: 

No auth. A Kubernetes MCP server behind a tunnel with cluster-level RBAC as the only gate. Fine for a kind cluster or a low-stakes internal tool; not something to run against a production API server without at least a bearer token in front of it. 

Static bearer token. A Grafana service-account token, stored in a vault and matched to the MCP server's URL. This is the common case for read-only observability queries. Prometheus, Loki, and Grafana all support long-lived service-account tokens, and the vault handles rotation. 

OAuth client-credentials → JWT. The hard case, since most internal APIs weren't built expecting a bearer-only connector. It sends Authorization: Bearer and nothing else, so the vault has to complete the OAuth exchange up front and hand back a plain bearer token. Sending the client secret as a separate field instead produces a 400 unknown field error, since the connector's schema only expects the one header. For a straightforward approach to handling this see the repo at the end of this post.  

The generalizable rule holds regardless of which of the three cases you're in: the connector's interface is Authorization: Bearer. Anything more complex belongs in the vault, not in the agent's configuration. 

Once that's wired up and authenticated, the reachability problem is addressed, and the real work can begin.  

Access isn't understanding 

An agent with full read access to Kubernetes, Prometheus, and your log pipeline still reasons poorly about the system it is investigating. Even the richest telemetry does not come with a model of what depends on what, or more specifically, what can cause what. Metric series, multiple alerts firing, log lines, and even traces provide the agent with rich data to investigate and reason with, but the access to this data is not structured in a way that makes it effective. 

Give an agent kubectl, a PromQL endpoint, and a log query tool, and it can retrieve almost anything but retrieval isn't diagnosis. This is not a knock on the underlying telemetry. OpenTelemetry's semantic conventions exist precisely to make spans, metrics, and logs consistent enough to reason over but consistent data isn't the same as a causal model of the system that produced it.  

We've written before about why LLMs alone hallucinate root causes for the same underlying reason: pattern-matching over telemetry isn't the same as reasoning over a dependency graph. The distinction between monitoring, knowing something is wrong, and observability, knowing why, makes a related point from a different angle. An agent that can query telemetry has solved the first problem. Solving the second requires encoding dependency and causality structure somewhere: in a semantic layer, a service graph, or an explicit causal model, so the agent has something to reason against, not just something to query. 

FAQ 

How do I connect a cloud AI agent to infrastructure it can't reach directly? Deploy an MCP server for the system you want the agent to reach, then bridge it with an outbound-only tunnel. For local development or a kind cluster, a quick tunnel like cloudflared works in minutes. For production, use native MCP tunnels: a gateway inside your network opens a single outbound connection, so no inbound port or public MCP endpoint is ever exposed. 

Does giving an agent more system access make it more reliable? Not by itself. Raw access to Kubernetes, Prometheus, and logs lets an agent retrieve data, but that doesn't mean it can diagnose. Without an encoded model of service dependencies and event ordering, an agent can't reliably tell a root cause from a downstream symptom. It can query more, not reason better. Access solves reachability; understanding requires a separate structural layer. 

What's the difference between an agent reaching a system and an agent understanding it? Reaching a system means the agent can call an MCP server and retrieve metrics, logs, or cluster state. Understanding means the agent can tell which of those signals is the cause and which are downstream effects, something that requires a dependency graph or causal model, not just a wider set of tools to query. 

How do I authenticate a managed agent to an internal API that uses OAuth? Resolve the OAuth client-credentials flow inside your credential vault, not in the agent's configuration. The Managed Agents MCP connector only ever sends an Authorization: Bearer header matched to the server's URL. It won't perform a token exchange or accept additional OAuth fields, so the vault has to mint the JWT and return a plain bearer token. See the reference repo below for code examples.  

What to do next 

Read the three-auth-pattern, kind-runnable reference repo to see all three cases running end to end with a Claude managed SRE agent accessing a real cluster. The open question this setup raises is whether adding a causal or semantic layer on top of raw telemetry access measurably changes how well an agent investigates an incident. We're running a reproducible, measured comparison on this next. Stay tuned.  

I’ve yet to meet an engineer who enjoys being on call. Some see it as a rite of passage or, at best, a learning and growth experience. However, much more has been written about why being on call sucks. I won’t rehash that here. I want to go deeper on one key point Charity Majors makes in her canonical piece on this subject.

“if you get paged about flappy alerts or the same problems week after week; of course you’ll hate it.”

When something breaks, you get paged. You start digging. Then you get paged again. The explanation has changed. An alert you looked at ten minutes ago is back. You acknowledge it again. You re-explain it in the Slack thread. You rebuild the context in your head. And the cycle continues… This is not just a tooling or thresholding problem. And it is not going away by putting AI agents on call.

In a large, dynamic environment like a microservice architecture, a single problem manifests through multiple symptoms that propagate across related entities. Not every symptom caused by the problem is present at any given moment. The environment's dynamic nature makes these symptoms flaky. They are present at one moment, gone the next. Common observability systems trigger on these symptoms via anomaly detection, but no anomaly-detection mechanism is immune to noise. Missing, spurious, and flaky symptoms are unavoidable.

This means a reasoning system can only draw a conclusion from what is present at that moment. At any point, the inferred cause is simply the explanation that best fits the current observations. And as those observations shift, the conclusion can shift with them. That's the real challenge. Knowing when the system can safely commit to a diagnosis and act on it.

It's tempting to think agents solve this by default. Agents don't get tired. They don't get annoyed. But the re-triage tax doesn't disappear when an agent takes over; it gets worse. When the symptoms or conclusions keep shifting underneath it, an agent starts over every time: it re-reads everything, reasons from scratch, and burns tokens doing it.

From cause to action

Causely already does the heavy lifting on call. It does not just process telemetry and flag anomalies; it understands how your system is connected and reasons about cause and effect across it. So instead of a hose of alerts, you get the cause. As one customer, a global bank running thousands of services, recently observed, dozens of alerts are explained by one Causely notification.

That reasoning is continuous. As observations change, the diagnosis updates to best explain them. The next challenge is deciding what action to take. In other words, confirming that the current diagnosis needs to be resolved, not just observed.

That confirmation is what a person needs to act with confidence, and what an agent needs to follow a problem through resolution. We call it an Issue.

What an Issue is

An Issue is the persistent thread that ties every diagnosis of the same underlying problem together into one thing to acknowledge, one thing to share with your team, one thing to resolve.

Here is how it works. An Issue causes anomalies, which trigger Signals: alerts, log messages, events… At any point during the life of the Issue, the system may infer a Diagnosis — the best explanation of the observations at that moment. As anomalies appear and clear, as affected services shift, as alerts fire and stop, the Signals change, and the Diagnosis updates to match.

The Issue is what holds it all together across all those changes. In each context, the odds that the same Signal is caused by two different Issues at once are very low. So, if two Diagnoses share even one Signal — present now or present at any point — they are treated as an expression of the same Issue. In other words, the Diagnosis can keep changing shape, but as long as Diagnoses keep sharing a Signal, it’s still one Issue. The Diagnosis evolves. The Issue doesn’t.

When you resolve an Issue, a leftover noise will not drag you back in. It only resurfaces if something genuinely related returns.

For on-call engineers and agents

For engineers on call: you get one stable thread for the persistent problem, start to finish. You stop re-triaging things you have already seen. And you can trust that when an Issue pings you, it is new, not the same problem wearing a different face.

For agents, this matters even more. An agent only knows what you hand it. A person can hold continuity in their head when the tooling falls short; an agent cannot. Hand it a stream of changing diagnoses, and it behaves like an inexperienced engineer who restarts the investigation every five minutes. It thrashes, repeats work, takes conflicting actions, and files two tickets for one problem.

Give an agent a stable Issue, and that stops. It has one thread to reason and act on. The diagnosis can keep evolving underneath without throwing it off, because the agent is working on the Issue, not chasing the latest hypothesis. It acts once. It does not start over. It does not burn tokens re-deriving what it already knew.

So, an agent needs a stable Issue more than a person does, not less.

One problem, one thread

The re-triage tax is what makes on-call miserable for people and wasteful for agents alike, and it isn’t solved by working faster through the same noise. It’s solved by giving whoever, or whatever, is responding to a single, stable thing to hold onto while the underlying diagnosis does its work underneath.

Issues are now live as early access. This is just the start. Issues are becoming the primary way Causely notifies you and the primary surface for agents working through MCP.

Check out Issues in your environment and tell us what you think.

TL;DR: Causely ingests the rich telemetry Dynatrace provides and uses it to construct an environment-specific causal model. When alerts fire, Causely pinpoints the cause that explains them, even when the originating entity itself never fired an alert. One diagnosed cause, instead of dozens of alerts, is what makes agents better, faster, and cheaper. Validated at a Fortune 50 enterprise across EKS, ECS, and VMware.

The gap that even the best instrumentation leaves

Dynatrace does a great job instrumenting hybrid environments: Kubernetes and serverless workloads, public cloud, and on-prem, all under one platform instead of five different tools stitched together. That breadth is exactly why, when a service fails or becomes congested, Dynatrace can see the whole blast pattern: dozens of alerts fire across multiple services when real problems arise. As a result, dozens of symptoms across connected services generate an alert storm and it becomes very difficult to differentiate between cause and effect. 

This is what we commonly see across production environments running hundreds to thousands of microservices. Rich telemetry from sources like Dynatrace tells you what is happening, but leave a gap in terms of understanding why it is happening (i.e., the diagnosis best matching the observed and unobserved symptoms). That gap typically gets closed by on-call engineers and incident channels operating under time pressure. Increasingly, teams are bringing AI agents into the loop (more on that topic here).

Tens of alerts, one triage tax

Dynatrace's tracing shows you the call path: a payment service calls a queue, which calls a downstream cache. What it doesn't understand is causation: which failure on that path produced the anomalies showing up everywhere else. A payment service timing out, a queue backing up, and a cache miss all fire as separate alerts because of their dependencies. The on-call engineer's job becomes manual correlation. Open ten tabs. Cross-reference deploy timelines. Guess at the chain. Page the wrong team at least once before landing on the cause.

That triage tax doesn't appear as a metric, but it's why MTTR climbs every time the service graph gets one layer deeper. Usually, it isn't a single path with a single point of failure. It's multiple paths at once, and the alert explosion engineers have to chase grows dramatically, driving the triage tax even higher.

How Causely improves Dynatrace

Dynatrace provides rich instrumentation for the first three layers in Causely's six-layer semantic model. Causely uses that data to do three things:

1. Extract the inventory of entities to instantiate the entity model,
2. Map the relationships between them to build the topology model, and
3. Use alerts generated in Dynatrace - their way of flagging that a symptom is present - as inputs to the behavior model.

Dynatrace stops at layer three. The layers above it, starting with causality, are what Causely brings to any environment (with or without Dynatrace). Causely maintains - in advance - an encoded set of potential causes and the symptoms each one produces, kept current as the environment changes. When ten alerts fire, Causely treats them as observed evidence, checks that evidence against the encoded causes, and identifies which encoded cause explains them. The output isn't a promoted alert. It's a diagnosis.

What a Fortune 50 Dynatrace customer achieved

In a recent engagement with a Fortune 50 enterprise, the validated result of adding Causely to the environment was a reduction in alert noise of more than 90%. 

A flood of Dynatrace alerts describing symptoms were consolidated into a single notification containing the actual cause explaining the observed symptoms. The cause and its blast radius were available in seconds and a triage call was avoided. For an engineering leader building the case to their CTO, there is no stronger proof that Causely reduces on-call load and MTTR.

Better, faster, cheaper agents

While noise reduction is a strong benefit, it’s just the starting point. A verified cause provides an even more important foundation for the on-call agents.

Better. An agent grounded in a diagnosed cause acts on the right service. An agent reasoning over ten raw alerts or investigating different fault tree analysis paths is guessing, and a wrong guess in production doesn't just fail; it can make the incident worse. In a 72-experiment benchmark, agents grounded in a causal model detected faults with zero false positives: no hallucinated causes to chase down

Faster. The diagnosis is available the moment alerts fire, because the causal model was built before the incident, not reconstructed during it. The agent skips the investigation phase entirely and starts at the remediation step. 63% faster in that same benchmark.

Cheaper. Feeding an agent ten of alerts, the logs behind them, and the topology around them, and asking it to reconstruct the causal knowledge for each of them every time the alerts are fired burns tokens on every query while the agent re-derives what a causal model already knows. Feeding it one diagnosed cause with its blast radius is a fraction of the context and none of the wasted remediation cycles. 57% lower operating cost across the same study.

Dynatrace's own 2026 release direction points toward autonomous prevention, remediation, and optimization at scale. The whole industry is moving toward letting agents act on findings rather than just surface them. Causal grounding must come before agentic action, not alongside it.

See the hard data behind the numbers.  

Who sees the data, and who acts on it

Grounding AI agents isn't only about whether the reasoning is correct. For enterprises in regulated industries, it's also about who sees the data and which model gets to act on it. Causely runs in the customer's environment, connects to Dynatrace wherever it's deployed, and processes data locally. The causal model and the LLM stay separate layers: the causal model runs in the customer's environment, and the customer controls which LLM is used to ingest the causal output. Remediating an issue never requires sending topology or log data to an LLM the customer didn't choose.

Expanded Dynatrace support

With our latest release, Causely now ingests how data flows between application components and underlying infrastructure directly from Dynatrace instrumentation, including public cloud and on-prem environments such as VMware. Your existing Dynatrace infrastructure alerts now automatically inform the model, mapping to infrastructure symptoms Causely already tracks, alongside the application-layer alerts already mapped.

This makes it easier to put Causely's semantics and causal model on top of your Dynatrace instrumentation wherever your applications run, so you reach the cause of alerts faster and dramatically reduce alert noise and MTTR. There's no separate integration project: adding a Dynatrace target is the only step needed. From there, you can connect that semantics to an AI agent for autonomous remediation, either through the Causely remote MCP server with your agent of choice, or, for more control over the data, running the whole thing in your environment.

FAQ

What does "cause in seconds" actually mean? It means the causal model has already encoded the potential causes in your environment and the symptoms each one produces, before the incident happens. Each potential cause is encoded as a unique signature of symptoms it may cause. When alerts fire, they're evidence checked against the pre-computed signatures, not the start of a live investigation. The diagnosis is inference against a known model, which is why it takes seconds.

Does this replace Dynatrace, or sit alongside it? It sits on top of an existing Dynatrace deployment. Causely doesn't re-instrument the environment. It ingests the entities, topology, and alerts that Dynatrace already emits and adds the causal layer, cause and effect, that turns them into a diagnosis.

Isn't Dynatrace's telemetry already enough on its own? The telemetry is the foundation, and it's genuinely rich, covering Kubernetes, serverless, cloud, and on-prem. What it doesn't include is causality, the model that says which problems produce which symptoms. Causely builds that layer on top of what Dynatrace already collects.

Does this work outside Kubernetes, or only for container platforms? Yes. The expanded integration covers EKS, ECS, and VMware, so the pattern holds whether the topology is Kubernetes-based, ECS-based, or VMware-based in your own data center.

Do I need to change my Dynatrace setup to use this? No. Adding a target through the Causely integration page is the only step. No separate instrumentation or migration project is required.

What to do next

Connect a Dynatrace target if you're already running on EKS, ECS, or VMware and want to see the diagnosis against your own topology. Get started here.

Why this matters now

As of mid-2026, many reliability teams have wired an LLM into some part of on-call: triage summaries, an agent that reads alerts and proposes causes, a chat interface over observability data. Most of these systems work by pushing more context into the prompt. More logs, more history, more runbooks. A study across 11 frontier models, including GPT-5, Claude 3.5 Sonnet, Gemini 2.5, and DeepSeek R1, found that this instinct is off. Accuracy on realistic tasks started dropping at a few hundred to a few thousand tokens, not at the million-token ceilings vendors publish. An on-call agent whose context grows with the incident gets less trustworthy exactly when you need it most.

How much of a model's context window can LLMs actually use?

LLMs do not effectively use nearly as much of the context window as the spec sheets claim.  Paulsen's study of 11 frontier models introduced the concept of the Maximum Effective Context Window (MECW): the point at which additional input begins to degrade a model's output. That's distinct from the Maximum Context Window (MCW) that providers advertise. The gap was large. Most models showed severe accuracy degradation by roughly 1,000 tokens, a few failed with as little as 100 tokens in context, and every model tested fell short of its advertised MCW by up to 99%+.

This isn't a new observation. Liu et al. (2023) first documented that models favor information at the start and end of a prompt, while losing content buried in the middle. Researchers call this "lost in the middle." Chroma's 2025 technical report tested 18 models, including GPT-4.1, Claude 4, and Gemini 2.5, and found that all of them degraded as input length increased, even on simple retrieval tasks (Hong et al., 2025). Paulsen's contribution shows that this holds across a wider range of task types and the specific token counts in which it occurs. Product manager Teresa Torres ties this same body of research to day-to-day AI usage in her write-up on context rot, worth reading if you want the pragmatic view alongside the research.

Do more reasoning steps mean worse accuracy?

What predicts accuracy is how much data a model must actively track when it produces its final answer, not how many logical steps the task requires. Paulsen's study found that summarization, which sounds simplest since it's a single operation, performed worse than the find-and-sum task, which requires two. Find-and-sum lets the model discard irrelevant facts early, so only a short filtered list has to stay in context for the final calculation. Summarization has no discard step, so the model has to keep the entire dataset in context throughout.

That's the mechanism: fewer steps don't help if the model still has to track the full dataset at the end. Filtering out what's irrelevant before the final reasoning step is what protects accuracy.

Why are on-call and SRE agents especially exposed?

Incident triage is a multi-step, context-accumulating task. Every additional signal an agent pulls in (a new log line, a related trace, an escalation message) pushes it further past the point where the next reasoning step degrades. An agent investigating a cascading failure doesn't get a clean, bounded prompt. It collects logs, metrics, topology, runbooks, and a growing chat history, all of which are appended as the investigation continues. That's close to the worst case in Paulsen's data: a multi-step reasoning task over an expanding context window.

Orchestrate agents together, and the math gets worse fast. Three agents at 70% accuracy each, a modest and individually reasonable number, compound to a 34.3% system success rate. Each agent's 70% is often measured near or beyond its effective context limit, which makes the actual number in production likely lower.

Is this settled, or is there a counterargument?

Some newer models show little to no "lost in the middle" effect on simple fact retrieval. Reports on Gemini 2.5 Flash describe near-perfect recall regardless of where a fact sits in a long document. Epoch AI's analysis found that the input length at which top models hit 80% accuracy on long-context benchmarks has risen by more than 250x over nine months. Providers are getting better at simple retrieval over long inputs.

This depends on the task type. The counterevidence focuses on single-fact lookup, the easiest category in Paulsen's study and the one closest to being solved. The degradation that persists, and that researchers all converge on, is worst on multi-step reasoning: summarizing, sorting, correlating multiple signals into one conclusion. That's precisely what root cause analysis is.

What does a causal model do differently?

A causal model doesn't hand the LLM the raw data. In Causely's architecture, causal inference over your service and resource dependencies runs deterministically upstream of any LLM call. By the time a model is involved, it isn't reasoning over raw logs, traces, and a sprawling incident history. It's working from a small set of named diagnoses that the causal model has already isolated.

A raw incident context (logs, traces, topology dumps, chat history) can easily run into the thousands of tokens before an LLM even starts reasoning. A causal summary naming the one or two services responsible is a small fraction of that. The LLM's job shrinks from "find the cause in everything we've collected" to "remediate a cause we already found." That keeps it well inside the range where these models are still reliable.

What this means for your agentic architecture

To increase your agent's accuracy, you should treat effective context, not advertised context, as your design constraint. Scope each agent to a narrow task rather than one agent doing full triage end-to-end. Bound how much raw telemetry any single LLM call receives, regardless of the model's advertised window. Prefer structured, pre-filtered summaries (causal, symbolic, or otherwise) over raw data dumps whenever a model has to reason, not just retrieve. An agent whose accuracy drops as an incident drags on doesn't need a longer context window. It needs a shorter, engineered one.

FAQ

What is the Maximum Effective Context Window (MECW)?
MECW is the point in a model's input length where additional tokens start measurably degrading its output, as distinct from the Maximum Context Window (MCW) providers advertise. Research across 11 frontier models found MECW is often a small fraction of MCW, sometimes under 1,000 tokens even when the model supports millions.

Doesn't a bigger context window just solve this?
No. Bigger windows raise the ceiling, not the floor. Providers have made real progress on simple factoid retrieval over long inputs, but multi-step reasoning tasks like summarization and root cause correlation still degrade well before the window fills up, according to multiple independent studies.

How do I keep an on-call agent inside its effective context window?
Scope each agent to a narrow task, cap how much raw telemetry any single LLM call receives, and filter or summarize before the model reasons rather than after. An agent that reasons over a pre-filtered set of candidate causes will typically outperform one reasoning over the full incident history.

Is this a RAG problem or a context-window problem?
Both, one causes the other. Retrieval-augmented generation reduces hallucination when the retrieved content stays inside the model's effective context window, but research shows accuracy actively worsens once retrieval pulls in more than that. Naive RAG that isn't context-aware can make context rot worse, not better.

What to do next
If you want the token-cost side of this problem, read our benchmark on why AI agents burn tokens on every reliability query, or see how Causely turns an incident into a small set of named diagnoses before any LLM is involved.

Java's dynamic compilation makes traditional eBPF probe placement hard. The JIT compiler moves code around, so the static memory addresses that eBPF relies on aren't stable. And the moment traffic hits TLS, which it does for every call to a cloud-managed service, every JDBC connection pool, every gRPC stream, you lose visibility unless you can decrypt in-kernel.

At GrafanaCon, I presented this work alongside Nikola Grcevski, Principal Software Engineer at Grafana Labs. The OpenTelemetry eBPF Instrumentation project (OBI) solves these constraints by correlating kernel-level events with application semantics, without any JVM modification. A small dynamically injected agent handles TLS decryption at the kernel boundary, so traces flow through encrypted connections without touching application code.

For Causely customers, OBI closes a gap that has blocked full topology coverage of Java estates. Once OBI is in place, Causely's causal model has the service dependency data it needs to identify the root cause service in a complex microservices incident and to map the blast radius accurately. In a Java application with 30+ interconnected services, that difference is between "something is slow in checkout" and "the Keycloak token validation service is the origin, and these six downstream services are affected."

eBPF and Java agents are complementary, not interchangeable. Where you need full semantic depth, for example custom business metrics, rich span attributes, the OTel JDK agent is still the right call. Where you can't touch the runtime, OBI covers the gap.

If you’d like to try OBI out on a sample application here are three realistic Java service topologies, all open source:

• Spring Boot + Keycloak + HTTPS — auth flows with encrypted external calls
• Spring Boot + gRPC + Google Pub/Sub — async messaging over TLS
• Quarkus + Postgres (TLS) + Kafka — polyglot data tier with encrypted DB connections

To learn more, watch the full GrafanaCon talk for a deeper walkthrough of how Nikola and team solved this challenge and the tradeoffs to consider when using OBI. Active development is tracked via Issue #880 and PR #891.

Why does a 25-tool MCP server need Skills?

The Causely MCP server gives agents and AI assistants direct access to Causely’s live causal model — a continuously maintained representation of your environment's services, resources, dependencies, and failure states. Because the causal model encodes service dependencies, resource states, and failure causality, most reliability questions can be answered in a single well-chosen tool call. One call to get_service_summary, one call to get_root_causes, or one call to get_incident_impact . Rather than the multi-step reconstruction, an agent has to perform against raw telemetry.

That’s the baseline Causely establishes. Skills push it further. When a platform engineer or agent builder wires up the MCP server for the first time, they’re looking at 25 tools across five categories. The natural question is: which one do I actually call for this situation? Without guidance, teams end up building the answer themselves — writing system prompts that encode when to call which tool, or prompt-engineering the selection on every query.

Skills eliminate that work entirely. The goal is one prompt, one routed call, one causal answer. For most reliability scenarios — triage an alert, check a deploy, generate a postmortem — that’s exactly what happens. Some complex investigations require a short sequence of calls, but even then Skills handle the sequencing; you don’t.

What are Causely MCP Skills?

Causely MCP Skills are pre-built routing workflows that sit above the 25 MCP tools. There is one master router — causely-mcp — and six specialist skills, each covering a distinct class of reliability work.

causely-mcp activates on any observability or reliability question, matches the prompt against each specialist’s trigger set, and delegates automatically. You never invoke a specialist directly.

The six specialists and what they cover:

Each specialist runs the correct tool sequence for its scenario. causely-alert-triage, for example, calls get_alerts to fetch the raw alert payload, follows up with get_root_causes to surface the confirmed causal analysis, and maps the result back to the specific alert, without you specifying any of that.

If you want to route to a specific specialist directly, prefix your prompt with the skill name:

use causely-postmortem: write a retrospective for the payment incident on April 25

You can also bypass Skills entirely and call MCP tools directly for explicit programmatic control.

Why Skills compound the token efficiency Causely already provides

There are three tiers of agent efficiency when it comes to reliability work, and it helps to be explicit about where each one sits.

Tier 1 — Raw telemetry tools with no causal layer
The agent has to reconstruct the environment state from scratch on every query: pull metrics, scan logs, infer topology, correlate signals across services. Benchmarks across 72 experiments measured what this costs: token consumption 60% higher, time-to-diagnosis 63% slower, and hallucinated incidents on healthy clusters at 67%, compared to agents working with Causely’s causal context.

Tier 2 — Causely MCP tools without Skills
The agent has access to deterministic causal answers: named root causes, computed blast radii, live dependency graphs. It doesn’t need to reconstruct anything. That’s where the 60% token reduction comes from. But the agent still has to decide which of 25 tools to call, and may still require prompt engineering to get the selection right.

Tier 3 — Causely MCP tools with Skills
The router handles tool selection. For most scenarios — triage an alert, validate a deploy, generate a postmortem — one prompt produces one routed call and one causal answer. The prompt engineering overhead disappears. This is the target state.

We’ve written about why generic diagnoses break AI ops agents and what named root causes change. Skills are the interface layer that delivers those named causal answers without requiring the agent or the engineer to know which tool surfaces them.

Which clients support Skills today?

Skills are available now in Claude Code, Claude Desktop, and Cursor, with support for more clients on the way.

The Skills repository is also open source. If you’re building a custom reliability agent — a LangGraph workflow, a Claude prompt chain, an internal SRE bot — the Skills are a usable starting point for tool-selection logic and investigation sequencing. Fork, adapt, or reference them directly.

For any other MCP-compatible client — JetBrains IDEs, Windsurf, Zed, Kiro CLI, HolmesGPT — point it at https//api.causely.app/mcp to access all 25 MCP tools directly. We published a Cursor plugin last month if you want the full setup context for that client.

FAQ

What are Causely MCP Skills?
Causely MCP Skills are pre-built routing workflows for the Causely MCP server. A master router called causely-mcp sits above six specialist skills covering alert triage, change impact, correlated incidents, health reporting, Kubernetes investigation, and postmortems. You describe your situation in plain language; the router selects the right specialist, which runs the correct tool call sequence and returns a causal answer. No manual tool selection required.

Do I need to know which Skill to invoke?
No. The causely-mcp master router handles delegation automatically based on your prompt. If it picks the wrong specialist, you can override it with use causely-<skill-name>: as a prefix. You can also bypass Skills entirely and call the 25 MCP tools directly if you need programmatic control — the full tool reference covers all parameters.

What’s the difference between using Skills and calling MCP tools directly?
Direct tool calls give you explicit control over every parameter, useful when building a custom agent that needs to chain results programmatically. Skills are the right choice when you want the correct tool sequence for a known scenario (incident triage, post-deploy check, postmortem) without writing the orchestration yourself. Most conversational use cases — investigating an alert, checking post-deploy health, generating a postmortem — are better served by Skills.

Which AI clients support Causely Skills today?
Skills are available in Claude Code, Claude Desktop, and Cursor (one-click plugin or manual setup), with more clients on the way. Any MCP-compatible client can access the full 25-tool MCP server at https://api.causely.app/mcp directly. The Skills repository is also open source — if you’re building a custom agent on top of the Causely MCP, the Skills are a usable starting point for tool-selection logic and investigation sequencing.

How do Skills reduce token consumption?
There are three tiers: raw telemetry tools require multi-step environment reconstruction on every query; Causely MCP tools without Skills return deterministic causal answers that cut token consumption by 60% (measured across 72 benchmark experiments); Causely MCP tools with Skills add routing on top, so most queries resolve in a single call with no prompt engineering required. Each tier compounds the efficiency gain of the one before it.

What to do next

The Skills documentation is at docs.causely.ai/agent-integration/mcp-server/skills/. If you’re on Cursor, the one-click plugin is the fastest path. Claude Code and Claude Desktop setup takes about five minutes via the manual copy-from-repo instructions.

If you want to understand the efficiency case in depth before committing to the integration, the token benchmark post has the full methodology and results from 72 experiments.

The word context is overloaded in observability discourse to the point of losing precision. It is used to refer to shared labels, a topology map, temporal proximity, and propagated trace identifiers. Four distinct things, each useful, none of them sufficient.

This blog defines a precise, layered model of observability semantics intended for engineers and SREs building or operating complex distributed systems.

The Gap in Current Observability Tooling

Most observability tooling today implements a shallow semantics of entity and some form of topology, often referred to as the knowledge graph – typically derived from trace data – with partial reach into behavior through threshold-based alerting and anomaly detection. A knowledge graph is a generic term that does not define the semantics of what it captures. It is typically assumed to be a list of services and their topological relationships, but as we point out later, this is not sufficient to answer questions. It is also a dynamic, ever-changing graph; capturing it only once, or once a week, is not a reliable representation for real-time, continuous, automated operation. Deeper semantics are largely absent from tooling and instead exist as undocumented knowledge held by individual engineers. 

This lack of semantics has predictable operational consequences:

• Alert fatigue: alerting systems fire on symptoms without the ability to identify the root cause, producing storms where a single fault generates dozens of notifications
• Slow incident response: root cause reasoning and impact analysis happen manually, in real time, under pressure, because the causal and dependency knowledge exists only in human memory
• Blast-radius estimation is guesswork: a lack of knowledge of the impact of failure. There is no systematic way to ask what would happen if a given component failed, which dependent services would be affected, which symptoms would surface, and in what order.
• Causal knowledge is reactive, not preventive: no proactive reasoning about causes. Engineers cannot ask what classes of failure could produce a given set of symptoms, or which components in the current system state are most likely to trigger an incident.
• No counterfactual reasoning: there is no way to assert that a specific condition could cause a specific failure, or conversely that a given failure mode cannot produce a set of observed symptoms. Without formal causal and dependency knowledge, both directions of reasoning – “could this cause that?” and “could that have caused this?” – rely entirely on individual experience
• Knowledge loss: engineers who have internalized the causal and constraint knowledge leave, and the organization reverts to slower, less reliable incident response

The data to address these problems is largely available in service meshes, infrastructure APIs, configuration management systems, and telemetry pipelines. The gap is not data. It is the absence of explicit semantic models that make that data legible to AI Agents. The sections below define the models. 

A Layered Semantic Model for Observability

The following six layers describe the semantic structure required to move from “we see a signal” to “we understand what is happening and why.” Each layer builds on the previous. Tooling that skips layers will exhibit predictable gaps in reasoning capability.

It is important to note that the semantics model at each layer has two parts, a knowledge base and a dynamic graph. The knowledge base captures the generic knowledge of what may be observed. The knowledge base captures abstract semantic descriptions common across all environments. The dynamic graph instantiates the knowledge base to generate a real-time model of the managed environment at a given point in time. The dynamic graph is the actual observed reality, specific to a given environment at a given point in time, and keeps adapting as the environment changes.

Layer 1: Entity Model

The entity model defines the named objects in the system and the attributes that describe them. An entity is any discrete thing with identity and an observable state.

The entity knowledge base captures the types of entities that may be discovered, e.g., a service instance, a host, a Kubernetes pod, a database cluster, a message queue, a CDN edge node, an AI Agent, an MCP server with tools or an LLM model and its provider, and the attributes that may be observed:

• Identity: UUID with associated attributes like name, namespace, cluster, region, environment, labels
• Configuration: resource limits, replica count, connection pool size, JVM flags
• Runtime state: CPU utilization, memory used, request rate, error rate, queue depth

The dynamic entity model instantiates the knowledge base with the discovered entities and serves as an inventory of all entities in the environment at a given point in time. For example, the list of services, hosts, pods, databases, message queues, etc., in the environment.

Without an entity model, telemetry is a stream of dimensioned numbers. With one, it is a description of a system composed of named, typed, queryable objects. This semantics is a prerequisite for all higher-order reasoning.

Layer 2: Topology Model

The topology model encodes the structural relationships between entities: which services call which databases, which pods run on which nodes, which queues are consumed by which workers, etc.

The topology knowledge base captures the relationship that may exist between types of entities, For exaple, a service may be connected to another service, a service may be accessing a database, a service may be layered over a pod.

The dynamic topology graph represents the actual discovered relationships between discovered entities. For example, the Coordinator Agent is connected to the Research, Coding, and Ops subagents; the GitHub MCP server, which includes the create pull request tool; and the GPT-4o model, which is layered over the OpenAI model provider (see Figure 1).

It is important to note that the knowledge base captures design-time assertions, not runtime observations. This distinction is critical. A service dependency map derived from trace data reflects what happened during the observation window. The dynamic topology graph reflects what is. Traces may be incomplete (non-instrumented paths, async operations, batch jobs). The dynamic topology graph enables blast-radius analysis; given that entity foo is degraded, which other entities are structurally dependent on it and therefore at risk.

Layer 3: Behavior Model

A behavior model captures the observable signals of normal and abnormal system behavior. It defines symptoms – semantically meaningful deviations from expected state – and events – discrete state transitions that represent changes in entity health or configuration.

A raw metric is not a symptom. The number 95.3 on a CPU utilization gauge is not meaningful without knowing the entity it describes, the threshold at which that utilization constitutes an anomaly, and the operational context in which that threshold applies. A symptom is the semantic interpretation: this entity is exhibiting CPU saturation relative to its configured capacity and expected workload.

The behavior model is what converts monitoring data into an actionable signal. Alerting systems that operate directly on raw metrics without a behavior model produce high false-positive rates because they lack the semantic layer needed to distinguish signal from noise.

The behavior knowledge base captures the possible observable anomalies an entity may experience. For example, a service may be degraded, experiencing high latency, or a node may be highly utilized, experiencing high CPU utilization.

The dynamic behavior model represents the environment's actual runtime state based on what is being observed. For example, the Research subagent is degraded and experiencing high latency, or the Postgres MCP is congested and experiencing high query times.

Layer 4: Causality Model

The causality model captures the cause-and-effect relationship between potential causes and the symptoms or events they may cause.

The causal knowledge base describes the potential causes, the symptoms they may cause, and how they may propagate. Propagation captures how a cause occurring on one entity may propagate to another related entity.  

The causal knowledge base captures generic knowledge about the types of causes that may occur in an environment and how they will manifest, i.e., the symptoms that may be observed when they occur.

For example:

• An MCP service may be congested and when it is congested it may cause its tool calls to be slow, manifested as a high tool-call latency symptom, and may propagate and cause degradation of subagents that depend on it.
• A subagent may be degraded and when it is degraded it may cause its responses to be slow, manifested as a slow response time symptom, and may propagate and cause degradation of the coordinator agent that orchestrates it.
• A model provider may be degraded and when it is degraded it may cause its inference requests to be slow, manifested as high inference latency, and may propagate to subagents using its models causing them to be degraded.

As the above examples illustrate, the causal knowledge base is completely independent of a given environment. It is independent of how many providers, MCP services, and/or subagents are in the environment, and which models run on which providers, which subagents access which MCPs, or which coordinator orchestrates which subagents.

The dynamic causal graph represents the actual runtime causality in the environment at a given point in time. It is a function of the dynamic topology graph and represents all potential specific causes that may occur in the environment, along with the specific symptoms they may cause. The figure below illustrates a concrete example: the Postgres MCP may be congested, and the fault may propagate through the execute_sql tool call to the Coding and Ops subagents, and then onward to the Coordinator, with symptom probability decreasing at each hop. Similarly, Anthropic provider degradation may propagate through Claude inference to the Research, Coding, and Ops subagents, and then to the Coordinator.

In this example, the dynamic causal graph captures that a congested Postgres MCP may cause degradation of the Coding and Ops subagents and the Coordinator. It is important to note that the keyword semantically is “may”. A congested Postgres MCP may degrade downstream entities, but that doesn’t necessarily mean the Postgres MCP is congested – Anthropic provider degradation could produce overlapping symptoms. To conclude, the cause of an observed symptom requires analyzing the state of all potential symptoms.

Cause (left) propagates rightward through topology edges. Probabilities indicate the likelihood of each symptom given the cause. Multiple symptoms may share a cause; diagnosis evaluates the full set of observed symptoms against this graph.

The causality model is what makes root cause analysis tractable. Without it, a single infrastructure fault that produces symptoms across dozens of dependent services generates an alert storm with no structural differentiation between cause and effect. With it, symptoms can be suppressed or annotated as consequences once their cause is pinpointed.

Layer 5: Attributes Dependency Model

Where the causality model operates at the symptom level, the attributes dependency model operates at the attribute level. As its name suggests, it captures attribute dependencies. The dependent attributes can be of the same entity or of different entities. Dependency knowledge captures that attributes are correlated, along with the directionality and function of the correlation. The function can be defined in the model or learned.

The attribute knowledge base captures the generic knowledge of which attribute depends on which. Examples of such dependencies:

• Subagent response latency is a function of the tool-call duration of the MCP services it invokes
• Coordinator response latency is a function of the response latencies of the subagents it orchestrates
• Subagent inference latency is a function of the inference latency of the model provider serving its model
• A subagent's token throughput is a function of the rate at which the coordinator dispatches requests to it.
• The end-to-end coordinator latency is a function of both per-call latencies and the number of subagent calls per task. A non-linear relationship, since a coordinator may fan out to multiple subagents in parallel, retry on failure, or recurse on intermediate results.
• Total tokens consumed by a subagent are a function of both the per-call token cost and the number of model-provider inference calls, which scales non-linearly with conversation depth, tool-call retries, and self-reflection loops.

The dynamic attribute dependency graph represents the actual run-time dependencies in the environment at a given point in time. It is a function of the dynamic topology model. Given the above static model examples, the dynamic model examples would be:

• If the Coding subagent is accessing the Postgres MCP, Coding subagent latency is a function of execute_sql tool-call duration
• If the Coordinator is orchestrating the Coding subagent, Coordinator latency is a function of Coding subagent latency. Note that based on this, Coordinator latency can be computed as a function of execute_sql tool-call duration, even though the Coordinator does not directly access the Postgres MCP.
• If the Research subagent is using Claude served by Anthropic, the Research subagent's inference latency is a function of Anthropic's Claude inference latency.
• For the Ops subagent, its token throughput is a function of the rate of requests the Coordinator dispatches to it.
• If the Coordinator fans out a single user task to the Research, Coding, and Ops subagents in parallel, total Coordinator latency is approximately the max of the three subagent latencies, but if any subagent retries on tool-call failure, that subagent's effective latency multiplies by its retry count, so end-to-end latency grows non-linearly with failure rate.
• If the Coding subagent calls the Postgres MCP once per row in a result set, Coding subagent latency is a function of execute_sql duration multiplied by row count, so a query returning N rows produces N tool calls, and end-to-end latency scales with N rather than staying constant.
• If the Research subagent invokes Claude in a multi-turn reasoning loop with branching factor B and depth D, total Anthropic inference calls scale as B^D – so doubling reasoning depth roughly squares the inference cost, even though the Research subagent's per-task topology entry is unchanged.

The attribute dependency graph enables change impact analysis: before modifying a configuration parameter, an operator or automation system can trace the dependency graph to identify which runtime attributes will be affected, on which entities, with what expected magnitude.

Layer 6: Constraint Model

The constraint model defines the desired state. The desired state is one in which goals are achieved while constraints are satisfied. Each entity in an environment may have a desired state, but, more interestingly, the environment as a whole may have one as well.

A desired environment state is one in which all applications deliver on their goals while running within their operational constraints. Usually, the goals are performance goals of latency and throughput, and service level goals in terms of errors and availability, while the operational constraints are capacity, budget, compliance, and configuration.

An entity's desired state is the configuration space within which the entity and its attributes operate without triggering symptoms or degradation events. It is the semantic encoding of the system’s operational envelope.

The constraint model captures both the environment's desired state and the entity's desired state. These can be defined independently, but by using the attribute dependency model, the entity's desired state can be generated/inferred from the environment's desired state, resulting in a very compelling and powerful system.

As in all other layers, the constraint knowledge base captures generic knowledge about constraints, and the dynamic constraint model represents the runtime constraints at a given point in time based on the discovered topology. Constraints are not just thresholds on metrics. They are relationships between configuration parameters and the runtime conditions required for healthy operation:

• Headroom constraint: Container memory limit >= 1.4 x used heap
• Budget constraint: Application cost < budget
• Latency constraint: Service latency < 100ms

When the constraint model is explicit, it becomes possible to answer operational questions that today require senior engineering intuition: “What is the minimum safe memory limit for this service at current traffic?” “If we reduce the connection pool by 40%, at what request rate will we begin to see queuing symptoms?” The constraint model is the missing link between observability (what is happening) and operations (what configuration produces health). It is the semantic layer connecting configuration management to runtime behavior. 

What “Context” Actually Covers and Where It Falls Short

Most observability platforms use the word loosely. Here is what the four common implementations actually encode, and where each one stops.

Shared Labels

Common attributes – service, environment, region, version – are attached to metrics, logs, and traces. This allows tooling to pivot between signal types for a given entity. The relationship is implied by label equality, not by any explicit model of dependency or causation. Two data points sharing service=checkout are navigable together; they are not necessarily related by anything other than that label. This is a partial implementation of the entity model (layer 1) that lacks declarative topological relationships and behavior. 

Topology Map

A topology map encodes structural relationships between entities, which services call which databases, which pods run on which nodes, and which workloads share a cluster. It is the most semantically substantive form of “context” in common tooling and is typically derived from trace data, service-mesh telemetry, or infrastructure APIs. This is a partial implementation of the topology model (layer 2): it captures that entity X is connected to entity Y, but not what behavior propagates along that edge, why a fault on X causes a symptom on Y, or which of X’s attributes drives which of Y’s. A topology map shows the wiring; it does not encode behavior (layer 3), causality (layer 4), attribute dependency (layer 5), or constraints (layer 6). Two services connected in the topology may be tightly coupled in failure propagation or entirely decoupled. The map alone cannot distinguish them. In practice, this means a topology map supports navigation (“show me what this service talks to”) but not reasoning (“if this service degrades, what symptoms should I expect, on which dependents, and why”). That reasoning requires the higher layers.

Temporal Proximity 

Events co-located on a timeline are assumed by operators to be potentially related. The system carries no model of that relationship; pattern recognition is entirely delegated to the human. This is correlation by coincidence, not by structure. It addresses none of the six semantic layers explicitly; it is a UI affordance for human reasoning, not a semantic model.

Distributed Trace Propagation

A trace context header (W3C traceparent, B3, etc.) propagated across service boundaries encodes a genuine causal chain: span B was initiated by span A. This is the most semantically rich form of “context” in common use. It partially implements the topology model (layer 2) for instrumented call paths, and provides raw material for the behavior model (layer 3). However, it is constrained to synchronous, instrumented code paths and does not capture async operations, batch jobs, or infrastructure-level dependencies. It carries no causality, dependency, or constraint models.

In summary, shared labels, temporal proximity, topology, and trace propagation collectively reach into layers 1 and 2. Layers 4 through 6 – causality, dependency, and constraints – are entirely outside the scope of what “context” addresses. That is the gap.

Conclusion 

Context – shared labels, topology map, temporal proximity, trace propagation – is a necessary foundation for navigating observability data. It is not sufficient for understanding systems. The jump from “we can see the signals” to “we understand what is happening and why” requires explicit semantic models at six distinct layers: entity, topology, behavior, causality, dependency, and constraint.

The practical implication for SREs and platform engineers is that runbooks, tribal knowledge, and escalation paths are informal implementations of layers 4 through 6. The question is not whether to encode this knowledge, which already exists, but whether to leave it implicit in human memory or make it explicit in a form that tooling can reason about.

Making semantics explicit is what makes a system legible to an AI agent. An agent reasoning over shared labels, topology, and temporal proximity alone is restricted to the same navigation problem a human faces, pivoting between signals without a model of why they are related. An agent reasoning over layers 4 through 6 can answer the questions that actually drive incident response, capacity planning, and change approval: what is causing this symptom, what is the blast radius of a degraded component, what configuration change would restore the constraint, and what would happen if that change were made. These questions do not require new data; the telemetry, configuration, and trace propagation already carry the raw signal. What they require is a model that the agent can reason against. One that encodes the causal, dependency, and constraint knowledge that today lives only in the engineers who happen to remember.

Semantics is not an observability feature. It is the foundation that makes observability useful.

Why do traditional observability approaches miss GenAI failures?

Traditional observability is not designed for GenAI applications.  A latency spike or HTTP 5xx is easy to detect. An agent that starts calling tools in the wrong order, consuming three times the expected tokens, or failing to complete requests under load may generate no metric anomaly at all. Three properties of LLMs make this systematically hard.

Unpredictable inputs. A production LLM endpoint receives free text prompts whose distribution shifts as usage grows. A change in how users phrase requests, or a change in the context an upstream service injects, can alter agent behavior with no code change and no deployment event. Without capturing prompt content as a structured span attribute, there is no way to correlate a behavioral shift to its source.

Unpredictable outputs. The same prompt can produce structurally different responses across calls: different tool selections, different response lengths, different completion rates. These variations affect downstream services that consume the output. Without content captured in the trace, you have a status code and a duration. You do not have visibility into what the agent did.

Silent drift. Agent behavior can degrade gradually with no hard threshold to cross. Token consumption creeps up. Completion rates fall. Tool call patterns shift. None of these fire an alert on their own. By the time the degradation is visible in user-facing metrics, the operational signal has been present for days.

The OpenTelemetry community has been working to close this gap. The GenAI semantic conventions, now in active development, define a standard schema for exactly the signals that traditional observability misses in LLM workloads.

What do OpenTelemetry's GenAI semantic conventions give you?

OpenTelemetry's GenAI semantic conventions define a standard schema for the signals that matter for LLM workloads: spans covering LLM and tool calls, events capturing prompts and completions, and metrics for token usage and latency. The GenAI semantic conventions are now stable enough for production use, and enterprises are leveraging tools like langfuse or Arize Phoenix for their production agents.

The key signals the conventions define:

• Spans for each LLM call, embedding call, and tool call. Each span covers model name, provider, request parameters, and outcome.
• Events (gen_ai.user.message, gen_ai.assistant.message, gen_ai.tool.message) capture the full conversation history when capture_message_content is enabled.
• Metrics: gen_ai.client.token.usage (prompt and completion tokens) and gen_ai.client.operation.duration (latency histogram).

Once this instrumentation is in place, you have a structured, correlated record of every LLM interaction in your system. That record is the prerequisite for evaluations, root cause analysis, cost attribution, and autonomous operations.

Liudmila Molkova’s KubeCon talk on GenAI observability clearly makes the operational point. Classic telemetry signals (latency and errors) are necessary but not sufficient. Understanding what an agent did — which tools it called, in what order, with what inputs and outputs — requires capturing content alongside structure. Latency and status codes tell you that something happened. Spans and events tell you what.

What does OTel GenAI instrumentation give you?

Once OpenTelemetry's GenAI semantic conventions are in place, every LLM interaction in your application becomes a structured, correlated span. You can see what your agent is doing: which model it called, with what parameters, how long it took, how many tokens it consumed, what tool it invoked, and what came back.

GenAI applications do not exist in isolation. LLM calls sit inside a broader service graph. Upstream services that assemble context and inject it into prompts, downstream services that act on model outputs, and infrastructure dependencies that affect latency and availability across the entire chain. Connecting GenAI telemetry to that existing graph, with its service dependencies, deployment history, and causal relationships, is what turns isolated LLM spans into a coherent operational picture.

How Causely Addresses the Gaps in GenAI Application Reliability

Instrumentation gives you signals to know what is changing, but not why. Here are four examples:

1.     Latency climbed. Where is the bottleneck? In a multi-hop agent workflow, latency compounds across LLM calls, tool calls, embedding lookups, and the infrastructure serving them. A slow p95 on the agent endpoint could originate anywhere in that chain. Finding it requires understanding the full call graph and how each hop contributed, not just identifying the slowest individual span.

2.     Error rates are up. What failed and why? An increase in errors across an agent workflow could mean a tool is failing, a model is rejecting certain inputs, a dependency the agent relies on has degraded, or the agent is looping and exhausting retries. Each has a different fix. Without causal attribution, the investigation starts from scratch every time.

3.     Tool calls are failing or slowing down. Why? A spike in tool call errors or latency could mean that an underlying service the tool depends on has degraded, that the tool is being called with parameters it cannot handle, or that a recent change to the tool's interface broke compatibility with how the agent invokes it. The call completes without a timeout or connection failure, but the response indicates a problem (e.g., an error code or an empty result) that propagates silently throughout the rest of the workflow. Attributing the cause requires connecting the tool call signal to the service dependency graph and recent deployment events, not just inspecting the span in isolation.

4.     Something that worked at low volume is breaking at scale. Moving from thousands of tasks to millions changes the failure profile. Bottlenecks that were negligible become systemic. Failure modes that were rare become frequent. The causal path from signal to root cause grows longer as the agent graph grows more complex.

Connecting observed signals to their causes across model calls, tool calls, prompt changes, model updates, and the services that surround them is a core problem you can solve by deploying Causely in your environment. If you are working through any of these challenges today, we would love to talk. 

FAQ

Why is alerting on latency and error rate for my GenAI agent not enough?
Latency and error rates catch infrastructure failures. They do not catch operational failures specific to GenAI workloads. An agent that starts looping through tool calls, consuming three times the expected tokens, or failing to complete requests under load, can look perfectly healthy in your existing dashboards. The failure is in the behavior of the agent workflow, not in the underlying infrastructure. Capturing it requires visibility into what the agent did — which tools it called, in what order, how many tokens it consumed — not just whether the calls returned a 200.

What's the difference between GenAI observability and classic APM for AI workloads?
GenAI observability builds on top of classic APM. It does not replace it. Latency, error rates, and resource consumption remain essential. The OTel GenAI semantic conventions extend the existing OpenTelemetry schema to add the signals specific to LLM workloads: which model was called, with what parameters, how many tokens were consumed, what tools were invoked, and optionally the prompt and completion content. An SRE still cares about whether the call took 3.2 seconds. A different stakeholder — one responsible for the agent's behavior and outputs — needs the content layer on top of that. The conventions are designed to serve both.

How does token consumption relate to cost and quality?
Token consumption is both a cost signal and a quality signal. A spike in prompt tokens could mean users are sending more context, a template change bloated the system prompt, or an upstream service is injecting data it should not. A spike in completion tokens could mean the model is being more verbose, looping, or hitting edge cases in its instructions. Neither direction is inherently bad, but both warrant investigation when unexpected. Without prompt and completion content captured in the trace, you cannot tell which scenario you are in.

Where do I start if I am running GenAI workloads today?
Start with instrumentation. Enable OTel GenAI semantic conventions for your LLM client calls to get spans, token metrics, and (optionally) prompt and completion content into your observability stack. Once the telemetry is in place, the gaps described above become concrete rather than hypothetical, and the path to causal attribution becomes tractable. 

What to do next
If you are building or running GenAI workloads today and want to understand what causal modeling looks like in the context of LLM call chains, talk to our team.

That gap is invisible to standard OpenTelemetry instrumentation, and in a microservice or agentic architecture, it's one of the most frustrating latency sources to diagnose because nothing in your dashboard points to it.

At KubeCon + CloudNativeCon Europe 2026, I presented alongside Nikola Grcevski, Principal Software Engineer at Grafana Labs, on how eBPF-based instrumentation, specifically the OpenTelemetry eBPF Instrumentation project has been extended to surface DNS tracing and metrics directly from the kernel. No code changes, no sidecar proxies. The kernel sees the lookup; eBPF captures it.

Why this matters more now: agentic pipelines fan out. A single agent step may call four or five MCP servers in sequence or parallel. Each call starts with a DNS lookup. If any of those lookups are slow — stale TTL, misconfigured resolver, transient NXDOMAIN — you'll see latency on the agent response and no obvious cause in your traces. As MCP adoption scales, this blind spot scales with it.

Watch the talk for the full technical walkthrough, including how DNS spans are correlated back to the originating service and surfaced through standard OTel pipelines.

Most AI ops agents fail for the same structural reason. They have no semantic layer over the environment, so when an incident hits, the only way to reason about it is to pull mountains of data and ask an LLM to figure out what matters. 

That approach scales inefficiently and delivers inconsistent results. The more telemetry you have, the more data the agent must pull before it can answer anything. The answers are only as good as the model’s ability to find the proverbial needle in the haystack. This increases latency and token consumption without necessarily increasing accuracy.   

Causely is built on the opposite principle. The system’s causal context layer encodes how your services depend on each other and how failures propagate, so an agent can ask what caused this? and get a deterministic answer without an ad-hoc scan of the whole environment.  

Tools for the agents you’re already building 

There’s a category of vendor right now selling a packaged AI SRE. These are mostly LLM wrappers sitting on top of your environment, claiming to run investigations as a human would, and offering to hand you plausible answers as output. In many cases, these tools require access to your source code. 

We don’t think that’s the right solution shape. The teams who run production systems at scale are already building their own agents. What those teams need is not yet another agent. They need infrastructure that optimizes the performance and efficiency of the agents they’re building. 

That’s where Causely fits. The Causely MCP server gives any human or agent a continuously maintained ontology and deterministic  causal context  about the behavior of their environment. Causely gives engineering teams the substrate that makes their agentic ops workflows reliable You can read more about our MCP server here: https://docs.causely.ai/agent-integration/mcp-server/. 

How Cursor and Causely Work Together 

Causely continuously scans your environment for emerging causes of congestion and malfunction, including when you haven't explicitly configured a monitor or alert. These are reliability risks that can be addressed proactively, before an incident occurs or is declared.  

In a pre-AI world, proactive maintenance work mostly sat in a backlog because it felt like there were never enough engineering hands to work through it. With Cursor and other coding agents, that constraint is loosened. 

Causely runs in your cluster and processes your telemetry locally. Cursor as your coding agent already has access to your codebase and git history. So rather than building our own code-aware remediation surface, we made Causely available directly inside Cursor. When Causely pinpoints a code-related cause of observed congestion or malfunction in your operating environment, Cursor can take that context and propose the fix in the code that produced it. 

You can install it here: https://cursor.directory/plugins/causely.  

Get started 

Search for Causely in the Cursor plugins directory or use the link provided above, install it, and point it at your Causely instance. Your agent gets deterministic causal context with no extra data access required. 

If you're building ops agents and want to see how the causal context layer fits into what you're working on, you can try Causely here: https://www.causely.ai/try.

TL;DR. AI ops agents fail in production not because they lack telemetry, but because they work from generic diagnoses like "database slow ." A named root cause — with a mechanism, a propagation path, and a remediation — collapses open-ended investigation into a structured answer. Without that grounding, agents hallucinate plausible-sounding explanations from symptoms rather than causes. We've been expanding the causal model to cover more of the failure modes production systems actually exhibit, and there's more to share soon.

Why this matters now

The 2024–2026 wave of LLM-based "SRE agents" has a common architecture: give the agent a set of tools, a system prompt, and enough context to pattern-match, then let it investigate when something breaks. This works in demos. In production, it produces agents that take fifteen tool calls to conclude "the database might be overloaded." We've written before about why LLMs alone aren't enough for root cause analysis. The pattern from our collaboration with Google Gemini, an LLM on top of a causal model, not LLM alone, is an alternative.

Why do generic diagnoses break AI ops, agents?

AI agents are designed to diagnose as a human engineer would, reviewing signals, forming hypotheses, and narrowing in on a cause. It pattern-matches toward the most plausible-sounding explanation. In a degraded system, that's almost always a symptom — the loudest alerting service — not the cause.

Named root causes collapse the search space and guard against unwarranted hallucinations. An agent that receives Lock Contention instead of "database slow " gets the mechanism (competing transactions are blocking each other), the propagation path (which upstream services are affected), and a remediation direction, none of which it could reliably reconstruct on its own. But only if those named causes are grounded in the types of failures your system encounters.

What does a named root cause actually change?

A named root cause collapses the investigation into a diagnosis. Instead of receiving signals and inferring a failure, the agent receives a structured answer: this failure mode, propagating through these services, owned by this team. That affects three things at once.

Accuracy. The agent works from named diagnoses instead of constructing narratives from ambiguous signals. It no longer has to distinguish cause from symptom. The causal graph has already done it.

Cost. A targeted causal query replaces a broad environment scan. In LLM terms, that's the difference between a prompt stuffed with raw telemetry and one carrying a structured answer.

Noise suppression. A richer causal model distinguishes downstream symptoms from genuine root causes rather than firing on both. An agent that can tell a lock-contention cascade from connection-pool exhaustion won't wake three teams for one incident.

Why a causal graph, compounds, and a signature library doesn't

Named root causes in a causal model are not list entries. They are nodes in a graph: connected to the symptoms they produce, the services they propagate through, and the other causes they can be mistaken for. A signature library adds value linearly: each new pattern covers one more situation, and if it doesn't match, you get nothing. A graph adds value non-linearly: when Idle-in-Transaction Accumulation enters, the graph can now distinguish it from connection pool exhaustion, lock contention, and checkpoint I/O pressure, and the agent's reasoning at every neighboring node becomes less ambiguous.

That's the compounding property: the nth named root cause improves diagnoses on the first n−1 by differentiating them.

What "Table Access Failure" tells an Ops agent

A downstream service starts returning errors. An agent without a causal model observes elevated latency and error rates in the data layer, concludes "database overloaded," and recommends scaling, which changes nothing, because the database isn't overloaded. It's inaccessible. Table Access Failure is a distinct failure mode: the database is reachable, but the table itself can't be opened, typically because of permission changes, a dropped or renamed object, or exhausted storage at the tablespace level. The remediation path is specific: verify pg_class for the object, check pg_tablespace for space, and audit recent DDL and permission changes. An agent working from "database slow  " will chase connection metrics, miss the permission audit entirely, and escalate to a human with nothing resolved.

What "Lock Contention" tells an agent that "database slow " doesn't

Lock Contention is the failure mode that agents misread as a capacity problem. JVM threads are blocking on synchronized blocks or object monitors; throughput drops; latency climbs. The signals look like an overload. An agent may reach for horizontal scaling or pod restarts, both of which do nothing when the bottleneck is thread contention within the application, not a shortage of instances.

The named root cause carries the mechanism: which threads are contending, which code paths hold the monitor, and the right remediation (reducing lock scope, moving to non-blocking concurrency primitives, or restructuring the critical section). Scaling adds more threads competing for the same lock and can make things worse.

What about failure modes that aren't named yet?

A named root cause is the best case. The agent gets a mechanism and a remediation. But the graph's worst telemetry. Dependency direction still applies. Propagation reasoning still applies. The graph still distinguishes cause-side services from symptom-side ones, still narrows the search space from "every component in the environment" to "these three upstream of the observed symptom," and still tells the agent which signals are consequences, and which are spurious.  The structure is always there; a named root cause sharpens it, and the absence of one doesn't remove it.

Why a sharper graph expands what an agent can do alone

The practical consequence is that more incidents land inside the surface area where an agent can act without human confirmation, because the reasoning is grounded in structure rather than inference. That's the prerequisite for autonomous remediation: you don't let an agent act on a guess, but you can let it act on a named, verified root cause.

FAQ

What is a "named root cause" in observability? A named root cause is a specific, mechanistic diagnosis — Postgres Idle-in-Transaction Accumulation, Redis Cache Miss Storm — rather than a generic classification like "database slow" or "high latency." Each named cause encodes the failure mechanism, how it propagates through dependent services, and the remediation path. For AI ops agents, the name itself is the interface: it replaces open-ended investigation with a structured diagnosis.

How is a causal knowledge base different from a signature library or AIOps detection catalog? A signature library matches patterns; if a pattern isn't in the catalog, the system has nothing to say. A causal knowledge base models dependencies, propagation, and mechanisms, thereby narrowing the search space and distinguishing cause from symptom even when a specific named diagnosis doesn't apply. It also compounds: each new named cause sharpens the diagnoses around it in the graph, rather than just adding one more isolated detector.

What happens when my environment has a failure mode Causely hasn't named? The graph still does work. Dependency direction, propagation reasoning, and cause-versus-symptom separation apply regardless of whether the terminal node has a name. The agent receives a narrowed candidate set and structured context, rather than a generic "investigate the environment" prompt. Unnamed failure modes that recur are also how the model identifies gaps and decides what to add next.

Can't an LLM agent figure out the root cause itself with enough context? With enough tools and enough patience, sometimes. But an LLM has no model of service dependencies or event ordering, so it tends to latch onto the most plausible-sounding explanation — often a loudly alerting symptom. It also burns tokens proportional to the search space. A causal model does the dependency reasoning once, ahead of time, so the agent doesn't have to reinvent it under time pressure.

What telemetry do I need for Causely to diagnose Postgres and Redis failures? Standard OpenTelemetry metrics and traces, plus the database-level metrics most teams already collect: Postgres statistics views (pg_stat_activity, pg_stat_replication, etc.) and Redis INFO output. The causal model does not require custom instrumentation. If you are already running an OpenTelemetry Collector with a Postgres or Redis receiver, you have what you need.

What to do next

• Browse the full Causely root cause reference to see the failure modes the model currently covers.
• If you are already running Causely, stay tuned; we'll be publishing specifics on new root causes shortly.
• If you are building an agent workflow, talk to us and we will show you the diff against your environment.

Why should you care about the efficacy of reliability agents?

Reliability agents are gaining momentum. Enterprises are adopting AI into their DevOps and platform reliability workflows. McKinsey’s 2025 State of AI report found that 78% of organizations have deployed AI in at least one business function, with platform teams amongst the most frequent adopters. A new class of purpose-built SRE agents is emerging alongside general-purpose coding agents such as Claude Code and Codex, and both are being deployed to production to support incident detection, root cause analysis, change safety, capacity planning, and post-incident reporting.

These AI agents are inefficient and inaccurate without causal context. To answer reliability questions, the agent must reconstruct the environment state from raw telemetry on every query: what services exist, how they depend on each other, navigating error logs and metrics data to investigate incidents, etc., for every new query. This access and investigation pattern burns tokens, adds latency, and produces uncertain inferences, especially when the telemetry is ambiguous. For complex queries in complex environments, the AI agent must maintain heterogeneous data from multiple sources in context to effectively support the reliability use case. This process requires multiple token-heavy tool calls and coordination, along with semantic understanding.

And the impact of all of this is going to get worse. Today, the cost of AI inference is hidden by subsidized subscription-based AI pricing. However, AI providers are increasingly struggling to make the economics of AI inference sustainable with rising GPU and energy costs as demand proliferates. As a result, major providers, such as OpenAI and Anthropic, are shifting to usage-based billing measured by token consumption. As AI providers shift to pure token-based billing, the true costs of inefficient AI agents become unsustainable.

What did we aim to prove?

The hard part of reliability work isn't running a query. It's knowing which query to run. Grafana Labs made this point in their o11y-bench release this week:

 "In a real incident, the hard part is rarely just writing a query. It's deciding which signal matters, figuring out whether a spike is noise or symptom, correlating metrics with logs and traces."

 Yasir Ekinci, Principal Software Engineer, and Jack Gordley, Senior Software Engineer, Grafana Labs

That's the same gap we set out to measure, but from a different angle. o11y-bench asks whether agents can execute observability tasks. We asked something upstream of that: can agents decide which task to execute, and do they need causal context to make good decisions?

Our hypothesis was that the limiting factor isn't the agent's ability to query telemetry. It's the agent's lack of a structured model of the environment. Without one, every investigation starts by reconstructing the topology, the dependencies, and the blast radius from raw signals. With Causely's MCP tool feeding the agent a live causal graph, we expected to see the reconstruction work collapse: fewer tool calls, fewer tokens, faster answers, and more accurate root-cause diagnoses. The benchmark was designed to test that hypothesis across four agent configurations and two scenarios.

What was the benchmark set up?

We ran 72 runs across Coding and SRE agent setups in a controlled Kubernetes environment. For coding agents, we evaluated Claude Code (using Sonnet) and Codex (using GPT-5.4-mini), and HolmesGPT with Gemini Pro 3 and Claude Sonnet backends as the AI SRE agent. We evaluated these AI agents with and without Causely, using a K8s-deployed application (i.e.,  OpenTelemetry Astronomy Shop).

We evaluated common SRE queries under an active-incident scenario (a multi-service impacted fault) and a healthy-baseline scenario (the same application without any injected defects). For each experiment run, we posed a common query and measured the total elapsed time for the AI agent to answer the query, the number of tool/MCP calls required, and the total processed tokens. We then evaluated our experiment data to understand both the baseline behavior of AI agents and the value of Causely’s causal intelligence layer. 

What did we find?

Our experiments found that access to Causely materially improved AI agent performance across time-to-correct-diagnosis, token consumption, tool-call volume, cost per correct diagnosis, and root-cause accuracy. We observed consistent improvements across both Coding and SRE agents, indicating that the gain is structural rather than specific to any model or framework.

Specifically, we found:

• 63% reduction in time-to-diagnosis. Causely maintains a real-time representation of the environment topology and active symptoms, removing the serial dependency between the CLI and MCP tool calls that agents would otherwise use to reconstruct that context on every query.
• 60% reduction in token consumption and 4.8x compression in tool calls per investigation. A single Causely query returns a pre-computed slice of causal environment state (topology, active root causes, blast radius) that would otherwise require five to ten narrower tool calls to assemble from raw telemetry.
• 62% reduction in cost per correct diagnosis. The combined reduction in token volume and in wrong answers shifts cost from a reconstruction-dominated profile, which scales with the difficulty of the search, to a query-dominated profile that is approximately constant across scenarios.
• Accuracy on root-cause diagnosis rose from 75% to 100%. Causely’s causality graph encodes cross-service propagation rules, which let the agent identify the originating defect rather than chase downstream symptoms.

We also observed that baseline AI agents consumed more time and tokens investigating a healthy cluster than an active fault, by up to 7.2x. Raw telemetry provides no principled stop condition for an agent asked whether something is wrong, so often the agent continued probing until normal variation was presented as a hallucinated incident. With Causely, hallucinated-incident rates fell accordingly:

• HolmesGPT (Claude Sonnet): 67% to 0%
• Codex: 67% to 33%

What does Causely change for SRE agents?

Causely improves SRE agents by providing semantically rich, causality-contextualized data via its MCP tools. A single Causely query returns the current topology, the active root causes, the symptoms each root cause produces, and the blast radius across dependent services.

The agent replaces five to ten primitive MCP calls with one or two dense Causely calls, resulting in tool-call footprints compressed by up to 6.1x. Diagnostic accuracy improves because Causely's causality graph encodes how failures propagate across services, so the agent can trace a symptom back to the originating defect rather than investigating each affected service independently and guessing at the relationship. For teams building an AI SRE, Causely is the layer the agent would otherwise be trying to reconstruct on every invocation, less reliably and at much higher cost.

What does Causely change for Coding agents?

Coding agents like Claude Code and Codex investigate environments through free-form shell access: kubectl, curl, grep, and ad-hoc scripts against whatever CLIs happen to be installed. At baseline, they already achieve high diagnostic accuracy for active faults, so the challenge is often not accuracy. It is the cost of getting there. Every investigation starts from scratch, probing the cluster one command at a time, absorbing mistyped commands and missing binaries along the way, and on a healthy cluster, the agent keeps probing because nothing in the raw shell output tells it when to stop. Causely preserves accuracy while reducing token consumption by 54% on average, compressing tool calls by 4.2x, and eliminating failed tool calls.

Causely improves coding agents by giving them a direct, structured answer to questions the shell can only answer indirectly. A single MCP call returns the current topology, active root causes, and blast radius, which removes the need to reconstruct that picture from kubectl, logs, and metrics on every invocation. Absence of fault is an explicit response rather than something the agent must infer from the absence of evidence, which is what collapses the runaway token consumption on healthy clusters. For teams pointing coding agents at production, Causely removes the reconstruction tax without touching the model's reasoning capability. 

What should you do next?

The benchmark results show the same pattern across Coding agents and SRE agents: AI agents do not fail in reliability workflows because they lack telemetry. They are inefficient and inaccurate due to a lack of access to structured environment data and a causal and semantic representation of the environment and its entities.

Causely provides a semantically and causality structured interpretation of the raw telemetry as a coherent environment state. As enterprise AI shifts to per-token billing, access to this kind of grounded environment state ceases to be a quality-of-reasoning improvement and becomes an economic precondition for running agent-driven reliability at scale.

Read the full benchmark methodology and per-configuration results at causely.ai/product/benchmark

If you're building an SRE or coding agent and hitting the reconstruction tax we measured, talk to our team. We'll walk through your agent's current tool-call pattern and where causal context would compress it.

Every SRE team has heard the pitch by now: AI agents that monitor your systems, diagnose incidents, and remediate problems autonomously — no pager, no runbook, no 1 am escalation. The vision is compelling. The reality, for most organizations, is an agent that reads dashboards faster than a human and then hands the problem back to an engineer.

This isn't as much a failure of AI's promise as much as it is a failure of context.

The industry has converged on a useful framework for thinking about AI SRE maturity: a progression from Read-Only (the agent observes and explains) through Advised (it recommends) and Approved (it acts with human sign-off) to Autonomous (it executes bounded remediations without waiting for a human). The widely-held view is that enterprises rarely skip stages; the curve is governed by trust, observability quality, and risk posture.

That framing is accurate, but it undersells the most solvable part of the equation. Trust and risk posture are cultural and organizational. Observability quality is a technical problem, and it's the one most teams aren't addressing before they build their AI layer on top.

The Read-Only Trap

Most AI SRE agents today operate in Read-Only or, at best, Advised mode. Teams are using LLMs to summarize alerts, correlate metrics, and produce incident summaries that otherwise require hours of investigation. That's genuinely useful, but it's also about as far as most organizations get, and they often chalk up the stall to "we're not ready to let AI act autonomously" or "our engineers don't trust it yet."

What they rarely examine is why the recommendations aren't trustworthy. The answer is almost always the same: the agent is working from raw telemetry.

Raw telemetry — metrics, logs, traces — tells you what is happening at any given point in time across individual services. It does not tell you why. When a latency spike ripples through five services simultaneously, the telemetry shows you five sick services. It does not show you that four of them are sick because of the fifth.

Feeding that data to an LLM and asking it to recommend a remediation produces exactly what you'd expect: a confident-sounding guess. Or, worse, a sorted list of guesses to sort through. And if your AI SRE agent is guessing, the correct human response is not to approve the action. You're stuck at Read-Only, not because your team isn't ready for autonomous operations, but because your agent is not giving you any reason to trust it.

What "Observability Quality" Actually Means

When the maturity model says the progression is governed by observability quality, it points to something deeper than data completeness. Having more logs, higher-resolution metrics, or full distributed tracing doesn't necessarily close the gap. Teams at large organizations have exhaustive telemetry and still spend hours in incident bridges arguing about what caused what.

The gap is causal clarity.

There's a difference between knowing that service A is responding slowly and service B's error rate just spiked — which telemetry gives you — and knowing that service A's slowness caused service B's errors through a dependency path, and that the root cause of both is a resource contention issue three layers down in a shared infrastructure component. The first kind of knowledge lets an agent describe the incident. The second kind lets an agent act on it with defensible confidence.

Consider a failure pattern that comes up often in practice: a certificate expiration in a shared dependency triggers cascading application errors across multiple services. Traditional monitoring surfaces the application failures loudly. The expiring certificate — the actual cause — sits quietly in a dependency that nobody thought to watch. Engineers spend hours chasing error rates in services that are innocent bystanders, because the telemetry they have doesn't encode the relationship between those services and the thing that broke.

The point isn't that certificate expirations are a hard problem to fix. It's that symptom-level observability systematically leads investigators — human or AI — to the wrong place.

The Telemetry Trap

There's a reason AI agents operating on raw telemetry tend to get stuck reasoning about the most alarming service rather than the root cause. Correlation is cheap; causation is expensive. When an LLM is given a flood of metrics from a degraded system, the signals that stand out are the loudest ones, which are usually the services closest to the user experiencing the failure, not the one that initiated it.

This matters enormously when you're trying to move up the maturity curve. An agent in Advised mode recommending "restart service B" when the actual problem is a saturated database connection pool upstream isn't moving you toward autonomy; it's moving you toward an incident that gets worse after an ill-advised remediation. Engineers learn fast that approving those recommendations is risky. Trust erodes, and the agent stays in a dashboard-summarizing role indefinitely.

The problem compounds in modern microservice architectures, where a single infrastructure failure can manifest as degraded latency, increased error rates, and elevated resource consumption across dozens of services simultaneously. The blast radius is wide, the telemetry is overwhelming, and a pattern-matching approach — whether human or AI — will chase symptoms in the wrong order.

Causal Context Changes the Calculus

The alternative is to give your AI agent something it can actually act on: a structured model of how your services interact and fail, not a stream of data points to pattern-match against.

When an incident begins, an agent with access to a causal model doesn't have to correlate its way to a hypothesis. It can identify the single root cause that explains all the observed symptoms — because the model encodes the dependency structure that makes certain failure paths deterministic. It can tell you not just that service A is the root cause, but which other services are at risk of degradation (blast radius), which team owns the affected component (owner), and whether this failure path has been seen before.

That's the kind of context that changes what a human approver has to evaluate. Instead of "the AI thinks maybe service B is the problem," you get "the causal model identified a saturated connection pool in the payments-db component as the root cause; it is impacting services X and Y, and service Z is in the blast radius but operating normally; the recommended action is to increase the pool limit or, alternatively, scale the downstream consumer." An autonomous agent with appropriate guardrails can now be trusted to take action.

This is why the path from Read-Only to Autonomous isn't primarily about the sophistication of your AI model or the maturity of your team's risk posture. It's about whether the agent has access not only to information, but to knowledge that's trustworthy enough to act on.

The Practical Path Forward

None of this requires replacing your observability stack. The telemetry you're already collecting with legacy APM tools, OpenTelemetry, or Prometheus is a necessary input. It feeds the detection layer that surfaces anomalies and triggers investigation. What changes is what sits between that telemetry and your AI agent: a causal reasoning layer that transforms noisy signals into structured, explainable system knowledge.

That context is continuously maintained by the system and delivered via MCP the moment an agent needs it. No broad environment scan, no waiting for context to be accumulated, no wasted tokens. In a benchmark study we'll soon share publicly, causal context cut an agent's average token consumption by 48% and mean query time by 63%.

When AI agents receive that context — root cause, symptoms, blast radius, etc. — they can do something far more useful than summarize an alert. They can reason about the system the way a team of senior engineers and architects with deep institutional knowledge would, but at an unmatched speed and scale.

That's what moving up the maturity curve actually requires. Not more data, and not braver approval workflows. A layer between the noise and the agent that turns what's happening into why it's happening, and what to do about it. Agents that stop guessing, burn fewer tokens, and act proactively.

Most AI SRE deployments will stay stuck on Read-Only until teams address this gap. The good news is that it's addressable.

Build ops agents you can trust at scale.

Causely's MCP server gives your agents deterministic causal context so they stop guessing, burn fewer tokens, and act proactively.

The goal: deliver a modern loan experience at speed.

The constraint: zero tolerance for backend blind spots.

The outcome: a high-stakes launch, executed with confidence.

A Fintech Innovator Built for Speed and Resilience

Humm Group isn’t your typical consumer lender. Headquartered in Australia and operating across multiple international markets, Humm Group differentiates by doing the hard things fast. From point-of-sale Buy Now Pay Later (BNPL) to commercial lending, their products are designed for immediacy - instant credit decisions, seamless merchant onboarding, rapid deployment of new features.

That speed is powered by serious technical investment. Humm Groups’s technology team has re-platformed its stack onto AWS, using EKS, RDS, Control Tower, and GuardDuty to enable secure, scalable, and globally extensible systems. Legacy Solaris environments have been replaced with containerized, modern infrastructure. Products have been rationalized from dozens to six core offerings, simplifying complexity while improving customer and merchant experiences.

Whether it's same-day underwriting for brokers or revamping a mobile feature with 400% adoption growth, technology is at the center of how Humm Group wins. But when launching an entirely new consumer product under a new brand, even the most tech-savvy teams need a way to know, with certainty, that everything is working as expected.

The Challenge: A High-Stakes Launch Without Blind Spots

With the introduction of Humm, Humm Group was replacing its legacy offering with a cleaner, more flexible consumer finance experience. But beneath that simplicity was a complex service mesh, spanning payment systems, customer identity, underwriting engines, and legacy integrations.

For Geoff Walsh, Head of DevOps, Infrastructure and Cloud, confidence didn’t just mean uptime or latency, it meant trust.

“It’s not just about performance dashboards. What I really care about is knowing our systems aren’t doing something terrible in the background. That’s the value. That we’re not having problems we can’t see.”

— Geoff Walsh DevOps, Infrastructure & Cloud, Humm Group

As traffic ramped up on their new services and customers began applying for loans in real time, the team needed assurance that nothing was breaking behind the scenes - no bottlenecks introduced under load, no regressions across asynchronous dependencies, no misconfigurations escaping detection.

The Solution: Causely, for Confidence at Launch

Causely gave Humm Group’s engineers confidence at every stage of their rollout. Instead of combing through logs, traces, and dashboards, the platform continuously traced cause-and-effect relationships across their distributed services—automatically mapping dependencies and proactively monitoring for cascading failures or behavior changes, with no configuration or code changes required.

That confidence wasn’t built overnight. Before go-live, Causely was running in development, staging, and UAT, giving the team a deep understanding of how the system behaved under realistic conditions. It didn’t just catch issues. It confirmed that nothing was silently going wrong. So, when launch day came, the team wasn’t hoping things would work. They knew they would.

“Causely gave us peace of mind during a high-impact launch. We weren’t guessing. We knew services were running properly as intended.”

— Geoff Walsh, Humm Group

And if something did break—whether it was a sudden queue backlog, a degraded API, or a misfiring release—Causely would instantly surface the true root cause and its downstream impact. No digging. No noise. Just answers.

Why It Matters

For companies like Humm Group that differentiate through digital agility, launches can’t be slowed by uncertainty. But speed only matters when you can trust what’s running behind it, and that’s exactly what Causely delivered.

By continuously mapping service dependencies and analyzing real-time telemetry against known failure patterns, Causely pinpoints the true root cause of issues as they emerge, not after they’ve cascaded into user-visible problems. There’s no waiting on dashboards, no stitching together logs and traces, just immediate clarity when it matters most.

With causal intelligence embedded into its cloud native stack, Humm Group turned a high-stakes rollout into a confident release. Teams stayed focused on building the future, not troubleshooting the past.

Want Confidence in Your Next Launch?

Learn how Causely can help you validate performance, prevent downtime, and ensure a clean go-live, even in the most complex cloud native environments.

Did the user succeed?

That question often gets neglected. Not because teams don’t care, but because most reliability management is organized around services, while the user experiences transactions.

Reliability Is Felt in Transactions, Not Services

Consider an end-user experience like placing an order in an application. From the user’s perspective, this is one thing they are trying to accomplish. They add an item, confirm, and expect the order to go through. For them, the outcome is binary: the order worked, or it didn’t.

From a system perspective, that same experience fans out across many services. Inventory and pricing services check availability, payment authorization happens synchronously, a machine-learning-based fraud detection system evaluates the order in real time, and downstream fulfillment, recommendation updates, and notification steps continue asynchronously.

Each service might look “fine” when viewed in isolation. Latency could be slightly elevated in one place; retries could be masking failures in another, and asynchronous steps could be lagging, without any single service showing a clear red flag.

This is not a theoretical edge case. It’s a natural consequence of composing many independently reliable components into an end-to-end experience.

Reliability is managed at the service level, but it is felt at the level of transactions and flows.

Why Transaction-Level Reliability Doesn’t Collapse Into One Simple Metric

You can try to model this by defining reliability objectives for individual transactions, and in some cases that works well. A critical HTTP path on a frontend service can and should have a clear objective.

But systems rarely stop there. Real products are full of flows that are asynchronous, multi-step, or partially decoupled. Password reset workflows, AI-assisted report generation, onboarding sequences, background processing pipelines; these don’t fit neatly into a single request-response boundary.

In these cases, the user-facing “success” signal isn’t always available at the moment a request returns. You often only know whether the user truly succeeded after downstream work completes and state converges—sometimes minutes later, sometimes after multiple systems reconcile.

This is also where many SLO programs end up skewing toward internal service-level contracts: useful for ownership and operations, but not always capturing whether the user intent completed successfully end-to-end.

So the challenge isn’t “measure reliability.” It’s: measure reliability at the level where success is actually determined.

Product Ownership Doesn’t Map Cleanly to Services

There is another dimension where this becomes visible: products and features are rarely owned by a single service. They span multiple microservices, owned by different teams, evolving at different speeds. Developers and product managers tend to think in terms of features and user journeys, not in terms of which service is emitting which metric.

In a DevOps model, that creates friction. The people responsible for improving a product’s reliability need to understand how changes affect end-to-end behavior. Service-level views are necessary for fixing issues, but they are not sufficient for reasoning how reliable a feature or product feels to users over time.

And this is where a lot of reliability programs stall: they’re excellent at service health and incident response, but weaker at answering:

• Which user-facing flows are degrading?
• Which products are accumulating reliability risk?
• What is reliability trending like from the user’s perspective?
• Where are we “green” locally but failing end-to-end?

Consider a subscription upgrade flow initiated through a support chatbot. A customer messages: “Upgrade me to Pro”. The bot confirms success immediately.

Behind the scenes, that single interaction fans out across a chain of services, agents and actions: billing is updated, an entitlements agent validates the new plan and pushes updated permissions, usage limits refresh, and internal automation agents orchestrate background jobs reconcile state across systems.

No single service is down. The agents are executing as designed. Everything looks acceptable in isolation.

But minutes or hours later, the customer hits a limit they shouldn’t have, or a feature remains locked. Support tickets spike. Revenue recognition is delayed. Trust erodes.

From a service view, everything looks acceptable. From a product view, the upgrade flow is unreliable.

Closing the Gap Requires Product- and Flow-Centric Views

Addressing that gap requires additional lenses. Reliability systems need ways to look at signals through transaction-centric and product-centric views. Views that reflect how users experience the system and how teams build products across services, not only how services behave in isolation.

Services remain the building blocks. Service-level discipline remains essential. But to understand what users are actually feeling, and how reliable a product truly is, reliability needs to be observable at the level where experience happens: across transactions, flows, and products composed from many services working together.

We’ve been working on addressing this gap in our own product. We are exploring how reliability systems can make transaction- and product-level views first-class, while strengthening the service foundations teams rely on today. We’ll share more about that soon.

In the meantime, we’re curious whether this resonates: Does this match what you’re seeing in your own systems, or do you think service level reliability already cover more of this than we’ve described?

Let us know! We love to hear and read what you think: reach out to us via email or discuss with us on LinkedIn!

That delay is the Page-to-Understanding Gap. It is the translation cost of turning a page into an explicit explanation of system behavior. This is why MTTR improvements plateau even in teams with mature alerting and observability. 

Engineers feel this cost most acutely during incidents today, but it is also what prevents alerts from being usable inputs to automated response workflows.

What comes next is closing that gap. That means grounding alerts directly in system behavior and causal context, so investigation starts with understanding instead of translation. 

The problem alerts create during investigation 

When an alert fires, the first requirement is not diagnosis. It is orientation. 

Engineers need to know what kind of behavior is happening, whether multiple alerts point to the same underlying issue, and where to focus first. Alerts do not answer those questions. They were never designed to. 

Most alerting systems encode thresholds, burn rates, or proxy signals that usually correlate with impact. They reflect past incidents and operational heuristics, not real-time system behavior. That is why the same alert can represent very different failure modes, and why many different alerts can describe the same underlying degradation. 

As a result, the first minutes of an incident are spent translating alert language into a mental model of the system. Engineers jump between dashboards, traces, and logs to figure out what the alert actually means right now. Slack fills with partial hypotheses. Context fragments before anyone agrees on what problem they are solving. 

This translation work is the gap. It is where time is lost and coordination breaks down. 

The missing bridge between alerts and understanding 

Alerts do carry some intent. They are not random noise. Each alert exists because someone believed a certain condition mattered. 

What has been missing is an explicit bridge between that alert intent and an explanation of system behavior that can drive consistent action. Without that bridge, alerts remain isolated signals. Engineers are left to do the mapping themselves under pressure. 

Closing the Page-to-Understanding Gap requires making that mapping explicit. An alert needs to land inside an explanation, not start a scavenger hunt for one. 

What changed in Causely 

Our recent release adds the missing bridge. 

Causely already ingests alerts from tools teams rely on, including Alertmanager, Prometheus or Mimir, incident.io, and Datadog. What’s new is that the relationship between those alerts and Causely’s symptom and causal model is now explicit and visible. 

Each ingested alert is mapped to the symptom it represents and shown directly in the context of the inferred root cause. 

Instead of treating alerts as separate problems, Causely treats them as evidence of system behavior. Multiple alerts that describe the same behavior collapse into a single symptom. Those symptoms are then connected through the causal model to the change, dependency, or resource actually driving the issue. 

Alerts are no longer just timestamps and labels. They become part of a coherent system story. 

You can also view alerts over time and see which symptoms and root causes they mapped to. This makes alert behavior visible and understandable, rather than noisy. 

How this changes the first minutes of an incident 

For the on-call engineer, the workflow changes immediately. 

When a page fires, the alert can be pasted directly into Ask Causely. Instead of starting with dashboards, the engineer sees which symptom the alert maps to and whether there is an active root cause. The first question shifts from what to check to whether action is required now. 

The same mapping is also available as structured context for automated workflows, so alerts no longer require manual interpretation to become actionable.

For teams dealing with alert noise, the change is cumulative. Over time, they can see which alerts consistently map to the same symptoms and causes. This builds trust that important behavior is covered and makes it easier to reason about which alerts are redundant versus genuinely distinct. 

For reliability leads, gaps become visible. Alerts that do not map cleanly to existing symptoms stand out. Those gaps are no longer discovered mid-incident. They become clear opportunities to extend coverage where it actually matters. 

From paging interface to investigation entry point 

Alerts still wake people up. That does not change. 

What changes is where investigation starts. Instead of beginning with translation and hypothesis building, teams start with an explanation that already accounts for how alerts relate to each other and to the system. 

Over time, this shifts behavior. Engineers stop debating which alert is primary. War rooms converge faster on a shared narrative. Investigation energy moves from interpretation to containment. More importantly, alerts become explicit, interpretable signals that can drive response consistently, whether the next step is taken by a human or a system.

Causely does not replace alerting. It turns alerts into inputs to understanding. 

Closing the gap 

Alerts are not the investigation. They never were. 

The cost teams pay today is not because they lack alerts or data, but because understanding arrives too late. By explicitly mapping alerts to symptoms and root causes, Causely closes the Page-to-Understanding Gap where most incidents stall, turning alerts from pages into grounded inputs for understanding and action.

Know what to chase when everything breaks. 

Instead of treating traces as a data stream we might analyze someday, we should be opinionated about what matters at the moment of decision. As we argued in The Signal in the Storm, raw telemetry only becomes useful when we extract meaningful patterns.

In this guide, you’ll build a repeatable workflow that turns OpenTelemetry database spans into span-derived metrics you can dashboard and alert on—so you can identify what’s slow, what matters most, and what just regressed.

We’ll make this concrete with slow SQL queries, serving two use cases:

• Optimization: Which queries yield the most value if made faster, weighted by traffic?
• Incident response: Which queries are behaving abnormally right now?

We’ll build a lab where your app emits OpenTelemetry traces, and we distill those into actionable metrics, starting with simple slow query detection, then adding traffic-weighted impact, and finally anomaly detection.

What Makes a Query Slow?

“Slow” isn’t a single problem. It’s a symptom with fundamentally different causes. A 50ms query might be fine for a reporting dashboard but catastrophic for checkout. As High Performance MySQL emphasizes, understanding why a query is slow determines how to fix it. Here are the most common problems that may cause slow queries:

Excessive Work

The database does more than necessary—typically full table scans due to missing or unusable indexes. Without an index on customer_id, a simple SELECT * FROM orders WHERE customer_id = $1 grows from 20ms at 10K rows to minutes at 10M rows. The query didn’t change; the data volume did. See Use The Index, Luke! for the fundamentals.

Aggregations and joins compound this. Even indexed queries can explode when the planner misjudges cardinality and chooses the wrong join strategy.

Resource Contention

Perfectly optimized queries can be slow when waiting for resources. Lock contention blocks queries until other transactions release rows. Connection pool exhaustion adds latency before the query even starts. A query spending 95% of its time waiting for locks won’t be fixed by query optimization—it needs transaction redesign.

Environmental Pressure

CPU saturation, I/O bottlenecks, and memory pressure can slow any query. The same SQL with the same plan performs completely differently under resource contention.

Plan Regressions

Performance degrades when execution plans change—even with identical queries and data. Parameter-sensitive plans optimize for one set of values but fail for others. Stale statistics after bulk loads cause the planner to choose terrible strategies. The PostgreSQL Performance Tips documentation covers how to catch these regressions.

Pathological Patterns

Some slowness doesn’t appear in slow query logs. The N+1 problem executes 100 fast queries (2ms each) sequentially, adding 200ms latency plus network overhead. No individual query is “slow,” but the pattern is catastrophic.

The classic workflow: DB-native tooling + manual triage

Databases ship with excellent diagnostic tools: slow query logs, query stores like PostgreSQL’s pg_stat_statements, and plan inspection with EXPLAIN. These tell you what’s expensive inside the database.

What they don’t provide is context. Which service triggered the slow query? Is it user-facing or background work? Does it correlate with the latency spike you’re investigating? You’re left with a list of slow queries and no signal about which ones matter most.

Typically, someone bridges this gap manually: a developer notices a slow endpoint, brings the query to a DBA, and they optimize it together. This works, but that manual linking is exactly what we can automate.

Bringing Context to Slow Queries

Database tools tell you what is slow, but not why it matters. When you find a slow query in your logs, you’re missing critical context: Which service triggered it? Is it user-facing or background work? Does it correlate with the latency spike you’re investigating?

Distributed traces provide this context. Each database span is embedded in a request context—it knows which service, endpoint, and user triggered it.

Instead of correlating database logs and traces after the fact, we analyze slow queries directly from traces with all the application context built in.

The Building Blocks

Now that we understand the philosophy and the value of context-rich traces, let’s look at the building blocks we’ll use to implement slow query analysis.

The Observability Stack

For our lab, we use the OpenTelemetry Collector paired with docker-otel-lgtm—a pre-packaged stack from Grafana that bundles Loki, Grafana, Tempo, and Mimir in a single container. This gives us a complete observability environment with minimal setup.

The Application

Our sample application is a simple Go-based “Album API” that serves music album data from PostgreSQL. It’s intentionally designed to produce the kind of intermittent slow queries that are common in production. They services use otelsql to instrument database calls, emitting spans with the stable OpenTelemetry database semantic conventions.

The Dashboards

We’ll build three dashboards, each adding a layer of insight:

1. A simple view of the queries by duration
2. Queries weighted by traffic to surface optimization opportunities
3. Anomaly detection to identify queries deviating from their normal behavior

Lab Setup

Let’s put the theory into practice. We’ll clone a sample application, start the observability stack, and explore three progressively more sophisticated approaches to slow query analysis. All you need is Docker installed.

Clone and Run

git clone https://github.com/causely-oss/slow-query-lab
cd slow-query-lab
docker-compose up -d

Once running, open Grafana at http://localhost:3001—that’s where we’ll explore our dashboards.

Queries by Duration

The first dashboard takes the most direct approach: query Tempo for database spans and aggregate them to find queries that take the longest time. This is what you’d naturally build when you first start exploring traces for slow query analysis.

What It Shows

The Slow SQL - By Duration dashboard queries traces directly using TraceQL:

{ span.db.system != "" } | select(span.db.query.text, span.db.statement)

This finds all spans with database attributes, then uses Grafana transformations to:

1. Group by root operation (API endpoint) and SQL statement
2. Aggregate duration into mean, max, and count
3. Sort by average duration (slowest first)

The result is a table showing your slowest queries, which endpoints triggered them, and how often they occur.

What’s Good About This

This approach gives you immediate visibility into queries with full application context:

• You can see exactly which SQL statements are taking the most time
• You know which API endpoints trigger them
• You have the count to understand frequency
• You can click through to individual traces for debugging

It’s a first improvement over raw database logs because you’re already seeing the application context that makes slow queries actionable.

The Limitation

Here’s the problem: sorting by average duration doesn’t tell you which queries matter most.

Consider two queries:

The complex report is “slower” by average duration, so it appears first. But the search query, despite being faster on average, runs 2,000 times more often. Its aggregate impact on your users is far greater.

This dashboard tells you what’s slow, but not what’s impactful. For that, we need to consider traffic volume.

Traffic-Weighted Impact Analysis

The second dashboard addresses this limitation by introducing an impact score: the product of average duration and call count.

What It Shows

The Slow SQL - Traffic Weighted dashboard uses the same TraceQL query but adds a calculated field:

Impact = Avg Duration × Count

This simple formula captures a key insight: a moderately slow query that runs thousands of times has more total impact than a very slow query that runs rarely. The dashboard sorts by impact score, surfacing the queries that matter most to your users.

The dashboard also adds:

• Service breakdown: See which service triggered each query
• Latency distribution: Visualize duration over time, not just averages
• Top queries by impact: A quick view of where to focus optimization efforts

What’s Good About This

Traffic-weighted impact gives you a much better prioritization signal for optimization work:

• High-volume, moderately-slow queries surface above rare-but-slow ones
• You can justify optimization work with concrete impact numbers
• The service and endpoint context helps you route issues to the right team

When someone asks “which slow queries should we optimize first?”, this dashboard gives you a defensible answer. It’s exactly what you need for planning performance improvements.

The Limitation

But this dashboard is for optimization, not incident response. Even with traffic-weighted impact, it can’t answer a critical question:

“What has changed?”

Suppose your search query has an impact score of 150,000. Is that normal? Is it higher than yesterday? Higher than last week? The dashboard shows you a snapshot of current state, but it has no concept of baseline.

This matters enormously during incidents. When latency spikes, you don’t just want to know “search queries are slow”—you want to know “search queries are slower than normal”. You need to distinguish between:

• A query that’s always been slow (known behavior, maybe acceptable)
• A query that just became slow (new problem, needs investigation)

Without a baseline, every slow query looks the same. You’re left manually comparing current values to your memory of what’s “normal,” or digging through historical data to establish context.

This is the gap that the third dashboard addresses.

Symptom Detection with Anomaly Baselines

Because of these limitations, the third dashboard changes our approach: instead of just querying traces, we distill metrics from spans and then apply anomaly detection to identify deviations from normal behavior.

The Setup

For this dashboard, we add the spanmetrics connector to the OpenTelemetry Collector. Here’s the relevant part of the collector configuration:

connectors:
  spanmetrics:
    dimensions:
      - name: db.system
        default: "unknown"
      - name: db.query.text
      - name: db.statement
      - name: db.name
        default: "unknown"
    exemplars:
      enabled: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [transform, batch]
      exporters: [spanmetrics, otlphttp/lgtm]
    
    metrics:
      receivers: [spanmetrics]
      processors: [batch]
      exporters: [otlphttp/lgtm]

The spanmetrics connector examines every database span and generates histogram metrics for query latency, labeled by:

• service_name: Which service made the query
• db_system: Database type (postgresql)
• db_query_text or db_statement: The SQL query
• db_name: Database name

These metrics are stored in Mimir (the Prometheus-compatible backend in docker-otel-lgtm), where we can apply PromQL-based anomaly detection.

Anomaly Detection with Adaptive Baselines

The sample app includes Prometheus recording rules from Grafana’s PromQL Anomaly Detection framework. These rules calculate:

• Baseline: A smoothed average of historical values (what’s “normal”)
• Upper band: Baseline + N standard deviations (upper threshold)
• Lower band: Baseline - N standard deviations (lower threshold)

When current values exceed the bands, we have an anomaly—a clear signal that something has changed.

What It Shows

The Slow SQL - Anomaly Detection dashboard displays:

1. Current latency plotted against the adaptive baseline bands
2. Anomaly indicators when latency exceeds normal bounds
3. Per-query breakdown so you can see which specific queries are anomalous

The key insight is the visual comparison: instead of just showing “p95 latency is 450ms”, it shows “p95 latency is 450ms, which is above the expected range of 200-350ms.”

Why This Is Better

This dashboard answers the question the previous one couldn’t: “What has changed?”

• A query that’s always slow (450ms baseline) won’t trigger anomalies when it runs at 450ms
• A query that’s normally fast (50ms baseline) will trigger anomalies if it suddenly runs at 200ms
• You get automatic context for what’s “normal” without maintaining manual thresholds

The anomaly detection acts as a symptom detector. It tells you: “This query is behaving differently than it usually does.” That’s a high-signal insight you can act on immediately.

From Metrics to Symptoms

Notice what we’ve achieved with this architecture:

1. Raw telemetry (traces) flows from the application
2. Distillation (spanmetrics connector) extracts metrics from those traces
3. Anomaly detection (Prometheus rules) identifies deviations from baseline
4. Symptoms (anomalous queries) surface for investigation

We went from thousands of trace spans to a handful of anomaly signals that tell you exactly where to look.

Taking This to Production

Metric Cardinality

Raw SQL in metric labels will explode your metrics backend—SELECT * FROM orders WHERE customer_id = 12345 becomes a separate series per customer. Use prepared statements (so instrumentation captures templates, not literals), normalize query text, or use aggregation_cardinality_limit in the spanmetrics connector.

Privacy

SQL may contain sensitive data. The Collector is the ideal place to redact: drop or transform sensitive attributes before shipping downstream. This aligns with distillation: sanitize at the edge, not centrally.

Anomaly Detection Baseline

Adaptive rules need 24-48 hours of data to establish baselines. Start with wider bands and tighten as confidence grows.

The Remaining Gap: From Symptoms to Root Causes

Even with anomaly detection, you’re still looking at symptoms. In real-world incident scenarios, especially in large environments, slow queries are just one of many symptoms that pop up at once. You’re not only trying to understand the cause of this one; you’re triaging a flood of alerts and correlating many symptoms to find the real root cause.

When the dashboard shows “search query latency spiked,” you know something changed. But you don’t know why it changed. The root cause might be:

• A missing index after a schema migration
• Query plan regression due to stale statistics
• Lock contention from a concurrent batch job
• Resource pressure from a noisy neighbor on the database host
• Upstream service degradation causing retry storms

Connecting the symptom (“search query is slow”) to the root cause (“index was dropped during last night’s migration”) requires causal reasoning—understanding the relationships between system components and tracing the chain of causation from effect back to cause.

You can absolutely do this reasoning yourself. Look at deployment timestamps, check for schema changes, investigate resource metrics, correlate with other symptoms. Good engineers do this every day.

But it’s manual, time-consuming, and doesn’t scale.

Going Beyond Symptoms with Causely

This is where Causely comes in: Causely extracts slow queries (and other symptoms) as distilled insights out of the box—the same pattern we implemented manually. But it goes further:

• Causal model: Slow queries are connected into a model of your system’s dependencies. You can see what they impact (which endpoints, which users) and what causes them (resource constraints, upstream failures, configuration changes).
• Root cause identification: Instead of showing you a list of symptoms to investigate, Causely traces causation chains to identify the underlying root cause. “Search queries are slow because the index was dropped.”
• Actionable recommendations: AskCausely helps you get to “what should we change?”—whether that’s adding an index, reverting a deployment, or addressing the upstream pressure that made the query slow in the first place.

The pattern we built in this post—distill, detect anomalies, surface symptoms—is the foundation. Causely is the natural next step: turning symptoms into root causes at scale.

Want to see how Causely connects your slow queries to their root causes? Try it yourself.

A queue starts to grow slowly. Nothing looks obviously broken at first. Publish calls are succeeding and consumers are still running, just not quite keeping up. Over time, messages begin to age out, and dead-letter queues start accumulating entries. Downstream services that depend on those messages begin to behave unpredictably. There are partial data, delayed processing and subtle customer-facing issues that are hard to tie back to a single event. By the time the impact is visible in latency or error rates elsewhere in the system, the original cause is buried several layers upstream and hours in the past. 

Teams do not miss these failures because they lack data. They miss them because the signals do not point clearly to the cause. 

Over the past several weeks, we’ve expanded Causely’s asynchronous and messaging queue capabilities to make these failures explicit, explainable, and actionable. This includes:  

• Expanding Causal Model for Amazon SNS and SQS and RabbitMQ 
• A new Producer Publish Rate Spike root cause   
• And adding Queue Size Growth and Dead-letter Queue as first class symptoms to our model  

Reliability Blind Spot in Messaging-Driven Architectures 

Asynchronous communication is foundational to how modern systems scale. The same advantages systems like Kafka and RabbitMQ provide, decoupling services and absorbing traffic spikes, also introduce new reliability challenges. 

The core issue is not that these systems fail quietly, but that cause and effect are separated. A producer can overload the system without returning errors. A broker can continue accepting traffic while consumers fall behind. By the time downstream symptoms appear, the triggering behavior has often already passed. 

For engineering managers and or those on the frontline of the on-call slack channel, this creates a familiar and frustrating dynamic. Reliability degrades without a clear trigger. Incident response turns into a debate about whether the producer or consumer is responsible. Teams chase anomalies across dashboards while backlogs continue to grow. By the time a decisive action is taken, the customer impact is already real. 

Why Traditional Observability Falls Short 

Metrics, logs, and traces are excellent at answering local questions. They tell you what a service is doing, how long an operation took, or how many messages are currently sitting in a queue. 

What they do not provide is causal understanding across asynchronous boundaries. 

In messaging-driven systems, cause and effect are separated in time and space. A spike in publish rate from one service may not create visible impact until hours later, in a different service, owned by a different team. A slow consumer may be the result of downstream backpressure rather than a defect in the consumer itself. Dead-letter queues tell you that messages failed, but not why the system reached that state. 

Without a causal model of how producers, exchanges, queues, and consumers interact, teams are forced to infer failures indirectly. That inference is slow, fragile, and heavily dependent on tribal knowledge. Under pressure, it leads to overcorrection, unnecessary rollbacks, and missed root causes. 

Expanding the Causal Model for Messaging Systems 

To close this gap, we have significantly expanded Causely’s causal model for asynchronous messaging systems. 

Rather than treating queues as opaque buffers, Causely now models messaging infrastructure the way it actually operates in production. Producers, exchanges, queues, and consumers are represented as distinct entities with explicit relationships and data flows. This applies across common technologies, including Amazon SQS, Amazon SNS, and RabbitMQ, whether used in simple queue mode or exchange-based pub/sub patterns. 

By modeling the topology directly, Causely can reason about how work enters the system, how it is routed, where it accumulates, and how pressure propagates across services. This makes it possible to explain failures that previously required intuition and guesswork. 

Making Queue Growth and Dead-Letter Failures First-Class Signals 

We have also expanded the causal model to treat queue size growth and dead-letter queue activity as first-class symptoms, not secondary indicators. 

This changes how asynchronous failures are diagnosed. Instead of surfacing queue metrics as passive signals, Causely reasons about them causally, linking backlog growth and dead-letter events directly to the producers, consumers, and operations involved. 

As a result, queue-related failures are no longer inferred indirectly from downstream latency or error spikes. The failure mode is explicit, explainable, and traceable to the point where intervention is most effective.  

A New Root Cause: Producer Publish Rate Spike 

One of the most common and least understood asynchronous failure modes is a sudden change in publish behavior. Causely now includes a dedicated root cause for this pattern: Producer Publish Rate Spike. 

This occurs when a service, HTTP path, or RPC method begins publishing messages at a significantly higher rate than normal. The increase may be triggered by a code change, a configuration update, or an unexpected shift in traffic patterns. Downstream queues absorb the initial surge, but consumers cannot keep up indefinitely. Queue depth grows, message age increases, and backpressure begins to affect the rest of the system. 

What makes this failure particularly dangerous is that the producer often looks healthy. Publish requests succeed, error rates remain low, and nothing appears obviously wrong at the source. Without causal reasoning, teams frequently blame consumers or infrastructure capacity, missing the true trigger entirely. 

Causely now detects this condition explicitly. It ties unexpected increases in publish rate to queue growth, consumer pressure, and downstream service degradation, making the failure both visible and explainable. 

What This Changes for Reliability Teams 

For teams responsible for revenue-critical services, these capabilities change how asynchronous failures are handled in practice. 

Instead of reacting after queues are saturated and customers are impacted, teams can see which producer initiated the failure, how pressure propagated through the messaging system, and where intervention will have the greatest effect. Slow consumers, misconfigured routing, and unexpected publish spikes are distinguished clearly rather than conflated into a single “queue issue.” 

This shortens incident response, reduces unnecessary mitigation, and eliminates the finger-pointing that often arises when failures span multiple teams. More importantly, it enables a proactive reliability posture in systems that are constantly changing. 

Asynchronous Reliability Without Guesswork 

Asynchronous architectures are essential for scale, but they demand a different approach to reliability than synchronous request paths. 

With its expanded messaging and asynchronous causal model, Causely provides deterministic, explainable reasoning over how data flows through your system. Teams do not need to stitch together dashboards to reconstruct timelines after the fact. They do not need to trust black-box AI summaries that cannot explain their conclusions. They no longer have to exhaustively eliminate possibilities to arrive at a root cause. 

Instead, they get clear answers to the questions that matter most: what is breaking, why it is breaking, and where to act first to protect reliability and revenue. 

If you’ve been on call, you’ve seen this movie.

This is also why plenty of “AI-powered observability” rollouts still don’t change the lived experience of on-call. Leadership expects response times to improve. On-call gets richer dashboards, smarter summaries, and more plausible explanations. To be fair, those do help with faster lookup and briefing in the room, but the social reality stays the same: alerts get silenced in PagerDuty, rules get tagged as “flappy,” and old pages keep firing long after anyone can justify what they were meant to protect. The problem isn’t effort. It’s that the page still doesn’t reliably collapse into shared understanding. Call it the Page-to-Understanding Gap: the time and coordination cost of turning a threshold into a system story.

Alerts are supposed to start an investigation. Too often, they start translation: what is the system doing right now? That translation slows containment, splinters context, and stretches customer impact. 

That decoding work is what incident response depends on, yet it’s rarely made explicit. It’s why MTTR gains often plateau even after teams invest heavily in monitoring and dashboards. 

Alerts are a paging interface, not a language of explanation 

Alerting is optimized for one job: interrupt a human at the right moment. 

So alerts are built from what’s easiest to express at scale: thresholds, proxy signals, and rules that “usually work.” They encode operational history, not system truth. 

That’s not a failure of alerting. It’s what alerting is for, but it also makes alerts a shaky foundation for understanding. They’re a simplified label over messy reality. When alerts aren’t grounded in clear definitions of “healthy” behavior, the signal loses meaning and teams stop trusting it. 

One alert name can mean multiple different realities 

Take a familiar page: “latency high,” or the modern equivalent: an SLO burn rate page. 

Burn rate fires on checkout. You assume checkout is slow, start in the service dashboard, see p99 up, then notice retries doubled. Meanwhile, Slack is already split: “DB” vs. “checkout.” Fifteen minutes later you realize a downstream dependency is brownouting and checkout is just drowning in retries. The giveaway is usually the shape: one hop shows rising timeouts and retry storms while upstream looks "healthy" until it saturates. The alert didn’t lie—it just didn’t tell you what you needed first. 

The page looks identical. The mechanism isn’t. 

So teams build muscle memory around “what usually causes this,” and it works—until the system changes just enough that it stops working. Scale and change are exactly what modern organizations optimize for, so the failure mode is guaranteed. When that happens, the alert doesn’t just wake you up. It points you in the wrong direction. 

Many alerts can describe the same underlying behavior 

The reverse problem happens just as often. A single degradation creates a cascade of pages across services: latency, errors, saturation, queueing, burn rate alarms. Each is technically “true,” but treating them as separate problems creates thrash. 

People split into parallel investigations, duplicate context gathering, and argue about which page is “the real one.” Context fragments across Slack threads and war rooms, ownership ping‑pongs, and escalations get noisy. By the time you agree which page is “primary,” you’ve already created a coordination incident. The outcome isn’t just wasted engineer time—it’s that nobody has one shared narrative everyone can repeat while impact is unfolding. The incident becomes less about understanding and more about sorting competing signals. This is how alert fatigue turns into incident fatigue. 

The silent gap: important behavior you don’t alert on 

The costly ones are the slow degradations that ship impact before they page. 

A dependency gets a little slower. Retries creep up. One critical route starts timing out for a slice of customers. Averages look fine. Thresholds don’t trip. You don’t notice until the page fires, or until customers do. 

It’s not that teams don’t care. It’s that these behaviors don’t fit neatly into alert rules: risk builds up, dependencies decay, partial impact hides in aggregates, and propagation only makes sense once you’ve traced it end to end. 

Most alerts only get defined after you’ve understood the behavior in the middle of an incident. The post-mortem produces a new rule and a brief feeling of closure. Then traffic shifts, dependencies evolve, and the next incident arrives with a different shape. You’re never done. 

So teams find these late, after impact is already underway, when time is most expensive. 

Why teams don’t switch investigation entry points 

When teams adopt a new investigation system, they often ask a simple question: “Does it match our alerts?” 

What they’re really asking is: “Can I trust this in the first 90 seconds?” Because in the first minute, the primary goal isn’t elegance, it’s not making it worse.

A system that generates more hypotheses doesn’t help if it can’t connect the page to what the system is doing in a way the on-call trusts. 

If the system describes an incident in a different language than the alert model engineers rely on, mismatches get interpreted as duplication, contradiction, or risk. The result is predictable: people consult it late, after they’ve already committed to a direction. 

Even correct insights arrive too late to change behavior. 

Why this problem is getting worse 

Systems are becoming more dynamic: more dependencies, faster deploys, and more integration points. Deploy frequency keeps climbing—and AI-assisted coding is only accelerating it—so the number of failure paths keeps growing. Meanwhile, alert fatigue is already high, and teams are hesitant to change workflows mid-incident. 

Better tooling can speed up lookup and correlation, but it can’t compensate for an alerting model that no longer maps cleanly to real system behavior. 

So the interpretation workload keeps rising. Every page demands more interpretation, more cross-checking, more manual stitching of symptoms into a coherent story. 

What’s actually broken 

Most organizations are operating with two different languages: the language of paging and the language of understanding. The persistent MTTR plateau is the Page-to-Understanding Gap between them.

Incidents start with the first, but the work happens in the second. 

A better way to think about alerts 

Alerts are not the investigation. They’re a notification that something is going sideways. 

The goal is not to tune thresholds until the noise feels tolerable. It’s to shorten the time from page to shared understanding: what behavior is emerging, what changed, what’s being impacted—and whether it matters to the business. 

Treating that translation as unwritten know‑how is not a workflow quirk. It’s a structural weakness. If your incident response starts with decoding alerts, you’re spending your best engineers on interpretation instead of containment. 

When these systems fail, the failure rarely shows up where it starts.

Teams often notice stale data, degraded behavior, or latency spikes elsewhere in the system. By the time those symptoms appear, the underlying problem has usually been present for some time.

In many real-world failures, two signals appear earlier: queue growth and dead-letter queues. They are widely monitored, but they are still widely misunderstood.

The common misunderstanding 

Queues are often treated as infrastructure components rather than behavioral signals. When a queue grows, it is attributed to load. When messages land in a DLQ, it is treated as a retry policy doing its job. Investigation tends to focus downstream, where symptoms are visible.

This framing obscures where asynchronous systems actually break. In many failures, message brokers continue to accept traffic normally. Producers succeed. Nothing looks obviously down. The problem is that consumers are no longer able to keep up reliably or consistently. That distinction matters.

Queue growth is not just volume 

Queue growth occurs when messages arrive faster than they can be processed successfully over time. This does not require a traffic spike. It can result from: 

• Consumers slowing due to code changes or resource pressure 
• Dependencies becoming latent or unreliable 
• Retry rates increasing 
• Backpressure failing to engage 
• Partition skew concentrating work unevenly 

In these cases, the broker behaves correctly. Messages are accepted. The queue grows quietly. What is accumulating is lag. A sustained backlog means work is no longer flowing through the system at the intended rate, even if no explicit failures are visible yet.  

Why this matters before anything looks broken 

Asynchronous systems are designed to absorb instability. Queues buffer mismatches. Retries smooth over failures. Backlogs delay visible impact. This is useful, but it also postpones feedback. As queues grow: 

• Processing time increases 
• Derived state falls behind 
• Downstream services operate on increasingly stale or incomplete data 

The transition from “degraded” to “broken” often appears sudden because the system has been accumulating lag for some time before any external threshold is crossed. 

Dead-letter queues signal a different failure 

Dead-letter queues exist to capture messages that cannot be processed successfully. Messages land there after repeated failures, timeouts, or deterministic errors. DLQs prevent infinite retries and protect the main pipeline. What they represent is not transient instability, but persistent processing failure under current system behavior. 

A non-empty DLQ means some class of messages cannot be handled as the system is currently operating. That incompatibility can come from: 

• Broken contracts between producers and consumers 
• Partial or skewed deployments 
• Schema drift 
• Unhandled edge cases 
• Dependencies that fail consistently rather than intermittently 

DLQs often grow alongside backlogs, but they can also appear independently. 

Why these problems are so common 

In real systems, producers and consumers evolve independently. Load shifts. Dependencies degrade. Retry behavior changes system dynamics in non-obvious ways. It is common for: 

• Brokers to continue accepting traffic 
• Queues to grow steadily 
• Consumers to fail intermittently or slow down 
• Processing failures to accumulate quietly 

Operationally, queues and DLQs sit between services. They rarely have clear ownership. They are easy to monitor superficially and hard to reason about in context. As a result, many teams only notice these issues once downstream behavior degrades.  

Queue growth and DLQs are related, but distinct 

Queue growth and DLQs are often discussed together, but they answer different questions. 

Queue growth asks: "Are messages flowing through the system fast enough?"

DLQs ask:  "Are some messages failing to be processed at all?"

In many incidents, sustained queue growth precedes DLQs. Consumers slow down, retries increase, and retry limits are eventually exceeded. In others, DLQs appear immediately due to deterministic processing failures, even while queue depth looks healthy. Treating one as a proxy for the other creates blind spots. 

The deeper diagnostic challenge 

Most teams diagnose asynchronous failures indirectly. They look at:

• Latency spikes 
• Error rates 
• Timeouts 
• User-visible symptoms 

Those signals matter, but they are downstream effects. Earlier and more precise signals exist inside the message pipeline itself: where messages are accepted, where they slow down, and where they fail to be processed reliably. When those signals are ignored or misinterpreted, teams spend time chasing symptoms rather than isolating where the workflow is actually breaking.  

A better way to think about queues 

Queues are not just buffers. Queue growth is not harmless backlog. Dead-letter queues are not operational exhaust. They are indicators of whether asynchronous workflows are functioning as intended or quietly degrading under real conditions. 

The goal is not to watch queue depth, instead, it’s to continuously understand flow: where work accumulates, why it accumulates, and which downstream interactions it impacts. 

Understanding them is not an optimization. It is foundational to operating reliable, event-driven systems. 

This limitation becomes especially visible in modern, hybrid architectures. Services span Kubernetes clusters, standalone EC2 instances, ECS tasks, and legacy infrastructure, all connected through real production traffic. Datadog can surface signals across these environments, but understanding how failures propagate across those boundaries remains a manual, error-prone exercise. The result is slower recovery, repeated incidents, and reduced confidence in change. 

With this expanded Datadog integration, Causely gives teams a unified, causal model of their entire application across Kubernetes and non-Kubernetes environments. This model that explains why services are impacted, not just where symptoms appear. 

From Observability Signals to System Understanding 

Many teams already rely on Datadog APM, infrastructure metrics, and monitors as the backbone of their observability stack. With Causely’s expanded support, those same Datadog signals can now be used to build a complete and accurate causal model of the system without changing existing instrumentation. 

Causely supports Datadog APM dual shipping, which allows trace data to be sent directly from the Datadog collector into Causely’s mediator. Teams continue using Datadog exactly as they do today, while Causely consumes the same traces for causal reasoning. This approach avoids additional agents, avoids data duplication, and does not introduce new egress costs. 

Just as importantly, Causely now supports services running outside Kubernetes and keeps causality intact across the hybrid boundary. By tagging Datadog APM traces with host identity metadata, Causely can stitch together services running on EC2 with those running inside Kubernetes clusters. What previously broke at environment boundaries becomes a single, end-to-end behavioral model of how the application actually runs in production. 

Datadog monitors can also be ingested directly into Causely and treated as symptoms rather than conclusions. Instead of reacting to alerts in isolation, Causely uses them as signals that inform its understanding of what is happening in the system and why. That’s how you get faster convergence, fewer false leads, and higher confidence in the fix. 

A Real-World Hybrid Application Scenario 

Consider a typical production application. Customer-facing APIs and frontend services run in a Kubernetes cluster. Background workers, billing services, or legacy processing jobs run on standalone EC2 instances. The application depends on shared infrastructure such as Postgres, Redis, and external APIs. Datadog is already deployed across all of it. 

Under normal conditions, everything appears healthy. Then, during a traffic spike, latency starts creeping up in one of the Kubernetes services. Shortly after, Datadog monitors begin firing for elevated error rates in downstream components. Engineers open dashboards, inspect traces, and try to correlate timelines across environments. The symptoms are visible, but the cause is not obvious. 

This is where Causely changes the workflow. 

Automatically Pinpointing the True Cause 

Using Datadog traces, infrastructure metadata, and monitor events, Causely continuously reconstructs end-to-end request paths and dependencies across Kubernetes and EC2. That continuity holds across environment boundaries, so you don’t have to manually stitch together “what talks to what” in the middle of an incident.  Instead of reacting to individual alerts, Causely continuously builds and maintains a behavioral model of the entire system. This model captures how services, infrastructure, and data flows interact, and how specific failure modes produce observable symptoms. 

Datadog APM traces provide the raw evidence of system behavior, including service interactions, request paths, and downstream dependencies. Datadog monitors are ingested and mapped as symptoms within Causely’s knowledge base. Together, these signals allow Causely to maintain an up-to-date causal model that explicitly links observed symptoms to the conditions and changes that produced them. 

Because this causal model is updated continuously, Causely can explain not just what is failing, but what changed first, how the impact propagated, and why specific services or endpoints are affected. The result is a precise, system-level explanation of performance degradation that teams can act on immediately. 

From Incident Response to Reliability Assurance 

This expanded Datadog integration is not just about faster root cause analysis during incidents. By continuously modeling system behavior, Causely enables teams to validate reliability before changes reach production, monitor how reliability evolves over time, and detect drift caused by infrastructure or configuration changes. 

Modern systems are hybrid by default, and reliability problems do not respect environment boundaries. To operate confidently at scale, teams need more than visibility. They need to understand how their systems behave and why failures occur. 

With expanded Datadog support across Kubernetes and EC2, Causely helps teams move from alert-driven firefighting to causal reliability engineering. The result is fewer war rooms, faster resolution, and the confidence to ship changes without fear. 

To learn more about using Causely with Datadog, explore the integration guide or reach out to see a unified service graph in action. 

In the first one, we said thanks to Grafana for donating Beyla and making it easier for teams to get to usable telemetry quickly. This time we want to zoom out to something that quietly runs under the hood at Causely every day: GitOps with FluxCD.

Causely is a member of the Cloud Native Computing Foundation (CNCF). That’s not just a logo on the website for us: our entire product and our own operations lean heavily on CNCF projects. We build on OpenTelemetry, we run on Kubernetes, and we dogfood our own reliability engine against that stack.

Another key piece in that puzzle is FluxCD. It is what takes “the desired state in git” and makes it true in our clusters, repeatedly. It’s the heartbeat behind our weekly releases.

Why Flux Helps

If you’re operating modern Kubernetes environments, you’ve probably felt this tension. On one hand, you want velocity: teams push changes constantly, services multiply, and configurations evolve all the time. On the other hand, every manual kubectl apply is a potential one-off change that no one can fully reconstruct later. Over time, clusters drift away from whatever was last written down as “how things should be,” and you are left relying on muscle memory and shell history.

Flux solves exactly that problem by turning git into the control surface and reacting to changes within seconds of a merge.

For us, that has very practical consequences.

• Releases are commits, not hand-crafted ceremonies.
• A new Causely version is rolled out by changing a tag or a value in git; Flux notices the change within seconds, reconciles the cluster, and either converges to the new state or loudly tells us why it couldn’t.
• Environments stay in sync because the same manifests back our test clusters, staging, and production. The differences between them are intentional and visible in overlays, not hidden in one-off fixes on a live production cluster.
• And drift becomes a signal, not a mystery: when the cluster does not match git, Flux shows it, which turns the familiar “what changed?” question into a quick investigation rather than a full-blown incident archaeology.

The net effect is that we can move quickly without giving up control. Our reliability engine depends on a stable substrate; Flux helps us keep it that way.

Best Practices We’ve Learned Using Flux

We didn’t get there on day one. It took a set of habits to turn FluxCD from a cool project into a core platform primitive.

1. Treat manifests like product code
2. Keep Kustomize overlays boring
3. Watch Flux like any other production controller
4. Make promotion a path, not an event

Treat manifests like product code

All of our Kubernetes manifests live in git. Not most of them, not “the important ones” – all of them. That sounds obvious, but it changes behavior in subtle ways.

• Reviews happen before things break, because a change to a HelmRelease or a Kustomize overlay goes through the same review process as a feature change.
• The commit history becomes an operational log: when we see a strange spike in errors, we can line it up against recent git changes, including configuration tweaks that would otherwise live only in someone’s bash history.
• Our issue tracker also stays connected to reality, because we reference issue numbers in commit messages, so the question “why did we change this setting?” always has a direct link back to the discussion that justified it.

In the early days, we still made the occasional manual change in production, usually in the name of speed. Those changes always came back to haunt us as confusing states that no one could fully explain. Once we committed to git as the only source of truth and forced ourselves to route every change through it, the platform became much more predictable.

Keep Kustomize overlays boring

We use Kustomize to manage environment-specific differences across test clusters, staging, production, and chaos environments. The rule we eventually settled on is simple: overlays describe differences, not alternative universes.

In practice, that means we maintain a clean base with shared resources such as namespaces, common HelmReleases, and shared configuration. On top of that base, we keep the environment overlays as thin as possible. They patch what truly needs to change, such as cluster names, resource limits, or a particular feature flag, rather than redefining whole stacks.

Whenever we tried to be clever with external references or overlays that diverged heavily from one another, troubleshooting became harder. Keeping overlays compact and predictable means we can scan a diff and understand at a glance what will change in a given cluster. Before committing, we render Kustomize configs locally as a quick sanity check that catches typos and misaligned paths before Flux has to complain about them.

Watch Flux like any other production controller

GitOps is not “set and forget.” Flux is a control loop running in production and, when it is unhappy, your platform will slowly drift.

We treat Flux like a critical controller. We watch reconciliation health and consider a stuck HelmRelease or Kustomization as important as any failing deployment. When Flux cannot talk to git, or when an apply keeps failing, that is something we alert on rather than something we notice days later in a dashboard. And when “nothing seems to be changing” in a cluster despite recent commits, Flux logs are one of the first places we look.

This mindset becomes even more important when GitOps extends beyond just core applications and starts to manage your observability stack, gateways, and even Causely itself.

Make promotion a path, not an event

Flux really shines when you treat deployments as a series of git-based promotions instead of isolated production pushes. A typical Causely release starts with a change landing in a test environment: we use clusters like test1 and test2 for this. We verify that the change behaves as expected there, including how it interacts with telemetry and Causely’s own reasoning about incidents. Once we are happy, we promote the same change to staging by updating the relevant overlay or values. Only after staging behaves as expected do we roll the change into production.

Alongside this path, we maintain dedicated chaos clusters, chaos1 and chaos2, where we deliberately break things to see how the system responds. Because everything flows through git, we can rehearse failure modes without fear of leaving behind strange manual fixes that only exist on one cluster. Keeping cluster-specific configuration isolated and well documented is what allows us to run realistic experiments in those chaos clusters without letting that complexity bleed into production.

Try Flux On Your Own

To really understand Flux, it helps to feel git driving your cluster. The smallest useful experiment is a git repository, a local kubernetes cluster, and Flux bootstrapped from that repository. The nice part is that flux bootstrap already does most of the heavy lifting: it creates the repository, installs the controllers, and wires everything together for you.

You can run the following guide on your laptop with kind.

Start by installing the Flux CLI. The easiest way is via the official install script; if you prefer Homebrew, apt, or other package managers, the Flux documentation lists those options as well in the Flux installation guide.

curl -s https://fluxcd.io/install.sh | sudo bash

Next, export your GitHub credentials so Flux can authenticate and create the repository for you. If you are logged in with the GitHub CLI (gh auth login), you can derive both the user name and token directly from it.

export GITHUB_USER="$(gh api user --jq '.login')"
export GITHUB_TOKEN="$(gh auth token)"

Now, create a local Kubernetes cluster and verify that Flux can run there.

kind create cluster --name flux-playground

When your cluster is ready, bootstrap Flux into it. This command will create a repository called flux-playground-gitops under your GitHub account, install Flux into the flux-system namespace, and configure it to track the ./clusters/flux-playground path in that repo.

flux bootstrap github \
  --owner=$GITHUB_USER \
  --repository=flux-playground-gitops \
  --branch=main \
  --path=./clusters/flux-playground \
  --personal

Clone the newly created repository to your machine and change into it.

gh repo clone flux-playground-gitops
cd flux-playground-gitops

You are now ready to define the OpenTelemetry demo as a git-managed workload by adding a manifest for the demo under clusters/flux-playground.

cat > clusters/flux-playground/oteldemo.yaml <<'EOF'
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: open-telemetry
  namespace: flux-system
spec:
  interval: 1m
  url: https://open-telemetry.github.io/opentelemetry-helm-charts---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: otel-demo
  namespace: flux-system
spec:
  interval: 1m
  chart:
    spec:
      chart: opentelemetry-demo
      sourceRef:
        kind: HelmRepository
        name: open-telemetry
        namespace: flux-system
  targetNamespace: otel-demo
  install:
    createNamespace: true
EOF

At this point your git repository fully describes both Flux itself and the OpenTelemetry demo that Flux will deploy.

Commit and push these changes so Flux can reconcile the new state.

git add .
git commit -m "Add OpenTelemetry demo via Flux"
git push origin main

Within seconds of the push, Flux will see the new revision, apply the changes, and start rolling out the OpenTelemetry demo. You can watch the pods come up.

watch kubectl get pods -n otel-demo

After a few minutes you will see the OpenTelemetry demo microservices starting in the otel-demo namespace.

You now have a real application being managed by Flux from git: the desired state lives in a repository, Flux reconciles it into the cluster within seconds of your merge, and you never had to use kubectl apply for the actual deployment.

Flux + Causely: GitOps All the Way Down 

If you followed the small lab above, you already have a local cluster, Flux installed, and the OpenTelemetry demo running under GitOps control. From there, adding Causely is just one more git-driven change. 

Conceptually, the flow is simple. You obtain a Causely access token, store it as a Kubernetes Secret, and add the Causely FluxCD manifests to the same git repository that Flux already manages. Git drives the rollout, Flux reconciles it into the cluster, the OpenTelemetry demo generates realistic behavior, and Causely explains what is going on. 

Our documentation has a full FluxCD installation guide with more background and variations. The example below is meant to be something you can copy and adapt directly from your existing flux-playground-gitops setup. 

First, retrieve your Causely access token from Causely and keep it handy. Then, create a namespace for Causely and a kubernetes secret with your token. The Causely FluxCD manifests expect a secret named causely-secrets and use Flux's native post-build substitution to inject CAUSELY_TOKEN into the HelmRelease, so the token never has to be committed to git: 

kubectl create namespace causely 
kubectl create secret generic causely-secrets \ 
  --from-literal=CAUSELY_TOKEN=your-actual-gateway-token-here \ 
  -n causely 

Next, from inside your GitOps repository, clone the public causely-deploy repository and copy the FluxCD manifests into your own cluster configuration. 

cd flux-playground-gitops 
git clone https://github.com/causely-oss/causely-deploy.git  
mkdir -p clusters/flux-playground/causely 
cp causely-deploy/kubernetes/fluxcd/causely/*.yaml clusters/flux-playground/causely/

With this setup, your git repository now contains everything Flux needs to deploy Causely, and you can commit and push these changes so Flux can reconcile the new state.

git add clusters/flux-playground 
git commit -m "Add Causely via Flux" 
git push origin main 

After the push, Flux notices the change, applies the new manifests, and starts deploying Causely into your cluster. You can watch it the same way you watched the OpenTelemetry demo roll out. 

flux get kustomizations -A 
kubectl get pods -n causely

 Once the Causely pods are healthy, you can return to the Causely portal, where the cluster you just configured will appear. Over time, topology fills in and, as issues arise, you will see root cause views associated with the services in your demo. 

At that point, you have a complete loop on a single laptop: Git drives change, Flux applies it, the OpenTelemetry demo generates behavior, and Causely explains what happens when things go wrong. If you want more variations, production-grade knobs, or to run this across multiple clusters, the FluxCD installation guide in our docs walks through additional options in detail. 

Closing: Thank You, Flux 

FluxCD is a great example of the kind of infrastructure we love in the CNCF ecosystem. It nudges teams toward good habits, turns the question “how did this get here?” into something with a clear, auditable answer, and helps keep complex Kubernetes estates boring and predictable.

As a CNCF member building on OpenTelemetry, Kubernetes, and the wider cloud-native stack, we are genuinely grateful for projects like Flux that quietly raise the floor for everyone.

So: thank you to the Flux maintainers and community for building and maintaining an engine that lets us practice what we preach about control, desired state, and autonomous reliability. If you are running kubernetes and still relying on manual deploys or one-off scripts, Flux is worth a serious look.  

And if you want to see what happens when you combine GitOps with causal reasoning, we are always happy to show you how Causely fits into that picture.

Book a demo today -> 

At the pace modern cloud-native systems move, reacting after symptoms appear is not enough. Teams need a reliability operating system that works continuously alongside their applications, maintains an up-to-date understanding of how everything fits together, and provides the context required for safe automation and proactive reliability. 

Why this recognition matters 

Modern cloud-native applications evolve constantly. Every deploy, configuration change, traffic surge, and infrastructure adjustment can reshape system behavior. The pace is so fast that even the best teams struggle to reason about what is happening and why. 

Traditional observability dashboards can show what happened after symptoms appear, and AI copilots can help speed up triage, but reacting after the fact isn’t good enough for business-critical applications. 

Teams need something that runs continuously alongside their systems, understands how everything fits together, and helps them see how changes will affect performance before they land in production. That’s the foundation for proactive reliability, not just faster incident response. 

What Gartner recognized about Causely 

Gartner recognized Causely for taking a fundamentally different approach to reliability in modern systems. Instead of reacting to symptoms after they spread, Causely maintains a live causality graph that reflects how services, dependencies, and performance constraints relate to one another as the environment evolves. By continuously analyzing telemetry, Causely identifies the underlying driver behind emerging changes in golden signals, even when failures cascade across multiple services, including the code change, configuration update, or operational event that first introduced risk. 

This continuous causal inference is what enables proactive reliability. Causely provides clear direction on where to focus and what action is most likely to reduce performance risk, long before issues escalate. The same causal model supports both pre-production and production, helping teams understand how behavior will shift during testing, rollout, and real-world load. 

Causely runs locally as a lightweight, containerized system and processes telemetry without exporting raw data. This eliminates the need for central pipelines, avoids sampling or data volume constraints, and gives teams high-fidelity insight that integrates directly into their engineering workflows through APIs, webhooks, and an MCP server. This structured context supports both human decisions and safe automation. 

For us, this recognition validates the direction we have been building toward. The future of reliability is proactive, predictive, and grounded in causal understanding. 

Who should care? 

Causely is designed for teams responsible for building and operating modern distributed systems. It gives engineering and operations organizations a clearer understanding of how system behavior changes over time and how those changes affect reliability. Whether preparing a release, managing growth, or diagnosing unexpected behavior, teams need deeper clarity to make confident decisions. Continuous causal inference provides that clarity. 

Looking ahead 

We’re grateful to Gartner for this recognition and excited about what it represents. Reliability is entering a new chapter. Systems are more dynamic, AI workloads are growing, and teams need deeper clarity to keep everything running smoothly. 

Causely’s mission is to provide that reliability. Continuous causal inference helps teams prevent issues before they escalate, support high-velocity engineering without sacrificing reliability, and give both humans and automation the context they need to act safely. 

We’re excited for what’s ahead and proud to help shape the future of reliable, intelligent, and resilient systems. 

Want to learn what this could look like for your organization? Get started with Causely today.  

Gartner subscribers can view the full report for more information. 

You take a breath. Maybe even consider sleeping. 

Then Slack lights up: 

“Did it work?” — CTO 

You stare at dashboards. Nothing’s red. But you still don’t actually know: 

• Did reliability improve, or quietly regress? 
• Did the change shift bottlenecks or introduce new stress points? 
• Are you now closer to the edge under peak load? 

In a 50 to 100+ microservice environment with dense service-to-service dependencies, even small regressions can cascade silently. And slowing down isn’t an option. Leadership needs faster delivery and fewer incidents. 

This is exactly why we built Reliability Delta. 

Reliability Delta: A Deterministic Answer to “Did This Change Make Things Better or Worse?”

Reliability Delta turns subjective guesswork (like manual diffing of dashboards, correlation hunts, “nobody is complaining” anecdotes) into clear, evidence-based reliability signals. 

It’s powered by Causely’s continuously updated understanding of your environment: 

Causality Mapping 

Causely builds a Bayesian network that models how issues propagate across services, enabling true cause-and-effect visibility. 

Attribute Dependency Graph 

A DAG of functional dependencies generated from live topology and Causely’s attribute models, highlighting how attributes influence one another. 

These models allow Causely to compare two snapshots—two releases, two load tests, or two moments in time—and determine whether system behavior: 

• Improved 
• Regressed 
• Or shifted in ways you need to investigate 

The result: deterministic signals engineers can trust.  

Use Cases for Reliability Delta 

1. Validate Every Release Instantly 

Know immediately whether your change introduced risk. 

Feature flags and canaries help, but they don’t guarantee safety. What matters is whether the system is behaving normally. 

Reliability Delta automatically surfaces: 

• Behavior changes isolated to a specific flag, tenant, or traffic segment 
• Downstream effects in pipelines, async jobs, and data flows 
• Subtle regressions that don’t trip alerts but violate known patterns of normal 

This isn’t “no alerts fired = good.” 

This is evidence-based release confidence.  

2. Understand Load Test Results Beyond Pass/Fail 

“With Causely’s Reliability Delta, we can quantify how each release behaves under identical load. It surfaces changes in bottlenecks, stress patterns, and causal relationships that traditional load tests miss. At our scale, having that level of confidence before shipping is critical.” 

Cade Moore, Performance Engineering Lead at Hard Rock Digital
 

Did this release push you closer to the breaking point? 

A load test passing doesn’t mean you’re safe. 

Reliability Delta shows: 

• How bottlenecks shifted compared to last time 
• Whether the same load now produces more stress 
• Early signs of fragility or shrinking performance margins 

It answers the question load tests never answer: 

“Are we drifting toward failure or away from it?” 

3. Detect Reliability Drift Over Time 

Systems naturally drift through config changes, dependency updates, scaling events, and organic load shifts. 

By capturing snapshots periodically, you can: 

• Spot slow-building risk 
• Track reliability trends 
• Validate that ongoing changes are improving SLO posture, not eroding it 

This moves teams from reactive firefighting to proactive reliability assurance. 

4. Validate Experiments with Confidence 

Know immediately whether your experiment improved or degraded system behavior. 

Teams frequently adjust timeouts, concurrency, sampling, queue behavior, or other system parameters, but these changes rarely trigger alerts, and standard dashboards make it hard to see their true impact. 

Reliability Delta lets you validate experiments with clear before-and-after evidence by automatically highlighting: 

• Shifts in bottlenecks or stress patterns across services 

• Degradations hidden behind “passing” performance metrics 

• Unexpected side effects in downstream dependencies 

• Whether the experiment made the system more resilient or more fragile 

This isn’t trial-and-error tuning. It is evidence-based experiment validation. 

Why Reliability Delta Matters for Modern Engineering Teams 

If you’re accountable for revenue-critical systems—measured by 99.9%+ SLOs, delivery pace, and incident reduction—you need more than observability dashboards. You need a deterministic framework for evaluating how change affects system behavior.  

Reliability Delta gives you: 

• Objective, repeatable comparisons between versions 
• Root-cause-aware analysis using causal models 
• Clear guardrails leadership can trust 
• Confidence to ship fast without risking SLOs or customer experience 

It transforms subjective judgment into trusted, actionable reliability signals—so every release, load test, and system change is safer, faster, and more predictable.  

Ship Faster with Confidence  

Reliability isn’t something you can eyeball anymore. With Reliability Delta, engineering leaders get the missing layer between observability and automation: clear causal evidence of how changes affect system behavior. It ensures your team can move fast, protect SLOs, and deliver with the confidence that every release is safer than the last. 

To learn more, see our docs: https://docs.causely.ai/in-action/reliability-delta/  
 

An Intellyx Brain Candy Brief

Causely monitors a real-time Bayesian network of semantically abstracted runtime operational telemetry data in order to observe and alert engineers to highly probable causes of issues and failure conditions, so they can ideally be resolved before they can emerge as customer-facing incidents.

Their “Causal Inference System” is a directed acyclic graph populated with tons of possible failure mode indicators. You can think of these indicators like micro-CVEs for observability, so that the system can know what to look for as it passively observes payloads within OTel logs and traces alongside golden signals such as latency or errors. It’s not another AI SRE, as the inferences it surfaces are deterministically based on live indicators that are semantically enriched with context.

When causes are observed, they can be captured as feedback so DevOps teams can flow through changes in the next CI/CD cycle, or reported into the enterprise’s incident management, ITSM and observability tools of choice, with direct links to contextual insights.

Sure, you could still do root cause hunting with any major observability platform worth its salt. However, to do that for a large distributed enterprise system, you would need to define thousands of policies that collect and tag telemetry data, and set up triggers and automation for each, such that the cognitive load and cloud data costs to keep it current might be prohibitively high. There’s always another way to do things!

Organizations collect far more telemetry than they use. The result? Exponential costs, PII risks, and still no root cause when incidents happen.

In this technical session, Yuri Oliveira (OllyGarden) and Severin Neumann (Causely) demonstrate:

• How to identify instrumentation problems at the source
• Why traces and semantic conventions enable purposeful telemetry
• How quality telemetry enables causal reasoning at scale

SRE teams are hitting a breaking point as Kubernetes environments scale faster than traditional workflows can keep up. Alerts pile up, incidents drag on, and teams lose hours in reactive firefighting instead of building reliability into their systems. At KubeCon + CloudNativeCon in Atlanta, Severin Neumann, Head of Community at Causely, offered a clear perspective on how AI can finally shift SRE practice from reactive to proactive. But this requires more than feeding logs into an LLM — it requires causal reasoning.

For years, the industry has talked about proactive SRE, but the reality rarely matches the aspiration. Neumann has spent more than a decade in observability and monitoring, and in his experience, teams always circle back to the same pattern: there’s an alert, teams react, things stabilize, and then the cycle starts again. The promise of AI offered a potential breakthrough, but early attempts fell into a familiar trap.

When companies say “AI SRE,” many immediately think of a large language model interpreting alerts and suggesting fixes. Neumann stressed that this approach often works only at the surface level. LLMs are powerful in pattern matching and summarization, but they don’t understand systems the way SREs need them to. They can connect correlations, but they can’t determine causation. And in high-pressure incident situations, an AI hallucination is more than an annoyance — it can waste hours and cost companies real money.

Causely takes a different approach. Their platform is built on a causal model that understands the structural relationships inside Kubernetes environments. Instead of giving an LLM raw observability data and hoping it finds meaning, Causely creates a deterministic model of how services interact and what components influence one another. The model knows which symptoms map to which possible root causes and can identify the single cause that explains them all.

This is a critical shift. Instead of chasing noisy correlations, SRE teams can rely on a reasoning engine that explains exactly why a problem is happening. LLMs do come into the picture, but only after the causal model has already identified the issue. At that point, a language model can help generate explanations, provide remediation guidance, or suggest kubectl or Helm commands to fix the problem. The heavy lifting — the understanding — remains deterministic.

Neumann also explained why this approach stands apart in the industry. Many teams tried the LLM-first path and burned their fingers. Some built internal solutions, others adopted competitors, but most eventually discovered the limitations. LLMs often fail when precision matters most, and in distributed systems, precision is everything. One misleading answer can waste hours in the middle of a firefight.

Causely’s causal reasoning avoids that by grounding all decisions in a model built around real system behavior. This also sets the stage for a more ambitious goal: shifting reliability left. Instead of waiting for symptoms, the model can analyze normal system behavior and identify bottlenecks or weak points before they cause downtime. This turns SRE work from reaction into prevention.

The conversation also explored the role of MCP (Model Context Protocol). Causely launched their MCP Server at KubeCon, enabling developers to pull causal insights directly into tools they already use. Through MCP, the remediation guidance — including detailed commands — can appear directly in an IDE or command line. The SRE no longer has to dig through logs or dashboards to figure out what’s going wrong. The causal model does that work and surfaces the fix.

Neumann outlined a vision where, over time, teams can set boundaries that allow AI to autonomously remediate certain classes of issues. If a service needs to scale up or memory needs to be increased, those actions could eventually be automated within limits defined by humans. This is where AI becomes a genuine partner — one that teams can trust to handle repetitive corrective tasks while they focus on system design, SLOs, and reliability architecture.

Trust is a key theme. Introducing AI into SRE workflows doesn’t eliminate human responsibility; it shifts it. Humans now become orchestrators who decide when to delegate and when to intervene. As Neumann put it, the more trust teams build in the system, the more they can remove themselves from situations where they’re no longer needed. This opens space for deeper reliability engineering, building guardrails, and designing better services.

A real-world example makes the value of causal reasoning clear. Imagine a financial services company running hundreds of microservices. Suddenly, everything turns red and user transactions fail. Traditional debugging would mean looking across alerts, logs, and traces to guess where the bottleneck is. Causely can cut straight through the noise, identify the one overloaded service, and surface the exact command needed to scale it up or adjust its memory. The time shaved off matters — for reliability, user experience, and cost.

Throughout the conversation, it became evident that AI isn’t replacing SREs. It’s allowing them to finally escape the cycle of constant firefighting. With causal reasoning, proactive reliability, and tools like MCP, SREs can shift to roles that emphasize guidance, architecture, and strategic improvements rather than crisis management.

Yotam Yemini, CEO of Causely, explains how the company integrates reliability engineering into product development, tackling challenges in cloud-native applications. The evolving role of Site Reliability Engineers (SREs) is discussed, highlighting the need to understand cause and effect in system performance. Causely provides tools for developers to enhance performance and reliability, promoting collaboration among engineering teams within a vibrant open source and cloud technology community.

Watch the interview here.

For a long time, observability has meant buying into a full stack you can’t really change: proprietary agents to collect the data, a proprietary protocol to move it, and a proprietary backend to look at it. Your telemetry lived inside a walled garden.

OpenTelemetry (OTel) is breaking that pattern. With community-standard instrumentation and the OTel Collector acting as a translation and routing engine, your metrics, logs, and traces are no longer trapped in that garden.

Observability isn’t a monolith 

There’s nothing inherently wrong with proprietary software; plenty of great systems are closed source. The problem is when your data becomes proprietary within these systems.

When collection, transport, and storage are tightly coupled to one vendor, your options shrink. Want support for a less common programming language? You might wait quarters for an agent. Want to change vendors? Prepare for weeks of reinstrumentation. Even simple ideas, like experimenting with a second backend in parallel, can become “projects.”

OTel changes the economics. Today, you can instrument almost everything consistently, and, yes, it has never been easier to swap your observability platform without touching application code. But, it’s not just about reducing vendor lock-in; when you own how your data moves, you can send it anywhere.

I promised not to rant about this again, but I need to come back to the “observability” debate once again: the way “observability” is used as a marketing term makes it seem like collecting, processing, and storing telemetry is one big monolith. It isn’t. Your pipeline is inherently composable, and the most leverage shows up at the tail: the “backend.” Treat that tail as a junction, not a culdesac.

What does SODA mean? 

That’s the idea behind SODA: Send Observability Data Anywhere.

SODA is simple on the surface: send a sensible combination or subset of your signals to the best tool for the job at hand. Under the hood, it means being deliberate about what you send and where you send it, keeping a durable copy you control, and refusing to recreate a new walled garden in the name of convenience. The OTel Collector makes this practical: you can enrich events, redact sensitive fields, apply sampling and routing policies, and fan out to multiple consumers, without touching application code.

We should understand that observability data is data, and often has value outside of just observability.

What does SODA look like in practice? 

By owning the transport of your data, your architecture becomes pluggable, and can adapt to how your needs change over time.

In the immediate-term, it means you can look at the most pressing concern most teams face: cost. You may not be able to move observability vendors overnight, but by instrumenting your app with OTel, you solve one of the biggest hurdles to getting started: testing a new observability stack is one configuration change away, and you can run multiple tools in parallel for comparison or during a migration.

For long-term retention, OTel makes it easy to store your complete historical data as a durable copy in inexpensive object storage in open formats. Doing so frees you from keeping all of your history in an expensive vendor bucket, and gives you replay: you can hydrate any alternative tool later without asking teams to reinstrument.

From there, you can send slices of telemetry to systems that act. Use signals inside the cluster to make decisions, like autoscaling with KEDA or the Horizontal Pod Autoscaler (HPA), gating progressive delivery so a canary only promotes when golden metrics and dependency health agree, or flipping feature flags when error budgets trend in the wrong direction. Security teams can mirror selected logs and traces to their SIEM, while the raw truth stays immutable in cold storage.

As AI enters the observability space, you are free to adopt new platforms that innovate in specific areas that matter to you, without being restricted to the capabilities of your core observability platform. You can use Causely to apply model-driven causal inference over your topology and recent changes to pinpoint likely causes for reliability issues and recommend or auto-apply safe actions. Or OllyGarden to help improve the quality of your OTel implementation.

Your telemetry is also product and business data in disguise, and can enable customer-facing teams to see product usage, user journeys, or customer impact as a result of an incident. Observability data is often silo’d and unavailable to internal analytics, but OTel allows you to bridge that gap whether you have separated data platforms, or a unified data stack. Platforms like ClickStack build on-top of flexible, open datastores like ClickHouse, and demonstrate how observability and business data can be co-located and correlated within one platform.

When the pipeline is open, you can experiment with, and adopt, new tools without rearchitecting your stack.

Composed, not siloed 

OpenTelemetry already breaks silos at the source by giving us a shared schema and transport for metrics, logs, and traces. SODA applies the same principle at the tail. Keep signals together as shared context on a durable stream and let specialized tools subscribe to that context, hand off seamlessly, and act without duplicating or fragmenting the truth. In a composed flow, a root cause identified in one place becomes the pivot to quantify impact and drive recovery in another—without losing the thread.

With freedom comes responsibility 

I’d like to stress an important note of caution: “send anywhere” is not the same as “send everything everywhere.”

Splitting your metrics, logs, and traces across disjointed backends that cannot be correlated is a fast path to longer MTTR and fingerpointing. If you want a thoughtful breakdown of why unified access matters for investigations, this article is a good primer.

The SODA posture helps to keep a coherent, durable source of truth under your control, then route purposeful subsets to the systems that extract additional value.

Drink some SODA and let us know what you think 

If you’ve read this far, you probably already have a mental list of places you wish your telemetry could go but currently doesn’t. That list is your SODA plan.

Maybe you start by dual-writing to object storage and turning on replay. Maybe you add a lightweight autoscaling signal for a spiky service. Maybe you route instrumentation health to a specialized tool while keeping unified access for investigations. The point isn’t to chase a shiny mesh of destinations; it’s to get more leverage from the data you already collect: safely, cheaply, and under your control.

We’d love to hear how you’re doing SODA today. Where are you sending telemetry beyond the one vendor you pay for? Which use cases are you covering: cost reduction, faster incident response, safer rollouts, richer product insights? Which ones do you want to see covered outside the “standard observability pipeline”?

It's back! This week is KubeCon, North America—one of the most beloved technology conferences in the world, centered around one of the most beloved enterprise technologies: Kubernetes. Born out of Google, Kubernetes has changed the very infrastructure on top of which modern applications are built. More than half of all enterprises have adopted Kubernetes, with millions of developers deploying their applications on top of Kubernetes every day.

But like any other technology, it comes with its trade-offs...mainly with the complexity of managing performance and debugging issues, which is why hundreds, if not thousands, of software vendors are touting their Kubernetes solutions this week. To help you separate the noise from the value, we want to highlight [] companies that truly help deliver value to enterprises using Kubernetes.

Gremlin

We're very excited to see this company ramp back up their public persona. Gremlin was founded by ex-Netflix and Amazon engineers in 2017 and made a lot of noise around the cutting-edge discipline of Chaos Engineering. For those of you who are unfamiliar, Gremlin has built a platform that empowers companies to run experiments on their systems in order to proactively identify where the weaknesses are.

Like getting people to the gym, it isn't always easy to convince people to change their habits—but the enterprises that build Chaos Engineering into their routines benefit from much healthier, reliable systems. At this KubeCon, Gremlin is announcing a strategic partnership with Dynatrace—a leader in observability and application performance monitoring—to help Kubernetes users keep their applications in a desired state.

Kubernetes services are automatically discovered within Gremlin, powered by Dynatrace's AI-driven observability and topology mapping. Health checks are then applied to Kubernetes' objects, allowing organizations to efficiently implement standardized reliability testing and gain deeper insights into their environments.

Mezmo

Mezmo is pioneering the concept of active telemetry in order to provide AI agents with better data and context. The company recently launched an RCA agent (root cause analysis) aimed at Kubernetes users that automatically identifies and fixes common issues such as deployment failures, resource issues, configuration errors, application-level failures, and more.

The proof is in the pudding: their AI agent consistently resolves issues in complex cloud environments faster and more accurately than other AI agents and models. According to the company's blog, "we're entering an era where incidents resolve themselves before engineers even know they exist." By leveraging agentic AI workflows, Mezmo rapidly analyzes telemetry data to pinpoint root causes, eliminate noise, and recommend actionable remediation steps.

Causely

Causely is pioneering the category of AI SRE—a category that will undoubtedly explode in popularity in 2026. Now more than ever, engineering teams are drowning in too much data and too many alerts. Having a system like Causely on the market—founded by a veteran of the industry with two prior startups in IT Operations—is a natural and critical response to the rise of AI code-generation tools that are shipping code faster than humans can reasonably understand it or manage it.

At this year's KubeCon, Causely is announcing an MCP Server that seamlessly integrates into any MCP-compatible IDE and enables developers to automatically diagnose, understand, and remediate complex issues within Kubernetes and application code using natural language prompts.

It works by analyzing the real-time state of the system, identifying whether the cause of an issue is in the infrastructure or application layer, recommending the precise code changes, configuration changes, or helm chart updates, and presenting these suggestions inline within the developer's IDE for review, refinement, or approval.

Komodor

It's hard to talk about Kubernetes troubleshooting and not mention Komodor. Founded by an ex-Google engineer, Komodor has been dedicated to the Kubernetes ecosystem since it launched out of stealth in 2021. Their management platform simplifies operations, provides automated troubleshooting, and helps teams manage complex environments. It tracks changes, analyzes their impact, and provides actionable context for issues, which reduces troubleshooting time and improves delivery velocity. Key features include automated drift detection, root cause analysis, and monitors for cluster health and resource optimization.

If you'll be in Atlanta, Georgia, this week for KubeCon 2025—stop by the booth of each of these companies. They are adding tremendous value to enterprises looking to maximize the performance and benefits of using Kubernetes.

Causely today unveiled a Model Context Protocol (MCP) server that enables developers to automatically diagnose, understand, and remediate complex issues within Kubernetes and application code using natural language prompts from within their integrated developer environment (IDE).

Severin Neumann, head of community for Causely, said the Causely MCP Server will make it easier for application developers to troubleshoot Kubernetes issues using a platform that already uses causal artificial intelligence (AI) models that were created to augment site reliability engineers (SREs).

The overall goal is to provide developers and SREs with insights needed to reduce downtime by analyzing the state of the system in real time to identify whether the cause of an issue is in the infrastructure or application layer. The Causely platform will then recommend precise code, configuration or Helm chart changes for developers to review, refine, or approve within their IDE.

Additionally, the Causely platform will also generate patches for Terraform, Helm, or application code to prevent issues from recurring.

It’s not clear at this point just how many software engineering teams are relying on AI to automate the management of Kubernetes workflows, but given the complexity of these environments, there is a significant opportunity to reduce stress and toil. The simple truth is that many organizations are limiting the number of Kubernetes clusters they might deploy simply because there isn’t enough SRE expertise available to manage them.

Many developers, meanwhile, have historically been intimidated by the complexity of Kubernetes clusters. The MCP server developed by Causely makes it simpler for developers to resolve issues on their own, both before and after cloud-native applications are deployed.

It’s not likely AI platforms such as Causely will replace the need for SREs any time soon, but they do present an opportunity for fewer SREs to successfully manage a larger number of Kubernetes clusters at scale at a time when the number of cloud-native applications being deployed continues to steadily increase. The issue, as always, is reducing mean time to remediation whenever all but inevitable incidents occur.

In fact, AI advances should enable SREs to spend more time on strategic issues such as ensuring availability versus constantly performing tactical incident management tasks, said Neumann.

Ultimately, multiple AI models that are being invoked by AI agents will soon be pervasively embedded across every DevOps workflow. The next major challenge will be finding a way to orchestrate the management of AI agents that will be accessing causal, predictive and generative AI models to optimize IT environments. Hopefully, those advances will significantly reduce the level of burnout that many software engineering teams experience as they are overwhelmed by repetitive manual tasks, especially when managing complex Kubernetes clusters that can be easily misconfigured.

In fact, it’s even conceivable that as AI starts to eliminate many of the bottlenecks that exist in software engineering workflows, much of the joy that attracted SREs and application developers to IT in the first place might soon be rediscovered.

If you haven't heard the term "AI SRE," you will soon. Following the boom of AI-generated code helping developers significantly increase their output, it was only a matter of time before more of the operations-related tasks around debugging, root cause analysis, and incident resolution were also AI-assisted.

One of the startups pioneering the AI SRE category is Causely—founded by IT Ops veteran Shmuel Kliger, who successfully sold two of his prior companies. Causely is unique in its ability to leverage causal inference to arrive at the root cause of issues accurately and more quickly than other solutions on the market. They are also shifting reliability left and identifying concerns before the code even gets shipped to production.

Today, the company announced the release of the new Causely MCP Server, a powerful tool designed to streamline and automate troubleshooting within Kubernetes. The company is making the announcement ahead of the popular conference KubeCon next week. This innovative solution integrates seamlessly with any MCP-compatible IDE, empowering developers to diagnose, understand, and resolve complex system issues using simple natural language prompts.

As Kubernetes' scalability and flexibility grow, so does the complexity of managing these systems. Issues like resource conflicts, unexpected pod evictions, and DNS delays often lead engineers to patch symptoms without uncovering root causes. Traditional monitoring tools offer valuable data but can make troubleshooting a manual, time-consuming process.

Causely's MCP Server aims to change that by embedding advanced causal reasoning directly into the developer workflow. Once integrated into popular IDEs such as Cursor or Claude, the system allows engineers to describe problems or desired outcomes conversationally, removing the need for manual searches or scripting.

Key features of the Causely MCP Server include:

• IDE-Centric Integration: Easy installation into MCP-compatible IDEs without significant infrastructure changes.
• Natural Language Prompts: Developers communicate issues and fixes naturally, streamlining problem reporting and resolution.
• Context-Aware Recommendations: The system uses real-time data and causal models to suggest effective fixes at the runtime, configuration, or code level.
• Upstream Fixes: Generates patches for Terraform, Helm, or application code to prevent similar issues in future deployments.
• Immediate Review & Refinement: Recommendations appear inline for iterative review before applying changes.

The MCP Server analyzes real-time system states to pinpoint whether an issue stems from infrastructure or application layers, then recommends precise modifications—be it code, configuration, or Helm chart adjustments. This approach simplifies maintaining systems in their desired state.

Karthik Ramakrishan, VP of Artificial General Intelligence at Amazon, praised the innovation: 

"Language models are powerful, but they require structured causal context to make the right decisions. Causely fills this gap, enabling real-time automation and reliable microservice operations."

By embedding intelligent, causal remediation directly into developers' workflows, Causely makes maintaining Kubernetes applications more straightforward and efficient than ever before. Certainly a company worth checking out if you'll be in Atlanta for KubeCon...or even if you won't be there, for that matter.

Kubernetes scalability and flexibility come with increased complexity. Services conflict for resources, pods are evicted unexpectedly, DNS queries lag, etc. When outages occur, engineers are often left patching symptoms without understanding the cause of these observed problems. Traditional monitoring and observability tools provide useful data, but troubleshooting remains a manual and time-consuming process.

“Causely’s MCP Server accelerates incident response by placing sophisticated causal reasoning directly in the hands of developers,” said Ben Yemini, Head of Product at Causely. “Once integrated into IDEs such as Cursor or Claude, the MCP Server allows engineers to describe problems or desired outcomes using simple natural language commands.”

Key Features of the Causely MCP Server include:

• IDE-Centric Integration: Installs seamlessly into any MCP-compatible IDE, requiring no significant infrastructure overhaul.
• Natural Language Prompts: Developers communicate problems or fixes conversationally, without needing to write scripts or manually search dashboards.
• Context-Aware Recommendations: The system uses real-time system data and causal models to propose specific, effective fixes at runtime, configuration, or code level.
• Upstream Fixes: Generates patches for Terraform, Helm, or application code to prevent issues from recurring in future deployments.
• Immediate Review & Refinement: Developers see recommendations inline, allowing iterative improvements before applying changes.

Causely’s new MCP server works by analyzing the real-time state of the system; identifying whether the cause of an issue is in the infrastructure or application layer; recommending the precise code changes, configuration changes, or helm chart updates; and presenting these suggestions inline within the developer’s IDE for review, refinement, or approval.

"If you’re serious about automating reliability in microservices you need what Causely is doing,” said Karthik Ramakrishan, VP of Artificial General Intelligence at Amazon. “Language models are powerful, but they can’t make the right calls without structured causal context. That’s the gap Causely fills, and it’s what makes real-time automation possible."

By embedding intelligent, causal remediation into the developer workflow, Causely makes it simpler than ever to maintain Kubernetes applications in their desired state. To learn more, read the announcement blog post or stop by their booth at KubeCon, Atlanta November 11-13.

About Causely

Causely is an AI startup dedicated to transforming Site Reliability Engineering through innovative automation, causal reasoning, and developer-centric tools. Their solutions help organizations manage complex distributed systems more efficiently and reliably.

Media Contact

Adam LaGreca

Founder, 10K Media

adam@10kmedia.co

The Problem: Complexity Hides the Real Issue 

Kubernetes gives teams the power to scale fast, but it also introduces new layers of complexity. Services contend for memory. Pods get evicted. DNS queries slow to a crawl. When something breaks, symptoms often show up far away from the real cause. 

A latency spike might surface at an API gateway, but the actual issue could be a congested message queue. Pod evictions might trace back to a misconfigured limit in a different service. Engineers end up chasing alerts, patching downstream effects, and firefighting without ever fully closing the loop. 

In systems like this, even well-intentioned remediation often lands in the wrong place. The action isn’t wrong; it’s just applied to the symptoms, not the cause.  

How Causely Approaches This 

Causely was built for distributed systems where problems propagate in non-obvious ways. It is designed to understand cause and effect across services and layers of the stack. 

At the core is a Causal Reasoning Engine (CRE) that applies domain-specific causal models to real-time telemetry. It maps causes to the symptoms they cause with high precision. It understands how services interact, how constraints emerge, and how changes ripple through the environment. This allows it to pinpoint the cause of service degradations, even in noisy environments where symptoms may be missing or spurious.  

Once the cause is pinpointed, Causely drives resolution at the appropriate layer, whether that’s a runtime adjustment, a configuration change, or a code fix. 

Closing the Loop with the MCP Server  

The new MCP Server connects this reasoning engine directly to the developer workflow. Through integrations with MCP-compatible editors like Cursor and Claude, Causely now: 

• Generates upstream patches to Terraform, Helm charts, or application code to prevent the same issue from recurring. 
• Remediates runtime issues in Kubernetes automatically, including CPU starvation, noisy neighbor interference, and memory exhaustion. 
• Delivers these remediations directly into the IDE for review, approval, or refinement, with full causal context. 

This isn't about writing scripts or building brittle rules. Causely analyzes the environment in real time and proposes the correct remediation for the right layer. 

Example: Slow Database Queries 

Let’s say your service slows down due to database query latency. Most tools might point you to the spike. Causely uses its CRE to map causes to the symptoms they cause, like a slow-running query may cause elevated latencies across multiple HTTP paths. It combines observed signals with domain-specific causal models to deterministically infer the cause of the observed symptoms. The MCP Server then surfaces the recommended fix directly in your IDE, with full causal context. 

Remediation, Where Engineers Work 

The MCP Server brings this reasoning and remediation into the tools engineers already use. There’s no jumping between dashboards, terminals, and editors to track an issue from signal to fix. Everything happens in place, with the right context. 

Whether you’re using Cursor, Claude, or any MCP-compatible editor, Causely now provides inline, explainable remediations. Developers can fix what’s broken—runtime, config, or code—without jumping between tools or digging through dashboards. 

Here’s another look at how this works in practice. Causely identifies a CPU resource contention issue that’s degrading multiple services, traces it to a misconfigured Helm value, and proposes the corrected setting—delivered directly into the IDE via the MCP Server. 

Built for Complex Systems; Kubernetes is Just the Start

We’ve focused first on Kubernetes because it concentrates so many reliability challenges in one environment: ephemeral workloads, misaligned configurations, and distributed dependencies. But the underlying problems we solve go beyond the cluster. 

Whether the root cause lives in your service mesh, Terraform plan, or application code, Causely’s causal model surfaces it, and delivers the fix where it matters. Kubernetes is one environment we operate in. The goal is broader: to make reliability engineered, not reactive, across every layer of modern software delivery.  

See It in Action 

If you’re attending KubeCon North America, come see it in action at Causely Booth #1661. We’ll show you how Causely detects what’s wrong, explains why it’s happening, and remediates it, right where the fix belongs.  

And if you’re ready to explore more on your own, start here: https://docs.causely.ai/ask-causely/mcp-server/  

“Bringing Causely to Google Cloud Marketplace will help customers quickly deploy, manage, and grow the Causal Reasoning Engine on Google Cloud's trusted, global infrastructure," said Dai Vu, Managing Director, Marketplace & ISV GTM Programs at Google Cloud.

Causely’s reasoning engine models how distributed systems behave, identifying the cause of reliability risks, their impacts, and the actions required to assure performance. With Google’s Gemini models, Causely automatically generates clear, context-rich explanations and remediation guidance that help teams act with speed and confidence.

“Our causal engine determines why services experience increased latency and error rates. Google’s Gemini models help communicate what to do next, translating our causal inference into actionable, automated guidance. We’re eliminating the need for incident war rooms and enabling proactive reliability,” said Yotam Yemini, CEO of Causely.

Two new features leverage Gemini models within Causely:

• Ask Causely: A chat interface for using Causely's inferencing engine to understand the cause of increased service latency and error rates and the corresponding blast radius.
• Enhanced Root Cause Descriptions: Causely prompts Gemini models with the metrics, symptoms, events, and logs associated with each root cause, allowing it to summarize the issue and expand Causely’s remediation with detailed, context-specific guidance.

Causely customers have reported up to 75 percent faster recovery and 25 percent fewer incidents, improving uptime and productivity. While optimized for Google Cloud, Causely remains multi-cloud and model-agnostic, operating across public clouds and on-prem environments and integrating with a range of large language models.

About Causely

Causely is an AI startup dedicated to transforming Site Reliability Engineering through innovative automation, causal reasoning, and developer-centric tools. Their solutions help organizations manage complex distributed systems more efficiently and reliably.

Media Contact

Adam LaGreca
Founder of 10KMedia
adam@10kmedia.co

Causely addresses this gap through its Causal Reasoning Engine, which models how dependencies interact in real time and accurately determines the cause of observed service latency and errors. By inferring the cause of performance degradation and understanding the affected dependencies, Causely enables automated actions to assure performance.  

Now, through a new collaboration with Google Gemini, engineering teams can act on those insights faster and more intuitively. 

Why We Started with Gemini  

Reliability engineering depends on both accuracy and trust. LLMs excel at interpreting vast, unstructured data, but without a principled understanding of cause and effect, their outputs are prone to hallucination.  

Causely provides that missing foundation. Its Causal Reasoning Engine models how services, dependencies, and resources interact. These causal models provide deterministic truth about the causes of performance anomalies, and their blast radius. LLMs build on this foundation by translating the results of this causal inference into natural language explanations and action plans that help teams act with confidence. The result is a real-time, closed loop between insight and action. 

We chose Gemini because of its contextual reasoning, enterprise-grade security, and deep integration with Google Cloud workloads. Gemini’s ability to interpret natural language, generate structured code, and summarize technical context complements Causely’s deterministic causal inference engine, turning complex telemetry into clear and reliable insights. 

How Causely Uses Gemini to Enhance Autonomous Reliability 

While Causely remains interoperable with any LLM that a customer wishes to use, we’ve integrated Gemini into Causely for two new features that make interacting with our Causal Reasoning Engine more intuitive and powerful. 

Ask Causely  

Ask Causely empowers users to ask complex questions about their environment, check service health, and identify both existing root causes and potential failure points. Ask Causely leverages Gemini’s natural language understanding and generation capabilities to deliver a conversational and seamless experience. It uses multiple Gemini models and takes advantage of Gemini’s generative features to provide a white-glove reliability experience. 

Gemini is integrated into several stages of the Ask Causely pipeline, offering a high degree of flexibility, control, and integration within Causely’s autonomous reliability framework. To ensure timely and accurate results, Causely uses low-latency Gemini models to support data-intensive operations such as log summarization, entity extraction, and contextual signal analysis across diverse telemetry sources.  

Key aspects of the integration include: 

• Adaptive Model Selection: Causely strategically deploys low-latency Gemini models to ensure quick responses while using higher-capability reasoning models to convert Causely’s causal diagnoses into clear, actionable remediations. 
• Grounded search for reliable knowledge: Ask Causely uses Gemini’s grounded search capability to deliver accurate, context-aware remediations based on trusted external sources such as vendor documentation, Stack Overflow, and GitHub. 
• Tool calling and code generation for live system intelligence: Ask Causely uses Gemini’s tool calling and code generation to query live services, interpret telemetry, and surface insights from Causely’s causal engine on identified symptoms and root causes. 
• Code generation for automation: Gemini’s code generation enables Causely’s Code Agents to analyze time series and topology data, generate diagnostic workflows, automate remediations, and perform dynamic analysis during active incidents. 
• Entity recognition: Gemini’s strong entity recognition helps Causely rapidly locate and correlate critical services, nodes, and components within complex environments. 
• Embeddings for enterprise grounding: Gemini embeddings enable Causely to integrate internal documentation and historical incidents to deliver organization-aware, contextually grounded insights. 
• Interpretable causal insights: Gemini translates Causely’s causal signal extraction from unstructured telemetry into clear, human-readable explanations and actionable remediations.  

Causal Explanation and Remediation 

Causely uses Gemini to generate SLO-aware, application-specific explanations and actionable remediations grounded in verified causal data and analysis. Causely infers the precise cause of observed anomalies and gathers the most relevant logs and events from across the environment to enable automated action. Gemini then contextualizes this evidence with Causely’s live causal graph to produce coherent, human/machine-readable descriptions and remediations that reflect the underlying issue, its operational impact, and suggested remediation steps 

Supporting features include: 

• Log and event contextualization: Gemini interprets logs and events selected by Causely’s reasoning engine, connecting raw telemetry to observed symptoms and their SLO implications. 
• Causal-grounded remediation actions: Recommendations are based on observed symptoms and tailored to the affected application or service context. 
• Automated post-incident summaries: Gemini compiles structured summaries that capture causal explanations, operational impact, and applied remediations, ensuring consistency and traceability across incidents.  

Open and Flexible 

We started with Gemini as an initial and foundational proof point. But our platform was built to be multi-cloud and model-agnostic. Causely runs across public clouds or on-prem environments, and we’ll continue to develop integrations with other large language models, giving customers flexibility without lock-in. If you have a particular model and use case in mind, please contact us! 

The Future: From Reactive to Autonomous Reliability 

Causely and Google Gemini together mark a step toward autonomous service reliability, where systems can understand, explain, and prevent issues before users are affected. This shift moves reliability from reactive firefighting to proactive, explainable prevention.  

See It in Action 

Explore the new Gemini-powered capabilities in Causely:

• View Causely on Google Cloud Marketplace ➜
• Request a Demo ➜ 

They cut through the hype and explore:

• Why AI SREs are gaining so much attention
• How to make AI benefits in operations tangible
• The limitations of language models in SRE work

Yotam shares insights from his unique journey from psychology to tech, and offers a fresh perspective on building reliable systems in the age of AI. Listen below or watch it on YouTube.

The goal: modernize without compromising performance. 

The constraint: no in-place upgrade path.

The outcome: a seamless migration, with zero regressions.

The Stakes: Migrating a Backbone Cluster Under Load

Quantum Metric is the customer-centered digital analytics platform for today’s leading organizations, enabling global brands to make smarter, faster decisions about their digital customer experiences. Their platform ingests and processes petabytes of data each day to help these brands avoid revenue loss and deliver more seamless digital journeys across their web and mobile applications.

Operating at this scale presents numerous challenges and Quantum Metric’s engineering team is constantly looking for ways to best serve their customers. The team wanted to upgrade to Google Cloud’s Dataplane V2 as a way to address operational challenges related to networking, but the team knew this wouldn’t be a simple migration to execute. The cluster in question hosted some of their highest-throughput services, including internal request handlers and real-time data processing pipelines. Several of these services alone handled over 30,000 requests per second.

With no in-place upgrade available, success meant spinning up a new cluster, migrating services in stages, and deprecating the old one without impacting production.

Stepwise Migration with No Room for Error

The team adopted a blue-green migration strategy. They started by shifting lower-risk, stateless services with minimal dependencies, which was ideal for early validation. From there, they moved to heavier components with deeper integrations and higher throughput.

Every step introduced risk. The services being moved were foundational; any missed dependency or regression could create a cascade of downstream failures.

Why Causely Was Essential

For Kevin Ard, Staff Platform Engineer at Quantum Metric and leading the migration, confidence throughout the process came from one source: Causely.

“Because of Causely, I didn’t need to do any custom telemetry work or worry about if something was off as changes were rolled out. The system made it simple to complete our migration with confidence.”

Causely continuously builds a live model of service and data flows using lightweight, eBPF-based tracing - no manual instrumentation, dashboard-building or query-writing is required. As Kevin shifted traffic, Causely proactively showed and analyzed real-time changes in service dependencies and system health. If something started to show signs of degradation, it showed exactly where and why so that changes could be applied ahead of any major problems. 

Example: Causely’s service and dataflow graphs show how requests and data move across services—and surface the true root cause directly within that flow.

Rather than stitching together dashboards or relying on intuition, Kevin was able to rely on Causely as a real-time copilot that analyzed cause-and-effect for him as changes were being made. 

The Result: Two Weeks Saved, No Fire Drills

Without Causely, Kevin estimates he would’ve spent two weeks building dashboards, coordinating instrumentation with at least three other engineers, and piecing together a view of the system. Instead, he had a single, always-on copilot keeping an eye on things as traffic shifted.

“With the sheer volume of telemetry our systems can generate, filtering the noise to focus on causality was a game-changer. Knowing that Causely would proactively spot degradations without needing to configure any rules or alerts was icing on the cake.”

The result:

✅ Zero regressions

✅ No performance degradations

✅ No fire drills

✅ A smooth cutover of a high-throughput cluster

Making High-Scale Reliability Practical

Even in environments with strong observability, Causely adds something critical: real-time causal reasoning. It understands not just what changed, but why – and it does this automatically without custom dashboards or complex rules. 

At Quantum Metric’s scale, reliability means preventing issues before they become incidents. During a high-risk migration, Causely gave Kevin and his team a new kind of clarity rooted in cause-and-effect across dynamic systems. This helped them improve how they think about managing complexity at scale and move fast without breaking things. 

Kubernetes has become the default backbone of cloud native architecture. But does it actually help you ship services more reliably, or is it just more moving parts?

Despite Betteridge’s law of headlines, the answer is yes, for the vast majority of companies. Kubernetes has proven essential for reliable service delivery in today’s cloud native world. But not because it magically sprinkles reliability over your stack. Rather, it gives you what you need to build reliability like an engineer, not a magician. Kubernetes is a means, never the end. But let me explain before you skeet this out of context…

Kubernetes Is Not the End, but a Means

Like any technology, Kubernetes is just a building block supporting your goals. Treat it that way. If you adopt it simply because nobody gets fired for choosing Kubernetes, you’re setting yourself up for pain. Kubernetes is powerful and unforgiving — there’s no shortage of real‑world lessons and failure stories from teams led by hype and not purpose.

Instead of chasing hype, start with the essentials. Cut to the core by asking a deceptively simple question: “Who cares?”

Who cares about the benefits —reliability, consistency and automation — that Kubernetes gives you? Teams running services on your platform expect those properties to hold under load and during change. Who cares about that? Your organization, offering those services to users. Users, who have their own goals. Those goals aren’t yours to reach, but you can make the path smooth and successful. When users succeed, they’re delighted — and that’s our end.

Why Delighted Users Matter to Engineers

Keeping that mindset, you might ask: “Why should I, as an engineer, care about delighted users?” You might wonder why you should care at all. Sure, you could go eat pretzels and scroll silly websites. But engineers don’t stop there. We look deeper — because by profession, you are a systematic thinker. You solve human needs with science and engineering.

So instead of just answering the initial question (“Do we even need Kubernetes for reliable service delivery?”), we approached it like engineers. We used a structured method called causal analytics — asking “Who cares?” — until the real goal emerges: Does Kubernetes deliver the level of service reliability that we need to delight our users?

All that remains is to break it down in your specific context: Who are your users? What is a delightful experience for them? What expectations for reliability do they have?

I am certain that after answering those questions, 95% of you will conclude that your needs for reliability are not exceptional, and that Kubernetes is the right choice to achieve those goals. This should not be surprising, because as with most requirements, there’s a bell curve; most teams need a reasonable amount of reliability, some need a little and a few need the extreme.

The teams needing extreme reliability are building systems where lives are literally at stake, if they fail — aviation, medicine and similar fields.

For the teams that need less than average reliability, here’s a story: Years ago, I asked the ops team of a world-class soccer club how much reliability mattered for their online store. Their answer surprised me — “Our users are fans! If they can’t get a jersey now, they’ll wait. Reliability isn’t our differentiator — we focus on a unique fan experience.“

Indeed, fans happily wait. Half the fun is in the waiting, which deepens their sense of belonging.

What I learned from that is simple — it’s perfectly fine to be average in most areas, as long as you know where not to compromise.

You’re an Engineer – Behave Like One!

As you might have guessed, the point of this text is not to settle the question of whether you need Kubernetes for reliable service delivery. It is to remind you that you should not only like building and tinkering like an engineer but also enjoy thinking and making decisions like one. It’s your job to know when it is perfectly fine to deliberately pick the boring, established standard and when you need to go against the grain.

When you make decisions that way, you gain clarity and calmness. You will read yet another blog post from teams ditching Kubernetes and understand that they might be one of the outliers that need something different. But because you made the choice to adopt it with your context in mind, you won’t regret it.

Even if you reach the point where you need to revisit your decision, as new information and context emerge, you’ll approach the problem with the right questions: Given our current context, is moving away from Kubernetes the right thing to do? Or is it preferable to continue using it, even if your enthusiasm has waned?

Finally, it’s not just about Kubernetes. Every technology you add is a means, not the end and should be evaluated in your context. This mindset is especially useful for approaching hype cycles. Right now, everyone is trying to solve every problem with LLMs. Next year it will be something else. The cycle never stops.

But as an engineer, your job isn’t to chase the latest shiny thing. It’s to understand your context, weigh the trade-offs and decide deliberately. Sometimes that means adopting the new; sometimes it means sticking with the boring. The strength lies in knowing the difference.

Causely already mediates to OpenTelemetry, Datadog, and Dynatrace to consume traces, metrics, alerts and logs. Today we’re adding IBM Instana Observability to that list.

This is yet another step in our journey to autonomous service reliability: Our Causal Reasoning Engine (CRE) takes the flood of data you already collect and turns it into action by automatically inferring and pinpointing the causes of service degradations, reliability issues, and emerging risks. Furthermore, Causely provides context-rich explanations and remediation guidance—so responders move faster and with confidence. 

Why This Matters 

Causely eliminates the need for war rooms. Causely’s reasoning engine analyzes telemetry, traces, logs and alerts from your observability stack—including IBM Instana Observability—minimizing MTTR, protecting SLOs, turning dashboards into decisions, and enabling reliability.

“Our goal is to collapse the war room—use the Instana data you already have to identify true causes and risks, then deliver clear, guided remediation,” says Ben Yemini, Head of Product at Causely. 

Causely’s users stop chasing alerts and act on clear, context-rich explanations and guided remediation. Causely maintains a continuously updated topology graph of your services, data flows and infrastructure and a causality graph that maps causes to symptoms. Using probabilistic inference, CRE pinpoints causes, evaluates blast radius, and prioritizes remediation—so teams converge faster with fewer escalations. Because the engine runs continuously, it also highlights emerging risks before they become incidents. 

As one customer put it:

“We don’t have a shortage of telemetry—we’ve been using Instana for years. We’re overwhelmed by data and dashboards. Dealing with performance issues is labor-intensive, and it’s been nearly impossible to prevent them.”

This integration is designed to change that. 

Key Features of the IBM Instana Observability Integration 

• Automatic root cause analysis pinpointing cause and risk in real time. Causely’s CRE contiguously ingests Instana signals and pinpoints the causes of service degradations and failures and surfacing emerging risks before they become incidents.  
• Automatic impact analysis and blast radius mapping. Causely’s CRE continuously infers the impacted services, endpoints, databases, and SLOs to drive the right prioritization.  
• Automatic remediation. Causely provides context-rich explanation of the root causes, but more importantly can automatically remediate the cause by providing the specific fix at runtime, configuration, or code level.  
• Fast, low-lift enablement. Connect Instana via API and start diagnosing in minutes without changes to your services. 

The result:  

• Continuously assuring service reliability 
• Minimizing MTTR 
• Improving operational efficiencies  
• Increasing productivity 

Learn More

To learn more about using Causely with IBM Instana Observability, see the integration guide. 

That matters to us because Causely turns your observability data into distilled insights to reason over cause and effect to put teams in control. The faster you get baseline, trustworthy signals, the faster we can do that work. 

Why Beyla Helps 

Teams come to us in two states. Some already have a healthy OpenTelemetry footprint; for them, we plug Causely in and get to causal reasoning quickly. Others are still early: traces are patchy, metrics are inconsistent, logs are everywhere. Beyla makes that second state less painful. You turn it on and a picture forms: service boundaries, dependency maps, request paths. Suddenly they’re no longer guessing. Our engine creates a consistent baseline of telemetry they can trust, and the engine can begin attributing symptoms to specific causes with confidence.  

The effect shows up in the first week: fewer blind spots, clearer causal analytics, and the ability to move from “seeing an error” to “knowing why it happened and what to change.” Beyla helps create the raw material; Causely turns it into decisions. 

What the OBI Donation Signals 

Donating Beyla to the OTel ecosystem signals a commitment to standards and longevity. It’s a move toward shared building blocks rather than one-off integrations. We value that because our promise — autonomous reliability without drama — depends on predictable inputs and open interfaces. The more OTel wins, the healthier the whole stack becomes. 

How This Looks With Causely 

Our agents roll out Beyla by default. Setup is minimal: deploy Beyla, establish a baseline of traces/metrics, and immediately begin causal reasoning. Beyla provides out-of-the-box visibility; Causely layers causal inference, risk scoring, and proposed actions.

You get: 

• A consistent baseline of traces/metrics powered by eBPF auto-instrumentation. 
• High-confidence root cause detection that doesn’t drown you in correlations. 
• Guardrails that respect zero-trust boundaries — you keep your data; Causely reasons over signals, not secrets. 

It adds up to a simple promise:

Faster time to control, not just faster dashboards.  

Try Beyla on Your Own 

If you’d like to see what we’re talking about, the easiest path is to stand up a tiny environment locally and watch Beyla fill in the picture. The steps below take you from a single service to a small conversation between services, and then into traces and metrics you can actually explore. 

Single Service + Beyla (Docker) 

Pick a simple HTTP service; the example below instruments a demo app running on port 8080 and prints spans to the console — no OTLP required to get started. 

# Terminal 1 — start your app 
docker run --rm --name demo -p 5678:5678 golang:1.23 go run github.com/hashicorp/http-echo@latest -text=hello 

# Terminal 2 — run Beyla next to it (console output only) 
docker run --rm \ 
  --name beyla \ 
  --privileged \ 
  --pid="container:demo" \ 
  -e BEYLA_OPEN_PORT=5678 \ 
  -e BEYLA_TRACE_PRINTER=text \ 
  grafana/beyla:latest 

Open the app in your browser (http://localhost:5678), click around to generate traffic, and watch spans print in your Beyla terminal. Save a short snippet of that output — we’ll use it below as an example of what “good” looks like. 

If you want to send data to an OpenTelemetry Collector, Tempo, or Jaeger, add the following to the Beyla container:  

  -e OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \ 
  -e OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 

A Small Web of Services (Docker Compose) 

To make this concrete, the snippet below starts two services that talk to each other, runs Beyla against the frontend, and wires up local backends (an OpenTelemetry Collector that feeds Jaeger for traces, and Prometheus scraping Beyla’s metrics). 

services: 
  frontend: 
    image: golang:1.23 
    command: ["go", "run", "github.com/hashicorp/http-echo@latest", "-listen=:5678", "-text=hello"] 
    ports: 
      - "5678:5678" 
    environment: 
      - BACKEND_URL=http://backend:9090 
 
  backend: 
    image: ealen/echo-server:latest 
    environment: 
      - PORT=9090 
    expose: 
      - "9090" 
 
  beyla: 
    image: grafana/beyla:latest 
    privileged: true 
    pid: "service:frontend" 
    environment: 
      - BEYLA_OPEN_PORT=5678 
      - OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf 
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4318 
      - BEYLA_PROMETHEUS_PORT=8999 
      - BEYLA_TRACE_PRINTER= 
    depends_on: 
      - frontend 
      - jaeger 
 
  jaeger: 
    image: cr.jaegertracing.io/jaegertracing/jaeger:2.10.0 
    ports: 
      - "16686:16686"  # Jaeger UI 
      - "4318:4318"    # OTLP/HTTP ingest (native in v2) 
 
  prometheus: 
    image: prom/prometheus:latest 
    volumes: 
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro 
    ports: 
      - "9090:9090" 

Create the Prometheus config next to your compose file: 

prometheus.yml 

global: 
  scrape_interval: 5s 
scrape_configs: 
  - job_name: "beyla" 
    static_configs: 
      - targets: ["beyla:8999"] 

Bring it up: 

docker compose up -d 

Hit the frontend to generate traffic: 

curl -s http://localhost:8080/ | head -n1 

When the stack is up, open the Jaeger UI at http://localhost:16686 and search for the frontend service to browse traces. For metrics, visit Prometheus at http://localhost:9090 and try queries like http_server_request_duration_seconds_count or http_client_request_duration_seconds_count to see call patterns emerge.

What to Look for 

Generate a little pressure (for example, hey -z 30s http://localhost:8080/). In Jaeger, follow a slow trace end-to-end and note where p95 shifts between the frontend and the backend call. In Prometheus, line up the client and server RED metrics to see where latency actually accumulates — it’s a simple way to separate symptoms from causes.

Where to Read More 

If you want the exact steps, the canonical source is the Beyla documentation and the OBI pages in OpenTelemetry.

Our own docs show how Beyla and the Causely agents fit together in a few minutes of setup. 

Closing the Loop 

Beyla is the right kind of infrastructure: minimal friction, maximal signal, and donated to the place where open standards live.

If you’re ready to move from “seeing it” to “controling it,” we’d be happy to show how Causely turns that signal into confident action. 

Ready when you are. 

The Stakes: Real-Time Bets, Real-Time Risk 

For this industry-leading platform, performance is crucial. During live events, hundreds of thousands of users rely on the app to place bets in real time. If a service slows down, bets fail. If latency spikes, revenue is lost. There’s no buffer for outages, and no time for guesswork. 

Many of the platform’s highest-traffic moments, such as marquee games, playoff weekends, or last-minute betting surges, occur on weekends or late at night. This adds strain on a small team that's already stretched thin, often pulling engineers into incident response during off hours, when the toll on focus, sleep, and morale is highest. 

The company had modern observability tools in place, including Grafana dashboards, Prometheus metrics, and alerting pipelines. But when something went wrong, their SREs still had to hunt for the cause across a vast array of charts and logs. They needed something that could connect the dots for them, quickly and accurately. 

The Challenge: Observability Without Clarity 

Like many teams operating large-scale microservices architectures, they have invested in and standardized on an observability platform that provides lots of visibility. What the team lacked was understanding to enable rapid decision-making. They could see when something was breaking, but not why, or what to fix first. 

The complexity came not just from the number of services, but from the nature of the system itself. Services communicated asynchronously over messaging queues, wrote to distributed databases, and scaled dynamically based on real-time load. These components operated independently yet were deeply interdependent with one another. During peak sporting events, traffic could spike dramatically in seconds. This created unpredictable cascades of latency, retries, and congestion. 

Under these conditions, their observability tools struggled to identify the real cause of issues. Dashboards showed metric anomalies. Alerts fired across disconnected tools. However, understanding what was actually happening required engineers to mentally reconstruct a constantly shifting topology and then guess at the most likely root cause based on limited data. 

Each incident became a manual, high-stakes investigation. Teams would dig through logs and traces, cross-reference alerts, and escalate issues across multiple teams to determine where to start. That meant long triage loops, missed SLOs, and far too much time reacting instead of preventing. 

What they needed was a system that could keep up with the complexity, something that could analyze the real-time behavior of the entire environment, understand how changes ripple across services, and surface what matters before user impact.

The Solution: Causal Reasoning Embedded in Grafana 

To close this gap, the team rolled out Causely’s Grafana plugin. It gave their SREs a new layer of intelligence, right inside the dashboards they already use. 

Causely continuously consumes telemetry from their existing observability tools and applies causal reasoning to identify what matters. It answers three essential questions: 

Where is the problem?
What is it?
And why did it happen? 

With that context, Causely enables remediation, either by guiding engineers to the right action or automating it through AI. 

In this case, when service latency began trending in the wrong direction, Causely didn’t just trigger an alert. It identified the actual cause, explained the potential blast radius, and gave the team the confidence to act quickly, without second-guessing or escalating the situation. 

Watch the demo:

The Outcome: Reliability Under Constant Change  

During a major live event, the platform began showing early signs of degradation. But instead of spinning up a war room, the SRE team used Causely to trace the issue to a misconfigured connection pool in an upstream service, before end users were ever impacted. 

They resolved the issue fast, stayed within SLOs, and avoided revenue loss during one of their busiest betting windows. There was no guesswork, no back-and-forth, and no missed opportunities.  

See It at ObservabilityCon 

Causely will be at Grafana’s ObservabilityCon, showing how causal reasoning is helping engineering teams stay ahead of incidents—and focus on building, not firefighting. 

Want a 1:1 walkthrough before the event? 

👉 Book time with our team.  

Causely turns observability into action. 

It brings clarity, speed, and confidence to every incident—right from your existing Grafana workflows. 

When a system fails, identifying and resolving the technical root cause is what brings services back to health. But the story doesn't end there. To fully recover, teams also need to understand the impact outside their systems: which customers were affected, how many transactions failed, and what actions to take next. Traditional observability tools stop at symptoms, leaving engineers without a clear view from failure to business impact.

Causely changes that. By continuously reasoning over your telemetry, Causely identifies the true root cause in real time and delivers it straight into Slack — no guesswork required. From there, engineers can drill into the Causely UI to validate technical details and confirm the non-technical context behind the failure.

Combined with ClickStack by ClickHouse, teams can move beyond the technical fix to quantify and remediate the wider impact. Starting from the root cause Causely highlights, ClickStack makes it easy to pivot directly to the affected users, measure the scope of the issue, and take recovery actions like targeted win-back campaigns. Together, they close the loop: Causely restores system health, while ClickStack ensures customer trust is recovered just as quickly.

Watch the video to see how Causely and ClickStack by ClickHouse help you fix failures confidently — and address their real-world impact.

Incidents create noise, pull in too many teams, and waste valuable engineering time chasing symptoms instead of causes.

That’s why we’re excited to share our new integration with incident.io. By combining Causely’s causal reasoning engine with incident.io’s powerful automation platform, engineering teams can identify the true root cause of incidents faster and respond with greater focus.

The result: fewer people distracted, faster MTTR, and incidents contained before they spread.

In this short demo, you’ll see how Causely analyzes incoming alerts in real time, maps them onto live dependency graphs, and pinpoints the single root cause behind cascading failures. incident.io then orchestrates the response, routing the incident to the right team with the right context from the start. Together, Causely and incident.io replace reactive firefighting with clarity, precision, and automation so your teams can spend less time triaging and more time building.

That’s why we’re excited to share how Causely now integrates with incident.io. By combining Causely’s causal reasoning engine with incident.io, engineering teams with complex microservices environments can go from incident to resolution much faster.

Why This Matters

When an incident begins, the first question everyone asks themselves is, “Why?” 

In modern microservices environments, answering that causality question is hard. Trying to do it in a programmatic way that enables automation feels to many as being near impossible. Dependencies and data flows shift unexpectedly, and latency and errors ripple outward in ways that are difficult to untangle. Without an understanding of causal relationships and emergent behaviors across services, multiple service owners often get pulled into noisy investigations and waste time chasing downstream effects instead of the actual cause.

We love incident.io as an automation platform because it has a great user experience and provides the structure and automation needed for automating incident response. For customers using incident.io in large and dynamic environments, Causely adds critical intelligence by applying causal reasoning directly to live infrastructure, services, and data flows to infer the single cause behind a flood of alerts. This ensures the right service owner is reached with the right context from the start.

The result: faster resolution, fewer people distracted, and incidents contained before they spread.

How It Works

1. Alerts flow into incident.io from multiple sources such as Alertmanager, cloud platforms, and observability tools.
2. Causely analyzes those alerts in real time, mapping them onto live dependency maps of services, infrastructure, and data flows it discovers. It applies causal inference to identify the cause of the anomalies. This requires no complex rules, policies, or prior training, and scales naturally to large, dynamic microservices environments.
3. incident.io orchestrates the response with this context: routing incidents to the right owners, engaging on-call engineers, opening Slack channels, automating where possible, and keeping stakeholders informed.

For large engineering organizations with hundreds or thousands of dependencies, the value is clear. Instead of multiple teams chasing scattered symptoms, the integration ensures the right team gets the right context immediately, and impacted teams get the fastest path to innocence.

The Joint Value

By combining causal inference with automated orchestration, teams get:

• Clarity without chaos: Causely distinguishes cause from effect across live dependency maps, so only the right team is engaged.
• Seamless execution: incident.io automates the response, removing friction from every operational step.
• Aligned productivity: The integration prevents cross-team thrash, reduces MTTR, and frees engineers to stay focused on building.

Together, Causely and incident.io replace reactive firefighting with precise, dependency-aware incident response and automation.

See It in Action

We’ll be showcasing this integration at incident.io’s Sev0 Conference in San Francisco on September 23rd. Stop by the Causely booth to see how our joint solution can work for you, or request a free consultation here.

These aren’t interchangeable hats. They’re disciplines with fundamentally different goals. Blurring them may help you survive at 20 engineers, but at 200 it will strangle velocity and reliability. Even Netflix and Spotify, the poster children for speed at scale, had to abandon blended roles once the stakes got high.  

This article cuts through the noise: what these roles really are, where they overlap, and why betting on “one-size-fits-all ops” is a costly mistake.   

The Three Disciplines 

DevOps: A Cultural Foundation, Not a Job Title 

DevOps is more than just a role: It is a mindset. Born to tear down the wall between dev and ops, DevOps emphasizes shared responsibility for software delivery. 

• Goal: Deliver software quickly, safely, and consistently 
• Focus Areas: Collaboration, automation, CI/CD 
• Common Activities: Pipelines, IaC, release orchestration 
• Metric: Deployment frequency and lead time for changes  

Where companies go wrong: hiring “DevOps engineers” as if DevOps were just another box on the org chart. Rather than existing as a role or even a team, DevOps should be considered the baseline culture for modern software delivery. 

Site Reliability Engineering (SRE): Reliability as a Feature 

SRE, created at Google, applies engineering discipline to operations. It treats reliability not as an afterthought but as a product feature with measurable outcomes. 

• Goal: Keep systems reliable, scalable, and performant 
• Focus Areas: SLOs, error budgets, capacity planning, incident response 
• Common Activities: Defining SLIs/SLOs, automating toil, postmortems 
• Metrics: Error rates, availability, latency 

Where DevOps creates shared accountability, SRE enforces it with data and rigor. Reliability becomes something you can track, budget for, and improve. 

Platform Engineering: Scaling Without Chaos 

Platform Engineering is the newest of the three, and it is rapidly becoming essential. Its job is to build internal products for developers, standardized pipelines, self-service infrastructure, and golden paths that reduce cognitive load. 

• Goal: Improve developer productivity and consistency 
• Focus Areas: Internal developer platforms (IDPs), golden paths, service catalogs 
• Common Activities: CI/CD frameworks, self-service infra, observability integrations 
• Metrics: Developer satisfaction, time-to-value for new features  

Where SRE enforces reliability, Platform Engineering makes speed sustainable by removing friction and standardizing how teams build and ship. 

How They Interrelate 

These disciplines do not compete, they reinforce one another: 

• DevOps creates the culture of shared responsibility. 
• SRE makes reliability measurable and actionable. 
• Platform Engineering productizes infrastructure so both can scale. 

Together, they form a virtuous cycle: culture drives collaboration, reliability ensures quality, and platforms remove friction. 

Why The Lines Blur 

In small companies, blending roles is inevitable. One engineer might build the pipeline, run on-call, and hack together infra automation. That works when survival depends on speed. 

But the trade-offs are real: constant context switching, shallow specialization, and bottlenecks that emerge as systems grow. What looks like efficiency at 20 engineers becomes fragility at 200. 

Common blends include: 

• DevOps + SRE: one ops-focused engineer juggling deployments and reliability. 
• SRE + Platform: reliability engineers building tooling to reduce toil. 
• DevOps + Platform: CI/CD pipelines evolving into internal platforms. 

These hybrids buy time early, but they do not scale. 

The Growth Path: From Blended Roles to Specialization 

As headcount and complexity rise, most organizations follow a predictable arc: 

• Early Stage (1–20 engineers): Generalists do everything. Speed over rigor. 
• Growth (20–100 engineers): “DevOps” teams emerge, basic SRE practices are introduced. 
• Scaling (100–500 engineers): Dedicated SRE and Platform teams form to manage reliability and developer experience. 
• Enterprise (500+ engineers): Clear ownership is established across disciplines, with DevOps principles embedded everywhere. 

Ignore this progression and you will pay the price, either in velocity lost to chaos or outages caused by brittle systems. 

Lessons from Netflix and Spotify 

Even the best had to evolve. 

• Netflix: Started with infra-savvy generalists. They embraced end-to-end DevOps culture, pioneered Chaos Monkey, and eventually built platforms like Spinnaker (CI/CD) and Titus (containers). Today, reliability is a shared mandate, supported by platform teams with a product mindset. 
• Spotify: Grew fast with generalist ops, then adopted squad-based DevOps. Over time, they created dedicated reliability teams and built Backstage (later open-sourced) to tame service sprawl. Platform and SRE teams now enable hundreds of squads to ship quickly without burning out.  

Both prove the same point: blended roles help in the sprint, but specialization wins the marathon. 

Conclusion: One-Size-Fits-All Ops is a Trap 

SRE, DevOps, and Platform Engineering are not buzzwords or interchangeable hats. They are complementary disciplines that companies must deliberately balance as they scale. 

The playbook is clear: 

1. Start with DevOps as cultural glue. 
2. Invest in SRE to treat reliability as a product feature. 
3. Build platforms once scale demands it.  

Done right, this progression does not just prevent outages. It transforms operational chaos into a lasting competitive advantage. 

Two topics stood out most in the series. The first is the ongoing wrestling match between the terms Application Performance Management (APM) and observability. The second is how AI will change this space over the coming years. Let’s take them one by one.  

APM vs. Observability — The Bikeshedding Debate 

When we talk about observability, it’s important to distinguish between the technical definition and the marketing definition. In control theory, observability means “a measure of how well internal states of a system can be inferred from knowledge of its external outputs” (Wikipedia). We borrowed that definition from control theory into the domain of software, where those “internal states” are the real things that are happening invisible to us: CPU cycles, memory consumption, packet flows, the actual bits moving through our systems. The “outputs” are what we measure: traces, metrics, logs, profiles, and ultimately what the end user sees. 

It’s a good definition because it clearly states how we can make a system objectively more observable: by providing increasingly better external outputs that describe the internal states as closely as possible, until in theory one could determine the entire system’s behavior from its outputs. This is how engineers think, which is why it resonates so well with us.  

Furthermore, we can see this approach in action, as open projects and standards like Prometheus and OpenTelemetry evolve, adding new signals, more domains, and even semantic conventions to steadily improve what can be inferred from telemetry. Observability in the technical sense is just about making more of the system’s inner state visible. 

But then there’s the marketing use of the term “observability,” particularly in how vendors brand and sell their offerings as “observability platforms.” Look under the hood and you’ll see evolution, not revolution. Many features touted as “new” under the observability banner — distributed tracing, combined app and infra visibility, end-user monitoring, business-centric insights — were already present in APM solutions long before.

APM, traditionally defined as the discipline of monitoring and managing the performance and availability of software applications, had long offered these capabilities — the key difference is that they were locked inside vendor walled gardens, fragmented across products, and often poorly implemented. Observability didn’t invent them; it generalized them and, crucially, opened them up.  

That’s the real accomplishment of observability, in its marketing sense: expanding a space once dominated by closed APM vendors, breaking down their walls, and making telemetry collection a commodity. 

Beyond that, does it really matter whether we call it APM or observability? The debate may have been useful to move the industry forward, but today it’s a distraction and a waste of energy; it’s bikeshedding. Use the terms however they suit you — what really matters is ensuring systems consistently deliver the service levels our users expect in the years ahead. And that’s where AI comes in. 

AI in Observability — Hype, Reality, and What’s Missing 

The APMdigest series also collected a wide range of AI predictions. Many of them, we at Causely agree with wholeheartedly. Machine learning is already reducing alert fatigue, clustering related events, and detecting anomalies faster than humans. LLMs are making telemetry more accessible in many languages, so engineers can query systems in their own tongue without having to learn domain-specific query languages. These are real wins. 

Where we diverge is in how far LLMs can take us. At their core, LLMs are pattern-matching machines. They can generate plausible explanations, but they don’t actually understand causality. That’s a critical limitation when the task is diagnosing why a system failed. Correlation is not causation — and if you’ve been on-call at 3 a.m., you know how costly the wrong guess can be. 

LLMs are also inherently reactive. If I have to ask the chatbot why something is broken, it’s already too late. They’re not designed to proactively detect precursors of failure, to flag the subtle changes that precede an outage. That requires something different: causal reasoning. 

Causal reasoning is what turns noise into signal. It builds an explicit model of how services depend on each other and how failures propagate. That model acts as a pre-processor layer: it distills raw telemetry into precise insights about what is really going on. Once you have that, LLMs can actually shine. They can take the output of causal analysis and do what they do best — generate natural-language explanations, propose remediations, even open pull requests to apply safe fixes. 

In other words, causal reasoning is the missing layer between raw observability data and generative AI. Without it, we risk drowning in correlations. With it, we unlock the path to truly autonomous reliability. We’ve written more about this in InfoQ, on our own blog, and in industry pieces like CIO Dive if you want to dive deeper. 

Closing Thoughts 

The APM vs. observability debate may make for lively discussion, but the more interesting question is how AI will actually change the way we run systems. At Causely, we agree with much of the optimism in the APMdigest series, but we believe the crucial next step is to go beyond correlation and embrace causal reasoning. That’s how we move from data to control, from dashboards to autonomous systems. 

It’s a privilege to be part of a community that debates these questions so openly. We’re grateful to APMdigest for hosting the series, and we look forward to future editions. In the meantime, we’d love to hear your perspective — drop us a note at community@causely.ai or join the conversation with us on social media. 

It’s a reassuring idea, but it’s a myth. Microservices don’t isolate failure; they multiply it. 

In real-world distributed systems operating at scale, failures do not stay politely in their lanes. They leak across queues, caches, retries, and shared state. They multiply through invisible dependencies. And the more services you run, the harder it gets to see where the blast radius actually stops and, critically, what the cause is. 

Microservices do not automatically deliver fault isolation by design. They replace one obvious forest fire with a sprawling network of subtle, cascading brush fires. 

The Promise of Microservices 

On paper, microservices look like a natural cure for fragility: 

• Each service is independent, so one crash should not cascade to others. 
• Circuit breakers, bulkheads, and fallbacks can contain failures. 
• Advanced designs like cell-based architectures further limit blast radius. 

This sounds good in theory. And in tightly disciplined environments with mature engineering practices, some of these promises hold. 

The Reality 

In practice, fault isolation is rarely automatic and microservices make it harder to understand and control the blast radius.  

• New failure modes emerge. Latency, coordination bugs, partial outages, and data drift become common. 
• Shared dependencies betray isolation. A database, queue, or cache hiccup can silently spread impact across dozens of services. 
• Partial degradation is worse than full failure. Services stuck in retry storms or serving stale data prolong incidents instead of containing them. 
• Operational burden grows. Effective isolation requires top-tier observability, disciplined retry policies, and carefully engineered degradation strategies. 

Without disciplined engineering focus, fault isolation remains more promise than reality. 

Our Perspective at Causely 

We do not believe resilience is a side effect of microservices. It is a design goal that must be deliberately engineered, monitored, and maintained. 

That is why our system focuses on causal reasoning: 

• Mapping dependencies. We continuously uncover how services and infrastructure actually connect, not just how architects think they do. 
• Analyzing blast radius. We model how failures propagate, not just where they originate. 
• Pinpointing cause and effect. We distinguish symptoms from true root causes, even when failures ripple through retries, caches, and queues. 

The challenge is not shrinking the blast radius. It is being able to understand it clearly and programmatically mitigate the damage. That is the gap our system closes. 

TL;DR? 

Resilience does not come from the architecture you choose. It comes from how well you understand causality inside it. 

For engineering teams, that means: 

• Continuously mapping dependencies and blast radius across services. 
• Designing isolation explicitly instead of assuming microservices will provide it. 
• Using systems that reason about cause and effect, so humans are not left guessing when it matters most. 

The myth is that microservices give you fault isolation for free. The reality is that they make causal reasoning non-optional. 

If your experience has confirmed (or contradicted) this reality, we would love to hear it. Engineers get stronger when we challenge the myths together. 

Introduction 

“AI SRE” – shorthand for applying artificial intelligence (AI) to Site Reliability Engineering (SRE) – has been bubbling up in conference talks, blog posts, and vendor marketing. SRE is a discipline that applies software engineering principles to ensure systems meet defined reliability targets. The idea of automating or augmenting this work with AI is compelling: fewer mistakes, less downtime, faster resolution, and lower cost. Not to mention, most engineering leaders would acknowledge a talent shortage in trying to fill these roles. 

But the conversation is often fuzzy due to: 

• A deafening level of marketing hype disconnected from reality. 
• Varied understanding and adoption of SRE as a job function. 
• Different interpretations and implementations of AI. 
• Technology being implemented that is a poor fit for the task at hand. 

In this article, I’ll strip away the buzzwords and define what I mean by AI SRE in precise, engineering-grounded terms. I’ll clarify how “AI” is being used today and what it should mean in the context of reliability engineering. From there, I’ll examine where it adds value, where it falls short, and outline a more effective starting point built on structured causal reasoning that can empower the safe and effective automation of reliability work. 

SRE: One Discipline, Different Meanings 

The SRE role traces its roots to Google in the 2000s, when Ben Treynor Sloss described it as “what happens when you ask a software engineer to design an operations function.” At its core, SRE is about taking accountability for reliability at a defined level that balances user and business needs while preserving engineering velocity. 

To support this, the discipline introduced a set of practices that gave teams a measurable way to manage reliability: Service Level Objectives (SLOs) to make goals explicit, error budgets to balance those goals with the pace of change, and blameless postmortems to learn from failure. Automation followed as a way to reduce toil and eliminate human error. Together, these practices positioned reliability as an engineering problem rather than an afterthought. 

However, in practice, many teams struggle to adopt these ideals. Reliability work often takes a backseat to feature delivery, and SLOs are often only partially implemented across sprawling microservices. What Google codified in its SRE handbook often reaches production environments as a patchwork. 

Organizations also implement the role itself in different ways: embedded with product teams, focused on platform infrastructure, parachuting in as consultants, or owning services end to end. Most end up with some hybrid. 

No matter the org structure or maturity, the discipline of reliability engineering is well understood: applying engineering principles to keep systems available, performant, and predictable. While the goals are clear, most engineering leaders would agree there is still plenty of room for improvement in practice. 

What “AI” Actually Means 

Before defining the ideal AI SRE, we need to clarify what “AI” means and how the term is used today.  

In its original sense, Artificial Intelligence refers to systems capable of completing tasks that normally require human intelligence: reasoning, learning, problem-solving, perception, and natural language understanding.  

Classical AI approached these challenges by decomposing complex tasks into structured representations of knowledge, explicit planning procedures, and inference engines that could compute outcomes from evidence. For example, building a diagnostic assistant for medicine required an ontology of diseases and symptoms, probabilistic rules for how symptoms map to likely conditions, and an inference engine to suggest tests or treatments based on the evolving patient state. Historically, AI has included: 

• Rule-based systems
• Search and planning algorithms
• Probabilistic reasoning
• Constraint satisfaction

For deeper study, I recommend: 

• Artificial Intelligence: A Guide for Thinking Humans – Melanie Mitchell
• Artificial Intelligence: A Modern Approach – Stuart Russell & Peter Norvig
• The Book of Why – Judea Pearl
• Judea Pearl on Cause and Effect (Sean Carroll Podcast) 

These systems performed well when the domain was well defined and the relationships between causes, observations, and interventions could be explicitly modeled. But they broke down when faced with ambiguity, missing context, or natural language. 

In common usage, “AI” today typically refers to large language models (LLMs), and in some cases, smaller-scale variants known as small language models (SLMs). These models generate language by predicting the most likely next word given the previous context, using patterns learned from massive training datasets. They have become popular because they handle a wide range of natural language tasks with impressive fluency, such as text generation, summarization, question answering, and code completion.  

But while language models are effective at producing language and code, they have no inherent understanding of system state, live telemetry, or causal dependencies.  

Why Language Models Are the Wrong Starting Point for Reliability Engineering 

Reliability engineering requires exactly this kind of causal reasoning, which is why language models alone are the wrong foundation. The irony is that in its foundational sense, AI is a natural fit for reliability engineering: diagnosing outages, predicting failures, and recommending fixes are structured decision problems that align well with decades-old AI techniques. But when “AI” is reduced to LLMs and SLMs, the fit becomes like shoving a square peg through a round hole. 

Surveying the market, most self-proclaimed AI SRE approaches fall into one or more of these categories: 

• Postmortem narratives – After-the-fact write-ups shaped as much by bias as data, crafted to explain what went wrong in hindsight. 
• Correlation engines – Systems that surface co-occurring anomalies and related events during incidents but conflate correlation with causation. 
• Data-fetching assistants – Interfaces that summarize telemetry and suggest plausible explanations without guarantees they are correct or verifiable. Some resemble traditional automation engines, requiring heavy setup in data formatting, rules, and conditions before delivering value. 

When language models are used as the foundation for reliability engineering, four weaknesses undermine their effectiveness: 

• Spurious causes – Coherent but incorrect diagnoses from hallucination, logical inconsistency, or lack of live-environment awareness. 
• Unprincipled reasoning – Mimicking the language of reasoning without performing structured inference. 
• Causal identification failures – Difficulty pinpointing causes in dynamic systems, especially when new evidence contradicts learned assumptions. 
• Runaway costs – Without precise prompting and context, LLMs consume large amounts of compute to generate answers that may still be inaccurate. 

For reliability engineering, starting with a language model is starting in the wrong place. These approaches assume you already know which signals are worth chasing, when in reality multiple noisy alerts often trace back to a single root cause. Without causal reasoning at the foundation, both humans and AI waste time chasing symptoms instead of causes. Whether you are building internally at enterprise scale or looking for an off-the-shelf capability, any viable AI SRE must begin with causal analysis as its core. 

What Effective AI for Reliability Engineering Looks Like 

An effective AI SRE isn’t just a chatbot or a rule engine. It’s a framework that combines structured causal knowledge, probabilistic inference, and agentic capabilities. This foundation enables advanced reasoning and supports automation of real reliability engineering work. 

Such a system needs at least three interdependent capabilities: 

1. A live causal representation of the environment

A domain-specific causal model that encodes how components in a distributed system can fail, how those failures propagate, and the symptoms they produce, paired with a continuously updated topology graph showing the real-time structure of services, infrastructure, and their interconnections. 

Why it matters: Replaces fuzzy LLM pattern-matching with a verifiable system map, enabling deterministic reasoning. 

Counter to LLMs: Addresses unprincipled reasoning by grounding analysis in a causal Bayesian network that models directionality as probabilities. 

Example: The model knows that a latency spike in a database layer can propagate through dependent APIs three layers away and can quantify that likelihood based on current conditions. 

2. Real-time probabilistic inference over live telemetry

Continuous ingestion of metrics, traces, and logs, mapped against the causal model to identify the most likely root cause at any given moment. This inference layer reflects both structural dependencies and observed patterns of failure propagation, updating conclusions the instant new evidence arrives. 

Why it matters: Dynamic systems change fast, with new code, new dependencies, and shifting load. LLMs cannot adapt without retraining, while a probabilistic inference engine adjusts instantly. 

Counter to LLMs: Addresses causal identification failures, especially in counterfactual scenarios where new observations contradict prior assumptions. 

Example: If a new deployment creates a previously nonexistent dependency, the model incorporates it into its reasoning immediately, without new rules or configs. 

3. Cross-attribute and cross-service dependency reasoning

Beyond linking causes to symptoms, the system maps how performance attributes such as latency, throughput, and utilization depend on one another across services and infrastructure layers. By modeling these relationships, it can trace how a change in one part of the system cascades elsewhere, identify emerging bottlenecks, and detect when operational constraints are at risk. An added benefit is a sharp reduction in alert fatigue, because by isolating the true point of failure, only the responsible service owner is notified rather than multiple teams chasing downstream symptoms. 

Why it matters: Many incidents stem from chains of interdependent changes, not a single fault. Modeling these relationships eliminates blind spots and false leads. 

Counter to LLMs: Addresses spurious causes by ruling out explanations that violate known constraints or dependencies. 

Example: If a queue length increase is due to a downstream service slowdown rather than local CPU saturation, the model identifies the real cause directly. 

Either way, the same foundation of causal reasoning, probabilistic inference, and dependency modeling is required. With that foundation in place, language models finally have a meaningful role to play. 

The Role of Language Models in an Effective AI SRE 

Earlier, I noted that large and small language models can add value to reliability workflows. When provided with the right context, they can help accelerate diagnosis, generate remediations, and improve resolution timelines. But on their own, they lack causal understanding. Without grounding in system state and structure, even the most advanced model tends to produce responses that are plausible but operationally useless. 

Modern AI SRE solutions must transcend summarizing symptoms or correlating metrics. To truly support autonomous site reliability, they must be able to understand complex IT environments, reason over live telemetry, and intervene intelligently to maintain SLOs and keep services healthy. AI for SRE should not be defined by language generation alone. It must include: 

• A continuously updated causal model of the environment
• The ability to perform probabilistic inference over live telemetry
• The capacity to reason across attributes, services, and dependencies
• An agentic interface that can propose or execute safe and effective actions

Taken together, these capabilities reframe AI SRE from buzzword to engineering discipline. 

Conclusion 

The conversation about AI SREs is too important to leave to buzzwords. “AI SRE” should not mean a chatbot guessing its way through your telemetry. Treating “AI” narrowly as LLMs or SLMs misses the chance to apply decades of proven AI techniques — causal reasoning, probabilistic inference, and constraint satisfaction — to one of the highest-impact opportunities in modern software engineering: autonomous service reliability. 

The future of AI SRE will not be built on pattern-matching language models alone, but on systems that can reason causally about live environments and adapt in real time. Language models then become a powerful augmentation layer. Without that foundation, AI SRE is just another buzzword. 

Whether you are an enterprise building internally and looking to integrate causal reasoning into your architecture, or a mid-market company that needs a turnkey solution, the same principle applies: causal reasoning must come first. At Causely, we provide the flexibility to address both situations - a platform that can serve as a foundation for your own build, or a ready-to-deploy solution that works out of the box. 

When an upstream API slows down or quietly rate-limits requests, your customers feel the pain long before your team finds the cause. Error rates spike, retries pile up, queues fill, and internal services appear unreliable. The hard part is proving it isn’t you. Traditional observability tools only show symptoms inside your stack — latency, 5xx errors, and timeouts across services — making it nearly impossible to tell whether the real problem is internal or hiding in an external provider.

Causely changes that. With eBPF auto-instrumentation, Causely sees client communication with external services and maps those dependencies directly into your topology and causal reasoning engine. When a provider slows down, Causely shows exactly how the impact ripples across your services and identifies the external API as the root cause. That means your team can stop chasing internal alerts and start taking action — escalating with proof in hand or triggering fallback strategies.

Watch the video to see how Causely helps you go from symptoms to root cause instantly, even when the root cause isn’t yours.

Key Takeaways

• Large language models (LLMs) in observability excel at turning high-volume telemetry such as logs, traces, and metrics into concise human-readable narratives, but they lack structural system knowledge and struggle to isolate root causes in complex distributed architectures.
• Current LLM and agentic AI approaches are prone to hallucinating plausible but incorrect explanations, mistaking symptoms for causes, and ignoring event ordering, which leads to misdiagnosis and incomplete remediation.
• Causal reasoning models service and resource dependencies explicitly, accounts for event temporality, and supports inference under partial or noisy observations, enabling more accurate root cause identification.
• Causal graphs and Bayesian inference allow for counterfactual and probabilistic reasoning, which lets engineers evaluate remediation options and their likely impact before taking action.
• Integrating LLM-based interfaces with continuously updated causal models and abductive inference engines provides a practical path to reliable, explainable, and eventually autonomous incident diagnosis and remediation in cloud native systems.

The central goal of IT operations and site reliability engineering (SRE) is to maintain the availability, reliability, and performance of services while enabling safe and rapid delivery of changes. Achieving this requires a deep understanding of how systems behave during incidents and under operational stress. Observability platforms provide the foundation for this understanding by exposing telemetry data (logs, metrics, traces) that support anomaly detection, performance analysis, and root cause investigations. However, modern applications are increasingly difficult to manage as cross-service calls, event-driven workflows, and distributed data stores introduce complex and dynamic interactions.

For instance, in July 2024, a faulty configuration update in CrowdStrike’s Falcon sensor caused widespread crashes on millions of Windows systems across industries worldwide. In another case, the 2016 removal of the tiny but widely used left-pad package from npm briefly broke thousands of builds and disrupted major websites until it was restored, revealing the fragility of transitive dependencies at scale. Whether the trigger is an external contingency or a rare emergent interaction within a highly coupled system, modern IT infrastructure can experience widespread service outages due to complex cross service dependencies. Implicit dependencies, asynchronous communication, and distributed state make it challenging to pinpoint the source of incidents or understand the chain of effects across the system.

A new class of AI-based observability solutions built on LLMs is gaining traction as they promise to simplify incident management, identify root causes, and automate remediation. These systems sift through high-volume telemetry, generate natural-language summaries based on their findings, and propose configuration or code-level changes. Additionally, with the advent of agentic AI, remediation workflows can be automated to advance the goal of self-healing environments. However, such tools remain fundamentally limited in their ability to perform root-cause analysis for modern applications. LLM-based solutions often hallucinate plausible but incorrect explanations, conflate symptoms with causes, and disregard event ordering, leading to misdiagnosis and superficial fixes.

Fundamentally, LLMs operating only on observed symptoms gleaned from telemetry are attempting to deduce root causes by traversing logs and shallow topologies. LLMs lack, however, an a priori understanding of the environment as a dynamic system with evolving interdependencies. As a result, underlying issues will persist even if symptoms are partially remediated in the short term.

Effective root-cause analysis in complex, distributed systems requires understanding the causal structure of events, services, and resources. Causal knowledge and reasoning remain critical missing components in modern AI-based observability solutions. Causal knowledge is codified into causal graphs, which model inter-service and resource dependencies explicitly. By supporting counterfactual inquiry, causal inference enables root-cause isolation and systematic remediation analysis. Augmenting LLMs and agentic AI with continuously updated causal models and an abductive inference engine (which identifies the best explanation for observed symptoms using causal reasoning), offers a path toward autonomous service reliability.

In this article, we begin by outlining the strengths of LLMs and agentic AI in observability and incident management. We then examine their limitations in performing accurate root cause analysis and driving effective remediation. Next, we introduce how causal knowledge and inference engines provide the missing context for precise incident diagnosis and response. Finally, we discuss how combining causal reasoning with AI agents enables proactive incident prevention, automated remediation, and the path toward autonomous service reliability.

The Strengths and Promise of LLMs and Agentic AI

The term "AI" has become increasingly overloaded with marketing hype and public fascination. AI now applies to everything from threshold-based alerting scripts to autonomous agents capable of planning and acting across complex workflows. For simplicity, AI solutions in the observability space can be categorized as rule-based systems, LLM-based tools, and agentic AI systems. Rule-based systems include hand-crafted logic and statistical models configured to monitor baseline deviations, detect known signal patterns, and apply threshold-based alerting across logs, metrics, and traces.

LLM-based solutions leverage the generative and language-understanding capabilities of language models to support natural-language interactions with observability data. LLMs can process unstructured telemetry such as logs, traces, and alert descriptions to generate summaries, interpret errors, and create remediation plans. Agentic AI allows LLMs to act in a managed environment by providing multi-step planning, tool-assisted execution, and direct code and configuration changes. Next, we examine the specific strengths of LLMs and agentic AI in the context of observability.

LLMs are neural architectures pretrained on large-scale corpora of natural language, code, and other text-based resources. Despite being fundamentally next-token predictors trained to model the conditional probability of text, when scaled to billions of parameters and exposed to terabytes of diverse data, LLMs become highly effective at producing coherent language and supporting a wide range of language-based tasks. Modern LLMs have been further fine-tuned for instruction-following, factual recall, code generation, and domain-specific question answering, used subsequently to explain errors, answer technical questions, and generate code, scripts, and configuration changes.

In observability contexts, LLMs can interpret complex logs and trace messages, summarize high-volume telemetry, translate natural-language queries into structured filters, and synthesize scripts or configuration changes to support remediation. Most LLM solutions rely on proprietary providers such as OpenAI and Anthropic, whose training data is opaque and often poorly aligned with specific codebases or deployment environments. More fundamentally, LLMs can only produce text. They cannot observe system state, execute commands, or take action. These limitations gave rise to agentic systems that extend LLMs with tool use, memory, and control.

Agentic AI comes closest to delivering on the speculative promise of AI. In practice, agentic systems commonly follow the ReAct framework introduced by Yao et al. (2022), which integrates reasoning and action in an interleaved loop. In this setup, the LLM generates intermediate reasoning steps, selects actions such as querying tools or retrieving information, and incorporates feedback from those actions to inform the next step. This cycle of thought, action, and observation allows the system to iteratively plan, update context, and progress toward a goal. With these capabilities, LLMs are able to write applications, solve multi-step reasoning problems, generate code based on system feedback, and interact with external services to complete goal-directed tasks.

Agentic AI shifts observability workflows from passive diagnostics to active response by predicting failure paths, initiating remediations, and executing tasks such as service restarts, configuration rollbacks, and state validation. However, current agentic systems lack a priori structural and causal models of the environment, which limits their ability to anticipate novel failure modes or explain observed behavior beyond surface-level associations. While these constraints remain, agentic AI represents a necessary step toward autonomous, tool-integrated systems capable of reasoning and acting within complex managed environments.

The ultimate promise of applying agentic AI to IT operations is autonomous service reliability. An ideal system continuously monitors telemetry, identifies potential failures, evaluates impact, and applies targeted interventions with limited human oversight. Integrated into the observability and operations stack, agentic AI should function as the control layer. It reasons over system state, coordinates diagnostics, and orchestrates remediations so that services operate reliably and in alignment with defined service level objectives (SLOs), which specify availability, latency, or other performance targets. Ultimately, autonomous service reliability reduces operational complexity, accelerates incident resolution, and improves service reliability across large-scale, dynamic environments.

On The Limitations of Modern AI and the Need for Causal Reasoning

Modern service architectures often rely on shared infrastructure and layered services, where dependencies are opaque and incident signals surface far from their origin. Imagine the simple service topology shown in Figure 1, which consists of two shared resources (R1 and R2) and a set of services (S1–S5) connected through overlapping dependencies. Now consider that we observe elevated latency and request timeouts at service S5. The underlying root cause is connection exhaustion on resource R2, which intermittently blocks new connections. This condition is not surfaced through direct telemetry because service S2, which depends on resource R2, reports only latency and timeouts without exposing the underlying resource-level failure. Tracing the issue upstream, service S3 shows increased request latency, and service S2 exhibits degraded performance. Meanwhile, service S1 also reports elevated CPU usage and latency, though its downstream service S6 remains unaffected.

An LLM-based agent begins with observable symptoms (see Figure 2). It queries telemetry, inspects logs, and follows trace spans starting at S5. Along the path through S3 and S2, it observes anomalies based on latency and request failures. It also notices performance degradation on S1 and considers it a potential contributor. The agent restarts S1 and S2 and observes that the latency and timeouts at S5 are mitigated, leading it to conclude the issue is resolved. However, the problem resurfaces once connection limits on resource R2 are hit again. This scenario illustrates two key challenges.

First, spurious signals can misdirect diagnosis by drawing attention to unrelated events. Second, some root causes cannot be directly observed through telemetry and must instead be inferred from incomplete or indirect symptoms. For instance, in this scenario, the connection exhaustion on R2 emits no direct signal. It must be inferred by reasoning over the observed symptoms across S2, S3, and S5 in combination with structural knowledge of the system. Therefore, resolving such incidents requires principled causal reasoning and structural causal knowledge.

Causal knowledge maps the various causal relationships typically found in modern service architectures. The architectural knowledge provides a structural understanding of the various resources, services, and data dependencies. Finally the causal graphs provide a probabilistic framework for inferring root causes given the observed symptoms in context of the architecture.

Causal knowledge represents the relationship between root causes and their observable symptoms. Such knowledge can be abstracted to support principled downstream reasoning over system behavior such as fault localization, impact analysis, and proactive mitigation. Causal graphs provide a formal structure for encoding this knowledge. Popularized by Judea Pearl, causal graphs are directed acyclic graphs that represent cause-effect relationships among variables. When applied to reliability engineering, causal graphs describe how specific failure conditions (e.g., memory exhaustion, resource saturation, lock contention, etc.) produce observable symptoms (e.g., latency, connection errors, service timeouts, etc.).

Unlike telemetry signals capturing only runtime observations or dependency graphs, which represent observed service call relationships, causal graphs provide a richer structural understanding of how faults propagate across services and resources. This knowledge can be utilized to support inferential reasoning and identify root causes from partially observed symptoms, even when the underlying issue is not directly visible. This can be accomplished by abductive causal reasoning.

Abductive causal reasoning provides a principled, logical, and inferential framework for identifying the most likely explanation for observed symptoms. Provided a set of plausible candidate root causes and a graphical model of their symptoms, abduction selects the cause that best accounts for the observed evidence. This approach offers several advantages for modern applications: it supports inference under partial observability, selects explanations based on causal sufficiency, and provides formal guarantees about the inferred root cause. When paired with causal Bayesian graphs extending causal graphs with probabilistic reasoning, abductive inference becomes tractable in large systems. These graphs encode prior knowledge about potential root causes and their associated symptoms, allowing the system to compute the most probable root cause without requiring prior training. Furthermore, likelihoods can be updated over time using posterior observations, enabling continuous refinement in dynamic environments.

Let’s revisit our scenario with the abductive reasoning framework (Figure 4). The framework begins with a defined set of all possible root causes. Each root cause is represented by a causal graph that maps the cause to its expected symptoms with associated prior probabilities. These priors capture the likelihood of each symptom occurring if the root cause is true, derived from historical incidents and domain expertise. The abductive process identifies which root cause best explains the observed symptoms while also accounting for those that are unobserved. It computes a likelihood score for each candidate by updating the prior probabilities of its associated symptoms with observational data.

In our scenario, the observed symptoms include timeouts at service S5 and latency at services S2 and S3, which align with the causal graph for connection exhaustion on resource R2. The graph also expects latency on S4; although it is not observed, that absence is included in the likelihood estimation. Competing explanations, such as CPU starvation on S1 or network congestion on S3, receive lower likelihoods because they either fail to explain all observed symptoms or have too few expected symptoms confirmed observationally. The key distinction between abductive causal reasoning and deductive reasoning is that abduction evaluates all candidate root causes against both the observed symptoms and the expected symptoms defined by the causal models.

The agent-based approach missed the actual root cause because it lacked structural causal knowledge and stopped at the first plausible symptom path. In contrast, the abductive process leverages causal models and Bayesian graphs to infer the most likely root cause even when observations are incomplete or include spurious symptoms. Abductive causal reasoning uses these structured probabilities to isolate the most coherent explanation despite partial observations and spurious symptoms.

Limitations of Causal Reasoning

While causal reasoning is a powerful approach, it does have limitations. Constructing causal models requires significant domain knowledge and ongoing effort. The graphs must accurately capture service and resource dependencies and need regular updates as architectures evolve. Coverage is another constraint. A reasoning engine can only work with root causes that are defined in the model. If a candidate cause is missing, the engine has no basis to infer it, which reduces its effectiveness in novel or poorly understood failure cases. There are also computational challenges. In large distributed environments, some root causes map to broad sets of symptoms. Computing conditional probabilities and running Bayesian inference across these sets can become costly, especially when multiple competing explanations must be evaluated in real time.

While these limitations do not diminish the value of causal reasoning, but they do highlight that such methods are more effective under specific circumstances. In service architectures with linear and well-understood dependencies, the root cause of an incident is often self-apparent, and a reasoning engine may not be required. The challenge arises in complex distributed environments where cross-service dependencies, asynchronous communication, and distributed state make it difficult to trace symptoms back to their causes. Humans can resolve these incidents, but the process is slow, resource-intensive, and often results in longer periods of downtime and service unavailability. Therefore, meeting this challenge requires developing more effective solutions that integrate causal reasoning with AI, moving beyond current limitations and enabling progress toward autonomous service reliability.

Towards the Promise of Autonomous Service Reliability

Causal knowledge is essential for diagnosing incidents in modern service architectures. LLM-based solutions and current agentic AI often miss the forest for the trees. These systems operate over logs, traces, and metrics, but lack the structural context required to reason comprehensively about system-level behavior. While LLMs can provide useful insights by aggregating and parsing telemetry, they are ultimately bounded by the inputs they receive. With incomplete observations, LLMs tend toward speculative explanations that often result in hallucinations. Without a causal understanding of how failures propagate through services and infrastructure, modern AI solutions cannot move beyond summarization of observed events to reliably identify underlying causes. Causal knowledge and abductive causal reasoning provide the missing keys that, when combined with LLMs and agentic AI, unlock effective identification of likely root causes from incomplete and partial observations in complex, distributed environments.

Therefore, it is necessary to augment modern LLMs with a causal reasoning engine to achieve effective incident analysis and autonomous reliability. A causal reasoning engine combines three key components: causal models that encode known root causes and their associated symptoms, Bayesian causal graphs that apply probabilistic reasoning over service topologies, and an abductive inference engine that selects the most likely root cause given partial or noisy observations. This engine can operate as an external reasoning layer, providing structured causal context that modern LLMs lack.

Drawing from the principles of neuro-symbolic reasoning, the LLM serves as the flexible language interface, while the causal reasoning engine verifies hypotheses, refines candidate root causes, and performs advanced reasoning that goes beyond the LLM’s predictive text generation. By integrating this engine, LLM-based agents can transition from surface-level incident triage to precise root-cause identification and actionable remediation, constructing a path toward proactive, autonomous service reliability.

Causal agents provide a meaningful step in the pursuit of autonomous service reliability. By integrating causal knowledge and abductive reasoning, these systems move beyond reactive response and manual triage. They enable proactive incident prevention by identifying emerging risks, support targeted remediation through structural awareness, and drive self-healing by isolating and addressing root causes without human intervention. The result is a system that not only detects symptoms but understands their context. This shift reduces downtime, accelerates resolution, and aids teams in managing reliability at scale with minimal manual intervention.

OTel’s Established Best Practices for Logging

The OTel project has laid a strong foundation for logs to work alongside metrics and traces: 

• Consistent Structure – OTel’s log data model ensures every log record has a predictable shape, making it machine-parseable and easier to route or query.  
• Context propagation – By embedding trace IDs, span IDs, and resource attributes in every log entry, OTel makes it possible to pivot seamlessly between logs, traces, and metrics in your analysis tools, without guesswork.  
• Unified Collection – The OTel Collector can ingest logs from multiple sources — applications, infrastructure, third-party services — and normalize them before routing to storage or analysis tools. 
• Enrichment and Filtering – Processors in the Collector allow you to enrich logs with metadata (region, environment, service version) and filter out noise or limit verbosity. 

If your team is following these practices, congratulations: yYou already have the building blocks for the next step. 

Why Logs Need Context to Be Useful 

Logs are essential to troubleshooting — they carry the fine-grained evidence of what went wrong. But in large, OTel-instrumented systems, collecting every log line from every service creates new challenges: 

• Mountains of data – terabytes of logs that make queries slow and expensive. 

• Information overload – too many log lines without guidance on which ones matter. 

• Slow incident response – engineers spend precious time searching instead of fixing. 

The goal isn’t to reduce the importance of logs — it’s to use them in context. By correlating logs with metrics and traces, you can elevate logs from raw text into causal signals: proof tied directly to the root cause of a problem. 
 
Instead of asking engineers to manually dig through a warehouse of logs, the right approach is to automatically pull the relevant 0.1% of logs that explain the incident. Logs don’t lose their role; they gain clarity by being connected to the surrounding signals. 

Tutorial: Implementing Contextual Logs in OTel 

Here’s a practical outline for how to set this up today using OTel: 

Step 1 – Ensure Rich Context in Logs 

Start with a simple log statement, for example: 

log.warn("Database connection pool exhausted – queries may fail");

On its own, this message is useful but limited. In a large distributed system, you need more context to understand which service emitted it, in what environment, and how it relates to the rest of a trace. 

That’s where OTel comes in. When the OTel SDK is properly configured, it automatically enriches log records with resource attributes (service name, version, environment, region, etc.) and correlation identifiers (trace_id, span_id). 

For example, the same log record might look like this once enriched: 

resource: 
  attributes: 
    service.name: "checkout-service" 
    service.version: "1.2.3" 
    deployment.environment: "production" 
    cloud.region: "us-east-1" 
trace_id: "91be8f92d45e471ea0bf1c25be8e3f1c" 
span_id: "2d4b1c73c9c3e7f0" 
severity_text: "WARN" 
body: "Database connection pool exhausted – queries may fail"

This enrichment happens automatically if you’ve instrumented with OTel, which means developers don’t have to manually add metadata. They just log as usual, and OTel ensures those logs are correlatable with traces and metrics. 

Step 2 – Collect and Route Logs Through the OTel Collector 

service: 
  pipelines: 
    logs: 
      receivers: [otlp, filelog] 
      processors: [attributes, filter] 
      exporters: [otlphttp]

Step 3 – Use Metrics/Traces to Narrow the Scope 

In your analysis tool, start with: 

• Error rate, latency, or saturation metrics to find affected services.
• Trace waterfalls to spot where requests slow down or fail. 

Step 4 – Pull Only Relevant Logs 

service.name = "checkout-service" 
AND timestamp >= "2025-02-10T14:03:00Z" 
AND timestamp <= "2025-02-10T14:08:00Z"

If using the OTel Collector with filtering, you can even forward only scoped logs to your log store during incidents. 

Step 5 – Confirm and Remediate 

Use the filtered logs to confirm the root cause hypothesis, capture the evidence you need for a fix, and move directly to remediation. 

How Causely Operationalizes This Workflow 

The tutorial above shows one way to approximate a more efficient log workflow by narrowing scope before querying logs. Causely builds on this approach by treating logs as integral to causal analysis — automatically surfacing the ones tied to the root cause. 

Here’s how we do it automatically: 

• Top-down causal reasoning – Causely continuously builds a model of your environment’s dependencies, behaviors, and known failure patterns.  
• Automatic root cause inference with contextual logs – Metrics, traces, logs, and topology context are analyzed together in real time to pinpoint the single most likely root cause. Relevant error logs are automatically surfaced in that context — tied to the specific service, time window, and failure mode.  
• Remediation specificity – Contextual logs and causal context are routed to an LLM, producing precise and actionable remediation paths. 
• Bring your own logs – Whether shipped via OTel or another pipeline, Causely ingests your logs and automatically pulls the right ~0.1% on demand. No pipeline reconfiguration or log warehouse required. 

The result: Logs remain first-class citizens. You see them in context of metrics and traces, as proof and evidence of the root cause and remediation path — not a warehouse to sift through, and not a second-class afterthought. 

Wrapping Up 

The OTel community has done the hard work of making logs consistent, enriched, and correlated with other telemetry. The next step is using them more effectively: elevating logs from raw records to causal signals. That shift makes troubleshooting faster, cheaper, and more reliable. 
 
If you’re following OTel logging best practices, you’ve laid the foundation. With Causely, you can take it to the next level: automatically inferring the root cause in real time, surfacing logs in context, and generating a clear remediation path. All without workflow changes, pipeline reconfiguration, or trade-offs between log richness and operational efficiency. 

This post explores four architecture patterns where standalone Docker is not only justified but recommended. We’ll dive into the technical reasons behind each pattern, explore the operational tradeoffs, and show how Causely reduces the complexity of managing performance across these hybrid systems.  

Architecture Pattern #1: Edge & Performance‑Critical Standalone Docker Services 

Use Case:

Real-time inference, inline data transformation, high-throughput gateways, protocol termination (e.g., gRPC or TLS offloading) 

Why Docker Makes Sense: 

• These services often sit at the ingress point of traffic or at the edge of the network, where ultra-low latency and deterministic behavior are required. 
• Standalone Docker provides tight control over CPU affinity, memory isolation, and scheduling—where Kubernetes can introduce variability through abstraction. 
• With no orchestrator overhead and direct access to the host network or hardware, teams can squeeze out performance optimizations critical to meeting service level expectations.  
• Example: A fraud scoring engine on a GPU-backed instance running outside the cluster for sub-10ms response time. 

Tradeoffs: 

• Requires homegrown automation for lifecycle management. 
• Correlating failures to upstream/downstream systems can be more difficult without orchestrated metadata. 

When to Consider This:

When you need deterministic performance, such as in ML inference, high-frequency trading, or streaming telemetry ingestion. 

Architecture Pattern #2: Specialized, Customer‑Specific Components  

Use Case: Multi-tenant SaaS platforms or platforms with dedicated microservices per enterprise customer or vertical 

Why Docker Makes Sense: 

• Standalone containers allow per-customer isolation without polluting the core K8s control plane. 
• Services can be versioned, tuned, and deployed independently, which is ideal when customers have differing SLAs, data residency needs, or workload profiles. 
• Enables progressive feature rollout, compliance isolation (e.g., GDPR), or regional data processing without touching the shared cluster. 
• Example: Deploying a custom analytics pipeline for an EU customer with stricter data locality rules. 

Tradeoffs: 

• More surface area to monitor and secure. 
• Requires tooling to manage container sprawl and avoid operational drift. 

When to Consider This:

When serving enterprise customers with tailored requirements or isolating sensitive data flows.  

Architecture Pattern #3: Asynchronous Messaging Backbone 

Use Case:

Internal event buses, Kafka sidecars, queue processors, or background workers handling fan-out/fan-in traffic patterns 

Why Docker Makes Sense: 

• These components often require tight control over retry logic, dead-letter queues, and message deduplication strategies. 
• Docker allows dedicated tuning of each message processor (e.g., CPU throttling, JVM heap tuning) without being subject to shared limits imposed by K8s resource quotas. 
• Services can scale independently of the producers and consumers managed in K8s. 
• Supports use cases like Command Query Responsibility Segregation (CQRS), or event sourcing, where replay behavior and ordering matter.  
• Example: Kafka sidecar processors that transform and route event data before entering the service mesh. 

Tradeoffs: 

• Failures can ripple across multiple services but be misattributed.  
• Requires deeper visibility into both the message layer and consuming systems. 

When to Consider This:

When messaging is your system’s backbone and needs dedicated, configurable components.  

Architecture Pattern #4: CI/CD, Build & Test Workloads 

Use Case:

Ephemeral build/test containers in CI pipelines, sandbox environments for pre-merge validation 

Why Docker Makes Sense: 

• CI systems like GitHub Actions, CircleCI, and GitLab are built around standalone container workloads. 
• Docker ensures parity between dev and test environments, reducing false positives from environment drift. 
• Local development with Docker Compose can mirror production service graphs with mocking or synthetic traffic. 
• Example: nightly regression tests that spin up mocked API services using Docker Compose, inject synthetic latency, and validate SLA adherence before promoting a build to staging. 

Tradeoffs: 

• Pre-prod performance issues may not emerge until you shift into the orchestrated production environment unless your test setup includes sufficient load or concurrency. 

When to Consider This:

If you need fast, reliable CI loops, or are troubleshooting performance regressions before services hit production.  

Why Performance Management Gets Hard in These Hybrid Architectures 

In a system composed of Kubernetes clusters, standalone Docker containers, and managed services like MongoDB or Kafka, performance issues don’t respect boundaries. 

Symptoms Emerge Far From Their Root Cause:  

• A degraded Kafka sidecar causes timeouts in upstream K8s services.  
• A CPU-hungry Docker-based customer module introduces latency that looks like a DB issue. 
• A slow edge container obscures the real issue in a downstream managed Redis instance. 

These Environments Break Traditional Monitoring Assumptions: 

• No single telemetry system captures the full execution path.  
• Logs, metrics, and traces must be stitched manually across platforms.  
• Minute-level delay in diagnosis equals dollars lost. 

You need to know not just what failed, but why. And you need it now. 

eBPF-Based Auto-Instrumentation: Zero-Overhead Insight 

eBPF provides kernel-level insight into what containers are doing, without needing in-app instrumentation, SDKs, or manual agents. 

In standalone Docker environments, Causely uses eBPF to:  

• Capture syscall activity, I/O latency, CPU/memory contention, and network issues directly from the host.  
• Observe containers even if they’re running outside Kubernetes or disconnected from the service mesh.  
• Collect minimal, high-value telemetry without generating a flood of low-signal noise. 

And because eBPF instrumentation is automatic,  

• There’s no overhead to developers 
• Visibility is always on 
• And you can trace performance across heterogeneous systems without writing a line of config. 

Causely also enables end-to-end tracing across Kubernetes, Docker, and managed environments—stitching together execution flows in real time and letting you see how a single failure propagates. 

Causely Makes Standalone Docker First-Class 

While eBPF gives us the raw signals, Causely’s causal reasoning engine makes them useful. 

• It applies a growing library of causal models which capture failure patterns across services, queues, and infrastructure. 
• It builds a real-time causal graph from observed behavior. 
• It reasons probabilistically to pinpoint the root cause—even across multiple failure domains. 

We Treat Docker Containers as Full Participants in the Production Graph

With Causely, you can:  

• Automatically discover Docker-hosted services and map their upstream/downstream relationships.  
• Trace symptoms across Kubernetes, Docker, and managed services without needing to standardize instrumentation.  
• Get clear, prioritized, explainable root causes in real time. 

You don’t have to fear hybrid complexity; you just need a system that understands it. 

Ready to Make Standalone Docker Part of Your Reliability Strategy, Not a Blind Spot? 

Causely installs in minutes. With our automated instrumentation and causal analysis, you’ll spend less time debugging and more time building. 

Learn more about how to add stand-alone Docker to your Causely deployment.  

When a message queue starts to back up, every second counts. One slow consumer can stall a Kafka topic, delay downstream services, and turn healthy SLOs into a wall of red.

The hard part? Your observability tools only show the symptoms: rising lag, service timeouts, and growing error rates. With dozens of potential culprits — from a code change on one service to a misconfigured broker or even a hidden infrastructure issue — finding the exact cause can feel like guesswork.

Causely takes the uncertainty out of the equation. By combining deep domain knowledge with your Kafka topology and real-time telemetry, Causely identifies the precise reason behind the lag — whether it’s inefficient garbage collection, resource contention, or broker congestion — and shows you exactly how it’s impacting your services. No endless alert-chasing. No service-by-service elimination. Just clear, evidence-backed answers so your team can act fast and keep the backlog from turning into a crisis.

Watch the video to see how Causely turns “Lag High” chaos into confident, informed action in seconds.

We'll explore five key techniques that power automatic instrumentation: monkey patching, bytecode instrumentation, compile-time instrumentation, eBPF, and language runtime APIs. Each technique leverages the unique characteristics of different programming languages and runtime environments to add observability without code changes.

What is Automatic Instrumentation?

According to the OpenTelemetry glossary, automatic instrumentation refers to “telemetry collection methods that do not require the end-user to modify application’s source code. Methods vary by programming language, and examples include bytecode injection or monkey patching.”

It’s worth noting that “automatic instrumentation” is often used to describe two related but distinct concepts. In the definition above and in this blog post, it refers to the specific techniques (like bytecode injection or monkey patching) that can be used to enable observability without code changes. However, when people use "automatic instrumentation" in conversations, they often mean complete zero-code solutions like the OpenTelemetry Java agent.

The distinction is important: there's actually a three-layer hierarchy here. At the bottom are the automatic instrumentation techniques (bytecode injection, monkey patching, etc.) that we explore in this blog post. These techniques are used by instrumentation libraries that target specific frameworks, for example, libraries that instrument Spring and Spring Boot, Express.js, Laravel, or other popular frameworks. Finally, complete solutions like the OpenTelemetry Java agent bundle these instrumentation libraries together and add all the boilerplate configuration for exporters, samplers, and other building blocks.

There are ongoing debates in the observability community about the right terminology, and this blog post won’t attempt to resolve those discussions.

Also note that what appears "automatic" to one person might be "manual" to another: if a library developer integrates the OpenTelemetry API into their code, the users of that library will get traces, logs, and metrics from that library “automatically” when they add the OpenTelemetry SDK to their application.

Want to try the techniques yourself?

This blog post contains small code snippets to illustrate the concepts. A full working examples can be found in a lab repository at https://github.com/causely-oss/automatic-instrumentation-lab where you can try them out yourself.

Before we explore these techniques, it’s important to note that you should not build your own automatic instrumentation from scratch, especially not using this blog post as a blueprint. The examples here are simplified for educational purposes and skip many complex details that you would encounter in real-world implementations. There are established tools and mechanisms available that handle much of the complexity and edge cases you would face when building instrumentation from the ground up. If you’re interested in diving deeper into this field, the best approach is to contribute to existing projects like OpenTelemetry, where you can learn from experienced maintainers and work with production-ready code.

Automatic Instrumentation Techniques

Now let’s explore how these techniques work under the hood.

Monkey Patching: Runtime Function Replacement

Monkey patching is perhaps the most straightforward automatic instrumentation technique, commonly used in dynamic languages like JavaScript, Python, and Ruby. The concept is simple: at runtime, we replace existing functions with instrumented versions that inject telemetry before and after calling the original function.

Here's how this works in Node.js:

const originalFunction = exports.functionName;

function instrumentedFunction(...args) {  
  const startTime = process.hrtime.bigint();  
  const result = originalFunction.apply(this, args); 
  const duration = process.hrtime.bigint() - startTime;   
  console.log(`functionName(${args[0]}) took ${duration} nanoseconds`);  
  return result;
}

exports.functionName = instrumentedFunction;

The require-in-the-middle library allows us to perform this replacement at module load time, intercepting the module loading process to modify the exported functions before they’re used by the application:

const hook = require("require-in-the-middle");
hook(["moduleName"], (exports, name, basedir) => {
  const functionName = exports.fibonacci; 
  ...
  exports.functionName = instrumentedFunction;
  return exports;
});

However, monkey patching has limitations. It can't instrument code that's already been compiled to machine code, and it may not work with functions that are called before the instrumentation is loaded. Additionally, the overhead of function wrapping can be significant for performance-critical applications. Monkey patching is also brittle when the implementation of the instrumented code changes significantly, as the instrumentation code needs to be updated to match the new interface.

To try this out yourself, take a look at the Node.js example from the lab.

If you’d like to see actively used implementations of monkey patching, you can take a look into the instrumentation libraries provided by OpenTelemetry for JavaScript or Python.

Bytecode Instrumentation: Modifying the Virtual Machine

For languages that run on virtual machines, bytecode instrumentation offers a powerful approach. This technique works by modifying the compiled bytecode as it’s loaded by the virtual machine, allowing us to inject code at the instruction level.

Java’s Instrumentation API provides the foundation for this approach. When a Java agent is specified with the -javaagent flag, the JVM calls the agent’s premain method before the main application starts. This gives us the opportunity to register a class transformer that can modify any class as it’s loaded.

public static void premain(String args, Instrumentation inst) {
    new AgentBuilder.Default()
        .type(ElementMatchers.nameStartsWith("com.example.TargetApp"))
        .transform((builder, typeDescription, classLoader, module, protectionDomain) ->
            builder.method(ElementMatchers.named("targetMethod"))
                   .intercept(MethodDelegation.to(MethodInterceptor.class))
        ).installOn(inst);
}

The interceptor then wraps the original method call with timing logic:

@RuntimeType
public static Object intercept(@Origin String methodName,
                            @AllArguments Object[] args,
                            @SuperCall Callable<?> callable) throws Exception {
    long startTime = System.nanoTime();
    Object result = callable.call();
    long duration = System.nanoTime() - startTime;
   
    System.out.printf("targetMethod(%s) took %d ns%n", args[0], duration);
    return result;
}

Bytecode instrumentation is particularly powerful because it works at the JVM level, making it language-agnostic within the JVM ecosystem. It can instrument Java, Kotlin, Scala, and other JVM languages without modification.

The main advantage of bytecode instrumentation is its comprehensive coverage—it can instrument any code that runs on the JVM, including code loaded dynamically or from external sources. However, it comes with some overhead due to the bytecode transformation process.

In real implementations, ByteBuddy is the go-to library for bytecode instrumentation in Java, providing a powerful and flexible API for creating Java agents. It abstracts away much of the complexity of bytecode manipulation and provides a clean, type-safe way to define instrumentation rules.

To try this out yourself, take a look at the Java example from the lab.

If you’d like to see actively used implementations of byte code instrumentation, you can take a look into the instrumentation libraries provided by OpenTelemetry for Java or .NET.

Compile-Time Instrumentation: Baking Observability into the Binary

For statically compiled languages like Go, compile-time instrumentation offers a different approach. Instead of modifying code at runtime, we transform the source code during the build process using Abstract Syntax Tree (AST) manipulation.

The process involves parsing the source code into an AST, modifying the tree to add instrumentation code, and then generating the modified source code before compilation. This approach ensures that the instrumentation is baked into the final binary, providing zero runtime overhead for the instrumentation mechanism itself.

func instrumentFunction() {
    fset := token.NewFileSet()
    file, err := parser.ParseFile(fset, "app/target.go", nil, parser.ParseComments)
   
    // Find the target function and add timing logic
    ast.Inspect(file, func(n ast.Node) bool {
        if fn, ok := n.(*ast.FuncDecl); ok && fn.Name.Name == "targetFunction" {
            // Add defer statement for timing
            deferStmt := &ast.DeferStmt{
                Call: &ast.CallExpr{
                    Fun: &ast.CallExpr{
                        Fun: &ast.Ident{Name: "trace_targetFunction"},
                    },
                },
            }
            fn.Body.List = append([]ast.Stmt{deferStmt}, fn.Body.List...)
        }
        return true
    })
   
    // Write the modified file back
    printer.Fprint(f, fset, file)
}

Compile-time instrumentation has several advantages. It provides zero runtime overhead for the instrumentation mechanism, and the resulting binary contains all the code it needs. This approach works well with compiled languages and can be integrated into existing build processes.

That said, it does come with trade-offs. It requires access to the source code and build system, which makes it impractical for instrumenting third-party applications or libraries. It also demands more sophisticated tooling to manipulate the abstract syntax tree (AST) correctly and consistently adding complexity to the build pipeline and potentially requiring changes to your CI/CD workflows.

To try this out yourself, take a look at the Go compile-time example from the lab.

If you’d like to see actively used implementations of compile-time instrumentation, you can take a look into the OpenTelemetry Go Compile Instrumentation project.

eBPF Instrumentation: Kernel-Level Observability

eBPF (Extended Berkeley Packet Filter) represents a fundamentally different approach to automatic instrumentation. Instead of modifying application code or bytecode, eBPF works at the kernel level, attaching probes to function entry and exit points in the running application.

eBPF programs are small, safe programs that run in the kernel and can observe system calls, function calls, and other events. For automatic instrumentation, we use uprobes (user-space probes) to attach to specific functions in our application.

#!/usr/bin/env bpftrace

uprobe:/app/fibonacci:main.fibonacci
{
    @start[tid] = nsecs;
}

uretprobe:/app/fibonacci:main.fibonacci /@start[tid]/
{
    $delta = nsecs - @start[tid];
    printf("fibonacci() duration: %d ns\n", $delta);
    delete(@start[tid]);
}

This bpftrace script attaches a probe to the function in our application. When the function is called, it records the start time. When the function returns, it calculates the duration and prints the result.

eBPF instrumentation is language-agnostic and works with any language running on Linux. It provides deep system-level observability without requiring any modifications to the application code or build process. The overhead is minimal since the instrumentation runs in the kernel.

However, eBPF instrumentation has some limitations. It requires Linux and root privileges to run, making it less suitable for containerized environments or applications that can’t run with elevated permissions.

For real-world use cases, bpftrace is just one of many eBPF tools available. While it’s excellent for learning and prototyping, production environments typically use more sophisticated frameworks like BCC (BPF Compiler Collection) or libbpf, which provide better performance, more features, and stronger safety guarantees.

To try this out yourself, take a look at the Go eBPF example from the lab.

If you’d like to see actively used implementations of compile-time instrumentation, you can take a look into the OpenTelemetry eBPF Instrumentation project (short “OBI”), which is the outcome of the donation of Belay by Grafana.

Language Runtime APIs: Native Instrumentation Support

Some languages provide built-in APIs for instrumentation, offering a more integrated approach. PHP’s Observer API, introduced in PHP 8.0, is a prime example of this approach.

The Observer API allows C extensions to hook into the PHP engine’s execution flow at the Zend engine level. This provides deep visibility into PHP application behavior without requiring code modifications.

static void observer_begin(zend_execute_data *execute_data) {
    if (execute_data->func && execute_data->func->common.function_name) {
        const char *function_name = ZSTR_VAL(execute_data->func->common.function_name);
        if (strcmp(function_name, "fib") == 0) {
            start_time = clock();
        }
    }
}

static void observer_end(zend_execute_data *execute_data, zval *retval) {
    if (execute_data->func && execute_data->func->common.function_name) {
        const char *function_name = ZSTR_VAL(execute_data->func->common.function_name);
        if (strcmp(function_name, "fib") == 0) {
            clock_t end_time = clock();
            double duration = (double)(end_time - start_time) / CLOCKS_PER_SEC * 1000;
            php_printf("Function %s() took %.2f ms\n", function_name, duration);
        }
    }
}

The Observer API provides a clean, supported way to add instrumentation to PHP applications. It operates at the language runtime level, similar to how other languages implement their instrumentation APIs. This approach is efficient and well-integrated with the language ecosystem.

However, it requires writing C extensions, which adds complexity and makes it less accessible to developers who aren’t familiar with C or PHP’s internal APIs. It’s also specific to PHP, so the knowledge doesn’t transfer to other languages.

To try this out yourself, take a look at the PHP Observer API example from the lab.

If you’d like to see actively used implementations of byte code instrumentation, you can take a look into the instrumentation libraries provided by OpenTelemetry for PHP.

A Note on Context Propagation

While we've covered the core techniques of automatic instrumentation, there's an important aspect we haven't discussed: context propagation. This involves injecting trace context information (trace IDs, span IDs) into HTTP headers, message metadata, and other communication channels to enable distributed tracing across service boundaries.

Unlike the purely observational techniques we've explored, context propagation actively modifies your application's behavior by altering data transmitted across service boundaries. This introduces additional complexity that deserves its own dedicated blog post.

Automatic Instrumentation as telemetry quality assurance

At Causely, we rely on automatic instrumentation to ensure consistent, high-quality telemetry, even when customers haven't instrumented their code manually. Our agent comes with automatic instrumentation powered by Grafana Beyla (now also known as OpenTelemetry eBPF Instrumentation), leveraging eBPF and OpenTelemetry out-of-the-box to ensure fast time-to-value and dependable insights from day one.

Conclusion

We've explored the core techniques behind automatic instrumentation, from monkey patching to bytecode instrumentation to eBPF probes. Each approach leverages the unique characteristics of different programming languages and runtime environments.

These techniques power production observability tools like OpenTelemetry, enabling developers to quickly add telemetry without modifying source code. The most successful observability strategies combine automatic and manual instrumentation: automatic instrumentation provides broad coverage for common patterns, while manual instrumentation captures business-specific metrics.

If you'd like to try out these techniques yourself, you can use the Automatic Instrumentation Lab.

If you're interested in contributing to these technologies, consider getting involved with OpenTelemetry's various Special Interest Groups (SIGs).

When dozens of teams are deploying code across services and environments, even a small change can quietly spiral into system-wide latency, retries, and customer pain.

And when things go sideways, your team is left scrambling: 

Was it the payments API?
A config change in the cluster?
A new version of Postgres? 

In most tools, you’re chasing alerts across dashboards trying to guess your way to the root cause. That’s why we built Causely — to surface the answer before the war room even begins.

In this short video, we show how Causely pinpoints the exact code change that triggered cascading performance issues — without requiring you to sift through logs or build custom dashboards. Our Causal Reasoning Engine continuously maps relationships between telemetry, deploys, and system behavior to deliver real-time root cause insight. So when your team asks “what changed?”, Causely already knows — and shows you what to fix, what was impacted, and who needs to take action.

In my recent talk at OpenTelemetry Community Day, “The Signal in the Storm: Practical Strategies for Managing Telemetry Overload,” I laid out a different path forward, one focused not on volume, but on meaning. 

More telemetry doesn’t guarantee more understanding. In many cases, it gives you the illusion of control while silently eroding your ability to reason about the system. That illusion becomes expensive, especially when telemetry pipelines are optimized for ingestion, not insight. 

Observability Needs a Better Model 

Traditional observability relies on emitting and aggregating raw signals (spans, logs, and metrics), then querying across that pile post-hoc. That model assumes the data will be useful after something goes wrong. But today’s distributed, dynamic, multi-tenant AI-driven systems don’t give you that luxury. The cost of collecting all raw signals without semantics and without context is too high. 

We need a shift: from streams of telemetry to structured, semantic representations of how systems behave. That starts by modeling the actual components of the system (entities) and the relationships between them. Not as metadata bolted onto spans, but as first-class signals. This is the work being advanced by the OpenTelemetry Entities SIG, and it's central to how we think about observability at Causely. 

What It Looks Like in Practice 

At Causely, we’ve been applying these ideas in production. We observe system behavior, and instead of centralizing all that raw output, we extract anomalies in the context of semantic entities and relationship graphs at the edge. This gives us fine-grained insight without overwhelming the pipeline. 

The impact is real: 

• Less data, more clarity: structured signals replace noisy aggregates 
• Better performance and cost-efficiency: telemetry becomes lean and targeted 
• Stronger privacy: raw, user-level data never needs to leave the cluster 
• Faster debugging: understanding is built-in, not reverse-engineered 

The goal isn’t to eliminate data; it’s to collect with intent. To make the system itself understandable, not just observable. 

Watch the Talk 

In the session, I go deep into what this looks like technically, including: 

• How teams are adapting their telemetry strategies to manage scale and cost 
• The role of entities, ontologies, and semantic modeling in modern observability 
• Why centralized data lakes aren’t a sustainable long-term answer 
• What we’ve learned building systems that can reason about their own behavior 

If you’ve felt the limits of traditional observability, and you're looking for a more scalable, reliable, and thoughtful path forward, I hope you’ll check it out. 

See the slides, or watch the recording:

You did the right things. 

• Your microservices are fully instrumented. 
• You’ve got distributed tracing and a modern observability stack.  
• You even built custom dashboards. 

But during a major production incident, it still took hours to figure out what was wrong. 

Sound familiar? 

In our recent webinar, 'Rethinking Reliability for Distributed Systems,' Causely co-founder Endre Sara shared a story we hear far too often: a large-scale customer, running mature microservices in Kubernetes with full observability coverage, still struggles to understand what’s broken during a high-stakes business event. 

Why microservices break differently 

Distributed systems aren't just complex, they're dynamic. Services spin up and down. Async data flows hide causal relationships. Teams own different pieces. Dashboards fill up with symptoms, not answers. 

That's what happened to a large enterprise team Endre worked with. They had: 

• Mature Kubernetes operations 
• Kafka for async comms 
• Comprehensive tracing + telemetry 
• A seasoned SRE team 

And still, they couldn't find the root cause of a high-stakes incident. 

The problem wasn't a lack of data. It was a lack of causal reasoning.  

Watch the recording: Rethinking Reliability for Distributed Systems 

In this session, Endre walks through: 

• Why microservices environments overwhelm traditional observability 
• How causal reasoning changes incident response 
• What teams can do to move from firefighting to foresight 

Whether you’re drowning in alerts or struggling to explain why something broke, this talk offers a clear new perspective and a path forward.  

When something breaks in production, most tools make you start with a question: What’s going on? But with Ask Causely, you start with the answer.

Powered by our real-time Causal Reasoning Engine, Causely continuously analyzes your telemetry and system topology to surface the root cause the moment an issue begins — no prompt required. Unlike other AI assistants that reactively dig through logs or metrics after you ask, Causely already knows what’s wrong, why it’s happening, and what’s likely to break next.

In this short demo, we show how Ask Causely shifts incident response from a fire drill to a focused, high-context workflow. Whether you're on-call, in a war room, or mid-debugging, Ask Causely can tell you who owns the failing service, what downstream systems are impacted, and what to do next — all from Slack, Teams, or your browser. It’s observability that works like a teammate, not just a tool.

Watch the video to see how your team can go from alerts to action in seconds.

Let’s Not Rename Observability — Let’s Make It Work 

Every few months, a new term pops up: understandability, explainability, controllability. And sure, we all want better systems. But do we really need new language? Or do we need better outcomes? 

As I said in the panel: six people will have twelve opinions on what observability means. But the real point is this: users don’t care what we call it! They want to catch root causes in production early and minimize the business impact during an incident. And that requires systems that deliver causal, actionable insights that allow teams to return their software into a healthy state quickly. 

Renaming observability and packaging it as something else is not conducive to getting to the true outcome everyone wants – they are just words without the proper action and change to back it up. We saw this before, that “monitoring” was replaced with “observability” without no actual change. 

Observability should support the full software lifecycle, from design to incident to fix. If it’s not helping you move faster and safer, then it’s just more dashboards. And no one wants that. Let’s not get stuck in terminology and instead get back to building systems that help us move. 

Value Over Volume 

One of the loudest themes from the roundtable: cost. Observability spend is skyrocketing, while the value people get is… questionable. 

We are stuck in a cycle of collecting everything, just in case! But volume does not equal value. More logs, more traces, more storage doesn’t solve problems, it mostly just adds noise. 

Any vendor claiming to innovate in observability has to answer this question: how do you shift the focus from collecting more data to instead only delivering useful insights? 

Causely’s answer: we do mediation at the cluster (i.e., edge-based processing), and only send distilled insights to the cloud . Our system doesn’t aim to collect all the data. It aims to understand what’s wrong, fast. That means we’re not just watching metrics, we’re diagnosing causal chains. That’s how we make observability affordable, and more importantly useful. 

From Optional to Invisible: The Future of OpenTelemetry 

During the roundtable I said that OpenTelemetry will have won when people use it without realizing it. When the libraries, frameworks and programming languages that we use every day come with an out of the box integration. Developers will use it like any other core feature of their language, like if-statements, variables and comments. 

That said, this vision is more of a stretch goal than a milestone on the horizon. In many ways, OpenTelemetry has already “won.” Making code observable, whether through OpenTelemetry or another system, is no longer optional, it is expected. Observability has become a baseline capability, and OpenTelemetry helped set that standard. 

Smart Automation, Not AI Hype 

A few years ago, AIOps promised to make sense of our systems with artificial intelligence and mostly delivered confusion. The hype faded, the promises didn’t hold up, and most teams were left with “smart” alert suppressors or dashboards that looked impressive but didn’t help when it counted. 

Today’s AI wave is louder: LLMs that summarize incidents, tools that promise auto-remediation with natural language, anomaly detectors wrapped in glossy UI. But most of them suffer from the same core flaw: LLMs are general-purpose tools, they excel at pattern recognition, but lack deep real-time causal reasoning needed to adopt to novel, dynamic environments. 

At Causely, we’re not building a chatbot. We’re not doing log clustering or time-series correlation. We’re building a causal reasoning system. 

Our system encodes known failure modes as structured models. That means it can infer root causes even when the triggering signal isn’t directly observable. It doesn’t just suppress noise; it explains it. It doesn’t just summarize symptoms; it traces causality. 

This isn’t black-box “AI.” It’s smart, explainable automation rooted in mathematics — built to help engineers understand why things happen and what to do next. 

And the goal isn’t to replace humans! It’s to get them out of the loop for the boring stuff: the CPU spike, the misconfigured downstream service, the memory leak that shows up every Tuesday. These aren’t mysteries. They’re patterns. And they can be handled automatically.  

That way engineers can spend time on doing what they love, building.   

Closing Thoughts 

The roundtable showed there’s still strong alignment across the industry: observability remains essential, but it needs to deliver more than just data. We need clearer value, smarter automation, and systems that help us move faster, not just monitor more. 

Thanks to Adam, Charity, Paige, Shahar, and Avi for a thoughtful and honest discussion. It’s good to see real progress, and even better to debate what is coming next. 

Grafana gives teams the power to visualize everything - but on Day 0, when your dashboards are live and alerts start firing, what your team really needs is clarity. That’s why we built the new Causely plugin for Grafana. In just minutes, Causely connects to your telemetry sources and begins surfacing the root cause of performance degradations - right inside your existing dashboards. No code changes. No sidecars. Just answers.

In this video, you’ll see how Causely helps teams cut through the noise of modern microservice environments by pinpointing why systems are breaking, not just where. Our Causal Reasoning Engine builds a real-time cause-and-effect map of your services - so instead of drowning in symptoms, your team is focused on what matters most. Watch the video to see how Causely helps you deliver real insight from Day 0 - and drive better reliability from Day 1.

But here is the problem: none of these are actually solving the Root Cause Analysis problem. Alarm suppression is NOT Root Cause Analysis.  

For teams operating modern, distributed systems - microservices, data pipelines, container orchestration, multi-cloud dependencies - these limitations aren’t minor. They make it impossible to reason clearly about why performance degradations and failures happen, and how to prevent them. 
 
For example, at a financial services company, a cascading Kafka issue was misdiagnosed for hours as a frontend memory spike. The result? Missed SLOs and three teams paged on a Saturday. 

What These Tools Miss 

Today's “RCA” tools suffer from several critical limitations: 

• They can’t infer the root cause unless it’s already present in the observed alarms – i.e., the output must be in the input. 
• They don't explain why the symptoms occurred – they just hide redundant signals. 
• They require perfect, complete signals to function reliability.  
• They often produce misleading or spurious outputs. 

At Causely, we’re not building a log aggregator, a correlation engine, alarm suppression, nor a chatbot. We are building a causal reasoning system. To understand how this approach is different and why it matters, we first need to re-establish what “root cause analysis” is meant to solve.  

What is the Root Cause Analysis Problem? 

 In managed cloud environments, the RCA problem requires the identification of the most likely cause of observed symptoms, based on a structured understanding of the environment and causal interdependencies between services. Put simply: it’s about using what you know about your system to explain what you see—not just matching alerts to patterns, but reasoning through cause and effect. 

What Most RCA Tools Actually Do 

Most tools that claim to perform RCA fall into one of three categories: 

• Postmortem Narratives: These are after-the-fact writeups, often shaped more by human bias than data, constructed to provide a retrospective of what went wrong.  
• Correlation Engines: These systems surface anomalies and related signals during incidents but confuse correlation with causation. They provide visibility into what things happened around the same time, but they don’t know why.   
• LLM-Powered Assistants: These interfaces can produce plausible-sounding explanations by summarizing the data they have access to, but they often generate spurious or unverifiable answers.  

While all three of these approaches can be useful in some ways, none of them utilize structured causal knowledge and reasoning to solve the RCA problem. That’s the missing piece. 

What Actual Causal Analysis Looks Like 

To be clear, causal analysis is not about finding "what’s weird." It’s about inferring what is the root cause. At Causely, our reasoning platform is built on three foundational principles: 

1. Root Causes Are Explicitly Defined 

A root cause is an underlying issue that results in multiple degradations and disruptions in the managed environment. Formally, a root cause is defined by causes and effects, where the cause is the underlying issue (e.g., a DB experiencing inefficient locking), and the effects are the disruptions it creates in the environment (e.g., degraded service response times).  

Our system monitors your environment and identifies which anomalous behaviors (such as high error rates, slow responses, or service crashes) are symptoms, using metrics gathered from telemetry and observability tools. 

Causely represents each root cause as a closure - a signature uniquely defined by a specific set of expected symptoms. Each root cause and its closure are automatically generated from causal knowledge and the discovered topology. Utilizing Causal Bayesian Networks, Causely can effectively and accurately infer root causes by reasoning over the relationships between the root causes and symptoms, rather than relying on simple mapping or correlations.  

Causely constructs precise causal graphs that show not just what broke, but why. This allows issues to be resolved faster and more efficiently. 

2. Causal Reasoning Is Bayesian 

Causely infers the root cause based on observed symptoms, even when the observations are incomplete or noisy using causal Bayesian networks. Why Bayesian networks? Because you can’t assume perfect signals and should always assume that the observed symptoms will be noisy: 

• Symptoms aren’t always fully observable. 
• Real systems behave unpredictably. 
• And we need to reason probabilistically under uncertainty. 

Causely uses Bayesian causal graphs to represent possible root causes and their effects, assigning probabilities to capture uncertainty in real-world systems. The prior probabilities in our models are defined by experts with decades of experience in distributed systems, microservices, data pipelines, and container orchestration. During active incidents, Causely uses observed symptoms to calculate posterior probabilities over possible root causes. These probabilities are then used to identify the most likely root cause, enabling teams to respond quickly and decisively.  

3. Causal Graphs Are Customer-Specific 

Our causal models are not static or generalized templates. Causely automatically constructs topologically grounded causal graphs that are specific to the customer’s environment and dynamically adapt as environment services and dependencies change. The causal graphs map environment-specific causal dependencies and represent how root causes propagate and manifest themselves across services.  

Benefits of the Causely Approach 

Causely conducts root cause analysis in a principled way to ensure: 

High Precision 

• Root causes are curated and relevant to managed cloud environments. 
• Causal structures mirror actual deployment topologies. 
• False positives are mitigated by never guessing outside the defined causal space. 

Generalizability & Rapid Deployment 

• Bayesian methods identify root causes even with sparse observations. 
• Topologies are grounded in causal graphs to ensure root cause accuracy. 
• Dynamic topology updates with real-time telemetry data let us adapt to each customer’s specific patterns. 

Predictive Power 

• Causal graphs are constructed a priori to ensure the reasoning engine can predict which downstream symptoms and disruptions may emerge when monitoring active root causes.  
• Causely’s causal graphs enable corrective interventions before all root cause symptoms manifest. 
• In addition to being used for real-time operations, the causal reasoning system can also be used to anticipate future failures and help prevent them. Causely identifies critical services and their failure risks based on causal pathways.  

What About LLMs? 

Strengths of LLM-Based RCA 

LLM-based approaches to RCA are gaining traction. They offer unique strengths: 

• Natural Language Interface: You can ask follow-up questions in plain English and get responses without needing to write a query or scan dashboards. 
• Trained on Broad Knowledge: LLMs draw from a vast, pre-trained corpus. Most LLM’s knowledge spans Stack Overflow, GitHub issues, and decades of online technical discourse. This breadth allows them to generate plausible explanations across diverse technologies. 
• Rapid Response: Most LLMs will respond within seconds and can complete tedious tasks quickly.  

But these benefits come with real tradeoffs. 

Limitations of LLM-Based RCA 

• Spurious Causes: LLMs often draw conclusions that appear coherent but are factually incorrect and contextually invalid - due to hallucinations, logical inconsistencies, and insufficient understanding of the managed environment  
• Unprincipled Reasoning: LLMs mimic the language of reasoning without performing structured inference. Research shows that LLMs suffer from content effects, where prior biases interfere with logical reasoning and result in reasoning fallacies.  
• Causal Identification Failures: Research shows LLMs systematically struggle with causal prediction, especially in dynamic settings due to the causal sufficiency problem. 

While LLMs are quickly gaining traction, they remain limited in accurately identifying root causes and reasoning in complex environments. Causely combines the best of both worlds by using LLMs responsibly to support natural language conversations, while grounding root cause analysis in structured causal models to ensure precision and accuracy. 

RCA Isn’t a Buzzword. It’s a well-defined problem. 

If you are calling something a “root cause,” you should be able to show how it caused the observed effects - not just that it co-occurred or provide an explanation that sounds plausible. 

At Causely we solve the RCA problem by being structured, explainable, and rooted in decades of engineering expertise. We don’t guess. We infer. We don’t react. We reason. 

Let’s stop diluting RCA into dashboards and chatbots. Let’s build systems that actually understand why things break and how to prevent future failures. 
 
If you’re tired of dashboards that guess and chatbots that bluff, it’s time to reason instead. Start your journey with Causely today. 

👉Access our sandbox and free trial environment 

👉Reach out for a customized demo 

When it comes to discussing how AI will impact the future of software development and IT management, most vendors hold on to the objective that it’s important to keep the human in the loop. They are afraid to publicly acknowledge what’s always been true — that machines are better than humans at many things. Meanwhile, that list of things continues to grow.

When it comes to observability and IT operations, our goal should be to get humans out of the loop as much as possible. In a world run by software, where SLAs guarantee 99.999% uptime, even one incident in a month — which requires human intervention — is enough downtime to violate your commitments to your customers.

With the rise of distributed, complex applications exacerbated by more and more AI-generated code, it is difficult for traditional IT operations to keep up across troubleshooting, provisioning infrastructure, performance monitoring, security, etc. Developers now deploy code 70% faster using AI, according to the latest DORA report — you can’t have enough SREs and security personnel to keep up with this kind of activity. The companies that succeed will leverage technology to get humans out of the way as quickly as possible. 

Failed Promises of AIOps 

It is worth stating the obvious that there’s a difference between where things are today and where they are heading in the future. As someone who has been in IT management for decades, I have seen all the buzzwords as well as both marketing nonsense and genuine progress.  

The term ‘AIOps’ was coined by Gartner in 2016 to address the increasing complexity and data volume in IT environments, aiming to automate processes such as event correlation, anomaly detection and causality determination. But as it turned out, many of the vendors who claimed to offer AIOps were nothing more than empty shells when you looked under the hood.  

It was essentially the same AI/ML that had been used for a decade beforehand, branded in a new way and making outsized claims that didn’t map to reality. We saw tech giants making acquisitions of point solutions, and then bundling them under the ‘AIOps’ category because it was trendy to do so and they had nowhere else better to put them. 

But I would argue that many of the companies that continued to trumpet these claims without actually delivering on the promises eventually suffered a hit to their reputation. We still aren’t at that stage where machines are acting autonomously to solve complex problems across performance, reliability and security. But they are doing more than ever before, and there’s every reason to believe we will reach that goal in the future. 

The Future of Autonomous Reliability 

Autonomous reliability platforms of the future will not only surface actionable insights, but they will also be competent enough to make autonomous decisions without human intervention. And why should this be impossible? If planes can mostly fly themselves, why can’t IT management become autonomous? 

The decades-long trend of collecting more and more data in the name of observability isn’t rendering autonomous service reliability. Nor is it feeding that data into a machine and magically hoping for answers. Machines trained on yesterday’s patterns might explain what went wrong in the past, but they can’t make the real-time decisions required to keep systems running, especially in dynamic, cloud-native environments. 

What we need is a paradigm shift from data collection to causal understanding. By capturing causal knowledge as part of an ontology, we can reason about cause-and-effect in complex, ever-changing systems — the key is to move beyond reactive alerting into autonomous reliability. Unlike the growing industry trend of offloading alerts to LLMs, causal reasoning gives us the context and clarity needed to take real control.

But even the clearest dashboard doesn’t give you what you need when something breaks. 

Observability allows you to see what is happening, but what about understanding why something is happening? With Causely, you can see the why behind what’s happening without having to leave your Grafana interface. 

That’s where Causely comes in. Our new plugin embeds root cause intelligence directly into your Grafana dashboards — so you can seamlessly shift from awareness to understanding and from incident response to proactive improvement, all in a single view. 

When Observability Stops at What 

Modern engineering teams operate in high-stakes, high-complexity environments:   

• Dozens (or hundreds) of microservices   
• CI/CD pipelines constantly push new code   
• SLOs that leave no margin for delay   

The result? Even with powerful observability tools like Grafana, most teams still end up in incident management mode: war rooms, escalations, and Slack threads at 2 a.m.   

Instead of building, you’re firefighting. Instead of improving systems, you're guessing at symptoms. 

Grafana tells you what happened.   

Causely shows you why — and what to do next. 

 Why Grafana + Causely Is Different 

This integration was built to help teams detect and truly understand problems—in context and real time. 

By embedding Causely’s reasoning engine inside Grafana, teams can now: 

• See the root causes for all active anomalies across your environment, not just alerts 
• Identify the highest-priority root cause impacting your services right now 
• Spot the root causes putting your SLOs at risk — before they’re breached 
• Reduce mean time to detect (MTTD) and resolve (MTTR) by surfacing not just data but decisions 

Causely continuously analyzes the relationships between services, symptoms, and anomalies — and uses causal reasoning to infer what’s driving those symptoms, all showcased natively in Grafana with the plugin.  

Why We Chose Beyla 

What’s the superpower behind this integration? It’s what we chose to run under the hood. 

Causely connects to your Kubernetes environment using Grafana Beyla, an open-source, eBPF-based instrumentation agent. It gives us deep visibility into your workloads — without sidecars, code changes, or custom configs. 

We chose Beyla because it enables our customers scaling on Kubernetes to gain value immediately and understand the root causes of issues impacting their applications.  Beyla enables quick time-to-value, minimal friction, and zero disruption to developer workflows. 

An important distinction here: Beyla isn’t part of the plugin — Beyla isn’t part of the plugin itself — it’s the component that connects to your environment and continuously monitors your services. The plugin then brings that intelligence into Grafana, making it actionable for your team." 

What You Get from Combining Causely and Grafana 

Bringing Causely into Grafana means you're gaining critical visibility when you need it most(no longer flying blind!). Together, they create a powerful loop: observability meets root cause analysis - inside your team's workflows. 

With Causely + Grafana, you get: 

• Instant visibility into root causes for all active anomalies and performance issues across your environment 
• The most urgent root cause of degrading your services surfaced right inside your Grafana dashboards 
• Proactive identification of issues putting your SLOs at risk - before violations occur 
• Context-rich visualizations and alert annotations that go beyond detection to show you what's broken, why it's happening, and the steps to take to resolve the issue 

Integrated with Grafana Alertmanager 

Causely's root cause alerts can be pushed directly into your existing Alertmanager workflows - so your teams don't just get notified that something's wrong; they get a head start on fixing it. That means fewer escalations, faster triage, and better SLO performance. 

Causely's reasoning engine powers all of this. It continuously maps how symptoms propagate through your services and precisely identifies the underlying causes. 

As mentioned, this is all enabled in minutes, thanks to Grafana Beyla. This open-source eBPF-based instrumentation layer lets Causely connect to your Kubernetes workloads without code changes, sidecars, or complex configurations. Beyla, let Causely start understanding your environment immediately - so you get value quickly without interrupting your team's flow. 

How to Use It 

Getting started is straightforward: 

• Install the Causely Plugin:   

   Please find it in the Grafana data source plugin catalog or install it via GitHub. 

• Link to Causely: 

Connect your Causely deployment to your Grafana instance. No changes to your instrumentation are needed, as Causely uses Beyla to automatically ingest telemetry from your Kubernetes workloads. 

• Visualize, Understand, Resolve:   

Add the Causely panel to your dashboards. See the root causes, understand the impact, and get ahead of issues before they become escalations. 

For complete setup steps, check out the Causely plugin docs.  

Start Seeing Root Causes, Not Just Symptoms  

The Causely Plugin for Grafana has officially launched and is ready for use. If you rely on Grafana to monitor your services, this is your next step in making observability genuinely actionable. 

Start Free Trial

Explore Documentation 

Request a Demo 

Modern observability tools help ensure engineers have all the data they could hope for at their disposal. But these tools cause a certain amount of “observability overload”; these tools do not have an understanding of whether an observed anomaly is a real problem and, more importantly, they do not on their own accurately infer the why (i.e. root cause) behind the what (i.e. observed symptoms).  

This was the focus of a recent conversation hosted by 10KMedia’s Adam LaGreca with guests Viktor Farcic (DevOps Toolkit) and Shmuel Kliger (founder of Causely). They explored why traditional observability approaches fall short on their own and what it will take to move beyond visibility toward real operational control. 

The Signal Problem No One's Solving 

The current state of reliability is defined by having too much data and not enough clarity. Alerts fire constantly and are often ignored. Dashboards contain all the data you could wish to have if you had all the time in the world to explore them and try to draw meaning from the data. What’s missing isn’t more data; it’s understanding. Without an understanding of cause and effect, teams are left guessing and leaning heavily on whichever domain experts they can grab at the moment.  

Causal Reasoning as a Path Forward 

The conversation centered on a new approach, causal reasoning, which leans away from bottom-up analysis of mountains of metrics, logs, and traces. By understanding service dependencies and identifying the cascading effect that load and code changes have on distributed systems, causal reasoning offers a way to eliminate the noise and focus on what truly matters for assuring reliable application performance. This helps reduce alert fatigue, speeds up troubleshooting, and frees engineers from the toil of reliability. 

Redefining the Role of AI 

While it makes logical sense to try to apply GenAI to the mountains of observability data we accumulate, leveraging LLMs without proper context will simply generate more noise at scale. By using causality to pinpoint the root cause of observed anomalies, a structured understanding of how systems behave can be used to get more efficient and effective outcomes from leveraging LLMs to automate operational work.  

The Future of Reliability 

Looking ahead, the vision is clear: fewer dashboards, fewer rabbit holes, and fewer hours lost to manual debugging. It’s a shift from observability to assurance. And for teams operating at the speed and scale of today’s distributed architectures, it’s a shift that can’t come soon enough. 

🔍 Want to see this approach in action? 
Check out the Causely sandbox or start a conversation with us: www.causely.ai 

If you’ve learned about how Causely works, you already know that our Causal Reasoning Platform includes a built-in causal knowledge base. This knowledge base guides system behavior by capturing the potential root causes in your environment and the symptoms they may cause. We are constantly exploring ways to expand that knowledge base, and one key source of inspiration for this is the work we do in the real-world with real users of our platform.

One of the most rewarding parts of my job is collaborating with awesome engineers who understand the value of our system and share ideas with me about how to make it better. Every customer scenario we encounter strengthens this knowledge base as we work through a four-step process:

Learn: Understand the root causes and observable symptoms of the scenario

Generalize: Capture the root causes and symptoms in the causality knowledge base

Implement: Develop the mediation to discover and monitor the required information

Deploy: Apply broadly and help all users benefit from this added knowledge

Whether it’s a metadata lock cascade in MySQL 8 or a Kubernetes resource noisy neighbor, this collaborative approach ensures that when one team faces a problem, the entire community can benefit from the expansion of our knowledge base. The following article is one such example of how this all played out in the real world with a real customer.

Modern Database Upgrades Can Be Painful

Last month I had a discussion with one of the engineering leaders at Yext, Peter Rimshnick. We explored a challenge many teams face: unexpected database locking in MySQL 8. Peter shared an incident where a routine schema change caused a bit of unexpected downtime. It turns out this scenario was tied to MySQL 8’s nuanced metadata locking behavior.

Metadata Locking in MySQL 8: What Changed?

MySQL 8 introduced critical improvements, but one underappreciated in their locking mechanism impacts teams daily. Before MySQL 8, A DDL statement (e.g., ALTER TABLE) locked only the target table. In MySQL 8, DDL operations now extend metadata locks to tables linked by foreign keys.

 Imagine this scenario:

A Migration Runs: A DROP COLUMN on the users table requests an exclusive MDL.

Dependencies Ignited: MySQL 8 locks the profiles table (linked via foreign key).

Queries Back Up: Reads/writes on both tables time out after lock_wait_timeout.

Symptoms Spread: APIs fail, dashboards freeze, teams chase false leads.

The Hidden Cost: Engineers manually trace foreign keys; customers see unrelated errors.

How Can Causely Help

Causely discovers dependencies such as users ↔ profiles, and observes how clients interact with the tables. Based on the symptoms it detects, it infers that the root cause is the DDL Locking during the database migration. 

Our platform can now infer root causes like DDL locking based on observed symptoms:

With Causely’s automated “DDL Excessive Lock” detection, engineers instantly pinpoint stalled schema changes—no more manual foreign-key tracing. This MySQL metadata-locking is just one of the many root causes we deliver out of the box. Explore the full library of insights to see how Causely can help you build reliable, resilient data pipelines.

The Network Effect of Shared Learning

This is how modern observability evolves: real problems solved once, scaled to many. Every collaboration like Yext’s isn’t just a fix; it’s a force multiplier that makes the platform more knowledgeable and eliminates manual troubleshooting. Every new root cause we learn from strengthens the entire platform. Join a community of engineers building a smarter, more resilient future.

 For engineers tired of playing whack-a-mole with outages:

• Learn more about Causely
• See Causely in action: Request a demo

But there are no free lunches -this adoption comes with a cost. As organizations adopt microservices, they encounter new operational challenges. Microservices architectures are dynamic, ever evolving and loosely coupled with intricate dependencies and interactions between services. Although built with standardized building blocks and common patterns, the emergent behavior of each system is unique and constantly changing as components are rewritten, upgraded, or replaced. The simplicity of small, loosely coupled services can quickly become overshadowed by the complexity of managing hundreds (or thousands) of interdependent components. 

Continuing the decades long trend in Observability of collecting more and more data won’t get us to the desired state of assuring service reliability. Furthermore, feeding this data to a machine and hoping the machine will magically generate the answers required to continuously assure service reliability is a false hope. Machines trained on yesterday's data may be able to understand the past but cannot make the real-time decisions necessary to continuously assure service levels, especially in cloud-native environments given the dynamic nature of such environments. 

At Causely, we believe that the key to overcoming these challenges lies in causal knowledge captured as part of an ontology, to enable reasoning about cause-and-effect relationships in complex systems. This stands in contrast to the industry's growing reliance on simply sending alerts to Large Language Models (LLMs), and we believe causal reasoning is the critical missing piece for autonomous service reliability. 

Why Traditional Approaches Fall Short 

Some organizations attempt to address reliability by dumping vast amounts of raw, unstructured telemetry into ML algorithms in the hope of surfacing meaningful patterns. When anomalies are detected, these observations are sometimes passed to LLMs to generate plausible explanations. While this can help contextualize events, it often falls short where it matters most. LLMs, by design, are general-purpose tools trained on broad, historical data. They excel at pattern recognition and language generation but lack the deep, real-time causal reasoning needed to adapt to novel, dynamic environments. They may describe "what" happened, but they struggle to uncover "why" it happened and "what" to do about it in a new, unseen context. Even when LLMs are fine-tuned on telemetry patterns, they generalize across environments. But reliability failures are context specific. What broke in one deployment doesn’t explain a novel failure in another deployment. 

Example 1 – Misleading Latency Correlations Across Services: 

In a microservices-based e-commerce platform, a simultaneous latency spike was observed in the checkout, inventory, and payment services. Observability tools showed strong correlations between these services, leading engineers to suspect the inventory service as the root cause. However, the real issue was a slow database query in the product-catalog service, which affected the three components above. The correlation misled the team into focusing on the wrong area, demonstrating the limitations of correlation without causal context. 

Example 2 – Misinterpreting Memory Spikes as Service Defects: 

An alert for repeated pod restarts in the recommendation engine, accompanied by memory spikes across backend services, led engineers to suspect a memory leak. Observability tools flagged backend services as anomalous based on correlated metrics. However, the actual root cause was a recent frontend change that dramatically increased the frequency and size of incoming requests. The backend services were merely reacting to an upstream trigger. This highlights how correlation can obscure the true origin of a problem, especially when external factors are involved. 

Relying on statistical correlation alone can be dangerous. Correlations can be misleading without causal grounding, leading teams to chase false positives, miss root causes, or implement ineffective remediations. 

Beyond Monitoring: Towards Autonomous Operations 

Traditional observability focuses on "what" happened. Causal Reasoning focuses on “why”. Causal Reasoning captures, represents, understands and analyzes cause-and-effect relationships and uses these, among other inferences, to automatically infer root cause based on observed anomalies. By embracing Causal Reasoning, organizations can move beyond the reactive model of monitoring and alerting to a world of autonomous operations, where systems can diagnose and heal themselves with minimal human intervention. This is essential for achieving the promise of resilient, scalable, always-on cloud-native applications. 

Causal Reasoning is driven by ontology. An ontology is a formal model that defines: 

• The types of entities, attributes and relationships in a domain, including root causes and symptoms 
• The relationships that can exist between entities, including the causality relationships between root causes and symptoms, and attribute dependencies  
• The behaviors or constraints (e.g., "a pod can be scheduled on one node ", "a pod can have multiple containers") 

It’s like the grammar and vocabulary for talking about a subject. 

Causal Reasoning uses a knowledge graph to organize the real-world information based on the ontology 

• Nodes are specific instances of entities (e.g., checkout-service, inventory-service, payment-service, production-database) 
• Edges are actual relationships between the instances (e.g., checkout-service -> depends_on -> production-database) 
• Edges can also be relationship between attribute dependencies (e.g. “calls to a user facing API invokes the backend GRPC method”, “backend GRPC method invocation produces async messages on a specific topic”) 
• Metadata: e.g. CPU usage, error logs, deployment time, configs 

The knowledge graph is the filled-out version of the ontology, populated with facts. The discovered topology of a microservices application is a knowledge graph, which describes the application components and their relationships using an ontology. It describes what is like a semantic network, but it doesn’t say anything about why or what will happen if something changes. 

Using the ontology and the knowledge graph Causal Reasoning automatically generates a causal graph. A causal graph is a directed acyclic graph (DAG) with focus on why things happen: 

• Nodes are specific causes and observations 
• Directed edges that represent causal links, not just association 
• Example: DatabaseMalfunction -> causes -> ClientServiceErrors 
• Allows you to ask "what if" questions: 
• What happens to service errors if the database is recovered? 

In short, a knowledge graph describes what is connected to what, while a causal graph, describes what causes what. 

Causal Reasoning: Engineering Intelligence into Service Operations 

The causal knowledge in the ontology captures essential system behaviors and relationships without getting lost in the weeds. Driven by the causal knowledge, causal reasoning enables engineering teams focus on what matters. Instead of reacting to every blip on a dashboard, casual reasoning drives a top-down focus on the critical causes that impact service reliability. 

Using causal reasoning, we can: 

• Understand why a service's performance degraded, not just that it did. 
• Infer the root causes instead of guessing based on symptoms. 
• Develop proactive, preventive strategies instead of reactive firefighting. 

Causal reasoning empowers teams to diagnose, remediate, and even predict and prevent risks to service-level objectives (SLOs) with clarity and confidence.   

Causely: A Purpose-built  Autonomous Reliability System 

At Causely, we’ve developed a purpose-built Causal Reasoning Platform for service reliability. The key tenets of the platform are: 

• Causal model: an ontology describing cloud-native application environments, including the causal knowledge of root causes, symptoms and causality between them 
• Topology graph: a knowledge graph of the specific managed environment. The graph is generated automatically by discovering the managed environment. 
• Abductive inference engine: an engine that automatically generates a causality graph from the ontology and the topology graph and used it in real time to infer root causes based on observed symptoms/anomalies   

Causely solves a problem that no other vendor is solving by delivering 

• Clarity in Complexity: Our models scale with your systems, maintaining meaningful insights even as your architecture evolves and grows more intricate. 
• Actionable Insights: We don't just flag anomalies—we infer root causes and deliver clear, prioritized paths to resolution. 
• Proactive Prevention: With an ontology and causal reasoning, we spot risks before they become incidents, shifting organizations from reactive to proactive. 
• Seamless Integration: Our platform integrates with your workflows and CI/CD pipelines, delivering instant value without requiring manual retraining or rule-writing. 

Conclusion 

We do see LLMs as a powerful technology that can benefit from being used in tandem with Causal reasoning, stay tuned for more on that.

Causal Reasoning is the foundation of Causely’s solution. It empowers organizations to cut through complexity, deliver precise insights, prevent downtime, and free engineers to focus on what matters most: innovation. 

Let us show you the power of causal reasoning and cross-organizational collaboration in cloud-native environments. See Causely for yourself. Book a meeting with the Causely team or start your free trial now. 

#cloudnative #servicereliability #causalreasoning #abstraction #siteReliabilityEngineering #sre #DevOps #observability #AI #ML 

Recently, we encountered an issue that perfectly illustrates the value of Causely’s Causal Reasoning Platform in action.  

A Crash That Didn’t Want to Be Found 

While reviewing our staging environment— which uses our existing OpenTelemetry instrumentation to identify root causes in our production deployment of Causely — I noticed something disturbing: a frequent crash failure had been quietly recurring over six days in one of our analytics microservices. 

This type of issue is notoriously difficult to detect. The logs were noisy. The failures were intermittent. The symptoms appeared scattered—some in one analysis module, others in a different component entirely. This issue would’ve gone unnoticed in most environments or been chalked up to transient behavior. But Causely identified the cause. Not as an alert or a spike in a dashboard – but as an active root cause that is impacting our system. It wasn’t just showing us individual failures. It was showing us causality - the propagation behind the symptoms. 

Different subcomponents—analysis-3, analysis-5—were failing in seemingly unrelated ways. One had high error rates; another had frequent panics. Without Causely, we would’ve been chasing these symptoms in isolation. Logging into pods, checking dashboards, comparing timelines. A manual, time-consuming wild goose chase. 

What Causely Did Differently 

 Causely automatically inferred these separate symptoms across services and time into a single root cause: a concurrency issue in a shared in-memory map used across multiple analysis workers. 

The issue? A concurrent write-read condition that would only manifest under higher-volume conditions – something nearly impossible to catch in development or test environments. But Causely connected the dots through symptom propagation and causal relationships, allowing us to zero in on the problem fast. 

Technical Deep Dive: The Bug Behind the Crash 

The crash was caused by concurrent access to a Go map, which is unsafe for concurrent reads and writes without explicit synchronization. 

We were using a plain map [string]SomeStruct to store intermediate analysis results across worker goroutines. This map was being: 

• Written to by one thread collecting event evidence in real time 
• Read from simultaneously by another thread responsible for emitting outputs 

Under low volume, this worked fine. But under sustained load in production, the race condition emerged—resulting in panics like: 

fatal error: concurrent map read and map write 

 This is a classic Go gotcha. The fix was straightforward: we wrapped the map with a sync.RWMutex to protect both reads and writes, ensuring thread safety. Alternatively, we could have used a sync.Map, but since our access patterns were relatively simple and performance-sensitive, the mutex approach was more appropriate. 

This was one of those bugs where: 

• It was impossible to reproduce locally 
• The crash symptoms varied depending on which thread accessed the map first 
• And the logs, while technically correct, gave no useful hint unless you already suspected the concurrency issue 

 Without Causely surfacing the underlying causality across services and across time, this would have remained an intermittent ghost bug. 

From Root Cause to Resolution – In Hours, Not Days 

Thanks to the inference Causely provided, we identified the concurrency bug, coded a fix, and shipped it – all within a couple of hours. 

This wasn’t just about speed. It was about precision. Without Causely, our team might have: 

• Spent days troubleshooting unrelated alerts and errors 
• Overlooked the root cause due to insufficient signal 
• Delayed resolving an issue that was actively degrading reliability 

Why This Matters 

Crashes like these don’t always surface in ways traditional monitoring or modern observability systems can catch. They emerge over time, appear intermittent, and produce symptoms that span components. 

Causely gave us what observability tools couldn’t: contextual understanding. Observability tools show you context. Causely helps you understand it. It didn’t just tell us something was broken. It told us what was breaking, why, and what needed to change. 

This is the power of causal reasoning over correlation. 

We built Causely to operate in the most demanding environments – and we hold ourselves to that same standard. This incident was a clear example of how our platform enables us to move faster, with more confidence, and resolve issues before they impact users. 

And that’s exactly the kind of reliability we aim to bring to every engineering team. 

Avoiding common observability pitfalls

We already know – based on decades of experience working in and with operations teams in the most challenging environments – that bridging the gap between the vast ocean of observability data and actionable insights has and continues to be a major pain point. This is especially true in the complex world of cloud-native applications.

Missing application insights

Application observability remains an elusive beast for many, especially in complex microservices architectures. While infrastructure monitoring has become readily available, neglecting application data paints an incomplete picture, hindering effective troubleshooting and operations.

Siloed solutions

Traditional observability solutions have relied on siloed, proprietary agents and data sources, leading to fragmented visibility across teams and technologies. This makes it difficult to understand the complete picture of service composition and dependencies.

To me, this is like trying to solve a puzzle with missing pieces – that’s essentially a problem that many DevOps teams face today – piecing together a picture of how microservices, serverless functions, databases, and other elements interact with one another, and underlying infrastructure and cloud services they run on. This makes it hard to collaborate and troubleshoot; it's a struggle to pinpoint the root cause of performance issues or outages.

Vendor lock-in

Many vendors’ products also lock customers’ data into their cloud services. This can result in customers paying through the nose, because licensing costs are predicated on the volume of data that is being collected and stored in the service providers’ backend SaaS. It can also be very hard to exit these services once locked in.

These are all pitfalls we wanted to avoid at Causely as we set out to build our our Causal Reasoning Platform.

The pillars of our observability architecture pointed us to OpenTelemetry

OpenTelemetry provides us with a path to break free from these limitations, establishing a common framework that transcends programming languages and platforms that we are using to build our services, and satisfying the requirements laid out in the pillars of our observability architecture:

Precise instrumentation

OpenTelemetry offers automatic instrumentation options that minimize the amount of work we need to do on manual code modifications and streamline the integration of our internal observability capabilities into our chosen backend applications.

Unified picture

By providing a standardized data model powered by semantic conventions, OpenTelemetry enables us to paint an end-to-end picture of how all of our services are composed, including application and infrastructure dependencies. We can also gain access to critical telemetry information, utilizing this semantically consistent data across multiple backend microservices even when written in different languages.

Vendor-neutral data management

OpenTelemetry allows us to avoid locking our application data into 3rd party vendors’ services by decoupling it from proprietary vendor formats. This gives us the freedom to choose the best tools on an ongoing basis based on the value they provide. If something new comes along that we want to exploit, we can easily plug it into our architecture.

Resource-optimized observability

With OpenTelemetry, we can take a top down approach to data collection, starting with the problems we are looking to solve and eliminating unnecessary information. In doing so, this minimizes our storage costs and optimizes compute resources we need to support our observability pipeline.

We believe that following these pillars and building our Causal Reasoning Platform on top of OpenTelemetry propels our product’s performance, enables rock-solid reliability, and ensures consistent service experiences for our customers as we scale our business. We also minimize ongoing operational costs, creating a win-win for us and our customers.

OpenTelemetry + causal analysis: scaling for performance and cost efficiency

Ultimately, observability aims to illuminate the behavior of distributed systems, enabling proactive maintenance and swift troubleshooting. Yet isolated failures manifest as cascading symptoms across interconnected services.

While OpenTelemetry enables back-end applications to use this data to provide a unified picture in maps, graphs and dashboards, the job of figuring out the cause and effect in the correlated data still requires highly skilled resources. This process can also be very time consuming, tying up personnel across multiple teams, with ownership for different elements of overall services.

There is a lot of noise in the industry right now about how AI and LLMs are going to magically come to the rescue, but reality paints a different picture. All of the solutions available in the market today focus on correlating data versus uncovering a direct understanding of causal relationships between problems and the symptoms they cause, leaving DevOps teams with noise, not answers.

Traditional AI and LLMs also require massive amounts of data as input for training and learning behaviors on a continuous basis. This is data that ultimately ends up being transferred and stored in some form of SaaS. Processing these large datasets is very computationally intensive. This all translates into significant cost overheads for the SaaS providers as customer datasets grow overtime – costs that ultimately result in ever increasing bills for customers.

At Causely, we're taking a different approach

Our causal reasoning software provides operations and engineering teams with an understanding of the “why”, which is crucial for effective and timely troubleshooting and decision-making.

Our Causal Reasoning Platform uses predefined models of how problems behave and propagate. When combined with real-time information about a system’s specific structure, Causely computes a map linking all potential problems to their observable symptoms.

This map acts as a reference guide, eliminating the need to analyze massive datasets every time the platform encounters an issue. Think of it as checking a dictionary instead of reading an entire encyclopedia.

The bottom line is, in contrast to traditional AI, Causely operates on a much smaller dataset, requires far less resources for computation and provides more meaningful actionable insights, all of which translate into lower ongoing operational costs and profitable growth.

Summing it up

There’s massive potential for causal analysis and OpenTelemetry to come together to tackle the limitations of traditional AI to get to the “why.” This is what we’re building at Causely – check out our recent integration news. Doing so delivers numerous benefits:

• Less time on Ops, more time on Dev: OpenTelemetry provides standardized data while Causely analyzes it to automate the root cause analysis (RCA) process, which will significantly reduce the time our DevOps teams have to spend on troubleshooting.
• Instant gratification, no training lag: We can eliminate AI’s slow learning curve, because Causely leverages OpenTelemetry’s semantic language and the Causal Reasoning Platform’s domain knowledge of cause and effect to deliver actionable results, right out of the box without massive amounts of data and with no training lag!
• Small data, lean computation, big impact: Unlike traditional AI’s data gluttony and significant computational overheads, Causely thrives on targeted data streams. OpenTelemetry’s smart filtering keeps the information flow lean, allowing Causely to identify the root causes with a significantly smaller dataset and compute footprint.
• Fast root cause identification: Traditional AI might tell us “ice cream sales and shark attacks rise together,” but causal reasoning reveals the truth – it’s the summer heat and not the sharks, driving both! By understanding cause-and-effect relationships, Causely cuts through the noise and identifies the root causes behind performance degradation and service malfunctions.

Having these capabilities is critical if we want to move beyond the labor intensive processes associated with how RCA is performed in DevOps today, and eventually achieve autonomous service reliability. This is why we are eating our own dog food and using Causely as part of our tech stack to manage the services we provide to customers.

Want to learn more about our integration with OpenTelemetry or see if Causely can help you build better, more reliable cloud-native applications? Book a meeting with the Causely team. We'd love to chat!

Our team at Causely has adopted OpenTelemetry within our own platform, and we recently announced a formal integration with OpenTelemetry, which prompted us to share a production-focused guide. Our goal is to help developers, DevOps engineers, software engineers, and SREs understand what OpenTelemetry is, its core components, and a detailed look at the OpenTelemetry Collector (OTel Collector). This background will help you use OTel and the OTel Collector as part of a comprehensive strategy to monitor and observe applications.

What Data Does OpenTelemetry Collect?

There are 3 types of data that are gathered by OpenTelemetry using the OTel Collector: logs, metrics, and traces.

Logs

Logs are records of events that occur within an application. They provide a detailed account of what happened, when it happened, and any relevant data associated with the event. Logs are helpful for debugging and understanding the behavior of applications.

OpenTelemetry collects and exports logs, providing insights into events and errors that occur within the system. For example, if a user reports a slow response time in a specific feature of the application, engineers can use OpenTelemetry logs to trace back the events leading up to the reported issue.

Metrics

Metrics are quantitative data that measure the performance and health of an application. Metrics help in tracking system behavior and identifying trends over time. OpenTelemetry collects metrics data, which helps in tracking resource usage, system performance, and identifying anomalies.

For instance, if a spike in CPU usage is detected using OpenTelemetry metrics, engineers can investigate the potential issue using the OTel data collected and make necessary adjustments to optimize performance.

Developers use OpenTelemetry metrics to see granular resource utilization data, which helps understand how the application is functioning under different conditions.

Traces

Traces provide a detailed view of request flows within a distributed system. Traces help understand the execution path, diagnose application behaviors, and see the interactions between different services.

For example, if a user reports slow response times on a website, developers can use trace data to help better identify which service is experiencing issues. Traces can also help in debugging issues such as failed requests or errors by providing a step-by-step view of how requests are processed through the system.

Introduction to OTel Collector

You can deploy the OTel Collector as a standalone agent or as a sidecar alongside your application. The OTel Collector also includes some helpful features for sampling, filtering, and transforming data before sending it to a monitoring backend.

How it Works

The OTel Collector works by receiving telemetry data from many different sources, processing it based on configured pipelines, and exporting it to chosen backends. This modular architecture allows for customization and scalability.

The OTel Collector acts as a central data pipeline for collecting, processing, and exporting telemetry data (metrics, logs, traces) within an observability stack.

Here’s a technical breakdown:

Data Ingestion:

• Leverages pluggable receivers for specific data sources (e.g., Redis receiver, MySQL receiver).
• Receivers can be configured for specific endpoints, authentication, and data collection parameters.
• Supports various data formats (e.g., native application instrumentation libraries, vendor-specific formats) through receiver implementations.

Data Processing:

• Processors can be chained to manipulate the collected data before export.
• Common processing functions include:
  • Batching: Improves efficiency by sending data in aggregates.
  • Filtering: Selects specific data based on criteria.
  • Sampling: Reduces data volume by statistically sampling telemetry.
  • Enrichment: Adds contextual information to the data.

Data Export:

• Utilizes exporters to send the processed data to backend systems.
• Exporters are available for various observability backends (e.g., Jaeger, Zipkin, Prometheus).
• Exporter configurations specify the destination endpoint and data format for the backend system.

Internal Representation:

• Leverages OpenTelemetry’s internal Protobuf data format (pdata) for efficient data handling.
• Receivers translate source-specific data formats into pdata format for processing.
• Exporters translate pdata format into the backend system’s expected data format.

Scalability and Configurability:

• Designed for horizontal scaling by deploying multiple collector instances.
• Configuration files written in YAML allow for dynamic configuration of receivers, processors, and exporters.
• Supports running as an agent on individual hosts or as a standalone service.

The OTel Collector is format-agnostic and flexible, built to work with various backend observability systems.

Setting up the OpenTelemetry (OTel) Collector

Starting with OpenTelemetry for your new system is a straightforward process that takes only a few steps:

1. Download the OTel Collector: Obtain the latest version from the official OpenTelemetry website or your preferred package manager.
2. Configure the OTel Collector: Edit the configuration file to define data sources and export destinations.
3. Run the OTel Collector: Start the Collector to begin collecting and processing telemetry data.

Keep in mind that the example we will show here is relatively simple. A large scale production implementation will require fine-tuning to ensure optimal results. Make sure to follow your OS-specific instructions to deploy and run the OTel collector.

Next, we need to configure some exporters for your application stack.

Integration with Popular Tools and Platforms

Let’s use an example system running a multi-tier web application using NGINX, MySQL, and Redis. Each source platform will have some application-specific configuration parameters.

Configuring Receivers

redisreceiver:

• Replace receiver_name with redisreceiver
• Set endpoint to the port where your Redis server is listening (default: 6379)
• You can configure additional options like authentication and collection intervals in the receiver configuration. Refer to the official documentation for details.

mysqlreceiver:

• Replace receiver_name with mysqlreceiver
• Set endpoint to the connection string for your MySQL server (e.g., mysql://user:password@localhost:3306/database)
• Similar to Redis receiver, you can configure authentication and collection intervals. Refer to the documentation for details.

nginxreceiver:

• Replace receiver_name with nginxreceiver
• No endpoint configuration needed as it scrapes metrics from the NGINX process.
• You can configure what metrics to collect and scraping intervals in the receiver configuration. Refer to the documentation for details.

The OpenTelemetry Collector can export data to multiple providers including Prometheus, Jaeger, Zipkin, and, of course, Causely. This flexibility allows users to leverage their existing tools while adopting OpenTelemetry.

Configuring Exporters

Replace exporter_name with the actual exporter type for your external system. Here are some common options:

• jaeger for Jaeger backend
• zipkin for Zipkin backend
• otlp/causely for Causely backend
• There are exporters for many other systems as well. Refer to the documentation for a complete list.

Set endpoint to the URL of your external system where you want to send the collected telemetry data. You might need to configure additional options specific to the chosen exporter (e.g., authentication for Jaeger).

There is also a growing list of supporting vendors who consume OpenTelemetry data.

Conclusion

OpenTelemetry provides a standardized approach to collecting and exporting logs, metrics, and traces. Implementing OpenTelemetry and the OTel Collector offer a scalable and flexible solution for managing telemetry data, making it a popular and effective tool for modern applications.

You can use OpenTelemetry as part of your monitoring and observability practice in order to gather data that can help drive better understanding of the state of your applications. The most valuable part of OpenTelemetry is the ability to ingest the data for deeper analysis.

How Causely Works with OpenTelemetry

At Causely, we use OpenTelemetry as one of many data sources to assure autonomous service reliability for our clients. OpenTelemetry data is ingested by our Causal Reasoning Platform, which is a model-driven AI system that automatically instruments your environment, defines SLOs, and pinpoints risk to reliability - without manual setup. By cutting through observability overload, Causely eliminates complex troubleshooting and ensures services meet reliability expectations. Engineering teams instantly improve service reliability, reduce downtime, and lower operational costs – freeing them to focus on innovation.

Click below to watch the interview, or check it out on Techstrong.tv here.

View the full article by Carl Williams on Tech Times.

Causely is announcing its integration with OpenTelemetry, bringing a fresh approach to observability that cuts through the noise and surfaces only what matters.

In today’s world of modern cloud applications, observability is both a necessity and a challenge. With countless services interacting dynamically, engineering teams are often flooded with logs, traces, and metrics, making it difficult to extract meaningful insights when something goes wrong. Traditional observability approaches rely on collecting everything—but the sheer volume of data can be overwhelming, turning troubleshooting into a time-consuming and frustrating process.

Causely is here to change that. Today, the company is announcing its integration with OpenTelemetry, bringing a fresh approach to observability that cuts through the noise and surfaces only what matters.

Yotam Yemini, CEO of Causely, highlights that OpenTelemetry adoption often requires significant effort, as teams must determine which data to collect, how to sample it, and how to extract useful insights. “Causely is focused on cutting through that noise to pinpoint what matters,” Yemini says. Causely does this by automatically filtering out unnecessary data and providing a structured understanding of system dependencies and potential failure points. The integration makes OpenTelemetry more practical and efficient, ensuring engineers receive only the most relevant insights.

The Challenge of Observability Today

Modern applications generate an immense amount of telemetry data, whether it’s from microservices, distributed systems, or cloud infrastructure. While OpenTelemetry has become the de facto standard for collecting and transmitting this data, its adoption often comes with significant hurdles. Engineers frequently struggle with:

• Too much data, too little insight – Logs and traces pile up, making it difficult to identify real issues.
• Slow root cause analysis – Teams spend too much time manually sifting through observability data.
• Uncertain data collection strategies – Debates about sampling vs. full collection leave engineers guessing about what’s really needed.

The result? An overload of information but very few actionable insights.

A Smarter Approach to Observability

Causely’s Causal Reasoning Platform takes a fundamentally different approach. Instead of just presenting raw telemetry data, it analyzes dependencies and causal relationships between different components of an application. This means that instead of looking at an overwhelming sea of logs, engineers can quickly pinpoint what is causing an issue and how it propagates through the system.

With the new OpenTelemetry integration, Causely makes it even easier for teams to:

• Seamlessly ingest OpenTelemetry traces and metrics without any complicated setup.
• Automatically filter out irrelevant data and highlight the most critical insights.
• Predict potential failures before they happen, allowing teams to be proactive instead of reactive.

By integrating with OpenTelemetry, Causely aims to bridge the gap between raw telemetry collection and intelligent observability. Teams can proactively manage service reliability, prevent latency issues, and optimize performance, making observability more effective and manageable.

How it Works

Causely provides two easy ways to integrate with OpenTelemetry. If your team is already using OpenTelemetry, you can simply point your existing collector at Causely. Within seconds, the system begins processing the telemetry data and identifying key dependencies, bottlenecks, and potential risks.

For teams that haven’t yet adopted OpenTelemetry, Causely offers an alternative: auto-instrumentation using eBPF. This allows teams to collect the necessary telemetry data without having to manually modify their applications. The best part? It all runs in your own cloud environment, ensuring data privacy and security while keeping infrastructure overhead minimal.

Beyond OpenTelemetry: Observability without Limits

While this new integration enhances how teams use OpenTelemetry, Causely isn’t limited to a single observability standard. The platform is designed to work with any telemetry source, whether proprietary or open-source, ensuring flexibility for engineering teams regardless of their stack.

By decoupling the intelligence layer from data collection, Causely provides a unified way to analyze and act on observability data, no matter where it comes from. This approach helps reduce the complexity caused by vendor sprawl, allowing teams to focus on application performance rather than managing observability infrastructure.

Who Benefits from Causely?

Causely is built for any engineering team dealing with complex, microservices-driven applications. Whether you’re operating a large-scale SaaS platform, managing a fintech infrastructure, or ensuring the reliability of a healthcare application, Causely helps keep things running smoothly by automating root cause detection and providing predictive insights.

Companies already leveraging Causely include publicly traded SaaS providers, financial institutions, telecom giants, and healthcare tech firms—all benefiting from the ability to reduce incident resolution times and improve service reliability.

Get Started with Causely

Observability doesn’t have to be overwhelming. With Causely’s OpenTelemetry integration, teams can move beyond raw data collection and gain instant, actionable insights that make troubleshooting faster and easier.

For those interested in trying it out, Causely offers a free trial where you can explore a demo environment or set up a real-world deployment in minutes. To get started, visit causely.ai and see how causal reasoning can transform your observability strategy.

Instead of spending hours digging through logs, let Causely do the heavy lifting—so you can focus on building great applications.

Guest: Yotam Yemini
Company: Causely
Show: Let’s Talk

Summary is written by Emily Nicholls

I’ve spent over three decades in IT Operations. Despite all the talk of transformation, many of the fundamental challenges remain unchanged, or have even worsened. The rise of modern DevOps and observability promised to revolutionize how we monitor and maintain systems, but in reality, we’ve simply scaled up the same old problems. More data, more dashboards, and more alerts haven’t led to better outcomes.  
 
The core issue? Our approach to observability has been misguided. We need a paradigm shift from a bottom-up approach to a top-down approach. Instead of bottoms up collecting everything hoping we find insights in the data, we need to start with the purpose or the insights we are looking for and collect only the data that will help us to infer these insights. 

Part 1: Looking Back at the Days of ITOps 

From the early days of IT Operations, we have been focusing on keeping the lights on. At the early days, systems were monolithic, monitoring was rudimentary, and troubleshooting often involved sifting through log files for hours on end. A major incident meant a war room filled with engineers manually correlating data, trying to pinpoint the root cause of an outage. 

As infrastructure grew more complex, the industry responded by layering on more tools, with each promising to make troubleshooting easier. But in practice, these tools often just created more dashboards, more logs, and more alerts, leading to information overload. IT Operations needed to evolve, which gave rise to modern DevOps, but we are still in the early innings of the game.  

Part 2: The Problem Today with Observability 

Observability was supposed to solve these challenges by giving teams a deeper understanding of their systems. The idea was that by collecting and analyzing vast amounts of telemetry – metrics, events, logs, and traces – organizations would gain better insights and respond to issues faster. 

Instead, we’ve seen an explosion of complexity. Today’s observability landscape is defined by: 

• Too much noise: The sheer volume of logs and alerts makes it nearly impossible for engineers to separate meaningful signals from the flood of data. 
• Too many tools: Companies rely on a fragmented ecosystem of monitoring, logging, and tracing solutions, leading to silos and inefficiencies. 
• Too much manual troubleshooting: Despite all this data, engineers still spend most of their time manually diagnosing incidents, correlating logs, and responding to false positives. 

The promise of observability has not been fully realized because organizations are still focusing on data collection instead of shifting their focus to intelligent data processing. 

Part 3: Causely’s Paradigm Shift 

At Causely, we believe that Observability must be disrupted and go through a paradigm shift from a bottom-up data collection to top-down purpose driven analytics that saves engineers from needing to spend hours sifting through data to figure out “the why behind the what.” We reject the notion that engineers will always need to spend time drilling through dashboards and making sense of the data coming from their tools. Instead, we believe that implementing the proper abstraction layer would allow systems to self-manage and take humans out of the troubleshooting loop. Instead of using tools that provide information for the engineers to analyze, the engineers should deploy systems that can actually make autonomous decisions. 

This journey to autonomous service reliability is built on a number of core principles that I outlined in a recent blog. This shift from passively collecting data to relying on a system that actively interprets and acts on these insights is the future of observability. 

Part 4: The Rise of AI and Agentic AI 

We are on the cusp of a new era where Agentic AI can fundamentally reshape IT Operations. Instead of engineers reacting to alerts, AI-driven systems will proactively maintain service reliability, predicting and preventing failures before they occur. 

I’m not talking about simple alerting rules or anomaly detection; I’m talking about true agentic AI that continuously analyzes system behavior and adapts in real-time to prevent service degradation.  

Causely is positioned to lead this transformation, helping organizations shift from reactive troubleshooting to proactive, AI-driven service reliability. This isn’t just a step forward in observability—it’s a paradigm shift in how we think about IT Operations. 

Conclusion 

For 30+ years, we’ve been fighting the same battles in IT Operations, just at a larger scale. It’s time to rethink observability, move beyond endless data collection, and embrace a future where AI-driven automation ensures service reliability with minimal human intervention. Organizations that adopt this new paradigm won’t just reduce downtime and incident costs—they’ll free their engineers to focus on what truly matters: building the future. 

Check out the interview below or view it on Spotify.

“For years, the IT industry has struggled to make sense of the overwhelming amounts of data coming from dozens of observability platforms and monitoring tools,” said Yotam Yemini, CEO of Causely. “OpenTelemetry is an important paradigm shift that gives teams a more flexible and vendor-neutral way to collect and use telemetry data, but it also exacerbates the big data problem of modern DevOps: deriving actionable insights from overwhelming amounts of telemetry data.”

Causely installs in cloud-native environments in under a minute and immediately bridges the gap between the overwhelming amount of observability data and the actionable insights that can be found within that data, maximizing the potential benefits of OpenTelemetry. Teams who have yet to adopt OpenTelemetry can leverage Causely’s eBPF-based auto-instrumentation to start collecting traces instantly. 

The Causely system works by automatically mapping an application’s topology and service dependencies, then applying a finite set of likely root causes to this data. This novel approach to observability is counter to traditional tooling and methods that encourage businesses to collect as much data as possible – a situation that hasn’t fundamentally changed in decades – which then requires human troubleshooting to respond to alerts, make sense of patterns, identify root cause, and ultimately determine the best action for remediation. 

“Traditionally an alert fires and we have to throw a bunch of subject matter experts at the problem to comb through all of the telemetry and troubleshoot why these things are going on,” said Matt Titmus, Engineering Manager at Yext. “I’m enthusiastic about Causely because it gets us to the root cause much faster and also helps us be more proactive."

Causely was founded by Shmuel Kliger who has been developing systems for IT Operations for over two decades. He was also the founder of Turbonomic (acquired by IBM) and the CTO of SMARTS (acquired by EMC), bringing together technical experience with a track record of successfully scaling companies.

About Causely

Causely leverages causal reasoning to cut through the observability noise and pinpoint what matters. Engineers are overwhelmed by too many tools, alerts, and data coming from their existing observability solutions. Causely automatically surfaces only the most critical risks to service reliability, enabling businesses to minimize operational overhead and maintain reliability without manual troubleshooting.

Contact

Adam LaGreca
Founder of 10KMedia
adam@10kmedia.co

Observability has become a cornerstone of application reliability and performance. As systems grow more complex—spanning microservices, third-party APIs, and asynchronous messaging patterns—the ability to monitor and debug these systems is both a necessity and a challenge. 

OpenTelemetry (OTEL) has emerged as a powerful, open source framework that standardizes the collection of telemetry data across distributed systems. It promises unprecedented visibility into logs, metrics, and traces, empowering engineers to identify issues and optimize performance across multiple languages, technologies and cloud environments. 

But with great visibility comes a hidden cost. While OTEL democratizes observability, it also exacerbates the “big data problem” of modern DevOps. 

This is where Causely comes in—today, we announced a new integration with OTEL that bridges the gap between OTEL's data deluge and actionable insights. In this post, we’ll explore the strengths and limitations of OpenTelemetry, the challenges it introduces, and how Causely transforms raw telemetry into precise, cost-effective analytics. 

The OpenTelemetry opportunity 

Microservices are a tangled web of interdependencies that communicate over REST or gRPC. Asynchronous systems like Kafka shuttle messages between loosely coupled services. Infrastructure dynamically scales resources to meet demand. Observability has become the glue that holds these systems together, enabling engineers to monitor performance, troubleshoot issues, and ensure reliability. 

At the heart of the observability revolution is OpenTelemetry (OTEL), an open-source standard that unifies the instrumentation and collection of telemetry data across logs, metrics, and traces. Its modular architecture, community-driven development, and broad compatibility with existing observability tools have made OTEL the de facto choice for modern DevOps teams. 

What does OpenTelemetry do? 

OpenTelemetry provides APIs, SDKs, and tools to capture three primary types of telemetry data: 

1. Logs: Detailed, timestamped records of system events (e.g., errors, warnings, and custom events). 
2. Metrics: Quantitative measurements of system health and performance (e.g., CPU usage, request latency, error rates). 
3. Traces: End-to-end views of requests flowing through distributed systems, mapping dependencies and execution paths. 

With OTEL, engineers can instrument their code to emit these telemetry signals, use an OpenTelemetry Collector to aggregate and process the data, and export it to observability backends like Prometheus, Tempo, or Elasticsearch. 

Why OpenTelemetry is a game-changer 

OpenTelemetry addresses a critical pain point in observability: fragmentation. Historically, different tools and platforms required unique instrumentation libraries, making it difficult to standardize observability across an organization. OTEL simplifies this by providing: 

• Vendor-Agnostic Instrumentation: A single API to instrument applications regardless of the backend. 
• Centralized Data Collection: The OpenTelemetry Collector serves as a pluggable data pipeline, consolidating telemetry from various sources. 
• Interoperability: Native support for popular backends like Prometheus, Tempo, and other vendors, allowing teams to integrate OTEL into their existing observability stack. 

Technical example: Debugging latency issues 

Consider a microservices-based e-commerce application experiencing high latency during checkout. With OTEL traces, you can get a lot of information about the performance of this service but it is hard to find out what is responsible for the latency. For example: https://github.com/esara/robot-shop/blob/instrumentation/dispatch/main.go#L172 

``` 

func processOrder(headers map[string]interface{}, order []byte) { 
    start := time.Now() 
    log.Printf("processing order %s\n", order) 
    tracer := otel.Tracer("dispatch") 
 
    // headers is map[string]interface{} 
    // carrier is map[string]string 
    carrier := make(propagation.MapCarrier) 
    // convert by copying k, v 
    for k, v := range headers { 
       carrier[k] = v.(string) 
    } 
 
    ctx := otel.GetTextMapPropagator().Extract(context.Background(), carrier) 
 
    opts := []oteltrace.SpanStartOption{ 
       oteltrace.WithSpanKind(oteltrace.SpanKindConsumer), 
    } 
    ctx, span := tracer.Start(ctx, "processOrder", opts...) 
    defer span.End() 
 
    span.SetAttributes( 
       semconv.MessagingOperationReceive, 
       semconv.MessagingDestinationName("orders"), 
       semconv.MessagingRabbitmqDestinationRoutingKey("orders"), 
       semconv.MessagingSystem("rabbitmq"), 
       semconv.NetAppProtocolName("AMQP"), 
    ) 

``` 

By exporting these traces to a backend like Tempo, engineers can visualize the request flow and identify bottlenecks, such as consuming messages from RabbitMQ in the dispatch service and inserting an order in a MongoDB database.  

The Big Data problem of observability 

OpenTelemetry’s ability to capture detailed telemetry data is a double-edged sword. While it empowers engineers with unprecedented visibility into their systems, it also introduces challenges that can hinder the very goals observability aims to achieve. The sheer volume of data collected—logs, metrics, and traces from thousands of microservices—can overwhelm infrastructure, slow down workflows, inflate costs, and most importantly drown engineers with data. This “big data problem” of observability is a natural consequence of OpenTelemetry’s strengths but must be addressed to make the most of its potential. 

OpenTelemetry collects a lot of data 

At its core, OpenTelemetry is designed to be exhaustive. This design ensures engineers can instrument their systems to capture every possible detail. For example: 

• A high-traffic e-commerce site might generate logs for every HTTP request, metrics for CPU and memory usage, and traces for each request spanning multiple services. 
• OpenTelemetry auto instrumentation libraries are an easy way to instrument HTTP, GRPC, messaging, database and caching libraries in all languages, but they generate metrics and traces for every call between every microservice, managed service, database and third-party API. 

Consider a production environment running thousands of microservices, each processing hundreds of requests per second. Using OpenTelemetry: 

• Logs: A single request might generate dozens of log entries, resulting in millions of logs per minute. 
• Metrics: Resource utilization metrics are emitted periodically, adding continuous streams of quantitative data. 
• Traces: Distributed traces can contain hundreds of spans, each adding its own metadata. 

While this level of detail is invaluable for debugging and optimization, it quickly scales beyond what many teams are prepared to manage. The amount of data makes it difficult to troubleshoot problems, manage escalations, be proactive about deploying new code, and plan for future investments. 

The cost of data 

The problem with this massive volume of telemetry data isn’t just about storage; it’s also about processing and time-to-insight. Let’s break it down: 

• Networking Costs: Transmitting telemetry data from distributed systems, microservices, or edge devices to central storage or processing locations incurs significant bandwidth usage. This can result in substantial networking costs, especially for real-time telemetry pipelines or when dealing with geographically dispersed infrastructure. 
• Storage Costs: Logs, metrics, and traces consume vast amounts of storage, often requiring specialized solutions like Elasticsearch, Amazon S3, or Prometheus’s TSDB. These systems must scale horizontally, adding significant operational overhead. 
• Compute Costs: Telemetry data needs to be parsed, indexed, queried, and analyzed. Complex queries, such as joining multiple traces to identify bottlenecks, can place a heavy burden on compute resources. 
• Time Costs: During a high-severity incident, every second counts. Pinpointing the root cause is like looking for a needle in a haystack. With OpenTelemetry, the haystack is much bigger, making the task harder and longer.  

Time-to-insight delays 

Imagine a scenario where an outage occurs in a distributed system. An engineer might start by querying logs for errors, then switch to metrics to identify anomalies, and finally inspect traces to pinpoint the failing service. Each query takes time, and engineers often waste effort chasing irrelevant leads. This delay increases Mean Time to Detect (MTTD) and Mean Time to Resolve (MTTR), directly impacting uptime and user satisfaction. 

Noise vs. signal 

Another challenge is separating the signal (useful insights) from the noise (redundant or irrelevant data). With OTEL: 

• Logs can be overly verbose, capturing routine events that clutter debugging efforts. 
• Metrics might lack the context needed to tie resource anomalies back to specific root causes. 
• Traces can become overwhelming in high-traffic systems, with thousands of spans providing more detail than is actionable. 

While OTEL excels at capturing data, it doesn’t inherently prioritize it. This creates a bottleneck for engineers who need actionable insights quickly. 

The need for top-down analytics 

Along with the benefits of modern observability tooling come challenges that need to be addressed. OpenTelemetry (OTEL) may unify telemetry data collection, but its bottom-up approach leaves teams drowning in redundant metrics, irrelevant logs, and sprawling traces. Without a clear purpose, teams end up collecting everything “just in case,” overwhelming engineers with noise and diluting the actionable insights needed to keep systems running. 

A top-down approach to observability flips the script. Instead of starting with what data is available, it begins with defining the goals: root cause analysis, SLO compliance, or performance optimization. By focusing on purpose, teams can build the analytics required to achieve those goals and then collect only the data necessary to power those insights.  

For example: 

• If the goal is root cause analysis, focus on traces that map dependencies across microservices, rather than capturing every granular log. 
• If the goal is performance optimization, prioritize metrics that highlight latency bottlenecks over exhaustive resource utilization data. 

This shift reduces noise, minimizes data storage and processing costs, and accelerates time-to-insight. 

The cost of ignoring purpose 

The current approach to observability is plagued by fragmentation. Point tools like APMs, native Kubernetes instrumentation, and cloud-specific monitors operate in silos, each with its own data model and semantics. This forces engineers to manually correlate information across dashboards, increasing time to resolution and undermining efficiency. Over time, the storage, compute, and human costs of managing fragmented data become unsustainable. 

Ask yourself: 

How much of your telemetry data is redundant or irrelevant? 

Are your engineers spending more time troubleshooting tools than resolving incidents? 

Is your observability stack delivering insights or merely adding complexity? 

Without a unified purpose and targeted analytics, observability becomes another “big data problem,” and your total cost of ownership (TCO) spirals out of control. 

Causely can help 

Causely transforms OpenTelemetry’s raw telemetry data into actionable insights by applying a top-down, purpose-driven approach. Instead of drowning in logs, metrics, and traces, Causely’s platform leverages built-in causal models and advanced analytics to automatically pinpoint root causes, prioritize issues based on service impact, and predict potential failures before they occur. This turns observability from a reactive big data challenge into a system that continuously assure application reliability and performance. 

How Causely brings focus 

Causely’s platform addresses these challenges head-on. Its causal reasoning starts with defining what matters: actionable insights to keep systems performing reliably and efficiently. Using built-in causal models and top-down analytics, Causely automatically pinpoints root causes and eliminates noise. By integrating with OTEL and other telemetry sources, Causely ensures that only the most critical data is collected, processed, and presented in real time. 

For example: 

• In a microservices architecture, Causely maps dependencies and pinpoints the root cause of cascading failures, reducing MTTR. 
• Similarly, with async messaging systems like Kafka, Causely pinpoints the bottlenecks that cause the consumer lag or delivery failures with actionable context, ensuring faster resolution. 
• In cases where a third-party software is a root cause of issues, Causely pinpoints the root cause   by analyzing services impact. 

This approach not only reduces the TCO of observability but also ensures teams can focus on delivering value rather than managing data. 

How Causely works with OpenTelemetry 

Causely Reasoning Platform is a model-driven, purpose-built Agentic AI system delivering multiple AI workers built on a common data model.   

Causely integrates seamlessly with OpenTelemetry, using its telemetry streams as input while applying context and intelligence to deliver precise, actionable outputs. Here’s how Causely solves common observability challenges: 

• Automated topology discovery: Causely automatically builds a dependency map of your entire environment, identifying how applications, services, and infrastructure components interact. OpenTelemetry’s traces provide raw data, but Causely’s topology discovery transforms it into a visual graph that highlights critical paths and dependencies.  
• Root cause analysis in real time: Using causal models, Causely automatically maps all potential root causes to the observable symptoms they may cause. Causely uses this mapping in real time to automatically pinpoint the root causes based on the observed symptoms, prioritizing those that directly impact SLOs. For instance, when  request latency spikes  are  detected across multiple services, Causely pinpoints whether the spikes stem from a database query (and which database), a messaging queue (and which queue), or an external API (and which one), reducing MTTD and MTTR. 
• Proactive prevention: Beyond solving problems, Causely helps prevent them. Its analytics can simulate “what-if” scenarios to predict the impact of configuration changes, workload spikes, or infrastructure upgrades. For example, Causely can warn you if scaling down a Kubernetes node pool might lead to resource contention under expected load. 

Example 1: Causely, OTEL, and microservices 

In a distributed e-commerce platform, a checkout service experiences intermittent failures. OpenTelemetry traces capture the flow of requests, but the data alone doesn’t explain the root cause. Causely’s causal models analyze the traces and identify that a dependent payment service is timing out due to a slow database query. This insight allows the team to address the issue without wasting time on manual debugging. 

Example 2: Causely, OTEL, and third-party software 

A team - using a third-party CRM API - notices degraded response times during peak hours. OpenTelemetry provides metrics showing increased latency, but engineers are left guessing whether the issue lies with their application or the external service. Causely reasons about the API latency and third-party requests and identifies that the CRM is rate-limiting requests, prompting the team to implement retry logic. 

Example 3: Causely, OTEL, and async messaging with Kafka 

A Kafka-based event pipeline shows sporadic delays in message processing. While OpenTelemetry traces highlight lagging consumers, it doesn’t explain why. Causely, reasoning about the behavior of the consumer microservices, identifies the root cause in the application’s mutex locking which is causing the slow consumption. The engineering team can focus on improving the locking of the data structure without the messaging infrastructure team having to scale up resources and waste time debugging Kafka. 

Reducing the big data burden 

Causely’s approach minimizes the data burden by focusing on relevance. Unlike traditional observability stacks that collect and store massive volumes of telemetry data, Causely processes raw metrics and traces locally, pushing only relevant context (e.g., topology and symptoms) to its backend analytics. This reduces storage and compute costs while ensuring engineers get the insights, they need, without delay. 

Conclusion: Transforming observability with Causely 

OpenTelemetry has redefined observability by standardizing how telemetry data is collected and processed, but its bottom-up approach leaves teams overwhelmed by the sheer volume of logs, metrics, and traces. Observability shouldn’t be about how much data you collect—it’s about how much insight you can gain to keep your systems running efficiently. Without clear prioritization and contextual insights, the observability stack can quickly become a costly burden—both in terms of infrastructure and engineering time. 

Causely integrates seamlessly with OpenTelemetry and helps bring order to the chaos, empowering teams to make smarter, faster decisions that directly impact reliability and user experience. Causely uses causal models, automated topology discovery and real-time analytics to pinpoint root causes, prevent incidents, and optimize performance. This reduces noise, eliminates unnecessary data collection, and allows teams to focus on delivering reliable systems rather than managing observability overhead. 

Ready to move beyond data overload and transform your observability strategy? Book a demo or start your free trial to see how Causely can help you take control of your telemetry data and build more reliable cloud-native applications. 

As an industry, we have made progress in many areas. But when it comes to technology operations and application management fundamentals, we are far from the desired state. No matter what new terms and buzz words people use, the industry still has a way to go. The reality is that no magic black box or new AI trend will get us there on its own. 

Autonomous Service Reliability requires a system that autonomically keeps all applications performing and meeting their objectives while satisfying their operational constraints (i.e. the “Desired State”). To continuously maintain the Desired State, the system needs to:  

• Assess whether all the applications are in their Desired State 
• Pinpoint the root cause(s) and identify the actions that will get the applications that are not in their Desired State back to their Desired State  
• Determine what actions will prevent applications from getting out of their Desired State in the first place 
• Continuously assess environment trends to identify what actions should be taken to prevent deviation from the Desired State 

These are the goals of our Causal Reasoning Platform, which is a model-driven, purpose-built Agentic AI system that includes multiple AI Workers built on a common data model. These AI Workers collaborate seamlessly to continuously assure application reliability and performance. Each of these workers utilizes specific analytics and they all share the common components of the Causal Reasoning Platform.  

There are seven AI Workers delivered by our Causal Reasoning Platform, sharing six common components. In this post, we’ll first introduce the six common components followed by a description of the seven workers, explaining what each worker is doing and how the platform works.  

Causal Models

The Causal Reasoning Platform is driven by Causal Models. Causely is delivered with built-in Causal Models that capture the root causes that can degrade application performance. These Causal Models enable Causely to automatically pinpoint root causes as soon as it is deployed in an environment with zero configuration.

There are at least a few important details to highlight about these Causal Models:  

• They capture potential root causes in a broad range of entities including applications, databases, caches, messaging, load balancers, DNS, compute, storage, and more. 
• They describe how the root causes will propagate across the entire environment and what symptoms may be observed when each of the root causes occurs.  
• They are completely independent from any specific environment and are applicable to any modern application environment.  

Attribute Dependency Models

Causely is delivered with built-in Attribute Dependency Models that extend the Causal Models to capture the dependencies between attributes across entities and the constraints attributes must satisfy. These Attribute Dependency Models enable Causely to automatically correlate performance trends across the entire environment, figure out the Desired State (as described earlier) and the actions to keep the environment in that state. 

There are at least a few important details to highlight about these Attribute Dependency Models:  

• They can capture attribute dependencies in a broad range of entities including services, applications, databases, caches, messaging, load balancers, DNS, compute, storage, and more. 
• They describe the functions between the attributes, but more importantly the functions can be learned.  
• They describe the desired state in terms of the applications' goals and the constraints they should operate within.  
• They are completely independent from any specific environment and are applicable to any modern application environment.

Automatic Topology Discovery

Cloud-native environments are a tangled web of applications and services layered over complex and dynamic infrastructure. Causely automatically discovers all the entities in the environment including the applications, services, databases, caches, messaging, load balancers, compute, storage, etc., as well as how they all relate to each other.

For each discovered entity, Causely automatically discovers its:  

• Connectivity - the entities it is connected to and the entities it is communicating with horizontally  
• Layering - the entities it is vertically layered over or underlying 
• Composition - what the entity itself is composed of  

Causely automatically stitches all of these relationships together to generate a Topology Graph, which is a clear dependency map of the entire environment. This Topology Graph updates continuously in real time, accurately representing the current state of the environment at all times. 

Automatic Causality Mapping Generation

Using the out-of-the-box Causal Models and the Topology Graph as described above, Causely automatically generates a causal mapping between all the possible root causes and the symptoms each of them may cause, along with the probability that each symptom would be observed when the root cause occurs. 

Causely automatically generates two data structures to capture the causality mapping:

• A Causality Graph is a directed acyclic graph (DAG), where the nodes are root causes and symptoms and the edges represent the causality, i.e., an edge from node A to node B means that A may cause B. The edges are labeled with the probability of the causality.
• A Codebook is a table where the columns represent the root causes and the rows represent the symptoms. Each column is a vector of probabilities defining a unique signature of the root cause. A cell in the vector represents the probability that the root cause may cause the symptom.

Automatic Attribute Dependency Graph Generation

Using the out-of-the-box Attributes Dependency Model and the Topology Graph as described above, Causely automatically generates an Attribute Dependency Graph. 

The Attribute Dependency Graph is a directed acyclic graph (DAG) where:

• The  nodes are attributes.
• The edges represent a dependency between the attributes. For example, an edge from attribute A to attribute B means that the value of B is a function of attribute A. 
• The edges are labeled with the functions. The functions can be defined in the Attributes Dependency Model or can be learned if they are not defined in the Model. 
• The nodes representing attributes that must satisfy a constraint will be decorated with the constraint the attribute must satisfy. 

Contextual Presentation

We believe explainability is a critical capability for AI-driven systems to demonstrate. For this purpose, the system presents its work intuitively in the Causely UI. This enables users to see the root causes, related symptoms, the service impacts and initiate actions. These insights can also be sent to external systems to initiate incident response workflows as well as to notify teams who are responsible for taking action and/or those whose services are impacted.

The Models, the automated topology discovery, and the automatic generation of the Causality Mapping and the Attribute Dependency Graph empower multiple AI workers that together deliver an autonomous application reliability system that continuously assures application performance.

Root Cause Analysis (RCA) Worker

The RCA Worker uses the Codebook described above to automatically pinpoint root causes based on observed symptoms in real time. No configuration is required for the worker to immediately pinpoint a broad set of root causes (100+), ranging from applications malfunctioning to services congestion to infrastructure bottlenecks.  

In any given environment, there can be tens of thousands of different root causes that may cause hundreds of thousands of symptoms. Causely prevents SLO violations by detangling this mess and pinpointing the root cause putting your SLOs at risk and driving remediation actions before SLOs are violated. For example, Causely proactively pinpoints if a software update changes performance behaviors for dependent services before those services are impacted. 

Performance Analysis Worker

The Performance Analysis Worker uses the Attribute Dependency Graph and Causality Graph to analyze microservices performance bottleneck propagation by automatically learning, based on your data:  

• The correlation between the loads on services, i.e., how a change in load of one cascades and impacts the loads on other services; 
• The correlation between services latencies, i.e., how latency of one cascades and impacts the latencies of other services; and  
• The likelihood a service or resource bottleneck may cause performance degradations on dependent services. 

Constraint Analysis Worker

The Constraint Analysis Worker uses the Attribute Dependency Graph decorated with performance goals like throughput and latency, as well as capacity and/or cost constraints, to automatically compute the Desired State of the environment and to figure out what actions need to be taken to assure the goals are accomplished while satisfying the defined constraints.  

Prevention Analysis Worker

The Prevention Analysis Worker uses the Causality Graph and the Attribute Dependency Graph to enable prevention analysis. Teams are empowered to analyze the potential impacts or problems of changes.

Teams can ask "what if'' questions to:

• Understand the services that may be degraded if a potential problem were to occur
• Understand the impact a planned change may have on services

In doing so, teams can support planning of service/architecture changes, maintenance activities, and service resiliency improvements, and assure that none of these cause unexpected outages that may dramatically impact the business.

Predictive Analysis Worker

The Predictive Analysis Worker uses machine learning (ML) together with the Causality Graph and the Attribute Dependency Graph for predictive analysis. Causely uses:

• ML to analyze the performance behavior of a small subset of attributes, e.g. some services loads, to predict their trends.
• The Attribute Dependency Graph and the predicted trends to predict the state of the environment, i.e., the state of all the attributes.
• The Causality Graph and the predicted future state to pinpoint potential bottlenecks and suggest actions that may prevent bottlenecks.

In doing so, Causely pinpoints the actions required to prevent future degradations, SLO violations, or constraint violations.

Service Impact Analysis Worker

The Service Impact Analysis Worker uses the Causality Graph to automatically analyze the impact of the root causes on SLOs, prioritizing the root causes based on the violated SLOs and those that are at risk. Causely automatically defines standard SLOs (based on latency and error rate) and uses machine learning to improve its anomaly detection over time. However, environments that already have SLO definitions in another system can easily be incorporated in place of Causely’s default settings. 

Postmortem Analysis Worker

The Postmortem Analysis Worker uses the Causality Graph to save the relevant context of prior incidents to enable postmortem analysis. Causely saves the root cause, the Causality Graph of the root cause, the symptoms in the Causality Graph and the relevant attribute trends. Teams can review prior incidents and see clear explanations of why these occurred and what the effect was, simplifying the process of postmortems and enabling actions to be taken to avoid re-occurrences.  

See Causely for Yourself!

Book a meeting with the Causely team and let us show you how to transform the state of escalations and cross-organizational collaboration in cloud-native environments, or start your free trial now. 

Collecting “more data” has been the defining characteristic of observability practices and tools for the last few decades. The allure of multiple tools that instrument virtually every layer of the stack, from applications to infrastructure and beyond, traps us into believing that more data equals more insights. But the reality is often the opposite: over-collection creates inefficiencies, noise, and cost without adding meaningful value. As I said during a recent podcast interview, it may make you less blind, but not necessarily smarter.   

Instead of clarifying what’s happening in your systems, excessive data leads to confusion. Teams are inundated with redundant metrics, irrelevant logs, and false-positive alerts. Each new piece of data requires engineers to interpret, process, and correlate, pulling focus away from actionable insights. As a result, observability becomes a “big data” problem, where the volume of information collected outweighs its usefulness. 

We always collected too much data, but microservices architectures and the new emerging instrumentation, eBPF and OpenTelemetry, exacerbate the problem. With tens of thousands of microservices, each collecting even a few metrics every few seconds, adds up very quickly to millions of data points requiring overloaded processing and terabytes of storage. The realization that we need application instrumentation to get insights about the higher layer on the stack and the interactions between services led to the adoption of eBPF and OpenTelemetry, but with these came a tsunami of traces that overwhelms observability tools and makes them practically useless. 

This trajectory must and can be changed. 

When collecting data offers diminishing returns 

Collecting data turns observability into a “big data problem” instead of its intended purpose: to keep systems up and running. As a result, you don’t simply apply a band-aid to a bullet hole, you end up applying band-aids upon band-aids upon band-aids. Each new big data problem results in more big data solutions, which result in more big data problems, and the cycle continues. 

Lack of purpose creates noise 

Observability data is often collected without a clear sense of purpose. Teams instrument everything they can, thinking it might be useful someday, but this creates a flood of irrelevant information. The result is alert fatigue, where low-priority or false-positive alerts drown out the critical signals engineers need to act on. In Kubernetes environments, for example, native instrumentation combined with third-party tools often generates more metrics than teams can effectively use, adding complexity instead of clarity. 

Earlier this week, we wrote about the difficulties of troubleshooting problems in production. Excessive data only makes the haystack larger; it doesn’t help you find the needle any faster. 

Ask yourself: what percentage of the collected and stored data is ever being looked at? What is the cost of processing and storing this data? 

Fragmentation of observability tools 

Observability is highly fragmented, with different tools collecting and processing data in silos. Each technology or layer of the application stack is monitored by a point tool that monitors that specific technology or layer. The applications may be monitored by an APM tool like Datadog, New Relic, or Dynatrace. The services layer may be monitored by eBPF and OpenTelemetry. The Kubernetes infrastructure may be monitored by the native Kubernetes instrumentation. Cloud services may be monitored by cloud provider’s tools like CloudWatch.  And the list goes on. 

Some of these tools collect metrics, some collect traces, and others ingest logs or events. Regardless of whether these point tools come from different vendors or are provided by a single vendor, each one of them has its own data model. They lack not only a unified data model, but a common semantic model to interpret the data to analyze the data across the entire stack and to provide actionable insights. Engineers are left to manually correlate information across dashboards, increasing time to resolution. This fragmentation creates inefficiencies that undermine the very purpose of observability: delivering actionable insights. 

Ask yourself:

How many different point tools are you using?
Which one do you use when and why? 
How much of the data is redundant? 

Overhead of storing and processing data 

Storing and analyzing vast amounts of observability data comes at a cost—both financial and operational. Many tools charge by data volume, making it cost-prohibitive to retain all collected data. Even for teams with generous budgets, processing and querying such large datasets consumes time and resources, often delivering diminishing returns. 

Ask yourself:

Are you in the business of collecting observability data or are you in the business of keeping your systems running reliably and efficiently? 

A top-down approach to observability 

 The solution isn’t more data—it’s the right data. A top-down approach flips the traditional model of observability on its head. Instead of starting with what data is available, start with the purpose: what are you trying to achieve?

Define the goals first—whether it’s root cause analysis, SLO compliance, or performance optimization—and then work down to identify the data needed to support those goals. 

Building the right analytics is critical. Analytics should be purpose-built to deliver insights that directly address the goals you’ve defined. For example, if the goal is to identify bottlenecks, focus on analytics that analyzes service latencies, instead of drowning in utilization metrics.   

Finally, only collect the data you need. With the purpose and analytics in place, you can focus on gathering metrics and logs that directly contribute to actionable insights. This reduces noise, eliminates unnecessary alerts, and lowers the overhead of storing and processing data. 

Causely helps you focus on what matters 

Our Causal Reasoning Platform is a model-driven, purpose-built AI system delivering multiple analytics built on a common data model. It is designed to minimize data collection while maximizing insight. (You can learn more about how it works here).

As a result, Causely is laser focused on collecting only the metrics, traces, and logs required as input for the above analytics. This dramatically reduces the amount of data collected by Causely. Furthermore, all the raw metrics and traces are processed locally in real time, and none if it is pushed out to the cloud. The only information pushed to the back-end analytics in the cloud is the topology and symptoms. The only data stored in the cloud are the pinpointed root causes and the contextual information associated with the root causes. This minimizes Causely's TCO! 

Conclusion 

Collecting more data doesn’t guarantee better observability. In fact, it often creates more problems than it solves. A top-down approach—starting with purpose, building targeted analytics, and focusing on necessary data—streamlines observability and makes it actionable. Platforms like Causely enable this approach, helping teams move beyond the big data trap and focus on delivering real value. Observability isn’t about how much data you collect—it’s about how much insight you can gain. 

This week, we covered numerous ways teams can shift their posture from reactive infrastructure fixes to more proactive observability. Organizations that are more proactive will enjoy several benefits, including: 

• Spending less time troubleshooting 
• Preventing escalations from devolving into finger-pointing and blame 
• Predicting the impact of changes before they are deployed 
• Planning for future environment and infrastructure changes 

Book a meeting with the Causely team and let us show you how you can bridge the gap between development and observability to build better, more reliable cloud-native applications. 

Yesterday, we talked about being more proactive about the code our developers write, anticipating how code changes will impact our systems and infrastructure more broadly. Now, we turn our attention to planning ahead for external factors that could affect our systems.  

Environmental changes are inevitable in any complex system. Whether it’s a sudden traffic spike, an unexpected hardware failure, or gradual infrastructure degradation, external pressures can quickly destabilize even the most carefully designed environments. The challenge lies in anticipating these changes before they disrupt services. In distributed systems, where small disruptions can ripple across dependencies, waiting until issues surface is simply not an option. 

Proactive observability provides a necessary safeguard against these challenges. By identifying potential risks in real time, predicting future demand, and adapting resources dynamically, teams can maintain reliability even under extreme conditions. This isn’t about eliminating unpredictability; it’s about building systems that respond intelligently to it. 

Challenges to being more proactive about environment changes 

Three core challenges make proactive management of environmental changes difficult: predicting load trajectories, gaining forward-looking visibility into the broader system, and scaling resources efficiently. While various tools exist to address parts of these problems, they often fall short in providing the level of predictability, visibility, and adaptability that modern systems require. 

Predicting future load is difficult 

Predicting how traffic patterns will evolve is one of the most complex aspects of managing modern systems. Seasonal events, marketing campaigns, or even unforeseen surges can create spikes in demand that exceed system capacity.  

When managing an increased number of users or handling heavier loads, developers must evaluate how the system handles scaling demands. This involves monitoring how many API calls are made to backend services for each user request and modeling the anticipated growth in API usage based on projected user activity. Monitoring tools can provide real-time data to understand current usage patterns, but they cannot predict the behavior of dependent services. Load-testing tools like Grafana Labs' k6 and Apache JMeter are commonly used during development to simulate expected traffic patterns. These tools are useful for pre-deployment stress testing but are static by design—they can’t account for real-time changes in traffic behavior or external variables like regional events or user behavior shifts. As a result, engineers often find themselves underprepared, scrambling to adjust capacity once demand has already outpaced infrastructure. 

Forward-looking visibility into evolving systems is limited 

Proactively managing environmental changes requires visibility not just into what’s happening now but into how the system is likely to evolve. For instance, if a service is handling a steadily increasing workload, engineers need to predict how soon it will hit a performance bottleneck and which dependent services may be affected.  

For systems using messaging architectures, such as event-driven designs, it’s crucial to evaluate the rate at which messages are published and consumed. Developers need to assess whether consumers can process messages fast enough to keep up with the increased production rate. This may involve checking the throughput of message queues (e.g., Kafka, RabbitMQ) and ensuring sufficient worker threads or instances are available to process the messages without creating backlogs. 

Database performance is another critical factor. As the number of user requests grows, so does the frequency of database queries. Developers must analyze the types of queries executed, the tables affected, and whether the database schema or indexes need optimization. Load testing can simulate increased query volume to predict the impact on database performance and resource usage. This analysis helps in determining whether the existing database can handle the load with vertical or horizontal scaling or if architectural changes, such as database sharding or read replicas, are necessary. 

Current tools often provide snapshots of real-time metrics but lack the ability to project those metrics forward into the future. 

Datadog and New Relic offer dashboards that display system dependencies and performance under current conditions, but they focus on reactive analysis. Similarly, Prometheus provides excellent real-time monitoring for resource metrics like CPU or memory utilization but does not include predictive capabilities for identifying future stress points.  

This lack of forward-looking insight forces engineers into a reactive posture, leaving them vulnerable to issues that could have been anticipated and avoided. 

Scaling resource efficiency is complicated 

Scaling infrastructure to meet demand is both a reliability and cost challenge. Reactive scaling, where systems respond only after resource thresholds are exceeded, risks under-provisioning during critical moments, leading to downtime. On the other hand, over-provisioning wastes resources and drives up operational costs.  

Tools like Kubernetes Horizontal Pod Autoscaler (HPA) and AWS Auto Scaling automate scaling based on resource metrics like CPU or memory utilization. While these tools are helpful, they are fundamentally reactive—they adjust only after usage patterns have shifted. They lack the predictive capabilities to anticipate future demand and scale resources proactively, leaving teams vulnerable to sudden surges or inefficiencies. 

What we require to be more proactive 

Proactively managing environmental changes requires observability systems that go beyond traditional monitoring and alerting. These systems must not only detect anomalies in real time but also provide predictive insights and adaptive workflows that empower teams to anticipate and mitigate risks before they impact users. Here are the critical capabilities needed to achieve this: 

• Real-time anomaly detection at scale. Identifying traffic surges, resource bottlenecks, or unexpected failures as they occur. Systems should process vast amounts of data with minimal latency, identifying deviations from normal patterns. Early detection prevents small issues from spiraling into widespread outages. 
• Predictive analytics for load and capacity needs. Beyond real-time data, teams also need analytics that use historical and live data to forecast future system behavior. These systems should forecast where and when additional capacity will be needed, including across long-term patterns such as seasonal demand spikes. Predictive insights allow teams to allocate resources proactively. 
• Integrated resource management. Armed with data about what to do, systems should be deeply integrated so that teams can act quickly on their predictive knowledge and act. 
• Comprehensive system modeling. Teams need the ability to test the impact of environmental changes before they happen. This includes simulating traffic surges, dependency failures, or configuration changes in a controlled environment to identify potential risks and validate resilience. 
• Clear, actionable insight. Systems must provide insights that are not only detailed but also actionable. This includes translating complex system behavior into clear recommendations that engineers and decision-makers can act on quickly. Think context-rich alerts and dashboards that highlight the root cause of anomalies, the potential impact of changes, and the recommended actions to mitigate it all. 

Achieving true proactive observability requires more than just reacting to anomalies. It demands systems that predict, adapt, and empower teams to address environmental changes before they become critical. With these capabilities in place, organizations can ensure their systems remain stable and reliable, no matter what external pressures arise. 

Causely delivers a proactive observability system 

Our Causal Reasoning Platform is a model-driven, purpose-built AI system delivering multiple analytics built on a common data model. You can learn more about its capabilities that uniquely help teams proactively anticipate and plan for environment changes here.

Conclusion 

Environmental changes like traffic spikes or hardware degradation are inevitable, but their impact doesn’t have to mean downtime. Reacting to problems after they occur leaves teams in firefighting mode, scrambling to restore stability. The key to reliability isn’t just responding quickly—it’s anticipating risks and adapting before issues arise. 

Proactive observability makes this possible. With real-time anomaly detection, predictive analytics, and adaptive workflows, teams can prevent disruptions and maintain stability under changing conditions. By adopting tools and practices that prioritize foresight over reaction, engineers can shift from firefighting to proactive system management, ensuring their systems remain reliable no matter what comes next. 

So far this week, we’ve talked about the difficulties involved with troubleshooting and the complexity of dealing with escalations. What if we could head off these problems before they even happen? 

Making changes to production environments is one of the riskiest parts of managing complex systems. Even a small, seemingly harmless tweak—a configuration update, a database schema adjustment, or a scaling decision—can have unintended consequences. These changes ripple across interconnected services, and without the right tools, it’s nearly impossible to predict their impact. 

Business needs speed and agility in introducing new capabilities and features but without scarifying reliability, performance and predictability. Hence, engineers need to know how a new feature, or even a minor change, will affect performance, reliability, and SLO compliance before it goes live. But existing observability tools lack the required capabilities to provide useful insights that enable engineers to safely deploy new features. The result? Reduced productivity, slowdown in feature development, and reactive firefighting when changes go wrong -- all leading to downtime, stress, and diminished user trust. 

Recently, we worked with a customer who sought to shift their reactive posture to one that more aggressively seeks out problems before they happen.  With this customer, a bug within one of their microservices (the data producer) caused the producer to stop updating the Kafka topic with new events.  This created a backlog of events for all the consumers of this topic.  As a result, their customers were looking at stale data.  Problems like this lead to poor customer experience and revenue loss, so many customers need to adopt a preventative mode of operations.

This post explores how this trend can be disrupted by providing the analytics and the reasoning capabilities to transform how changes are made, empowering teams to anticipate risks, validate decisions, and protect system stability—all before the first line of code is deployed. 

Being proactive is easier said than done 

Change management is a process designed to ensure changes are effective, resolve existing issues, and maintain system stability without introducing new problems. At its core, this process requires a deep understanding of both the system’s dependencies and its state before and after a change.  

Production changes are risky because engineers typically lack sufficient visibility into how changes will impact the behavior of entire systems. What seems like an innocuous change to an environment file or an API endpoint could have far-reaching ramifications that aren’t always obvious to the developer. 

While observability tools have come a long way in helping teams monitor systems, they lack the analytics required to understand, analyze, and predict the reliability and performance behavior of cloud-native systems. As a result, engineers are left to “deploy and hope for the best” ... and get a 3AM call when things didn’t work as expected. And, while we live in a veritable renaissance of developer tooling, most of these tools focus on developer productivity, not developer understanding, of whole systems and the consequences of changes made to one component or service. 

It’s hard to predict the impact of code changes 

When planning a change, the priority besides adding new functionality is to confirm whether it addresses the specific service degradations or issues it was designed to resolve. Equally important is ensuring that the change does not introduce new regressions or service degradations. 

Achieving these goals requires a comprehensive understanding of the system’s architecture, particularly the north-south (layering) dependencies and the east-west (service-to-service) interactions. Beyond mapping the topology, it is crucial to understand the data flow within the system—how data is processed, transmitted, and consumed—because these flows often reveal hidden interdependencies and potential impact areas.  

Even minor configuration changes can create cascading failures in distributed systems. For instance, adjusting the scaling parameters of an application might inadvertently overload a backend database, causing performance degradation across services. Engineers often rely on experience, intuition, or manual testing, but these methods can’t account for the full complexity of modern environments. 

Unpredictable performance behavior of microservices  

As we discussed in our post yesterday, loosely coupled microservices communicate with each other and share resources. But which services depend on which? And what resources are shared by which services? These dependencies are continuously changing and, in many cases, unpredictable.  

A congested database may cause performance degradations of some services that are accessing the database. But which one will be degraded? Hard to know. Depends. Which services are accessing which tables through what APIs? Are all tables or APIs impacted by the bottleneck? Which other services depend on the services that are degraded? Are all of them going to be degraded?  

These are very difficult questions to answer. As a result, analyzing, predicting, and even just understanding the performance behavior of each service is very difficult. Furthermore, using existing brittle observability tools to diagnose how a bottleneck cascades across services is practically impossible.  

There’s a lack of “what-if” analysis tools for testing resilience 

Even though it’s important to simulate and test the impact of changes before deployment, the tools currently available are sorely lacking. Chaos engineering tools like Gremlin and Chaos Monkey simulate failures, but don’t evaluate the impact of configuration changes. Tools like Honeycomb provide event-driven observability, but don’t help much with simulating what will happen with new builds. Inherently, if the tools can’t analyze the performance behavior of the services, they can’t support any “what-if” analysis. 

Developer tools are inherently focused on the “build and deploy” phases of the software lifecycle, meaning they prioritize pre-deployment validation over predictive insights. They don’t provide answers to critical questions like: “How will this change impact my service’s reliability or my system’s SLOs?” or “Will this deployment create new bottlenecks?” 

Predictive insights require correlating historical data, real-time metrics, dependency graphs, and most importantly deep understanding of the microservices performance behaviors. Developer tools simply aren’t built to ingest or analyze this kind of data at the system level. 

Developer and operations tools today are both insufficient 

Developer tools are essential for building functional, secure, and deployable code, but they are fundamentally designed for a different domain than observability. Developer tools focus on ensuring “what” is built and deployed correctly, while observability tools aim to identify “when” something is happening in production. The two domains overlap but rarely address the full picture. 

 Bridging this gap often involves integrating developer workflows—such as CI/CD pipelines—with observability systems. While this integration can surface useful metrics and automate parts of the release process, it still leaves a critical blind spot: understanding “why” something is happening. Neither traditional developer tools nor current observability platforms are designed to address the complexity of dynamic, real-world systems. 

To answer the “why,” you need a purpose-built system to unravel the interactions, dependencies, and behaviors that drive modern production environments. 

Building for reliability 

Building reliable performing applications was never easy, but it has become much harder. As David Shergilashvili correctly states in his recent post Microservices Bottlenecks, “In modern distributed systems, microservices architecture introduces complex performance dynamics that require deep understanding. Due to their distributed nature, service independence, and complex interaction patterns, microservices systems' performance characteristics differ fundamentally from monolithic applications.” 

Continuing to collect data and presenting it to developers in pretty dashboards with very little or no built-in analytics to provide meaningful insights won’t get us to build reliable distributed microservices applications. 

To accurately assess the impact of a change, the state of the system must be assessed both before and after the change is implemented. This involves monitoring key indicators such as system health, performance trends, anomaly patterns, threshold violations, and service-level degradations. These metrics provide a baseline for evaluating whether the change resolves known issues and whether it introduces new ones. However, the ultimate goal goes beyond metrics; it is to confirm that the known root causes of issues are addressed and that no new root causes emerge post-change.  

We need to build systems that enable engineers to introduce new features quickly, efficiently, and most importantly safely, i.e., without risking the reliability and performance of their applications. Reasoning platforms with built-in analytics need to provide actionable insights that anticipate implications and prevent issues. These systems must have the following capabilities: 

• Services dependencies. Be able to capture, represent and auto-discover service dependencies. Without this knowledge a developer can’t know and anticipate which services may be impacted by a change. 
• Causality knowledge. Be aware of all potential failures and the symptoms they may cause. Without these insights, a developer can’t assess the risk of introducing a change. Is the application code I am about to change a single point of failure? What will happen if for some reason my new code won’t perform as expected? 
• Performance analysis. Understanding the blast radius of performance bottlenecks. Without having a deep understanding of the interactions between the microservices as well as their shared resources, developers can’t predict all possible situations that may occur in production. Even the best tester can’t cover all possible scenarios. How can you know how a noisy neighbor will impact your application in production?  
• What-if. The ability to assess load fluctuations before they happen or potential impacts of code changes to be introduced. Is your service ready for the predicted Black Friday traffic? What about the dependent upstream service? 

Causely can help you build better cloud-native applications 

Our Causal Reasoning Platform is a model-driven, purpose-built AI system delivering multiple analytics built on a common data model. You can learn more about what makes it unique here.

With Causely, you can go from reactive troubleshooting to proactive design, development, and deployment of reliable cloud-native applications. 

Conclusion 

Making changes to production environments shouldn’t be a guessing game. Without the right systems, even the smallest change can introduce risks that compromise reliability, breach SLOs, and erode user trust. Reactive approaches to troubleshooting only address problems after they occur, leaving engineers stuck firefighting rather than innovating. 

A purpose-built reasoning platform can flip this narrative. By incorporating predictive analytics, “what-if” modeling, and automated risk detection, teams can anticipate issues before they arise and deploy with confidence. Platforms like Causely empower engineers with the insights they need to write reliable code, validate changes, and ensure system stability at every step. 

Book a meeting with the Causely team and let us show you how you can bridge the gap between development and observability to build better, more reliable cloud-native applications. 

Microservices architectures introduce complex, dynamic dependencies between loosely coupled components. In turn, these dependencies lead to complex, hard to predict interactions. In these environments, any resource bottleneck, or any service bottleneck or malfunction, will cascade and affect multiple services, crossing team boundaries. As a result, the response often spirals into a chaotic mix of war rooms, heated Slack threads, and finger-pointing. The problem isn’t just technical—it’s structural. Without a clear understanding of dependencies and ownership, every team spends more time defending their work than solving the issue. It’s a waste of effort that undermines collaboration and prolongs downtime. 

Yesterday, we resolved to spend less time troubleshooting in 2025. 

Troubleshooting and escalation are closely intertwined. A single unresolved bottleneck can ripple outward, forcing multiple teams into reactive mode as they struggle to isolate the true root cause. This dynamic creates inefficiencies and delays, with teams often focusing on band-aiding symptoms instead of remediating and solving the root causes. To eliminate this friction, we need systems that do more than detect anomalies—they must provide a seamless view of dependencies, understand and analyze the performance behaviors of the microservices, assign ownership intelligently, and guide engineers toward resolution with precision and context. 

Take, for example, an application developer who notices high request duration for users who are trying to interact with their application. This application communicates with many different services, and it happens to run within a container environment on public cloud infrastructure.  There are more than 50 possible root causes that might be causing the high request duration issue.  That developer would need to investigate garbage collection issues, disk congestion, app-locking problems, and node congestion among many other potential root causes until accurately determining that a congested database is the source of their problem.  The only proper way to determine root cause is by considering all the cause-and-effect relationships between all the possible root causes and the symptoms they may cause. This process can often take hours or days before the correct root cause is pinpointed, resulting in a variety of business consequences (unhappy users, missed SLOs, SLA violations, etc.). 

In this post, we’ll explore the challenges of multi-team escalations, and the capabilities needed to address them. From automated dependency mapping to explainable triage workflows, we’ll show how observability can be transformed from chaos into clarity, making escalations less contentious and far more productive. 

Escalations can cripple teams 

Escalations create inefficiencies that extend downtime, frustrate teams, and waste resources. These inefficiencies stem from a combination of structural and technical gaps in how dependencies are understood, root causes are isolated, and ownership is assigned. Here are some of the key challenges that make escalations so painful today: 

• There is a lack of cross-team visibility into dependencies 
• It can be hard to predict or analyze the performance behaviors of loosely coupled dependent microservices  
• It can be difficult to isolate the root cause among all affected services 
• Legacy observability tools must be stitched together to provide even partial visibility into issues 

Lack of cross-team visibility 

Microservices architectures are complex and full of deeply interconnected components. An issue in one can cascade into others. Without clear visibility into these dependencies, teams are left guessing which components are impacted and which team should take ownership. 

Your favorite observability tools help you visualize dependencies, but they lack real-time accuracy. These maps can quickly become outdated in environments with frequent changes. Some of them are great for aggregating logs, but don’t offer much insight into service relationships. Engineers are often left to piece together dependencies manually. 

Unpredictable performance behavior of microservices  

Loosely coupled microservices communicate with each other and share resources. But which services depend on which? And what resources are shared by which services? These dependencies are continuously changing and, in many cases, unpredictable.  

A congested database may cause performance degradations of some services that are accessing the database. But which one will be degraded? Hard to know. Depends. Which services are accessing which tables through what APIs? Are all tables or APIs impacted by the bottleneck? Which other services depend on the services that are degraded? Are all of them going to be degraded? These are very difficult questions to answer.  

As a result, predicting, understanding and analyzing the performance behavior of each service is very difficult. Using existing brittle observability tools to diagnose how a bottleneck cascades across services is practically impossible.  

Difficulty identifying root causes among all affected services 

Determining what’s a cause and what’s a symptom can be an incredibly time-consuming aspect of troubleshooting and escalations. Further, the person or team identifying a problem may well be looking at only their local maxima: the part of the system they work on or are directly affected by. They often don’t see the full picture of all intertwined systems. Identifying the root cause among all affected services can be inordinately difficult. 

Even if you have tools that are excellent for visualizing time-series data, you must still rely on engineers to manually correlate metrics. APM tools can help you examine application performance but require significant manual effort to link symptoms to underlying causes, especially in microservices-based, cloud-native applications. 

Legacy observability tooling only gives you partial functionality 

While both established and up-and-coming tools offer valuable capabilities, they often address only one part of the problem, leaving critical gaps. Dependency visibility, performance analysis and root cause isolation need to be integrated seamlessly to reduce the chaos of escalations. Today’s tools, however, are fragmented, requiring engineers to bridge the gaps manually, costing valuable time and effort during incidents. Solving these problems demands a holistic approach that ties all these elements together in real time. 

How escalations should be handled 

Escalations have negative consequences for organizations of all sizes. Let’s work together to build systems that render escalations less about blame and more about opportunities to foster trust and collaboration. These systems will have the following capabilities: 

• End-to-end discovery of service dependencies. Automatically discover and maintain, in real time, a complete view of how systems interact. 
• Workflow integration that directs root cause resolution to the correct team. Use the tools your organization has already invested in to turn root causes into actions for the correct team, reducing delays caused by miscommunication. 
• Performance analysis of bottlenecks propagations. Provide insights on how bottlenecks cascade across services. 
• Detailed identification of root causes across an entire system. Empower engineers to act confidently without over-reliance on senior team members. 

With these new systems, escalations can result in positive business outcomes: 

• Deep, real-time understanding of complete microservices systems 
• Better collaboration between teams to remediate issues as they arise 
• More innovation, less time in war rooms 
• Empowered engineering teams that solve problems instead of pointing fingers 

Causely helps you handle escalations quickly and confidently 

Our Causal Reasoning Platform is a model-driven, purpose-built AI system delivering multiple analytics built on a common data model. It includes several features to help you understand issues and handle escalations efficiently: 

• Out-of-the-box Causal Models. Causely is delivered with built-in causality knowledge capturing the common root causes that can occur in cloud-native environments. This causality knowledge enables Causely to automatically pinpoint root causes out-of-the-box as soon as it is deployed in an environment. There are at least a few important details to share about this causality knowledge:  
  • It captures potential root causes in a broad range of entities including applications, databases, caches, messaging, load balancers, DNS, compute, storage, and more. 
  • It describes how the root causes will propagate across the entire environment and what symptoms may be observed when each of the root causes occurs.  
  • It is completely independent from any specific environment and is applicable to any cloud-native application environment.  
• Automatic topology discovery. Cloud-native environments are a tangled web of applications and services layered over complex and dynamic infrastructure. Causely automatically discovers all the entities in the environment including the applications, services, databases, caches, messaging, load balancers, compute, storage, etc., as well as how they all relate to each other. For each discovered entity, Causely automatically discovers its:  
  • Connectivity - the entities it is connected to and the entities it is communicating with horizontally  
  • Layering - the entities it is vertically layered over or underlying 
  • Composition - what the entity itself is composed of  
    
    Causely automatically stitches all of these relationships together to generate a Topology Graph, which is a clear dependency map of the entire environment. This Topology Graph updates continuously in real time, accurately representing the current state of the environment at all times. 
• Root cause analysis. Using the out-of-the-box Causal Models and the Topology Graph as described above, Causely automatically generates a causal mapping between all the possible root causes and the symptoms each of them may cause, along with the probability that each symptom would be observed when the root cause occurs. Causely uses this causal mapping to automatically pinpoint root causes based on observed symptoms in real time. No configuration is required for Causely to immediately pinpoint a broad set of root causes (100+), ranging from applications malfunctioning to services congestion to infrastructure bottlenecks.  
  
  In any given environment, there can be tens of thousands of different root causes that may cause hundreds of thousands of symptoms. Causely prevents SLO violations by detangling this mess and pinpointing the root cause putting your SLOs at risk and driving remediation actions before SLOs are violated. For example, Causely proactively pinpoints if a software update changes performance behaviors for dependent services before those services are impacted. 
• Performance analysis. Causely analyzes microservices performance bottleneck propagation by automatically learning, based on your data:  
  • the correlation between the loads on services, i.e., how a change in load of one cascades and impacts the loads on other services; 
  • the correlation between services latencies, i.e., how latency of one cascades and impacts the latencies of other services; and  
  • the likelihood a service or resource bottleneck may cause performance degradations on dependent services. 
• Constraints analysis. Causely uses performance goals like throughput and latency, and capacity or cost constraints, and automatically figures out what actions need to be taken to assure the goals are accomplished while satisfying the constraints.  
• Prevention analysis. Teams can also ask "what if'' questions to understand the impact that potential problems might have if they were to occur to support the planning of service/architecture changes, maintenance activities, and service resiliency improvements.
• Predictive analysis. Causely automatically analyzes performance trends and pinpoints the actions required to prevent future degradations, SLO violations, or constraints.  
• Service impact analysis. Causely automatically analyzes the impact of the root causes on SLOs, prioritizing the root causes based on the violated SLOs and those that are at risk. Causely automatically defines standard SLOs (based on latency and error rate) and uses machine learning to improve its anomaly detection over time. However, environments that already have SLO definitions in another system can easily be incorporated in place of Causely’s default settings. 
• Contextual presentation. Results are intuitively presented in the Causely UI, enabling users to see the root causes, related symptoms, the service impacts and initiate remedial actions. The results can also be sent to external systems to alert teams who are responsible for remediating root cause problems, to notify teams whose services are impacted, and to initiate incident response workflows. 
• Postmortem analysis. Teams can also review prior incidents and see clear explanations of why these occurred and what the effect was, simplifying the process of postmortems, enabling actions to be taken to avoid re-occurrences.  

Conclusion 

Escalations don’t need to devolve into chaos, finger pointing, and frayed relationships. They can be opportunities for teams to solve real problems together. The key is having dependable, real-time information on service dependencies and root causes of problems. Armed with the right information, teams can work efficiently and collaboratively to maintain system reliability. 

Book a meeting with the Causely team and let us show you how to transform the state of escalations and cross-organizational collaboration in cloud-native environments. 

Troubleshooting is an unavoidable part of life for SREs and developers alike, and it often feels like an endless grind. The moment a failure occurs, the clock starts ticking. If the failure impacts a mission critical application, every second counts. Outages can cost hours of wasted productivity, to say nothing of lost revenue and pricing concessions when you’ve violated an SLO. Pinpointing the root cause requires sifting through piles of logs, metrics that blur together, and false positives. Troubleshooting becomes a search for a needle in a haystack, and to make things even more complex, the needle may not even be in the haystack. Furthermore, when the failure originates in your scope of control, the pressure intensifies—you’re expected to resolve it quickly, minimize downtime, and restore service without disrupting the rest of your work. It’s a reactive process, and it’s draining.  

But it doesn’t have to be this way. By adopting systems that solve the root cause analysis problem and automate troubleshooting, you can shift troubleshooting from a time-consuming, heavy-lifting chore to a streamlined task. Automated root cause analysis cuts through the noise and pinpoints the issue in no time.   

With the right approach, troubleshooting becomes a quick, manageable part of your day, freeing you to focus on building systems that don’t just react better but fail less often. 

What do we mean by troubleshooting? 

In a distributed microservices environment, troubleshooting often begins with an alert from the monitoring system or user feedback about degraded performance, such as increased latency or error rates. Typically, these issues are first observed in the service exposed to end users, such as an API gateway or frontend service. However, the root cause often lies deeper within the service architecture, making initial diagnosis challenging. The development team must begin by confirming the scope of the issue, correlating the alert with specific user-reported problems to identify whether it is isolated or systemic. 

The next step involves tracing the source of the alert within the service ecosystem. Using distributed tracing tools like OpenTelemetry, the team tracks requests as they propagate through various microservices, identifying where bottlenecks or failures occur. Concurrently, a service dependency map, often visualized through monitoring platforms, provides a bird’s-eye view of interactions between services, databases, caches, and other dependencies, helping to pinpoint potential hotspots in the architecture. 

Example service dependency map. Source: Grafana 

Once the potential hotspots are identified, developers turn to metrics and logs for further insights. Resource utilization metrics, such as CPU, memory, and disk I/O, are analyzed to detect bottlenecks, while logs reveal specific errors or anomalies like timeouts or failed database queries. This analysis attempts to correlate symptoms with the timeline of the issue, offering clues to its origin. Often, the team experiments with quick fixes, such as scaling up CPU, memory, or storage for the affected services or infrastructure. While these adjustments might temporarily relieve symptoms, they rarely address the root cause and must be rolled back if ineffective.  

When resource adjustments fail, a deeper dive into the affected components is necessary. Distributed traces provide detailed insights into slow transactions or failures, highlighting which services or calls are problematic. Developers then use continuous profiling tools to examine runtime data for each service, identifying resource-intensive methods, excessive memory allocations, or inefficient call paths. This granular analysis helps uncover inefficiencies or regressions in code performance. 

If the issue involves a database, further investigation focuses on query performance. Database profiling tools are used to analyze query execution times, frequency, and data volume. Developers assess whether queries are taking longer than usual, retrieving excessive data, or being executed too frequently. This step often reveals issues such as missing indexes, inefficient joins, or unoptimized queries, which could be contributing to overall service degradation. By iteratively analyzing and addressing these factors, the root cause of the problem is eventually resolved, restoring system stability and performance. 

Troubleshooting is reactive, time-consuming, and exhausting.  Developers should be focusing their time and energy (and their company’s investment) on innovation, yet troubleshooting forces them to turn their attention elsewhere. 

Troubleshooting doesn’t have to dominate your role; with the right systems, it can become efficient and manageable. 

Troubleshooting is hard 

When trying to find the root cause of service or application outages or degradations, developers face numerous challenges: 

• It’s hard to pinpoint which service is the source of the degradations amid a flood of alerts 
• It’s hard to diagnose and remediate the root cause 
• It’s hard to see the forest from the trees 

It’s hard to pinpoint which service is the source of the degradations amid a flood of alerts 

Failures propagate and amplify through the environment. A congested database or a congested resource will cause application starvation and service degradation that cascades throughout the system. Even if you deploy an observability tool to monitor the database or the resource, you may observe nothing on the database or the resource. And if you deploy an observability tool to monitor the applications and services, you will be flooded with alerts about application starvation and service degradations. Given the flood of alerts, pinpointing the bottleneck is very complex. As described above, it entails a time-consuming, heavy lifting, manual process under pressure.   

The more observability tools you deploy, the more data you collect and the harder the problem gets. More alerts, more noise, more data you need to sift through. This is a journey to nowhere, a trajectory you want to reverse. 

Diagnosing and remediating the root cause of a problem is hard 

Pinpointing the congested service, database or resource is hard and inefficient. Even if you know where the root cause is, you may not know what the root cause is. Without knowing what the root cause is, you can’t know what to remediate nor how to remediate.  

Whether pinpointing where the bottleneck is or pinpointing what the root cause is, engineers rely on manual workflows to sift through logs, metrics, and traces. While new observability tools have emerged over the past decade focusing on cloud-native application infrastructure, and the traditional old guards have expanded their coverage to monitor the new technology landscape, neither has solved the problem. Some may do a better job than others in correlating anomalies or slicing and dicing the information for you, but they leave it to you to diagnose and pinpoint the root cause, leaving the hardest part unsolved. Furthermore, most of them require time consuming setup and configuration, deep expertise to operate, and deep domain knowledge to realize their benefits. 

In practice, this means engineers are still performing most of the diagnostic work manually. The tools may be powerful, even elegant, but they don’t address the core challenge: diagnosing and remediating root causes remains a slow, resource-intensive process, particularly when time is of the essence during an incident. These gaps prolong resolution times, increase stress, and reduce time for proactive system improvements. 

It’s hard to see the forest from the trees 

Once the engineers turn to dashboards to investigate further, they stare at dashboards created by bottom-up tools. These tools collect a lot of data (often at great cost) and present this data in their dashboards without regard to the purpose of the information and the problem that needs to be solved. Engineers sift through metrics, logs, and time-series data, trying to understand context, composition, and dependencies so they can manually piece together patterns and correlations. This is highly labor-intensive and drives the engineer to get lost in the weeds without understanding the big picture of how the business or the service is impacted.  Are service level objectives (SLOs) being violated? Are SLOs at risk? 

Take your favorite observability tool. It probably excels at visualizing time-series data. However, it requires engineers to manually connect trends across dashboards and services, which can be especially challenging in distributed systems. Similarly, application performance management (APM) tools provide rich metrics and infrastructure insights, but the sheer volume of data presented in their dashboards can overwhelm users, making it difficult to focus on the most relevant information.  

These tools, while powerful, often fall short in helping engineers see the forest from the trees. Instead of guiding engineers toward the right priorities and actionable insights about the broader system or the root cause, or even better, automatically pinpointing the root cause and remediating, they frequently amplify the noise. Irrelevant data, ambiguous relationships, and false positives force engineers to wade through excessive details, wasting time and delaying resolution. The lack of a top-down perspective makes it harder to understand how symptoms connect to underlying problems, leaving engineers stuck in the weeds. 

The negative consequences of troubleshooting today 

The way troubleshooting is done today has serious ramifications for organizations, teams, and individuals. It affects business outcomes and quality of life. 

Failing to meet the SLAs 

Whether the goal is 5-nines, 4-nines, or even only 3-nines, if we continue to manually troubleshoot, we will never meet these SLAs. The table below illustrates how many minutes in a month the given SLA allows for downtime.  

Source: High Availability, Wikipedia

3-nines means 99.9% uptime—in other words, all services are performing reliably at least 99.9% of the time. So, if any of the services is degraded for more than 43.2 minutes in a month, the 3-nines SLA is not met. Because of the length of time manual troubleshooting entails, a single incident in the month will cause us to miss delivering on a 3-nines SLA. And 3-nines is not even so great! 

High Mean Time to Detect and Resolve (MTTD/MTTR) 

The longer it takes to detect and resolve an issue, the greater the impact on customers and the business. Traditional troubleshooting workflows, which often rely on reactive and manual processes, are inherently slow. Engineers are forced to navigate through an overwhelming volume of alerts, sift through logs, and correlate metrics without clear guidance. This delay can lead to: 

• Prolonged outages that damage user trust and satisfaction. 
• Breaches of service level objectives (SLOs), which can result in financial penalties for organizations with stringent service level agreements (SLAs) 
• Snowballing effects, where unresolved issues trigger secondary failures, compounding the problem and making resolution even more challenging. 

Individual stress and burnout from constant reactive tasks 

The reactive nature of troubleshooting takes a significant toll on individual engineers. When every incident feels like a race against the clock, the pressure to resolve issues quickly can become overwhelming. Engineers often work under constant stress, juggling: 

• Interruptions to their regular work, leading to disrupted schedules and decreased productivity. 
• Escalations where they are expected to step in as subject matter experts, often during nights or weekends. 
• Repeated exposure to alert noise, which can cause decision fatigue and desensitization to critical alerts. 

This relentless pace contributes to burnout. All it takes is a few hours of perusing the /r/sre subreddit to see that burnout is a very common issue among SREs and developers tasked with maintaining system reliability. Burnout not only affects individuals but also leads to higher attrition rates, disrupting team continuity and increasing hiring and training costs. 

Reduced time for proactive reliability engineering 

Troubleshooting dominates the time and energy of engineering teams, leaving little room for proactive reliability initiatives. As we will see later this week, proactive reliability engineering has extraordinary promise for the entire company: product/engineering, operations, business leaders. But instead of focusing on preventing incidents, engineers are stuck in a reactive loop. This trade-off results in: 

• Delayed implementation of improvements that could enhance system stability and scalability. 
• Accumulation of technical debt. 
• A vicious cycle where the lack of proactive work increases the likelihood of future incidents, perpetuating the troubleshooting burden. 

By constantly reacting to problems rather than proactively addressing underlying issues, teams lose the ability to innovate and build resilient systems. This dynamic not only affects engineering morale but also has broader implications for an organization’s ability to compete and adapt in fast-paced markets. 

How troubleshooting should look 

If we all recognize that the state of the art of troubleshooting is awful today, let’s work together to imagine a future where troubleshooting is routine and fast: 

• Systems automatically pinpoint root causes within your domain quickly and accurately. Modern troubleshooting workflows must prioritize speed and precision. Systems should go beyond flagging symptoms and directly pinpoint the underlying cause within your domain. 
• Actionable information provides necessary context upfront. Systems need to focus on identifying the actions and ideally automating the automatable. 
• Troubleshooting workflows are streamlined. Workflows should be intuitive and efficient, designed to minimize context switching and maximize focus with unified dashboards that integrate with your operational workflows. 

 These systems must have certain capabilities to be effective: 

• Causality. The ability to capture, represent, understand and analyze cause and effect relations. 
• Reasoning. Generic analytics that can reason about causality and automatically pinpoint root causes based on observed symptoms. 
• Automatic topology discovery. The ability to automatically discover the environment, the entities and the relationships between them. 

With these systems, proper troubleshooting can drive positive business outcomes, such as: 

• Delivering on SLOs and meeting SLAs. Reduce the number of incidents. 
• Faster issue resolution, minimizing downtime. Reduce mean time to detect (MTTD) and mean time to resolve or recover (MTTR), keeping systems operational and minimizing the impact on users. 
• Improved productivity by reducing time spent on reactive tasks. Enable engineers to focus on high-value innovation. 

Causely automates troubleshooting 

Our Causal Reasoning Platform is a model-driven, purpose-built AI system delivering multiple analytics built on a common data model. It is designed to make troubleshooting much simpler and more effective by providing:  

• Out-of-the-box Causal Models. Causely is delivered with built-in causality knowledge capturing the common root causes that can occur in cloud-native environments. This causality knowledge enables Causely to automatically pinpoint root causes out-of-the-box as soon as it is deployed in an environment. There are at least a few important details to share about this causality knowledge:  
  • It captures potential root causes in a broad range of entities including applications, databases, caches, messaging, load balancers, DNS compute, storage, and more. 
  • It describes how the root causes will propagate across the entire environment and what symptoms may be observed when each of the root causes occurs.  
  • It is completely independent from any specific environment and is applicable to any cloud-native application environment.  
• Automatic topology discovery. Cloud-native environments are a tangled web of applications and services layered over complex and dynamic infrastructure. Causely automatically discovers all the entities in the environment including the applications, services, databases, caches, messaging, load balancers, compute, storage, etc., as well as how they all relate to each other. For each discovered entity, Causely automatically discovers its:  
  • Connectivity - the entities it is connected to and the entities it is communicating with horizontally  
  • Layering - the entities it is vertically layered over or underlying 
  • Composition - what the entity itself is composed of 

Causely automatically stitches all of these relationships together to generate a Topology Graph, which is a clear dependency map of the entire environment. This Topology Graph updates continuously in real time, accurately representing the current state of the environment at all times. 

• Root cause analysis. Using the out-of-the-box Causal Models and the Topology Graph as described above, Causely automatically generates a causal mapping between all the possible root causes and the symptoms each of them may cause, along with the probability that each symptom would be observed when the root cause occurs. Causely uses this causal mapping to automatically pinpoint root causes based on observed symptoms in real time. No configuration is required for Causely to immediately pinpoint a broad set of root causes (100+), ranging from applications malfunctioning to services congestion to infrastructure bottlenecks.  

In any given environment, there can be tens of thousands of different root causes that may cause hundreds of thousands of symptoms. Causely prevents SLO violations by detangling this mess, pinpointing the root cause that’s putting your SLOs at risk, and driving remediation actions before SLOs are violated. For example, Causely proactively pinpoints if a software update changes performance behaviors for dependent services before those services are impacted. 

• Service impact analysis. Causely automatically analyzes the impact of the root causes on SLOs, prioritizing the root causes based on the violated SLOs and the ones that are at risk. Causely automatically defines standard SLOs (based on latency and error rate) and uses machine learning to improve its anomaly detection over time. However, environments that already have SLO definitions in another system can easily be incorporated in place of Causely’s default settings. 
• Contextual presentation. The results are intuitively presented in the Causely UI, enabling users to see the root causes, related symptoms, the service impacts and initiate remedial actions. The results can also be sent to external systems to alert teams who are responsible for remediating root cause problems, to notify teams whose services are impacted, and to initiate incident response workflows. 
• Prevention analysis. Teams can also ask "what if'' questions to understand the impact that potential problems might have if they were to occur to support the planning of service/architecture changes, maintenance activities and improving the resilience of services.  
• Postmortem analysis. Teams can also review prior incidents and see clear explanations of why these occurred and what the effect was, simplifying the process of postmortems, enabling actions to be taken to avoid re-occurrences.  

Conclusion 

Troubleshooting doesn’t have to dominate a developer’s or SRE’s nightmare when the right systems are in place. Empower yourself with the only system that solves the root cause analysis problem to make troubleshooting a small, manageable part of your job. 

Book a meeting with the Causely team and let us show you how to stop troubleshooting and consistently meet your reliability expectations in cloud-native environments. 

Best Use of AI

Winner: Causely

Many observability systems now claim to support Root Cause Analysis. At the same time though, most of these systems use algorithms – admittedly, advanced algorithms, which are based, fundamentally, on correlation rather than causation. For us Causely stands out as a system which truly embeds Causal AI in its reasoning and can therefore genuinely go beyond correlation and make more intelligent analysis of system behaviour.

Their full recap is available on the OpenTelemetry blog, and you can watch the full recording below.

What is CPU Throttling?

CPU throttling in containers occurs due to resource constraints set by control groups (cgroups). Kubernetes and other container orchestrators rely on cgroups to enforce resource limits. When a container attempts to use more CPU than its assigned quota, it gets throttled, delaying execution of tasks. (When containers have CPU limits defined, they will be converted to a cgroup CPU quota.)

Working on the vendor side for over a decade, I have seen the impact CPU throttling can have on different services across many industries.  Here are three top-of-mind examples, both from my days at Turbonomic and from recent conversations with customers at Causely:

Financial Systems

• • Example: A stock trading platform uses containers to handle real-time market data feeds and execute trades. Throttling during peak trading hours delays data processing, potentially causing missed opportunities or incorrect order placements.
  • Impact: Missed deadlines for transaction processing.

Gaming Servers

• • Example: Online multiplayer games hosted in containers experience throttling, leading to delayed responses (lag) during gameplay. Players may experience slow rendering of in-game actions or disconnections during high traffic.
  • Impact: Latency and poor user experience.

Video Streaming Platforms

• Example: A video-on-demand service runs encoding jobs in containers to transcode videos. Throttling increases encoding times, leading to delayed content availability or poor streaming quality for users.
• Impact: Degraded video quality and buffering issues.

How to Identify CPU Throttling

It’s often difficult to catch CPU throttling because it can happen even when the host CPU usage is low. It’s critical to have the right level of monitoring set up in order to see CPU throttling when it happens, or even better, before it becomes a problem.

Monitor Your cgroup Metrics

Linux cgroups provide detailed metrics about CPU usage and throttling. Look for the cpu.stat file within the container’s cgroup directory (usually under /sys/fs/cgroup):
Within the cpu.stat file there are three key metrics:

• nr_throttled: Number of times the container was throttled.
• throttled_time: Total time spent throttled – which I believe is in nanoseconds
• nr_periods: Total CPU allocation periods.

Example:
cat /sys/fs/cgroup/cpu/cpu.stat

Output:
nr_periods 12345
nr_throttled 543
throttled_time 987654321

If nr_throttled or throttled_time is high relative to nr_periods, then you have CPU throttling on your container.

Monitor Container Orchestration Metrics

If you’re running Kubernetes, you can use the kubectl top pod command to get metric data on the highest utilized pods.  Try the command below to get metrics for a pod and all the associated containers:
kubectl top pod --containers
This is a very manual process, and you will need to compare the CPU usage against the defined limits in the pod’s resource config.  If you run a describe command on the pod it will show you this information.  This also means you will need to know which pod is having an issue on it.  Usually when issues arise on an application it’s going to take some time to drill down to a component that might be performing poorly.  You will need the Kubernetes metric server to be installed in order to run commands like kubectl top but being able to access metrics like “container_cpu_cfs_throttled_periods_total” and “container_cpu_cfs_periods_total” offer valuable insights into CPU usage and throttling.

Application Performance Metrics

Although they can be expensive, application performance monitoring (APM) tools provide invaluable insights into CPU throttling, offering detailed visibility that can help uncover the issue. These tools can often track throttling over time, identify exactly when it first occurred, and, in some cases, even predict future throttling trends based on usage patterns. Many organizations use a combination of monitoring tools to get a comprehensive view of their systems. APM tools also highlight the symptoms of CPU throttling, which may manifest as:

• Prolonged request durations, leading to slower application response times.
• Decreased throughput, resulting in fewer transactions or tasks processed within a given timeframe.
• Irregular CPU usage patterns, which can signal performance instability or inefficiencies.

By combining the capabilities of APM tools with metrics collected from Kubernetes, teams can proactively address CPU throttling and ensure optimal application performance.

Best Practices to Manage CPU Throttling in Kubernetes

There are many ways to fix CPU throttling and even a few ways you can prevent it. Most root causes of CPU throttling are overcommitted nodes, or misconfigured CPU limits.  Below are some ways to fix CPU throttling when it occurs and some best practices to avoid it in the future.

Adjust CPU Limits

Update resource limits in your container or pod configuration. Kubernetes resource specs can be updated like in the example below.  Usually what I see from the customers I have worked with is they will set the limit just above the peak usage in the last 30,60, or even 90 days.  For non-critical workloads I have seen a few companies set this limit to 80% of max usage, and a few companies use more advanced techniques like calculating percentiles:

resources:
requests:
cpu: "500m"
limits:
cpu: "1000m"

• Increase the limits.cpu value to reduce throttling frequency.
• Set requests.cpu to ensure better performance during contention.  Note that if you do not set the request, Kubernetes will automatically set the request to the limit.

Use Autoscaling like Horizontal Pod Autoscaler

The Horizontal Pod Autoscaler (HPA) in Kubernetes helps address CPU throttling by dynamically adjusting the number of pods in a deployment based on real-time resource usage.  Resources like CPU and memory are monitored, and when certain thresholds are met, HPA kicks in to provision more pods.  In more idle periods it will also scale down the number of pods to help you run more efficiently.  By distributing the workload across more pods, HPA reduces the CPU demands on individual pods, thereby mitigating CPU throttling.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
targetAverageUtilization: 80

In this example, if average CPU utilization across all pods exceeds 80% then it will add more pods as necessary within the bounds of 2-10 (Min and Max Replicas).

Analyze Node Resource Allocation

Check overall CPU availability on nodes using a describe command:
kubectl describe node <node-name>
Ensure nodes aren’t overcommitted. Use taints and tolerations to control scheduling and ensure high-priority workloads run on dedicated nodes.  Nodes that are overcommitted run the risk of not having available CPUs.  If containers’ requests are higher than the node CPU availability, you are going to run into scheduling problems.  Or if the limits are set too high relative to CPU availability on the node and the workload suddenly increases, then you are going to have contention.

Tweak CPU CFS Settings

Containers use the Completely Fair Scheduler (CFS) by default. The CFS in Kubernetes is a mechanism inherited from the Linux kernel that enforces CPU usage limits on containers. It works by using two key parameters from Linux cgroups: cpu.cfs_quota_us and cpu.cfs_period_us. These parameters allow Kubernetes to control the amount of CPU time a container can use over a specific period:

• cpu.cfs_quota_us: Maximum microseconds of CPU time allowed per period.
• cpu.cfs_period_us: Length of a scheduling period in microseconds.

To prevent CPU throttling: Increase cpu.cfs_quota_us to provide more CPU time:
echo 200000 > /sys/fs/cgroup/cpu/cpu.cfs_quota_us
I have seen this create issues before though so be careful with this adjustment as it can lead to overcommitment.  In other words, if too many tasks are scheduled and you increase the amount of time a container can use the CPU, then it will create delays and throttling.  Start by playing around with this in Dev or Test clusters before you make any changes to prod… duh.

Use CPU Pinning

This is more of an edge case, but instead of using CPU shares and limits, pin containers to specific CPUs for predictable performance.  The Kubernetes CPU Manager controls how CPUs are allocated to containers.  The enable CPU pinning, the static CPU manager policy must be used, which provides exclusive CPU allocation.  Just enable the policy in the Kubelet configuration file:
cpuManagerPolicy: static
With the flag set to “static,” containers are allocated exclusive CPUs on the nodes in a cluster.  Kubernetes assigns the container specific CPUs and it runs only on those cores.  The big challenges with CPU pinning are overhead and scalability.  When you manage pinned workloads it requires detailed planning to avoid fragmentation and underutilization.  CPU pinning is good for workloads that are sensitive to CPU throttling but not ideal for volatile and dynamic workloads.

CPU Throttling is a Double-Edged Sword in Kubernetes

While CPU throttling plays a crucial role in resource management and stability, it can also hinder application performance if not managed correctly. By understanding how CPU throttling works and implementing best practices, you can optimize your Kubernetes environment, ensuring efficient resource use and enhanced application performance. As Kubernetes continues to grow and evolve, keeping a close eye on resource management will be key to maintaining robust and responsive applications.

In this post, we’ll explain what it means to be a Service Owner, outline key responsibilities associated with the role, and offer advice for companies looking to build a culture of service ownership.

How the Service Owner came to be

When the DevOps movement took off around 2010, it promised to fix the issues with fragmented teams and inefficient software lifecycle management that had been hindering application performance and reliability for years.  This new wave of IT fostered cross-team collaboration, communication, and transparency as a way to accelerate software delivery of software and deliver higher quality of service (QoS).

But there was something still missing: accountability across the end- to-end application lifecycle, from development to roadmap design to customer satisfaction. Hence the emergence of the Service Owner.

So, what is a Service Owner?

Many IT teams today, especially those using microservice architectures, employ multiple Service Owners.  The exact definition varies slightly from company to company, but most would define a Service Owner (SO) as the person who is responsible for making sure the application or service meets its designated service level objectives (SLOs). Implicit in this definition: Service Owners are accountable for the end-to-end lifecycle of applications, from development through operational performance production monitoring, to ensure uptime and customer satisfaction.

For a little more context, here’s how ITIL defines the role:

• The Service Owner is responsible for delivering a particular service within the agreed service levels.
• Typically, the Service Owner acts as the counterpart of the Service Level Manager when negotiating Operational Level Agreements (OLAs).
• Often, this role will lead a team of technical specialists or an internal support unit.

Usually, Service Owners are aligned to a specific product line for the business. Their formal title may be Product Owner or Engineering Manager. For example, imagine a healthcare tech company that sells solutions to businesses and hospitals.  They have 5 different products within their offering suite:

• AI platform
• Online therapy product
• Analytics product
• Telehealth service
• Financial management product

A Service Owner would be assigned to each of these product lines and assume full responsibility for its application lifecycle management and reliability engineering.  A Service Owner role can also be broken down by customer, by services within the products, or even by mobile vs desktop applications.

Responsibilities of the Service Owner

Service Owners’ responsibilities tend to vary slightly from company to company but they often include:

1. Design and Roadmap: Writing the code and overseeing the design and implementation of the service, ensuring it functions properly and is scalable. This includes managing the ongoing balance and prioritization of releasing new features vs. improving the reliability of existing functionality.
2. Maintenance and Support: Ensuring that the service is properly maintained, updated, and supported over its lifecycle.  This is where creating and adhering to SLOs comes into play.
3. Performance Monitoring: Monitoring the performance and reliability of the service, by implementing metrics and logging to track its health and flag when things break.  Performance Monitoring also includes implementing proactive monitoring to prevent downtime.
4. Collaboration: Working closely with other teams, such as Product Management, Sales, Platform Engineering, etc. to align the service with the business goals.
5. Documentation: Creating and maintaining comprehensive documentation for the service, including things like APIs, user guides, and architecture diagrams.
6. Governance and Compliance: Ensuring that the service adheres to relevant policies, standards, and regulatory requirements.
7. Stakeholder Communication: Acting as a point of contact for stakeholders, addressing their needs and concerns regarding the service.

Service Owners work closely with technical experts such as SREs, DevOps, or Developers to maintain service reliability, though they each have distinct roles and tool preferences:

• Service Owners typically use collaboration and project management tools like Jira or Asana and monitor high-level metrics on observability dashboards like Grafana.
• Technical Experts handle incident resolution and service reliability, relying primarily on observability and incident management tools like PagerDuty.

Tools like ServiceNow, Atlassian’s Service Management (OpsGenie, Jira), and PagerDuty’s workflow orchestration attempt to bridge the gap between these two roles by providing a unified space for planning, alerting, diagnosis, and response. This enables Service Owners and technical experts to operate more effectively together, allowing engineering teams to enhance alignment, transparency, and accountability.

Code it, ship it, own it

A service owner’s job goes beyond writing and compiling code and bug fixes.  They are responsible for their applications and services after they’ve been shipped to production.  When Service Owners own the lifecycle, organizations see improved QoS and faster MTTR.

If they wrote the code, they know how to fix it.  If something breaks, they are the first responders and take accountability for failures.  They have a deeper knowledge of issues within their service and application so they can fix and develop the fastest, but they must be held accountable for downtime.  No more finger pointing!

One thing service owners MUST do is align themselves with their customers.  They must understand what the customers’ needs and expectations are and then design the code, roadmap, and establish SLOs accordingly.

The benefits of this approach? Products directly solve customer pain, and in most cases, services are delivered faster to customers.  It reminds me of a funny meme I saw years ago — it couldn’t be more accurate.  This situation is exactly what Service Owners are preventing!

Source: Reddit

Building a culture of service ownership

Since service ownership is a culture and not a tool, it needs to grow over time; it can’t happen overnight, no matter how much pressure the business puts on IT.  There are ways to actively foster a shift in mentality, so teams thrive in an environment where they have more responsibility and where better services are being delivered to the customer.

Based on my conversations with leaders in IT, these are some common best practices for building a culture of service ownership.

1. Promote collaboration: Encourage open communication between teams—development, operations, and business units. Regular cross-functional meetings and collaborative tools can help break down silos faster and foster a shared understanding of service objectives.
2. Establish a customer-first mentality: As mentioned before, service ownership will not thrive if everyone has different ideas and goals.  Establishing a common goal like focusing on customers’ needs can align teams. If customer satisfaction is the north star, companies will have more satisfied customers, which means bigger checks 😉.  Defining customer specific SLOs is an excellent way to keep everyone aligned on the mission.  SLOs on latency, number of customer tickets/escalations, and even uptime are some standard ones I see most Service Owners using.
3. Embrace failure: Taking accountability and responsibility for something that directly impacts a business can be scary. That’s probably the single biggest reason why most software engineers are hesitant to adopt a service ownership role.  If leadership fosters a culture where failure is seen as progress and not regression, then it becomes more appetizing to developers.  No one wants to lose their job over a silly mistake, but they need to learn from these slips and drive towards a more reliable application architecture.

Building a culture of service ownership in IT requires a deliberate and consistent approach. By defining roles, fostering collaboration, and empowering teams, IT can create an environment where service ownership is continuously improved. This culture not only enhances QoS but also drives innovation and responsiveness, ultimately benefiting the health of any business.

FAQs

• What is application lifecycle management?
  Application lifecycle management is the end-to-end process of developing, building, deploying, and managing software applications over time to ensure consistent and ongoing quality, reliability, and resilience.
• What is reliability engineering?
  Reliability engineering is the practice of ensuring that applications, products, or systems function without failure. Reliability engineers focus on proactively identifying potential failures to determine their root cause and mitigation strategies before they happen.
• What is a Service Owner?
  A Service Owner is someone who is responsible for meeting agreed service levels. They ultimately usually own the overall engineering, management and governance of the lifecycle of a service.

And if you pay attention, you might start to notice some buzz building around one event in particular. Word on the KubeStreet is this will be one of those IYKYK type things, so I’m here to help you get in the know (cue Jaws music 🦈 🎶).

First, we need to set a bit of a foundation about the evolution of DevOps and what’s missing from the Observability market.

The beginning of DevOps

Patrick Debois, a tech consultant based in Belgium, is often cited with popularizing the term DevOps. This is because, in 2009, he organized the first DevOpsDays conference. However, the truth is the roots of DevOps extend further back.

Earlier that same year, John Allspaw and Paul Hammond’s influential talk at the Velocity Conference, 10+ Deploys per Day: Dev and Ops Cooperation at Flickr highlighted the importance of collaboration between development and operations to enable frequent, reliable deployments.

We can go back even further though, as the groundwork for DevOps was already being laid in 2001. Back then, The Agile Manifesto championed collaboration and iterative development, which are principles that would later shape the DevOps ethos. Meanwhile, tools like Puppet, Chef, and Jenkins were gaining traction in the oughts, helping bridge the gap between development and operations teams.

These influences helped shape what is commonly known as DevOps, which is a movement that champions collaboration, cultural transformation, and tooling integration to achieve high-frequency, reliable software delivery.

Then came Observability

As part of this movement, engineering teams started to adopt microservices architecture as a software development method of breaking applications down into small, independent services that interact with each other to perform the job of the application. This method became popular because of the speed and ease that it adds to the software development process.

On the other hand, a microservices approach makes it exponentially more difficult to diagnose, remediate, and prevent application performance problems. The growing popularity of DevOps practices and microservices architecture paved the way for advancements in monitoring and its evolution into what many know today as observability.

As DevOps led teams to release faster and decompose applications into microservices, traditional monitoring – which relies on static alerts based on isolated metrics – could no longer keep up with the resulting complexity. Observability tries to address this gap by offering a more holistic view to help engineers pinpoint issues, understand dependencies, and continuously improve system performance.

Companies large and small appreciate how observability tools provide their engineering teams with a single pane of glass, nifty dashboards, and helpful data correlation features, but the more discerning engineers have matured to view this market of tools as being more like a single glass of pain.

Cool, so what does this have to do with sharks?

Suppose I told you that ice cream consumption and shark attacks were correlated. From this you might infer that sharks love sugar. Of course, you’ve probably heard this example used before to help make the point that correlation does not equal causation. It’s certainly useful to understand that two metrics are temporally correlated, but it’s misleading that observability vendors claim this to be root cause analysis, and in some scenarios it can hurt more than it helps.

At Causely, we value the observability market and believe it’s provided a good start for engineers who want to reduce MTTD and MTTR. The more discerning engineers are coming to realize they will only reach the next frontier of autonomous service reliability if they start to look at the problem from a different, top-down perspective.

We’ll be happy to tell you more about this if you’re lucky enough to find the shark at the show next week. Here’s a hint: she’ll be holding something sweet. 🍨 🥞

Additional information about our presence at KubeCon is here.

Cloud-native and open source technologies are foundational to everything we’re building at Causely, and I’m looking forward to diving into the latest developments with observability tools that will shape the reliability of countless modern applications. We’re excited about how observability tools, such as Prometheus, OpenTelemetry, Grafana Beyla, and Odigos, are improving how systems are monitored and understood. These tools will underpin the reliability engineering strategy of countless modern application environments.

In this post, I’ll unpack some of the exciting innovations happening in the open source observability space, and highlight specific talks at KubeCon 2024 I’m looking forward to attending. Add them to your schedule if this is an area you’re following too, and I’ll see you there!

What’s New in the Observability Landscape

The observability space is evolving rapidly. Several key developments with OpenTelemetry and eBPF are reshaping how we approach monitoring and tracing in distributed systems. This helps make metric collection and auto-instrumentation easier and more flexible than ever.

Prometheus and OpenTelemetry are even better together

The collaboration between Prometheus and OpenTelemetry is delivering a more cohesive experience for capturing system metrics. Here’s what’s new:

• Using the OTel Collector’s Prometheus Receiver: OpenTelemetry now allows for streamlined ingestion of Prometheus metrics, creating a more unified data pipeline.
• Exploring OTel-Native Metric Collection Options: For those deeply involved in Kubernetes monitoring, tools like Kubernetes Cluster Receiver and the Kubeletstats Receiver provide robust options for flexible metric collection.

These integrations create a powerful foundation for system insights that can fuel everything from troubleshooting to strategic infrastructure optimizations.

Odigos makes migration to OpenTelemetry easy

Transitioning from proprietary observability tools can be challenging.  If you’re looking to transition from proprietary observability tools to OpenTelemetry, Odigos is a great enabler. Here’s why:

• Simplified Migration: Odigos takes the complexity out of moving to OpenTelemetry, helping reduce the friction and downtime that can accompany large-scale tooling changes.
• Access to Enhanced Capabilities: With OpenTelemetry, you gain access to a broader ecosystem, opening doors to new integrations and data visualizations that enrich your observability setup.

Streamline eBPF-based auto-instrumentation

Combining Grafana Beyla with Grafana Alloy makes eBPF-powered observability easy and minimally intrusive. Whether you’re working with standalone systems or Kubernetes clusters, this integration provides high-precision monitoring capabilities without heavy overhead.

KubeCon 2024 Deep Dives and Hands-On Sessions

The schedule at KubeCon North America 2024 will feature several sessions dedicated to observability. Here are some highlights I’m looking forward to:

1. OpenTelemetry: The OpenTelemetry Hero’s Journey – Working with Open Source Observability
   Takeaway: Dive into the current capabilities of OpenTelemetry, including correlated metrics, traces, and logs for a complete observability picture. The talk also addresses gaps in today’s open source observability tools.
2. Inspektor Gadget: eBPF for Observability, Made Easy and Approachable
   Takeaway: This project lightning talk will explain how Inspektor Gadget simplifies eBPF distribution and deployment, allowing you to build fast, efficient data collection pipelines that plug into popular observability tools.
3. Optimizing LLM Performance in Kubernetes with OpenTelemetry
   Takeaway: Speakers Ashok Chandrasekar (Google) and Liudmila Molkova (Microsoft) will help you gain practical insights into observing and optimizing Large Language Model (LLM) deployments on Kubernetes. This session covers everything from client and server tracing with OpenTelemetry to advanced autoscaling strategies.
4. Unifying Observability: Correlating Metrics, Traces, and Logs with Exemplars and OpenTelemetry
   Takeaway: Speakers Kruthika Prasanna Simha and Charlie Le (Apple) will help attendees learn how to correlate data across metrics, traces, and logs, with exemplars. This session demonstrates practical visualization techniques in Grafana, making it easy to move from high-level metrics down to detailed traces.
5. Now You See Me: Tame MTTR with Real-Time Anomaly Detection
   Takeaway: Speakers Kruthika Prasanna Simha and Raj Bhensadadia (Apple) will dive into the latest in real-time anomaly detection, with insights into applying machine learning to time series data in cloud-native environments.

Find Causely at KubeCon 2024!

If the above topics are also interesting to you, we’d love to meet up. At Causely, we’re passionate about helping organizations unlock the full potential of their observability data to continuously assure service reliability. We believe that flying with overwhelming volumes of observability data is just as bad as flying blind.

Here’s where you can find us at KubeCon:

• 🥂 Come to our happy hour! We’re co-hosting a happy hour with friends from NVIDIA, Alma Security, Edera, and 645 Ventures on November 12th.
• 🦈 Find the shark! Look for a shark at the show on the afternoon of November 13th. Here’s a hint: she’ll be holding something sweet.

Takeaways from eBPF Summit 2024

How are organizations applying eBPF to solve real problems in observability, security, profiling, and networking? It’s a question I’ve found myself asking as I work in and around the observability space – and I was pleasantly surprised when Isovalent’s recent eBPF Summit provided some answers.

For those new to eBPF, it’s an open source technology that empowers observability practices. Many organizations and vendors have adopted it as a data source (including Causely, where we use it to enhance our instrumentation for Kubernetes).

Many of the eBPF sessions highlighted real challenges companies faced and how they used eBPF to overcome them. In the spirit of helping others, my cliff notes and key takeaways from eBPF Summit are below.

Organizations like Netflix and Datadog are using eBPF in new, creative ways

The use of eBPF in Netflix

One of the Keynote presentations was delivered by Shweta Saraf who described specific problems Netflix overcame using eBPF, such as noisy neighbors. This is a common problem faced by many companies with cloud-native environments.

Shweta Saraf described Netflix’s use cases for eBPF

Netflix uses eBPF to measure how long processes spend in the CPU scheduled state.  When processes are taking too long, it usually indicates a performance bottleneck on CPU resources — like CPU throttling or over-allocation.  (Netflix’s compute and performance team released a blog recently with much more detail on the subject.)  In solving the noisy neighbor problem, the Netflix team also created a tool called bpftop which is designed to measure the CPU usage of the eBPF code they instrumented.

The Netflix team released bpftop for the community to use, and it will ultimately help organizations implement efficient eBPF programs.  This is especially useful if an eBPF program is hung, allowing teams to quickly identify any overhead that an eBPF program has.  We have come full circle: monitoring our monitoring programs 😁.

The use of eBPF in Datadog

Another use case for eBPF – and one that can be easily overlooked – is in chaos engineering.  Scott Gerring, a technical advocate at Datadog, shared his experience on the matter.  This quote resonated with me: “with eBPF… we have this universal language of destruction” – controlled destruction that is.

Scott Gerring discussed eBPF’s use in Datadog

The benefit of eBPF is that we can inject failures into cloud-native systems without having to re-write the code of an application.  Interestingly, there are open source projects out there for chaos engineering that already use eBPF, such as ChaosMesh.

Scott listed a few examples like Kernel Probes that are attached to the openat system that will cause access denied errors for 50% of calls made by system processes that a user can select or define.  Or, using the traffic control subsystem to drop packets for sockets on process you want to mark for failure.

eBPF will underpin AI development

Isovalent Co-founder and CTO Thomas Graf presented the eBPF roadmap and what he is most excited about.  Notably: eBPF will deliver value in enabling the GPU and DPU infrastructure wave fueled by AI.  AI is undoubtably one of the hottest topics in tech right now.  Many companies are using GPUs and DPUs to accelerate AI and ML (Machine Learning) tasks, because CPUs cannot deliver the processing power demanded by today’s AI models.

Thomas Graf talked about the value of eBPF in enabling GPU and DPU infrastructures

As Tom mentioned, whether the AI wave produces anything meaningful is up for debate, but companies will undoubtedly try, and they will make significant investments in GPUs and DPUs along the way.  The capabilities of eBPF will be applied to this new wave of infrastructure in the same manner they did for CPUs.

GPUs and DPUs are expensive, so companies do not want to waste processing power on programs that will drive up utilization. The efficiency of eBPF programs can help maximize the performance of costly GPUs. For example, eBPF can be used for GPU profiling by hooking into GPU events such as memory, sync, and kernel launches.  Unlocking this type of data can be used to understand which kernels are used most frequently, improving efficiencies of AI development.

eBPF support for Windows is growing

Another interesting milestone in eBPF’s journey is the support for Windows.  In fact, there is a growing Git Repository for eBPF programs on Windows that exists today: https://github.com/microsoft/ebpf-for-windows

The project supports Windows 10 or later and Windows Server 2019 or later, and while there is not feature parity yet to Linux, there is a lot of development in this space.  The community is hard at work porting over the same tooling for eBPF on Linux, but it is a challenging endeavor as the hook points for Linux eBPF components (like Just-In-Time compilation or eBPF bytecode signatures) will differ on Windows.

It will be exciting to watch the same networking, security, and observability eBPF capabilities on Linux become available for Windows.

The need for better observability is fueling eBPF ecosystem growth

eBPF tools have been created by the community for both applications and infrastructure use cases.  There a 9 major projects for applications and over 30 exciting emerging projects for applications.  Notably, while there are a few production-ready runtimes and tools within the infrastructure ecosystem (like Linux and LLVM Compiler), there are many emerging projects such as eBPF for Windows.

With a user base across Meta, Apple, Capital One, LinkedIn, and Walmart (just to name a few), we can expect the number of eBPF projects to grow considerably in the coming years.  The overall number of projects is actually forecasted in the triple digits by the end of 2025.

One of the top catalysts for growth? The urgent need for better observability.  Of all the topics at last year’s KubeCon in Chicago, observability ranked the highest, beating competing topics like cost and automation.  As with any other tool, eBPF can allow organizations gather a lot of data, but the “why” is important. Are you using that data to create more noise and more alerts, or can you apply it to get to the root cause of problems that surface, or for other applications?

It is exciting to watch the eBPF community develop and implement creative new ways to use eBPF and the 2024 eBPF summit was (and still is) an excellent source of real-world eBPF use cases and community-generated tooling.