Autonomous SRE Agent for Kubernetes: Safe Rollbacks

An autonomous SRE agent should not have unrestricted access to your Kubernetes cluster. Its job is narrower: detect an incident, collect evidence, propose a reversible change, obtain approval when required, execute within strict limits, and prove whether the cluster recovered. The hard part is not generating a kubectl command. It is controlling when that command can run, preserving the previous state, and failing closed when the diagnosis is uncertain.

Define bounded autonomy and the Kubernetes operating contract

Start with an operating contract. Treat it as executable policy, not a paragraph in a runbook.

The agent should move through explicit states:

OBSERVE
  -> DIAGNOSE
  -> PROPOSE
  -> AWAIT_APPROVAL
  -> EXECUTE
  -> VERIFY
  -> COMPLETE | ROLLBACK | ESCALATE

Each transition needs a machine-checkable condition. For example, PROPOSE -> EXECUTE is valid only when the action is allowed by policy, the approval is current, and a pre-change snapshot exists.

A practical risk matrix looks like this:

Risk	Example actions	Default behavior
Read-only	Get pods, inspect events, query metrics, read rollout status	Execute automatically
Reversible	Pause a rollout, restart an allowed workload, revert a Deployment image	Require policy validation and usually approval
Sensitive	Change RBAC, read Secrets, delete persistent data, modify cluster-level resources	Deny or escalate to a human

This follows the conservative pattern documented by ScaleOps: its SRE agent is read-only by default and requires explicit human approval before changing cluster state.

Define limits across several dimensions:

• Namespace: operate only in an allowlist.
• Resource: permit Deployments while denying Secrets and cluster-scoped objects.
• Verb: separate get, list, and watch from patch or update.
• Time: expire proposed changes and approvals.
• Blast radius: reject plans that affect unrelated workloads.
• Ownership: require a workload owner or escalation target.
• Evidence: do not remediate without observable incident signals.
• Reversibility: require an explicit reversal operation before execution.

Represent those rules as data:

policy:
  namespaces:
    allow:
      - application-production
  resources:
    deployments:
      read:
        - get
        - list
        - watch
      write:
        - patch
    secrets:
      read: []
      write: []
  require_approval:
    - patch
  deny_cluster_scoped_changes: true
  require_pre_change_snapshot: true
  require_rollback_operation: true

The model may recommend an action. It must not decide whether the action is authorized. Put that decision in deterministic code.

Design the multi-agent architecture around cluster signals and topology

Do not send every alert, metric, log line, and Kubernetes object to one general-purpose agent. Split the workflow by responsibility.

The open-source SRE-agent project describes an architecture with:

• A hybrid triage agent.
• A topology-aware planner.
• Parallel root-cause-analysis workers.
• A supervisor that synthesizes the findings.

Its triage stage combines deterministic heuristics for latency, errors, and saturation with LLM reasoning. This is a useful boundary. Code detects known conditions. The model interprets incomplete evidence and generates investigation plans.

A production workflow can use these components:

Scheduled health check or alert
        |
        v
Deterministic triage
        |
        v
Topology-aware planner
        |
        +----> Metrics worker
        +----> Logs worker
        +----> Traces worker
        +----> Kubernetes events worker
        |
        v
Supervisor
        |
        v
Policy validator -> Approval -> Executor -> Verifier

Keep workflow state outside the model context. Store the incident identifier, observed resources, evidence references, proposed diff, approval status, execution result, and verification result.

LangGraph is a reasonable fit when you need explicit transitions and resumable execution. Our guide to deploying LangGraph as a reliable production job covers checkpoints, recovery, scheduling, and secret handling.

Parallel workers should return structured findings rather than prose:

{
  "resource": "Deployment/api",
  "signal": "rollout_not_healthy",
  "evidence": [
    "kubernetes://events/application-production/api",
    "prometheus://query/api-error-signal"
  ],
  "confidence": "medium",
  "recommended_next_step": "compare_current_and_previous_revision"
}

The supervisor may synthesize those results. It should not silently replace conflicting findings with one confident answer. Contradictions must trigger more observation or human escalation.

Build the observation and diagnosis loop without overwhelming the model

The SRE-agent repository uses a custom Model Context Protocol server to interact with Prometheus, Jaeger, and the Kubernetes API. It preselects relevant metrics and logs before placing them in model context. It also uses cluster topology and dependency data to prioritize investigations.

Apply the same pattern to your tool layer:

def build_incident_context(alert, topology, tools):
    target = alert["resource"]
    related = topology.dependencies_of(target)

    return {
        "target": target,
        "related_resources": related,
        "pod_health": tools.pod_health(target),
        "events": tools.relevant_events(target),
        "rollout": tools.rollout_status(target),
        "hpa": tools.hpa_status(target),
        "node_conditions": tools.node_conditions(target),
        "recent_changes": tools.recent_changes(target),
        "metrics": tools.relevant_metrics(target, related),
    }

This code is intentionally selective. The tool server, not the LLM, decides which raw records are relevant enough to return.

Begin with deterministic checks:

• Is the alert still active?
• Is the affected object inside an allowed namespace?
• Did a rollout or configuration change precede the incident?
• Are unhealthy pods concentrated on one node?
• Is an HPA reporting an abnormal condition?
• Do traces connect the affected workload to an upstream dependency?
• Do current signals violate your own service health criteria?

A LangChain SRE example described on LinkedIn periodically checks pod health, HPA state, and node conditions before delegating investigations to specialized sub-agents. That sequence prevents the model from diagnosing a Deployment while ignoring an unhealthy node or autoscaler condition.

The context passed to the model should contain summaries plus evidence references. Preserve raw telemetry in your observability system. Do not use the prompt as the incident archive.

Generate and execute reversible Kubernetes remediations

Every remediation plan should be executable data. Do not accept “restart the service” as a complete plan.

Require these fields:

incident_id: incident-example
target:
  namespace: application-production
  kind: Deployment
  name: api
diagnosis:
  summary: Current rollout is unhealthy after an image change
  evidence:
    - kubernetes://deployment/application-production/api
action:
  command:
    - kubectl
    - rollout
    - undo
    - deployment/api
    - --namespace
    - application-production
  predicted_impact: Restore the preceding Deployment revision
rollback:
  description: Reapply the captured pre-change manifest
preconditions:
  - approval_valid
  - snapshot_persisted
  - target_resource_version_unchanged
verification:
  - rollout_completed
  - health_gate_recovered

Before execution:

1. Fetch and store the current manifest.
2. Record the resource version and rollout revision.
3. generate the proposed diff.
4. Run a server-side dry run where the operation supports it.
5. Submit the request to admission policies.
6. Confirm that the live resource has not changed since diagnosis.
7. Persist an idempotency key so retries cannot apply the same action twice.

Then execute one narrow operation. Avoid generated shell pipelines. Use an argument array and allowlisted commands.

import subprocess

ALLOWED_PREFIX = [
    "kubectl", "rollout", "undo", "deployment/api",
    "--namespace", "application-production",
]

def execute_approved_plan(command: list[str], approved: bool) -> str:
    if not approved:
        raise PermissionError("Human approval is required")

    if command != ALLOWED_PREFIX:
        raise PermissionError("Command does not match the approved plan")

    result = subprocess.run(
        command,
        check=True,
        capture_output=True,
        text=True,
    )
    return result.stdout

Start with actions that are narrow and reversible: pause an active rollout, revert an image revision, adjust a known probe configuration, or restart a failed workload under a documented runbook.

The broader autonomous-agent workflow guide explains how to separate reasoning, tools, state, and execution in production. That separation matters here because retries must resume from persisted state, not ask the model to reconstruct what already happened.

Put human approval and Kubernetes security between the agent and production

Use a dedicated Kubernetes service account. Give it only the verbs and resources required by the approved remediation catalog.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: application-production
  name: sre-agent
rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "patch"]
  - apiGroups: [""]
    resources: ["pods", "events"]
    verbs: ["get", "list", "watch"]

This example intentionally grants no Secret access, deletion permission, or cluster-wide authority. Use separate credentials for development, staging, and production. Restrict network egress to required telemetry, model, approval, and Kubernetes endpoints. Keep tool credentials out of prompts and logs.

For sensitive changes, send a structured approval request containing:

• Incident and workload identity.
• Diagnosis and evidence links.
• Exact diff or command.
• Expected impact.
• Risk classification.
• Rollback operation.
• Approval expiry.
• Current resource version.

The LangChain implementation described on LinkedIn allows a human to approve, reject, or modify proposed production actions in Slack. It uses an outbound Slack connection rather than exposing an inbound endpoint, and it does not modify production without approval.

Bind the approval to the exact plan hash and resource version. If either changes, invalidate it. For additional isolation patterns, see production AI agent security with least-privilege IAM and secret isolation.

Verify the rollback, handle failure, and measure operational readiness

A successful kubectl exit code proves only that Kubernetes accepted the request. It does not prove that the service recovered.

After execution, record:

• Before-and-after manifests and resource versions.
• Rollout status.
• Kubernetes events.
• Relevant logs and metrics.
• Approval identity.
• Executed command or patch.
• Health-gate result.
• Rollback result, if triggered.

Fail closed when tools are unavailable, diagnoses conflict, approval expires, the resource version changes, a patch applies partially, or health signals regress. In those cases, stop further mutation and escalate with the evidence already collected.

Test the full loop under controlled fault injection. The SRE-agent project uses AIOpsLab to prepare a cluster, inject faults, run the agent, evaluate the result, and clean up. Its documented scenarios include network delays, pod failures, and configuration errors.

The project evaluates detection precision, root-resource localization precision, and RCA quality using an LLM-as-a-Judge scale from 1 to 5. For remediation readiness, also track whether approved changes complete, whether recovery criteria pass, whether rollbacks are verified, and whether unsafe proposals reach the approval queue. Define these metrics against your own incident fixtures before production access.

An autonomous SRE agent is useful when its autonomy is bounded by deterministic policy. Let it collect evidence and coordinate investigation. Require reversible plans, narrow credentials, explicit approval, and post-change verification before trusting it with production. Once that workflow is reliable, you can run the observation and diagnosis stages as isolated scheduled jobs while keeping every cluster mutation behind the same approval and audit boundary.