Persistent State Serverless Jobs: Minimal AI Design

A serverless AI job can disappear after any step. Its memory, local files, and cached responses may disappear with it. If the next run cannot determine what already happened, retries create duplicate API calls, repeated model work, and inconsistent results.

The fix is not to preserve the runtime. Persist a small workflow record outside it. Store progress, stable identifiers, retry metadata, and the result data required for recovery. Everything else can remain transient or move to object storage.

Why persistent state matters when serverless runtimes disappear

Treat every serverless invocation as disposable. An in-memory variable can help within the current process, but it cannot be the source of truth for the next invocation. The same applies to a file written inside the runtime or a singleton held by your Python process.

This changes how you structure an agent.

A local script might keep its progress like this:

current_step = "summarize"
results = []

A production job needs to load that information from a durable system:

state = state_store.load(
    agent_id="competitor-monitor",
    workflow_id="weekly-example-com"
)

The distinction matters for scheduled research, SEO reporting, lead enrichment, and monitoring jobs. If a competitor-monitoring run collects source documents and times out during summarization, the next run should reuse the collected document identifiers. It should not start the workflow from scratch.

A streaming checkpoint illustrates the same recovery principle, but it is not a complete application state model. Databricks continuous mode restarts a task after it finishes, while the streaming checkpoint prevents data from being processed again (Databricks documentation). Your agent still needs its own record for workflow status, external API job IDs, results, and errors.

This is a core constraint of deploying AI agents on serverless runtimes: compute can be temporary, but workflow state cannot be.

Define the smallest useful state model for a multi-run agent

Do not serialize the entire agent object. Persist only what another run needs to make the next safe decision.

A compact relational schema can look like this:

CREATE TABLE agent_workflows (
    agent_id          TEXT NOT NULL,
    workflow_id       TEXT NOT NULL,
    status            TEXT NOT NULL,
    current_step      TEXT NOT NULL,
    state_version     BIGINT NOT NULL,
    input_id          TEXT,
    external_job_id   TEXT,
    attempt_count     INTEGER NOT NULL,
    lease_owner       TEXT,
    lease_expires_at  TIMESTAMPTZ,
    result_uri        TEXT,
    result_json       JSONB,
    error_code        TEXT,
    created_at        TIMESTAMPTZ NOT NULL,
    updated_at        TIMESTAMPTZ NOT NULL,
    PRIMARY KEY (agent_id, workflow_id)
);

Each field has a recovery purpose:

• agent_id identifies the deployed agent.
• workflow_id identifies one logical unit of work, such as a report or enrichment request.
• status represents the workflow state.
• current_step tells the next runtime where to resume.
• state_version protects conditional updates from stale workers.
• input_id points to the stable input rather than copying it into every checkpoint.
• external_job_id lets a later run poll work that was already submitted.
• attempt_count supports bounded retry decisions.
• lease_owner and lease_expires_at coordinate overlapping invocations.
• result_json holds a small result needed by the next step.
• result_uri points to a large or immutable artifact.
• error_code records a machine-readable failure category.
• Timestamps support operational queries and stuck-workflow detection.

Keep prompts, complete model responses, downloaded pages, and verbose logs out of this row. Store them only if the workflow requires them, and use a separate artifact store when they are large. A competitor report might keep its final object location and source IDs in the state record, while the rendered report lives in object storage.

Add a schema version if the record format will evolve independently from state_version. The former describes data structure. The latter controls concurrent updates.

Choose durable storage without overbuilding

Choose storage based on the operations your run loop needs.

A key-value store fits workflows that load one record by a composite key and update it conditionally. It keeps the access pattern narrow: fetch a checkpoint, verify its version, then replace it.

A relational database fits workflows that need constraints, transactions, filtering, or operational queries. It is useful when you need to find expired leases, inspect failures by error code, or associate workflow state with existing application records.

Object storage fits large, immutable artifacts. Examples include generated reports, normalized document collections, and exported datasets. It is a poor coordination mechanism because your worker still needs a compact record describing which object is current and which step owns it.

A queue delivers work. It should not own workflow truth. Messages can trigger workflow_id, but the durable state record decides whether the work is new, already running, or complete.

A minimal design usually combines:

1. A durable key-value or SQL record for progress, identifiers, leases, and result pointers.
2. Object storage for results too large for the workflow record.
3. A scheduler or queue that passes stable workflow identifiers.

This separation also helps with resumable AI tasks that can exceed a Lambda timeout. A bounded invocation completes one step, commits its checkpoint, and exits. Another invocation continues from that checkpoint.

Before choosing a backend, review its latency, request cost, retention controls, encryption options, backup process, and access model. Give the agent permission only to the records and artifact locations it needs. Retain state long enough to support recovery and investigation, then delete it under an explicit policy.

Make every step resumable and idempotent

A resumable run follows a strict order:

1. Load durable state.
2. Claim or renew a lease.
3. Verify the expected state version.
4. Execute one bounded step.
5. Persist its essential result.
6. Atomically move to the next step.

Use a deterministic idempotency key for every operation that may create a side effect:

import hashlib

def idempotency_key(
    agent_id: str,
    workflow_id: str,
    step: str,
    input_bytes: bytes,
) -> str:
    input_hash = hashlib.sha256(input_bytes).hexdigest()
    raw = f"{agent_id}/{workflow_id}/{step}/{input_hash}"
    return hashlib.sha256(raw.encode("utf-8")).hexdigest()

Pass that key to an external API when it supports idempotent requests. After submitting asynchronous work, persist the returned job ID immediately. Later invocations should poll that ID rather than submit another job.

Use a conditional update to advance state:

UPDATE agent_workflows
SET
    status = 'running',
    current_step = %(next_step)s,
    state_version = state_version + 1,
    result_json = %(result_json)s,
    result_uri = %(result_uri)s,
    updated_at = CURRENT_TIMESTAMP
WHERE agent_id = %(agent_id)s
  AND workflow_id = %(workflow_id)s
  AND state_version = %(expected_version)s
  AND lease_owner = %(run_id)s;

Check the affected row count. If no row changed, another worker modified the record or the lease is no longer yours. Reload state instead of overwriting it.

This is compare-and-swap: update the record only if its version still matches the version you read. A lease adds temporary ownership, but it is not permanent. Its expiry allows recovery when a runtime dies without releasing it.

The dangerous failure window sits between a side effect and its checkpoint. An API request may succeed, then the runtime may stop before saving the returned identifier. On retry, your code cannot safely assume that the request failed. A deterministic idempotency key, a provider-side lookup, or a locally reserved operation record is required to resolve that ambiguity.

Design for retries, overlap, and platform-specific limits

Model retries as normal control flow. Your state layer must tolerate crashes, duplicate deliveries, overlapping invocations, and late responses.

Use explicit states such as:

pending -> running -> succeeded
                   \-> failed

Keep workflow progress forward-moving. A retry can create a new attempt for the same step, but it should not silently move a completed workflow back to an earlier step. Classify errors as retryable or terminal, and stop retrying after the workflow’s configured attempt policy is exhausted.

Do not assume the platform will provide the concurrency semantics your application needs. Databricks continuous jobs permit only one active instance and recommend less than 60 seconds between the end of one run and the start of the next. Failed jobs restart automatically with exponential backoff, while job-level dependencies and retry policies are not supported. On serverless compute, bounded Structured Streaming triggers such as Trigger.AvailableNow are supported, but Trigger.ProcessingTime and Trigger.Continuous are not (source).

Those controls are platform-specific. Your persisted workflow record should remain correct even if scheduling behavior changes.

Shared compute memory is not a substitute. On Azure Databricks, tasks sharing compute use the same driver JVM, so class state and singletons can persist during that job run. Parallel tasks can overwrite mutable singleton state. The state does not provide durable cross-run persistence (source).

Pass parameters explicitly between steps. Persist cross-run progress externally. Treat shared memory as an implementation detail of the current run.

Operate the state layer safely in production

Every transition should emit a structured log containing:

{
  "workflow_id": "weekly-example-com",
  "run_id": "runtime-generated-id",
  "step": "summarize",
  "state_version": "current-version",
  "attempt": "current-attempt",
  "latency_ms": "measured-duration",
  "result_uri": "object-location"
}

Do not put complete prompts, secrets, credentials, or unnecessary source content in logs. Use identifiers that let you correlate the runtime, state record, and artifact.

Track:

• Expired or stuck leases.
• Retry frequency by step and error code.
• Duplicate operations suppressed by idempotency keys.
• Age of the latest checkpoint.
• Failed conditional updates.
• Invalid state transitions.
• Artifact writes without a corresponding state transition.

The same identifiers should appear in logs and traces. See the Python guide to AI agent observability for a practical instrumentation approach. Storage requests, repeated model calls, and retry loops also belong in your production agent cost model.

Before deployment, verify:

• [ ] State lives in an external durable store.
• [ ] Progress transitions are atomic.
• [ ] Side effects use deterministic idempotency keys.
• [ ] External job IDs are saved before polling.
• [ ] Expired leases can be recovered.
• [ ] Retryable and terminal failures are distinct.
• [ ] Large artifacts use dedicated storage.
• [ ] Retention and deletion policies are explicit.
• [ ] Storage encryption and least-privilege access are configured.
• [ ] Schema changes are versioned.
• [ ] Tests terminate the runtime after each side effect and verify safe recovery.

Conclusion

Persistent state for serverless jobs should be small and deliberate. Store the current step, stable IDs, version, attempt metadata, lease, error code, and essential result or artifact pointer. Make each step bounded and idempotent. Advance state with conditional writes.

This lets your agent survive runtime loss without preserving the runtime itself. A serverless deployment platform such as HollowHost can schedule and isolate the execution, while your compact state layer gives each new run enough context to continue safely.