AI Agent Cold Starts: Measure and Reduce Lambda Time¶
An AI agent cold start is not one delay. It is a sequence: Lambda initialization, Python imports, client setup, the model request, and the wait for the first streamed token. If you measure only total duration, you cannot tell which part to optimize. This guide adds structured timing to a Python Lambda, defines a repeatable cold-versus-warm benchmark, and maps each latency segment to a practical fix.
What an AI Agent Cold Start Actually Includes¶
A Lambda invocation has several latency segments:
- Lambda INIT: AWS creates an execution environment and initializes the runtime.
- Module import: Python imports your handler and its transitive dependencies.
- Dependency setup: Your code creates SDK clients, loads configuration, or initializes tools.
- Handler setup: The handler validates input and builds the model request.
- Model request: DNS, connection setup, TLS, and provider-side processing begin.
- Time to first token (TTFT): The interval from sending the request to receiving the first streamed token.
- Agent execution: The agent runs tools, updates state, and may issue more model requests.
Do not call all of this “cold-start time.” Lambda INIT and Python initialization happen before handler execution. Model latency happens after invocation starts. They need separate metrics.
For a conventional Lambda function, a cold start can range from under 100 milliseconds to more than one second, depending on the runtime. Agent bundles containing components such as LangChain, an OpenAI SDK, and a vector-store client commonly add another 250–450 milliseconds beyond runtime initialization, according to Blaxel’s analysis.
Agents also amplify downstream latency. If an agent performs tool discovery, retrieval, and summarization as sequential steps, every model or tool call extends the critical path. Parallelize independent work. Remove calls that do not affect the final result.
This distinction matters whether you run scheduled reports, lead enrichment, or competitor monitoring. A slow scheduled job may tolerate INIT latency. An event-driven agent responding to a webhook may not. See the broader tradeoffs in serverless AI agent deployment.
Instrument Import Time, Dependency Loading, and First-Token Latency Separately¶
Place the first timestamp before application imports. Then measure standard-library imports, the external HTTP dependency import, configuration loading, and HTTP client initialization at separate boundaries.
The following Lambda handler uses httpx as a real external dependency. It imports and initializes a reusable HTTP client at module scope, then calls an OpenAI-compatible streaming endpoint. Configure MODEL_URL, MODEL_NAME, and MODEL_API_KEY as environment variables.
import time
PROCESS_START = time.perf_counter()
import json
import os
BASE_IMPORTS_DONE = time.perf_counter()
import httpx
DEPENDENCY_IMPORTS_DONE = time.perf_counter()
MODEL_URL = os.environ["MODEL_URL"]
MODEL_NAME = os.environ["MODEL_NAME"]
MODEL_API_KEY = os.environ["MODEL_API_KEY"]
CONFIG_READY = time.perf_counter()
MODEL_CLIENT = httpx.Client(
headers={
"Authorization": f"Bearer {MODEL_API_KEY}",
"Content-Type": "application/json",
},
timeout=httpx.Timeout(60.0, connect=10.0),
)
CLIENT_READY = time.perf_counter()
IS_FIRST_INVOCATION = True
def elapsed_ms(start: float, end: float) -> float:
return round((end - start) * 1000, 2)
def log_metric(request_id: str, **fields) -> None:
print(json.dumps({
"event": "agent_latency",
"request_id": request_id,
**fields,
}, separators=(",", ":")))
def lambda_handler(event, context):
global IS_FIRST_INVOCATION
handler_start = time.perf_counter()
cold = IS_FIRST_INVOCATION
IS_FIRST_INVOCATION = False
prompt = event.get("prompt", "Summarize the supplied text.")
payload = {
"model": MODEL_NAME,
"stream": True,
"messages": [{"role": "user", "content": prompt}],
}
request_start = time.perf_counter()
first_token_at = None
chunks = []
with MODEL_CLIENT.stream(
"POST",
MODEL_URL,
json=payload,
) as response:
headers_received = time.perf_counter()
response.raise_for_status()
for line in response.iter_lines():
line = line.strip()
if not line.startswith("data:"):
continue
data = line.removeprefix("data:").strip()
if data == "[DONE]":
break
message = json.loads(data)
token = (
message.get("choices", [{}])[0]
.get("delta", {})
.get("content")
)
if token:
if first_token_at is None:
first_token_at = time.perf_counter()
chunks.append(token)
finished_at = time.perf_counter()
cold_only_metrics = {
"base_import_ms": (
elapsed_ms(PROCESS_START, BASE_IMPORTS_DONE)
if cold else None
),
"dependency_import_ms": (
elapsed_ms(BASE_IMPORTS_DONE, DEPENDENCY_IMPORTS_DONE)
if cold else None
),
"import_ms": (
elapsed_ms(PROCESS_START, DEPENDENCY_IMPORTS_DONE)
if cold else None
),
"config_load_ms": (
elapsed_ms(DEPENDENCY_IMPORTS_DONE, CONFIG_READY)
if cold else None
),
"dependency_setup_ms": (
elapsed_ms(CONFIG_READY, CLIENT_READY)
if cold else None
),
}
metrics = {
"request_id": context.aws_request_id,
"cold_process": cold,
**cold_only_metrics,
"handler_setup_ms": elapsed_ms(
handler_start, request_start
),
"request_to_headers_ms": elapsed_ms(
request_start, headers_received
),
"ttft_ms": (
elapsed_ms(request_start, first_token_at)
if first_token_at else None
),
"stream_ms": (
elapsed_ms(first_token_at, finished_at)
if first_token_at else None
),
"handler_total_ms": elapsed_ms(
handler_start, finished_at
),
}
log_metric(context.aws_request_id, **metrics)
return {
"statusCode": 200,
"body": json.dumps({
"output": "".join(chunks),
"metrics": metrics,
}),
}
import_ms is not the complete Lambda INIT duration. It measures imports after PROCESS_START, while dependency_import_ms isolates the external httpx import. Compare these values with the INIT duration in Lambda’s platform logs. The difference includes work outside your application timer.
dependency_setup_ms now measures actual client initialization rather than environment-variable reads. config_load_ms records those reads separately. Because imports and client initialization happen only once per execution environment, the handler returns null for those fields on warm invocations instead of implying that warm requests paid the same cost again.
request_to_headers_ms approximates connection and upstream response setup. The client object and its connection pool are reused across warm invocations, although the first request may still need to establish a network connection. ttft_ms includes that interval plus the wait for token generation. Keep both. Otherwise, a networking regression can look like a model regression.
Use structured fields rather than parsing prose. The same pattern fits a broader Python agent observability setup.
Build a Reproducible Cold-vs-Warm Lambda Benchmark¶
Do not compare one cold invocation with one warm invocation. Use controlled groups.
A practical protocol is:
- Deploy one immutable code artifact to a disposable benchmark function.
- Keep the code, model, prompt, AWS region, and model-provider region fixed.
- Set reserved concurrency to one during the benchmark to reduce environment switching between sequential invocations.
- Run at least 20–30 cold and warm repetitions.
- Store the Lambda request ID with every application metric.
- Separate successful responses, provider errors, throttles, and timeouts.
- Report p50, p95, and p99 for each latency segment.
- Record billed duration and INIT-related cost beside latency.
For the warm group, invoke the same environment repeatedly. For the cold group, create fresh execution conditions through your deployment or test process. Do not assume that a long pause guarantees a new environment. Confirm cold execution with platform INIT logs and your cold_process field.
The following executable script updates only the description of a disposable $LATEST benchmark function before each cold candidate. A configuration update recycles its execution environments. The script then immediately sends a second invocation as the warm candidate, classifies both responses using cold_process, extracts INIT and billed duration from the Lambda log tail, and calculates p50, p95, and p99.
import argparse
import base64
import json
import math
import re
import time
from collections import defaultdict
import boto3
INIT_PATTERN = re.compile(r"Init Duration:\s*([\d.]+)\s*ms")
BILLED_PATTERN = re.compile(r"Billed Duration:\s*([\d.]+)\s*ms")
def percentile(values: list[float], percent: int) -> float | None:
if not values:
return None
ordered = sorted(values)
index = max(0, math.ceil((percent / 100) * len(ordered)) - 1)
return round(ordered[index], 2)
def parse_platform_metrics(log_result: str | None) -> dict:
if not log_result:
return {}
log_text = base64.b64decode(log_result).decode(
"utf-8",
errors="replace",
)
init_match = INIT_PATTERN.search(log_text)
billed_match = BILLED_PATTERN.search(log_text)
return {
"init_ms": (
float(init_match.group(1))
if init_match else None
),
"billed_duration": (
float(billed_match.group(1))
if billed_match else None
),
}
def invoke(client, function_name: str, prompt: str) -> dict:
response = client.invoke(
FunctionName=function_name,
InvocationType="RequestResponse",
LogType="Tail",
Payload=json.dumps({"prompt": prompt}).encode("utf-8"),
)
payload = json.loads(response["Payload"].read())
if response.get("FunctionError"):
raise RuntimeError(json.dumps(payload))
body = payload.get("body", payload)
if isinstance(body, str):
body = json.loads(body)
metrics = body["metrics"]
metrics.update(parse_platform_metrics(response.get("LogResult")))
return metrics
def summarize(groups: dict[str, list[dict]]) -> dict:
summary = {}
for group_name, rows in groups.items():
values_by_metric = defaultdict(list)
for row in rows:
for name, value in row.items():
if isinstance(value, bool):
continue
if isinstance(value, (int, float)):
values_by_metric[name].append(float(value))
summary[group_name] = {
"samples": len(rows),
"metrics": {
name: {
"p50": percentile(values, 50),
"p95": percentile(values, 95),
"p99": percentile(values, 99),
}
for name, values in sorted(values_by_metric.items())
},
}
return summary
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--function", required=True)
parser.add_argument("--region")
parser.add_argument("--rounds", type=int, default=30)
parser.add_argument(
"--prompt",
default="Reply with one short sentence about cold starts.",
)
args = parser.parse_args()
client = boto3.client("lambda", region_name=args.region)
groups = {"cold": [], "warm": []}
for round_number in range(1, args.rounds + 1):
marker = f"cold-start-benchmark-{time.time_ns()}"
client.update_function_configuration(
FunctionName=args.function,
Description=marker,
)
client.get_waiter("function_updated").wait(
FunctionName=args.function
)
for planned_group in ("cold", "warm"):
metrics = invoke(
client,
args.function,
args.prompt,
)
actual_group = (
"cold" if metrics["cold_process"] else "warm"
)
groups[actual_group].append(metrics)
print(json.dumps({
"round": round_number,
"planned_group": planned_group,
"actual_group": actual_group,
**metrics,
}, separators=(",", ":")))
print(json.dumps(
{"summary": summarize(groups)},
indent=2,
sort_keys=True,
))
if __name__ == "__main__":
main()
Install the local benchmark dependency and run the script against a disposable function:
python -m pip install boto3
python benchmark_lambda.py \
--function ai-agent-cold-start-benchmark \
--region us-east-1 \
--rounds 30
The benchmark identity needs permission to call lambda:InvokeFunction, lambda:UpdateFunctionConfiguration, and lambda:GetFunctionConfiguration. Do not run this recycling procedure against a production function. Save the line-delimited invocation records and the final JSON summary as benchmark artifacts.
Correlate three sources:
- INIT: Lambda platform logs.
- INVOKE: handler and billed-duration records.
- Application: the JSON timing event emitted by your code.
Export these fields for analysis:
request_id
cold_process
init_ms
base_import_ms
dependency_import_ms
import_ms
config_load_ms
dependency_setup_ms
handler_setup_ms
request_to_headers_ms
ttft_ms
stream_ms
handler_total_ms
billed_duration
status
A study cited by Blaxel measured 750 milliseconds of total cold-start latency for a five-step chain, compared with 250 milliseconds in a warm environment. Treat that as context, not your baseline. Your package, region, provider, and tool graph determine your result.
Reduce Python Import and Dependency Loading Overhead¶
Start with the package. Agent frameworks are convenient, but a Lambda should not import an entire framework to make one model call.
Apply these changes in order:
- Remove unused frameworks, providers, vector-store clients, and document loaders.
- Import rarely used tools inside the branch that needs them.
- Avoid wildcard imports and convenience modules with large transitive graphs.
- Package only production code.
- Pin dependencies with a lockfile.
- Build native wheels for the exact Lambda runtime and architecture.
- Measure Layers rather than assuming they improve startup time.
For example, move an optional PDF parser out of module scope:
def extract_text(document_type: str, content: bytes) -> str:
if document_type == "pdf":
from your_pdf_package import parse_pdf
return parse_pdf(content)
return content.decode("utf-8")
Lazy imports shift work into the invocation that uses the tool. That is useful only when most invocations avoid it. Record tool-specific setup time so you do not hide the cost.
Agent dependency bundles commonly add 250–450 milliseconds beyond runtime initialization. Use the full process in this guide to building smaller, reproducible serverless Python packages.
Reduce Time-to-First-Token Without Hiding the Cost¶
Once imports are under control, optimize the request path:
- Initialize reusable SDK clients outside the handler.
- Reuse connections when your HTTP client supports pooling.
- Place the Lambda and model endpoint in appropriate regions.
- Stream responses when the caller can consume partial output.
- Minimize prompt content that does not affect the task.
- Remove redundant model calls.
- Run independent tools concurrently.
- Cache stable tool results with explicit freshness rules.
Streaming improves perceived response time, but it does not reduce total execution by itself. Keep TTFT, stream duration, and total duration as separate metrics.
Provisioned concurrency removes cold-start latency by keeping environments ready. Its efficiency depends on predictable traffic, however. Irregular and bursty agent workloads are harder to forecast, according to Blaxel.
Cost belongs in the same benchmark. Blaxel reports that AWS has billed Lambda INIT since August 2025. A packaging change that lowers INIT duration may therefore affect both latency and direct execution cost.
Package the Executable Example for Production and Choose the Right Runtime¶
The handler uses httpx as an external Python dependency. Pin it in requirements.txt:
Build a Lambda ZIP containing both the handler and its dependencies:
rm -rf package function.zip
mkdir package
python -m pip install \
--requirement requirements.txt \
--target package
cp lambda_function.py package/
cd package
zip -r ../function.zip .
cd ..
Deploy function.zip with the handler set to:
Configure these environment variables through your secrets and deployment system:
Then add production controls:
- A timeout aligned with the job’s latency budget.
- Memory tests using the same benchmark protocol.
- JSON logs with request and run identifiers.
- Alarms for errors, timeouts, and TTFT budget breaches.
- A pass/fail gate based on cold and warm percentiles.
- Redaction rules that keep prompts, tokens, and secrets out of logs.
Test memory configurations rather than assuming the smallest setting is cheapest. Compare INIT, import, dependency setup, TTFT, total duration, and billed cost together.
Use this runtime decision matrix:
| Runtime | Choose it when | Main tradeoff |
|---|---|---|
| Optimized on-demand Lambda | Jobs are intermittent and tolerate measured cold starts | Startup latency varies |
| Provisioned concurrency | Request latency matters and traffic is forecastable | Ready capacity must match demand |
| Always-on runtime | The agent needs continuously warm connections or sustained execution | You operate persistent capacity |
If the agent cannot finish safely within one invocation, cold-start tuning is no longer the main architectural issue. Split the workflow into checkpoints using a resumable Lambda design for AI agents.
Conclusion: Measure the timeline before changing the runtime. Separate Lambda INIT, Python imports, external dependency imports, client initialization, request connection time, and first-token latency. Then optimize the largest segment and rerun the same benchmark. A managed agent platform such as HollowHost can reduce the deployment work around packaging, secrets, logs, and scheduled execution, but the core rule remains the same: keep every latency boundary visible.