LangSmith Preview Builds: Compare Agents Before Production¶

LangSmith Preview Builds showing an isolated agent deployment where teams compare pull request changes, tool calls, and traces before production.

An agent can pass local tests and still fail after a prompt, tool, model, or dependency change. LangSmith Preview Builds reduce that risk by deploying a pull request branch into an isolated environment before you merge it. The deployment alone is not enough, though. You still need repeatable fixtures, a production baseline, measurable acceptance gates, and trace review. This guide combines those pieces into a practical preview-versus-production workflow.

What LangSmith Preview Builds are—and when to use them¶

A LangSmith Preview Build is a temporary, production-like deployment created from the source branch of a pull request. It remains connected to the parent deployment but does not update that parent.

Each pull request gets an isolated preview. Multiple previews can run in parallel, so reviewers do not have to keep replacing one shared staging environment as branches change. That makes Preview Builds useful for changes such as:

Updating a system prompt.
Adding or changing a tool.
Replacing a model or provider.
Upgrading agent dependencies.
Modifying LangGraph state or routing logic.
Changing error handling and fallback behavior.

Reviewers can interact with the deployed agent without cloning the repository or reproducing the developer’s local environment. Engineers can then inspect the traces and tool calls associated with those responses.

This isolation is the key benefit. A prompt change in one pull request cannot silently move the staging target used by another reviewer. Production remains unchanged until you approve and merge the code.

See LangChain’s Preview Builds announcement for the underlying deployment model. If you need context on how deployment fits with datasets, tracing, and evaluation, start with the complete LangSmith reliability loop.

Prerequisites and GitHub setup for Preview Builds¶

Preview Builds are in public beta. According to the LangSmith Preview Builds documentation, they are available only in LangSmith Cloud for deployments created through the GitHub integration.

To enable them:

Open the parent deployment in LangSmith Cloud.
Open its deployment settings.
Find the Preview Builds configuration.
Enable Preview Builds.
Select a trigger mode.
Configure cleanup and concurrency settings for your team.

You can choose between two trigger modes:

Every PR creates a Preview Build whenever a pull request targets the deployment branch. Use it when every proposed change should receive an isolated environment.

Label only waits until someone adds the configured GitHub label. Use it when previews should run only for selected changes. For example, documentation-only pull requests may not need an agent deployment.

The Preview Build uses the latest commit from the pull request’s source branch. Each new commit automatically creates a new revision of that preview deployment.

That behavior affects review. An evaluation report for an earlier commit is stale as soon as the author pushes another commit. Record the commit SHA with every test result, then verify it again before approval.

Build a deterministic evaluation set before opening a PR¶

Preview Builds provide the environment. They do not, based on the supplied Preview Builds sources, define a deterministic test harness or a quantitative production comparison. You need to add that layer.

Start with versioned fixtures in your repository:

evals/
├── cases.json
├── tool_fixtures/
│   ├── account_active.json
│   └── account_missing.json
└── graders.py

A compact cases.json might look like this:

[
  {
    "id": "known-account",
    "input": {
      "message": "Summarize the status of account acct_test_active",
      "now": "2026-09-23T12:00:00Z"
    },
    "expected": {
      "status": "active",
      "required_tool": "get_account"
    }
  },
  {
    "id": "missing-account",
    "input": {
      "message": "Summarize the status of account acct_test_missing",
      "now": "2026-09-23T12:00:00Z"
    },
    "expected": {
      "status": "not_found",
      "required_tool": "get_account"
    }
  }
]

A deterministic harness does not guarantee identical natural-language output. It controls the inputs and the grading conditions. Keep these elements stable:

User inputs and conversation history.
Evaluation timestamps.
Tool responses.
Test credentials and test data.
Expected structured fields.
Grading rules.
Test-set version.

Prefer structured assertions over full-text equality. An agent may phrase a correct answer differently while preserving the required status, evidence, and tool behavior.

For tools, route evaluation traffic to fixtures or isolated test services. Do not let a regression suite send email, update production records, or write to a live customer database. If a write must be tested, target a disposable resource and verify the resulting state.

Commit the fixtures with the agent code. A change to the tests then becomes visible in the same pull request. For a broader design covering reference datasets, graders, and regression testing, read the reproducible AI agent evaluation guide.

Run the same cases against production and the Preview Build¶

Treat production as the baseline and the Preview Build as the candidate. Send the same fixture to both endpoints, then compare:

Case pass or failure.
Task-specific score.
Required tool selection.
Tool arguments.
Errors.
Response latency.
Reported token or cost data, when your response or tracing adapter exposes it.

This comparison harness is a recommended CI workflow. It is not described as a native Preview Builds capability in the supplied sources.

The following Python script expects both endpoints to accept JSON and return a JSON object containing status and tool_calls. Adapt call_agent() to your deployment contract.

import json
import os
import time
import urllib.request

PRODUCTION_URL = os.environ["PRODUCTION_URL"]
PREVIEW_URL = os.environ["PREVIEW_URL"]
COMMIT_SHA = os.environ.get("GITHUB_SHA", "local")
TEST_SET_VERSION = os.environ.get("TEST_SET_VERSION", "working-tree")


def call_agent(url, payload):
    body = json.dumps(payload).encode("utf-8")
    request = urllib.request.Request(
        url,
        data=body,
        headers={"Content-Type": "application/json"},
        method="POST",
    )

    started = time.perf_counter()
    with urllib.request.urlopen(request, timeout=60) as response:
        result = json.loads(response.read())
    latency_ms = round((time.perf_counter() - started) * 1000, 2)

    return result, latency_ms


def grade(result, expected):
    tool_names = [
        call.get("name")
        for call in result.get("tool_calls", [])
    ]
    checks = {
        "status": result.get("status") == expected["status"],
        "required_tool": expected["required_tool"] in tool_names,
    }
    return checks, all(checks.values())


with open("evals/cases.json", encoding="utf-8") as file:
    cases = json.load(file)

report = {
    "commit": COMMIT_SHA,
    "test_set_version": TEST_SET_VERSION,
    "cases": [],
}

for case in cases:
    baseline, baseline_latency = call_agent(
        PRODUCTION_URL, case["input"]
    )
    candidate, candidate_latency = call_agent(
        PREVIEW_URL, case["input"]
    )

    baseline_checks, baseline_passed = grade(
        baseline, case["expected"]
    )
    candidate_checks, candidate_passed = grade(
        candidate, case["expected"]
    )

    report["cases"].append({
        "id": case["id"],
        "production": {
            "passed": baseline_passed,
            "checks": baseline_checks,
            "latency_ms": baseline_latency,
        },
        "preview": {
            "passed": candidate_passed,
            "checks": candidate_checks,
            "latency_ms": candidate_latency,
        },
    })

with open("comparison-report.json", "w", encoding="utf-8") as file:
    json.dump(report, file, indent=2)

regressions = [
    case["id"]
    for case in report["cases"]
    if case["production"]["passed"]
    and not case["preview"]["passed"]
]

if regressions:
    raise SystemExit(f"Preview regressions: {regressions}")

Run it after the preview endpoint becomes available:

export PRODUCTION_URL="https://production.example.test/invoke"
export PREVIEW_URL="https://preview.example.test/invoke"
export TEST_SET_VERSION="$(git rev-parse HEAD:evals)"
python compare_agents.py

Store comparison-report.json as a CI artifact. Include the preview revision, commit SHA, test-set version, and grader version. Averages alone can hide important failures, so retain case-level results.

Inspect traces, tools, secrets, and failure cases before approval¶

Automated scores tell you where behavior changed. Traces help explain why.

LangSmith Preview Builds let reviewers test prompts, tool calls, configured dependencies, edge cases, and failure scenarios in isolation. Review the traces for cases where:

The final answer passed but the wrong tool ran.
A tool received malformed or excessive arguments.
The agent retried a non-idempotent action.
A fallback hid an upstream error.
A dependency change altered routing.
The agent exposed internal tool output.

Trace review should complement deterministic grading, not replace it. A passing response can still contain an unsafe execution path. The Python agent observability guide explains how to connect logs and traces to actionable runtime signals.

Secrets need separate attention. A Preview Build inherits secrets from its parent when it is created. You can override them on the preview. Later changes to parent secrets do not propagate to existing previews.

Do not assume inherited credentials are appropriate for testing. Replace production credentials with restricted preview credentials. Isolate databases, MCP servers, queues, object storage, and external write actions. Use least-privilege access and disposable data where possible.

Set acceptance gates, clean up previews, and deploy safely¶

Define approval rules before reviewing the pull request. A practical policy can require:

No regression on critical fixtures.
The configured minimum quality score.
Correct tools and arguments for protected actions.
Latency within your team’s accepted bound.
No unexpected external writes.
Trace review for changed or failed cases.
Reviewer approval.

Also verify that the evaluated commit matches the current Preview Build revision. Each new commit on the pull request creates a new preview revision, so previous approval should not automatically carry forward.

LangSmith provides lifecycle controls for these environments. Idle TTL defines how long a preview may remain inactive after its latest revision before automatic deletion. Max concurrent previews limits how many previews can run simultaneously for one parent deployment.

You can also delete a Preview Build manually. Deleting the parent deployment removes its Preview Builds as well.

The resulting path is straightforward: open the pull request, create the isolated preview, evaluate production and preview against the same fixtures, inspect changed traces, confirm the commit, approve, and merge. Only then should the production deployment move forward.

Preview Builds solve the environment-isolation part of agent review. Deterministic fixtures, restricted credentials, CI artifacts, and explicit gates solve the release-decision part. Together, they turn a pull request into a testable deployment candidate. The same pattern also applies when your production agent runs as a scheduled or serverless job: compare the candidate in isolation, preserve the evidence, and deploy only the reviewed commit.