Compound AI Systems: Groq, E2B, and Isolated Agents

A multi-tool agent often starts as one Python process. It plans, calls APIs, executes code, and formats a response. Then production requirements arrive: retries, permissions, timeouts, logs, persistent artifacts, and independent releases.

The fix is not a larger agent. It is decomposition.

This guide presents a reproducible Groq–E2B reference pattern. Groq sits behind the model boundary. E2B provides isolated execution for jobs that run code or handle untrusted inputs. The design is illustrative. It does not claim to document Groq’s private production architecture.

What compound AI systems are—and why decomposition matters

Databricks defines compound AI systems as applications that combine components such as models, agents, retrieval systems, databases, external tools, code interpreters, and permission systems.

A retrieval-augmented generation application already qualifies. It combines a model with a retrieval system. More elaborate systems can add planning, repeated model calls, external APIs, code execution, and output validation.

That composition matters because a model does not need to perform every operation itself. You can assign narrow responsibilities to specialized components:

• A model classifies or plans.
• A retriever finds approved source material.
• A tool reads from an internal API.
• A sandbox executes generated code.
• A validator checks the resulting artifact.
• An orchestrator applies retries, budgets, and permissions.

This also expands your cost–quality options. Databricks notes that combining language models with external tools creates more possible trade-offs than relying on an individual model at a fixed quality and cost level. A smaller model paired with search heuristics can sometimes be a less expensive option than a larger model.

The operational benefit is equally important. Each boundary gives you somewhere to enforce policy and observe failure. If retrieval returns irrelevant documents, you can test retrieval without blaming generation. If code execution times out, you can retry the job without repeating the planner call.

Decomposition turns “the agent failed” into a specific diagnosis.

The Groq–E2B reference architecture: model, agent, tool, and job boundaries

Use Groq and E2B as separate infrastructure boundaries rather than dependencies spread throughout your business logic.

In this reference pattern:

• Groq model adapter: Handles model inference behind one application interface.
• Planner agent: Converts a user request into an allowlisted action.
• Tool: Defines a narrow, typed capability.
• E2B sandbox adapter: Starts an isolated environment for code or untrusted work.
• Job: Performs one bounded unit of execution.
• Artifact validator: Rejects malformed, incomplete, or unauthorized output.
• Orchestrator: Owns state, retries, deadlines, and the final response.

The request flow is:

trigger
  -> planner agent
  -> typed tool call
  -> isolated E2B job
  -> validated artifact
  -> final response

Suppose your agent produces a weekly competitor-monitoring report. The planner can decide that it needs an analysis job. The tool contract accepts approved source records. The sandbox runs the transformation. A validator checks that the output contains the expected fields and source references. Only then does the response layer publish the report.

The model never receives unrestricted infrastructure access. It selects from capabilities exposed by the application.

This design follows a core property of compound systems described by IBM: specialized components can operate collaboratively or sequentially, and components can be replaced independently without rebuilding the entire system.

You can therefore change the model adapter without rewriting sandbox lifecycle logic. You can update the job image without changing the planner. You can also replace a tool while preserving its contract.

Illustrative Groq → E2B sandbox pipeline

The following Python program illustrates a Groq SDK call, the creation of an E2B sandbox, the transfer of a JSON input, the execution of a bounded job, the retrieval of a staged artifact, its validation, and its promotion to durable S3-compatible storage.

This is intentionally a Groq → sandbox pipeline, not a complete implementation of the planner-agent architecture described above. Groq only summarizes approved records; it does not select an action, emit a structured plan, or make a typed tool call. The orchestrator invokes run_isolated_report_job directly. A production planner would need to return a validated structured action whose name and arguments are checked against an allowlist before dispatch.

The dependency versions below form an illustrative snapshot, not a compatibility matrix or a set of SDK behaviors verified by the supplied fact dossier. Before running or adopting the example, compare it with the versioned references for the groq-python v0.31.1 client, the E2B Python SDK v1.5.1 source, and the boto3 1.40.21 documentation.

In particular, verify the construction of Sandbox, the filesystem methods, timeout units and semantics, cleanup behavior, and the fields returned by command execution against the E2B reference for the version you install. The example expects a command result with an exit_code attribute, but that expectation belongs to this illustrative adapter rather than to a generic E2B contract.

Install the illustrative dependency snapshot:

python -m pip install \
  "groq==0.31.1" \
  "e2b==1.5.1" \
  "boto3==1.40.21"

Set the required environment variables:

export GROQ_API_KEY="..."
export E2B_API_KEY="..."
export ARTIFACT_BUCKET="compound-agent-artifacts"
export AWS_REGION="us-east-1"
export GROQ_MODEL="llama-3.3-70b-versatile"

Save this program as run_report.py:

import json
import os
import uuid
from typing import Any

import boto3
from e2b import Sandbox
from groq import Groq


SANDBOX_INPUT = "/tmp/input/report-input.json"
SANDBOX_STAGING_ARTIFACT = "/tmp/staging/report.json"


def validate_artifact(raw_artifact: str, expected_run_id: str) -> dict[str, Any]:
    artifact = json.loads(raw_artifact)

    required = {"run_id", "status", "summary", "source_ids"}
    missing = required - artifact.keys()
    if missing:
        raise ValueError(f"Artifact is missing fields: {sorted(missing)}")

    if artifact["run_id"] != expected_run_id:
        raise ValueError("Artifact run_id does not match the requested run")

    if artifact["status"] != "completed":
        raise ValueError(f"Unexpected artifact status: {artifact['status']}")

    if not isinstance(artifact["source_ids"], list):
        raise TypeError("source_ids must be a list")

    return artifact


def create_model_summary(sources: list[dict[str, str]]) -> str:
    client = Groq(api_key=os.environ["GROQ_API_KEY"])
    model = os.environ.get("GROQ_MODEL", "llama-3.3-70b-versatile")

    response = client.chat.completions.create(
        model=model,
        temperature=0,
        messages=[
            {
                "role": "system",
                "content": (
                    "Summarize the supplied approved records in one concise "
                    "paragraph. Do not invent facts or sources."
                ),
            },
            {
                "role": "user",
                "content": json.dumps(sources),
            },
        ],
    )

    summary = response.choices[0].message.content
    if not summary:
        raise RuntimeError("Groq returned an empty summary")

    return summary


def run_isolated_report_job(
    run_id: str,
    sources: list[dict[str, str]],
    summary: str,
) -> str:
    payload = {
        "run_id": run_id,
        "sources": sources,
        "summary": summary,
    }

    sandbox = Sandbox.create(
        template="base",
        timeout=120,
        envs={
            "RUN_ID": run_id,
            "OUTPUT_PATH": SANDBOX_STAGING_ARTIFACT,
        },
    )

    try:
        sandbox.files.write(
            SANDBOX_INPUT,
            json.dumps(payload),
        )

        job_program = r'''
import json
import os
import pathlib
import sys

input_path = sys.argv[1]
output_path = pathlib.Path(os.environ["OUTPUT_PATH"])
output_path.parent.mkdir(parents=True, exist_ok=True)

with open(input_path, encoding="utf-8") as input_file:
    payload = json.load(input_file)

source_ids = []
for source in payload["sources"]:
    source_id = source.get("id")
    if source_id:
        source_ids.append(source_id)

artifact = {
    "run_id": os.environ["RUN_ID"],
    "status": "completed",
    "summary": payload["summary"],
    "source_ids": source_ids,
}

with output_path.open("w", encoding="utf-8") as output_file:
    json.dump(artifact, output_file)
'''

        sandbox.files.write("/tmp/report_job.py", job_program)

        command = sandbox.commands.run(
            f"python /tmp/report_job.py {SANDBOX_INPUT}",
            timeout=30,
        )

        if command.exit_code != 0:
            raise RuntimeError(
                f"E2B report job failed with exit code {command.exit_code}"
            )

        return sandbox.files.read(SANDBOX_STAGING_ARTIFACT)
    finally:
        # Cleanup is explicit even if the sandbox has a configured lifetime.
        sandbox.kill()


def promote_artifact(
    raw_artifact: str,
    run_id: str,
    bucket: str,
) -> str:
    artifact = validate_artifact(raw_artifact, expected_run_id=run_id)
    artifact_key = f"reports/{run_id}/report.json"

    s3 = boto3.client(
        "s3",
        region_name=os.environ.get("AWS_REGION", "us-east-1"),
    )
    s3.put_object(
        Bucket=bucket,
        Key=artifact_key,
        Body=json.dumps(artifact).encode("utf-8"),
        ContentType="application/json",
        Metadata={
            "run-id": run_id,
            "contract-version": "1",
        },
    )

    return f"s3://{bucket}/{artifact_key}"


def main() -> None:
    run_id = str(uuid.uuid4())
    sources = [
        {
            "id": "source-001",
            "title": "Approved product update",
            "text": "The monitored product added batch export support.",
        },
        {
            "id": "source-002",
            "title": "Approved pricing update",
            "text": "The monitored product introduced a new team plan.",
        },
    ]

    summary = create_model_summary(sources)
    staged_artifact = run_isolated_report_job(
        run_id=run_id,
        sources=sources,
        summary=summary,
    )
    durable_uri = promote_artifact(
        raw_artifact=staged_artifact,
        run_id=run_id,
        bucket=os.environ["ARTIFACT_BUCKET"],
    )

    print(json.dumps({
        "run_id": run_id,
        "status": "completed",
        "artifact_uri": durable_uri,
    }))


if __name__ == "__main__":
    main()

Run it with:

python run_report.py

The Groq credential remains in the orchestrator process. The E2B sandbox receives only RUN_ID, OUTPUT_PATH, and the transferred input file. AWS credentials also remain outside the sandbox: the orchestrator validates the retrieved staging artifact before uploading it to durable storage.

Because the SDK calls and return fields are illustrative and version-sensitive, add adapter-level integration tests before using this pipeline in production. Those tests should verify the installed SDK versions, sandbox lifecycle, timeout behavior, file transfer methods, command result shape, cleanup, and S3 upload behavior against the linked versioned documentation.

Turn one multi-tool agent into independently deployable jobs

Start by separating source code according to runtime responsibility:

compound-agent/
├── agents/
│   └── planner.py
├── tools/
│   └── analyze_records.py
├── jobs/
│   └── report_job.py
├── contracts/
│   └── report.py
├── evaluators/
│   └── report_quality.py
├── infra/
│   └── runtime.py
├── deploy/
│   ├── planner.Dockerfile
│   ├── report-job.Dockerfile
│   └── kubernetes/
│       ├── planner.yaml
│       └── report-job.yaml
└── tests/
    ├── test_contracts.py
    └── test_report_job.py

The contract is more important than the framework. It lets the planner, tool, and job change independently.

Here is a minimal executable contract using only the Python standard library:

from dataclasses import dataclass
from typing import Any


@dataclass(frozen=True)
class ReportJobInput:
    run_id: str
    sources: list[dict[str, Any]]
    output_name: str


@dataclass(frozen=True)
class ReportJobResult:
    run_id: str
    status: str
    artifact_path: str | None
    error_code: str | None


def validate_input(payload: dict[str, Any]) -> ReportJobInput:
    required = {"run_id", "sources", "output_name"}
    missing = required - payload.keys()

    if missing:
        raise ValueError(f"Missing fields: {sorted(missing)}")

    if not isinstance(payload["sources"], list):
        raise TypeError("sources must be a list")

    return ReportJobInput(
        run_id=str(payload["run_id"]),
        sources=payload["sources"],
        output_name=str(payload["output_name"]),
    )

Every deployable job should define:

• A versioned input schema.
• A versioned output schema.
• An idempotency key, such as run_id.
• An explicit timeout supplied by configuration.
• Named success and failure states.
• A stable artifact location.
• Rules for retryable and terminal errors.

Idempotency means that processing the same logical request again does not create an unintended duplicate result. Store run_id with the artifact or result record. Before running expensive work, check whether that identifier already completed.

Keep research, retrieval, code execution, and post-processing in different jobs when they have different permissions or failure modes. A retrieval retry should not rerun code execution. A formatting failure should not repeat external API reads.

Version and launch the planner and report job separately

Build the planner and job as separate images with independent immutable versions:

docker build \
  -f deploy/planner.Dockerfile \
  -t registry.example.com/compound-planner:1.4.0 \
  .

docker build \
  -f deploy/report-job.Dockerfile \
  -t registry.example.com/report-job:2.1.0 \
  .

docker push registry.example.com/compound-planner:1.4.0
docker push registry.example.com/report-job:2.1.0

The planner can run as a long-lived Kubernetes deployment:

 # deploy/kubernetes/planner.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: compound-planner
  labels:
    app: compound-planner
    component-version: "1.4.0"
spec:
  replicas: 2
  selector:
    matchLabels:
      app: compound-planner
  template:
    metadata:
      labels:
        app: compound-planner
        component-version: "1.4.0"
    spec:
      containers:
        - name: planner
          image: registry.example.com/compound-planner:1.4.0
          command: ["python", "-m", "agents.planner"]
          env:
            - name: REPORT_CONTRACT_VERSION
              value: "1"
            - name: GROQ_API_KEY
              valueFrom:
                secretKeyRef:
                  name: compound-agent-secrets
                  key: groq-api-key
          resources:
            requests:
              cpu: "100m"
              memory: "128Mi"
            limits:
              cpu: "500m"
              memory: "512Mi"

The report worker can be launched as a bounded Kubernetes job with its own image version, timeout, service account, and retry policy:

 # deploy/kubernetes/report-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: report-job-example
  labels:
    app: report-job
    component-version: "2.1.0"
spec:
  backoffLimit: 2
  activeDeadlineSeconds: 180
  ttlSecondsAfterFinished: 600
  template:
    metadata:
      labels:
        app: report-job
        component-version: "2.1.0"
    spec:
      restartPolicy: Never
      serviceAccountName: report-job
      containers:
        - name: report-job
          image: registry.example.com/report-job:2.1.0
          command: ["python", "-m", "jobs.report_job"]
          env:
            - name: RUN_ID
              value: "example-run-001"
            - name: REPORT_CONTRACT_VERSION
              value: "1"
            - name: JOB_TIMEOUT_SECONDS
              value: "120"
          resources:
            requests:
              cpu: "250m"
              memory: "256Mi"
            limits:
              cpu: "1"
              memory: "1Gi"

Apply or upgrade each component independently:

kubectl apply -f deploy/kubernetes/planner.yaml
kubectl apply -f deploy/kubernetes/report-job.yaml

A planner release does not require rebuilding the job image:

kubectl set image \
  deployment/compound-planner \
  planner=registry.example.com/compound-planner:1.5.0

Likewise, a new job version can be launched without changing the planner deployment. For each real run, generate a unique Kubernetes Job name and pass the run_id through the job input rather than reusing report-job-example.

The contract version is the compatibility boundary between those releases. Deploying planner 1.5.0 and report job 2.2.0 independently is safe only while both support the configured report contract version.

For a broader deployment breakdown, see the anatomy of an AI job deployment. As throughput grows, queues and component boundaries also help address the common bottlenecks in scaling AI agents.

Choose the control plane: code, agent routing, or a hybrid

Your control plane decides what runs next.

A programmatic orchestrator uses explicit branches:

def route(action: str) -> str:
    allowed = {
        "retrieve_sources": "retrieval_job",
        "analyze_records": "analysis_job",
        "render_report": "render_job",
    }

    if action not in allowed:
        raise PermissionError(f"Action not allowed: {action}")

    return allowed[action]

An agent-driven router asks a model to choose the next tool. This is flexible, but it moves control into probabilistic output. Databricks notes that compound-system control can live in code or be assigned to a language model, with a potential reliability trade-off for model-driven control.

A hybrid works well for multi-tool production systems:

1. Code defines the allowlist and validates arguments.
2. The agent selects a tool from that allowlist.
3. Code checks permissions, state, and budget.
4. The orchestrator dispatches the job.
5. Code validates the result before another model call.

sequenceDiagram
    participant T as Trigger
    participant O as Orchestrator
    participant A as Planner agent
    participant J as Isolated job
    participant V as Validator

    T->>O: Submit request
    O->>A: Provide state and allowed tools
    A-->>O: Return typed tool selection
    O->>O: Validate policy and arguments
    O->>J: Execute bounded job
    J-->>O: Return artifact reference
    O->>V: Validate artifact
    V-->>O: Accept or reject
    O-->>T: Return result

Control mode	Latency	Cost control	Flexibility	Failure handling
Code	Predictable routing	Explicit budgets	Low	Deterministic retries
Agent	Additional model decisions	Harder to bound	High	Requires output validation
Hybrid	Depends on selected path	Policy enforced in code	High within an allowlist	Explicit recovery with flexible selection

Keep permissions, retry limits, and critical workflow transitions in code. Let the model choose among safe, typed capabilities.

Isolate execution without creating an operations monolith

Code execution should not share the orchestrator’s process, filesystem, or long-lived credentials.

Create a short-lived E2B sandbox for each bounded job. In the illustrative adapter above, the intended lifecycle is:

1. Sandbox.create(timeout=120, ...) supplies an example sandbox-level timeout value.
2. commands.run(..., timeout=30) supplies a separate example command-level timeout value.
3. envs passes only RUN_ID and the staging output path when the sandbox is created.
4. files.write transfers the approved input and the static job program.
5. The adapter expects commands.run to return a command result with an exit_code field and checks it before artifact retrieval.
6. files.read retrieves the artifact from /tmp/staging/report.json.
7. validate_artifact checks the artifact outside the sandbox.
8. put_object promotes the accepted artifact to durable storage.
9. sandbox.kill() runs in a finally block so the adapter attempts cleanup after command, transfer, or validation failures.

The numerical timeout values are examples, not recommended production defaults. Their units, exact scope, enforcement, and interaction with sandbox termination must be confirmed against the E2B Python SDK v1.5.1 source and tested with the installed service and SDK. Likewise, verify exit_code, files.write, files.read, commands.run, Sandbox.create, and sandbox.kill() rather than treating them as stable fields or methods across E2B releases.

The Groq call and S3 promotion are also version-sensitive. Confirm the model response shape against the groq-python v0.31.1 client reference, and confirm put_object arguments against the versioned boto3 1.40.21 S3 put_object reference before adopting the sample.

The Groq, AWS, and other long-lived credentials are not passed into the E2B environment. If an isolated task needs a credential, provide a narrowly scoped, short-lived token instead of copying the orchestrator’s full credential set.

The durable layer should retain:

• Workflow ID and current state.
• Idempotency key.
• Tool selection and validated arguments.
• Artifact references.
• Retry and failure state.
• Trace context.

The ephemeral E2B layer should contain:

• Temporary input files such as /tmp/input/report-input.json.
• Job-specific dependencies.
• Short-lived credentials when strictly required.
• Generated or static job code.
• Staging output such as /tmp/staging/report.json.

Do not store orchestration state only inside the sandbox. If E2B terminates the environment because its configured lifetime or another service limit is reached, the orchestrator still needs enough information to decide whether to retry, resume, or stop.

Treat partial artifacts as untrusted. Keep them in the sandbox staging path until the command reports successful completion according to the verified SDK result contract. Retrieve and validate their schema, type, expected run_id, and ownership before writing them to a durable URI such as:

s3://compound-agent-artifacts/reports/<run_id>/report.json

If validation or promotion fails, do not report the run as completed. Record the failure outside E2B, retain the idempotency key, and let the orchestrator decide whether promotion can be retried without rerunning the model or sandbox job.

For a deeper execution pattern, see this guide to an isolated E2B agent sandbox. If your orchestration uses checkpoints, the same separation applies to durable LangGraph production jobs.

Test and observe every agent, tool, and job independently

Use one trace ID across the entire request:

import json
import logging
import uuid

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("compound-agent")


def start_run() -> str:
    trace_id = str(uuid.uuid4())
    logger.info(json.dumps({
        "event": "run_started",
        "trace_id": trace_id,
    }))
    return trace_id

Attach that ID to model calls, planner decisions, tool invocations, sandbox jobs, and artifacts. Logs should identify:

• Component and contract version.
• Selected tool.
• Job and trace IDs.
• Latency and model token usage.
• Retries and exit state.
• Applied permission profile.
• Artifact reference.
• Failure category.

Test each layer separately.

Contract tests should reject missing fields and incompatible versions. Tool tests should verify argument validation. E2B sandbox integration tests should determine the actual timeout behavior, cleanup behavior, available environment variables, filesystem methods, and command-result fields using the same pinned SDK version as production. Artifact tests should reject staging output with the wrong run_id or schema. Job tests should use fixed fixtures. Planner evaluations should check whether the selected tool is valid for the request.

Version-sensitive integration tests should fail explicitly when the installed Groq, E2B, or boto3 version differs from the version validated by the application. Review the corresponding versioned documentation and rerun those tests before changing a dependency pin.

Then run end-to-end evaluations for task completion and artifact quality.

This separation follows Databricks’ guidance that compound systems can be evaluated both end to end and by component. In RAG, retrieval and generation often need distinct evaluation because a poor answer can originate in either stage.

A repeatable dataset and regression suite make those checks useful across releases. The production guide to reproducible AI agent evaluation covers that workflow in more depth.

Conclusion

A compound AI system stays manageable when models, agents, tools, and jobs have explicit boundaries. Put Groq behind a model adapter. Let the planner choose only typed, allowlisted tools. Run risky or untrusted work in short-lived E2B environments. Keep durable workflow state outside those sandboxes.

The illustrative pipeline in this guide demonstrates the model and sandbox boundaries, artifact validation, and credential separation. Its dependency versions, timeout values, SDK calls, and return-field expectations must be verified against the linked versioned documentation and integration-tested before production use. It does not itself implement the planner or typed-tool-selection boundary; that layer must be added and evaluated separately when the orchestrator needs model-driven routing.

Most importantly, deploy and test each job independently. A failure should identify a component, contract, and trace—not disappear inside one agent loop.

A managed agent runtime such as HollowHost can provide the scheduling, isolation, secrets, and execution lifecycle around this pattern. The architecture remains the same: small deployable units, durable state, narrow permissions, and observable handoffs.