Running Evaluations

SMOLTRACE evaluations can be driven from the command line (smoltrace-eval) or programmatically via the Python API. This guide covers both, plus advanced options. For the complete flag list, see the CLI Reference.

Basic Command

smoltrace-eval \
  --model mistral/mistral-small-latest \
  --provider litellm \
  --agent-type both \
  --enable-otel

• --model — the model ID (required).
• --provider — one of litellm, inference, transformers, ollama (default litellm). See Model Providers.
• --agent-type — tool, code, or both (default both).
• --enable-otel — enable OpenTelemetry tracing and metrics.

Example Output

A tool-agent run prints a console summary:

TOOL AGENT SUMMARY
Total: 5, Success: 4/5 (80.0%)
Tool called: 100%, Correct tool: 80%, Avg steps: 2.6

[SUCCESS] Evaluation complete! Results pushed to HuggingFace Hub.
  Results: https://huggingface.co/datasets/{username}/smoltrace-results-20250125_143000
  Traces: https://huggingface.co/datasets/{username}/smoltrace-traces-20250125_143000
  Metrics: https://huggingface.co/datasets/{username}/smoltrace-metrics-20250125_143000
  Leaderboard: https://huggingface.co/datasets/{username}/smoltrace-leaderboard

Advanced Usage

Model Generation Parameters

Generation settings are sent to the model on every call (0.2.11). Unset means the provider's default.

# Sampling settings
smoltrace-eval \
  --model openai/gpt-4.1-nano \
  --provider litellm \
  --temperature 0.2 --top-p 0.9 --max-new-tokens 2048 \
  --enable-otel

# Reasoning effort, for models with a reasoning control
smoltrace-eval \
  --model anthropic/claude-sonnet-4-5 \
  --provider litellm \
  --reasoning-effort low

# Thinking switch of an open-weight chat-template model (OpenAI-compatible endpoint, Ollama, transformers)
smoltrace-eval \
  --model qwen3:8b \
  --provider ollama \
  --top-k 20 --enable-thinking false

A setting the provider cannot apply is never dropped silently: the run prints [GENERATION] not applied for <provider>: <setting> (<reason>) and the leaderboard row's generation_settings records what was requested, applied and not applied. The per-provider mapping is in SPEC v0.2.11.

--model-args key=value ... is something else: smolagents hands those values to the agent as task variables (additional_args). They do not reach the model.

MCP Tools Integration

Run evaluations with tools from one or more MCP servers. Repeat the flag and optionally prefix each URL with a stable server name to avoid tool-name collisions:

# Start your MCP servers, then:
smoltrace-eval \
  --model openai/gpt-4 \
  --provider litellm \
  --agent-type code \
  --mcp-server-url search=http://localhost:8000/sse \
  --mcp-server-url files=http://localhost:8001/mcp/ \
  --enable-otel

Transport is detected from the endpoint (/sse or streamable HTTP /mcp/), or can be selected explicitly with --mcp-transport.

Custom Prompt Templates

Supply a prompt configuration YAML with --prompt-yml:

smoltrace-eval \
  --model openai/gpt-4 \
  --provider litellm \
  --agent-type code \
  --prompt-yml smoltrace/prompts/code_agent.yaml \
  --enable-otel

Built-in templates in smoltrace/prompts/:

• code_agent.yaml — standard code agent prompts
• structured_code_agent.yaml — structured JSON output format
• toolcalling_agent.yaml — tool calling agent prompts

Additional Python Imports for CodeAgent

Allow the CodeAgent to import extra modules with --additional-imports:

smoltrace-eval \
  --model openai/gpt-4 \
  --provider litellm \
  --agent-type code \
  --additional-imports pandas numpy matplotlib \
  --enable-otel

Note

Make sure the specified modules are installed in your environment.

Parallel Execution

Speed up evaluations with --parallel-workers — 10-50x faster for API models, where operations are I/O bound:

smoltrace-eval \
  --model openai/gpt-4.1-nano \
  --provider litellm \
  --parallel-workers 8 \
  --agent-type both \
  --enable-otel

Warning

Use --parallel-workers 1 (default) for GPU models to avoid memory issues.

Python API

from smoltrace.core import run_evaluation
import os

# Simple usage — everything is auto-configured
all_results, trace_data, metric_data, dataset_used, run_id = run_evaluation(
    model="openai/gpt-4",
    provider="litellm",
    agent_type="both",
    difficulty="easy",
    enable_otel=True,
    enable_gpu_metrics=False,  # False for API models (default), True for local models
    hf_token=os.getenv("HF_TOKEN"),
)

print(f"Evaluation complete! Run ID: {run_id}")
print(f"Total tests: {len(all_results.get('tool', []) + all_results.get('code', []))}")
print(f"Traces collected: {len(trace_data)}")

Results are automatically pushed to the HuggingFace Hub as the four smoltrace-* datasets.

Advanced: MCP Tools, Custom Prompts, and Additional Imports

from smoltrace.core import run_evaluation
from smoltrace.utils import load_prompt_config
import os

prompt_config = load_prompt_config("smoltrace/prompts/code_agent.yaml")

all_results, trace_data, metric_data, dataset_used, run_id = run_evaluation(
    model_name="openai/gpt-4",
    agent_types=["code"],
    test_subset="medium",
    dataset_name="kshitijthakkar/smoltrace-tasks",
    split="train",
    enable_otel=True,
    verbose=True,
    debug=False,
    provider="litellm",
    prompt_config=prompt_config,
    mcp_server_url="http://localhost:8000/sse",
    additional_authorized_imports=["pandas", "numpy", "matplotlib", "json"],
    enable_gpu_metrics=False,
)

Advanced: Manual Dataset Management

For full control over dataset creation and pushing, compose the lower-level helpers:

from smoltrace.core import run_evaluation
from smoltrace.utils import (
    get_hf_user_info,
    generate_dataset_names,
    push_results_to_hf,
    compute_leaderboard_row,
    update_leaderboard,
)
import os

hf_token = os.getenv("HF_TOKEN")
user_info = get_hf_user_info(hf_token)
username = user_info["username"]

results_repo, traces_repo, metrics_repo, leaderboard_repo = generate_dataset_names(username)

all_results, trace_data, metric_data, dataset_used, run_id = run_evaluation(
    model="meta-llama/Llama-3.1-8B",
    provider="transformers",
    agent_type="both",
    enable_otel=True,
    enable_gpu_metrics=True,  # Auto-enabled for local models
    hf_token=hf_token,
)

push_results_to_hf(
    all_results=all_results,
    trace_data=trace_data,
    metric_data=metric_data,
    results_repo=results_repo,
    traces_repo=traces_repo,
    metrics_repo=metrics_repo,
    model_name="meta-llama/Llama-3.1-8B",
    hf_token=hf_token,
    private=False,
    run_id=run_id,
)

leaderboard_row = compute_leaderboard_row(
    model_name="meta-llama/Llama-3.1-8B",
    all_results=all_results,
    trace_data=trace_data,
    metric_data=metric_data,
    dataset_used=dataset_used,
    results_dataset=results_repo,
    traces_dataset=traces_repo,
    metrics_dataset=metrics_repo,
    agent_type="both",
    run_id=run_id,
    provider="transformers",
)

update_leaderboard(leaderboard_repo, leaderboard_row, hf_token)

See the Python API reference for the full set of functions.

Common Use Cases

Easy tasks onlyCompare tool vs code agentsGPU model with metricsPublic results

smoltrace-eval \
  --model mistral/mistral-small-latest \
  --provider litellm \
  --difficulty easy \
  --output-format json

smoltrace-eval --model openai/gpt-4 --provider litellm --agent-type tool
smoltrace-eval --model openai/gpt-4 --provider litellm --agent-type code

smoltrace-eval \
  --model meta-llama/Llama-3.1-8B \
  --provider transformers \
  --agent-type both \
  --enable-otel

smoltrace-eval \
  --model your-model \
  --provider litellm \
  --output-format hub \
  --public

Hub datasets are private by default in 0.1.0. Public publication requires the explicit --public flag.