I shipped a Weights & Biases sink for robot evals. Here is how it works.

I just published inspect-robots-wandb v0.1.0, a Weights & Biases logging sink for Inspect Robots, the open-source evaluation framework from Robocurve. This post covers what it does, how it plugs in, and the few decisions that mattered in a package this small.

What Inspect Robots is

Inspect Robots is "Inspect AI for robotics". It evaluates vision-language-action policies on real and simulated robots. You plug in a policy (the model) and an embodiment (the robot or simulator), pick a task, and the framework runs closed-loop trials, scores them, and writes an immutable evaluation log.

Think of a driving test. The policy is the student driver, the embodiment is the car, and the task is the test route. The JSON log is the examiner's signed score sheet: complete, permanent, and stored on your machine.

The missing league table

A signed score sheet is the right primitive for one evaluation. Once you are running dozens of them across checkpoints, seeds, and robots, you want every result in one place, side by side, with charts you can share with a team. That is what Weights & Biases is for, and that is the gap this plugin fills.

Using it

pip install inspect-robots-wandb
wandb login

The package registers a wandb sink in the inspect_robots.sinks entry-point group. After installing it, the sink shows up in inspect-robots list with no change to the core. From Python:

from inspect_robots import eval
from inspect_robots.logging import JsonLogSink
from inspect_robots_wandb import WandbSink

logs = eval(
    "cubepick-reach",
    "scripted",
    "cubepick",
    sinks=[
        JsonLogSink("logs"),
        WandbSink(project="robot-evals", tags=("cubepick", "nightly")),
    ],
)

What gets logged

WandbSink creates one W&B run per evaluation. The full evaluation specification is stored as run config. When the evaluation ends, the run gets:

• final status, scene count, trial count, and errored trials,
• total steps and duration in seconds,
• every aggregate result metric, under a metric/ prefix.

The same sink instance can be reused by eval_set(), and each task gets its own run. Options mirror W&B's: project, entity, name, group, tags, mode (including offline and disabled), and dir.

Design choices

The core stays NumPy-only. The maintainers' bar is a dependency-light core, so W&B support lives in its own package. The wandb SDK is imported lazily, only when a run is actually requested, and a missing SDK raises a clear install hint rather than an obscure import error.

Run lifecycle is explicit. If a new evaluation starts while a previous run is still open, the old run is finished with a failure exit code instead of being silently abandoned. On a normal end, the exit code reflects the evaluation status, and the run is finished even if logging itself raises.

Aborted evaluations get cleaned up. If an evaluation dies with an exception, the sink logs eval/status: error with the exception text and finishes the run with exit code 1. That relies on a small optional hook, on_eval_error, which I added to the core in a separate PR. With an older core the sink still works, but an aborted run is left for the W&B SDK to close when the process exits.

Tests need no network. The test suite replaces the W&B SDK with an in-process double, so it runs without an account or a connection, and the project holds 100% coverage with strict mypy and ruff. Releases go out through PyPI trusted publishing, so there are no API tokens to leak.

From PR #434 to a package

This started as PR #434 inside the main repository. The review was kind ("well put together"), and the maintainers asked for W&B support to live outside the repo so the core would not take on the dependency. That was the right call. The work split into three pieces: this package, the core hook (PR #550), and a community-plugins section in the docs (PR #551).

Status of the related PRs

I would rather be exact than optimistic, so here is where things stand as of Oct 4, 2026.

• The package itself is published: PyPI and source.
• PR #550 (the on_eval_error hook) is open.
• PR #551 (docs) is open.
• PR #434 is closed by agreement, superseded by the split above.