ct run monitor
Score a finished eval log with more monitors, writing a monitored copy
Usage
ct run monitor [OPTIONS] SOURCE
Arguments
| Argument | Type | Required |
|---|---|---|
SOURCE | text | Yes |
Options
| Option | Description |
|---|---|
-m, --monitors TEXT | The monitors, as YAML: a list of packaged prompt names or mappings of Monitor fields, e.g. '[simple, {prompt: strong, model: judge, cot_level: cot, samples: 2}]'. model names a role bound with --model-role. simple asks about an action, trajectory about a whole run, so pick the one --kind is for. [required] |
--kind [action|trajectory] | Score every action before it ran, or the whole trajectory once. [default: action] |
--model-role TEXT | Bind a model role, as name=model or name=<JSON model spec> (multiple allowed). A monitor answers on the role its model names, monitor by default, falling back to trusted. |
-o, --save-dir PATH | Where to write a base log's monitored copy (default: ./data/monitoring); a monitored copy given as SOURCE is updated in place. |
--help | Show this message and exit. |