smevals - a small eval suite for evaluating models, prompts, and harnesses

Smevals is a small, self-contained evaluation suite designed to test and compare the capabilities of different artificial intelligence models, prompts, and agent harnesses. Developed alongside Jesse Vincent's Prime Radiant applied AI research lab, the tool allows users to run targeted evaluations and grade the results.

An evaluation suite is structured as a directory containing YAML files made up of individual tasks, which represent specific challenges like writing a haiku or generating a vector graphic. Users execute these tasks against one or more configurations, which specify the models or system parameters to test. The tool keeps execution and grading completely separate. Runs record what happens during execution, while graders assess the outcomes against defined checks that range from simple string matching to custom scripts and model-based evaluations.

Users can interact with the tool by instructing a coding agent to read the documentation, building an eval directory, and executing commands to run tests, grade outputs, and view results. The software supports launching a local web server to explore the findings or building a static HTML report for hosting anywhere. This framework represents the third iteration of a multi-year effort to create a streamlined, effective approach for evaluating generative artificial intelligence.