Using DSPy to evaluate and improve Datasette Agent's SQL system prompts

The Datasette Agent uses system prompts to help users answer questions by executing read-only SQL queries. To improve the accuracy and efficiency of these interactions, this project uses the DSPy framework to evaluate and refine the agent's core instructions. The testing process involves a harness that runs the agent against a live, in-process Datasette instance, using an auto-generated gold-standard dataset to measure performance.

The research identified specific weaknesses in the agent's current prompting strategy. For instance, the system prompt previously instructed the agent to avoid calling a table description tool if it already possessed information. This led the model to guess column names incorrectly, resulting in frequent error-retry loops. The findings suggest that providing explicit column names in the initial schema listing or relaxing the restriction on using the description tool significantly improves performance.

By automating the evaluation process with DSPy, developers can systematically test different prompt variations using models like GPT-4.1 mini and nano. This approach allows for the identification of subtle logic errors that cause the agent to struggle with database queries. Refining these prompts is essential for creating more reliable AI agents that can accurately interpret and interact with structured data without unnecessary errors.