Quickstart

We’re going to walk through setting up Evalite in an existing project.

1. Install evalite and vitest.

   pnpm add -D evalite vitest

2. Add an eval:dev script to your package.json:

   {
     "scripts": {
       "eval:dev": "evalite watch"
     }
   }

3. Create your first eval:

   my-eval.eval.ts

   import { evalite } from "evalite";
   import { exactMatch } from "evalite/scorers";
   
   evalite("My Eval", {
     // An array of test data
     // - TODO: Replace with your test data
     data: [{ input: "Hello", expected: "Hello World!" }],
     // The task to perform
     // - TODO: Replace with your LLM call
     task: async (input) => {
       return input + " World!";
     },
     // The scoring methods for the eval
     scorers: [
       {
         scorer: ({ output, expected }) =>
           exactMatch({ actual: output, expected }),
       },
     ],
   });

   Tip

   .eval.ts is the extension Evalite looks for when scanning for evals.

4. Run pnpm run eval:dev.

   pnpm run eval:dev

   This runs evalite, which runs the evals:

   • Runs the data function to get the test data
   • Runs the task function on each test data
   • Scores the output of the task function using the scorers
   • Saves the results to a sqlite database in node_modules/.evalite

   It then:

   • Shows a UI for viewing the traces, scores, inputs and outputs at http://localhost:3006.
   • If you only ran one eval, it also shows a table summarizing the eval in the terminal.

5. Open http://localhost:3006 in your browser to view the results of the eval.