Configuration

Since Evalite is based on Vitest, you can configure eval behavior using Vitest’s configuration options. Each data point in your eval becomes a separate Vitest test case, which means all Vitest configuration options work with Evalite.

Evalite Configuration

You can configure Evalite-specific options using evalite.config.ts:

evalite.config.ts

import { defineConfig } from "evalite/config";

export default defineConfig({
  testTimeout: 60000, // 60 seconds
  maxConcurrency: 100, // Run up to 100 tests in parallel
  scoreThreshold: 80, // Fail if average score < 80
  hideTable: false,
  server: {
    port: 3006,
  },
});

Available Options

• testTimeout: Maximum time (in milliseconds) a test can run before timing out. Default is 30000ms (30 seconds).
• maxConcurrency: Maximum number of test cases to run in parallel. Default is 5.
• scoreThreshold: Minimum average score (0-100). Process exits with code 1 if average score falls below this threshold.
• hideTable: Hide the results table in terminal output. Default is false.
• server.port: Port for the Evalite UI server. Default is 3006.
• trialCount: Number of times to run each test case. Default is 1. Useful for measuring variance in non-deterministic evaluations.
• setupFiles: Array of file paths to run before tests (e.g., for loading environment variables).
• cache: Enable or disable caching of AI SDK model outputs. Default is true. See Vercel AI SDK for details.
• viteConfig: Pass through Vite/Vitest configuration options. This allows you to import and use your existing vite.config.ts explicitly.
• forceRerunTriggers: Extra file globs that trigger eval reruns in watch mode. This maps onto Vitest’s test.forceRerunTriggers option (globs resolved relative to the directory where you run Evalite). See Watching Additional Files.

Important Configuration Options

maxConcurrency

Control how many test cases run in parallel. Default is 5.

Configure in evalite.config.ts:

evalite.config.ts

import { defineConfig } from "evalite/config";

export default defineConfig({
  maxConcurrency: 100, // Run up to 100 tests in parallel
});

This is useful for optimizing performance and managing API rate limits.

testTimeout

Set the maximum time (in milliseconds) a test can run before timing out. Default is 30000ms in Evalite.

Configure in evalite.config.ts:

evalite.config.ts

import { defineConfig } from "evalite/config";

export default defineConfig({
  testTimeout: 60000, // 60 seconds
});

Passing Vitest Configuration

You can pass your existing Vitest configuration to Evalite using the viteConfig option.

evalite.config.ts

import viteConfig from "./vite.config.ts";
import { defineConfig } from "evalite/config";

export default defineConfig({
  viteConfig: viteConfig,
});

This allows you to reuse your existing Vitest configuration file and plugins if needed.

TypeScript Path Aliases

If you’re using TypeScript path aliases in your tsconfig.json, Evalite can’t read them by default.

For example, in a Next.js app, you might have:

tsconfig.json

{
  "compilerOptions": {
    "paths": {
      "@/*": ["./*"]
    }
  }
}

You’ll need to configure Evalite to resolve these paths - by passing the viteConfig option.

1. Install vite-tsconfig-paths:

   pnpm add -D vite-tsconfig-paths

2. Create a vite.config.ts file:

   vite.config.ts

   import { defineConfig } from "vite";
   import tsconfigPaths from "vite-tsconfig-paths";
   
   export default defineConfig({
     plugins: [tsconfigPaths()],
   });

3. Pass this config to Evalite:

   evalite.config.ts

   import viteConfig from "./vite.config.ts";
   import { defineConfig } from "evalite/config";
   
   export default defineConfig({
     viteConfig,
   });