Skip to content
Evalite
These are the docs for the beta version of Evalite. Install with pnpm add evalite@beta

defineConfig()

Type-safe helper for defining Evalite configuration in evalite.config.ts.

Signature

defineConfig(config: {
  storage?: () => Evalite.Storage | Promise<Evalite.Storage>;
  server?: {
    port?: number;
  };
  scoreThreshold?: number;
  hideTable?: boolean;
  testTimeout?: number;
  maxConcurrency?: number;
  trialCount?: number;
  setupFiles?: string[];
  cache?: boolean;
  viteConfig?: ViteUserConfig;
  forceRerunTriggers?: string[];
}): Evalite.Config

Usage

Create an evalite.config.ts file in your project root:

evalite.config.ts
import { defineConfig } from "evalite/config";


export default defineConfig({
  testTimeout: 60000,
  maxConcurrency: 100,
  scoreThreshold: 80,
});

Options

storage

Type: () => Evalite.Storage | Promise<Evalite.Storage>

Default: In-memory storage (data lost after process exits)

Factory function to create a custom storage backend. Use createSqliteStorage() for persistent storage.

import { defineConfig } from "evalite/config";
import { createSqliteStorage } from "evalite/sqlite-storage";


export default defineConfig({
  storage: () => createSqliteStorage("./custom.db"),
});

See Storage for more details.

server.port

Type: number

Default: 3006

Port for the Evalite UI server.

export default defineConfig({
  server: {
    port: 8080,
  },
});

scoreThreshold

Type: number (0-100)

Default: 100

Minimum average score threshold. If the average score falls below this threshold, the process will exit with code 1.

export default defineConfig({
  scoreThreshold: 80, // Fail if average score < 80
});

Useful for CI/CD pipelines where you want to fail the build if evals don’t meet a quality threshold.

hideTable

Type: boolean

Default: false

Hide the detailed results table in terminal output. Keeps the score summary but removes the detailed table.

export default defineConfig({
  hideTable: true,
});

Useful when debugging with console.log to see logs more clearly.

testTimeout

Type: number (milliseconds)

Default: 30000 (30 seconds)

Maximum time a test can run before timing out.

export default defineConfig({
  testTimeout: 60000, // 60 seconds
});

maxConcurrency

Type: number

Default: 5

Maximum number of test cases to run in parallel.

export default defineConfig({
  maxConcurrency: 100, // Run up to 100 tests in parallel
});

Useful for optimizing performance and managing API rate limits.

trialCount

Type: number

Default: 1

Number of times to run each test case. Useful for measuring variance in non-deterministic evaluations.

export default defineConfig({
  trialCount: 3, // Run each test case 3 times
});

Can also be set per-eval in the evalite() function.

setupFiles

Type: string[]

Default: []

Array of file paths to run before tests. Useful for loading custom environment setup.

export default defineConfig({
  setupFiles: ["./custom-setup.ts"],
});

Note: .env files are loaded automatically via dotenv/config - no need to configure them here.

cache

Type: boolean

Default: true

Enable or disable caching of AI SDK model outputs. See Vercel AI SDK caching for details.

export default defineConfig({
  cache: false, // Disable cache entirely
});

viteConfig

Type: ViteUserConfig

Pass-through Vite/Vitest configuration options. This allows you to import and use your existing vite.config.ts explicitly.

import { defineConfig } from "evalite/config";
import viteConfig from "./vite.config.ts";


export default defineConfig({
  viteConfig: viteConfig,
});

Note: testTimeout, maxConcurrency, and setupFiles must be configured at the root level of evalite.config.ts, not in viteConfig.test.

forceRerunTriggers

Type: string[]

Default: []

Extra file globs that trigger eval reruns in watch mode. This maps onto Vitest’s forceRerunTriggers option.

export default defineConfig({
  forceRerunTriggers: [
    "src/**/*.ts", // helper / model code
    "prompts/**/*", // prompt templates
    "data/**/*.json", // test data
  ],
});

Useful when your evals depend on files that Vitest can’t automatically detect as dependencies (e.g., prompt templates, external data files).

Complete Example

evalite.config.ts
import { defineConfig } from "evalite/config";
import { createSqliteStorage } from "evalite/sqlite-storage";


export default defineConfig({
  // Persistent storage
  storage: () => createSqliteStorage("./evalite.db"),


  // Server configuration
  server: {
    port: 3006,
  },


  // Quality threshold
  scoreThreshold: 75,


  // Test execution
  testTimeout: 60000,
  maxConcurrency: 50,
  trialCount: 1,


  // UI preferences
  hideTable: false,


  // Setup
  setupFiles: ["./test-setup.ts"],


  // Caching
  cache: true,


  // Watch mode triggers
  forceRerunTriggers: ["src/**/*.ts", "prompts/**/*"],
});

Supported File Names

Evalite will look for configuration in these files (in order):

  • evalite.config.ts
  • evalite.config.mts
  • evalite.config.js
  • evalite.config.mjs

Vitest Integration

Since Evalite is built on Vitest, you can also use vitest.config.ts for backward compatibility. However, evalite.config.ts is the recommended approach and takes precedence when both files exist.

See Also