[READ-ONLY] Mirror of https://github.com/vitest-dev/vitest. Next generation testing framework powered by Vite.
vitest.dev
test
testing-tools
vite
docs: add a guide for the custom benchmark provider
Vladimir Sheremet
(Jul 22, 2026, 3:19 PM +0200)
d14b73d8
96ee499b
+102
-6
+4
docs/.vitepress/config.ts
+4
docs/.vitepress/config.ts
+8
-1
docs/config/benchmark.md
+8
-1
docs/config/benchmark.md
···
46
46
47
47
Include the `samples` array of per-iteration timings on every benchmark result. Disabled by default to reduce memory usage; enable when a custom reporter or API consumer needs the raw samples.
48
48
49
+
## benchmark.provider
50
+
51
+
- **Type:** `string`
52
+
- **Default:** `undefined` (uses the built-in provider)
53
+
54
+
The benchmark provider that executes registered benchmarks and returns their results. Set this to a module path whose default export implements `BenchmarkProvider`. Relative paths are resolved from the project root.
55
+
56
+
See the [Custom Benchmark Provider](/guide/advanced/benchmark-provider) guide for setup instructions and the provider API.
49
57
50
58
## benchmark.suppressExportGetterWarnings
51
59
···
53
61
- **Default:** `false`
54
62
55
63
Suppress the warning printed when a benchmark accesses module export getters too many times. Vitest tracks getter access during benchmark runs because Vite's module runner wraps every export in a getter, and excessive access can dominate the measurement (see [Module Runner Overhead](/guide/benchmarking#module-runner-overhead)). Enable this when you've intentionally accepted the overhead, or when the warning is noisy for benchmarks where the getter cost is negligible.
56
-
+7
-5
docs/guide/benchmarking.md
+7
-5
docs/guide/benchmarking.md
···
4
4
5
5
# Benchmarking
6
6
7
-
Vitest lets you write benchmarks alongside your tests using the `bench` fixture from the [test context](/guide/test-context). Benchmarks are powered by [Tinybench](https://github.com/tinylibs/tinybench) and are defined inside regular `test()` calls, giving you access to the full power of Vitest's test runner: retries, lifecycle hooks, filtering, and assertions.
7
+
Vitest lets you write benchmarks alongside your tests using the `bench` fixture from the [test context](/guide/test-context). The built-in benchmark provider is powered by [Tinybench](https://github.com/tinylibs/tinybench), and benchmarks are defined inside regular `test()` calls, giving you access to the full power of Vitest's test runner: retries, lifecycle hooks, filtering, and assertions.
8
8
9
9
## Defining a Benchmark
10
10
···
23
23
The `bench()` function registers a benchmark without executing it. Calling `.run()` runs the benchmark and returns the result. After the test completes, Vitest prints a single-row version of the [comparison table](#comparing-benchmarks) (ops/sec, mean time, percentiles, etc.), so you get the same output for a one-off benchmark as you do for `bench.compare()`.
24
24
25
25
::: warning
26
-
The `bench` fixture is only available in files matched by [`benchmark.include`](/config/#benchmark-include) (default: `**/*.{bench,benchmark}.?(c|m)[jt]s?(x)`). Using `{ bench }` inside a regular test file will throw an error.
26
+
The `bench` fixture is only available in files matched by [`benchmark.include`](/config/benchmark#benchmark-include) (default: `**/*.{bench,benchmark}.?(c|m)[jt]s?(x)`). Using `{ bench }` inside a regular test file will throw an error.
27
27
28
28
Whether a file participates in the benchmark run is decided by the filename, not by whether the test uses the `bench` fixture. Renaming `parser.test.ts` to `parser.bench.ts` (or adjusting `benchmark.include`) is what moves it into the benchmark project.
29
29
:::
30
30
31
31
## Running Benchmarks
32
32
33
-
Benchmark files are matched by [`benchmark.include`](/config/#benchmark-include) (default: `**/*.{bench,benchmark}.?(c|m)[jt]s?(x)`) and run in their own project, separate from your regular tests. There are three ways to run them, depending on whether you want to skip them, run them alongside tests, or run them on their own.
33
+
Benchmark files are matched by [`benchmark.include`](/config/benchmark#benchmark-include) (default: `**/*.{bench,benchmark}.?(c|m)[jt]s?(x)`) and run in their own project, separate from your regular tests. There are three ways to run them, depending on whether you want to skip them, run them alongside tests, or run them on their own.
34
34
35
35
### `vitest` (default)
36
36
37
-
Without [`benchmark.enabled`](/config/#benchmark-enabled), the `vitest` command only runs regular tests. Benchmark files are ignored entirely. This is the default and the right choice for day-to-day development, since benchmarks are slow and noisy and shouldn't run on every save.
37
+
Without [`benchmark.enabled`](/config/benchmark#benchmark-enabled), the `vitest` command only runs regular tests. Benchmark files are ignored entirely. This is the default and the right choice for day-to-day development, since benchmarks are slow and noisy and shouldn't run on every save.
38
38
39
39
### `vitest` with `benchmark.enabled`
40
40
···
71
71
# only benchmarks whose test name matches "JSON"
72
72
vitest bench -t JSON
73
73
```
74
+
75
+
To execute benchmarks with another benchmarking engine or execution strategy, see the [Custom Benchmark Provider](/guide/advanced/benchmark-provider) guide.
74
76
75
77
## Comparing Benchmarks
76
78
···
330
332
331
333
Benchmarks are inherently flaky: CPU load, thermal throttling, GC pressure, and background processes all affect results. Vitest takes several steps to minimize this noise:
332
334
333
-
- **Separate project**: Benchmark files are grouped into their own project based on the [`benchmark.include`](/config/#benchmark-include) pattern. The `bench` fixture is only exposed in files matched by that pattern. Using it inside a regular test file will throw an error.
335
+
- **Separate project**: Benchmark files are grouped into their own project based on the [`benchmark.include`](/config/benchmark#benchmark-include) pattern. The `bench` fixture is only exposed in files matched by that pattern. Using it inside a regular test file will throw an error.
334
336
- **No concurrency**: Tests within a benchmark file always run sequentially. Benchmark files themselves also run one at a time, never in parallel. This prevents benchmarks from interfering with each other.
335
337
336
338
To further improve stability:
+83
docs/guide/advanced/benchmark-provider.md
+83
docs/guide/advanced/benchmark-provider.md
···
1
+
# Custom Benchmark Provider <Version type="experimental">5.0.0</Version> <Badge type="danger">advanced</Badge> {#custom-benchmark-provider}
2
+
3
+
::: warning
4
+
This is an advanced, experimental API. If you only need to run benchmarks with Vitest's built-in provider, read the [Benchmarking](/guide/benchmarking) guide instead.
5
+
:::
6
+
7
+
Vitest uses a benchmark provider to execute the functions registered with `bench` and convert their measurements into results that Vitest can report. The built-in provider uses [Tinybench](https://github.com/tinylibs/tinybench), but you can replace it to use another benchmarking engine or execution strategy.
8
+
9
+
## Setup
10
+
11
+
Set [`benchmark.provider`](/config/benchmark#benchmark-provider) to the path of your provider module. Relative paths are resolved from the project root.
12
+
13
+
```ts [vitest.config.ts]
14
+
import { defineConfig } from 'vitest/config'
15
+
16
+
export default defineConfig({
17
+
test: {
18
+
benchmark: {
19
+
provider: './benchmark-provider.ts',
20
+
},
21
+
},
22
+
})
23
+
```
24
+
25
+
The module must has a default export with an object that implements `BenchmarkProvider`. This example wraps Tinybench to demonstrate how registrations and results flow through a provider. If you use Tinybench in your provider, add it as a direct dependency of your project.
26
+
27
+
```ts [benchmark-provider.ts]
28
+
import type { BenchmarkProvider } from 'vitest'
29
+
import { Bench } from 'tinybench'
30
+
31
+
const provider = {
32
+
async run({ test, config, registrations, options }) {
33
+
const bench = new Bench({
34
+
signal: test.context.signal,
35
+
retainSamples: config.retainSamples,
36
+
...options,
37
+
})
38
+
39
+
for (const { name, fn, fnOpts } of registrations) {
40
+
bench.add(name, fn, fnOpts)
41
+
}
42
+
43
+
await bench.run()
44
+
45
+
return bench.tasks.map((task) => {
46
+
const result = task.result
47
+
48
+
if (result.state === 'errored') {
49
+
throw result.error
50
+
}
51
+
if (result.state !== 'completed') {
52
+
throw new Error(`Benchmark "${task.name}" ended in the "${result.state}" state`)
53
+
}
54
+
55
+
return {
56
+
...result,
57
+
name: task.name,
58
+
}
59
+
})
60
+
},
61
+
} satisfies BenchmarkProvider
62
+
63
+
export default provider
64
+
```
65
+
66
+
## Provider API
67
+
68
+
Vitest calls `provider.run(group)` when a registration's `.run()` method is called, or once for all runnable registrations passed to `bench.compare()`. The `group` contains:
69
+
70
+
- `test`: the test that registered the benchmarks. `test.context.signal` is aborted when the run is cancelled.
71
+
- `config`: the resolved benchmark configuration for the current project.
72
+
- `registrations`: runnable benchmarks in registration order. Every registration contains `name`, `fn`, and optional `fnOpts` for lifecycle hooks, cancellation, async behavior, and sample retention.
73
+
- `options`: benchmark run options passed to `.run()` or `bench.compare()`, if any.
74
+
75
+
The provider is responsible for honoring the run and registration options and for running every benchmark function and its `beforeAll`, `beforeEach`, `afterEach`, and `afterAll` hooks according to the benchmarking engine's lifecycle. If execution fails, throw the error to fail the test.
76
+
77
+
`run` must resolve to one `BenchResult` for every runnable registration. Results are matched to registrations by `name` and are the source for `.run()` return values, comparison tables, reporters, and saved benchmark results. A custom engine must convert its measurements into the Tinybench-compatible `BenchResult` shape exported by `vitest`.
78
+
79
+
Registrations created by `bench.from()` are loaded by Vitest and are not passed to the provider.
80
+
81
+
## Provider Lifetime
82
+
83
+
Vitest imports the provider module on first use and caches its default export for the lifetime of the worker. The API does not have separate setup or teardown hooks; keep worker-scoped state on the provider object when needed.