[READ-ONLY] Mirror of https://github.com/vitest-dev/vitest. Next generation testing framework powered by Vite.
vitest.dev
test
testing-tools
vite
feat(benchmark): add pluggable benchmark provider API (#10799)
Co-authored-by: Vladimir Sheremet <sleuths.slews0s@icloud.com>
authored by
Guillaume Lagrange
Vladimir Sheremet
and committed by
GitHub
(Jul 22, 2026, 3:54 PM +0200)
86c700ed
b19e5ece
+493
-128
+4
docs/.vitepress/config.ts
+4
docs/.vitepress/config.ts
+8
-1
docs/config/benchmark.md
+8
-1
docs/config/benchmark.md
···
46
46
47
47
Include the `samples` array of per-iteration timings on every benchmark result. Disabled by default to reduce memory usage; enable when a custom reporter or API consumer needs the raw samples.
48
48
49
+
## benchmark.provider
50
+
51
+
- **Type:** `string`
52
+
- **Default:** `undefined` (uses the built-in provider)
53
+
54
+
The benchmark provider that executes registered benchmarks and returns their results. Set this to a module path whose default export implements `BenchmarkProvider`. Relative paths are resolved from the project root.
55
+
56
+
See the [Custom Benchmark Provider](/guide/advanced/benchmark-provider) guide for setup instructions and the provider API.
49
57
50
58
## benchmark.suppressExportGetterWarnings
51
59
···
53
61
- **Default:** `false`
54
62
55
63
Suppress the warning printed when a benchmark accesses module export getters too many times. Vitest tracks getter access during benchmark runs because Vite's module runner wraps every export in a getter, and excessive access can dominate the measurement (see [Module Runner Overhead](/guide/benchmarking#module-runner-overhead)). Enable this when you've intentionally accepted the overhead, or when the warning is noisy for benchmarks where the getter cost is negligible.
56
-
+7
-5
docs/guide/benchmarking.md
+7
-5
docs/guide/benchmarking.md
···
4
4
5
5
# Benchmarking
6
6
7
-
Vitest lets you write benchmarks alongside your tests using the `bench` fixture from the [test context](/guide/test-context). Benchmarks are powered by [Tinybench](https://github.com/tinylibs/tinybench) and are defined inside regular `test()` calls, giving you access to the full power of Vitest's test runner: retries, lifecycle hooks, filtering, and assertions.
7
+
Vitest lets you write benchmarks alongside your tests using the `bench` fixture from the [test context](/guide/test-context). The built-in benchmark provider is powered by [Tinybench](https://github.com/tinylibs/tinybench), and benchmarks are defined inside regular `test()` calls, giving you access to the full power of Vitest's test runner: retries, lifecycle hooks, filtering, and assertions.
8
8
9
9
## Defining a Benchmark
10
10
···
23
23
The `bench()` function registers a benchmark without executing it. Calling `.run()` runs the benchmark and returns the result. After the test completes, Vitest prints a single-row version of the [comparison table](#comparing-benchmarks) (ops/sec, mean time, percentiles, etc.), so you get the same output for a one-off benchmark as you do for `bench.compare()`.
24
24
25
25
::: warning
26
-
The `bench` fixture is only available in files matched by [`benchmark.include`](/config/#benchmark-include) (default: `**/*.{bench,benchmark}.?(c|m)[jt]s?(x)`). Using `{ bench }` inside a regular test file will throw an error.
26
+
The `bench` fixture is only available in files matched by [`benchmark.include`](/config/benchmark#benchmark-include) (default: `**/*.{bench,benchmark}.?(c|m)[jt]s?(x)`). Using `{ bench }` inside a regular test file will throw an error.
27
27
28
28
Whether a file participates in the benchmark run is decided by the filename, not by whether the test uses the `bench` fixture. Renaming `parser.test.ts` to `parser.bench.ts` (or adjusting `benchmark.include`) is what moves it into the benchmark project.
29
29
:::
30
30
31
31
## Running Benchmarks
32
32
33
-
Benchmark files are matched by [`benchmark.include`](/config/#benchmark-include) (default: `**/*.{bench,benchmark}.?(c|m)[jt]s?(x)`) and run in their own project, separate from your regular tests. There are three ways to run them, depending on whether you want to skip them, run them alongside tests, or run them on their own.
33
+
Benchmark files are matched by [`benchmark.include`](/config/benchmark#benchmark-include) (default: `**/*.{bench,benchmark}.?(c|m)[jt]s?(x)`) and run in their own project, separate from your regular tests. There are three ways to run them, depending on whether you want to skip them, run them alongside tests, or run them on their own.
34
34
35
35
### `vitest` (default)
36
36
37
-
Without [`benchmark.enabled`](/config/#benchmark-enabled), the `vitest` command only runs regular tests. Benchmark files are ignored entirely. This is the default and the right choice for day-to-day development, since benchmarks are slow and noisy and shouldn't run on every save.
37
+
Without [`benchmark.enabled`](/config/benchmark#benchmark-enabled), the `vitest` command only runs regular tests. Benchmark files are ignored entirely. This is the default and the right choice for day-to-day development, since benchmarks are slow and noisy and shouldn't run on every save.
38
38
39
39
### `vitest` with `benchmark.enabled`
40
40
···
71
71
# only benchmarks whose test name matches "JSON"
72
72
vitest bench -t JSON
73
73
```
74
+
75
+
To execute benchmarks with another benchmarking engine or execution strategy, see the [Custom Benchmark Provider](/guide/advanced/benchmark-provider) guide.
74
76
75
77
## Comparing Benchmarks
76
78
···
330
332
331
333
Benchmarks are inherently flaky: CPU load, thermal throttling, GC pressure, and background processes all affect results. Vitest takes several steps to minimize this noise:
332
334
333
-
- **Separate project**: Benchmark files are grouped into their own project based on the [`benchmark.include`](/config/#benchmark-include) pattern. The `bench` fixture is only exposed in files matched by that pattern. Using it inside a regular test file will throw an error.
335
+
- **Separate project**: Benchmark files are grouped into their own project based on the [`benchmark.include`](/config/benchmark#benchmark-include) pattern. The `bench` fixture is only exposed in files matched by that pattern. Using it inside a regular test file will throw an error.
334
336
- **No concurrency**: Tests within a benchmark file always run sequentially. Benchmark files themselves also run one at a time, never in parallel. This prevents benchmarks from interfering with each other.
335
337
336
338
To further improve stability:
+83
docs/guide/advanced/benchmark-provider.md
+83
docs/guide/advanced/benchmark-provider.md
···
1
+
# Custom Benchmark Provider <Version type="experimental">5.0.0</Version> <Badge type="danger">advanced</Badge> {#custom-benchmark-provider}
2
+
3
+
::: warning
4
+
This is an advanced, experimental API. If you only need to run benchmarks with Vitest's built-in provider, read the [Benchmarking](/guide/benchmarking) guide instead.
5
+
:::
6
+
7
+
Vitest uses a benchmark provider to execute the functions registered with `bench` and convert their measurements into results that Vitest can report. The built-in provider uses [Tinybench](https://github.com/tinylibs/tinybench), but you can replace it to use another benchmarking engine or execution strategy.
8
+
9
+
## Setup
10
+
11
+
Set [`benchmark.provider`](/config/benchmark#benchmark-provider) to the path of your provider module. Relative paths are resolved from the project root.
12
+
13
+
```ts [vitest.config.ts]
14
+
import { defineConfig } from 'vitest/config'
15
+
16
+
export default defineConfig({
17
+
test: {
18
+
benchmark: {
19
+
provider: './benchmark-provider.ts',
20
+
},
21
+
},
22
+
})
23
+
```
24
+
25
+
The module must has a default export with an object that implements `BenchmarkProvider`. This example wraps Tinybench to demonstrate how registrations and results flow through a provider. If you use Tinybench in your provider, add it as a direct dependency of your project.
26
+
27
+
```ts [benchmark-provider.ts]
28
+
import type { BenchmarkProvider } from 'vitest'
29
+
import { Bench } from 'tinybench'
30
+
31
+
const provider = {
32
+
async run({ test, config, registrations, options }) {
33
+
const bench = new Bench({
34
+
signal: test.context.signal,
35
+
retainSamples: config.retainSamples,
36
+
...options,
37
+
})
38
+
39
+
for (const { name, fn, fnOpts } of registrations) {
40
+
bench.add(name, fn, fnOpts)
41
+
}
42
+
43
+
await bench.run()
44
+
45
+
return bench.tasks.map((task) => {
46
+
const result = task.result
47
+
48
+
if (result.state === 'errored') {
49
+
throw result.error
50
+
}
51
+
if (result.state !== 'completed') {
52
+
throw new Error(`Benchmark "${task.name}" ended in the "${result.state}" state`)
53
+
}
54
+
55
+
return {
56
+
...result,
57
+
name: task.name,
58
+
}
59
+
})
60
+
},
61
+
} satisfies BenchmarkProvider
62
+
63
+
export default provider
64
+
```
65
+
66
+
## Provider API
67
+
68
+
Vitest calls `provider.run(group)` when a registration's `.run()` method is called, or once for all runnable registrations passed to `bench.compare()`. The `group` contains:
69
+
70
+
- `test`: the test that registered the benchmarks. `test.context.signal` is aborted when the run is cancelled.
71
+
- `config`: the resolved benchmark configuration for the current project.
72
+
- `registrations`: runnable benchmarks in registration order. Every registration contains `name`, `fn`, and optional `fnOpts` for lifecycle hooks, cancellation, async behavior, and sample retention.
73
+
- `options`: benchmark run options passed to `.run()` or `bench.compare()`, if any.
74
+
75
+
The provider is responsible for honoring the run and registration options and for running every benchmark function and its `beforeAll`, `beforeEach`, `afterEach`, and `afterAll` hooks according to the benchmarking engine's lifecycle. If execution fails, throw the error to fail the test.
76
+
77
+
`run` must resolve to one `BenchResult` for every runnable registration. Results are matched to registrations by `name` and are the source for `.run()` return values, comparison tables, reporters, and saved benchmark results. A custom engine must convert its measurements into the Tinybench-compatible `BenchResult` shape exported by `vitest`.
78
+
79
+
Registrations created by `bench.from()` are loaded by Vitest and are not passed to the provider.
80
+
81
+
## Provider Lifetime
82
+
83
+
Vitest imports the provider module on first use and caches its default export for the lifetime of the worker. The API does not have separate setup or teardown hooks; keep worker-scoped state on the provider object when needed.
+2
-2
packages/vitest/src/defaults.ts
+2
-2
packages/vitest/src/defaults.ts
···
1
1
import type {
2
-
BenchmarkUserOptions,
3
2
CoverageOptions,
3
+
ResolvedBenchmarkOptions,
4
4
UserConfig,
5
5
} from './node/types/config'
6
6
import type { FieldsWithDefaultValues } from './node/types/coverage'
···
14
14
'**/node_modules/**',
15
15
'**/.git/**',
16
16
]
17
-
export const benchmarkConfigDefaults: Required<BenchmarkUserOptions> = {
17
+
export const benchmarkConfigDefaults: ResolvedBenchmarkOptions = {
18
18
enabled: false,
19
19
include: ['**/*.{bench,benchmark}.?(c|m)[jt]s?(x)'],
20
20
exclude: defaultExclude,
+107
-1
test/e2e/test/benchmarking.test.ts
+107
-1
test/e2e/test/benchmarking.test.ts
···
1131
1131
expect(stderr).toMatchInlineSnapshot(`
1132
1132
"stderr | getter-warning.bench.ts > getter warning
1133
1133
Benchmark Warning
1134
-
Benchmark "getter warning 1" accessed module export getters too many times.
1134
+
Benchmark "getter warning" accessed module export getters too many times.
1135
1135
1136
1136
This can make results unreliable because export getters add overhead.
1137
1137
See https://vitest.dev/guide/benchmarking#module-runner-overhead
···
1320
1320
.toThrow(TypeError)
1321
1321
expect(() => expect({ foo: 'bar' }).toBeSlowerThan(fakeResult(1.0)))
1322
1322
.toThrow(TypeError)
1323
+
})
1324
+
1325
+
test('`benchmark.provider` runs a custom provider whose returned results are authoritative', async () => {
1326
+
const tasks: TestBenchmarkTask[] = []
1327
+
const { stderr } = await runInlineTests(
1328
+
{
1329
+
// A provider that never runs the benchmark functions. It returns
1330
+
// fabricated statistics keyed by registration name so the assertions
1331
+
// below can only pass if Vitest reports exactly what the provider
1332
+
// returned, not anything measured off tinybench.
1333
+
'my-provider.ts': /* ts */`
1334
+
function stats(mean) {
1335
+
return {
1336
+
aad: 0, critical: 0, df: 0, mad: 0, max: mean, samples: undefined,
1337
+
mean, min: mean, moe: 0, p50: mean, p75: mean, p99: mean,
1338
+
p995: mean, p999: mean, rme: 0, samplesCount: 1, sd: 0, sem: 0, variance: 0,
1339
+
}
1340
+
}
1341
+
export default {
1342
+
async run({ registrations }) {
1343
+
return registrations.map((reg, i) => ({
1344
+
name: reg.name,
1345
+
state: 'completed',
1346
+
latency: stats((i + 1) * 100),
1347
+
throughput: stats(1 / ((i + 1) * 100)),
1348
+
period: (i + 1) * 100,
1349
+
totalTime: (i + 1) * 1000,
1350
+
}))
1351
+
},
1352
+
}
1353
+
`,
1354
+
'custom.bench.ts': /* ts */`
1355
+
import { test } from 'vitest'
1356
+
test('custom provider', async ({ bench }) => {
1357
+
await bench.compare(
1358
+
bench('first', () => { throw new Error('provider must not call fn') }),
1359
+
bench('second', () => { throw new Error('provider must not call fn') }),
1360
+
)
1361
+
})
1362
+
`,
1363
+
},
1364
+
{
1365
+
benchmark: { enabled: true, provider: './my-provider.ts' },
1366
+
reporters: [{
1367
+
onTestCaseBenchmark(_tc, benchmark) {
1368
+
tasks.push(...benchmark.tasks)
1369
+
},
1370
+
}],
1371
+
},
1372
+
)
1373
+
1374
+
expect(stderr).toBe('')
1375
+
// the benchmark fns throw — a passing run proves the provider never invoked
1376
+
// them, and the reported numbers are the provider's fabricated values
1377
+
const byName = Object.fromEntries(tasks.map(t => [t.name, t.latency.mean]))
1378
+
expect(byName).toEqual({ first: 100, second: 200 })
1379
+
// ranking is applied by Vitest over the provider's results
1380
+
expect(tasks.map(t => ({ name: t.name, rank: t.rank })).sort((a, b) => a.rank - b.rank))
1381
+
.toEqual([{ name: 'first', rank: 1 }, { name: 'second', rank: 2 }])
1382
+
})
1383
+
1384
+
test('`benchmark.provider` receives the registrations with raw fn and fnOpts', async () => {
1385
+
// A provider that actually runs each fn once, so we can prove the raw
1386
+
// benchmark function and its lifecycle hooks reach the provider untouched.
1387
+
const { stderr, results } = await runInlineTests(
1388
+
{
1389
+
'runner-provider.ts': /* ts */`
1390
+
function stats(mean) {
1391
+
return {
1392
+
aad: 0, critical: 0, df: 0, mad: 0, max: mean, samples: undefined,
1393
+
mean, min: mean, moe: 0, p50: mean, p75: mean, p99: mean,
1394
+
p995: mean, p999: mean, rme: 0, samplesCount: 1, sd: 0, sem: 0, variance: 0,
1395
+
}
1396
+
}
1397
+
export default {
1398
+
async run({ registrations }) {
1399
+
const out = []
1400
+
for (const reg of registrations) {
1401
+
await reg.fnOpts?.beforeAll?.()
1402
+
await reg.fn()
1403
+
await reg.fnOpts?.afterAll?.()
1404
+
out.push({
1405
+
name: reg.name, state: 'completed',
1406
+
latency: stats(1), throughput: stats(1), period: 1, totalTime: 1,
1407
+
})
1408
+
}
1409
+
return out
1410
+
},
1411
+
}
1412
+
`,
1413
+
'raw.bench.ts': /* ts */`
1414
+
import { test, expect } from 'vitest'
1415
+
test('raw registration', async ({ bench }) => {
1416
+
const calls = []
1417
+
await bench('x', {
1418
+
beforeAll: () => { calls.push('beforeAll') },
1419
+
afterAll: () => { calls.push('afterAll') },
1420
+
}, () => { calls.push('fn') }).run()
1421
+
expect(calls).toEqual(['beforeAll', 'fn', 'afterAll'])
1422
+
})
1423
+
`,
1424
+
},
1425
+
{ benchmark: { enabled: true, provider: './runner-provider.ts' } },
1426
+
)
1427
+
expect(stderr).toBe('')
1428
+
expect([...(results[0]?.children.allTests() ?? [])][0]?.result()?.state).toBe('passed')
1323
1429
})
1324
1430
1325
1431
declare module 'vitest' {
+6
packages/vitest/src/public/index.ts
+6
packages/vitest/src/public/index.ts
···
38
38
export type {
39
39
Bench,
40
40
BenchCompareOptions,
41
+
BenchFn,
41
42
BenchFnOptions,
42
43
BenchFromSource,
44
+
BenchmarkGroup,
45
+
BenchmarkProvider,
46
+
BenchOptions,
43
47
BenchRegistration,
48
+
BenchRegistrationInput,
44
49
BenchResult,
50
+
BenchRunOptions,
45
51
BenchStorage,
46
52
} from '../runtime/benchmark'
47
53
+191
-102
packages/vitest/src/runtime/benchmark.ts
+191
-102
packages/vitest/src/runtime/benchmark.ts
···
5
5
TaskResultCompleted,
6
6
TaskResultRuntimeInfo,
7
7
TaskResultTimestampProviderInfo,
8
-
Task as TinybenchTask,
9
8
} from 'tinybench'
10
9
import type { SerializedConfig } from './config'
10
+
import type { TestModuleRunner } from './moduleRunner/testModuleRunner'
11
11
import type { BaselineData, Test, TestBenchmark, TestBenchmarkTask } from './runner/types'
12
12
import { isAbsolute, relative } from 'pathe'
13
-
import { Bench as Tinybench } from 'tinybench'
14
13
import c from 'tinyrainbow'
14
+
import { createDefaultBenchmarkProvider } from './benchmark/default-provider'
15
15
import { rpc } from './rpc'
16
-
import { TestRunner } from './runners/test'
17
16
import { getWorkerState } from './utils'
18
-
19
-
const now = globalThis.performance
20
-
? globalThis.performance.now.bind(globalThis.performance)
21
-
: Date.now
22
17
23
18
const kRegistration: unique symbol = Symbol('registration')
24
19
const kFromSource: unique symbol = Symbol('fromSource')
···
30
25
[K in keyof T]: T[K] extends BenchRegistration<infer N> ? N : never
31
26
}[number], never>
32
27
33
-
// We throw an error if benchmark did not complete, so it will always be TaskResultCompleted
34
-
export type BenchResult = TaskResultCompleted & TaskResultRuntimeInfo & TaskResultTimestampProviderInfo
28
+
/**
29
+
* A benchmarked function. Vitest-owned and engine-agnostic.
30
+
*/
31
+
export type BenchFn = Fn
32
+
33
+
/**
34
+
* Per-benchmark initialization options passed to `bench(name, options, fn)`.
35
+
*/
36
+
export type BenchOptions = FnOptions
37
+
38
+
/**
39
+
* Engine-neutral run options forwarded to a {@link BenchmarkProvider}. These
40
+
* come from the optional trailing options argument of `bench.compare()` and
41
+
* `registration.run()`.
42
+
*/
43
+
export type BenchRunOptions = BenchCompareOptions
44
+
45
+
/**
46
+
* The result of a single benchmark, produced by a {@link BenchmarkProvider}.
47
+
* Structurally compatible with tinybench's task statistics so it flows
48
+
* unchanged into the reporter, `bench.from()` baselines and comparison tables.
49
+
* The `name` matches the corresponding {@link BenchRegistration}.
50
+
*/
51
+
export interface BenchResult extends TaskResultCompleted, TaskResultRuntimeInfo, TaskResultTimestampProviderInfo {
52
+
/** The registered benchmark name this result belongs to. */
53
+
name: string
54
+
}
55
+
56
+
/**
57
+
* A benchmark defined by `bench()` (or `bench.from()`), as seen by a
58
+
* {@link BenchmarkProvider}. `fn`/`fnOpts` are absent for `bench.from()`
59
+
* registrations, which are resolved from a stored baseline instead of run.
60
+
*/
61
+
export interface BenchRegistrationInput {
62
+
name: string
63
+
fn: BenchFn
64
+
fnOpts?: BenchOptions
65
+
}
66
+
67
+
/**
68
+
* The set of benchmarks registered by a single test, handed to
69
+
* {@link BenchmarkProvider.run}.
70
+
*/
71
+
export interface BenchmarkGroup {
72
+
/** The test that registered these benchmarks. */
73
+
test: Test
74
+
/** The resolved benchmark configuration for the current project. */
75
+
config: SerializedConfig['benchmark']
76
+
/** The runnable registrations, in registration order. */
77
+
registrations: BenchRegistrationInput[]
78
+
/** Engine-neutral run options, when provided by the caller. */
79
+
options?: BenchRunOptions
80
+
}
81
+
82
+
/**
83
+
* Executes the benchmarks of a single test and returns their results.
84
+
*
85
+
* A custom provider module must default-export an object of this shape.
86
+
*
87
+
* The returned array is the sole source of results: it's what `bench().run()`
88
+
* resolves to and what the reporter serializes. Results are matched to
89
+
* registrations by `name`; return one {@link BenchResult} per registration.
90
+
*
91
+
* @experimental
92
+
*/
93
+
export interface BenchmarkProvider {
94
+
run: (group: BenchmarkGroup) => Promise<BenchResult[]>
95
+
}
96
+
97
+
let cachedProvider: Promise<BenchmarkProvider> | undefined
98
+
99
+
async function loadProviderModule(
100
+
provider: string,
101
+
moduleRunner: TestModuleRunner,
102
+
): Promise<BenchmarkProvider> {
103
+
let mod: Record<string, any>
104
+
try {
105
+
mod = await moduleRunner.import(provider)
106
+
}
107
+
catch (error) {
108
+
throw new Error(
109
+
`Failed to load benchmark provider from "${provider}".`,
110
+
{ cause: error },
111
+
)
112
+
}
113
+
if (mod.default == null) {
114
+
throw new Error(
115
+
`Benchmark provider loaded from "${provider}" did not have a default export.`,
116
+
)
117
+
}
118
+
return mod.default
119
+
}
120
+
121
+
/**
122
+
* Resolves the benchmark provider for the current worker, importing a custom
123
+
* provider module on first use. The result is cached for the lifetime of the
124
+
* worker so a custom provider is imported at most once.
125
+
*/
126
+
export function resolveBenchmarkProvider(
127
+
config: SerializedConfig,
128
+
moduleRunner: TestModuleRunner,
129
+
): Promise<BenchmarkProvider> {
130
+
if (!cachedProvider) {
131
+
const provider = config.benchmark.provider
132
+
cachedProvider = !provider
133
+
? Promise.resolve(createDefaultBenchmarkProvider(config))
134
+
: loadProviderModule(provider, moduleRunner)
135
+
}
136
+
return cachedProvider
137
+
}
35
138
36
139
export interface BenchStorage<T extends string> {
37
140
get: (name: T) => BenchResult
···
40
143
export type { BenchOptions as BenchCompareOptions } from 'tinybench'
41
144
42
145
/**
43
-
* Options accepted by `bench(name, options, fn)`. Extends tinybench's
44
-
* `FnOptions` with Vitest-specific fields.
146
+
* Options accepted by `bench(name, options, fn)`. Extends the per-benchmark
147
+
* lifecycle hooks with Vitest-specific fields.
45
148
*/
46
-
export interface BenchFnOptions extends FnOptions {
149
+
export interface BenchFnOptions extends BenchOptions {
47
150
/**
48
151
* Path (relative to the project root) where the benchmark result is written
49
152
* after a successful run. The string `${projectName}` is substituted with
···
65
168
/**
66
169
* The benchmark function. Absent for registrations created via `bench.from()`.
67
170
*/
68
-
fn?: Fn
171
+
fn?: BenchFn
69
172
/**
70
173
* Per-benchmark options (`beforeEach`, `beforeAll`, etc.). Absent for
71
174
* registrations created via `bench.from()`.
72
175
*/
73
-
fnOpts?: FnOptions
176
+
fnOpts?: BenchOptions
74
177
/**
75
178
* @internal
76
179
*/
77
180
[kRegistration]: true
78
-
run: (options?: BenchCompareOptions) => Promise<BenchResult>
181
+
run: (options?: BenchRunOptions) => Promise<BenchResult>
79
182
}
80
183
81
184
interface BenchCompare {
82
185
<Args extends BenchRegistration<any>[]>(...args: Args): Promise<BenchStorage<ExtractBenchNames<Args>>>
83
-
<Args extends BenchRegistration<any>[]>(...args: [...Args, BenchCompareOptions]): Promise<BenchStorage<ExtractBenchNames<Args>>>
186
+
<Args extends BenchRegistration<any>[]>(...args: [...Args, BenchRunOptions]): Promise<BenchStorage<ExtractBenchNames<Args>>>
84
187
}
85
188
86
189
interface BenchFactory {
87
-
<Name extends string>(name: Name | Function, fn: Fn): BenchRegistration<Name>
88
-
<Name extends string>(name: Name | Function, options: BenchFnOptions, fn: Fn): BenchRegistration<Name>
190
+
<Name extends string>(name: Name | Function, fn: BenchFn): BenchRegistration<Name>
191
+
<Name extends string>(name: Name | Function, options: BenchFnOptions, fn: BenchFn): BenchRegistration<Name>
89
192
}
90
193
91
194
export interface BenchFromSource {
···
104
207
}
105
208
106
209
interface RunnableRegistration<Name extends string> extends BenchRegistration<Name> {
107
-
fn: Fn
108
-
fnOpts?: FnOptions
210
+
fn: BenchFn
211
+
fnOpts?: BenchOptions
109
212
[kWriteResult]?: string
110
213
[kPerProject]?: true
111
214
}
···
122
225
return template.replace(/\$\{projectName\}/g, projectName ?? '')
123
226
}
124
227
125
-
export function createBench(test: Test, config: SerializedConfig): Bench {
126
-
let benchIdx = 0
228
+
export function createBench(
229
+
test: Test,
230
+
config: SerializedConfig,
231
+
moduleRunner: TestModuleRunner,
232
+
): Bench {
127
233
const pending = new Set<BenchRegistration<any>>()
128
-
const createTinybench = (options?: BenchCompareOptions) => {
129
-
const currentIndex = ++benchIdx
130
-
return new Tinybench({
131
-
signal: test.context.signal,
132
-
name: `${test.fullTestName} ${currentIndex}`,
133
-
retainSamples: config.benchmark.retainSamples,
134
-
...options,
135
-
now,
136
-
})
137
-
}
138
234
139
235
const resolveTemplate = (template: string) => substitutePath(template, config.benchmark.projectName)
140
236
···
164
260
})
165
261
166
262
const createCompareStorage = <T extends string>(
167
-
bench: Tinybench,
263
+
results: Map<string, BenchResult>,
168
264
fromResults?: Map<string, BaselineData>,
169
265
): BenchStorage<T> => {
170
266
return {
···
173
269
if (stored) {
174
270
return stored as BenchResult
175
271
}
176
-
const task = bench.getTask(name)
177
-
if (!task) {
272
+
const result = results.get(name)
273
+
if (!result) {
178
274
throw new Error(`task "${name}" was not defined`)
179
275
}
180
-
return task.result as BenchResult
276
+
return result
181
277
},
182
278
}
183
279
}
···
185
281
interface TaskMeta { perProject?: true }
186
282
187
283
const serializeBenchmark = (
188
-
tinybenchTasks: TinybenchTask[],
189
-
name: string | undefined,
284
+
results: BenchResult[],
285
+
name: string,
190
286
taskMeta?: Map<string, TaskMeta>,
191
287
fromTasks?: TestBenchmarkTask[],
192
288
): TestBenchmark => {
193
-
const tasks: TestBenchmarkTask[] = tinybenchTasks.map((t) => {
194
-
const result = t.result
195
-
if (result.state === 'errored') {
196
-
throw result.error
197
-
}
198
-
if (result.state !== 'completed') {
199
-
throw new Error(`task "${t.name}" did not complete: received "${result.state}"`)
200
-
}
201
-
return {
202
-
name: t.name,
203
-
latency: result.latency,
204
-
throughput: result.throughput,
205
-
period: result.period,
206
-
totalTime: result.totalTime,
207
-
rank: 0,
208
-
...taskMeta?.get(t.name),
209
-
}
210
-
})
289
+
const tasks: TestBenchmarkTask[] = results.map(result => ({
290
+
name: result.name,
291
+
latency: result.latency,
292
+
throughput: result.throughput,
293
+
period: result.period,
294
+
totalTime: result.totalTime,
295
+
rank: 0,
296
+
...taskMeta?.get(result.name),
297
+
}))
211
298
if (fromTasks) {
212
299
tasks.push(...fromTasks)
213
300
}
···
216
303
task.rank = idx + 1
217
304
})
218
305
return {
219
-
name: name || test.fullTestName,
306
+
name,
220
307
tasks,
221
308
}
222
309
}
223
310
224
311
const recordBenchmark = async (
225
-
tinybenchTasks: TinybenchTask[],
226
-
name: string | undefined,
312
+
results: BenchResult[],
313
+
name: string,
227
314
taskMeta?: Map<string, TaskMeta>,
228
315
fromTasks?: TestBenchmarkTask[],
229
316
) => {
230
-
const serializedBenchmark = serializeBenchmark(tinybenchTasks, name, taskMeta, fromTasks)
317
+
const serializedBenchmark = serializeBenchmark(results, name, taskMeta, fromTasks)
231
318
test.benchmarks.push(serializedBenchmark)
232
319
await rpc().onTestBenchmark(test.id, serializedBenchmark)
233
320
}
···
243
330
await rpc().writeBenchmarkResult(resolved, data)
244
331
}
245
332
246
-
const runBenchmarks = async (tinybench: Tinybench) => {
333
+
const groupName = (options: BenchRunOptions | undefined) => options?.name ?? test.fullTestName
334
+
335
+
const runGroup = async (
336
+
registrations: BenchRegistrationInput[],
337
+
options: BenchRunOptions | undefined,
338
+
): Promise<Map<string, BenchResult>> => {
247
339
const workerState = getWorkerState()
248
340
const getterTracker = workerState.getterTracker
249
341
getterTracker?.resetInvocations()
250
342
try {
251
-
return await TestRunner.runBenchmarks(tinybench)
343
+
const provider = await resolveBenchmarkProvider(config, moduleRunner)
344
+
const results = await provider.run({ test, config: config.benchmark, registrations, options })
345
+
const byName = new Map<string, BenchResult>()
346
+
for (const result of results) {
347
+
byName.set(result.name, result)
348
+
}
349
+
return byName
252
350
}
253
351
finally {
254
352
const excessiveInvocations = config.benchmark.suppressExportGetterWarnings
···
261
359
console.warn(
262
360
[
263
361
c.yellow(c.bold('Benchmark Warning')),
264
-
`Benchmark ${c.bold(`"${tinybench.name}"`)} accessed module export getters too many times.`,
362
+
`Benchmark ${c.bold(`"${groupName(options)}"`)} accessed module export getters too many times.`,
265
363
'',
266
364
'This can make results unreliable because export getters add overhead.',
267
365
'See https://vitest.dev/guide/benchmarking#module-runner-overhead',
···
276
374
277
375
const runSingle = async (
278
376
name: string,
279
-
fn: Fn,
280
-
fnOpts: FnOptions | undefined,
281
-
options: BenchCompareOptions | undefined,
377
+
fn: BenchFn,
378
+
fnOpts: BenchOptions | undefined,
379
+
options: BenchRunOptions | undefined,
282
380
meta: TaskMeta | undefined,
283
381
writeResult: string | undefined,
284
382
): Promise<BenchResult> => {
285
-
const tinybench = createTinybench(options).add(name, fn, fnOpts)
286
-
const tasks = await runBenchmarks(tinybench)
287
-
const task = tinybench.getTask(name)!
288
-
if (task.result.state === 'errored') {
289
-
throw task.result.error
383
+
const results = await runGroup([{ name, fn, fnOpts }], options)
384
+
const result = results.get(name)
385
+
if (!result) {
386
+
throw new Error(`benchmark provider did not return a result for "${name}"`)
290
387
}
291
-
await recordBenchmark(tasks, tinybench.name, meta ? new Map([[name, meta]]) : undefined)
388
+
await recordBenchmark([result], groupName(options), meta ? new Map([[name, meta]]) : undefined)
292
389
if (writeResult) {
293
-
await writeResultArtifact(writeResult, task.result as BenchResult)
390
+
await writeResultArtifact(writeResult, result)
294
391
}
295
-
return task.result as BenchResult
392
+
return result
296
393
}
297
394
298
395
const runFrom = async (
···
309
406
return data as BenchResult
310
407
}
311
408
312
-
const bench: Bench = (nameOrFunction: string | Function, a: Fn | BenchFnOptions, b?: Fn | BenchFnOptions) => {
409
+
const bench: Bench = (nameOrFunction: string | Function, a: BenchFn | BenchFnOptions, b?: BenchFn | BenchFnOptions) => {
313
410
validateBenchmarkProject(config)
314
411
const { fn, fnOpts, writeResult, perProject } = normalizeBenchArgs(a, b)
315
412
const name = typeof nameOrFunction === 'function' ? nameOrFunction.name || '<anonymous>' : nameOrFunction
···
319
416
name,
320
417
fn,
321
418
fnOpts,
322
-
run: (options?: BenchCompareOptions) => {
419
+
run: (options?: BenchRunOptions) => {
323
420
pending.delete(registration)
324
421
return runSingle(name, fn, fnOpts, options, meta, writeResult)
325
422
},
···
359
456
bench.compare = async (...args) => {
360
457
validateBenchmarkProject(config)
361
458
362
-
// extract optional trailing BenchCompareOptions argument
459
+
// extract optional trailing BenchRunOptions argument
363
460
const lastArg = args.at(-1)
364
461
const isOptions = lastArg != null && typeof lastArg === 'object' && !(kRegistration in lastArg)
365
-
const benchOptions = isOptions ? args.pop() as BenchCompareOptions : undefined
462
+
const benchOptions = isOptions ? args.pop() as BenchRunOptions : undefined
366
463
const registrations = args as BenchRegistration<any>[]
367
464
368
465
// Mark every passed-in registration as consumed before validation so a
···
416
513
}
417
514
}
418
515
419
-
const tinybench = createTinybench(benchOptions)
420
-
runnable.forEach((reg) => {
421
-
tinybench.add(reg.name, reg.fn, reg.fnOpts)
422
-
})
423
-
424
-
let tasks: TinybenchTask[] = []
516
+
let results = new Map<string, BenchResult>()
425
517
if (runnable.length > 0) {
426
-
tasks = await runBenchmarks(tinybench)
427
-
const errors = tinybench.tasks
428
-
.filter(task => task.result.state === 'errored')
429
-
.map(task => (task.result as any).error)
430
-
if (errors.length > 0) {
431
-
throw new AggregateError(errors, 'Some benchmarks failed')
432
-
}
518
+
results = await runGroup(
519
+
runnable.map(reg => ({ name: reg.name, fn: reg.fn, fnOpts: reg.fnOpts })),
520
+
benchOptions,
521
+
)
433
522
}
434
523
435
-
await recordBenchmark(tasks, tinybench.name, taskMeta, fromTasks)
524
+
await recordBenchmark(Array.from(results.values()), groupName(benchOptions), taskMeta, fromTasks)
436
525
437
526
// write artifacts for every runnable registration that requested it. We
438
527
// do this after recording so a write failure can't be confused with a
···
441
530
runnable
442
531
.filter(reg => reg[kWriteResult] != null)
443
532
.map((reg) => {
444
-
const task = tinybench.getTask(reg.name)!
445
-
return writeResultArtifact(reg[kWriteResult]!, task.result as BenchResult)
533
+
const result = results.get(reg.name)!
534
+
return writeResultArtifact(reg[kWriteResult]!, result)
446
535
}),
447
536
)
448
537
449
-
return createCompareStorage(tinybench, fromResults)
538
+
return createCompareStorage(results, fromResults)
450
539
}
451
540
452
541
bench[kFinalize] = () => {
···
477
566
}
478
567
479
568
function normalizeBenchArgs(
480
-
a: Fn | BenchFnOptions,
481
-
b: Fn | BenchFnOptions | undefined,
482
-
): { fn: Fn; fnOpts: FnOptions | undefined; writeResult: string | undefined; perProject: boolean } {
569
+
a: BenchFn | BenchFnOptions,
570
+
b: BenchFn | BenchFnOptions | undefined,
571
+
): { fn: BenchFn; fnOpts: BenchOptions | undefined; writeResult: string | undefined; perProject: boolean } {
483
572
if (typeof a === 'function') {
484
573
if (b !== undefined) {
485
574
throw new TypeError('`bench()` does not accept options as the third argument. Pass options as the second argument instead: `bench(name, options, fn)`.')
···
491
580
}
492
581
// Strip vitest-specific fields only when present so we don't allocate a new
493
582
// object — preserving referential identity matters: users inspect
494
-
// `registration.fnOpts` and tinybench's `add` sees the same object the
495
-
// caller passed in.
583
+
// `registration.fnOpts` and the provider sees the same object the caller
584
+
// passed in.
496
585
if (a.writeResult === undefined && a.perProject === undefined) {
497
-
return { fn: b, fnOpts: a as FnOptions, writeResult: undefined, perProject: false }
586
+
return { fn: b, fnOpts: a as BenchOptions, writeResult: undefined, perProject: false }
498
587
}
499
588
const { writeResult, perProject, ...fnOpts } = a
500
589
return {
501
590
fn: b,
502
-
fnOpts: Object.keys(fnOpts).length > 0 ? fnOpts as FnOptions : undefined,
591
+
fnOpts: Object.keys(fnOpts).length > 0 ? fnOpts as BenchOptions : undefined,
503
592
writeResult,
504
593
perProject: perProject ?? false,
505
594
}
+1
packages/vitest/src/runtime/config.ts
+1
packages/vitest/src/runtime/config.ts
+3
-3
packages/ui/client/composables/client/static.ts
+3
-3
packages/ui/client/composables/client/static.ts
···
70
70
rpc: undefined!,
71
71
reconnect: () => registerMetadata(),
72
72
waitForConnection: async () => {},
73
-
})
73
+
}) as VitestClient
74
74
75
-
ctx.state.filesMap = reactive(ctx.state.filesMap)
76
-
ctx.state.idMap = reactive(ctx.state.idMap)
75
+
ctx.state.filesMap = reactive(ctx.state.filesMap) as StateManager['filesMap']
76
+
ctx.state.idMap = reactive(ctx.state.idMap) as StateManager['idMap']
77
77
78
78
async function registerMetadata() {
79
79
const content = await window.HTML_REPORT_METADATA!
+6
packages/vitest/src/node/config/resolveConfig.ts
+6
packages/vitest/src/node/config/resolveConfig.ts
···
294
294
...benchmarkConfigDefaults,
295
295
...resolved.benchmark,
296
296
}
297
+
if (resolved.benchmark.provider) {
298
+
resolved.benchmark.provider = resolvePath(
299
+
resolved.benchmark.provider,
300
+
resolved.root,
301
+
)
302
+
}
297
303
298
304
const inspector = resolved.inspect || resolved.inspectBrk
299
305
+1
packages/vitest/src/node/config/serializeConfig.ts
+1
packages/vitest/src/node/config/serializeConfig.ts
···
134
134
benchmark: {
135
135
enabled: config.benchmark.enabled,
136
136
retainSamples: config.benchmark.retainSamples,
137
+
provider: config.benchmark.provider,
137
138
suppressExportGetterWarnings: config.benchmark.suppressExportGetterWarnings,
138
139
projectName: config.benchmark.projectName,
139
140
},
+14
packages/vitest/src/node/types/benchmark.ts
+14
packages/vitest/src/node/types/benchmark.ts
···
29
29
retainSamples?: boolean
30
30
31
31
/**
32
+
* The benchmark provider that executes registered benchmarks and produces
33
+
* their results. Provide a path to a module whose default export implements
34
+
* `BenchmarkProvider`. The path is resolved relative to the project
35
+
* root. If not specified, the built-in provider is used.
36
+
*
37
+
* @experimental
38
+
*/
39
+
provider?: string
40
+
41
+
/**
32
42
* Disable warnings when a benchmark accesses module export getters too many times.
33
43
* @default false
34
44
*/
···
42
52
* @internal
43
53
*/
44
54
projectName?: string
55
+
}
56
+
57
+
export type ResolvedBenchmarkOptions = Omit<Required<BenchmarkUserOptions>, 'provider'> & {
58
+
provider?: string | undefined
45
59
}
+3
-3
packages/vitest/src/node/types/config.ts
+3
-3
packages/vitest/src/node/types/config.ts
···
20
20
import type { TestSequencerConstructor } from '../sequencers/types'
21
21
import type { VCSProvider } from '../vcs/vcs'
22
22
import type { WatcherTriggerPattern } from '../watcher'
23
-
import type { BenchmarkUserOptions } from './benchmark'
23
+
import type { BenchmarkUserOptions, ResolvedBenchmarkOptions } from './benchmark'
24
24
import type { BrowserConfigOptions, BrowserServerContribution, ResolvedBrowserOptions } from './browser'
25
25
import type { CoverageOptions, ResolvedCoverageOptions } from './coverage'
26
26
import type { Reporter } from './reporter'
27
27
28
28
export type { CoverageOptions, ResolvedCoverageOptions }
29
-
export type { BenchmarkUserOptions }
29
+
export type { BenchmarkUserOptions, ResolvedBenchmarkOptions }
30
30
export type { RuntimeConfig, SerializedConfig } from '../../runtime/config'
31
31
export type { SequenceHooks, SequenceSetupFiles } from '../../runtime/runner/types'
32
32
export type { BrowserConfigOptions, BrowserInstanceOption, BrowserScript } from './browser'
···
1227
1227
cliExclude?: string[]
1228
1228
1229
1229
project: string[]
1230
-
benchmark: Required<BenchmarkUserOptions>
1230
+
benchmark: ResolvedBenchmarkOptions
1231
1231
shard?: {
1232
1232
index: number
1233
1233
count: number
+55
packages/vitest/src/runtime/benchmark/default-provider.ts
+55
packages/vitest/src/runtime/benchmark/default-provider.ts
···
1
+
import type { Task as TinybenchTask } from 'tinybench'
2
+
import type { BenchmarkGroup, BenchmarkProvider, BenchResult } from '../benchmark'
3
+
import type { SerializedConfig } from '../config'
4
+
import { Bench as Tinybench } from 'tinybench'
5
+
6
+
const now = globalThis.performance
7
+
? globalThis.performance.now.bind(globalThis.performance)
8
+
: Date.now
9
+
10
+
/**
11
+
* The built-in benchmark provider, backed by tinybench. Selected when
12
+
* `benchmark.provider` is not configured.
13
+
*/
14
+
export function createDefaultBenchmarkProvider(config: SerializedConfig): BenchmarkProvider {
15
+
let benchIdx = 0
16
+
return {
17
+
async run({ test, options, registrations }: BenchmarkGroup): Promise<BenchResult[]> {
18
+
const currentIndex = ++benchIdx
19
+
const tinybench = new Tinybench({
20
+
signal: test.context.signal,
21
+
name: `${test.fullTestName} ${currentIndex}`,
22
+
retainSamples: config.benchmark.retainSamples,
23
+
...options,
24
+
now,
25
+
})
26
+
for (const { name, fn, fnOpts } of registrations) {
27
+
tinybench.add(name, fn, fnOpts)
28
+
}
29
+
await tinybench.run()
30
+
31
+
const errors = tinybench.tasks
32
+
.filter(task => task.result.state === 'errored')
33
+
.map(task => (task.result as { error: unknown }).error)
34
+
if (errors.length === 1) {
35
+
throw errors[0]
36
+
}
37
+
if (errors.length > 1) {
38
+
throw new AggregateError(errors, 'Some benchmarks failed')
39
+
}
40
+
41
+
return tinybench.tasks.map(toBenchResult)
42
+
},
43
+
}
44
+
}
45
+
46
+
function toBenchResult(task: TinybenchTask): BenchResult {
47
+
const result = task.result
48
+
if (result.state !== 'completed') {
49
+
throw new Error(`task "${task.name}" did not complete: received "${result.state}"`)
50
+
}
51
+
return {
52
+
...result,
53
+
name: task.name,
54
+
}
55
+
}
+2
-11
packages/vitest/src/runtime/runners/test.ts
+2
-11
packages/vitest/src/runtime/runners/test.ts
···
1
1
import type { SpanOptions } from '@opentelemetry/api'
2
2
import type { ExpectStatic } from '@vitest/expect'
3
-
import type { Bench as Tinybench, Task as TinybenchTask } from 'tinybench'
4
3
import type { ModuleRunner } from 'vite/module-runner'
5
4
import type { Traces } from '../../utils/traces'
6
5
import type { Bench } from '../benchmark'
···
240
239
let _bench: Bench | undefined
241
240
const runnerConfig = this.config
242
241
const benchInstances = this.benchInstances
242
+
const moduleRunner = this.moduleRunner
243
243
Object.defineProperty(context, 'bench', {
244
244
get() {
245
245
if (!_bench) {
246
-
_bench = createBench(context.task, runnerConfig)
246
+
_bench = createBench(context.task, runnerConfig, moduleRunner)
247
247
benchInstances.set(context.task, _bench)
248
248
}
249
249
return _bench
···
298
298
static setTestFn: typeof getFn = getFn
299
299
static matchesTags: typeof matchesTags = matchesTags
300
300
static createFileTask: typeof createFileTask = createFileTask
301
-
302
-
/**
303
-
* @experimental
304
-
* A function that runs tinybench tasks.
305
-
* Can be overriden to run tasks in a special environment.
306
-
*/
307
-
static async runBenchmarks(tinybench: Tinybench): Promise<TinybenchTask[]> {
308
-
return await tinybench.run()
309
-
}
310
301
}
311
302
312
303
function clearModuleMocks(config: SerializedConfig) {