[READ-ONLY] Mirror of https://github.com/vitest-dev/vitest. Next generation testing framework powered by Vite.
vitest.dev
test
testing-tools
vite
feat: print performance hints after the run (`experimental.diagnostics`)
Vladimir Sheremet
(Jul 23, 2026, 6:09 PM +0200)
fd3342a6
4b54a1a7
+1314
-11
+81
docs/config/experimental.md
+81
docs/config/experimental.md
···
413
413
test('my test', { tags }, () => {})
414
414
```
415
415
:::
416
+
417
+
## experimental.diagnostics <Version type="experimental">5.0.0</Version> {#experimental-diagnostics}
418
+
419
+
- **Type:**
420
+
421
+
```ts
422
+
interface DiagnosticsOptions {
423
+
/**
424
+
* Hint when `isolate: true` spends a significant amount of time spawning
425
+
* a fresh worker (and re-creating the environment) for every test file,
426
+
* estimating how much `isolate: false` could save.
427
+
* @default true
428
+
*/
429
+
isolate?: boolean
430
+
/**
431
+
* Hint when re-creating a DOM environment for every test file dominates
432
+
* the run and a `vm` pool would set it up once per worker.
433
+
* @default true
434
+
*/
435
+
environment?: boolean
436
+
/**
437
+
* Hint when test files repeatedly evaluate the same module graph
438
+
* (typical for barrel-file imports) and `isolate: false` would
439
+
* evaluate it once per worker.
440
+
* @default true
441
+
*/
442
+
import?: boolean
443
+
/**
444
+
* Hint when transforming modules dominates the run and
445
+
* `fsModuleCache` would persist the results across runs.
446
+
* @default true
447
+
*/
448
+
transform?: boolean
449
+
}
450
+
```
451
+
452
+
- **Default:** `true`
453
+
454
+
Print performance hints after the run when the collected timings show that a configuration change would make the run significantly faster:
455
+
456
+
```
457
+
Environment jsdom was created 40 times · 23.80s total, 79% of tracked time
458
+
create it once per worker with pool: 'vmThreads' (keeps per-file isolation) or isolate: false (shares it across files)
459
+
learn more: https://vitest.dev/guide/improving-performance#test-environments
460
+
```
461
+
462
+
Hints never suggest changing an option that was set explicitly: if the config defines `pool`, other pools are not suggested, and an explicitly configured `isolate` is never suggested to be disabled. Hints are also printed in CI. Set the option to `false` to disable all hints, or disable them individually.
463
+
464
+
### experimental.diagnostics.isolate {#experimental-diagnostics-isolate}
465
+
466
+
- **Type:** `boolean`
467
+
- **Default:** `true`
468
+
469
+
Hint when `isolate: true` spends a significant amount of time spawning a fresh worker (and re-creating the environment) for every test file, estimating how much `isolate: false` could save. Reused workers also keep evaluated modules alive, so files stop re-evaluating the module graph they share. Per-module evaluation times are only collected when [`experimental.importDurations`](#experimental-importdurations) is enabled; without it the estimate counts the worker startups alone and is reported as a lower bound ("at least").
470
+
471
+
### experimental.diagnostics.environment {#experimental-diagnostics-environment}
472
+
473
+
- **Type:** `boolean`
474
+
- **Default:** `true`
475
+
476
+
Hint when re-creating a DOM environment for every test file dominates the run and a `vm` pool would set it up once per worker.
477
+
478
+
### experimental.diagnostics.import {#experimental-diagnostics-import}
479
+
480
+
- **Type:** `boolean`
481
+
- **Default:** `true`
482
+
483
+
Hint when test files repeatedly evaluate the same module graph and `isolate: false` would evaluate it once per worker. This is typical for barrel-file imports: every test file imports a few symbols through an index file and evaluates the whole graph behind it. The duplication is measured from how often each module was served to the workers, so suites whose test files import mostly disjoint modules stay quiet: reusing workers would not reduce their import work.
484
+
485
+
```
486
+
Import 837 modules were evaluated 16740 times · 15.69s total, 64% of tracked time
487
+
~850ms faster with isolate: false — shared modules are evaluated once per worker instead of once per file
488
+
learn more: https://vitest.dev/guide/improving-performance#test-isolation
489
+
```
490
+
491
+
### experimental.diagnostics.transform {#experimental-diagnostics-transform}
492
+
493
+
- **Type:** `boolean`
494
+
- **Default:** `true`
495
+
496
+
Hint when transforming modules dominates the run. Without a persistent cache every `vitest run` transforms the whole module graph from scratch; [`fsModuleCache`](/config/fsmodulecache) stores the results on disk so repeated runs skip them. The hint estimates the time the next run would save. On CI the hint includes a note that the cache directory must be persisted between runs for the cache to take effect.
+28
docs/guide/cli-generated.md
+28
docs/guide/cli-generated.md
···
992
992
- **Config:** [experimental.preParse](/config/experimental#experimental-preparse)
993
993
994
994
Parse test specifications before running them. This will apply `.only` flag and test name pattern across all files without running them. (default: `false`)
995
+
996
+
### experimental.diagnostics.isolate
997
+
998
+
- **CLI:** `--experimental.diagnostics.isolate`
999
+
- **Config:** [experimental.diagnostics.isolate](/config/experimental#experimental-diagnostics-isolate)
1000
+
1001
+
Print a hint estimating how much time `isolate: false` would save when `isolate: true` spends a significant amount of time spawning a worker per test file. (default: `true`)
1002
+
1003
+
### experimental.diagnostics.environment
1004
+
1005
+
- **CLI:** `--experimental.diagnostics.environment`
1006
+
- **Config:** [experimental.diagnostics.environment](/config/experimental#experimental-diagnostics-environment)
1007
+
1008
+
Print a hint when re-creating a DOM environment for every test file dominates the run and a `vm` pool would set it up once per worker. (default: `true`)
1009
+
1010
+
### experimental.diagnostics.import
1011
+
1012
+
- **CLI:** `--experimental.diagnostics.import`
1013
+
- **Config:** [experimental.diagnostics.import](/config/experimental#experimental-diagnostics-import)
1014
+
1015
+
Print a hint when test files repeatedly evaluate the same module graph (typical for barrel-file imports) and `isolate: false` would evaluate it once per worker. (default: `true`)
1016
+
1017
+
### experimental.diagnostics.transform
1018
+
1019
+
- **CLI:** `--experimental.diagnostics.transform`
1020
+
- **Config:** [experimental.diagnostics.transform](/config/experimental#experimental-diagnostics-transform)
1021
+
1022
+
Print a hint when transforming modules dominates the run and `fsModuleCache` would persist the results across runs. (default: `true`)
+29
-2
docs/guide/improving-performance.md
+29
-2
docs/guide/improving-performance.md
···
8
8
Duration 3.76s (environment 79%, import 14%, transform 6%, tests 1%)
9
9
```
10
10
11
-
The percentages are relative to the sum of all tracked phases, not to the wall-clock time: phases run in parallel workers, so their sum is usually larger than the run itself. In a multi-project setup the percentages aggregate over all [projects](/guide/projects), so a phase that dominates one project can be diluted by the others.
11
+
The percentages are relative to the sum of all tracked phases, not to the wall-clock time: phases run in parallel workers, so their sum is usually larger than the run itself. In a multi-project setup the percentages aggregate over all [projects](/guide/projects), so a phase that dominates one project can be diluted by the others; the performance hints below analyze each project separately.
12
12
13
13
The phases map to configuration options:
14
14
15
-
- `environment` - creating the test environment (`jsdom`, `happy-dom`) for test files.
15
+
- `environment` - creating the test environment (`jsdom`, `happy-dom`) for test files. See [Test Environments](#test-environments).
16
16
- `transform` - transforming files with Vite. See [Caching Between Reruns](#caching-between-reruns).
17
17
- `import` - importing test files and their modules. When files import mostly the same modules (typical for barrel-file imports), isolation re-evaluates that shared graph for every file. See [Test Isolation](#test-isolation).
18
18
- `setup` - running [`setupFiles`](/config/setupfiles).
19
19
- `tests` - running the tests themselves. A run dominated by this phase has little to gain from configuration changes.
20
+
21
+
When the collected timings show that a configuration change would make the run significantly faster, Vitest also prints a hint after the summary, see [`experimental.diagnostics`](/config/experimental#experimental-diagnostics). Hints never suggest changing an option that was set explicitly.
20
22
21
23
## Test Isolation
22
24
···
90
92
})
91
93
```
92
94
:::
95
+
96
+
## Test Environments
97
+
98
+
DOM environments are expensive to create: `jsdom` costs roughly 200-500ms per import and `happy-dom` roughly 90-200ms, plus the time to construct the window. With an isolating pool (the default), that cost is paid for every test file, because every file gets a fresh worker. On DOM-heavy suites this is often the largest cost of the run; it appears as the `environment` share of the `Duration` breakdown.
99
+
100
+
Three configurations reduce this cost:
101
+
102
+
| configuration | environment created | isolation | trade-off |
103
+
|---|---|---|---|
104
+
| `pool: 'forks'`/`'threads'` + `isolate: true` (default) | once per file | fresh process/thread and environment per file | safest, slowest |
105
+
| `pool: 'vmThreads'` | once per worker | fresh VM context and `window` per file | test code runs in a VM realm: cross-realm `instanceof` edge cases with externalized packages, and memory is not reclaimed as reliably (see [`vmMemoryLimit`](/config/vmmemorylimit)) |
106
+
| `isolate: false` | once per worker | none - files in the same worker share the environment and module state | tests must not depend on a clean `window` or module state |
107
+
108
+
```ts [vitest.config.js]
109
+
import { defineConfig } from 'vitest/config'
110
+
111
+
export default defineConfig({
112
+
test: {
113
+
environment: 'jsdom',
114
+
pool: 'vmThreads', // environment per worker, fresh window per file
115
+
},
116
+
})
117
+
```
118
+
119
+
Prefer `isolate: false` with `threads` if the tests tolerate shared state: it is the fastest option and keeps memory behavior simple. Use `vmThreads` when every file needs a fresh `window` and the per-file environment cost dominates the run. `happy-dom` is cheaper to create than `jsdom` in every setup.
93
120
94
121
## Limiting Directory Search
95
122
+5
-1
test/test-utils/index.ts
+5
-1
test/test-utils/index.ts
···
212
212
},
213
213
// override cache config with the one that was used to run `vitest` from the CLI
214
214
fsModuleCache: rest.fsModuleCache ?? currentConfig.fsModuleCache,
215
-
...(cliOptions?.experimental ? { experimental: cliOptions.experimental } : {}),
215
+
experimental: {
216
+
// keep performance hints out of captured test output unless a test opts in
217
+
diagnostics: rest.experimental?.diagnostics ?? false,
218
+
...cliOptions?.experimental,
219
+
},
216
220
}, {
217
221
...viteConfig,
218
222
plugins: [
+184
test/unit/test/environment-diagnostics.test.ts
+184
test/unit/test/environment-diagnostics.test.ts
···
1
+
import { describe, expect, it } from 'vitest'
2
+
import { getEnvironmentDiagnostics, isSavingWorthHinting } from '../../../packages/vitest/src/node/reporters/diagnostics'
3
+
4
+
const domProject = {
5
+
name: 'dom',
6
+
environment: 'jsdom',
7
+
pool: 'forks',
8
+
isolate: true,
9
+
browser: false,
10
+
poolProvided: false,
11
+
isolateProvided: false,
12
+
environmentTime: 20_000,
13
+
environmentCount: 40,
14
+
trackedTime: 30_000,
15
+
// estimated saving: 20s/8 - 20s/40 = 2s
16
+
parallelism: 8,
17
+
executionTime: 10_000,
18
+
}
19
+
20
+
describe('getEnvironmentDiagnostics', () => {
21
+
it('fires for an isolating DOM project dominated by environment setup', () => {
22
+
expect(getEnvironmentDiagnostics([domProject])).toEqual([
23
+
{
24
+
name: 'dom',
25
+
environment: 'jsdom',
26
+
environmentTime: 20_000,
27
+
environmentCount: 40,
28
+
share: 20_000 / 30_000,
29
+
suggestIsolate: true,
30
+
},
31
+
])
32
+
})
33
+
34
+
it('fires for happy-dom the same way', () => {
35
+
expect(getEnvironmentDiagnostics([{ ...domProject, environment: 'happy-dom' }])).toEqual([
36
+
{
37
+
name: 'dom',
38
+
environment: 'happy-dom',
39
+
environmentTime: 20_000,
40
+
environmentCount: 40,
41
+
share: 20_000 / 30_000,
42
+
suggestIsolate: true,
43
+
},
44
+
])
45
+
})
46
+
47
+
it('fires for the threads pool', () => {
48
+
expect(getEnvironmentDiagnostics([{ ...domProject, pool: 'threads' }])).toHaveLength(1)
49
+
})
50
+
51
+
it('fires exactly at the minimum time and share thresholds', () => {
52
+
// 2s of setups over 8s of tracked time is exactly the 2s / 25% minimum;
53
+
// saving: 2s/4 - 2s/40 = 450ms, above the 200ms (5% of 4s) floor
54
+
expect(getEnvironmentDiagnostics([{
55
+
...domProject,
56
+
environmentTime: 2_000,
57
+
trackedTime: 8_000,
58
+
parallelism: 4,
59
+
executionTime: 4_000,
60
+
}])).toHaveLength(1)
61
+
})
62
+
63
+
it('fires for a serial run - all setups but one are avoidable', () => {
64
+
expect(getEnvironmentDiagnostics([{ ...domProject, parallelism: 1 }])).toHaveLength(1)
65
+
})
66
+
67
+
it('reports every affected project with its own numbers', () => {
68
+
const components = {
69
+
...domProject,
70
+
name: 'components',
71
+
environment: 'happy-dom',
72
+
environmentTime: 9_000,
73
+
environmentCount: 30,
74
+
trackedTime: 12_000,
75
+
}
76
+
expect(getEnvironmentDiagnostics([domProject, components])).toEqual([
77
+
{
78
+
name: 'dom',
79
+
environment: 'jsdom',
80
+
environmentTime: 20_000,
81
+
environmentCount: 40,
82
+
share: 20_000 / 30_000,
83
+
suggestIsolate: true,
84
+
},
85
+
{
86
+
name: 'components',
87
+
environment: 'happy-dom',
88
+
environmentTime: 9_000,
89
+
environmentCount: 30,
90
+
share: 9_000 / 12_000,
91
+
suggestIsolate: true,
92
+
},
93
+
])
94
+
})
95
+
96
+
it('does not suggest disabling isolation the user explicitly enabled', () => {
97
+
const diagnostics = getEnvironmentDiagnostics([{ ...domProject, isolateProvided: true }])
98
+
expect(diagnostics).toHaveLength(1)
99
+
expect(diagnostics[0].suggestIsolate).toBe(false)
100
+
})
101
+
102
+
it('stays quiet when the user explicitly configured the pool', () => {
103
+
expect(getEnvironmentDiagnostics([{ ...domProject, poolProvided: true }])).toEqual([])
104
+
})
105
+
106
+
it('stays quiet for vm pools - the environment is already per worker', () => {
107
+
expect(getEnvironmentDiagnostics([{ ...domProject, pool: 'vmThreads', isolate: false }])).toEqual([])
108
+
})
109
+
110
+
it('stays quiet without isolation - the environment is already per worker', () => {
111
+
expect(getEnvironmentDiagnostics([{ ...domProject, isolate: false }])).toEqual([])
112
+
})
113
+
114
+
it('stays quiet for the node environment', () => {
115
+
expect(getEnvironmentDiagnostics([{ ...domProject, environment: 'node' }])).toEqual([])
116
+
})
117
+
118
+
it('stays quiet for browser projects', () => {
119
+
expect(getEnvironmentDiagnostics([{ ...domProject, browser: true }])).toEqual([])
120
+
})
121
+
122
+
it('stays quiet when the setup cost is small in absolute terms', () => {
123
+
expect(getEnvironmentDiagnostics([{
124
+
...domProject,
125
+
environmentTime: 1_000,
126
+
trackedTime: 1_200,
127
+
}])).toEqual([])
128
+
})
129
+
130
+
it('stays quiet when the setup cost is small relative to the run', () => {
131
+
expect(getEnvironmentDiagnostics([{
132
+
...domProject,
133
+
environmentTime: 3_000,
134
+
trackedTime: 60_000,
135
+
}])).toEqual([])
136
+
})
137
+
138
+
it('stays quiet for a single file - there is nothing to amortize', () => {
139
+
expect(getEnvironmentDiagnostics([{ ...domProject, environmentCount: 1 }])).toEqual([])
140
+
})
141
+
142
+
it('stays quiet when the saving is negligible relative to a long run', () => {
143
+
// ~2s estimated saving is below 5% of a 5-minute run and below 10s absolute
144
+
expect(getEnvironmentDiagnostics([{ ...domProject, executionTime: 300_000 }])).toEqual([])
145
+
})
146
+
147
+
it('fires on long runs when the absolute saving is large', () => {
148
+
// 100s/8 - 100s/50 = 10.5s: only 2.6% of the run, but 10s+ is worth attention
149
+
expect(getEnvironmentDiagnostics([{
150
+
...domProject,
151
+
environmentTime: 100_000,
152
+
environmentCount: 50,
153
+
trackedTime: 150_000,
154
+
executionTime: 400_000,
155
+
}])).toHaveLength(1)
156
+
})
157
+
158
+
it('reports only the affected projects', () => {
159
+
const diagnostics = getEnvironmentDiagnostics([
160
+
domProject,
161
+
{ ...domProject, name: 'unit', environment: 'node' },
162
+
])
163
+
expect(diagnostics.map(diagnostic => diagnostic.name)).toEqual(['dom'])
164
+
})
165
+
})
166
+
167
+
describe('isSavingWorthHinting', () => {
168
+
it('requires a quarter of a second in absolute terms', () => {
169
+
expect(isSavingWorthHinting(200, 1_000)).toBe(false)
170
+
})
171
+
172
+
it('accepts savings above 5% of the run', () => {
173
+
expect(isSavingWorthHinting(300, 1_000)).toBe(true)
174
+
expect(isSavingWorthHinting(500, 10_000)).toBe(true)
175
+
})
176
+
177
+
it('rejects savings below the noise floor of a long run', () => {
178
+
expect(isSavingWorthHinting(300, 60_000)).toBe(false)
179
+
})
180
+
181
+
it('accepts 10s+ savings regardless of percentage', () => {
182
+
expect(isSavingWorthHinting(12_000, 400_000)).toBe(true)
183
+
})
184
+
})
+208
test/unit/test/import-diagnostics.test.ts
+208
test/unit/test/import-diagnostics.test.ts
···
1
+
import { describe, expect, it } from 'vitest'
2
+
import { estimateModuleEvaluationSaving, getImportDiagnostics, getTransformDiagnostics } from '../../../packages/vitest/src/node/reporters/diagnostics'
3
+
4
+
// the barrel-file shape: 20 test files that each re-evaluate the same
5
+
// 800-module graph to use a few symbols from it
6
+
const barrelProject = {
7
+
name: 'barrel',
8
+
pool: 'forks',
9
+
isolate: true,
10
+
browser: false,
11
+
isolateProvided: false,
12
+
importTime: 15_000,
13
+
trackedTime: 20_000,
14
+
fetchCounts: Array.from({ length: 800 }, () => 20),
15
+
fileCount: 20,
16
+
// duplication: (20 - 8) / 20 = 0.6, estimated saving: 15s * 0.6 / 8 = 1.125s
17
+
parallelism: 8,
18
+
executionTime: 4_000,
19
+
}
20
+
21
+
describe('getImportDiagnostics', () => {
22
+
it('fires when test files repeatedly evaluate a shared module graph', () => {
23
+
expect(getImportDiagnostics([barrelProject])).toEqual([
24
+
{
25
+
name: 'barrel',
26
+
importTime: 15_000,
27
+
share: 15_000 / 20_000,
28
+
totalFetches: 16_000,
29
+
uniqueModules: 800,
30
+
duplication: 0.6,
31
+
estimatedSaving: (15_000 * 0.6) / 8,
32
+
},
33
+
])
34
+
})
35
+
36
+
it('fires for the threads pool', () => {
37
+
expect(getImportDiagnostics([{ ...barrelProject, pool: 'threads' }])).toHaveLength(1)
38
+
})
39
+
40
+
it('counts only the fetches above the worker parallelism as avoidable', () => {
41
+
// 100 modules fetched 20 times (12 avoidable each on 8 lanes) and 100
42
+
// fetched twice (0 avoidable): duplication = 1200 / 2200
43
+
const fetchCounts = [
44
+
...Array.from({ length: 100 }, () => 20),
45
+
...Array.from({ length: 100 }, () => 2),
46
+
]
47
+
expect(getImportDiagnostics([{ ...barrelProject, fetchCounts }])).toEqual([
48
+
{
49
+
name: 'barrel',
50
+
importTime: 15_000,
51
+
share: 15_000 / 20_000,
52
+
totalFetches: 2_200,
53
+
uniqueModules: 200,
54
+
duplication: 1_200 / 2_200,
55
+
estimatedSaving: (15_000 * (1_200 / 2_200)) / 8,
56
+
},
57
+
])
58
+
})
59
+
60
+
it('fires exactly at the minimum duplication', () => {
61
+
// 10 fetches per module on 8 lanes: duplication (10 - 8) / 10 = 0.2
62
+
expect(getImportDiagnostics([{
63
+
...barrelProject,
64
+
fetchCounts: Array.from({ length: 800 }, () => 10),
65
+
}])).toHaveLength(1)
66
+
})
67
+
68
+
it('fires exactly at the minimum import time and share', () => {
69
+
// 2s of imports over 8s of tracked time is exactly the 2s / 25% minimum;
70
+
// saving on 4 lanes: 2s * 0.8 / 4 = 400ms, above the 250ms floor
71
+
expect(getImportDiagnostics([{
72
+
...barrelProject,
73
+
importTime: 2_000,
74
+
trackedTime: 8_000,
75
+
parallelism: 4,
76
+
}])).toHaveLength(1)
77
+
})
78
+
79
+
it('treats every repeated fetch as avoidable on a single lane', () => {
80
+
const diagnostics = getImportDiagnostics([{ ...barrelProject, parallelism: 1 }])
81
+
expect(diagnostics).toHaveLength(1)
82
+
expect(diagnostics[0].duplication).toBe(19 / 20)
83
+
expect(diagnostics[0].estimatedSaving).toBe(15_000 * (19 / 20))
84
+
})
85
+
86
+
it('reports every affected project', () => {
87
+
const diagnostics = getImportDiagnostics([barrelProject, { ...barrelProject, name: 'ui' }])
88
+
expect(diagnostics.map(diagnostic => diagnostic.name)).toEqual(['barrel', 'ui'])
89
+
})
90
+
91
+
it('stays quiet for disjoint per-file graphs - reused workers would not help', () => {
92
+
// every module belongs to one or two test files: no fetch count exceeds
93
+
// the parallelism, so nothing is re-evaluated beyond what lanes require
94
+
expect(getImportDiagnostics([{
95
+
...barrelProject,
96
+
fetchCounts: Array.from({ length: 800 }, (_, i) => (i % 2 === 0 ? 1 : 2)),
97
+
}])).toEqual([])
98
+
})
99
+
100
+
it('stays quiet when the tests dominate the run', () => {
101
+
expect(getImportDiagnostics([{
102
+
...barrelProject,
103
+
importTime: 2_500,
104
+
trackedTime: 30_000,
105
+
}])).toEqual([])
106
+
})
107
+
108
+
it('stays quiet when the import time is small in absolute terms', () => {
109
+
expect(getImportDiagnostics([{
110
+
...barrelProject,
111
+
importTime: 1_500,
112
+
trackedTime: 2_000,
113
+
}])).toEqual([])
114
+
})
115
+
116
+
it('does not suggest disabling isolation the user explicitly enabled', () => {
117
+
expect(getImportDiagnostics([{ ...barrelProject, isolateProvided: true }])).toEqual([])
118
+
})
119
+
120
+
it('stays quiet without isolation - the graph is already shared', () => {
121
+
expect(getImportDiagnostics([{ ...barrelProject, isolate: false }])).toEqual([])
122
+
})
123
+
124
+
it('stays quiet for vm pools - they re-create the graph per context regardless', () => {
125
+
expect(getImportDiagnostics([{ ...barrelProject, pool: 'vmThreads' }])).toEqual([])
126
+
})
127
+
128
+
it('stays quiet for browser projects', () => {
129
+
expect(getImportDiagnostics([{ ...barrelProject, browser: true }])).toEqual([])
130
+
})
131
+
132
+
it('stays quiet when files do not outnumber the workers', () => {
133
+
expect(getImportDiagnostics([{ ...barrelProject, fileCount: 8 }])).toEqual([])
134
+
})
135
+
136
+
it('stays quiet when the saving is negligible relative to a long run', () => {
137
+
// ~1.1s estimated saving is below 5% of a 5-minute run and below 10s absolute
138
+
expect(getImportDiagnostics([{ ...barrelProject, executionTime: 300_000 }])).toEqual([])
139
+
})
140
+
})
141
+
142
+
const coldProject = {
143
+
name: 'cold',
144
+
transformTime: 8_000,
145
+
trackedTime: 20_000,
146
+
fsModuleCache: false,
147
+
fsModuleCacheProvided: false,
148
+
executionTime: 10_000,
149
+
}
150
+
151
+
describe('getTransformDiagnostics', () => {
152
+
it('fires when transforms dominate and nothing persists them', () => {
153
+
expect(getTransformDiagnostics([coldProject])).toEqual([
154
+
{
155
+
name: 'cold',
156
+
transformTime: 8_000,
157
+
share: 8_000 / 20_000,
158
+
},
159
+
])
160
+
})
161
+
162
+
it('stays quiet when the fs module cache is already enabled', () => {
163
+
expect(getTransformDiagnostics([{ ...coldProject, fsModuleCache: true }])).toEqual([])
164
+
})
165
+
166
+
it('does not suggest a cache the user explicitly configured away', () => {
167
+
expect(getTransformDiagnostics([{ ...coldProject, fsModuleCacheProvided: true }])).toEqual([])
168
+
})
169
+
170
+
it('stays quiet when the transform time is small in absolute terms', () => {
171
+
expect(getTransformDiagnostics([{ ...coldProject, transformTime: 1_500 }])).toEqual([])
172
+
})
173
+
174
+
it('stays quiet when the tests dominate the run', () => {
175
+
expect(getTransformDiagnostics([{
176
+
...coldProject,
177
+
transformTime: 3_000,
178
+
trackedTime: 30_000,
179
+
}])).toEqual([])
180
+
})
181
+
182
+
it('stays quiet when the saving is negligible relative to a long run', () => {
183
+
expect(getTransformDiagnostics([{
184
+
...coldProject,
185
+
transformTime: 2_500,
186
+
trackedTime: 9_000,
187
+
executionTime: 300_000,
188
+
}])).toEqual([])
189
+
})
190
+
})
191
+
192
+
describe('estimateModuleEvaluationSaving', () => {
193
+
it('spreads the avoidable evaluations across the worker lanes', () => {
194
+
// a module evaluated by 20 files at 10ms each on 8 lanes keeps 8
195
+
// evaluations: (200 * 12/20) / 8 = 15ms saved per module
196
+
const modules = Array.from({ length: 10 }, () => Array.from({ length: 20 }, () => 10))
197
+
expect(estimateModuleEvaluationSaving(modules, 8)).toBe(150)
198
+
})
199
+
200
+
it('saves nothing when no module is evaluated more often than the lanes', () => {
201
+
const modules = [Array.from({ length: 8 }, () => 10), [10, 10]]
202
+
expect(estimateModuleEvaluationSaving(modules, 8)).toBe(0)
203
+
})
204
+
205
+
it('treats every evaluation past the first as avoidable on a single lane', () => {
206
+
expect(estimateModuleEvaluationSaving([[10, 10, 10, 10]], 1)).toBe(30)
207
+
})
208
+
})
+29
-2
packages/vitest/src/node/state.ts
+29
-2
packages/vitest/src/node/state.ts
···
34
34
* of magnitude.
35
35
*/
36
36
transformTime = 0
37
+
/**
38
+
* Transform busy time split by project name, measured the same way as
39
+
* `transformTime`. Used by performance diagnostics to relate a project's
40
+
* transform cost to the rest of its tracked time.
41
+
*/
42
+
transformTimes: Map<string, number> = new Map()
43
+
/**
44
+
* Total time spent starting test workers (spawning the process/thread, loading
45
+
* the worker bundle and setting up the test environment). Used to surface the
46
+
* cost of `isolate: true`, which spawns a fresh worker per test file.
47
+
*/
48
+
startupTime = 0
49
+
/** Number of test workers that were started during the run. */
50
+
workersSpawned = 0
37
51
private _transformsInflight = 0
38
52
private _transformsBusyStart = 0
53
+
private _projectTransformsInflight: Map<string, number> = new Map()
54
+
private _projectTransformsBusyStart: Map<string, number> = new Map()
39
55
40
-
transformStarted(): void {
56
+
transformStarted(projectName: string): void {
41
57
if (this._transformsInflight++ === 0) {
42
58
this._transformsBusyStart = performance.now()
43
59
}
60
+
const inflight = this._projectTransformsInflight.get(projectName) || 0
61
+
this._projectTransformsInflight.set(projectName, inflight + 1)
62
+
if (inflight === 0) {
63
+
this._projectTransformsBusyStart.set(projectName, performance.now())
64
+
}
44
65
}
45
66
46
-
transformFinished(): void {
67
+
transformFinished(projectName: string): void {
47
68
if (--this._transformsInflight === 0) {
48
69
this.transformTime += performance.now() - this._transformsBusyStart
70
+
}
71
+
const inflight = (this._projectTransformsInflight.get(projectName) || 1) - 1
72
+
this._projectTransformsInflight.set(projectName, inflight)
73
+
if (inflight === 0) {
74
+
const busy = performance.now() - this._projectTransformsBusyStart.get(projectName)!
75
+
this.transformTimes.set(projectName, (this.transformTimes.get(projectName) || 0) + busy)
49
76
}
50
77
}
51
78
+18
packages/vitest/src/node/cli/cli-config.ts
+18
packages/vitest/src/node/cli/cli-config.ts
···
949
949
preParse: {
950
950
description: 'Parse test specifications before running them. This will apply `.only` flag and test name pattern across all files without running them. (default: `false`)',
951
951
},
952
+
diagnostics: {
953
+
description: 'Print performance hints after the run when a configuration change would make it significantly faster. Hints never suggest changing options that were set explicitly. (default: `true`)',
954
+
argument: '',
955
+
subcommands: {
956
+
isolate: {
957
+
description: 'Print a hint estimating how much time `isolate: false` would save when `isolate: true` spends a significant amount of time spawning a worker per test file. (default: `true`)',
958
+
},
959
+
environment: {
960
+
description: 'Print a hint when re-creating a DOM environment for every test file dominates the run and a `vm` pool would set it up once per worker. (default: `true`)',
961
+
},
962
+
import: {
963
+
description: 'Print a hint when test files repeatedly evaluate the same module graph (typical for barrel-file imports) and `isolate: false` would evaluate it once per worker. (default: `true`)',
964
+
},
965
+
transform: {
966
+
description: 'Print a hint when transforming modules dominates the run and `fsModuleCache` would persist the results across runs. (default: `true`)',
967
+
},
968
+
},
969
+
},
952
970
},
953
971
},
954
972
// disable CLI options
+39
packages/vitest/src/node/config/resolveConfig.ts
+39
packages/vitest/src/node/config/resolveConfig.ts
···
170
170
}
171
171
}
172
172
173
+
/**
174
+
* Records which options the user provided explicitly. Must be computed from
175
+
* the raw user config sources BEFORE `configDefaults` is merged in - the
176
+
* merged object cannot distinguish a default from a user-provided value.
177
+
*/
178
+
export function captureProvidedOptions(
179
+
...sources: (UserConfig | undefined)[]
180
+
): ResolvedConfig['providedOptions'] {
181
+
return {
182
+
pool: sources.some(source => source?.pool != null),
183
+
isolate: sources.some(source => source?.isolate != null),
184
+
environment: sources.some(source => source?.environment != null || source?.dom),
185
+
fsModuleCache: sources.some(source =>
186
+
source?.fsModuleCache != null
187
+
|| (source?.experimental as { fsModuleCache?: boolean } | undefined)?.fsModuleCache != null),
188
+
}
189
+
}
190
+
173
191
// warn only once, check one PER PROCESS, not per instance,
174
192
// that's why it's on a module-level
175
193
let warnedTypeCheck = false
···
205
223
options.environment = 'happy-dom'
206
224
}
207
225
226
+
// provenance must be captured from the raw options BEFORE `configDefaults`
227
+
// is merged in - the merged object cannot distinguish a default from a
228
+
// user-provided value; `viteConfig.test` is not resolved yet at this point,
229
+
// the call sites assign the resolved config to it after this function returns
230
+
const providedOptions = captureProvidedOptions(
231
+
options,
232
+
viteConfig.test as UserConfig | undefined,
233
+
)
234
+
208
235
const resolved = deepMerge({}, configDefaults, options) as ResolvedConfig
209
236
resolved.root = viteConfig.root
237
+
resolved.providedOptions = providedOptions
210
238
211
239
// Coverage is collected once for the whole run using the root config, so projects
212
240
// share its resolved coverage. Each project's setup/test/config files are then
···
991
1019
resolved.experimental.importDurations.thresholds ??= {} as any
992
1020
resolved.experimental.importDurations.thresholds.warn ??= 100
993
1021
resolved.experimental.importDurations.thresholds.danger ??= 500
1022
+
1023
+
const diagnostics = (resolved.experimental.diagnostics as boolean | { isolate?: boolean; environment?: boolean; import?: boolean; transform?: boolean } | undefined)
1024
+
?? true
1025
+
resolved.experimental.diagnostics = typeof diagnostics === 'boolean'
1026
+
? { isolate: diagnostics, environment: diagnostics, import: diagnostics, transform: diagnostics }
1027
+
: {
1028
+
isolate: diagnostics.isolate ?? true,
1029
+
environment: diagnostics.environment ?? true,
1030
+
import: diagnostics.import ?? true,
1031
+
transform: diagnostics.transform ?? true,
1032
+
}
994
1033
995
1034
if (typeof resolved.experimental.vcsProvider === 'string' && resolved.experimental.vcsProvider !== 'git') {
996
1035
resolved.experimental.vcsProvider = resolvePath(resolved.experimental.vcsProvider, resolved.root)
+6
-4
packages/vitest/src/node/environments/fetchModule.ts
+6
-4
packages/vitest/src/node/environments/fetchModule.ts
···
26
26
* actual work by orders of magnitude.
27
27
*/
28
28
export interface TransformClock {
29
-
transformStarted: () => void
30
-
transformFinished: () => void
29
+
transformStarted: (projectName: string) => void
30
+
transformFinished: (projectName: string) => void
31
31
}
32
32
33
33
class ModuleFetcher {
34
34
private tmpDirectories = new Set<string>()
35
35
private fsCacheEnabled: boolean
36
+
private projectName: string
36
37
// the module type is only needed by the evaluator to decide if CJS
37
38
// variables should be provided to the module, so don't waste time
38
39
// on the detection when every module receives them
···
47
48
) {
48
49
this.fsCacheEnabled = config.fsModuleCache === true
49
50
this.detectModuleType = config.injectCjsGlobals === false
51
+
this.projectName = config.name || ''
50
52
}
51
53
52
54
async fetch(
···
325
327
moduleGraphModule: EnvironmentModuleNode,
326
328
options?: FetchFunctionOptions,
327
329
): Promise<VitestFetchResult> {
328
-
this.clock.transformStarted()
330
+
this.clock.transformStarted(this.projectName)
329
331
try {
330
332
const moduleRunnerModule = await fetchModule(
331
333
environment,
···
344
346
return result
345
347
}
346
348
finally {
347
-
this.clock.transformFinished()
349
+
this.clock.transformFinished(this.projectName)
348
350
}
349
351
}
350
352
+7
packages/vitest/src/node/pools/poolRunner.ts
+7
packages/vitest/src/node/pools/poolRunner.ts
···
184
184
this._operationLock = createDefer()
185
185
186
186
let startSpan: Span | undefined
187
+
const startedAt = performance.now()
187
188
try {
188
189
this._state = RunnerState.STARTING
189
190
···
233
234
await startPromise
234
235
235
236
this._state = RunnerState.STARTED
237
+
238
+
// record how long it took to spawn this worker, load its bundle and set up the
239
+
// environment, so the reporter can surface the cost of `isolate: true`
240
+
const { state } = this.project.vitest
241
+
state.startupTime += performance.now() - startedAt
242
+
state.workersSpawned += 1
236
243
}
237
244
catch (error: any) {
238
245
this._state = RunnerState.START_FAILURE
+343
-1
packages/vitest/src/node/reporters/base.ts
+343
-1
packages/vitest/src/node/reporters/base.ts
···
6
6
import type { Reporter, TestRunEndReason } from '../types/reporter'
7
7
import type { TestCase, TestCollection, TestModule, TestModuleState, TestResult, TestSuite, TestSuiteState } from './reported-tasks'
8
8
import { readFileSync } from 'node:fs'
9
+
import { availableParallelism } from 'node:os'
9
10
import { performance } from 'node:perf_hooks'
10
11
import { toArray } from '@vitest/utils/helpers'
11
12
import { parseStacktrace } from '@vitest/utils/source-map'
12
13
import { relative } from 'pathe'
13
14
import c from 'tinyrainbow'
14
15
import { groupBy } from '../../utils/base'
15
-
import { isTTY } from '../../utils/env'
16
+
import { isCI, isTTY } from '../../utils/env'
16
17
import { getSuites, getTestName, getTests, hasFailed, hasFailedSnapshot } from '../../utils/tasks'
17
18
import { generateCodeFrame, printStack } from '../printError'
19
+
import { estimateModuleEvaluationSaving, getEnvironmentDiagnostics, getImportDiagnostics, getTransformDiagnostics, isSavingWorthHinting } from './diagnostics'
18
20
import { computeDurationBreakdown, formatDurationBreakdown } from './durationBreakdown'
19
21
import { BENCH_TABLE_HEAD, computeBenchColumnWidths, padBenchRow, renderBenchmarkRow } from './renderers/benchmark-table'
20
22
import { F_CHECK, F_DOWN_RIGHT, F_POINTER } from './renderers/figures'
···
657
659
658
660
this.reportImportDurations()
659
661
662
+
// at most one hint per family: the environment hint is the most specific,
663
+
// the import hint explains the same `isolate` remedy through module data,
664
+
// and the transform hint (a cache, helping the *next* run) only speaks up
665
+
// when neither applies
666
+
const hinted = this.reportEnvironmentDiagnostic(files)
667
+
|| this.reportImportDiagnostic(files)
668
+
|| this.reportTransformDiagnostic(files)
669
+
if (!hinted) {
670
+
this.reportIsolateDiagnostic(files)
671
+
}
672
+
660
673
this.log()
674
+
}
675
+
676
+
private getEffectiveMaxWorkers(): number {
677
+
const configured = this.ctx.config.maxWorkers
678
+
return typeof configured === 'number' && configured > 0
679
+
? configured
680
+
: Math.max(1, availableParallelism() - 1)
681
+
}
682
+
683
+
/**
684
+
* Surfaces the cost of re-creating a DOM environment for every test file:
685
+
* with an isolating pool, `jsdom`/`happy-dom` are imported and set up once
686
+
* per file. When that repeated setup dominates the run, hint that a `vm`
687
+
* pool sets the environment up once per worker while keeping per-file
688
+
* isolation, and that `isolate: false` shares it across files.
689
+
*/
690
+
private reportEnvironmentDiagnostic(files: File[]): boolean {
691
+
// merged blob reports replay durations of past runs: no environments were
692
+
// created by this process, and the per-project transform split needed for
693
+
// an accurate share is not part of the blob format
694
+
if (this.ctx.config.watch || this.ctx.state.blobs || !this.ctx.config.experimental.diagnostics.environment) {
695
+
return false
696
+
}
697
+
698
+
const executionTime = this.end - this.start
699
+
const maxWorkers = this.getEffectiveMaxWorkers()
700
+
const transformTimes = this.ctx.state.transformTimes
701
+
const inputs = this.ctx.projects.map((project) => {
702
+
const projectFiles = files.filter(file => (file.projectName || '') === project.name)
703
+
let environmentTime = 0
704
+
let environmentCount = 0
705
+
let trackedTime = transformTimes.get(project.name) || 0
706
+
for (const file of projectFiles) {
707
+
if (file.environmentLoad) {
708
+
environmentTime += file.environmentLoad
709
+
environmentCount++
710
+
}
711
+
trackedTime
712
+
+= (file.environmentLoad || 0)
713
+
+ (file.setupDuration || 0)
714
+
+ (file.collectDuration || 0)
715
+
+ (file.result?.duration || 0)
716
+
}
717
+
return {
718
+
name: project.name,
719
+
environment: project.config.environment,
720
+
pool: project.config.pool,
721
+
isolate: project.config.isolate,
722
+
browser: project.config.browser.enabled,
723
+
poolProvided: project.config.providedOptions.pool,
724
+
isolateProvided: project.config.providedOptions.isolate,
725
+
environmentTime,
726
+
environmentCount,
727
+
trackedTime,
728
+
parallelism: Math.max(1, Math.min(environmentCount, maxWorkers)),
729
+
executionTime,
730
+
}
731
+
})
732
+
733
+
const diagnostics = getEnvironmentDiagnostics(inputs)
734
+
if (!diagnostics.length) {
735
+
return false
736
+
}
737
+
738
+
for (const diagnostic of diagnostics) {
739
+
const project = this.ctx.projects.find(p => p.name === diagnostic.name)
740
+
this.log()
741
+
this.log(
742
+
padSummaryTitle('Environment'),
743
+
formatProjectName(project)
744
+
+ c.yellow(`${diagnostic.environment} was created ${diagnostic.environmentCount} times`)
745
+
+ c.dim(` · ${formatTime(diagnostic.environmentTime)} total, ${Math.round(diagnostic.share * 100)}% of tracked time`),
746
+
)
747
+
const alternative = diagnostic.suggestIsolate
748
+
? c.dim(' (keeps per-file isolation) or ') + c.yellow('isolate: false') + c.dim(' (shares it across files)')
749
+
: c.dim(' (keeps per-file isolation)')
750
+
this.log(
751
+
padSummaryTitle(''),
752
+
c.dim('create it once per worker with ')
753
+
+ c.yellow(`pool: 'vmThreads'`)
754
+
+ alternative,
755
+
)
756
+
this.log(
757
+
padSummaryTitle(''),
758
+
c.dim('learn more: https://vitest.dev/guide/improving-performance#test-environments'),
759
+
)
760
+
}
761
+
762
+
return true
763
+
}
764
+
765
+
/**
766
+
* Surfaces repeated evaluation of the same module graph: with `isolate: true`
767
+
* every test file re-imports its whole graph, so suites where files share
768
+
* most of their modules (typically through barrel files) pay the graph cost
769
+
* once per file. The duplication is measured from server-side fetch counts,
770
+
* so suites with disjoint per-file graphs stay quiet.
771
+
*/
772
+
private reportImportDiagnostic(files: File[]): boolean {
773
+
if (this.ctx.config.watch || this.ctx.state.blobs || !this.ctx.config.experimental.diagnostics.import) {
774
+
return false
775
+
}
776
+
777
+
const executionTime = this.end - this.start
778
+
const maxWorkers = this.getEffectiveMaxWorkers()
779
+
const transformTimes = this.ctx.state.transformTimes
780
+
const inputs = this.ctx.projects.map((project) => {
781
+
const projectFiles = files.filter(file => (file.projectName || '') === project.name)
782
+
let importTime = 0
783
+
let trackedTime = transformTimes.get(project.name) || 0
784
+
for (const file of projectFiles) {
785
+
importTime += file.collectDuration || 0
786
+
trackedTime
787
+
+= (file.environmentLoad || 0)
788
+
+ (file.setupDuration || 0)
789
+
+ (file.collectDuration || 0)
790
+
+ (file.result?.duration || 0)
791
+
}
792
+
const durations = this.ctx.state.metadata[project.name]?.duration
793
+
return {
794
+
name: project.name,
795
+
pool: project.config.pool,
796
+
isolate: project.config.isolate,
797
+
browser: project.config.browser.enabled,
798
+
isolateProvided: project.config.providedOptions.isolate,
799
+
importTime,
800
+
trackedTime,
801
+
fetchCounts: durations ? Object.values(durations).map(times => times.length) : [],
802
+
fileCount: projectFiles.length,
803
+
parallelism: Math.max(1, Math.min(projectFiles.length, maxWorkers)),
804
+
executionTime,
805
+
}
806
+
})
807
+
808
+
const diagnostics = getImportDiagnostics(inputs)
809
+
if (!diagnostics.length) {
810
+
return false
811
+
}
812
+
813
+
for (const diagnostic of diagnostics) {
814
+
const project = this.ctx.projects.find(p => p.name === diagnostic.name)
815
+
this.log()
816
+
this.log(
817
+
padSummaryTitle('Import'),
818
+
formatProjectName(project)
819
+
+ c.yellow(`${diagnostic.uniqueModules} modules were evaluated ${diagnostic.totalFetches} times`)
820
+
+ c.dim(` · ${formatTime(diagnostic.importTime)} total, ${Math.round(diagnostic.share * 100)}% of tracked time`),
821
+
)
822
+
this.log(
823
+
padSummaryTitle(''),
824
+
c.dim(`~${formatTime(diagnostic.estimatedSaving)} faster with `)
825
+
+ c.yellow('isolate: false')
826
+
+ c.dim(' — shared modules are evaluated once per worker instead of once per file'),
827
+
)
828
+
this.log(
829
+
padSummaryTitle(''),
830
+
c.dim('learn more: https://vitest.dev/guide/improving-performance#test-isolation'),
831
+
)
832
+
}
833
+
834
+
return true
835
+
}
836
+
837
+
/**
838
+
* Surfaces transform-dominated runs: without the fs module cache every
839
+
* `vitest run` transforms the whole module graph from scratch. Enabling
840
+
* `fsModuleCache` persists the results so the next run skips them.
841
+
*/
842
+
private reportTransformDiagnostic(files: File[]): boolean {
843
+
if (this.ctx.config.watch || this.ctx.state.blobs || !this.ctx.config.experimental.diagnostics.transform) {
844
+
return false
845
+
}
846
+
847
+
const executionTime = this.end - this.start
848
+
const transformTimes = this.ctx.state.transformTimes
849
+
const inputs = this.ctx.projects.map((project) => {
850
+
const projectFiles = files.filter(file => (file.projectName || '') === project.name)
851
+
const transformTime = transformTimes.get(project.name) || 0
852
+
let trackedTime = transformTime
853
+
for (const file of projectFiles) {
854
+
trackedTime
855
+
+= (file.environmentLoad || 0)
856
+
+ (file.setupDuration || 0)
857
+
+ (file.collectDuration || 0)
858
+
+ (file.result?.duration || 0)
859
+
}
860
+
return {
861
+
name: project.name,
862
+
transformTime,
863
+
trackedTime,
864
+
fsModuleCache: project.config.fsModuleCache === true,
865
+
fsModuleCacheProvided: project.config.providedOptions.fsModuleCache,
866
+
executionTime,
867
+
}
868
+
})
869
+
870
+
const diagnostics = getTransformDiagnostics(inputs)
871
+
if (!diagnostics.length) {
872
+
return false
873
+
}
874
+
875
+
for (const diagnostic of diagnostics) {
876
+
const project = this.ctx.projects.find(p => p.name === diagnostic.name)
877
+
this.log()
878
+
this.log(
879
+
padSummaryTitle('Transform'),
880
+
formatProjectName(project)
881
+
+ c.yellow(`transforming modules took ${formatTime(diagnostic.transformTime)}`)
882
+
+ c.dim(` · ${Math.round(diagnostic.share * 100)}% of tracked time, re-done on every run`),
883
+
)
884
+
this.log(
885
+
padSummaryTitle(''),
886
+
c.dim('persist transforms across runs with ')
887
+
+ c.yellow('fsModuleCache: true'),
888
+
)
889
+
if (isCI) {
890
+
this.log(
891
+
padSummaryTitle(''),
892
+
c.dim('on CI this only helps when the cache directory is persisted between runs'),
893
+
)
894
+
}
895
+
this.log(
896
+
padSummaryTitle(''),
897
+
c.dim('learn more: https://vitest.dev/guide/improving-performance#caching-between-reruns'),
898
+
)
899
+
}
900
+
901
+
return true
902
+
}
903
+
904
+
/**
905
+
* Surfaces the cost of `isolate: true`: with isolation enabled Vitest spawns a
906
+
* fresh worker (and re-creates the test environment) for every test file. When
907
+
* that repeated startup cost is significant, hint that `isolate: false` would
908
+
* reuse workers across files.
909
+
*/
910
+
private reportIsolateDiagnostic(files: File[]): void {
911
+
// opt-out via `experimental.diagnostics.isolate`; the timers it relies on
912
+
// are only shown for a full (non-watch) run
913
+
if (this.ctx.config.watch || !this.ctx.config.experimental.diagnostics.isolate) {
914
+
return
915
+
}
916
+
917
+
const state = this.ctx.state
918
+
const numWorkers = state.workersSpawned
919
+
// only meaningful when at least one non-browser project isolates workers
920
+
// without the user having explicitly chosen isolation
921
+
const isolates = this.ctx.projects.some(
922
+
project => project.config.isolate
923
+
&& !project.config.browser.enabled
924
+
&& !project.config.providedOptions.isolate,
925
+
)
926
+
if (!numWorkers || !isolates) {
927
+
return
928
+
}
929
+
930
+
const numFiles = files.length
931
+
// `startupTime` is the summed (across workers) time spent spawning the worker,
932
+
// loading its bundle and setting up the environment. The environment setup is
933
+
// already part of this window, so it is not added separately.
934
+
const startupTime = state.startupTime
935
+
const avgStartup = startupTime / numWorkers
936
+
937
+
// with `isolate: false` the same files would run in ~`parallelism` reused
938
+
// workers instead of spawning a fresh worker for every file
939
+
const parallelism = Math.max(1, Math.min(numFiles, this.getEffectiveMaxWorkers()))
940
+
941
+
// nothing was actually spawned per-file (e.g. a single worker handled everything)
942
+
if (numWorkers <= parallelism) {
943
+
return
944
+
}
945
+
946
+
// Spawns are spread across ~`parallelism` lanes, so the wall-clock cost is the
947
+
// summed startup divided by parallelism. Reusing workers leaves ~1 spawn per
948
+
// lane, so the reducible wall-clock time is the rest.
949
+
const wallStartup = startupTime / parallelism
950
+
951
+
// Reused workers also keep evaluated modules alive, so every module a later
952
+
// file would re-evaluate is saved as well. Per-module evaluation times are
953
+
// only collected when `experimental.importDurations` is enabled — without
954
+
// them the spawn saving is reported as a lower bound ("at least").
955
+
const measuresModules = this.ctx.config.experimental.importDurations.limit > 0
956
+
let moduleSavings = 0
957
+
if (measuresModules) {
958
+
// vm pools re-create the module graph per VM context regardless of
959
+
// `isolate`, so only files of `forks`/`threads` projects count
960
+
const eligibleProjects = new Set(this.ctx.projects
961
+
.filter(project => (project.config.pool === 'forks' || project.config.pool === 'threads')
962
+
&& project.config.isolate
963
+
&& !project.config.browser.enabled
964
+
&& !project.config.providedOptions.isolate)
965
+
.map(project => project.name))
966
+
const moduleSelfTimes = new Map<string, number[]>()
967
+
for (const file of files) {
968
+
if (!eligibleProjects.has(file.projectName || '') || !file.importDurations) {
969
+
continue
970
+
}
971
+
for (const moduleId in file.importDurations) {
972
+
let times = moduleSelfTimes.get(moduleId)
973
+
if (!times) {
974
+
times = []
975
+
moduleSelfTimes.set(moduleId, times)
976
+
}
977
+
times.push(file.importDurations[moduleId].selfTime)
978
+
}
979
+
}
980
+
moduleSavings = estimateModuleEvaluationSaving(moduleSelfTimes.values(), parallelism)
981
+
}
982
+
983
+
const estimatedSavings = wallStartup - avgStartup + moduleSavings
984
+
985
+
if (!isSavingWorthHinting(estimatedSavings, this.end - this.start)) {
986
+
return
987
+
}
988
+
989
+
this.log()
990
+
this.log(
991
+
padSummaryTitle('Isolate'),
992
+
c.yellow(`${numWorkers} workers spawned`)
993
+
+ c.dim(` · ~${formatTime(avgStartup)} startup each (spawn + environment, per file)`),
994
+
)
995
+
this.log(
996
+
padSummaryTitle(''),
997
+
c.dim(`${measuresModules ? '' : 'at least '}~${formatTime(estimatedSavings)} faster with `)
998
+
+ c.yellow('isolate: false')
999
+
+ c.dim(measuresModules
1000
+
? ' — reuses workers across files and evaluates shared modules once per worker'
1001
+
: ' — reuses workers across files instead of one per file'),
1002
+
)
661
1003
}
662
1004
663
1005
private reportImportDurations() {
+288
packages/vitest/src/node/reporters/diagnostics.ts
+288
packages/vitest/src/node/reporters/diagnostics.ts
···
1
+
const DOM_ENVIRONMENTS = new Set(['jsdom', 'happy-dom'])
2
+
3
+
/** Minimum summed environment setup time before the hint is worth printing. */
4
+
const MIN_ENVIRONMENT_TIME = 2_000
5
+
/** Minimum share of the project's tracked time spent setting up environments. */
6
+
const MIN_ENVIRONMENT_SHARE = 0.25
7
+
8
+
/**
9
+
* A hint has to be worth acting on: the estimated saving must be noticeable.
10
+
* Run-to-run noise of real suites is commonly a few percent, so anything below
11
+
* ~5% of the wall time cannot even be confirmed by trying the change - except
12
+
* on long runs, where 10 seconds is worth attention regardless of percentage.
13
+
*/
14
+
export function isSavingWorthHinting(saving: number, executionTime: number): boolean {
15
+
if (saving < 250) {
16
+
return false
17
+
}
18
+
return saving >= executionTime * 0.05 || saving >= 10_000
19
+
}
20
+
21
+
export interface EnvironmentDiagnosticInput {
22
+
name: string
23
+
environment: string
24
+
pool: string
25
+
isolate: boolean
26
+
browser: boolean
27
+
/** The user explicitly configured the pool, so don't suggest changing it. */
28
+
poolProvided: boolean
29
+
/** The user explicitly configured isolation, so don't suggest disabling it. */
30
+
isolateProvided: boolean
31
+
/** Summed time spent creating the environment, across all files. */
32
+
environmentTime: number
33
+
/** Number of files that created an environment. */
34
+
environmentCount: number
35
+
/** Summed time of all tracked phases of this project. */
36
+
trackedTime: number
37
+
/** How many workers the environment setups were spread across. */
38
+
parallelism: number
39
+
/** Wall time of the whole run. */
40
+
executionTime: number
41
+
}
42
+
43
+
export interface EnvironmentDiagnostic {
44
+
name: string
45
+
environment: string
46
+
environmentTime: number
47
+
environmentCount: number
48
+
/** Share of the project's tracked time, 0-1. */
49
+
share: number
50
+
/** Whether suggesting `isolate: false` is appropriate. */
51
+
suggestIsolate: boolean
52
+
}
53
+
54
+
/** Minimum summed import time before the hint is worth printing. */
55
+
const MIN_IMPORT_TIME = 2_000
56
+
/** Minimum share of the project's tracked time spent importing modules. */
57
+
const MIN_IMPORT_SHARE = 0.25
58
+
/**
59
+
* Minimum fraction of module fetches that re-evaluate an already evaluated
60
+
* module. Below this the test files import mostly disjoint graphs and
61
+
* reusing workers would not meaningfully reduce the import work.
62
+
*/
63
+
const MIN_IMPORT_DUPLICATION = 0.2
64
+
65
+
export interface ImportDiagnosticInput {
66
+
name: string
67
+
pool: string
68
+
isolate: boolean
69
+
browser: boolean
70
+
/** The user explicitly configured isolation, so don't suggest disabling it. */
71
+
isolateProvided: boolean
72
+
/** Summed time spent importing test files and their module graphs. */
73
+
importTime: number
74
+
/** Summed time of all tracked phases of this project. */
75
+
trackedTime: number
76
+
/**
77
+
* How many times each module was served to a worker. With `isolate: true`
78
+
* every test file re-fetches (and re-evaluates) its whole module graph, so
79
+
* counts above the worker parallelism are repeated evaluations of the same
80
+
* module that `isolate: false` would avoid.
81
+
*/
82
+
fetchCounts: number[]
83
+
fileCount: number
84
+
/** How many workers the imports were spread across. */
85
+
parallelism: number
86
+
/** Wall time of the whole run. */
87
+
executionTime: number
88
+
}
89
+
90
+
export interface ImportDiagnostic {
91
+
name: string
92
+
importTime: number
93
+
/** Share of the project's tracked time, 0-1. */
94
+
share: number
95
+
totalFetches: number
96
+
uniqueModules: number
97
+
/** Fraction of fetches that re-evaluated an already evaluated module, 0-1. */
98
+
duplication: number
99
+
/** Estimated wall-clock saving of `isolate: false`. */
100
+
estimatedSaving: number
101
+
}
102
+
103
+
/**
104
+
* Detects projects where test files repeatedly evaluate the same module
105
+
* graph. Typical for barrel-file imports: every test file pulls hundreds of
106
+
* shared modules to use a few of them, and `isolate: true` re-evaluates that
107
+
* graph for every file. The duplication is measured from the server-side
108
+
* fetch counts, so suites with disjoint per-file graphs - where reusing
109
+
* workers would not help - stay quiet.
110
+
*/
111
+
export function getImportDiagnostics(
112
+
projects: ImportDiagnosticInput[],
113
+
): ImportDiagnostic[] {
114
+
const diagnostics: ImportDiagnostic[] = []
115
+
for (const project of projects) {
116
+
if (
117
+
// vm pools re-create the module graph per VM context regardless of
118
+
// `isolate`, so reusing workers would not reduce the import work
119
+
(project.pool !== 'forks' && project.pool !== 'threads')
120
+
|| !project.isolate
121
+
|| project.isolateProvided
122
+
|| project.browser
123
+
|| project.fileCount <= project.parallelism
124
+
|| project.importTime < MIN_IMPORT_TIME
125
+
|| project.trackedTime <= 0
126
+
|| project.importTime / project.trackedTime < MIN_IMPORT_SHARE
127
+
) {
128
+
continue
129
+
}
130
+
const parallelism = Math.max(1, project.parallelism)
131
+
let totalFetches = 0
132
+
let avoidableFetches = 0
133
+
for (const count of project.fetchCounts) {
134
+
totalFetches += count
135
+
// reused workers still fetch a module once per lane that needs it
136
+
avoidableFetches += Math.max(0, count - parallelism)
137
+
}
138
+
if (totalFetches === 0) {
139
+
continue
140
+
}
141
+
const duplication = avoidableFetches / totalFetches
142
+
if (duplication < MIN_IMPORT_DUPLICATION) {
143
+
continue
144
+
}
145
+
// imports are spread across the worker lanes, so the reducible wall time
146
+
// is the duplicated share of the summed import time divided by lanes
147
+
const estimatedSaving = (project.importTime * duplication) / parallelism
148
+
if (!isSavingWorthHinting(estimatedSaving, project.executionTime)) {
149
+
continue
150
+
}
151
+
diagnostics.push({
152
+
name: project.name,
153
+
importTime: project.importTime,
154
+
share: project.importTime / project.trackedTime,
155
+
totalFetches,
156
+
uniqueModules: project.fetchCounts.length,
157
+
duplication,
158
+
estimatedSaving,
159
+
})
160
+
}
161
+
return diagnostics
162
+
}
163
+
164
+
/**
165
+
* Estimates the wall-clock time saved by evaluating shared modules once per
166
+
* worker instead of once per test file. `moduleSelfTimes` holds, per module,
167
+
* the module's own evaluation time in every test file that evaluated it.
168
+
* Reused workers keep ~`parallelism` evaluations of each module (one per
169
+
* lane); the rest of the summed time is avoidable and spread across the
170
+
* lanes.
171
+
*/
172
+
export function estimateModuleEvaluationSaving(
173
+
moduleSelfTimes: Iterable<number[]>,
174
+
parallelism: number,
175
+
): number {
176
+
const lanes = Math.max(1, parallelism)
177
+
let avoidable = 0
178
+
for (const times of moduleSelfTimes) {
179
+
if (times.length <= lanes) {
180
+
continue
181
+
}
182
+
let sum = 0
183
+
for (const time of times) {
184
+
sum += time
185
+
}
186
+
avoidable += (sum * (times.length - lanes)) / times.length
187
+
}
188
+
return avoidable / lanes
189
+
}
190
+
191
+
/** Minimum summed transform time before the hint is worth printing. */
192
+
const MIN_TRANSFORM_TIME = 2_000
193
+
/** Minimum share of the project's tracked time spent transforming modules. */
194
+
const MIN_TRANSFORM_SHARE = 0.25
195
+
196
+
export interface TransformDiagnosticInput {
197
+
name: string
198
+
/** Summed time spent transforming and serving modules. */
199
+
transformTime: number
200
+
/** Summed time of all tracked phases of this project. */
201
+
trackedTime: number
202
+
/** The fs module cache is already enabled - transforms persist across runs. */
203
+
fsModuleCache: boolean
204
+
/** The user explicitly configured the cache, so don't suggest enabling it. */
205
+
fsModuleCacheProvided: boolean
206
+
/** Wall time of the whole run. */
207
+
executionTime: number
208
+
}
209
+
210
+
export interface TransformDiagnostic {
211
+
name: string
212
+
transformTime: number
213
+
/** Share of the project's tracked time, 0-1. */
214
+
share: number
215
+
}
216
+
217
+
/**
218
+
* Detects projects that spend the run transforming modules. Without the fs
219
+
* module cache every `vitest run` starts from scratch and transforms the
220
+
* whole module graph again; `fsModuleCache` persists the results on disk so
221
+
* repeated runs skip them.
222
+
*/
223
+
export function getTransformDiagnostics(
224
+
projects: TransformDiagnosticInput[],
225
+
): TransformDiagnostic[] {
226
+
return projects
227
+
.filter((project) => {
228
+
if (
229
+
project.fsModuleCache
230
+
|| project.fsModuleCacheProvided
231
+
|| project.transformTime < MIN_TRANSFORM_TIME
232
+
|| project.trackedTime <= 0
233
+
|| project.transformTime / project.trackedTime < MIN_TRANSFORM_SHARE
234
+
) {
235
+
return false
236
+
}
237
+
// the next run skips the persisted transforms, so the (mostly serial,
238
+
// main-thread) transform time itself bounds the saving
239
+
return isSavingWorthHinting(project.transformTime, project.executionTime)
240
+
})
241
+
.map(project => ({
242
+
name: project.name,
243
+
transformTime: project.transformTime,
244
+
share: project.transformTime / project.trackedTime,
245
+
}))
246
+
}
247
+
248
+
/**
249
+
* Detects projects where re-creating a DOM environment for every test file
250
+
* dominates the run. With an isolating pool the environment is set up once
251
+
* per file; `vmThreads`/`vmForks` set it up once per worker while still
252
+
* giving every file a fresh VM context.
253
+
*/
254
+
export function getEnvironmentDiagnostics(
255
+
projects: EnvironmentDiagnosticInput[],
256
+
): EnvironmentDiagnostic[] {
257
+
return projects
258
+
.filter((project) => {
259
+
if (
260
+
!DOM_ENVIRONMENTS.has(project.environment)
261
+
|| (project.pool !== 'forks' && project.pool !== 'threads')
262
+
|| !project.isolate
263
+
|| project.browser
264
+
|| project.poolProvided
265
+
|| project.environmentCount <= 1
266
+
|| project.environmentTime < MIN_ENVIRONMENT_TIME
267
+
|| project.trackedTime <= 0
268
+
|| project.environmentTime / project.trackedTime < MIN_ENVIRONMENT_SHARE
269
+
) {
270
+
return false
271
+
}
272
+
// setups are spread across the worker lanes; a vm pool would still pay
273
+
// one setup per lane, so the reducible wall time is the rest
274
+
const parallelism = Math.max(1, project.parallelism)
275
+
const saving
276
+
= project.environmentTime / parallelism
277
+
- project.environmentTime / project.environmentCount
278
+
return isSavingWorthHinting(saving, project.executionTime)
279
+
})
280
+
.map(project => ({
281
+
name: project.name,
282
+
environment: project.environment,
283
+
environmentTime: project.environmentTime,
284
+
environmentCount: project.environmentCount,
285
+
share: project.environmentTime / project.trackedTime,
286
+
suggestIsolate: !project.isolateProvided,
287
+
}))
288
+
}
+49
-1
packages/vitest/src/node/types/config.ts
+49
-1
packages/vitest/src/node/types/config.ts
···
1008
1008
* This will apply `.only` flag and test name pattern across all files without running them.
1009
1009
*/
1010
1010
preParse?: boolean
1011
+
1012
+
/**
1013
+
* Print performance hints after the run when the collected timings show
1014
+
* that a configuration change would make the run significantly faster.
1015
+
* Hints are never printed for options that were set explicitly.
1016
+
*
1017
+
* Set to `false` to disable all hints, or disable them individually:
1018
+
* - `isolate`: hint when `isolate: true` spends a significant amount of
1019
+
* time spawning a fresh worker (and re-creating the environment) for
1020
+
* every test file, estimating how much `isolate: false` could save.
1021
+
* - `environment`: hint when re-creating a DOM environment for every test
1022
+
* file dominates the run and a `vm` pool would set it up once per worker.
1023
+
* - `import`: hint when test files repeatedly evaluate the same module
1024
+
* graph (typical for barrel-file imports) and `isolate: false` would
1025
+
* evaluate it once per worker.
1026
+
* - `transform`: hint when transforming modules dominates the run and
1027
+
* `fsModuleCache` would persist the results across runs.
1028
+
* @default true
1029
+
*/
1030
+
diagnostics?: boolean | {
1031
+
/** @default true */
1032
+
isolate?: boolean
1033
+
/** @default true */
1034
+
environment?: boolean
1035
+
/** @default true */
1036
+
import?: boolean
1037
+
/** @default true */
1038
+
transform?: boolean
1039
+
}
1011
1040
}
1012
1041
1013
1042
/**
···
1264
1293
tagsFilter?: string[]
1265
1294
mergeReportsLabel?: string
1266
1295
1267
-
experimental: Omit<Required<UserConfig>['experimental'], 'importDurations'> & {
1296
+
experimental: Omit<Required<UserConfig>['experimental'], 'importDurations' | 'diagnostics'> & {
1268
1297
importDurations: {
1269
1298
print: boolean | 'on-warn'
1270
1299
limit: number
···
1274
1303
danger: number
1275
1304
}
1276
1305
}
1306
+
diagnostics: {
1307
+
isolate: boolean
1308
+
environment: boolean
1309
+
import: boolean
1310
+
transform: boolean
1311
+
}
1312
+
}
1313
+
1314
+
/**
1315
+
* Options that were explicitly provided by the user, as opposed to resolved
1316
+
* defaults. Used by diagnostics to avoid suggesting changes to options the
1317
+
* user chose deliberately.
1318
+
* @internal
1319
+
*/
1320
+
providedOptions: {
1321
+
pool: boolean
1322
+
isolate: boolean
1323
+
environment: boolean
1324
+
fsModuleCache: boolean
1277
1325
}
1278
1326
1279
1327
cliOptions: CliOptions