chore: migrate tests to vitest (#3381)
managed to migrate the engine leak & memory reporters, although it took a bit of rigamarole. Mainly because, as opposed to karma, the reporter runs in node and not the browser. So I have to track the data needed separately in some global hooks _within_ the browser environment, which the reporter then reads and creates the logs if needed.
However, it seems the memory tracking is regularly reporting >1 mb, so I'm not sure if it's tracking properly or if things have changed with the new browsers. My only theory is that because the timing where memory is read is during an `afterEach` _before_ the test's `afterEach`, it's running before any potential cleanup. The timing of this is not something that's easy to control, unfortunately. I did try to prove this theory by doing the memory analysis on the next test's `beforeEach`, but it didnt seem to change the results, so it may not be an issue.
Implementation is done in `src/spec/vitest/__reporters__/memory.ts` and `src/spec/vitest/__reporters/memory.setup.ts`
---
I noticed CouroutineSpec was an offender for >10mb memory, which spins up additional engines. I added a short 100ms wait after the engine.dispose and the memory usage did drop, so it does seem like this is prone to scheduled garbage collecting.
---
I was able to run the garbage collector (if exposed, currently only on chrome) and this makes the reports more accurate
chore: migrate tests to vitest (#3381)
managed to migrate the engine leak & memory reporters, although it took a bit of rigamarole. Mainly because, as opposed to karma, the reporter runs in node and not the browser. So I have to track the data needed separately in some global hooks _within_ the browser environment, which the reporter then reads and creates the logs if needed.
However, it seems the memory tracking is regularly reporting >1 mb, so I'm not sure if it's tracking properly or if things have changed with the new browsers. My only theory is that because the timing where memory is read is during an `afterEach` _before_ the test's `afterEach`, it's running before any potential cleanup. The timing of this is not something that's easy to control, unfortunately. I did try to prove this theory by doing the memory analysis on the next test's `beforeEach`, but it didnt seem to change the results, so it may not be an issue.
Implementation is done in `src/spec/vitest/__reporters__/memory.ts` and `src/spec/vitest/__reporters/memory.setup.ts`
---
I noticed CouroutineSpec was an offender for >10mb memory, which spins up additional engines. I added a short 100ms wait after the engine.dispose and the memory usage did drop, so it does seem like this is prone to scheduled garbage collecting.
---
I was able to run the garbage collector (if exposed, currently only on chrome) and this makes the reports more accurate
chore: split out vitest tests into visual & unit (#3508)
This splits out our vitest test suite into `unit` and `visual` tests. `unit` will run on chromium, while `visual` will run on chrome. I've removed firefox/webkit browsers as we were not running these in CI, and there's a discussion to be had about whether they even make sense to use in unit tests, or are they actually only valuable for visual tests. If the latter, we need to figure out how to not rely on Chrome for consistent screenshotting.
Tests that are visual simply need to contain `@visual` somewhere in it's test name or `describe` name. Unit tests will run for all tests without `@visual`, while visual tests will run all that contain `@visual`. I had considered splitting them out into different files under different folders, but it's nice to re-use the same setup/teardown logic. Otherwise I had to copy & paste all of that in both places.
I also made some other tweaks to our vitest config:
- I've removed the conditional exemption of `--disable-gpu` flag for Chromium on macOS, which although slower it ensures the visual tests pass. I suppose this is because it uses a software renderer.
- Changed the test pooling from `thread` to the default. I never saw a difference in speed but in theory the default (`fork`) is supposed to be faster
- Enabled fileParallelism for tests. I had run into WebGL context issues before, but now they seem to be gone, so this should lead to faster tests
- I've switched from Chrome to actual Chromium for the visual tests, and then lowered the tolerance on the TextSpec so that it could pass both in Windows/Ubuntu CI as well as macOS locally. I'm fine to undo this, but a lower tolerance on font tests might make sense to do anyway and this frees us from relying on Windows+Chrome to run these tests.
