Follow-on from #2751. Further shards linter benchmarks so each fixture runs in its own job.
This reduces total time to run benchmarks by another ~75 secs. So approx 2.5 mins shaved off in total.
This PR shards benchmarks when running on CI. Each benchmark (parser, minifier etc) runs as a separate job, and then a final job combines the results and uploads to Codspeed.
A bit of a hacky implementation. Uses a small NodeJS HTTP server to intercept the results from `codspeed-runner`, and then another NodeJS script to combine them all together, and upload to CodSpeed.
I will submit PRs on Codspeed's runner + action to do it properly, but as I imagine it'll be a slow process getting that merged upstream, I wanted to see if it worked first. We can replace this once it's supported upstream.
Sharding only reduces total time to run the benchmarks by about 70 secs at present, because linter benchmark takes 6 mins alone and holds up the whole process (all the rest are done in ~2 mins). If we can split up the linter benchmark, we can likely get total run time down to around 3 mins. I'll try that in a follow-on PR.
I guess the other upside is we can now add as many benchmarks as we like with impunity - they'll run in parallel, and so won't slow things down overall.
#2724 added CodSpeed benchmarks for NodeJS `oxc-parser`.
Unfortunately it turns out CodSpeed's results are wildly inaccurate.
Unclear why, but have raised an issue with CodSpeed
(https://github.com/CodSpeedHQ/action/issues/96). In meantime it seems
best to remove the benchmarks as they're not useful at present.
Closes#2616.
Adds benchmarks for NodeJS NAPI build. Measurement includes `JSON.parse`
of the AST on JS side, since that's how it'll be used 99% of the time.
Benchmarks run against same files as Rust parser benchmarks, so we can
see the overhead of transferring AST to JS.
I've recently switched over from hacking on a fork to working directly
on upstream, in order to be able to use Graphite.
One thing I miss now is ability to create branches and run CodSpeed on
them while developing. Compared to running benches locally, CodSpeed has
the advantage of giving you a visual history, showing how each commit
you push moves the dials.
This PR makes the benchmarks run for any branch named `bench-*`, in
order to enable this workflow on upstream, without having to open a
draft PR for every experiment.
Bumps [CodSpeedHQ/action](https://github.com/codspeedhq/action) from 1
to 2.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/codspeedhq/action/releases">CodSpeedHQ/action's
releases</a>.</em></p>
<blockquote>
<h2>v2.0.0</h2>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/CodSpeedHQ/action/compare/v1...v2.0.0">https://github.com/CodSpeedHQ/action/compare/v1...v2.0.0</a></p>
<p>This major update greatly simplifies the action and moves all the
core logic to the <a
href="https://github.com/CodSpeedHQ/runner">https://github.com/CodSpeedHQ/runner</a>
repo.</p>
<p>Updating is as easy as changing <code>CodSpeedHQ/action@v1</code> to
<code>CodSpeedHQ/action@v2</code>.</p>
<h2>What's changed</h2>
<ul>
<li>feat: integrate the native runner with a composite action by <a
href="https://github.com/art049"><code>@art049</code></a> and <a
href="https://github.com/adriencaccia"><code>@adriencaccia</code></a>
in <a
href="https://redirect.github.com/CodSpeedHQ/action/pull/86">CodSpeedHQ/action#86</a></li>
</ul>
<h3>Breaking changes</h3>
<ul>
<li><code>upload_url</code> input is now <code>upload-url</code></li>
<li><code>pytest-codspeed</code> is no longer installed automatically by
this action</li>
</ul>
<h2>v1.8.1</h2>
<h2>What's Changed</h2>
<ul>
<li>fix: update apt registry before installing valgrind by <a
href="https://github.com/adriencaccia"><code>@adriencaccia</code></a>
in <a
href="https://redirect.github.com/CodSpeedHQ/action/pull/80">CodSpeedHQ/action#80</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/CodSpeedHQ/action/compare/v1.8.0...v1.8.1">https://github.com/CodSpeedHQ/action/compare/v1.8.0...v1.8.1</a></p>
<h2>v1.8.0</h2>
<h2>What's Changed</h2>
<h3>Added</h3>
<ul>
<li>Add the <code>working-directory</code> input parameter by <a
href="https://github.com/art049"><code>@art049</code></a> in <a
href="https://redirect.github.com/CodSpeedHQ/action/pull/77">CodSpeedHQ/action#77</a></li>
</ul>
<h3>Internals</h3>
<ul>
<li>Update documentation examples by <a
href="https://github.com/adriencaccia"><code>@adriencaccia</code></a>
in <a
href="https://redirect.github.com/CodSpeedHQ/action/pull/76">CodSpeedHQ/action#76</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/CodSpeedHQ/action/compare/v1.7.1...v1.8.0">https://github.com/CodSpeedHQ/action/compare/v1.7.1...v1.8.0</a></p>
<h2>v1.7.1</h2>
<h2>🎉 What's Changed</h2>
<ul>
<li>Prepare for release on GitHub Action Marketplace</li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/CodSpeedHQ/action/compare/v1.7.0...v1.7.1">https://github.com/CodSpeedHQ/action/compare/v1.7.0...v1.7.1</a></p>
<h2>v1.7.0</h2>
<h2>🎉 What's Changed</h2>
<ul>
<li>
<p>A <a href="https://github.com/CodSpeedHQ/valgrind-codspeed">fork of
valgrind</a> is now used, allowing us to <a
href="https://docs.codspeed.io/features/trace-generation/">generate
execution traces</a> with Node and Python.</p>
</li>
<li>
<p>V8 flags previously set as an environment variable
<code>CODSPEED_V8_FLAGS</code> are now handled directly by the
integration library and injected to Nodejs through our <a
href="https://github.com/CodSpeedHQ/action/blob/main/dist/bin/node">custom
node executable</a>.</p>
</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/CodSpeedHQ/action/blob/main/CHANGELOG.md">CodSpeedHQ/action's
changelog</a>.</em></p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="3cb272459e"><code>3cb2724</code></a>
Release v2.0.1 🚀</li>
<li><a
href="6790ba286f"><code>6790ba2</code></a>
feat: simplify action output</li>
<li><a
href="0c6eabe26d"><code>0c6eabe</code></a>
fix: quiet the codspeed-runner installation</li>
<li><a
href="4f1d7b4ec1"><code>4f1d7b4</code></a>
Release v2.0.0 🚀</li>
<li><a
href="de30d2b7e6"><code>de30d2b</code></a>
chore: use full version in script and stop relying on pnpm</li>
<li><a
href="5a53587953"><code>5a53587</code></a>
chore(ci): add multiline check</li>
<li><a
href="e2bcafd32c"><code>e2bcafd</code></a>
feat: integrate the native runner with a composite action</li>
<li><a
href="e8372ebb5f"><code>e8372eb</code></a>
chore: update readme with vitest</li>
<li>See full diff in <a
href="https://github.com/codspeedhq/action/compare/v1...v2">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
benchmark execution time reduced from 15 mins to 12 mins. Time reduction
should be more noticeable for large code changes which require longer
compile time.
rustup show always install the profiles set in rust-toolchain.toml,
I added an action to remove this line so we can install components selectively.
This should reduce Rust installation time, which can be significant on Windows.
