oxc/justfile
no-yan 07196ae0a8
perf(justfile): optimize submodule cloining with shallow fetch (#6292)
fix #6291, where `just submodules` takes around 8 minutes to clone
repositories.

This change reduces the clone time from around 8 minutes to 42 seconds
in reported scenario.
 
<img width="534" alt="image"
src="https://github.com/user-attachments/assets/bf74e412-f1ed-419e-b180-09ed96a8eb67">

<details><summary>Command log: time just submodules</summary>
<p>

```
❯ time just submodules
just clone-submodule tasks/coverage/test262 git@github.com:tc39/test262.git d62fa93c8f9ce5e687c0bbaa5d2b59670ab2ff60
cd tasks/coverage/test262 || git init tasks/coverage/test262
bash: 1 行: cd: tasks/coverage/test262: No such file or directory
Initialized empty Git repository in /Users/noyan/ghq/github.com/oxc-project/oxc/tasks/coverage/test262/.git/
cd tasks/coverage/test262 && git remote add origin git@github.com:tc39/test262.git || true
cd tasks/coverage/test262 && git fetch --depth=1 origin d62fa93c8f9ce5e687c0bbaa5d2b59670ab2ff60 && git reset --hard d62fa93c8f9ce5e687c0bbaa5d2b59670ab2ff60
remote: Enumerating objects: 53504, done.
remote: Counting objects: 100% (53504/53504), done.
remote: Compressing objects: 100% (19436/19436), done.
remote: Total 53504 (delta 39436), reused 39428 (delta 33947), pack-reused 0 (from 0)
Receiving objects: 100% (53504/53504), 12.32 MiB | 9.06 MiB/s, done.
Resolving deltas: 100% (39436/39436), done.
From github.com:tc39/test262
 * branch              d62fa93c8f9ce5e687c0bbaa5d2b59670ab2ff60 -> FETCH_HEAD
Updating files: 100% (51664/51664), done.
HEAD is now at d62fa93c [explicit-resource-management] Complete exception handling
just clone-submodule tasks/coverage/babel git@github.com:babel/babel.git 3bcfee232506a4cebe410f02042fb0f0adeeb0b1
cd tasks/coverage/babel || git init tasks/coverage/babel
bash: 1 行: cd: tasks/coverage/babel: No such file or directory
Initialized empty Git repository in /Users/noyan/ghq/github.com/oxc-project/oxc/tasks/coverage/babel/.git/
cd tasks/coverage/babel && git remote add origin git@github.com:babel/babel.git || true
cd tasks/coverage/babel && git fetch --depth=1 origin 3bcfee232506a4cebe410f02042fb0f0adeeb0b1 && git reset --hard 3bcfee232506a4cebe410f02042fb0f0adeeb0b1
remote: Enumerating objects: 36357, done.
remote: Counting objects: 100% (36357/36357), done.
remote: Compressing objects: 100% (21830/21830), done.
remote: Total 36357 (delta 8527), reused 30780 (delta 7455), pack-reused 0 (from 0)
Receiving objects: 100% (36357/36357), 9.34 MiB | 9.24 MiB/s, done.
Resolving deltas: 100% (8527/8527), done.
From github.com:babel/babel
 * branch              3bcfee232506a4cebe410f02042fb0f0adeeb0b1 -> FETCH_HEAD
Updating files: 100% (31735/31735), done.
HEAD is now at 3bcfee23 Fix printing of TS `infer` in compact mode (#16788)
just clone-submodule tasks/coverage/typescript git@github.com:microsoft/TypeScript.git a709f9899c2a544b6de65a0f2623ecbbe1394eab
cd tasks/coverage/typescript || git init tasks/coverage/typescript
bash: 1 行: cd: tasks/coverage/typescript: No such file or directory
Initialized empty Git repository in /Users/noyan/ghq/github.com/oxc-project/oxc/tasks/coverage/typescript/.git/
cd tasks/coverage/typescript && git remote add origin git@github.com:microsoft/TypeScript.git || true
cd tasks/coverage/typescript && git fetch --depth=1 origin a709f9899c2a544b6de65a0f2623ecbbe1394eab && git reset --hard a709f9899c2a544b6de65a0f2623ecbbe1394eab
remote: Enumerating objects: 69880, done.
remote: Counting objects: 100% (69880/69880), done.
remote: Compressing objects: 100% (52067/52067), done.
remote: Total 69880 (delta 17267), reused 39053 (delta 16227), pack-reused 0 (from 0)
Receiving objects: 100% (69880/69880), 32.80 MiB | 8.54 MiB/s, done.
Resolving deltas: 100% (17267/17267), done.
From github.com:microsoft/TypeScript
 * branch                a709f9899c2a544b6de65a0f2623ecbbe1394eab -> FETCH_HEAD
Updating files: 100% (72575/72575), done.
HEAD is now at a709f9899 Update deps, dprint plugins (#59810)
just clone-submodule tasks/prettier_conformance/prettier git@github.com:prettier/prettier.git 52829385bcc4d785e58ae2602c0b098a643523c9
cd tasks/prettier_conformance/prettier || git init tasks/prettier_conformance/prettier
bash: 1 行: cd: tasks/prettier_conformance/prettier: No such file or directory
Initialized empty Git repository in /Users/noyan/ghq/github.com/oxc-project/oxc/tasks/prettier_conformance/prettier/.git/
cd tasks/prettier_conformance/prettier && git remote add origin git@github.com:prettier/prettier.git || true
cd tasks/prettier_conformance/prettier && git fetch --depth=1 origin 52829385bcc4d785e58ae2602c0b098a643523c9 && git reset --hard 52829385bcc4d785e58ae2602c0b098a643523c9
remote: Enumerating objects: 10259, done.
remote: Counting objects: 100% (10259/10259), done.
remote: Compressing objects: 100% (6662/6662), done.
remote: Total 10259 (delta 970), reused 8226 (delta 696), pack-reused 0 (from 0)
Receiving objects: 100% (10259/10259), 12.85 MiB | 10.34 MiB/s, done.
Resolving deltas: 100% (970/970), done.
From github.com:prettier/prettier
 * branch            52829385bcc4d785e58ae2602c0b098a643523c9 -> FETCH_HEAD
HEAD is now at 5282938 Release 3.3.3
just submodules  1.56s user 12.29s system 32% cpu 42.408 total
``` 

</p>
</details> 

## Solution
To optimize the cloning process:
- Fetch only the specified commit, eliminating the need for a git clone
(which fetches HEAD and subsequently updates with differences).
- Initialize it with git init when the repository doesn't exist locally.
- Use `git fetch --depth=1` to prevent fetching previous commits.

See: https://stackoverflow.com/a/3489576

I initially suggested using `git fetch --depth=1`, but after testing, I
found that removing clone could speed up the process by about 20
seconds. Therefore, I've changed the approach.
2024-10-05 23:58:03 +08:00

195 lines
5.2 KiB
Makefile
Executable file

#!/usr/bin/env -S just --justfile
set windows-shell := ["powershell"]
set shell := ["bash", "-cu"]
_default:
@just --list -u
alias r := ready
alias c := conformance
alias f := fix
alias new-typescript-rule := new-ts-rule
# Make sure you have cargo-binstall installed.
# You can download the pre-compiled binary from <https://github.com/cargo-bins/cargo-binstall#installation>
# or install via `cargo install cargo-binstall`
# Initialize the project by installing all the necessary tools.
init:
cargo binstall cargo-watch cargo-insta typos-cli cargo-shear dprint -y
# When ready, run the same CI commands
ready:
git diff --exit-code --quiet
typos
just fmt
just check
just test
just lint
just doc
just ast
git status
# Clone or update submodules
submodules:
just clone-submodule tasks/coverage/test262 git@github.com:tc39/test262.git d62fa93c8f9ce5e687c0bbaa5d2b59670ab2ff60
just clone-submodule tasks/coverage/babel git@github.com:babel/babel.git 3bcfee232506a4cebe410f02042fb0f0adeeb0b1
just clone-submodule tasks/coverage/typescript git@github.com:microsoft/TypeScript.git a709f9899c2a544b6de65a0f2623ecbbe1394eab
just clone-submodule tasks/prettier_conformance/prettier git@github.com:prettier/prettier.git 52829385bcc4d785e58ae2602c0b098a643523c9
# Install git pre-commit to format files
install-hook:
echo -e "#!/bin/sh\njust fmt" > .git/hooks/pre-commit
chmod +x .git/hooks/pre-commit
# --no-vcs-ignores: cargo-watch has a bug loading all .gitignores, including the ones listed in .gitignore
# use .ignore file getting the ignore list
# Run `cargo watch`
watch command:
cargo watch --no-vcs-ignores -i '*snap*' -x '{{command}}'
# Run the example in `parser`, `formatter`, `linter`
example tool *args='':
just watch 'run -p oxc_{{tool}} --example {{tool}} -- {{args}}'
# Generate AST related boilerplate code.
# Run this when AST definition is changed.
ast:
cargo run -p oxc_ast_tools
just check
# Format all files
fmt:
cargo shear --fix # remove all unused dependencies
cargo fmt --all
dprint fmt
# Run cargo check
check:
cargo ck
# Run all the tests
test:
cargo test
# Lint the whole project
lint:
cargo lint -- --deny warnings
[unix]
doc:
RUSTDOCFLAGS='-D warnings' cargo doc --no-deps --document-private-items
[windows]
doc:
$Env:RUSTDOCFLAGS='-D warnings'; cargo doc --no-deps --document-private-items
# Fix all auto-fixable format and lint issues. Make sure your working tree is clean first.
fix:
cargo clippy --fix --allow-staged --no-deps
just fmt
typos -w
git status
# Run all the conformance tests. See `tasks/coverage`, `tasks/transform_conformance`, `tasks/minsize`
coverage:
cargo coverage
cargo run -p oxc_transform_conformance -- --exec
cargo run -p oxc_prettier_conformance
cargo minsize
conformance *args='':
cargo coverage -- {{args}}
# Watch oxlint
watch-oxlint *args='':
just watch 'run -p oxlint -- {{args}}'
# Build oxlint in release build
oxlint:
cargo oxlint
# Get code coverage
codecov:
cargo codecov --html
# Run the benchmarks. See `tasks/benchmark`
benchmark:
cargo benchmark
# Automatically DRY up Cargo.toml manifests in a workspace.
autoinherit:
cargo binstall cargo-autoinherit
cargo autoinherit
# Test Transform
test-transform *args='':
cargo run -p oxc_transform_conformance -- {{args}}
cargo run -p oxc_transform_conformance -- --exec {{args}}
# Install wasm-pack
install-wasm:
cargo binstall wasm-pack
watch-wasm:
cargo watch --no-vcs-ignores -i 'npm/oxc-wasm/**' -- just build-wasm dev
build-wasm mode="release":
wasm-pack build --out-dir ../../npm/oxc-wasm --target web --{{mode}} --scope oxc crates/oxc_wasm
# Generate the JavaScript global variables. See `tasks/javascript_globals`
javascript-globals:
cargo run -p javascript_globals
# Create a new lint rule by providing the ESLint name. See `tasks/rulegen`
new-rule name:
cargo run -p rulegen {{name}}
new-jest-rule name:
cargo run -p rulegen {{name}} jest
new-ts-rule name:
cargo run -p rulegen {{name}} typescript
new-unicorn-rule name:
cargo run -p rulegen {{name}} unicorn
new-react-rule name:
cargo run -p rulegen {{name}} react
new-jsx-a11y-rule name:
cargo run -p rulegen {{name}} jsx-a11y
new-oxc-rule name:
cargo run -p rulegen {{name}} oxc
new-nextjs-rule name:
cargo run -p rulegen {{name}} nextjs
new-jsdoc-rule name:
cargo run -p rulegen {{name}} jsdoc
new-react-perf-rule name:
cargo run -p rulegen {{name}} react-perf
new-n-rule name:
cargo run -p rulegen {{name}} n
new-promise-rule name:
cargo run -p rulegen {{name}} promise
new-vitest-rule name:
cargo run -p rulegen {{name}} vitest
new-security-rule name:
cargo run -p rulegen {{name}} security
clone-submodule dir url sha:
cd {{dir}} || git init {{dir}}
cd {{dir}} && git remote add origin {{url}} || true
cd {{dir}} && git fetch --depth=1 origin {{sha}} && git reset --hard {{sha}}
website path:
cargo run -p website -- linter-rules --table {{path}}/src/docs/guide/usage/linter/generated-rules.md --rule-docs {{path}}/src/docs/guide/usage/linter/rules
cargo run -p website -- linter-cli > {{path}}/src/docs/guide/usage/linter/generated-cli.md
cargo run -p website -- linter-schema-markdown > {{path}}/src/docs/guide/usage/linter/generated-config.md