Let's help help help devs.
Target: <1000ms. Several important LLM CLI tools take multiple seconds. PRs welcome ❤️
| library | cold | warm (10 runs) | version | measured on |
|---|---|---|---|---|
vllm --help | 15095ms | 7172ms | 0.22.1+cpu | 2026-06-05T15:47Z |
VLMEvalKit --help | 15757ms | 6397ms | v0.2 | 2026-06-05T15:50Z |
sglang --help | 28834ms | 5252ms | v0.5.12.post1 | 2026-06-05T15:45Z |
tensorrt-llm --help | 6722ms | 2109ms | 1.2.1 | 2026-06-05T15:43Z |
datasets --help | 3092ms | 788ms | 5.0.0 | 2026-06-05T15:38Z |
llm --help | 1399ms | 618ms | 0.31 | 2026-06-05T15:38Z |
openai --help | 1201ms | 525ms | 2.34.0 | 2026-06-07T12:06Z |
hf --help | 1148ms | 392ms | 1.18.0 | 2026-06-05T15:37Z |
langchain-cli --help | 863ms | 295ms | 0.0.37 | 2026-06-05T15:39Z |
lm-eval --help | 1577ms | 246ms | 0.4.12 | 2026-06-05T15:40Z |
ollama --help | 15ms | 14ms | 0.30.5 | 2026-06-05T15:37Z |
llama.cpp --help | 14ms | 11ms | b9529 | 2026-06-05T15:39Z |
transformers --help | 1ms | 0ms | 5.10.2 | 2026-06-05T15:41Z |