Let's help help help devs.
Target: <200ms. Several important LLM CLI tools take multiple seconds. PRs welcome ❤️
| library | cold | warm (10 runs) | version | measured on |
|---|---|---|---|---|
vllm --help | 18115ms | 8157ms | 0.19.0+cpu | 2026-04-14T14:02Z |
sglang --help | 13130ms | 5464ms | v0.5.10.post1 | 2026-04-14T13:58Z |
VLMEvalKit --help | 13244ms | 5338ms | v0.2 | 2026-04-14T14:04Z |
tensorrt-llm --help | 6517ms | 2183ms | 1.2.0 | 2026-04-14T13:57Z |
datasets --help | 3241ms | 975ms | 4.8.4 | 2026-04-14T13:51Z |
llm --help | 1174ms | 539ms | 0.30 | 2026-04-14T13:51Z |
openai --help | 1069ms | 535ms | 2.31.0 | 2026-04-14T13:52Z |
hf --help | 1351ms | 389ms | 1.10.2 | 2026-04-14T13:50Z |
langchain-cli --help | 844ms | 257ms | 0.0.37 | 2026-04-14T13:52Z |
lm-eval --help | 751ms | 251ms | 0.4.11 | 2026-04-14T13:53Z |
llama.cpp --help | 16ms | 14ms | b8784 | 2026-04-14T13:53Z |
ollama --help | 14ms | 12ms | 0.20.7 | 2026-04-14T13:50Z |
transformers --help | 1ms | 0ms | 5.5.4 | 2026-04-14T13:53Z |