Let's help help help devs.
Target: <200ms. Several important LLM CLI tools take multiple seconds. PRs welcome ❤️
| library | cold | warm (10 runs) | version | measured on |
|---|---|---|---|---|
vllm --help | 14015ms | 6269ms | 0.20.2+cpu | 2026-05-12T09:22Z |
sglang --help | 13656ms | 5536ms | v0.5.11 | 2026-05-12T09:19Z |
VLMEvalKit --help | 13984ms | 5484ms | v0.2 | 2026-05-12T09:24Z |
tensorrt-llm --help | 5743ms | 2190ms | 1.2.1 | 2026-05-12T09:17Z |
datasets --help | 3424ms | 973ms | 4.8.5 | 2026-05-12T09:10Z |
llm --help | 1412ms | 644ms | 0.31 | 2026-05-12T09:11Z |
hf --help | 1038ms | 380ms | 1.14.0 | 2026-05-12T09:10Z |
lm-eval --help | 1910ms | 330ms | 0.4.12 | 2026-05-12T09:13Z |
langchain-cli --help | 819ms | 262ms | 0.0.37 | 2026-05-12T09:12Z |
ollama --help | 15ms | 13ms | 0.23.2 | 2026-05-12T09:09Z |
llama.cpp --help | 13ms | 12ms | b9114 | 2026-05-12T09:12Z |
transformers --help | 0ms | 0ms | 5.8.0 | 2026-05-12T09:13Z |
openai --help | 1ms | 0ms | 2.36.0 | 2026-05-12T09:11Z |