Let's help help help devs.
Target: <200ms. Several important LLM CLI tools take multiple seconds. PRs welcome ❤️
| library | cold | warm (10 runs) | version | measured on |
|---|---|---|---|---|
vllm --help | 15610ms | 7473ms | 0.14.1+cpu | 2026-01-28T21:50Z |
sglang --help | 13875ms | 5490ms | v0.5.7 | 2026-01-05T00:09Z |
VLMEvalKit --help | 14551ms | 4990ms | v0.2 | 2026-01-05T00:28Z |
transformers --help | 8799ms | 3102ms | 4.57.3 | 2026-01-05T00:23Z |
tensorrt-llm --help | 8208ms | 2372ms | 1.0.0 | 2026-01-05T00:04Z |
datasets --help | 3351ms | 842ms | 4.4.2 | 2026-01-05T00:21Z |
llm --help | 1257ms | 573ms | 0.28 | 2026-01-22T16:52Z |
openai --help | 1591ms | 505ms | 2.14.0 | 2026-01-05T00:11Z |
langchain-cli --help | 749ms | 253ms | 0.0.37 | 2026-01-05T00:03Z |
hf --help | 805ms | 218ms | 1.2.3 | 2026-01-05T00:14Z |
lm-eval --help | 49ms | 43ms | 0.4.9.2 | 2026-01-05T00:08Z |
llama.cpp --help | 26ms | 17ms | b7626 | 2026-01-05T00:12Z |
ollama --help | 14ms | 14ms | 0.13.5 | 2026-01-05T00:30Z |