Kilo Gateway
An NVIDIA developer-forums performance report from May 4, 2026 documents running the 128B Mistral Medium 3.5 weights in NVFP4 quantization alongside Mistral's official EAGLE draft model on a single DGX Spark / GB10 node using vLLM. The test reported target model memory of 72.91 GiB and a KV cache supporting roughly 89,500 tokens at 16k context. Generation benchmarks using temperature 0.0 measured 6.9 tok/s on a 256-token Q&A, 8.8–8.9 tok/s on 512–701 token code and JSON outputs, and 9.1 tok/s on a 2048-token long-code task. The report notes that the --load-format fastsafetensors loader caused OOM instability at 16k context while --load-format auto was stable but slower.
