what is vllm-int8-kv-cache-for-ampere-sm86 fr?

This project is a patch, a small set of code changes, for vLLM, a popular tool used to run large language models efficiently on GPUs. It specifically targets Ampere generation Nvidia GPUs, a hardware family that lacks support for a newer, more efficient number format called FP8. Because these GPUs cannot use FP8 for a part of the model called the KV cache, which stores information the model needs to remember earlier parts of a conversation, this patch adds an alternative that uses an older, more widely supported number format called INT8 instead. Using INT8 for the KV cache roughly halves the memory needed for it compared to the more common BF16 or F16 formats. Since the KV cache is often what limits how much conversation history or document length a model can handle at once, this patch lets users fit longer contexts into the same amount of GPU memory, at the cost of a small amount of accuracy and some added processing overhead. An optional feature also emulates a different format called FP8-E4M3 specifically for one part of the cache, to preserve more numeric range than plain INT8 alone would allow. The patch builds on earlier work by another developer who created a similar patch for an older version of vLLM, and this project brings that idea forward to a newer vLLM release, version 0.24.0, with some refinements. It was mainly tested using two specific large language models, and the author notes that results may vary with other models. Using this project requires downloading a specific version of the vLLM source code, applying the patch file to it, and then building and installing vLLM from that modified source, which is a developer level task requiring a compatible Nvidia GPU and a working Python and CUDA environment. The README includes benchmark numbers comparing plain BF16 performance against the INT8 patch, showing the tradeoff in speed for the memory savings, along with detailed step by step instructions for calibrating and applying the per layer scaling values the INT8 conversion needs.