模型长什么样,机房就得长什么样
prefill 拼算力,decode 拼内存带宽,context 每长一倍,一张卡能同时服务的人就少一半。模型怎么服务用户,决定了机房要堆多少内存和带宽。
prefill 拼算力,decode 拼内存带宽,context 每长一倍,一张卡能同时服务的人就少一半。模型怎么服务用户,决定了机房要堆多少内存和带宽。
Prefill is limited by compute, decode by memory bandwidth, and every doubling of context halves how many users a GPU can serve at once. How a model serves people decides how much memory and bandwidth the building has to hold.
© Xingfan Xia 2024 - 2026 · CC BY-NC 4.0