Q4 isn't an extreme quant, and I average 75 toks/s on code, 45 tok/s on prose wi... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

naasking 18 days ago | parent | context | favorite | on: Speculative Decoding in vLLM on AMD GPUs

Q4 isn't an extreme quant, and I average 75 toks/s on code, 45 tok/s on prose with MTP.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact