Shoutout to the dev on r/LocalLLaMA who told me to quantize my model to 4-bit back in March
I thought he was nuts, but my 32GB rig in my garage went from crawling to running the whole thing smooth. Now I want to try a bigger model, so what's the next step up from 4-bit without buying new hardware?