Comment on Is anyone self-hosting their own AI?
melfie@lemmy.zip 3 days agoI have nearly the same setup, except OpenCode for the harness. I run 200k context for anything that doesn’t need vision, or 150k otherwise. Typically get about 30 t/s decode, starts at close to 800 t/s for prefill and drops to the 200s for a long context. I’d imagine huger decode speeds are possible with some tinkering, but I haven’t gotten around to it.
trougnouf@lemmy.world 2 days ago
I get about 45 t/s after some tweaking, getting the thermals under control helped a lot; mainly by reducing the power consumption to 261 W (and downclocking a little bit)