Comment on Is anyone self-hosting their own AI?

<- View Parent
melfie@lemmy.zip ⁨3⁩ ⁨days⁩ ago

I have nearly the same setup, except OpenCode for the harness. I run 200k context for anything that doesn’t need vision, or 150k otherwise. Typically get about 30 t/s decode, starts at close to 800 t/s for prefill and drops to the 200s for a long context. I’d imagine huger decode speeds are possible with some tinkering, but I haven’t gotten around to it.

source
Sort:hotnewtop