How much electricity does it burn? And how much does it cost to let it run overnight? I feel like this is the same problem that the companies have, its way too expensive to use it on a lots of things and it is way cheaper to use it on someone else’s server…
Comment on Is anyone self-hosting their own AI?
trougnouf@lemmy.world 3 days ago
I use Qwen 3.8 27b Q4 with 100k context (llama.cpp and the Mistral AI harness) on a 24 GB AMD Radeon 7900 XTX and it’s been great. It can handle pretty much anything I throw at it and I can let it work indepedently overnight / over days on long tasks.
dieTasse@feddit.org 2 days ago
trougnouf@lemmy.world 2 days ago
It uses 261 W, probably more expensive than getting a 2nd Mistral subscription with the cost of power where I live (Belgium), but it will be free heat once I have solar panels and a battery.
dieTasse@feddit.org 2 days ago
Thank you, that is not as bad as I expected. 👍
melfie@lemmy.zip 3 days ago
I have nearly the same setup, except OpenCode for the harness. I run 200k context for anything that doesn’t need vision, or 150k otherwise. Typically get about 30 t/s decode, starts at close to 800 t/s for prefill and drops to the 200s for a long context. I’d imagine huger decode speeds are possible with some tinkering, but I haven’t gotten around to it.
trougnouf@lemmy.world 2 days ago
I get about 45 t/s after some tweaking, getting the thermals under control helped a lot; mainly by reducing the power consumption to 261 W (and downclocking a little bit)