trougnouf

@trougnouf@lemmy.world

This is a remote user, information on this page may be incomplete. View at Source ↗

⁨Comment⁩ on ⁨Is anyone self-hosting their own AI?⁩ ⁨⁨2⁩ ⁨days⁩ ago⁩:
It uses 261 W, probably more expensive than getting a 2nd Mistral subscription with the cost of power where I live (Belgium), but it will be free heat once I have solar panels and a battery.
⁨Comment⁩ on ⁨Is anyone self-hosting their own AI?⁩ ⁨⁨2⁩ ⁨days⁩ ago⁩:
I get about 45 t/s after some tweaking, getting the thermals under control helped a lot; mainly by reducing the power consumption to 261 W (and downclocking a little bit)
⁨Comment⁩ on ⁨Is anyone self-hosting their own AI?⁩ ⁨⁨3⁩ ⁨days⁩ ago⁩:
I use Qwen 3.8 27b Q4 with 100k context (llama.cpp and the Mistral AI harness) on a 24 GB AMD Radeon 7900 XTX and it’s been great. It can handle pretty much anything I throw at it and I can let it work indepedently overnight / over days on long tasks.