Comment on Is anyone self-hosting their own AI?
binaryqueen@programming.dev 4 days ago
I don’t want to create an extra post, so I ask here.
Where do you guys get your hardware from, for an affordable price?
I’d like to run at least a 27b model but my hardware is too weak. I’m on a 16GB unified mem, M4. Tried running Bonsai 27b (it’s Qwen architecture) but I had to close all other apps and it is not in a state that I could work with it.
ikidd@lemmy.dbzer0.com 4 days ago
A Tesla V100 is fairly affordable but older and won’t run newer quants like FP8 or attention kernel routines. But it will run a pile of models, even most new models are compatible or usable.
binaryqueen@programming.dev 4 days ago
I will ask Santa Claus for that GPU <3