What’s your rig?
Comment on Is anyone self-hosting their own AI?
hoshikarakitaridia@lemmy.world 13 hours ago
- Qwen3.6 35b a3b Q4_K_M running on llama.cpp
- Open-WebUI as a frontend
- integrations with an embedding model and searxng & crawl4ai
- doing lots of different stuff with it, asking basic stupid questions, recipes, tech stuff, etc
BooBees@fedinsfw.app 13 hours ago
hoshikarakitaridia@lemmy.world 2 hours ago
1070ti and some hexacore something cpu that’s alright with around 24gb ddr4, I think one of the slower speeds. I also got a faster PC with another llama.cpp instance and I vibe coded a quick and dirty proxy which I can use to switch between the different backends. It’s genuinely useful because I got code editors but also my Open-WebUI using it.
Sounds elaborate but generally it’s all pretty primitive.
jwiggler@lemmy.dbzer0.com 12 hours ago
Pretty much my setup as well. Though I’m using qwen3.8 27b-Q4_K_M under ollama. Everything running with docker. Add nginx and either a domain+port forwarding or zerotier/tailscale you’ve got an app you can get to from outside your house.
Right now not sure how I’m going to leverage it not probably to streamline my self hosted stuff, help with config mgmt and documentation, web page summarization, etc. we’ll see how it does, just got it setup yesterday
robber@lemmy.ml 4 hours ago
In case you are unaway, frieds don’t let friends use ollama. The llama.cpp has become pretty much equally easy to use, plus since you use Docker there’s no need to compile anything.
hoshikarakitaridia@lemmy.world 2 hours ago
Oh damn what a useful link. Gonna check out those orchestrators, maybe I can replace my janky ass proxy.
jwiggler@lemmy.dbzer0.com 3 hours ago
Oh shit I wasn’t aware! Thanks for the heads up, I’ll swap to llama.cpp
mortalic@lemmy.world 9 hours ago
Random Question, I’ve been trying to offload some of my stuff handled by claud to basically this exact setup. However one of the things I manage has a pair of MCP servers that are required. I might not be smart enough, but how do you get Qwen to use an MCP?
hoshikarakitaridia@lemmy.world 2 hours ago
MCP is a really dumb server that just takes requests and sends back data.
Somewhere in your frontend settings (Open-WebUI has it under “integrations” I believe, some backends can set these things too) you see a section called “MCP server”. You need to keep your MCP server running somewhere (it’s very slim, so not a huge deal usually) and then in those settings you put your IP and port for the server, for example “192.168.1.15:8080”. Usually if you did it right (sometimes you can check if it works in the settings), you can ask the model if it has access to the tools and it will tell you.
What’s happening in the background is that every conversation the AI does, it gets a short preamble with some context about what it can do, and if you connect web search, custom skills or an MCP server (it’s just skills but they are run through their own server basically), the AI sees all that in this preamble if it’s set up correctly. And then when it feels like it needs to use those tools, it will check the description on how to do that, and then it uses them.
mortalic@lemmy.world 38 minutes ago
Thank you, I’ll give this a go
eodur@piefed.social 11 hours ago
I’ve got similar running on my RX 7600 XT. It is not a great experience for interactive use, but fine for async tasks. I’m also running nemo-speak for live local dictation and meeting transcription.
hoshikarakitaridia@lemmy.world 2 hours ago
Yeah running in the background type beat. Not gonna trust this with anything important but it’s great to have someone just do shit for you or get a second opinion.
BuckFutter@timeyak.com 11 hours ago
Yah doing similar on 5060ti.
hoshikarakitaridia@lemmy.world 2 hours ago
That’s gonna be quick af with the same models.
BuckFutter@timeyak.com 1 hour ago
It is, i also use the gemma4b model on it for description generation for my immich photos and its super fast.
wreckedcarzz@lemmy.world 3 hours ago
Very similar, but I’m using Gemma4 12b. I could run 26b but that would exhaust my vram after a short interaction. And ollama. Open WebUI + Home Assistant hooked up with it, a couple very basic scripts I threw together, and connected to Kagi.
I’m trying to get my folks off the Gemini/Google bandwagon for the privacy aspect, but also because it’s just cool from a tech geek perspective. For me, any questions throughout the day that I want info on, but not enough to do a deep-dive for an hour+, gets fed to Steve, my models name. Or I’ll use it as a springboard and then properly read the citations for more info. And I feed it HA automations when I get stuck scratching my head on an edge case or whatever - it’s alright but helps me explain what I want from the automation, and when it understands my goal, great, and when it doesn’t I reflect on what I have setup and can usually go ‘ah ha’ and fix the issue. It’s like a rubber ducky/pair programmer that you can both think is an idiot sometimes, but it doesn’t judge you when you’re the idiot.
hoshikarakitaridia@lemmy.world 2 hours ago
That’s a pretty good way to describe those kinds of models. Somewhere between rubber duck debugging and a really lazy intern. Better than an algorithm, faster than a human, but less trustworthy than an intern. It’s great if you know how and when to use it. Maybe it’ll become an intern soon enough, but for now let’s focus on decompiling adobe into bankruptcy.