Is anyone self-hosting their own AI?

⁨220⁩ ⁨likes⁩

Submitted ⁨⁨3⁩ ⁨weeks⁩ ago⁩ by ⁨StateMachine@lemmy.zip⁩ to ⁨selfhosted@lemmy.world⁩

https://lemmy.zip/pictrs/image/360687a3-031f-432e-aa02-3df6ff606923.avif

Does anyone run their own AI service?

If so,

Which one?

What hardware?

Do you have a way to access it from outside or from a phone?

And, if you’re open to sharing, what do you do with it?

I know AI-enabling hardware is insanely expensive right now, but some of the smaller (<32GB) more efficient models (think deepseek- or qwen-based) don’t absolutely need specialized hardware to run and perform well enough for many purposes.

I am so aware of the absolutely abysmal privacy practices of basically all AI companies that I’m not comfortable using them to help me with web searches and the such, because there’s no way that all this information won’t be going into a profile. This makes me want to have my own “helper” service.

source

Comments

Sort:hotnew top

violetrage@lemmy.blahaj.zone ⁨3⁩ ⁨weeks⁩ ago
I self-host my own neural network, all within the confines of my own skull.

source
- MidnightMarauder@lemmy.dbzer0.com ⁨3⁩ ⁨weeks⁩ ago
  That 33% downtime tough
  
  source
  - frongt@lemmy.zip ⁨3⁩ ⁨weeks⁩ ago
    It’s scheduled maintenance so it doesn’t count toward downtime. Besides, I’m not trying to use it then anyway.
    
    source
    -> View More Comments
  - arudesalad@piefed.ca ⁨3⁩ ⁨weeks⁩ ago
    It’s better than github
    
    source
  - SpaceCowboy@lemmy.ca ⁨2⁩ ⁨weeks⁩ ago
    The downtime isn’t the biggest issue, you can plan around that. The real problem is the random lag on some queries and volatile memory being lost (usually when there’s a context change).
    
    source
  - BartyDeCanter@piefed.social ⁨3⁩ ⁨weeks⁩ ago
    Only 33% downtime? Damn, that must be nice. I really need to get a hardware update.
    
    source
  - andrewoidxyz@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
    Downtime is the friends made along the way
    
    source
- Valmond@lemmy.dbzer0.com ⁨3⁩ ⁨weeks⁩ ago
  How much RAM is it using?
  
  source
  - mike_wooskey@lemmy.thewooskeys.com ⁨3⁩ ⁨weeks⁩ ago
    Made me think of Johnny Mnemonic. www.youtube.com/watch?v=jS2GFqT5ycU
    
    source
    -> View More Comments
- A_norny_mousse@piefed.social ⁨3⁩ ⁨weeks⁩ ago
  I’m just glad that this is both Top & Hot comment. 🧠✊
  
  source
hoshikarakitaridia@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
Qwen3.6 35b a3b Q4_K_M running on llama.cpp

Open-WebUI as a frontend

integrations with an embedding model and searxng & crawl4ai

doing lots of different stuff with it, asking basic stupid questions, recipes, tech stuff, etc
source
- BooBees@fedinsfw.app ⁨3⁩ ⁨weeks⁩ ago
  What’s your rig?
  
  source
  - hoshikarakitaridia@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
    1070ti and some hexacore something cpu that’s alright with around 24gb ddr4, I think one of the slower speeds. I also got a faster PC with another llama.cpp instance and I vibe coded a quick and dirty proxy which I can use to switch between the different backends. It’s genuinely useful because I got code editors but also my Open-WebUI using it.
    
    Sounds elaborate but generally it’s all pretty primitive.
    
    source
- jwiggler@lemmy.dbzer0.com ⁨3⁩ ⁨weeks⁩ ago
  Pretty much my setup as well. Though I’m using qwen3.8 27b-Q4_K_M under ollama. Everything running with docker. Add nginx and either a domain+port forwarding or zerotier/tailscale you’ve got an app you can get to from outside your house.
  
  Right now not sure how I’m going to leverage it not probably to streamline my self hosted stuff, help with config mgmt and documentation, web page summarization, etc. we’ll see how it does, just got it setup yesterday
  
  source
  - robber@lemmy.ml ⁨2⁩ ⁨weeks⁩ ago
    In case you are unaway, frieds don’t let friends use ollama. The llama.cpp has become pretty much equally easy to use, plus since you use Docker there’s no need to compile anything.
    
    source
    -> View More Comments
- wreckedcarzz@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
  Very similar, but I’m using Gemma4 12b. I could run 26b but that would exhaust my vram after a short interaction. And ollama. Open WebUI + Home Assistant hooked up with it, a couple very basic scripts I threw together, and connected to Kagi.
  
  I’m trying to get my folks off the Gemini/Google bandwagon for the privacy aspect, but also because it’s just cool from a tech geek perspective. For me, any questions throughout the day that I want info on, but not enough to do a deep-dive for an hour+, gets fed to Steve, my models name. Or I’ll use it as a springboard and then properly read the citations for more info. And I feed it HA automations when I get stuck scratching my head on an edge case or whatever - it’s alright but helps me explain what I want from the automation, and when it understands my goal, great, and when it doesn’t I reflect on what I have setup and can usually go ‘ah ha’ and fix the issue. It’s like a rubber ducky/pair programmer that you can both think is an idiot sometimes, but it doesn’t judge you when you’re the idiot.
  
  source
  - hoshikarakitaridia@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
    That’s a pretty good way to describe those kinds of models. Somewhere between rubber duck debugging and a really lazy intern. Better than an algorithm, faster than a human, but less trustworthy than an intern. It’s great if you know how and when to use it. Maybe it’ll become an intern soon enough, but for now let’s focus on decompiling adobe into bankruptcy.
    
    source
- eodur@piefed.social ⁨3⁩ ⁨weeks⁩ ago
  I’ve got similar running on my RX 7600 XT. It is not a great experience for interactive use, but fine for async tasks. I’m also running nemo-speak for live local dictation and meeting transcription.
  
  source
  - hoshikarakitaridia@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
    Yeah running in the background type beat. Not gonna trust this with anything important but it’s great to have someone just do shit for you or get a second opinion.
    
    source
- BuckFutter@timeyak.com ⁨3⁩ ⁨weeks⁩ ago
  Yah doing similar on 5060ti.
  
  source
  - hoshikarakitaridia@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
    That’s gonna be quick af with the same models.
    
    source
    -> View More Comments
- mortalic@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
  Random Question, I’ve been trying to offload some of my stuff handled by claud to basically this exact setup. However one of the things I manage has a pair of MCP servers that are required. I might not be smart enough, but how do you get Qwen to use an MCP?
  
  source
  - hoshikarakitaridia@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
    MCP is a really dumb server that just takes requests and sends back data.
    
    Somewhere in your frontend settings (Open-WebUI has it under “integrations” I believe, some backends can set these things too) you see a section called “MCP server”. You need to keep your MCP server running somewhere (it’s very slim, so not a huge deal usually) and then in those settings you put your IP and port for the server, for example “192.168.1.15:8080”. Usually if you did it right (sometimes you can check if it works in the settings), you can ask the model if it has access to the tools and it will tell you.
    
    What’s happening in the background is that every conversation the AI does, it gets a short preamble with some context about what it can do, and if you connect web search, custom skills or an MCP server (it’s just skills but they are run through their own server basically), the AI sees all that in this preamble if it’s set up correctly. And then when it feels like it needs to use those tools, it will check the description on how to do that, and then it uses them.
    
    source
    -> View More Comments
kensand@sopuli.xyz ⁨3⁩ ⁨weeks⁩ ago
Running Qwen 3.8 27B dense on some R9700s, and Qwen 3.8 flash next on dgx sparks. I use it for code gen outside work, including an automated Dev pipeline that rubs on its own. When I did the math comparing to token API pricing of Opus 4.8, it covered its own costs in a few months. With the sparks I get about 40tg/s, 1500pp/s, but can get concurrency up to 5 stably. With the R9700s, I get ~100tg/s, 3000pp/s and can also get up to 5 concurrency.

I access it over wireguard to the network that it is on, and use termux to ssh into a build machine for dev over CLI using pi.dev

I also started building fcli.dev to make integrations easier; the idea being that LLMs should just implement simple reusable functions instead of repeatedly making complex curl requests to integrate with a service.

I also use LLMs heavily for work; Qwen 3.8 seems almost on par to Opus 4.8 IMO. I also don’t have to worry about token costs, and it often responds faster than Claude would because there usually isn’t a queue on my machines. At this point, I wouldn’t complain if my work provided me 2x B6000 instead of a Claude subscription; I’d still be able to get all my work done.

source
- FrankLaskey@lemmy.ml ⁨3⁩ ⁨weeks⁩ ago
  Would be interested to know your experience with fcli if you’d share more. Just hearing about it.
  
  source
  - kensand@sopuli.xyz ⁨3⁩ ⁨weeks⁩ ago
    I mean, I haven’t really started discussing it outside IRL friends, so not surprised you haven’t heard of it 😅
    
    In essence, the problem is twofold:
    
    LLMs tend to use CLIs more effectively than MCP, at least in the domains I use. However, installing all the CLI tools and configuring them can be cumbersome and not easily repeated. CLIs also cannot be modified, and are often quite verbose.
    
    LLMs will repeat very similar commands over and over with just a few tweaks. This often happens when there is a gap in a tool or poor documentation.
    
    So fcli is intended to be an easily extended, configured, and authenticated CLI. The “one CLI” to do everything your LLM needs. Ideally, you set up a personal repo to store ‘ucks’ (f packages) in, and you set up credentials for that repo so your LLM can push any new ucks to it. You can then share that repo with other users if they could also benefit from any of that functionality.
    
    As you go along in a session, when you notice the LLM repeatedly running similar verbose commands, you can have it create an uck for it instead and save those tokens.
    
    Likewise, if, say, an MCP server doesn’t have full functionality, you can have your LLM import the SDK from npm into an uck and write a basic fcli wrapper around it to start using.
    
    The other nice thing is that it makes your tool configuration repeatable between LLMs and harnesses. You don’t need to set up your MCP servers in both Claude and Codex and Pi, just in f and tell them to use f.
    
    There’s also a creds uck to make your secrets more portable and (a bit) safer.
    
    In general, I’m trying to model it a bit like pi.dev, but as part of the tooling instead of the harness itself. I think they work quite well together :)
    
    Interesting paper comparing MCP and CLI
    
    source
    -> View More Comments
JasonDJ@lemmy.zip ⁨3⁩ ⁨weeks⁩ ago
What a timely comment.

I recently gave my kid a gaming PC for his birthday…which I used as an excuse to upgrade my GPU from a 1660 Super to a 5060ti 16GB

Combined with a 5600x3d and 32gb of system memory.

I just started messing with local LLMs yesterday. “Getting started” is rather easy. pacman -S lm-studio (I use arch, btw), launch it, download a model that your hardware supports, and start talking.

Right now I’m working on getting it to talk to vscode.

source
- ShadowZone@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
  I used Cline as my VSCode chat plugin for models in LMStudio. Worked pretty well but I have moved my local LLM stack to Hermes in the meantime.
  
  source
  - JasonDJ@lemmy.zip ⁨2⁩ ⁨weeks⁩ ago
    I tried using zoo with qwen3.6 and 2.5 but it failed horrendously. My card could only handle like a 16k context window with those models, and the prompt alone was like 11k.
    
    Tried opencode and that was fast…but can’t get it to interact with system yet. More playing to do.
    
    source
db0@lemmy.dbzer0.com ⁨2⁩ ⁨weeks⁩ ago
Not only do I run and share it. I’ve built an entire service to allow everyone else to do so safely. aihorde.net

source
PetteriPano@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
Yes, were all on c/LocalLLaMa

I’m running qwen3.8-next-flash on a rx7900xtx plus 128gb of ram. It serves my hermes agent. I direct it through matrix. I also forward an endpoint to my VPS to use for anything. Mostly opencode.

source
- xSikes@feddit.online ⁨3⁩ ⁨weeks⁩ ago
  Is that the only community? I see the last activity was 2 months ago.
  
  source
  - BeefAndPoultry@lemmus.org ⁨2⁩ ⁨weeks⁩ ago
    
    I see the last activity was 2 months ago.
    
    that means no one from your instance is subscribed, if you subscribe then you’ll get new posts
    
    you can see that there are new posts here sh.itjust.works/c/localllama
    
    source
    -> View More Comments
  - PetteriPano@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
    Posts from 16 minutes and 2 days ago here. Maybe your instance is not federating with theirs.
    
    source
    -> View More Comments
auzy1@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
We did the maths. Didn’t seem worthwhile and the payback period was a very long time

source
- MagicShel@lemmy.zip ⁨3⁩ ⁨weeks⁩ ago
  I think the value is in the privacy. You can hook it up to knowledge and capabilities you’d never share with a cloud AI. But if your use case is strictly coding, I agree.
  
  source
- laz@lemmy.dbzer0.com ⁨3⁩ ⁨weeks⁩ ago
  True but if you happen to broken macs you can just take thier board with all the unified ram it’s become quite the thing recently even Asus duo with 32gb Intel 9 core ultra board’s have been started to be repurposed like mini pcs for ai use
  
  source
state_electrician@discuss.tchncs.de ⁨3⁩ ⁨weeks⁩ ago
I wish I could. I don’t have the hardware for it.

source
- vsis@feddit.cl ⁨2⁩ ⁨weeks⁩ ago
  I run Gemma-4 4B.
  
  It’s not the best model, but it runs with CPU only and with 6 ~ 9 G of RAM.
  
  source
uzername@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
24GB GPU Qwen 3.8 ( 2 variants ) Pangolin>OpenWebUI>lama.cpp for general use and available outside the homelab Zoo Code>OpenWebUI>lama.cpp for coding , mostly scripts and html

Qwen 3.8 was a game changer for me.

source
- TheFinn@discuss.tchncs.de ⁨2⁩ ⁨weeks⁩ ago
  What variants? And why two?
  
  source
  - uzername@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
    For coding with large context: Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf Anything else: Qwen3.8-27B-GGUF:Q5_K_M
    
    source
magic_smoke@lemmy.blahaj.zone ⁨2⁩ ⁨weeks⁩ ago
I write my own code, skid.

source
trougnouf@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
I use Qwen 3.8 27b Q4 with 100k context (llama.cpp and the Mistral AI harness) on a 24 GB AMD Radeon 7900 XTX and it’s been great. It can handle pretty much anything I throw at it and I can let it work indepedently overnight / over days on long tasks.

source
- melfie@lemmy.zip ⁨2⁩ ⁨weeks⁩ ago
  I have nearly the same setup, except OpenCode for the harness. I run 200k context for anything that doesn’t need vision, or 150k otherwise. Typically get about 30 t/s decode, starts at close to 800 t/s for prefill and drops to the 200s for a long context. I’d imagine huger decode speeds are possible with some tinkering, but I haven’t gotten around to it.
  
  source
  - trougnouf@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
    I get about 45 t/s after some tweaking, getting the thermals under control helped a lot; mainly by reducing the power consumption to 261 W (and downclocking a little bit)
    
    source
- dieTasse@feddit.org ⁨2⁩ ⁨weeks⁩ ago
  How much electricity does it burn? And how much does it cost to let it run overnight? I feel like this is the same problem that the companies have, its way too expensive to use it on a lots of things and it is way cheaper to use it on someone else’s server…
  
  source
  - trougnouf@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
    It uses 261 W, probably more expensive than getting a 2nd Mistral subscription with the cost of power where I live (Belgium), but it will be free heat once I have solar panels and a battery.
    
    source
    -> View More Comments
Shimitar@downonthestreet.eu ⁨3⁩ ⁨weeks⁩ ago
Yes i do.

I esperiment a lot, but my stable setup is Qwen 3.8 27B Q6_K on my home server. Hardware is Ryzen 9 5950x with 64gb ddr4 and two old but effective NVIDIA RTX A4000 for a total of 32gb VRAM.

Qwen 3.8 is the best model I can run in terms of output quality. It can perform all tasks pretty well and it’s reasonably fast at 500ts prefill and 30/40ts processing. There are much faster models but honestly not as good.

I can also run gemma4 all on CPU, with no GPU at all.

The key point is having both GPU on pci-e CPU, not on chip set bridge, so I am using a PCI riser 8x8x to share the 16x slot between the two GPUs. Inference speed is unaffected, but prefill benefits immensely.

Software side, use llama.cpp on Linux, you get the best performance possible at the expense of a bit more complexity compared to other engines.

From a power consumption perspective, I tined down my CPU governor and maximum frequency, and also the GPUs maximum watts. At ide the system is around 80w. At peak, it’s about 280w with the limiters, and inference speed doesn’t suffer too much. Without limiters I can easily reach 650w.

source
MagicShel@lemmy.zip ⁨3⁩ ⁨weeks⁩ ago
This is something I will do probably within the year, but it feels to me like the area is still moving quite rapidly. The model will naturally depend a bit on the hardware. I’m looking at used AI cards because it looks like currently for about $2k you can get one that will run ~350B models, and GLM is nice in that space, though I will experiment with other models once I can run them.

Things I will do:

try to replace Alexa for voice activated smart home management — this will let me go locally controlled I hope (my wife is big on convenience and could give a fuck about privacy so the solution has to be solid).

a personal journal so I can have it remember things so later on I can ask things like when is the last time a friend and I hung out, or what was that orchard we went to last October, or what was that great B&B we stayed at, let’s do that again. This is likely the thing that would require a good model — everything else is pretty simple orchestration, aggregation, and summary. Arguably it’s the thing that would make the biggest difference in my life, too. There is a lot of beauty in life but my brain is wired to constantly churn solving problems and be irritated and I rarely stop to appreciate the moments or remember the accomplishments.

I have a ton of ideas for conveniences AI could do if it had total privacy and security. In short, orchestrate all of the tools and knowledge that I already rely on constantly. Integrate calendar and private info like SSNs, VINs, who has what doctors for what, or who was sick when, smart alerts like the overnight temp will be below freezing next week so make sure the house is winterized, or I wanted to see this movie and it’s releasing or will be leaving theaters soon — oh and next weekend you’re visiting the grandkid, so it’s either now or I’ll remind you again when it comes out on streaming.

Just having a personal knowledge base always on and totally private would be handy. It would be like having a personal assistant maybe concierge if it could be trusted to that degree.
source
- button_masher@lemmy.ml ⁨3⁩ ⁨weeks⁩ ago
  Your use cases are quite similar to what I’ve been trying to action.
  
  One question I keep wondering about is the “security” aspect of it. And how the inputs to the model might work. Yes it can read your calendar and you could “Jeeves, please remind me to hang out with Wooster next week” but it’s easy to forget and I have a hard time logging spontaneous events.
  
  Any ideas on how you imagine the friend/activity tracker could work?
  
  Was thinking maybe it could read my smartwatch location history. Or scan the photos I take. But that gets dystopian fast (yes it’ll be local but then I’m worried about security).
  
  Fun to ponder!
  
  source
  - MagicShel@lemmy.zip ⁨2⁩ ⁨weeks⁩ ago
    I think having a chat interface is part of it so it’s friendly and convenient to use. I think subverting suggested a matrix bot and that would continue the theme of privacy. You could have different channels or groups for different personas. Maybe one is your Majordomo, another is a simpler calendar bot, another is for journaling, etc.
    
    Then, wire in some cron jobs to invoke the AI to prompt you on various subjects. “Anything you want to share?” every day at 9pm. If you miss a day or two because you were busy or asleep, you can catch it up later or not at all. You could even have it ask what you did at work if your life is like mine and they kinda blend together and you don’t know how many hours you worked.
    
    source
Casuls_Die_Thrice@lemmy.zip ⁨3⁩ ⁨weeks⁩ ago
I’d like to, but what’s “good” that can run on a 16GB RAM, i5 12th gen, 3070-equipped desktop?

source
- anamethatisnt@sopuli.xyz ⁨3⁩ ⁨weeks⁩ ago
  Gemma 4 12b unified as a q6 gguf fits with about 24k context inside your vram. koboldcpp+sillytavern works well on the software side.
  
  source
  - laz@lemmy.dbzer0.com ⁨3⁩ ⁨weeks⁩ ago
    Sc117
    
    source
- Wildmimic@anarchist.nexus ⁨3⁩ ⁨weeks⁩ ago
  Might wanna try a ternary model, which are a lot denser in comparison.
  
  Ternary Bonsai 27b is 7.17GB in size, which might fit on your 3070 depending on how much free VRAM you have (and with CPU offloading it should work in all cases). You can also try their edge-computing model designed to be working on smart phones here - that one is less than 4GB in size, which should fit in any case. In my experience they do work very well compared to other models in that size class.
  
  Be aware that they need a specific llama.cpp build found here
  
  source
- eodur@piefed.social ⁨3⁩ ⁨weeks⁩ ago
  This is the question to spend your Claude tokens on. It helped me tweak my setup to get better performance.
  
  source
carrylex@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
No, I prefer using my brain or other things that usually work 100x more efficient without hallucinations and don’t required a shit ton of RAM that are way to expensive to buy right now.

source
- dieTasse@feddit.org ⁨2⁩ ⁨weeks⁩ ago
  Unless your brain is hallucinating this whole reality 😀
  
  source
  - bier@lemmy.blahaj.zone ⁨2⁩ ⁨weeks⁩ ago
    I wish
    
    source
aurion@piefed.aurionblack.com ⁨2⁩ ⁨weeks⁩ ago
Yes! I’ve got quite a few projects going right now. Probably too many of them to keep track

I am the only user, so it’s relatively easy to run multiple models for what ever task. Ollama is pretty good at balancing which models are working at what time to not bog down they system

I’m running gpt-oss 120 for a general purpose Hermes agent. Out of the box, it was ok. Not very snappy, but it was one of the best ones for orchestrating other agents.

Glm flash (4.something?) does pretty good as well.

I’ve found a hybrid of cloud and local is the best solution. Use local for smaller or more defined tasks. Use cloud for orchestrating, planning, and…fixing the local agents mistakes lol

I’ve got an LLM maintained Obsidian wiki that is written and edited by a few Ollama agents based on chats via my own self hosted matrix instance. I’ve taken it to a point where I’m pretty happy with the results. But I think if I had enough time and energy, I could really dial it in.

One of my favorite use cases for this is when I’m researching topics like UFOs/high strangeness/conspiracy theories. I have my very own LLM maintained Obsidian vault filled with curated notes and links to sources.

Literally, no one cares about this and they think it’s stupid and I’m stupid and I should feel bad for spending time on this or having the audacity of sharing it… they just don’t appreciate the amount of work and knowledge it took to get what I have set up.

I think it’s cool as fuck. I geek out every time that it works.

source
- zer0squar3d@lemmy.dbzer0.com ⁨2⁩ ⁨weeks⁩ ago
  Do what makes you happy mate! Tell the naysayers to fk off.
  
  source
rtxn@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
Not me. Call me old-fashioned, but I prefer organic.

source
Seralth@piefed.seralth.com ⁨3⁩ ⁨weeks⁩ ago
Three models my self, qwen 27b as a dev assistant, a small model rigged up to handle voice input as an assistant that’s connected to my intercom system though out the house. Mostly to run my lights/music/audiobook play back. It’s fun having jarvis control the house after all. Then a image generator.

Llama and swap on a 7900xtx.

I haven’t gotten around to setting up phone access but wireguard for remote access form work in my laptop.

Honestly it’s all basically over priced toys. I’m baffled that people rely on this stuff. I look at it the same way people do with model airplanes or RC cars.

Over priced toys that are hell of a lot of fun. Break frequently and require more care then they are probably worth to use safely lol

source
Sibbo@sopuli.xyz ⁨3⁩ ⁨weeks⁩ ago
I find the idea interesting. However I wouldn’t get the commercial service you are advertising. Maybe something that can run on an older CPU as well. Doesn’t need to be fast, but if it uses up the idle compute I have on my rented server, then why not.

source
- StateMachine@lemmy.zip ⁨3⁩ ⁨weeks⁩ ago
  I’m not advertising any service. I added an image because Lemmy wouldn’t let me post the question without one. I think I may have chosen the wrong post type or something.
  
  source
  - Sibbo@sopuli.xyz ⁨2⁩ ⁨weeks⁩ ago
    No worries. Yeah, the user interface is difficult. Thanks anyways for the post, it motivated me to self host my own agent, and it works quite well actually.
    
    source
twilightwolf90@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
A quick glance shows that I’m in the minority. I grabbed a Bosgames M5 with 128GB UMA RAM when I discovered openclaw (at the time claudebot). I’m running Lemonade, which is a wrapper for llama.cpp, sd-cpp, flux, and others. I’m happy to have paid a “mere” $2400 compared to what similar systems cost now, and now I’m toying with fine tuning my own models. Though, I primarily run Hermes Agent paired with Qwen3.6 35b A3B (using a heretic ablitieration for 0 refusals).

source
- mal3oon@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
  Try gufo with qwen 3.8 flash next. Much much better than 3.6-35B.
  
  source
  - twilightwolf90@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
    How well does it handle concurrency? I could see this as a vast improvement on llamacpp, but I was already speccing out vLLM.
    
    source
ns1@feddit.uk ⁨2⁩ ⁨weeks⁩ ago
If you haven’t seen it already, take a look at !localllama@sh.itjust.works

source
greybeard@feddit.online ⁨3⁩ ⁨weeks⁩ ago
I recently gave Strata a shot for my local AI. I’m very impressed with the performance they have eeked out of my hardware. I’ve got a rather beefy gaming desktop I built as an AI server back when the current craze started, before prices started going up, but models up until this point really failed to take full advantage of that hardware. Strata (which uses a Qwen model) both simplified the setup process, and it maxes out the hardware when running.

Strata acts like an OpenAI compatible API server, I then have Hermes as a “harness” to make an “agent”, which I interact with on “buzz”, which is like discord, but made for bots. Although Hermes also supports discord and several other chat platforms.

If you have the hardware to run Strata, I recommend setting it up, it comes with a basic chat interface, which you can then grill to assist you with setting up Hermes and whatever else.

One thing I didn’t mention, is security. I highly recommend you treat AI as unpredictable, unreliable, and capable of being taken over by bad actors. This isn’t a problem for a pure chat interface, but Hermes gives the AI tools to do things, and those tools could be used to take over your computer, scan your network, and create all kinds of havoc. For my part, Hermes is running in a virtual machine which is on an isolated network. I keep tight control on what it does and doesn’t have access to in my network main network. That may not be something you have the hardware or knowledge to do, at the very least I recommend running Hermes in a docker container, and not giving it access to your personal desktop nor credentials to things like your email account.

source
joseki@lemmy.zip ⁨3⁩ ⁨weeks⁩ ago
Yes, with OpenClaw I can expose my agents via discord apps on my home discord server, I like that route because of the rich media and UI options (agents can give me drop down selector lists and approve/reject buttons). I have two 24gb Nvidia cards for local models usually Qwen, Mistral, and nemo based models for different purposes. I can chat with the agents anywhere I’m logged into discord and I expose privileged control surfaces and nodes via tailscale and family/shared endpoints via caddy and cloudflared

source
panda_abyss@lemmy.ca ⁨3⁩ ⁨weeks⁩ ago
I run lemonade on my strix halo box, I change the models I use, but the qwens are pretty good.

I access it via Tailscale on my phone, then I have a page I can use. The Lemonade interface forces you into the app, so you’ll want some other interface for your LLMs, as the app is junk.

I used to run llama.cpp and llama swap, but lemonade combines them and makes managing different ai backends easier.

source
- Seralth@piefed.seralth.com ⁨3⁩ ⁨weeks⁩ ago
  I’ve been eyeing lemonade as I am on llama and swap my self. How much of a pita was the migration. I’m a lazy bastard.
  
  source
  - MalReynolds@slrpnk.net ⁨3⁩ ⁨weeks⁩ ago
    Lemonade is pretty good, especially for AMD GPUs and the lazy, but it’s pretty opinionated (like ollama but not as toxic) and it’s not easy to directly control llama-server parameters. It feels a bit commercial / we want to IPO.
    
    Seeing as you already have your parameters, llama-swap might be a better fit, lets you configure llama-server directly in a yaml, and switch between them. You can even have multiple versions of the same model (e.g. thinking or not, try out different parameters) and change without reloading.
    
    Both work fine with e.g. Open WebUI and Pi (lemonade its own integration and llama-server needs an extension).
    
    source
    -> View More Comments
  - panda_abyss@lemmy.ca ⁨3⁩ ⁨weeks⁩ ago
    I pointed lemonade at my llama.cpp model folder but it uses json files to configure them. I think it was fine for importing them or you could have an agent write them for you. After that you can manage models inside the app
    
    Overall it wasn’t a huge deal, it’s easier than managing llama.cpp directly with rocm updates.
    
    source
xylogx@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
With just my NVidia Geforce 3070 card with 8GB of VRAM I can get 40 tokens per second on smallish models such as llama3.1 or qwen 3.8

ollama list NAME ID SIZE MODIFIED
qwen3:8b 500a1f067a9f 5.2 GB 5 months ago
llama3.1:8b 46e0c10c039e 4.9 GB 5 months ago

source
- Schilling2304@thelemmy.club ⁨2⁩ ⁨weeks⁩ ago
  What game servers ? Have you looked into qwen3.5 9b ? Gemma e4b ? There is plenty of new small models after yours. There is r/lowendlocalai, a community of people with low computing resources but it is in reddit.
  
  MiniCPM5 may be good.
  
  source
boudica@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
Llama.cpp running qwen 3.8 on a mini PC with 32GB ram and 8GB vram. Not the must speedy but it works. I only access it within my network and I’m also set up with tailscale.

source
- andrewoidxyz@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
  So you’re using CPU offloading? Or the model’s Bonsai2 variation?
  
  source
hendrik@palaver.p3x.de ⁨3⁩ ⁨weeks⁩ ago
Sure. I have Home Assistant with STT and TTS. And I use KoboldCPP to experiment with smaller LLMs. Nothing fancy, it’ll run on my old laptop, albeit very slow.

source
floquant@lemmy.dbzer0.com ⁨3⁩ ⁨weeks⁩ ago
90€/mo to self-host something? Lol

source
irotsoma@piefed.blahaj.zone ⁨2⁩ ⁨weeks⁩ ago
Even ignoring all of the functional issues with LLMs/"AI” related to their design to never admit to not knowing and preferring to make stuff up instead (hallucinate), I have an ethical qualms about the way they are trained. Two primary issues are that they aggressively scrape data from sources that explicitly do not consent to sharing their work and the data centers which have made hardware impossibly expensive for everyone but them and are so impactful to our underfunded electrical grids that they cause brown outs that keep people from heating their homes among other things, cause major spikes that cause house fires and destroy home appliances, and create an excuse to generate power using the worst polluting generators that we had finally gotten shut down to slow climate change (personally I’m tired of the wildfires and extreme storms from climate change).

So, I personally consider it extremely unethical to use the technology for that and several other reasons which are hard to touch on exhaustively in a post like this.

source
zorflieg@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
If you are playing around at home I suggest seeking a MoE model. You get less penalty from having less vram and offloading to normal ram. I run Qwen 3.6 quant 6 on 250k context on a 16gb vram 64gb ram ddr4 PC. I get about 31tps which is enough for my use. After a little tweaking I have put results of it’s output into Claude Pro for vetting and it hasn’t found any flaws in the simple scripts and things I’ve asked it to make. I can only run Qwen 3.8 which is a dense model and not MoE. At low quant with not as much context. It spits out accurate short context answers if that’s all you need. So I prefer 3.6. Tiel is faster but Claude picked up more errors in what I was doing. Orinith kept getting into thinking loops for me so I stopped using it. Anyway I’m just a little guy.

source

-> View More Comments