Is anyone self-hosting their own AI?

Comments

• hardcoreufo@lemmy.world ⁨25⁩ ⁨minutes⁩ ago

  Yes, I have a desktop with a radeon 9700 ai pro with 32GB of VRAM. Its more than doubled since I bought it but its basically a down clocked 9070xt with double the VRAM. I also have a little arc b50 in my server.

  Mostly yeah use it for privacy reasons. I run llama.CPP directly and access via open webui. I host a searx instance for it to use as web search. Mostly run qwen3.6/3.8 or gemma4 on the radeon and almost exclusively gptoss 20b on the B50 along with a second small model for tags and titles in open webui. Yeah I just use it mostly as a basic chatbot or as an agent in obsidian using the engram plugin. Helps search and summarize years of notes and works pretty well.

  I’ve tried to use it as a coding assistant in zed without much luck so usually just post my code into open webui for feedback and it works alot better for me.

  Yeah I can access it with other machines on my network including phone with conduit app. I can get to it on the go by connecting to home network with wireguard and sending a magic packet to wakeup desktop.

  I havent done any image gen nor have much use for it outside of maybe using it to make app icons in the style of papirus icons for programs they don’t and won’t support.

  It definitely feels more comprable to first ai wave than current frontier models but the privacy and price trade off is worth it to me. I’d say using it in obsidian has been the best use of it so far and one I certainly wouldn’t trust to an external AI.

  source
• SirMaple__@lemmy.ca ⁨29⁩ ⁨minutes⁩ ago

  I have the following hardware

  2 x ASUS GX10 128GB(each) 2 x Mac Mini M4 Pro 24GB(each)

  The GX10s are clustered and run DeepSeek-V4-Flash-0731-Abliterated. The M4 Pros run various utility LLMs such as Qwen 3.8

  I have the following software running

  OpenWebUI Hermes Agent LiteLLM Llama.cpp vLLM

  StoryBinder - custom web app that I use for story/script writing when in the mood

  LASM - Local AI Stack Manger - custom web interface to manage pretty much all aspects of my local stack. Agents run on the 4 nodes as well as my NAS(Download GGUF/HF snapshots once and copy out to the nodes)

  Have yet to connect my Home Assistant into the stack.

  source
• chaospatterns@lemmy.world ⁨32⁩ ⁨minutes⁩ ago

  Yes, I run Ollama on a NVIDIA 1080 Ti at home. I run granite4:3b for my Home Assistant assistant for my voice assistant, paperless-ngx, OpenWebUI for chat, and a few other things. For speech to text, I run a custom-built hybrid speech-to-phrase + parakeetTTS (requests that match speech-to-phrase take priority over parakeet.) I run GOModel to route requests to my local Ollama instance, or DeepInfra (cloud-based for larger models), or Tinfoil (cloud-based for encrypted LLM chats.)

  As much as I love to use local-everything, only my smarthome+basic chats go local. Trying to fit both the STT and LLM models on the same GPU at the same time while also using this computer as a media console is challenging. I’ve recently started looking at the Hailo m.2 accelerator board to see if I can shift the STT to it.

  source
• violetrage@lemmy.blahaj.zone ⁨10⁩ ⁨hours⁩ ago

  I self-host my own neural network, all within the confines of my own skull.

  source

  • MidnightMarauder@lemmy.dbzer0.com ⁨10⁩ ⁨hours⁩ ago

    That 33% downtime tough

    source

    • frongt@lemmy.zip ⁨9⁩ ⁨hours⁩ ago

      It’s scheduled maintenance so it doesn’t count toward downtime. Besides, I’m not trying to use it then anyway.

      source
    • SpaceCowboy@lemmy.ca ⁨4⁩ ⁨hours⁩ ago

      The downtime isn’t the biggest issue, you can plan around that. The real problem is the random lag on some queries and volatile memory being lost (usually when there’s a context change).

      source
    • arudesalad@piefed.ca ⁨9⁩ ⁨hours⁩ ago

      It’s better than github

      source
    • BartyDeCanter@piefed.social ⁨7⁩ ⁨hours⁩ ago

      Only 33% downtime? Damn, that must be nice. I really need to get a hardware update.

      source
  • Valmond@lemmy.dbzer0.com ⁨10⁩ ⁨hours⁩ ago

    How much RAM is it using?

    source

    • mike_wooskey@lemmy.thewooskeys.com ⁨10⁩ ⁨hours⁩ ago

      Made me think of Johnny Mnemonic. www.youtube.com/watch?v=jS2GFqT5ycU

      source

      • -> View More Comments
  • A_norny_mousse@piefed.social ⁨8⁩ ⁨hours⁩ ago

    I’m just glad that this is both Top & Hot comment. 🧠✊

    source
• JasonDJ@lemmy.zip ⁨7⁩ ⁨hours⁩ ago

  What a timely comment.

  I recently gave my kid a gaming PC for his birthday…which I used as an excuse to upgrade my GPU from a 1660 Super to a 5060ti 16GB

  Combined with a 5600x3d and 32gb of system memory.

  I just started messing with local LLMs yesterday. “Getting started” is rather easy. pacman -S lm-studio (I use arch, btw), launch it, download a model that your hardware supports, and start talking.

  Right now I’m working on getting it to talk to vscode.

  source
• hoshikarakitaridia@lemmy.world ⁨11⁩ ⁨hours⁩ ago

  • Qwen3.6 35b a3b Q4_K_M running on llama.cpp
  • Open-WebUI as a frontend
  • integrations with an embedding model and searxng & crawl4ai
  • doing lots of different stuff with it, asking basic stupid questions, recipes, tech stuff, etc

  source

  • wreckedcarzz@lemmy.world ⁨57⁩ ⁨minutes⁩ ago

    Very similar, but I’m using Gemma4 12b. I could run 26b but that would exhaust my vram after a short interaction. And ollama. Open WebUI + Home Assistant hooked up with it, a couple very basic scripts I threw together, and connected to Kagi.

    I’m trying to get my folks off the Gemini/Google bandwagon for the privacy aspect, but also because it’s just cool from a tech geek perspective. For me, any questions throughout the day that I want info on, but not enough to do a deep-dive for an hour+, gets fed to Steve, my models name. Or I’ll use it as a springboard and then properly read the citations for more info. And I feed it HA automations when I get stuck scratching my head on an edge case or whatever - it’s alright but helps me explain what I want from the automation, and when it understands my goal, great, and when it doesn’t I reflect on what I have setup and can usually go ‘ah ha’ and fix the issue. It’s like a rubber ducky/pair programmer that you can both think is an idiot sometimes, but it doesn’t judge you when you’re the idiot.

    source
  • BooBees@fedinsfw.app ⁨10⁩ ⁨hours⁩ ago

    What’s your rig?

    source

    • hoshikarakitaridia@lemmy.world ⁨6⁩ ⁨minutes⁩ ago

      1070ti and some hexacore something cpu that’s alright with around 24gb ddr4, I think one of the slower speeds. I also got a faster PC with another llama.cpp instance and I vibe coded a quick and dirty proxy which I can use to switch between the different backends. It’s genuinely useful because I got code editors but also my Open-WebUI using it.

      Sounds elaborate but generally it’s all pretty primitive.

      source
  • jwiggler@lemmy.dbzer0.com ⁨10⁩ ⁨hours⁩ ago

    Pretty much my setup as well. Though I’m using qwen3.8 27b-Q4_K_M under ollama. Everything running with docker. Add nginx and either a domain+port forwarding or zerotier/tailscale you’ve got an app you can get to from outside your house.

    Right now not sure how I’m going to leverage it not probably to streamline my self hosted stuff, help with config mgmt and documentation, web page summarization, etc. we’ll see how it does, just got it setup yesterday

    source

    • robber@lemmy.ml ⁨1⁩ ⁨hour⁩ ago

      In case you are unaway, frieds don’t let friends use ollama. The llama.cpp has become pretty much equally easy to use, plus since you use Docker there’s no need to compile anything.

      source

      • -> View More Comments
  • eodur@piefed.social ⁨8⁩ ⁨hours⁩ ago

    I’ve got similar running on my RX 7600 XT. It is not a great experience for interactive use, but fine for async tasks. I’m also running nemo-speak for live local dictation and meeting transcription.

    source
  • BuckFutter@timeyak.com ⁨9⁩ ⁨hours⁩ ago

    Yah doing similar on 5060ti.

    source
  • mortalic@lemmy.world ⁨6⁩ ⁨hours⁩ ago

    Random Question, I’ve been trying to offload some of my stuff handled by claud to basically this exact setup. However one of the things I manage has a pair of MCP servers that are required. I might not be smart enough, but how do you get Qwen to use an MCP?

    source
• amgine@lemmy.world ⁨2⁩ ⁨hours⁩ ago

  Epyc 7551? 256gb ddr4, r9700 + mi50. Running two models depending on what I’m doing . Llama.cpp + comfyUI or two llama.cpp for model and advisor in OMP. Main model is Qwen 27 abliterated

  source
• kensand@sopuli.xyz ⁨9⁩ ⁨hours⁩ ago

  Running Qwen 3.8 27B dense on some R9700s, and Qwen 3.8 flash next on dgx sparks. I use it for code gen outside work, including an automated Dev pipeline that rubs on its own. When I did the math comparing to token API pricing of Opus 4.8, it covered its own costs in a few months. With the sparks I get about 40tg/s, 1500pp/s, but can get concurrency up to 5 stably. With the R9700s, I get ~100tg/s, 3000pp/s and can also get up to 5 concurrency.

  I access it over wireguard to the network that it is on, and use termux to ssh into a build machine for dev over CLI using pi.dev

  I also started building fcli.dev to make integrations easier; the idea being that LLMs should just implement simple reusable functions instead of repeatedly making complex curl requests to integrate with a service.

  I also use LLMs heavily for work; Qwen 3.8 seems almost on par to Opus 4.8 IMO. I also don’t have to worry about token costs, and it often responds faster than Claude would because there usually isn’t a queue on my machines. At this point, I wouldn’t complain if my work provided me 2x B6000 instead of a Claude subscription; I’d still be able to get all my work done.

  source

  • FrankLaskey@lemmy.ml ⁨9⁩ ⁨hours⁩ ago

    Would be interested to know your experience with fcli if you’d share more. Just hearing about it.

    source

    • kensand@sopuli.xyz ⁨8⁩ ⁨hours⁩ ago

      I mean, I haven’t really started discussing it outside IRL friends, so not surprised you haven’t heard of it 😅

      In essence, the problem is twofold:

      1. LLMs tend to use CLIs more effectively than MCP, at least in the domains I use. However, installing all the CLI tools and configuring them can be cumbersome and not easily repeated. CLIs also cannot be modified, and are often quite verbose.
      2. LLMs will repeat very similar commands over and over with just a few tweaks. This often happens when there is a gap in a tool or poor documentation.

      So fcli is intended to be an easily extended, configured, and authenticated CLI. The “one CLI” to do everything your LLM needs. Ideally, you set up a personal repo to store ‘ucks’ (f packages) in, and you set up credentials for that repo so your LLM can push any new ucks to it. You can then share that repo with other users if they could also benefit from any of that functionality.

      As you go along in a session, when you notice the LLM repeatedly running similar verbose commands, you can have it create an uck for it instead and save those tokens.

      Likewise, if, say, an MCP server doesn’t have full functionality, you can have your LLM import the SDK from npm into an uck and write a basic fcli wrapper around it to start using.

      The other nice thing is that it makes your tool configuration repeatable between LLMs and harnesses. You don’t need to set up your MCP servers in both Claude and Codex and Pi, just in f and tell them to use f.

      There’s also a creds uck to make your secrets more portable and (a bit) safer.

      In general, I’m trying to model it a bit like pi.dev, but as part of the tooling instead of the harness itself. I think they work quite well together :)

      Interesting paper comparing MCP and CLI

      source
• uzername@lemmy.world ⁨7⁩ ⁨hours⁩ ago

  24GB GPU Qwen 3.8 ( 2 variants ) Pangolin>OpenWebUI>lama.cpp for general use and available outside the homelab Zoo Code>OpenWebUI>lama.cpp for coding , mostly scripts and html

  Qwen 3.8 was a game changer for me.

  source
• PetteriPano@lemmy.world ⁨8⁩ ⁨hours⁩ ago

  Yes, were all on c/LocalLLaMa

  I’m running qwen3.8-next-flash on a rx7900xtx plus 128gb of ram. It serves my hermes agent. I direct it through matrix. I also forward an endpoint to my VPS to use for anything. Mostly opencode.

  source

  • xSikes@feddit.online ⁨7⁩ ⁨hours⁩ ago

    Is that the only community? I see the last activity was 2 months ago.

    source

    • BeefAndPoultry@lemmus.org ⁨5⁩ ⁨hours⁩ ago

      I see the last activity was 2 months ago.

      that means no one from your instance is subscribed, if you subscribe then you’ll get new posts

      you can see that there are new posts here sh.itjust.works/c/localllama

      source

      • -> View More Comments
    • PetteriPano@lemmy.world ⁨5⁩ ⁨hours⁩ ago

      Posts from 16 minutes and 2 days ago here. Maybe your instance is not federating with theirs.

      source

      • -> View More Comments
• GreenKnight23@lemmy.world ⁨5⁩ ⁨hours⁩ ago

  • 96gb ddr5
  • intel i7 12th gen
  • RTX 3060

  it also doubles as my media ripper. I initially built it specifically for that before AI was even a thing.

  source
• auzy1@lemmy.world ⁨10⁩ ⁨hours⁩ ago

  We did the maths. Didn’t seem worthwhile and the payback period was a very long time

  source

  • MagicShel@lemmy.zip ⁨9⁩ ⁨hours⁩ ago

    I think the value is in the privacy. You can hook it up to knowledge and capabilities you’d never share with a cloud AI. But if your use case is strictly coding, I agree.

    source
  • laz@lemmy.dbzer0.com ⁨10⁩ ⁨hours⁩ ago

    True but if you happen to broken macs you can just take thier board with all the unified ram it’s become quite the thing recently even Asus duo with 32gb Intel 9 core ultra board’s have been started to be repurposed like mini pcs for ai use

    source
• greybeard@feddit.online ⁨7⁩ ⁨hours⁩ ago

  I recently gave Strata a shot for my local AI. I’m very impressed with the performance they have eeked out of my hardware. I’ve got a rather beefy gaming desktop I built as an AI server back when the current craze started, before prices started going up, but models up until this point really failed to take full advantage of that hardware. Strata (which uses a Qwen model) both simplified the setup process, and it maxes out the hardware when running.

  Strata acts like an OpenAI compatible API server, I then have Hermes as a “harness” to make an “agent”, which I interact with on “buzz”, which is like discord, but made for bots. Although Hermes also supports discord and several other chat platforms.

  If you have the hardware to run Strata, I recommend setting it up, it comes with a basic chat interface, which you can then grill to assist you with setting up Hermes and whatever else.

  One thing I didn’t mention, is security. I highly recommend you treat AI as unpredictable, unreliable, and capable of being taken over by bad actors. This isn’t a problem for a pure chat interface, but Hermes gives the AI tools to do things, and those tools could be used to take over your computer, scan your network, and create all kinds of havoc. For my part, Hermes is running in a virtual machine which is on an isolated network. I keep tight control on what it does and doesn’t have access to in my network main network. That may not be something you have the hardware or knowledge to do, at the very least I recommend running Hermes in a docker container, and not giving it access to your personal desktop nor credentials to things like your email account.

  source
• MagicShel@lemmy.zip ⁨9⁩ ⁨hours⁩ ago

  This is something I will do probably within the year, but it feels to me like the area is still moving quite rapidly. The model will naturally depend a bit on the hardware. I’m looking at used AI cards because it looks like currently for about $2k you can get one that will run ~350B models, and GLM is nice in that space, though I will experiment with other models once I can run them.

  Things I will do:

  • try to replace Alexa for voice activated smart home management — this will let me go locally controlled I hope (my wife is big on convenience and could give a fuck about privacy so the solution has to be solid).
  • a personal journal so I can have it remember things so later on I can ask things like when is the last time a friend and I hung out, or what was that orchard we went to last October, or what was that great B&B we stayed at, let’s do that again. This is likely the thing that would require a good model — everything else is pretty simple orchestration, aggregation, and summary. Arguably it’s the thing that would make the biggest difference in my life, too. There is a lot of beauty in life but my brain is wired to constantly churn solving problems and be irritated and I rarely stop to appreciate the moments or remember the accomplishments.

  I have a ton of ideas for conveniences AI could do if it had total privacy and security. In short, orchestrate all of the tools and knowledge that I already rely on constantly. Integrate calendar and private info like SSNs, VINs, who has what doctors for what, or who was sick when, smart alerts like the overnight temp will be below freezing next week so make sure the house is winterized, or I wanted to see this movie and it’s releasing or will be leaving theaters soon — oh and next weekend you’re visiting the grandkid, so it’s either now or I’ll remind you again when it comes out on streaming.

  Just having a personal knowledge base always on and totally private would be handy. It would be like having a personal assistant maybe concierge if it could be trusted to that degree.

  source

  • button_masher@lemmy.ml ⁨7⁩ ⁨hours⁩ ago

    Your use cases are quite similar to what I’ve been trying to action.

    One question I keep wondering about is the “security” aspect of it. And how the inputs to the model might work. Yes it can read your calendar and you could “Jeeves, please remind me to hang out with Wooster next week” but it’s easy to forget and I have a hard time logging spontaneous events.

    Any ideas on how you imagine the friend/activity tracker could work?

    Was thinking maybe it could read my smartwatch location history. Or scan the photos I take. But that gets dystopian fast (yes it’ll be local but then I’m worried about security).

    Fun to ponder!

    source

    • MagicShel@lemmy.zip ⁨6⁩ ⁨hours⁩ ago

      I think having a chat interface is part of it so it’s friendly and convenient to use. I think subverting suggested a matrix bot and that would continue the theme of privacy. You could have different channels or groups for different personas. Maybe one is your Majordomo, another is a simpler calendar bot, another is for journaling, etc.

      Then, wire in some cron jobs to invoke the AI to prompt you on various subjects. “Anything you want to share?” every day at 9pm. If you miss a day or two because you were busy or asleep, you can catch it up later or not at all. You could even have it ask what you did at work if your life is like mine and they kinda blend together and you don’t know how many hours you worked.

      source
• state_electrician@discuss.tchncs.de ⁨10⁩ ⁨hours⁩ ago

  I wish I could. I don’t have the hardware for it.

  source
• Shimitar@downonthestreet.eu ⁨9⁩ ⁨hours⁩ ago

  Yes i do.

  I esperiment a lot, but my stable setup is Qwen 3.8 27B Q6_K on my home server. Hardware is Ryzen 9 5950x with 64gb ddr4 and two old but effective NVIDIA RTX A4000 for a total of 32gb VRAM.

  Qwen 3.8 is the best model I can run in terms of output quality. It can perform all tasks pretty well and it’s reasonably fast at 500ts prefill and 30/40ts processing. There are much faster models but honestly not as good.

  I can also run gemma4 all on CPU, with no GPU at all.

  The key point is having both GPU on pci-e CPU, not on chip set bridge, so I am using a PCI riser 8x8x to share the 16x slot between the two GPUs. Inference speed is unaffected, but prefill benefits immensely.

  Software side, use llama.cpp on Linux, you get the best performance possible at the expense of a bit more complexity compared to other engines.

  From a power consumption perspective, I tined down my CPU governor and maximum frequency, and also the GPUs maximum watts. At ide the system is around 80w. At peak, it’s about 280w with the limiters, and inference speed doesn’t suffer too much. Without limiters I can easily reach 650w.

  source
• rtxn@lemmy.world ⁨6⁩ ⁨hours⁩ ago

  Not me. Call me old-fashioned, but I prefer organic.

  source
• Casuls_Die_Thrice@lemmy.zip ⁨10⁩ ⁨hours⁩ ago

  I’d like to, but what’s “good” that can run on a 16GB RAM, i5 12th gen, 3070-equipped desktop?

  source

  • anamethatisnt@sopuli.xyz ⁨10⁩ ⁨hours⁩ ago

    Gemma 4 12b unified as a q6 gguf fits with about 24k context inside your vram. koboldcpp+sillytavern works well on the software side.

    source

    • laz@lemmy.dbzer0.com ⁨10⁩ ⁨hours⁩ ago

      Sc117

      source
  • Wildmimic@anarchist.nexus ⁨8⁩ ⁨hours⁩ ago

    Might wanna try a ternary model, which are a lot denser in comparison.

    Ternary Bonsai 27b is 7.17GB in size, which might fit on your 3070 depending on how much free VRAM you have (and with CPU offloading it should work in all cases). You can also try their edge-computing model designed to be working on smart phones here - that one is less than 4GB in size, which should fit in any case. In my experience they do work very well compared to other models in that size class.

    Be aware that they need a specific llama.cpp build found here

    source
  • eodur@piefed.social ⁨8⁩ ⁨hours⁩ ago

    This is the question to spend your Claude tokens on. It helped me tweak my setup to get better performance.

    source
• ns1@feddit.uk ⁨6⁩ ⁨hours⁩ ago

  If you haven’t seen it already, take a look at !localllama@sh.itjust.works

  source
• joseki@lemmy.zip ⁨9⁩ ⁨hours⁩ ago

  Yes, with OpenClaw I can expose my agents via discord apps on my home discord server, I like that route because of the rich media and UI options (agents can give me drop down selector lists and approve/reject buttons). I have two 24gb Nvidia cards for local models usually Qwen, Mistral, and nemo based models for different purposes. I can chat with the agents anywhere I’m logged into discord and I expose privileged control surfaces and nodes via tailscale and family/shared endpoints via caddy and cloudflared

  source
• Seralth@piefed.seralth.com ⁨10⁩ ⁨hours⁩ ago

  Three models my self, qwen 27b as a dev assistant, a small model rigged up to handle voice input as an assistant that’s connected to my intercom system though out the house. Mostly to run my lights/music/audiobook play back. It’s fun having jarvis control the house after all. Then a image generator.

  Llama and swap on a 7900xtx.

  I haven’t gotten around to setting up phone access but wireguard for remote access form work in my laptop.

  Honestly it’s all basically over priced toys. I’m baffled that people rely on this stuff. I look at it the same way people do with model airplanes or RC cars.

  Over priced toys that are hell of a lot of fun. Break frequently and require more care then they are probably worth to use safely lol

  source
• BeefAndPoultry@lemmus.org ⁨5⁩ ⁨hours⁩ ago

  32GB RAM, RTX 2080 8GB VRAM, I use Tiel Coder 35b-a3b (it’s a fine tune of Qwen 3.6) with Zoo Code

  source
• panda_abyss@lemmy.ca ⁨10⁩ ⁨hours⁩ ago

  I run lemonade on my strix halo box, I change the models I use, but the qwens are pretty good.

  I access it via Tailscale on my phone, then I have a page I can use. The Lemonade interface forces you into the app, so you’ll want some other interface for your LLMs, as the app is junk.

  I used to run llama.cpp and llama swap, but lemonade combines them and makes managing different ai backends easier.

  source

  • Seralth@piefed.seralth.com ⁨10⁩ ⁨hours⁩ ago

    I’ve been eyeing lemonade as I am on llama and swap my self. How much of a pita was the migration. I’m a lazy bastard.

    source

    • MalReynolds@slrpnk.net ⁨9⁩ ⁨hours⁩ ago

      Lemonade is pretty good, especially for AMD GPUs and the lazy, but it’s pretty opinionated (like ollama but not as toxic) and it’s not easy to directly control llama-server parameters. It feels a bit commercial / we want to IPO.

      Seeing as you already have your parameters, llama-swap might be a better fit, lets you configure llama-server directly in a yaml, and switch between them. You can even have multiple versions of the same model (e.g. thinking or not, try out different parameters) and change without reloading.

      Both work fine with e.g. Open WebUI and Pi (lemonade its own integration and llama-server needs an extension).

      source
    • panda_abyss@lemmy.ca ⁨9⁩ ⁨hours⁩ ago

      I pointed lemonade at my llama.cpp model folder but it uses json files to configure them. I think it was fine for importing them or you could have an agent write them for you. After that you can manage models inside the app

      Overall it wasn’t a huge deal, it’s easier than managing llama.cpp directly with rocm updates.

      source
• Sibbo@sopuli.xyz ⁨10⁩ ⁨hours⁩ ago

  I find the idea interesting. However I wouldn’t get the commercial service you are advertising. Maybe something that can run on an older CPU as well. Doesn’t need to be fast, but if it uses up the idle compute I have on my rented server, then why not.

  source

  • StateMachine@lemmy.zip ⁨8⁩ ⁨hours⁩ ago

    I’m not advertising any service. I added an image because Lemmy wouldn’t let me post the question without one. I think I may have chosen the wrong post type or something.

    source

    • Sibbo@sopuli.xyz ⁨4⁩ ⁨hours⁩ ago

      No worries. Yeah, the user interface is difficult. Thanks anyways for the post, it motivated me to self host my own agent, and it works quite well actually.

      source
• hendrik@palaver.p3x.de ⁨10⁩ ⁨hours⁩ ago

  Sure. I have Home Assistant with STT and TTS. And I use KoboldCPP to experiment with smaller LLMs. Nothing fancy, it’ll run on my old laptop, albeit very slow.

  source
• floquant@lemmy.dbzer0.com ⁨10⁩ ⁨hours⁩ ago

  90€/mo to self-host something? Lol

  source
• curbstickle@anarchist.nexus ⁨7⁩ ⁨hours⁩ ago

  As a service? No, not really.

  I use llms for specific tasks, usually small models, for things like identification and tagging (immich, paperless), translation or transcription (whisper), etc. I have a bunch of larger models on a Mac mini and my work laptop for various testing purposes, and have used them to make summaries and other such things.

  They aren’t great for search IMO, and search summaries can be quite bad.

  I do find them useful to interrogate an unknown rs232 protocol or similar, though, which is where larger models come in handy.

  Corporate LLMs are generally garbage to me not only because they are just data farming, but also because they really aren’t good at everything. They might claim it, but the reality is a tailored model is often the better approach.

  Using openwebui as a front end for llama.cpp is a decent strategy, combined with sub-agent routing (so generic model hands off to tailored models then returns the right info). This can be useful for some scenarios.

  A caveat on coding - IMO, they are mostly bad at it. If you are using it to generate a small part of something that needs to be repeated they are handy, but at scale they aren’t great, meaning for the entire project (or even an entire plugin or module, depending on complexity). Maintainability becomes a serious issue. While they have improved somewhat, there are limits to what they can do.

  Macs with M series chips are good options for a hardware base since the memory is shared, and IMO is one of the most cost effective options over a dgpu with a ton of memory. A dgpu with a decent bit of memory (say 8-12GB) is a solid choice for a model thats larger (27b with 3 or 4b active as a mixture-of-experts model) especially if you have an iGPU for regular graphics tasks. Since its all shared on the M series macs, just get something that fits entirely in memory there.

  source
• Wildmimic@anarchist.nexus ⁨8⁩ ⁨hours⁩ ago

  Hardware is a 5070 12GB VRAM, an AMD 5950x and 64G of DDR4 running on Nobara (Fedora spinoff).

  I use 3 models for most things: Ternary Bonsai 27b or and Gemma 4 E4B Uncensored Q8_0 for everyday stuff, and Qwen Agentworld 35B A3B Q8_0 with CPU offloading for tasks that require to keep the model on track for long periods of time.

  Everyday performance is best with Bonsai for me - it comfortably fits in my VRAM and is noticeably better with staying on task than similarily sized models. But if i want it to work overnight on a task without issue, the lower speed of the agentic Qwen is outweighed by its capability to do this.

  They are allowed tool calls for websearch and reading webpages and have a sandboxed directory for R/W operations, but anything regarding command line and R/W access in other areas has to be manually authorized by me (i’m not willing to play with fire here).

  They are currently not reachable from outside, but i am currently building a smaller machine as a home server, which will then be using a 3070Ti and 32G RAM to provide probably the Bonsai Model for me on the go per VPN.

  The smaller models are here for quickly creating documents i do not care to do myself, RAG for technical documentation, creating quick python scripts for automating small tasks and teaching me RegEx and Python(mostly by analyzing the LLM output and playing around with it) and also some RP as NPC in a fantasy setting. The agentic model is currently working on cataloguing my games collection (around 36000 titles) by looking up games, their genres and descriptions and filling in blanks in the database. (Funnily, this task was refused by any model freely available because of the size of the database. Even the local models moan and bitch that it takes so long. No shit, this is why i outsource this!)

  source
• pech@lemmy.world ⁨9⁩ ⁨hours⁩ ago

  I’ve been using NixOS/nix containers to host some llama.cpp servers and Hermes Agent. Though I wrote a flake to get Project Maya working so I can serve GLM-5.3-Flash and in legit blown away by how well it works. It’s a fork of strata/llama.cpp that focuses on where the relevant experts are loaded and has a neat dashboard to track their use.

  My server is assembled into an openair mining rig with a Tesla P40, RTX 3080, and a few Tesla V100s. All of which I bought used off eBay.

  I purchased my first domain to self host a matrix server that I can access via my phone and I also have tailscale set up so I can ssh into the server via my laptop while traveling

  source
• ch00f@lemmy.world ⁨8⁩ ⁨hours⁩ ago

  I just got qwn3-4b-instruct working on an Intel A750 with 8 gigs of vram.

  I use it to power Home Assostant/Voice Assistant on a few M5Stack echos around the house to replace cloud based smart speakers.

  source