Submitted 6 hours ago by Alvaro@lemmy.blahaj.zone to selfhosted@lemmy.world
What tools and projects do you use for your local LLM chats and agents?
I’m trying to sift through the mess of LLM harness and tools to find how to really get the most out of local LLMs.
The anti-AI sentiment here is ridiculous. They were asking a legitimate question!
I just use ollama locally but I keep meaning to setup OpenRouter because I like to play with larger models that my hardware can’t handle and I’d like to be able to pick the best deals 🤷♀️
Isn’t ollama made by Meta?
I think you’re thinking of the Llama family of llm models which are owned/made by Meta.
Ollama is an open source server software and model repository for llm models. I won’t say Meta hasn’t contributed to the project, but as far as I know they don’t own it and didn’t start it.
I just use llama.cpp with Zoo Code, the VSCode extension
join us !localllama@sh.itjust.works
This community won’t be much helpful but there is one specific for local LLMs. I think it was ai_ and some other…
That’s it.
I’ve got an M4 Mac mini with 32 GB of unified memory, so I can run MoE models in the <40B parameter range quite comfortably. I use LM Studio as the backend with Hermes as the frontend. Works great for everyday chat, web search, and RAG.
It depends on how complex you want to make it.
The simplest way would be to use a GUI tool like LM Studio, it has the benefit of selecting the model quants that are likely to work on your hardware and having sane defaults for the various parameters that you can work from.
Typically you’ll want to increase the context window, the default is good for short one-off questions but if you’re going to do RAG or coding tasks you will need a much larger context window. You will typically find this through trial and error, increasing until you hit memory limits or notice a large slowdown on tokens/second.
You can also have the model respond to an OpenAI-like API interface on a designated port/interface if you want to run non-chat tools through the same model.
Then, depending on the task that you’re trying to solve, you may want to find an MCP server built for that task which can provide the ability for the AI to interact with tools in a way that’s more controlled (giving an AI raw terminal access on a user’s machine is typically a bad idea).
If you need something more flexible that you can incorporate into scripts and automation, llama.cpp has been popular for a while. Like Linux distros, everyone is going to have an opinion on the best and whatever you decide on will be the wrong choice. There’s also Ollama and vLLM if you want to run a datacenter.
Model-wise it completely depends on the task. Do you need a multi-modal model that supports vision, is it going to be coding, is it trained to use tools/MCP servers, does it use reasoning, etc. The Qwen models are open weight and available in many different quants. The version 3.5 models run well on consumer hardware and can use vision, reasoning and tools.
I’m running it through Llama.cpp but it will likely be available via LM Studio as well.
Llama-swap running llama.cpp and strata. For the tools I use oh my pi and nvim with plugins
StrawberryPigtails@discuss.tchncs.de 32 minutes ago
Currently, I’m using Hermes and OpenCode mostly, backed by Ollama running a couple of different models. Usually I’m running the Gemma4 or Gwen3.5 models.
Memory management is done through Obsidian as every harness seems to have really good skills for it.