vhstape

@vhstape@lemmy.ml

This is a remote user, information on this page may be incomplete. View at Source ↗

I make computers

⁨Comment⁩ on ⁨[AIT] what is your selfhosted local LLM stack?⁩ ⁨⁨2⁩ ⁨weeks⁩ ago⁩:
I’ve got an M4 Mac mini with 32 GB of unified memory, so I can run MoE models in the <40B parameter range quite comfortably. I use LM Studio as the backend with Hermes as the frontend. Works great for everyday chat, web search, and RAG.