Not OP but running local LLMs for exactly the privacy reasons you describe, some practical notes:

• For a no-GPU start: llama.cpp (or Ollama as a wrapper) with Qwen3-4B/8B or Llama-3.1-8B at Q4_K_M runs fine on a plain x86 box with 16GB RAM. Speed on CPU is ~5-15 tok/s for 4B-class models, which is usable for search summarization and drafting, painful for long chats.
• Quantization is the whole game: 8B Q4 is ~5GB and close to fp16 quality for everyday tasks; don’t run fp16 on consumer hardware, run the bigger model more quantized instead.
• For your “web search without profiling” use case the actual stack that matters is: SearXNG (self-hosted metasearch, no tracking) -> fetch top N pages -> feed text to the local model for summarization. The LLM part is almost the trivial bit; the search layer is what fixes the privacy problem.
• Remote access: don’t expose Ollama/Open WebUI directly. WireGuard (or Tailscale if you don’t want to manage keys) and you get it on the phone with zero ports open.
• One honest caveat: local small models are noticeably worse at factual recall than cloud ones. Where they shine is processing text you hand them (summaries, extraction, rewriting). Design your workflows around that and the gap mostly disappears.