brucethemoose
@brucethemoose@lemmy.world
- Comment on The Media's Pivot to AI Is Not Real and Not Going to Work 2 weeks ago:
, especially since something like a Mixture of Experts model could be split down to base models and loaded/unloaded as necessary.
It doesn’t work that way. All MoE experts are ‘interleaved’ and you need all of them loaded at once, for every token. Some API servers can hotswop wholes models, but its not fast, and rarely done since LLMs are pretty ‘generalized’ and tend to serve requests in parallel on API servers.
The closest to what you’re thinking of is LoRAX (which basically hot-swaps Loras efficiently). But it needs an extremely specialized runtime derived from its associated paper, hence people tend to not use it since it doesn’t support quantization and some other features as well: github.com/predibase/lorax
There is a good case for pure data processing, yeah… But it has little integration with LLMs themselves, especially with the API servers generally handling tokenizers/prompt formatting.
But, all of its components need to be localized
They already are! Local LLM tooling and engines are great and super powerful compared to ChatGPT (which offers no caching, no raw completion, primitive sampling, hidden thinking, and so on).
- Comment on The Media's Pivot to AI Is Not Real and Not Going to Work 2 weeks ago:
SGLang is partially a scripting language for prompt building leveraging its caching/logprobs output, for doing stuff like filling in fields or branching choices, so it’s probably best done in that. It also requires pretty beefy hardware for the model size (as opposed to backends like exllama or llama.cpp that focus more on tight quantization and unbatched performance), so I suppose theres not a lot of interest from more local tinkerers?
It would be cool, I guess, but ComfyUI does feel more geared for diffusion. Image/video generation is more multimodel and benefits from dynamically loading/unloading/swapping all sorts of little submodels, loras and masks, applying them, piping them into each other and such.
LLM running is more monolithic: you have the 1 big model, maybe a text embeddings model as part of the same server, and everything else is just processing strings to build the prompts which one does linearly om python or whatever. Stuff like CFG and Loras do exist, but aren’t used much.
- Comment on The Media's Pivot to AI Is Not Real and Not Going to Work 2 weeks ago:
Not specifically. Ultimately, ComfyUI would build prompts/API calls, which I tend to do in Python scripts.
I tend to use Mikupad or Open Web UI for more general testing.
There are some neat tools with ‘lower level’ integration into LLM engines, like SGlang (which leverages caching and constrained decoding) to do things one can’t do over standard APIs: docs.sglang.ai/frontend/frontend.html
- Comment on The Media's Pivot to AI Is Not Real and Not Going to Work 2 weeks ago:
I mean, I run Nemotron and Qwen every day, you are preaching to the choir here :P
- Comment on The Media's Pivot to AI Is Not Real and Not Going to Work 2 weeks ago:
AI is a tool (sorry)
This should be a bumper sticker. Also, thanks for this, bookmarking 404, wish I had the means to subscribe.
My hope is that the “AI” craze culminates in a race to the bottom where we end up in a less terrible state: local models on people’s phones, reaching out to reputable websites for queries and redirection.
And this would be way better for places like 404, as they’d have to grab traffic individually and redirect users there.
- Comment on Confirmed - China bans NVIDIA chips and accelerates its total independence from US technology 1 month ago:
Yeah honestly the Nvidia ban was stupid.
Everyone in the AI research space was saying it, but no, our old policymakers are captured by Altman, Musk and tech bros who would burn anything for their two years of pure anticompetitiveness.
The running joke is that the Nvidia ban was the best thing to ever happen to Chinese research, as it made them thrifty, while big US companies are lazily burning huge GPU farms scaling up and… not improving anything.
- Comment on X's new 'encrypted' XChat feature seems no more secure than the failure that came before it 1 month ago:
I have to wonder who this appeals to?
Most are already trapped in something established like Discord, WeChat, FB Messenger. As said, security isn’t a strong point, and there’s no engagement angle.
I guess if you already spend tons of time on X it’s kinda convenient?
- Comment on EA never grasped Dragon Age's value as an RPG, says Inquisition writer 2 months ago:
Side note, but even with all their troubles/turnover, I still love RPS’s hint of bite in their news writing (outside the columns).
- Comment on Tiny Corp heralds world's first AMD GPU driven via USB3 — eGPUs tested on Apple Silicon, with Linux and Windows also supported 2 months ago:
Tinycorp generates these headlines every once in awhile, but as far as I can tell no one uses it. At least not in the tinkerer space I can see.
It’d be cool if they can eat away at PyTorch, XLA and whatever else… Some day…
- Comment on Researchers unveil LegoGPT, an AI model that designs physically stable Lego structures from text prompts and currently supports eight standard brick types 2 months ago:
A pretty long time.
Niche models are tons of fun though.
- Comment on Consumers make their voices heard as Microsoft's huge venture flatlines in popularity 2 months ago:
Comment from the source:
Microsoft poisoned their own well with all the changes they have been forcing on users lately. The update nagging, resetting the default browser to edge, the the ads in windows features, and integrating bing into the start menu have all trained users that when Microsoft starts pushing something new, it probably isn’t great and should just be ignored, like ads in phone apps.
That ^. So much that.
Also, the copilot llm itself sucks. Local models are neat within their limitations, and they’d be even better if Microsoft made them trainable/customizable, did better RAG, or whatever, but they just shoved a bad thing down user’s throats, and now they’ve poisoned another well.
- Comment on Microsoft removes Windows 11 24H2 official support on 8th 9th 10th Gen Intel CPUs 5 months ago:
As crazy as that is, playing devil’s advocate, Comet Lake is basically the aging Skylake architecture.
Ice Lake though? WTF.
- Comment on Meta asks the US government to block OpenAI’s switch to a for-profit 7 months ago:
The limit is already (apparently) starting to be data… and capital, lol.
There could be a big computer breakthrough like , say, fast bitnet training that makes the multimodal approach much easier to train though.
- Comment on Meta asks the US government to block OpenAI’s switch to a for-profit 7 months ago:
Well for one, I directly disagree with Altman’s fundamental proposition, they don’t need to “scale” AI so dramatically to make it better.
See: Qwen 2.5 from Alibaba, a fraction of the size, made with a tiny fraction of the H100 GPUs and highly competitive (and (mostly) Apache licensed). And frankly, OpenAI is pointedly ignoring all sorts of open research that could make their models notably better or more powerful efficient, even with the vast resources and prestige they have… they seem most interested in anticompetitive efforts to regulate competitors that would make them look bad, using the spectre of actual AGI (which has nothing to do with transformers LLMs) to scare people.
Even if doing it for the wrong reasons, I feel like Google would be right to oppose Mozilla axing the nonprofit division if they were somehow in a similar position to OpenAI. Their mission of producing a better, safer browser would basically be lying through their teeth.
- Comment on Meta asks the US government to block OpenAI’s switch to a for-profit 7 months ago:
And in that case, will the Llama fork be the same as the Meta fork? We are talking about AI that has a considerable development, companies would probably not participate because it is not an open source license and its clause limits in those aspects.
Llama has tons of commercial use even with its “non open” license, which is basically just a middle finger to companies the size of Google or Microsoft. And yes, companies would keep using the old weights like nothing changed… because nothing did. Just like they keep using open source software that goes through drama.
Also you have to think that if the new version of Llama with the new license is 3 times better than Llama with the previous license, do you really think that the community will continue to develop the previous version?
Honestly I have zero short term worries about this because the space is so fiercely competitive. Also much of the ecosystem (like huggingface and inference libraries) is open source and out of their control.
And if they go API only, honestly they will just get clobbered by Google, Claude, Deepseek or whomever.
In the longer term… transformers will be obsolete anyway.
- Comment on Meta asks the US government to block OpenAI’s switch to a for-profit 7 months ago:
A small startup called Arcee AI actually “distilled” logits from several other models (Llama, Mistral) and used the data to continue train Qwen 2.5 14B (which itself is Apache 2.0). It’s called supernova medius, and it’s quite incredible for a 14B model… SOTA as far as I know, even with their meager GPU resources.
A company called upstage “expands” models to larger parameter counts by continue training them. Look up the SOLAR series.
And quite notably, Nvidia continue trained Llama 3.1 70B and published the weights as Nemotron 70B. It was the best 70B model for awhile, and may still be in some areas.
And some companies like Cohere continuously train the same model slowly, and offer it over API, but occasionally publish the weights to promote them.
- Comment on Meta asks the US government to block OpenAI’s switch to a for-profit 7 months ago:
Call them ClosedAI like the community does, and it’s much easier, lol.
- Comment on Meta asks the US government to block OpenAI’s switch to a for-profit 7 months ago:
Also competition is stiff. Alibaba is currently handing their butts to them with Qwen 2.5. Deepseek (a Chinese startup), tencent and Mistral (French) are giving them a run for their money too, and there are even some that “continue train” their old weights.
- Comment on Meta asks the US government to block OpenAI’s switch to a for-profit 7 months ago:
No, they can’t, because you can just pull the git repo with the old license as use them as they were at the time of upload, just like any software on a git repository. And too many people have them downloaded to delete them from the internet.
There are also finetunes inheriting the old license, and those orga are not going to pull the weights.
- Comment on Meta asks the US government to block OpenAI’s switch to a for-profit 7 months ago:
It’s not open source, but open weights, documented, relatively permissively licensed and all the inference/finetuning libraries for it are open source.
- Comment on Meta asks the US government to block OpenAI’s switch to a for-profit 7 months ago:
They don’t use OpenAI at all.
- Comment on Meta asks the US government to block OpenAI’s switch to a for-profit 7 months ago:
I know Meta has some heavy sins, but they’ve pushed ‘open’ ML a long time. They developed, and continue to fund PyTorch, and they basically standardized the open LLM architecture with Llama to the point literally everyone uses it almost unmodified now, just to name two examples.
They also have a commercial interest in their open weight model ecosystem succeeding over OpenAI’s completely closed models and research. And TBH they have a good shot, as OpenAI really seem to have stagnated.
Also, Altman is a straight up con artist. I wouldn’t be surprised if Facebook employees hate him.
- Comment on Nintendo shuts down Ryujinx 9 months ago:
Twitter screenshot of this linked in slack that evening.
The modern internet in a nutshell, lol.
- Comment on Nintendo shuts down Ryujinx 9 months ago:
Discord is even worse, as you need to find an invite to a specific Discord, and sometimes go through a lengthy sign up process for each Discord.
Some won’t let you sign up without a phone #.
- Comment on Nintendo shuts down Ryujinx 9 months ago:
Matrix.
And… Lemmy.
It doesn’t matter though, the problem is the critical mass is migrating to Discord and shunting everything out of view.
- Comment on Nintendo shuts down Ryujinx 9 months ago:
I’m a bit bitter this was apparently only announced through discord?
The future of social media is fragmented siloes, I guess.
- Comment on Astronomers discover technique to spot AI fakes using galaxy-measurement tools 1 year ago:
Simple, but quite clever.
Theoretically they could finetune for consistent eye reflections, but even then I suspect image models wouldn’t get it right. Physical correctness is not really something they actually do.