Rendered at 06:24:48 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
karimf 3 minutes ago [-]
Not sure why it's on the front page now, but I highly recommend using llama.cpp for running AI model locally vs using other inference framework, unless you have a very specific requirement.
ggerganov and the team have done a stellar job maintaining the quality while still being fast to implement new models/improvements.
walrus01 53 seconds ago [-]
At this point the options are llama-server or vLLM if you're serious about running things at your desk in the under 256GB RAM size class (70B, 120B size models). In addition to, of course, 27B to 35B size things. With of course a ton of compile time build customization options for whatever specific hardware platform you want to run either llama or vllm on.
bhouston 5 minutes ago [-]
It seems that llama.app is a direct competitor to ollama.com
I can understand the desire for the llama.cpp project to want to own the end user relationship, it is true that previous to this they were a tool provider and not really owning the end user experience.
tosh 44 minutes ago [-]
I was a bit suspicious of the url but it is also listed on llama.cpp github
>Anything that suggests curl into bash just plain sketches me out.
How is it different than trusting any other method of installation? If URL has https and is from an author you trust i dont see the difference.
walrus01 3 minutes ago [-]
I mean it's probably not, I just haven't got used to it yet. It's about the same level of security as installing a windows app on win2000 25 years ago and blindly downloading a .exe off the internet and running it to get into the install wizard. But indeed I also kind of blindly trust that whatever I'm getting from the debian trixie officially gpg-signed packages isn't backdoored.
One thing I do not do as a matter of practice is install things with a ridiculous number of recursive npm dependencies.
ur-whale 9 minutes ago [-]
> Anything that suggests curl into bash just plain sketches me out.
Yeah, 100% and it's becoming more and more of a thing, see rust install for example.
OTOH, if you're installing llama.cpp, you're more than likely planning to run an LLM on you Linux box with an agentic harness, so a curl into bash thing might be the least of your security concerns, :-)
jakkos 2 minutes ago [-]
> security concerns
Yeah I recently tried the coding harness that's recommended here, Pi, in a bubble wrap sandbox and was horrified to learn that it spams multiple warnings at you if you don't give it write access to its own config/extension folder... Everyone else is rawdogging it I guess.
walrus01 6 minutes ago [-]
One way I prevent possible catastrophic fuckups is that the 'doing code work' box that runs opencode or pi or whatever, is its entirely own separate VM and desktop environment (running as a xen or kvm guest and with its own LVM logical volume as boot/root and /home disk), than the machine running llama-server itself.
The harness gets the openai-compatible endpoint fed into it to talk to llama-server across the network, but the VM has no access whatsoever to my personal files, mail, backups/deep storage, fileserver, Documents folder, etc.
nexawave-ai 10 minutes ago [-]
I think I can probably run Gemma 3 12B on my macbook M3 pro with 18GB. The question is, should I do it? This small model is probably not capable of doing a lot or advanced coding or reasoning. What else could it be used for, since it can run locally and privately?
helsinkiandrew 2 minutes ago [-]
I'm confused, is this from Meta? There's no attribution anywhere. Surely releasing an AI tool called llama breaks their trademark if not
dlcarrier 29 minutes ago [-]
I tried to run in on my Arc A770, but all of the binary releases I could find were compiled without OpenVINO support enabled. I tried compiling it myself, but after two days of the compiler running it failed.
madushan1000 24 minutes ago [-]
Two days sounds like a lot, both llama.cpp and openvino only takes a few minutes to compile on any decent modern cpu.
gnull 3 minutes ago [-]
It also even easier to get working and integrate into your system in a sustainable manner with NixOS. Do it by hand or throw an LLM at it, it will get you a declarative patch for your NixOS config that brings llama-cpp into your config that you can review and add under version control (no random `make install` build artifacts contaminating your system, no wondering "what was it that I ran? what are all these files? how do I do the same with a newer version?" a couple months later). There's also likely some build cache where Nixoids have already build what you want.
I had a great experience with llama-cpp with Nvidia backend on NixOS.
(Sorry for being that guy.)
walrus01 9 minutes ago [-]
the full set of llama.cpp binaries builds in under 5 minutes with an unmodified build workflow straight from their github page on a literally ten year old dual xeon.
puzzlingcaptcha 20 minutes ago [-]
I'm getting roughly twice the pp speed using ik_llama fork with a Cascade Lake Xeon, but their MTP implementation seems faulty compared to llama.cpp so token generation is often slower. Can't have nice things.
whateveracct 6 minutes ago [-]
meta is gonna fuck it all up lol. hf samA and dario LOLLLL ur cooked
ggerganov and the team have done a stellar job maintaining the quality while still being fast to implement new models/improvements.
I can understand the desire for the llama.cpp project to want to own the end user relationship, it is true that previous to this they were a tool provider and not really owning the end user experience.
https://github.com/ggml-org/llama.cpp
(I still deeply distrust curlpipes in general though.)
Git clone llama.cpp and build it, it's not hard.
https://github.com/ggml-org/llama.cpp/blob/master/docs/build...
literally just a few steps for the basics:
git clone https://github.com/ggml-org/llama.cpp
cmake -B build
cmake --build build --config Release
How is it different than trusting any other method of installation? If URL has https and is from an author you trust i dont see the difference.
One thing I do not do as a matter of practice is install things with a ridiculous number of recursive npm dependencies.
Yeah, 100% and it's becoming more and more of a thing, see rust install for example.
OTOH, if you're installing llama.cpp, you're more than likely planning to run an LLM on you Linux box with an agentic harness, so a curl into bash thing might be the least of your security concerns, :-)
Yeah I recently tried the coding harness that's recommended here, Pi, in a bubble wrap sandbox and was horrified to learn that it spams multiple warnings at you if you don't give it write access to its own config/extension folder... Everyone else is rawdogging it I guess.
The harness gets the openai-compatible endpoint fed into it to talk to llama-server across the network, but the VM has no access whatsoever to my personal files, mail, backups/deep storage, fileserver, Documents folder, etc.
I had a great experience with llama-cpp with Nvidia backend on NixOS.
(Sorry for being that guy.)