I was watching the RFK Jr questioning today and when Bernie was talking about healthcare and wages I felt he was the only one who gave a real damn. I also thought “Wow he’s kinda old” so I asked my phone how old he actually was. Gemini however, wouldnt answer a simple, factual question about him. What the hell? (The answer is 83 years old btw, good luck america)
By the way…
Local LLM gang represent! ✌️
Local LLMs aren’t perfect but are getting more usable. There’s abliterated models and uncensored fine tunes to choose from if you don’t like your LLM rejecting your questions.
I have dabbled with running llms locally, I’d absoluteley love to but for some reason amd dropped support for my GPU in their ROCm drivers, which are needed for using my GPU for ai on Linux.
When I tried it fell back to using my cpu and I could only use small models because of the low vram of my RX 590 😔
Dont give up you got this! I had luck with using vulkan in kobold.cpp as a substitute for rocm with my amd rx 580 card.
How do I start?
First you need to get a program that reads and runs the models. If you are an absolute newbie who doesn’t understand anything technical your best bet is llamafiles. They are extremely simple to run just download and follow the quickstart guide to start it like a application They recommend llava model you can choose from several prepackaged ones. I like mistral models.
Then once you get into it and start wanting to run things more optimized and offloaded on a GPU you can spend a day trying to setup kobold.cpp.
They both start a local server you can point your phone or other computer on WiFi network to it with local ip address and port forward for access on phone data.
My primary desktop has a typical gaming GPU from 4 yrs ago Primary fuck around box is an old Dell w onboard GPU running proxmox NAS has like no gpu Also have a mini PC running HAOS And a couple of unused pi’s Can I do anything with any of that?
Your primary gaming desktop gpu will be best bet for running models. First check your card for exact information more vram the better. Nvidia is preferred but AMD cards work.
First you can play with llamafiles to just get started no fuss no muss download them and follow the quickstart to run as app.
Once you get it running learn the ropes a little and want some more like better performance or latest models then you can spend some time installing and running kobold.cpp with cublas for nvidia or vulcan for amd to offload layers onto the GPU.
If you have linux you can boot into CLI environment to save some vram.
Connect with program using your phone pi or other PC through local IP and open port.
In theory you can use all your devices in distributed interfacing like exo.