Across all industries, the speed at which AI-related apps and services are being launched or updated continues to be breathtaking. Unfortunately, while these foundation-model level updates offer greater functionality (sometimes with the gentlest of ‘hey this just might end the world’ caveats to go along with them), privacy and data security continues to feel like an after-thought.
That said, when it comes to privacy, there is a faint-ish-but-growing beacon of hope on the horizon in the form of increasingly powerful Open Source Local large language models (LLMs). Let’s look at what this means.
Foundation models like ChatGPT, Claude, Gemini, etc. run on servers hosted by their parent companies, which means that any time that you ask a question, upload a document, or set them on an agentic workflow, all of the information in that request passes through their servers. While these companies do their best to convince users that their data is safe, and will not be accessed by their employees or any other ne’er-do-wells, the hard truth is that once this data is uploaded to someone else’s servers, you cannot guarantee anything in terms of privacy. This might not be a big deal when you’re asking it why your Peach Cobbler came out burnt when you followed the recipe, but it absolutely is a big deal when your requests involve sensitive data, like building custom curriculum from a client’s proprietary internal processes that are under NDA, or a coach or facilitator running sensitive conversation transcripts for analysis.
Some companies have ‘Private AI’ or ‘Custom AI’ solutions (looking at you, Copilot) that are meant to be fully secure, not allow data to be sent out of the company’s virtual walls, and therefore should be fully trusted. Trust is on a spectrum though, right? In theory there shouldn’t be any security issues with setups like this, but putting all of your trust and your company or customer data in their hands feels…tenuous? Risky? Foolish?
For those instances where you want to be absolutely certain that your data (learner records, assessment results, or client-specific materials) never leaves your computer, local LLM models are here to save the day. The ‘local’ in local LLM means that you install and run these models on your own machine. The prompts, files, and workflows are handled completely on your hardware, to the point where you can even prevent them from accessing outside computer networks and they will still run fine. The upside to this is a marked difference in security – secrets are only secret until they’re said out loud, and a local LLM will make absolutely certain that nothing gets shared unless you explicitly tell it to.
There are certainly compromises to local LLM setups, especially in two areas – capabilities, and speed. Companies like ChatGPT and Anthropic have raised a staggering amount of money, and most of that goes towards hardware that allows their AI services to run as powerfully and as quickly as possible. Your personal computer is almost certainly only a fraction as fast as dedicated AI servers, so questions that come back in milliseconds on traditional AI systems might take seconds, or even minutes to run on a local LLM setup. The good news is that consumer hardware is getting more and more capable, and the local models are getting hugely closer in functionality to their network-based brethren with every release, so the future is promising.
So, how does one create a local LLM setup? That will come in part two!