Skip to content

Knowledge that lights the world

What Are Open-Weight AI Models? Running AI on Your Own Computer with Llama, Mistral, Qwen, DeepSeek and Gemma

Some AI models run only on a company’s servers, while others have their “weights” shared publicly and can run on your own computer. The difference between open weights and open source, well-known model families, tools for running them locally, hardware needs, advantages and limits.

What Are Open-Weight AI Models? Running AI on Your Own Computer with Llama, Mistral, Qwen, DeepSeek and Gemma

When you use assistants such as ChatGPT, Gemini or Claude, what you type is sent to the company’s servers and the answer is produced there. Alongside them, “open-weight” models, whose trained files anyone can download and run on their own computer or server, are spreading fast.

What does “weights” mean?

A language model stores what it learns from text during training in billions of numerical values (parameters, or “weights”). The model file consists mainly of these weights. With an open-weight model this file is shared, so the model can run without an internet connection or the company’s servers.

Is open weight the same as open source?

Not always. According to the “Open Source AI Definition” published by the Open Source Initiative (OSI) in 2024, for a model to count as open source, the training code and sufficient information about the training data must also be shared, and use must be unrestricted. Many popular models share only their weights and come with their own licence terms, for example requiring extra permission for companies above a certain number of users or restricting certain uses. So read a model’s licence before using it commercially.

Well-known model families

  • Llama: Developed by Meta, a pioneer in popularising open-weight models.
  • Mistral: Models from the French company Mistral AI.
  • Qwen: Alibaba’s multilingual model family.
  • DeepSeek: Models from the Chinese company DeepSeek, noted especially for reasoning and coding.
  • Gemma: Google’s relatively small, efficient open models.

These families release new versions often; always check their official pages for the current state and licences.

How to run them on a computer

Free tools such as Ollama, LM Studio and llama.cpp let you download and run open models with a few clicks or a single command. Models are usually compressed with a method called “quantization” so they need less memory. Small models can run even on a recent laptop, while large models need lots of memory and a powerful graphics card. As a rule, the bigger the model, the more capable it is and the more hardware it needs.

Advantages

  • Privacy: Data never leaves your device, a big plus when working with sensitive documents.
  • Offline use: Once downloaded, the model can run without a connection.
  • Customisation: Organisations can fine-tune the model on their own data for specific tasks.
  • Cost control: For heavy use, running on their own hardware can be more predictable for some organisations.

Limits and risks

  • They require hardware and some setup knowledge.
  • Small models can make mistakes and hallucinate more often than the largest closed models.
  • Because safety measures can easily be changed by users, debates about misuse continue.
  • Download model files only from trusted sources.

Licences: what to watch for with "open" models

Being able to download a model's weights does not mean you can use it freely for any purpose. The licence text is what really decides, and there are important differences between families:

  • Permissive licences: Some models, such as Mistral 7B, were released under the Apache 2.0 licence, and DeepSeek-R1 under the MIT licence. These licences generally allow commercial use and modification.
  • Community licences: Meta's Llama models are distributed under their own community licence; for example, companies with more than 700 million monthly active users must obtain a separate licence, and compliance with an acceptable use policy is expected.
  • Terms of use: Google's Gemma models are bound by a separate terms-of-use text and a list of prohibited uses.
  • Different licences within one family: In some families, such as Qwen, the licence can change with the model size, so the page of each release needs to be read separately.

If you are thinking of using a model in a product or service, check the licence and usage policy for the exact version you downloaded; if in doubt, seek legal advice.

What hardware is enough? A rough memory calculation

There is a simple way to estimate how much memory a model needs: multiply the number of parameters by the number of bytes per parameter. At 16-bit (2-byte) precision, a model with 7 billion parameters needs about 14 GB of memory for the weights alone. With 4-bit quantisation the same model drops to roughly 4–5 GB, which makes it runnable on many current laptops.

On top of this comes extra memory that grows with the length of the conversation (the context cache), so it is wise to leave some headroom above the estimate. The llama.cpp project, widely used for running models locally, and its GGUF file format make it possible to run quantised models on the processor even on computers without a graphics card; speed, however, varies considerably with the hardware.

Sources and further reading

  1. Open Source Initiative (OSI), “The Open Source AI Definition 1.0” (2024).
  2. Meta AI, Llama licence and acceptable use policy pages.
  3. Hugging Face, open model hub and documentation.
  4. Ollama and LM Studio official documentation.
  5. Apache Software Foundation – Apache License 2.0
  6. Meta – Llama 3 Community License Agreement
  7. ggml-org – llama.cpp (GitHub) and GGUF documentation
Share
Follow YulaHubGet the day's most important stories first.
RSS

Leave a comment

Your email address will not be published. Required fields are marked *