Skip to content

Knowledge that lights the world

How Does Generative AI Work? Large Language Models, Image Generation and Their Limits

AI tools that write text, answer questions and create images entered daily life within a few years. How large language models are trained, why they sometimes make up false information and how to use these tools wisely.

How Does Generative AI Work? Large Language Models, Image Generation and Their Limits

“Generative AI” is the general name for AI systems that can produce new text, images, audio or code. From late 2022, through chat assistants and image generators, these systems entered the daily lives of millions. But how can a computer program write an article or draw a landscape that never existed?

Large language models

Text-generating systems are built on “large language models”. These rely on a neural network architecture called the “transformer”, introduced in a 2017 paper by Google researchers. The model is trained on enormous amounts of written data — web text, books and code. The training task is surprisingly simple: see a piece of text and predict the next piece — the word or part of a word called a “token”. By attempting this prediction across billions of examples, the model learns the structure of language, relationships between concepts and a great deal of knowledge as statistical patterns.

Becoming a chat assistant

A model fresh from pre-training knows how to continue text but does not always understand what the user wants. So in a second stage it is fine-tuned with question-and-answer examples and with feedback in which people judge which responses are more helpful, accurate and safe. The model thus becomes an assistant that follows instructions, can hold a conversation and is more careful with harmful requests.

Image generation

Most text-to-image tools use “diffusion models”. During training these models learn how noise is added to images step by step and then how to remove it. In use, they start from random noise and remove it step by step in a way that fits the user’s description, creating a new image.

Limits and risks

  • Hallucination: Language models produce what is “likely”, not what is “true”. So they can sometimes produce entirely convincing but false information, non-existent sources or quotes.
  • Currency: A model’s knowledge is limited to when its training data was collected; some tools fill this gap with web search.
  • Bias: Biases in the training data can be reflected in the model’s answers.
  • Privacy: Check a tool’s data policy before sharing personal, financial or confidential business information.
  • Copyright and originality: Legal debates continue over rights in training data and generated content.

Using it wisely

Generative AI is a powerful helper for drafting, brainstorming, summarising long texts, explaining a topic in different ways and writing code. But on important matters such as health, law and finance, always verify information with reliable sources and experts, and keep the final decision and responsibility with yourself.

Frequently heard terms: a short glossary

  • Prompt: The instruction or question given to the model. The clearer and more context-rich the prompt, the more accurate the result.
  • Context window: The amount of text the model can "see" at once. If a conversation exceeds this limit, the oldest parts may no longer be taken into account.
  • Temperature: The setting for randomness in answers. A low value gives more consistent and repeatable results, a high value more varied and creative ones.
  • Fine-tuning: Further training of a pre-trained model with additional data for a specific task or domain.
  • Multimodal: Models that can understand and generate different types of data such as images, audio or video alongside text.

How can accuracy be improved? Retrieval-augmented generation (RAG)

Language models "remember" information from their training data, so on topics that are out of date or that they have never seen they can give made-up answers. A common method for reducing this problem is "retrieval-augmented generation" (RAG), described by Lewis and colleagues in 2020. In this approach the system first searches a trusted collection of documents for passages relevant to the question, then asks the model to answer based only on those passages.

RAG makes it easier to show which document an answer is based on and makes it possible to work with information the model has never seen, such as internal company documents. It does not fully solve the problem, however: if the search brings back the wrong document or the model misreads it, errors can still occur. For this reason, the 2024 generative AI risk profile of the US National Institute of Standards and Technology (NIST) also lists practices recommending human verification of outputs.

Frequently asked questions

Does AI really “understand”?

This is still debated among scientists and philosophers. Models can use complex patterns of language remarkably well, but there is no consensus on whether they have anything like human understanding.

Sources and further reading

  1. Ashish Vaswani et al., “Attention Is All You Need” (2017).
  2. Jonathan Ho et al., “Denoising Diffusion Probabilistic Models” (2020).
  3. OECD AI Principles; UNESCO Recommendation on the Ethics of Artificial Intelligence (2021).
  4. Lewis P. et al. – Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS (2020)
  5. NIST AI 600-1 – Generative Artificial Intelligence Profile (2024)
Share
Follow YulaHubGet the day's most important stories first.
RSS

Leave a comment

Your email address will not be published. Required fields are marked *