How Do Large Language Models (LLMs) Work?

What a Large Language Model Really Is

An LLM is not a reference work or a search engine but a probability model of language. Well-known examples include GPT (OpenAI), Claude (Anthropic), Gemini (Google) and Llama (Meta). They all rest on the same principle: they compute which word fragments most likely follow a given piece of text.

A model is created in two phases. During pretraining it learns general language patterns from very large amounts of text. It is then shaped through fine-tuning and methods such as RLHF (reinforcement learning from human feedback) to answer helpfully, safely and appropriately to the conversation. The knowledge does not sit in a database but in billions of trained parameters.

Tokens: How Text Becomes Numbers

An LLM does not process letters or whole words directly but tokens: small units of text, often word fragments. A tokenizer splits the input into these building blocks and assigns each a number. Each number is then translated into a vector (an embedding) that the network can compute with.

• As a rough rule of thumb, one token corresponds to roughly four characters in English; in German it often differs because of long compound words.
• Rare words, numbers and code are split into several tokens, while common words are often a single token.
• Costs and limits of commercial models are almost always billed in tokens, not words.

Next-Token Prediction: The Core Mechanism

At its core an LLM does just one thing: for the text so far it computes a probability distribution over all possible next tokens and picks one. That token is appended, and the whole process repeats token by token. This is called autoregressive generation: the text is produced step by step, always based on what has already been written.

A parameter called temperature controls how far the model strays from the most likely token. Low values give predictable, fact-oriented answers; higher values encourage variety and creativity but raise the risk of errors. The model is trained by comparing its prediction with the actual next token across billions of text passages and adjusting its parameters slightly.

Transformers and Attention: The Architecture

Almost all of today's LLMs are based on the transformer, introduced in the 2017 paper Attention Is All You Need. Its decisive component is the attention mechanism, more precisely self-attention: for each token the model assesses how relevant every other token in the context is and weights them accordingly.

This lets the model capture context-dependent meaning: in the bank of the river, bank means something different from money in the bank. Many such attention layers stacked on top of each other, combined with the weights stored in billions of parameters, allow complex relationships to be modelled. Because attention computes in parallel, models can be trained efficiently on huge amounts of data.

The Context Window: Short-Term Memory

The context window is the maximum amount of tokens a model can consider at once. The system instruction, your input, uploaded documents and the answer so far must all fit inside it. Once the window is full, older text drops out and the model loses that information.

Importantly, an LLM has no persistent memory of its own beyond a single session. Whatever it should know in a new chat must be supplied again. Very long contexts also increase compute and cost, and relevant details in the middle of long inputs are sometimes attended to less well than those at the start or end.

Why LLMs Hallucinate

A hallucination is a fluently phrased but factually wrong or invented statement. It does not stem from a malfunction but from the core principle: the model generates plausible-sounding text, it does not retrieve verified facts. If a false continuation is linguistically likely, it can be phrased just as convincingly as a correct one.

• Knowledge gaps: the training data contained too little or contradictory material on a topic.
• Knowledge cutoff: events after the training date are unknown to the model.
• Ambiguous questions and the incentive to always provide an answer rather than flag uncertainty.
• Countermeasures: retrieval-augmented generation (RAG) with reliable sources, requesting citations, lower temperature and consistent human verification.

What This Means for Practical Use

Understanding how LLMs work means using them correctly: as powerful assistants for drafts, summaries and ideas, not as the final authority on facts. Important statements, figures, quotes and legal details should always be verified. A human remains responsible for the result.

For companies in Switzerland, data protection is added to this. Before entering personal data or trade secrets into an AI tool, check the revised Data Protection Act (revDSG) and where the data is processed; the FDPIC (EDOEB) is the competent supervisory authority. Clear internal policies and the choice of suitable services, ideally hosted in Switzerland or the EU, build trust.