What Is a Large Language Model (LLM)?

#What a large language model is

A large language model (LLM) is a program that has learned, from a very large amount of text, which pieces of text tend to follow which others. You type a prompt, and the program produces its reply one small piece at a time. That is the whole mechanism at the top level. The rest of this article explains how such a simple job becomes useful, and where it breaks.

The word "large" refers to two things: the amount of text used to train the model, and the number of numeric values, called parameters, that training adjusts. The word "model" means the program assigns probabilities to sequences of text. It does not look up answers in a database, and it does not keep a list of facts that it checks before speaking. It generates text from patterns it absorbed during training.

If you have used an AI chat tool, you have used an LLM. The product wraps the model in an interface, adds instructions you never see, and often connects it to a search engine or other tools. The model underneath is the part this article describes. When the product's answer looks wrong, it helps to know which layer produced the mistake.

#Why it matters for security work

Security work runs on claims. A package is safe to install. A port is closed. A configuration disables a check. A vulnerability affects a particular version. An LLM writes every claim in the same fluent voice, whether the claim is true or not. If you do not know how the model produces text, it is easy to treat a confident paragraph as a finding.

Understanding the mechanism changes three habits. You stop expecting the model to know things it never saw. You stop reading fluency as evidence. And you start asking where each claim could be checked. The NIST Generative AI Profile, published as NIST AI 600-1, lists confabulation among the risks of generative AI systems. Confabulation is the production of confidently stated content that is wrong. The profile asks organizations to plan for it, so they do not assume it away.

#How it works, step by step

The model's job breaks into a few stages: text becomes tokens, tokens become vectors, attention mixes the context, the model predicts one token, and training and inference make that prediction useful. The sections below take them in order.

#Step 1: Text becomes tokens

The model does not read letters or whole words directly. It reads tokens, which are pieces of text chosen by a component called a tokenizer. A token can be a whole word, part of a word, a space, a punctuation mark, or a short byte sequence. Common words usually become one token. Rare words, code identifiers, and text in less common languages often split into several.

The tokenizer learns its pieces from data before the model is trained. One common method is byte-pair encoding, usually shortened to BPE. It starts with single characters and repeatedly merges the pair of neighbors that appears most often, until the vocabulary reaches a chosen size. The result is a fixed list of pieces and a rule for splitting any new text into them.

To make the idea concrete, the script below builds a tiny BPE from a few words. It is a toy, not the tokenizer of any real model, but the mechanism is the same. Save it as toy_bpe.py and run it with Python 3:

from collections import Counter

corpus = "low low low lower lowest newer newer newer wider wider new new"
words = Counter(tuple(w) + ("</w>",) for w in corpus.split())

def pair_counts(vocab):
    pairs = Counter()
    for word, freq in vocab.items():
        for a, b in zip(word, word[1:]):
            pairs[(a, b)] += freq
    return pairs

def merge(vocab, pair):
    out = Counter()
    for word, freq in vocab.items():
        new, i = [], 0
        while i < len(word):
            if i < len(word) - 1 and (word[i], word[i + 1]) == pair:
                new.append(word[i] + word[i + 1]); i += 2
            else:
                new.append(word[i]); i += 1
        out[tuple(new)] += freq
    return out

merges = []
vocab = words
for step in range(6):
    pairs = pair_counts(vocab)
    if not pairs:
        break
    best = max(pairs, key=lambda p: (pairs[p], p))
    merges.append(best)
    vocab = merge(vocab, best)
    print(f"merge {step + 1}: {best[0]!r} + {best[1]!r} -> {best[0] + best[1]!r} (count {pairs[best]})")

def tokenize(word):
    toks = list(word) + ["</w>"]
    for a, b in merges:
        i = 0
        while i < len(toks) - 1:
            if toks[i] == a and toks[i + 1] == b:
                toks[i:i + 2] = [a + b]
            else:
                i += 1
    return toks

for w in ["lowest", "newest", "widen"]:
    print(w, "->", tokenize(w))

The output below comes from running this script on Windows 11 with Python 3.12.10. Your output should match it line for line, because the script uses no randomness:

merge 1: 'r' + '</w>' -> 'r</w>' (count 6)
merge 2: 'e' + 'r</w>' -> 'er</w>' (count 6)
merge 3: 'w' + '</w>' -> 'w</w>' (count 5)
merge 4: 'n' + 'e' -> 'ne' (count 5)
merge 5: 'l' + 'o' -> 'lo' (count 5)
merge 6: 'w' + 'er</w>' -> 'wer</w>' (count 4)
lowest -> ['lo', 'w', 'e', 's', 't', '</w>']
newest -> ['ne', 'w', 'e', 's', 't', '</w>']
widen -> ['w', 'i', 'd', 'e', 'n', '</w>']

Read the merges first. The most frequent pair in the small corpus is a word-end marker next to r, so that merge comes first. Later merges build pieces such as er</w> and lo, which reflect repeated spellings in the corpus. Then look at the splits. "lowest" breaks into lo, w, e, s, t, and the end marker, because the toy vocabulary never learned a piece for est or owe. "widen" falls back almost to single characters. The tokenizer's vocabulary covers what it saw during its own training, and anything else costs more pieces.

Many real tokenizers work on bytes and train on far more text, so their pieces are longer and more numerous. The principle carries over. Text that appears often in the training data is cheap, and unfamiliar text costs more tokens. This has practical effects. Models limit input and output by token count, and services usually price requests by token count, so the same sentence can cost different amounts in different languages or in code.

#Step 2: Tokens become vectors

Each token has an integer identifier. The model maps that identifier to a vector, a list of numbers that is often hundreds or thousands of values long. These vectors are learned during training. Tokens that appear in similar contexts end up with similar vectors, so the model can treat "car" and "automobile" as related without anyone writing a rule that says so.

From this point on, the model works with numbers, not characters. It does not inspect the letters inside a token the way you would. That is one common explanation for why models sometimes miscount the letters in a word or get a simple spelling puzzle wrong. The letters were never presented to the model as separate items.

#Step 3: Attention and the transformer

Most current LLMs are built on the transformer architecture, described in "Attention Is All You Need" by Vaswani and colleagues, published in 2017 (arXiv:1706.03762). Earlier language models often read text one token after another, carrying a running summary forward. The transformer lets each position look at other positions directly through a mechanism called attention.

For each token in the context, attention computes how much weight to give every earlier token when it builds the representation of the current one. Stack many layers of attention and small feed-forward networks, and the model builds a rich picture of the context. It can link a pronoun to the noun it refers to, match a function call to a definition several lines earlier, or notice which part of a question the answer must address.

You do not need the mathematics to use these models well. The useful fact is that the model's reading of any token depends on the tokens around it, across the whole context. Change one word in a prompt and the representation of many other words can change with it. That is why small wording changes sometimes produce large changes in the reply.

#Step 4: Predicting the next token

At the top of the layers, the model produces one score for every token in its vocabulary, then converts the scores into probabilities. The next token is picked from that distribution, added to the text, and the process repeats with the longer text as input. A reply of 300 tokens is 300 separate predictions, each one conditioned on everything before it, including the tokens the model itself produced a moment earlier.

Take the phrase "The capital of France is". Nearly all of the probability goes to a token that continues with "Paris". The model did not consult a map. It learned that this continuation is very common in the text it saw. The same mechanism handles harder requests: a code comment followed by a function signature, an error message followed by a likely cause, a question followed by a plausible explanation. Each is a continuation, and the model's success depends on how well the patterns it learned fit the new text.

#Step 5: Training

Training adjusts the parameters so that the model's predictions match the text in its training data. In simplified form, the model reads a piece of text, hides the next token, predicts it, measures how far the prediction was from the real token, and adjusts the parameters slightly in the direction that would have helped. Repeat that over an enormous amount of text, using a large amount of computation, and the parameters come to encode much of the statistical structure of language.

Training usually happens in stages. The first stage, often called pre-training, is the next-token process above. Later stages commonly use examples of instructions and good answers, and sometimes feedback from people who rank replies. These stages shape how the model behaves when someone asks it a question: it becomes more willing to answer in a useful form, and less likely to continue the conversation as if it were a web page. The training data sets the range of topics the model can draw on, and its cutoff date sets how far its learning from text reaches.

Training is expensive, and the model's developer does it, not the person typing into a chat window. Ordinary chat sessions do not change the model's parameters.

#Step 6: Inference

Inference is the act of running a trained model to produce output. Every reply you see is inference. During inference the parameters stay fixed. The only thing that varies from one request to the next is the input, which includes the system instructions, the conversation so far, any attached files, and any tool results fed back into the context.

This explains a common surprise. A model can seem to forget a conversation the next day. Nothing you typed updated its parameters, so the model starts from the same state as before. Products that offer memory store notes outside the model and insert them into the context of later requests. The model reads the notes as text. It does not remember them the way a person does.

#Context window

The context window is the maximum number of tokens the model can take into account in one request. It must hold the instructions, the conversation, the documents you attached, and the reply being generated. Text outside the window is not visible to the model at that step, however important it was earlier. In long conversations, early messages can drop out, or they can lose influence to recent ones.

Window sizes differ between models and change over time. Check the current documentation for the model you use instead of trusting a number from an article, including this one. A bigger window also does not guarantee better use of it. Long contexts can bury the detail that matters, so a focused prompt with the right document often beats a long pile of text.

#Temperature and sampling

Sampling is the step that picks one token from the probability distribution. Temperature is a setting that controls how sharp that distribution is before the pick. At low temperature, the most probable tokens dominate, so replies are more repetitive and more stable from one run to the next. At high temperature, probability spreads across more tokens, so replies vary more and wander more often. Many chat interfaces do not expose this setting at all.

Temperature does not decide whether a reply is correct. A low setting can repeat a wrong claim just as confidently as a high one. A setting of zero also does not guarantee identical output in every deployment, because the underlying computation can differ across hardware and software versions. If you need repeatable output for a test, record the model version, the settings, and the input, and treat the output as a sample rather than a fixed truth.

#Why models sound confident and still get things wrong

A confident wrong answer follows from how the model was built, not from a rare glitch.

The model optimizes for plausible continuation. A false sentence that fits the pattern of the text can score as well as a true one. Training data also contains errors, and the model absorbs them as readily as it absorbs correct facts. At inference time it has no direct access to the world unless a tool supplies that access. Without a retrieval step or a tool, it answers from the patterns in its parameters.

Fluency is part of the problem. Confident, well-organized writing is common in the training text, so the model learns to produce it regardless of accuracy. The tone of a reply carries no information about whether its contents are true.

Gaps make things worse. When a detail is missing, the model fills it with something that fits the pattern. Specific details such as version numbers, function names, CVE identifiers, configuration keys, and URLs are the easiest to reconstruct wrongly, because they look regular but have to match a particular fact. NIST's name for this is confabulation, and the same pattern appears across almost every generative system.

#Common mistakes

1. Treating a reply as a search result. The model may paraphrase real sources, invent others, or mix two of them. Ask for the source, then open it.
2. Assuming the model ran the command or scan it describes. Unless a tool gave it real output, it is describing what such a run would usually look like.
3. Pasting secrets, credentials, or customer data into a prompt without checking how the service stores and uses inputs. Read the policy before you paste anything sensitive.
4. Using one answer as the final word on a version number, a default setting, a legal point, or a license term.
5. Assuming a longer context always helps. Extra text can dilute the part that matters.
6. Believing a citation is real because it has the right format. Authors, titles, and links can all be generated to look plausible.

#What this means for security work

Verify claims against primary sources. For a security question, that usually means the vendor's advisory, the relevant RFC or standard, the official documentation for the exact version you run, the CVE record, the NIST publication, or the code in your own repository. Use the model to find out what to check and how to test it, not as the final authority on what is true.

Keep a short record for every claim you rely on: the claim, the source, the date you checked it, and the version it applies to. This habit catches both model errors and your own errors. Treat any generated command as a draft to read before running. Run it only in a lab you own, and never point generated scanning or exploitation steps at a system you do not have written permission to test.

#How to go further

Read the first section and the architecture figure of the transformer paper. You will not need every equation to see the shape of the idea. Then run the toy tokenizer on your own words and watch where it splits them. The paper by Sennrich and colleagues introduced subword units built by byte-pair encoding, and the Hugging Face course chapter walks through the algorithm with code. Next, read the confabulation section of the NIST profile to see how the risk is framed for organizations.

The next article, What Is an LLM API, covers calling a model from your own code. That is how NotHoly exposes its models to developers, and the same verification habit applies to every response you receive through an API. For background on the kind of claim security work depends on, see What Is a Vulnerability.

#Sources

• Vaswani et al., "Attention Is All You Need": https://arxiv.org/abs/1706.03762
• NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative AI Profile: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf
• Sennrich, Haddow and Birch, "Neural Machine Translation of Rare Words with Subword Units": https://arxiv.org/abs/1508.07909
• Byte-Pair Encoding tokenization, Hugging Face LLM Course: https://huggingface.co/learn/llm-course/chapter6/5