Have you ever wondered how a large language model actually reads, processes and understands your words? The answer is a journey — from tiny fragments of text to a rich, semantic frame of reference we call context. Follow that journey and a lot of AI behaviour stops being mysterious.

1. The atoms of language: tokens

The fundamental truth of AI language processing is that the model does not see words — it sees tokens. Tokens are the atoms of AI language: small pieces of text that can be whole words, word fragments, numbers or symbols. The word “unbelievable,” for example, is often split into un | believe | able. Each token is then converted into a high-dimensional vector called an embedding, and the position of that vector in space encodes meaning — ideas that are related sit close together, which is how the machine can “feel” the relationship between concepts.

2. The core mechanism: next-token prediction

The engine behind every modern LLM is a simple loop: generate, predict, repeat. The model is constantly predicting the next most plausible token. In a vacuum that prediction is deeply ambiguous — begin a sentence with “I ___” and “am,” “was” and “want” are all equally plausible. This is where context steps in, because meaning does not exist in isolation.

3. Context: the frame that gives meaning

Meaning comes from tokens + context. Context acts as a lens that resolves abstract tokens into specific realities — a process worth calling dynamic disambiguation.

  • Resolving ambiguity. On its own, “The bank ___” leans toward financial words like “approved” or “loan.” Add the surrounding context — “The river overflowed its boundaries. The bank ___” — and the representation of “bank” instantly shifts from financial to geographical, favouring words like “muddy” or “collapsed.”
  • Collapsing possibility. An ambiguous word like “it” resolves to different nouns depending on the descriptive context around it.
  • Shifting perception. Context even colours judgement: a pen priced at ₹50 nudges the model toward “cheap,” while ₹5000 shifts the completion to “expensive” or “premium.”

4. The breakthrough of attention

Earlier approaches struggled with weak ambiguity handling, poor long-range context and tight memory limits. The breakthrough was the Transformer architecture, which offers full visibility: it processes all the tokens at once, in parallel, and lets every token attend to every other token. Attention is the model’s focus mechanism — it identifies the relevant words, connects dependencies, and dynamically decides what matters in order to resolve ambiguity.

5. Intelligence is bounded: the context window

Full visibility is powerful, but it comes with caveats. More context is not always better context — too much can become conflicting and confusing (asking for a pen that is somehow both “premium” and “cheap”). And the model works inside a context window with hard limits; once that window overflows, the earliest tokens simply fall away. So an LLM is really three things working in harmony: prediction of the next token, attention to focus on what matters, and context as the frame that gives everything meaning.

6. The new discipline: context engineering

Because context drives output quality, shaping it has become a discipline in its own right. A generic prompt like “write a plan” yields shallow, unusable output; a specific one — “write a SaaS business plan for a vendor-management platform in India under ₹10L” — delivers something specific and actionable. Context engineering comes down to four moves:

  • Designing inputs — shape the prompt with intent and precision.
  • Structuring constraints — set the boundaries so the model can focus.
  • Providing examples — anchor expectations with concrete patterns.
  • Managing memory — curate what stays inside the context window.

Why this mental model pays off

Once you can picture tokens, embeddings, attention, the context window and context engineering, most AI behaviour becomes explainable. Hallucinations usually trace back to missing or noisy context; “the model forgot what I said” is almost always a context-window limit; and better prompts are, more often than not, just better context. At Orthonity we put this understanding to work — engineering the context and structure around models so the AI systems we build are accurate, grounded and dependable.