Context Window vs Memory vs RAG: How AI Models Actually Remember Things

The bottom line

Three terms get mixed up constantly when people talk about what an AI model “knows” or “remembers”:

• Context window: the maximum amount of text (measured in tokens) a model can take in and produce in a single request. It is the model’s working space for that one conversation turn.
• Memory: a product feature that saves information between conversations and adds it back into future prompts.
• RAG (retrieval-augmented generation): a technique that searches an external collection of documents and inserts the most relevant pieces into the prompt before the model answers. [1]

All three end up in the same place—text inside the context window—but they get there in different ways and suit different jobs. Knowing which one you are dealing with explains most “why did the AI forget that?” moments.

The context window: the model’s desk

Think of the context window as a desk. Everything the model can use right now has to fit on it: your instructions, the conversation so far, any files you attached, and the answer it is writing. When the desk is full, something has to come off—usually the oldest part of the conversation.

Context windows have grown fast. In early 2024 Google announced a Gemini model that could process up to a million tokens in one go, and long windows are now common across leading models. [5] Mistral’s Small 4, for example, lists a 256,000-token window. (See our explainer on Mistral Small 4’s specifications.)

A rough guide: one token is about three-quarters of an English word, so 100,000 tokens is in the region of a long novel. But a bigger desk doesn’t mean the model reads everything on it equally well.

Lost in the middle

Researchers from Stanford and elsewhere tested how well models use information spread through long inputs. They found performance was best when the relevant fact was at the beginning or end of the context, and dropped noticeably when it sat in the middle—even for models built for long contexts. [2] Later benchmarks such as RULER found that many models’ effective context—the length at which they still perform well on harder tasks than simple look-up—was considerably shorter than the advertised maximum. [3]

What this means in practice: pasting a 300-page report into a prompt and asking a question will often work, but not reliably. If accuracy matters, put the key material near the start or end, ask the model to quote the passage it relied on, and check it.

Long contexts also cost more. Most providers charge per token, and longer inputs take longer to process.

Memory: what the app saves for you

When a chat assistant “remembers” that you are vegetarian or that you work in Pune, the model hasn’t learned anything. The application has stored that fact and quietly adds it to later prompts. OpenAI, for example, introduced memory in ChatGPT as a feature users can view, edit, delete or turn off. [4]

Key points:

• Memory is usually a short list of facts, not a full record of past chats.
• It is controlled by the app, so check the settings: you can typically see what is stored and delete it.
• It shares the same desk: remembered facts take up space in the context window.
• The underlying model’s knowledge comes from training. It doesn’t change because of your conversations unless the provider separately uses chat data for training, which is a different setting to check.

RAG: give the model a library card

Retrieval-augmented generation was introduced in a 2020 research paper that combined a language model with a searchable index of documents. [1] The idea is now the standard way to make AI answer questions about information it wasn’t trained on—company policies, product manuals, legal documents or last week’s news.

How it works:

1. Documents are split into chunks and indexed, usually as embeddings (numerical representations of meaning) in a vector database, often alongside a keyword index.
2. When a question arrives, the system retrieves the most relevant chunks.
3. Those chunks are inserted into the prompt, and the model generates an answer based on them—ideally citing which chunk it used.

RAG’s strengths are that it can cover far more material than any context window, stays current when documents change, and makes answers traceable to a source. Its weak point is retrieval: if the search step fetches the wrong passages, the model answers from the wrong material, often confidently.

Which one do you need?

Situation	Best fit	Why
Ask questions about one long document today	Long context window	Simple; no setup
An assistant that keeps your preferences across chats	Memory	Small set of stable facts
Answer questions over thousands of documents	RAG	Scales beyond any window; easy to update
Answers must cite an exact source	RAG (with citations)	Each answer links to retrieved passages
Agent working through a multi-step task	Context window + task notes	Keeps track of what it has already done

Many products combine all three: memory for preferences, RAG for documents, and a long context window to hold the result.

FAQ

Does a bigger context window mean a smarter model?

No. It means the model can read more at once. How well it reasons about that text depends on the model, and studies show accuracy can drop for information buried in the middle of long inputs. [2][3]

Is RAG still needed now that context windows are huge?

Usually yes for large or changing document collections. Re-sending millions of words with every question is slow and expensive, and retrieval makes answers easier to trace back to a source.

Does ChatGPT remember everything I say?

Not by default in the model itself. The app can save specific facts if memory is enabled, and keeps chat history you can delete. Check the memory and data-control settings in your account. [4]

What is a token?

A chunk of text the model processes—often part of a word. In English, one token is roughly four characters or three-quarters of a word, though it varies by language. Indian languages often use more tokens per word.

Why does the AI forget what I said earlier in a long chat?

The conversation may have exceeded the context window, so early messages were dropped or summarised. Or the detail is still there but buried in the middle of a long context, where models are less reliable. Restating key facts helps.

Sources

1. Lewis, P. et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS 2020, arXiv:2005.11401. https://arxiv.org/abs/2005.11401
2. Liu, N. F. et al., Lost in the Middle: How Language Models Use Long Contexts, Transactions of the ACL, 2024, arXiv:2307.03172. https://arxiv.org/abs/2307.03172
3. Hsieh, C.-P. et al., RULER: What’s the Real Context Size of Your Long-Context Language Models?, arXiv:2404.06654, 2024. https://arxiv.org/abs/2404.06654
4. OpenAI, Memory and new controls for ChatGPT, 13 February 2024. https://openai.com/index/memory-and-new-controls-for-chatgpt/
5. Google, Our next-generation model: Gemini 1.5, 15 February 2024. https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/

Methodology note

This explainer was researched and drafted with AI assistance and checked against the sources listed above. Product features change often; check your provider’s current documentation. See our editorial policy.