Learning from a mountain of text
Before a language model can write anything, it has to be trained. The first and biggest stage is called pretraining. The model is fed a huge amount of text, such as public web pages, books, articles, and code. The exact mix varies by company, and they don't always say exactly what they used.
During pretraining, the model plays a guessing game over and over. It sees some text with the next piece hidden, guesses what comes next, and checks its guess against the real text. Each time it's wrong, its inner settings are nudged a tiny bit so the next guess is a little better. Repeat this an enormous number of times, and the model ends up with a strong sense of how language, and the ideas in it, tend to fit together.
After pretraining, the model is good at continuing text but not yet good at being an assistant. Ask it a question and it might just add more questions, because that's what often follows a question on a web page.
Making it helpful
The next stage is fine-tuning: more training on a smaller, carefully chosen set of examples. These often show a request followed by a good, helpful answer. This teaches the model the shape of a conversation.
Many companies also use human feedback. People compare different answers from the model and mark which ones are better, for example clearer, more accurate, or safer. The model is then adjusted to give more answers like the preferred ones. This is a big part of why chatbots sound so friendly and eager to help.
The knowledge cutoff
Training uses data up to a certain date, then stops. That date is the model's knowledge cutoff. The model has no built-in knowledge of anything after it. Ask a plain model about last week's news and it may say it doesn't know, or worse, guess. Some apps add a web search step to fill the gap, but the model itself stays frozen at its cutoff until a new version is trained.
A mirror of its data
A model can only learn what's in its data, so it reflects the patterns there. If a topic is written about a lot online, the model usually knows it well. If a topic is rare, or mostly covered in other languages, the model may be shaky on it.
The same goes for slants and blind spots. If the text it learned from tends to describe certain jobs, places, or groups of people in a lopsided way, the model can repeat that. Companies try to reduce this, but it's hard to remove fully.
Try it
Ask a chatbot, "What is your knowledge cutoff date?" Then ask about something you know happened recently. See whether it admits it doesn't know, searches the web, or guesses. Models aren't always right about their own cutoff, which is a small lesson in itself.