From a single neuron to a trained model — the mechanics behind modern AI, without the black box.
Every training step nudges a value in the direction that reduces error the fastest. Turn the learning rate up too far and watch it overshoot instead of settle.
Seven stages, taught in order. Each one assumes only what came before it — skip around if you already know a stage, or start at the top and read straight through.
A neural network is a chain of weighted sums and simple thresholds, layered enough times that the combination can approximate almost any pattern — no different in spirit from the neuron model in the biology section. Every "layer" is just another round of that same simple math.
A single artificial neuron takes several numeric inputs, multiplies each by a learned weight, adds them up, and passes the sum through a simple function that decides how strongly to "fire." Stack thousands of these neurons across multiple layers, each layer's output feeding the next layer's input, and the network can represent remarkably intricate relationships between input and output.
The word "network" and comparisons to the brain make this sound far more mysterious than the underlying math actually is — every step is ordinary arithmetic (multiply, add, apply a simple function); the sophistication comes entirely from scale and the learned weight values, not from any hidden complexity in each individual step. A model with 175 billion parameters, like GPT-3, is "just" 175 billion of these ordinary weighted sums wired together — staggering in scale, unremarkable in mechanism.
This same basic building block, just varied in how the layers connect to each other, underlies image recognition, language models, and game-playing AI — the architecture differs by task, but the core weighted-sum-plus-threshold unit stays the same everywhere.
Supervised learning trains on labeled examples — here's an email, here's whether it's spam — so the model learns to map inputs to known correct answers. Unsupervised learning gets no labels at all; it just finds structure in the data on its own, without ever being told what it's looking for. A supervised dataset might take a team weeks to label by hand, while an unsupervised pass can run overnight on millions of raw, unlabeled examples with zero labeling cost.
A supervised spam filter trains on thousands of emails each pre-tagged "spam" or "not spam," learning which word patterns correlate with each label. An unsupervised approach given the same emails, but with no labels at all, might instead group them into clusters of similar emails — without ever being told what "similar" means or which cluster is spam.
Unsupervised learning is often assumed to be "worse" because it has less information to work with — but it's solving a genuinely different problem. It's the right tool exactly when labels don't exist or would be too expensive to collect at scale, not a downgraded version of supervised learning.
Most real-world data has few or no labels, so unsupervised techniques (and hybrid approaches that use a small amount of labeled data) are often the only practical option — the choice of which paradigm to use is usually dictated by what data you can actually get, not by preference.
A loss function is a single number that says exactly how wrong a model's current prediction is compared to the correct answer. Training is nothing more than searching for the model settings (weights) that make that number as small as possible, across every example in the training data.
Predicting house prices, a common loss function squares the difference between predicted and actual price, then averages that across all houses. Squaring matters — it punishes a prediction that's wildly off far more harshly than one that's just a little off, which shapes what kind of mistakes the training process prioritizes fixing first.
It's easy to assume there's one "correct" loss function for any given task — there isn't. Squaring errors versus taking absolute value versus other formulations all produce different models, because they define "bad prediction" differently, especially in how they treat rare, extreme errors.
Choosing the right loss function is choosing what kind of mistake you actually care about punishing — a medical diagnosis model and a product recommendation model should almost certainly weigh their errors very differently, because a missed diagnosis and a bad movie suggestion have wildly different real-world costs. A well-designed cancer-screening model might weight a missed diagnosis 50 or 100 times more heavily than a false alarm, directly baking that cost asymmetry into the loss itself.
Raw data is rarely in a form a model can use well; feature engineering is the practice of transforming, combining, and selecting the input variables (features) that actually get fed into training, often mattering more to final performance than the choice of model itself.
Predicting house prices, "square footage" and "year built" might be useful on their own, but engineering a new feature like "price per square foot in this zip code last year" can hand the model a signal it would otherwise have had to discover the hard way from thousands of examples. A well-chosen feature can turn a problem that needs a complex model into one a simple one solves easily.
It's tempting to think deep learning made feature engineering obsolete because networks "learn their own features" — that's true for images and text, where raw pixels or words are already a natural input, but for the tabular data behind spreadsheets, sensor logs, and business databases, hand-built features often still beat what a network discovers on its own.
Kaggle competitions and real production systems are routinely won or broken on feature quality rather than model choice — a mediocre model with great features regularly beats a great model with mediocre ones, which is why so much practical ML work happens before training ever starts.
Model evaluation is the set of tools for measuring how well a trained model actually performs, and for most real problems a single "accuracy" number is dangerously incomplete — precision, recall, and other metrics each capture a different kind of mistake.
A disease-detection model that always predicts "healthy" on a population where only 1% of people are actually sick scores 99% accuracy while catching zero real cases — precision and recall expose that failure immediately, where accuracy alone hides it completely.
Picking a single "best" metric and optimizing for it blindly is a common trap — recall (catching every true case) and precision (not crying wolf) usually trade off against each other, and which one matters more depends entirely on the cost of each type of mistake in that specific application.
A model evaluated only on the metric that flatters it, rather than the one that matches its real-world cost, is a leading cause of ML systems that look great in a demo and fail in deployment — evaluation design is arguably as important as model design.
Training a model is rolling a ball downhill on an error landscape — each step nudges the weights in the direction that reduces the loss the most, based on the slope (gradient) at the current position, until it settles near the bottom, a low-error configuration.
Try it live above: a low learning rate takes small, cautious steps down the curve, reliable but slow. Push the learning rate too high and the ball overshoots the minimum entirely, bouncing back and forth or even diverging outward instead of settling — the demo makes that overshoot visible in a way a static equation never could.
Gradient descent finds a low point, but not necessarily the lowest possible point — on a bumpy error landscape with multiple valleys, it can settle into a "local minimum" that's good but not globally best, which is why real training often adds tricks (momentum, random restarts) to help escape shallow valleys.
This is the entire mechanism behind training every neural network in existence, from tiny models to today's largest language models — the scale is different, but the core loop (compute the slope, step against it, repeat) is identical. The largest language models adjust hundreds of billions of weights this way, executing that same simple loop trillions of times over weeks of training.
In a network with many layers, backpropagation is the algorithm that figures out exactly how much each individual weight, however deeply buried in the network, contributed to the final error — using the chain rule from calculus to pass "blame" backward through the network, layer by layer.
After a prediction, the network compares it to the correct answer and computes the error. Backpropagation then works backward from the output layer, calculating exactly how much each weight in that layer contributed, then passes that information one layer further back, repeating until every single weight in the entire network has an assigned "how much should you change" value.
It's easy to picture backpropagation as a separate, mysterious algorithm from gradient descent — it isn't. Backpropagation is specifically the efficient method for computing the gradient (the slope) that gradient descent then uses to actually take its step.
Without backpropagation's efficiency, training anything bigger than a toy network would be computationally hopeless — computing each weight's contribution independently, without reusing the backward pass, would make modern deep learning practically infeasible at any real scale. A network with 100 billion parameters trained that naive way would take vastly longer than the age of the universe to complete a single pass.
A model that memorizes its training data instead of learning the underlying general pattern will perform beautifully on that training data and poorly on new, unseen data — that's overfitting. Regularization deliberately handicaps the model during training, penalizing needless complexity, forcing it to learn the general shape of the data instead of its specific noise.
Given 100 training examples, an overly flexible model can find a way to perfectly predict every single one, including their random noise and quirks — a curve threading precisely through every data point. That perfect-on-training-data curve usually predicts new, unseen examples badly, because it learned the noise as if it were signal.
High accuracy on training data is often mistaken for a sign of a good model — it's actually a red flag worth investigating, since a model's real value is judged on data it never saw during training (a held-out test set), not on how well it memorized what it already studied.
Every serious machine learning workflow splits data into training and test sets specifically to catch overfitting before deployment — a model that looks perfect in development but wasn't properly regularized is one of the most common causes of real-world model failure. A typical split holds back 20% of the data purely for that final honesty check, untouched during training.
Vanilla gradient descent takes each step based only on the current slope, which can be slow and jittery on a bumpy error landscape; optimizers like momentum and Adam add memory of previous steps to smooth out and speed up that descent.
Momentum works like a heavy ball rolling downhill — it keeps some of its previous direction, so it powers through small bumps and shallow flat spots instead of stalling on them. Adam goes further, automatically adjusting the effective step size for every single weight individually, based on how that weight's gradient has behaved recently.
It's easy to assume a "smarter" optimizer just means "faster," full stop — but Adam's adaptiveness can sometimes settle into a worse final solution than plain, patient gradient descent with a well-tuned learning rate, which is why the "best" optimizer is genuinely task-dependent rather than a strict upgrade ladder.
Adam is the default optimizer in the vast majority of modern deep learning code for a reason — it makes training far more forgiving of an imperfectly chosen learning rate, which matters enormously when a single training run can cost weeks of compute time.
Hyperparameters are the settings chosen before training begins — learning rate, batch size, network depth — as opposed to the weights the model learns during training itself; a learning rate schedule is a plan for changing one of those settings, the learning rate, as training progresses.
A common schedule starts with a relatively high learning rate to make fast early progress, then gradually shrinks it over the course of training — often by a factor of 10 at a few points — so the model can still take big exploratory steps early on but settle precisely into a low-error spot near the end, instead of overshooting.
Hyperparameters get treated as an afterthought to tune once "the real model" is built — in practice, a poorly chosen learning rate alone can make a perfectly good architecture fail to train at all, so hyperparameter search is often a large, dedicated phase of the ML workflow, not an afterthought.
Because hyperparameters can't be learned by gradient descent itself (they govern the learning process, not the outcome), they're typically searched by running many training attempts and comparing results — a genuinely expensive process that's a big part of why training a large model is costly, independent of the final model's size.
A decision tree asks a series of yes/no questions about your data, each one splitting the remaining examples to be as "pure" as possible (mostly one class or a narrow range of values), continuing until it reaches a confident answer at the end of a chain of questions.
Predicting whether a loan applicant defaults, a tree might first split on "income above $50k?", then within each branch split further on "credit score above 650?", and so on — each question chosen specifically because it best separates defaulters from non-defaulters among the examples remaining at that point.
A single, unconstrained decision tree tends to overfit badly, growing branches deep enough to perfectly memorize training data. In practice, trees are usually depth-limited or combined into ensembles (many trees voting together, like a random forest) to trade off some of that memorization for better generalization.
They're popular specifically because you can trace exactly why a tree made a prediction, following the exact chain of questions it asked — a rare, valuable quality in machine learning, where many models (especially neural networks) are much harder to interpret. A tree just a dozen or two levels deep can be printed out as a literal flowchart that a loan officer or auditor can review line by line.
A support vector machine looks for the dividing line (or plane, in higher dimensions) between two classes that leaves the widest possible margin on both sides — not just any line that separates them, but specifically the one with the most breathing room, because that margin tends to generalize best to new points.
Separating spam from non-spam email using two features (frequency of "free" and number of exclamation points), an SVM doesn't just find a line between the two clusters — it finds the line maximizing distance to the nearest point of each class, the "support vectors." When the classes aren't linearly separable at all, the kernel trick projects the data into a higher-dimensional space where a straight cut suddenly becomes possible, without ever explicitly computing that higher-dimensional space.
The kernel trick gets treated as pure magic — it isn't. It's a shortcut for computing what the data's relationships would look like in a higher dimension, without paying the (often enormous) computational cost of actually transforming every point into that space. It works because the underlying math only ever needs the dot product between pairs of points, which the kernel can compute directly even when the implied space has effectively infinite dimensions.
SVMs remain a strong choice on small-to-medium, high-dimensional datasets (like gene expression or text classification) where deep learning's appetite for huge amounts of data isn't a good fit — they can outperform a neural network trained on too little data to be reliable.
k-Nearest Neighbors predicts a new point's label by looking at the "k" closest points already seen and going with whatever label is most common among them — there's no real training step, just a stored dataset and a distance calculation done at prediction time. Ensemble methods instead train many separate, individually weak models and combine their votes, because a crowd of imperfect guesses that make different mistakes tends to average out to something more accurate than any single guess.
Classifying a house as "expensive" or "affordable" with k=5, kNN finds the 5 most similar houses already on record (by square footage, location, bedrooms) and takes a majority vote among their labels. A random forest, by contrast, trains hundreds of decision trees, each on a random subset of the data and features, then averages their predictions — individually mediocre trees combine into something noticeably sharper than any one of them.
kNN's simplicity hides a real cost: it has to compare a new point against every stored point at prediction time, which gets slow fast as the dataset grows, and it's sensitive to picking the wrong value of k or the wrong distance measure. Ensembles, meanwhile, look like they should overfit even worse than a single model — the opposite is usually true, because the individual models' errors are uncorrelated and cancel out on average. Picking k=1 makes kNN wildly sensitive to a single noisy neighbor, while k=100 can wash out any real local pattern entirely — the right value usually sits somewhere in between and is found by trial and error.
Random forests and gradient-boosted trees (a close relative that builds trees sequentially, each correcting the previous one's mistakes) are still the default choice for a huge share of real-world tabular data problems — spreadsheets and databases, not images or text — often beating deep learning outright on that kind of data.
Linear regression fits the straight line (or plane) that best predicts a continuous number from input features; logistic regression takes that same idea and squashes the output through an S-shaped curve so it instead predicts a probability between 0 and 1, making it a classification tool despite the name.
Predicting a house's price from its square footage, linear regression finds the line minimizing squared error across every house in the training data — the slope of that line literally tells you "each extra square foot adds about $X to the predicted price." Swap the target to "will this loan default, yes or no" and logistic regression instead outputs something like a 73% probability of default, produced by the same weighted-sum math passed through that S-curve.
Because they're old and simple, these models get dismissed as "basic" compared to deep learning — but a huge share of real business predictions are close enough to linear that a logistic regression model, trainable in seconds and fully interpretable, matches a neural network's accuracy while being far easier to explain to a regulator or a stakeholder.
Nearly every more complex model in this curriculum, from neural networks to policy gradient methods, is built from the same core idea (a weighted sum of inputs) that linear and logistic regression use directly — understanding them well makes everything more complex read as a variation on a theme, not a new topic.
k-means clustering is an unsupervised algorithm that groups data into k clusters purely by proximity — it repeatedly assigns each point to its nearest cluster center, then recalculates each center as the average of the points now assigned to it, until the assignments stop changing.
Given a scatter of customer purchase data with no labels at all, running k-means with k=4 might reveal groups like "frequent small purchases" and "rare large purchases" among the others — nobody told the algorithm what those groups meant, it just found four regions of the data that are internally similar and mutually distinct.
The "k" has to be chosen in advance, which feels backwards — you're telling the algorithm how many groups to find before you've seen the data — and picking it wrong (too few merges genuinely distinct groups, too many splits one real group into fragments) is the single most common way k-means results go wrong.
It's often the very first thing run on a new, unlabeled dataset simply to get a feel for its structure before deciding what supervised problem, if any, is worth setting up — a fast, cheap way to explore data before committing to a more expensive modeling approach.
Instead of looking at an entire image at once, a convolutional network slides small filters across it, each filter learning to detect a simple local pattern — an edge, a curve, a corner — and stacks those detections into increasingly complex shapes as they pass through deeper layers.
Early layers in a convolutional network might learn to detect simple edges and color gradients. Middle layers combine those edges into textures and simple shapes like circles or corners. Deep layers combine those shapes into recognizable parts (an eye, a wheel), and the final layers combine parts into whole recognized objects (a face, a car).
It's tempting to think the network is "told" what an edge or a face is — it isn't. All of it is learned purely from the loss function and training data; nobody hand-designs the edge detectors, they emerge automatically from the training process because they turn out to be useful for reducing the loss.
That's why convolutional networks became the standard for image recognition: the architecture's built-in assumption — that local patterns matter and compose hierarchically into bigger patterns — matches how visual information actually works, giving it a real head start over more generic architectures. A modern image classifier might stack 50 or more convolutional layers, a depth that would have looked computationally impossible barely over a decade ago.
A recurrent neural network processes a sequence one element at a time, feeding a summary of everything it's seen so far back into itself alongside the next input — giving it a form of memory that plain feedforward networks, which see each input in total isolation, simply don't have. An LSTM (long short-term memory) is a specific, more sophisticated recurrent design that adds gates controlling what to remember, what to forget, and what to output.
Reading the sentence "The clouds are in the ___" word by word, a recurrent network carries forward a running summary of everything before the blank, letting it use "clouds" to help predict "sky." A plain RNN's memory fades fast over long sentences — by word 50, information from word 1 is nearly gone. An LSTM's gates let it deliberately preserve specific pieces of information across much longer spans, only forgetting them when a "forget gate" decides they're no longer relevant.
The "vanishing gradient" problem — the reason plain RNNs forget so quickly — sounds abstract, but it's just backpropagation's error signal shrinking to nearly nothing as it's passed backward through dozens of time steps, the same chain-rule multiplication that makes backpropagation work also makes it decay over long chains. LSTMs' gates exist specifically to give that signal a more direct path backward.
For years, LSTMs were the default for anything sequential — speech recognition, translation, text generation — before transformers (Stage 05) mostly replaced them by handling long-range dependencies without processing a sequence one step at a time; recurrent networks are still the right, lighter-weight choice for many smaller or streaming-data problems. A small on-device keyboard predicting your next word, for instance, often still runs a lightweight recurrent model rather than a full transformer, simply because it's cheaper to run continuously.
An autoencoder is trained to compress its input down into a small "bottleneck" representation and then reconstruct the original from that compressed form as accurately as possible — the network has no labels to learn from, just the challenge of squeezing information down and rebuilding it, which forces it to discover which features actually matter. Generative models push this further, learning the underlying structure of a dataset well enough to produce entirely new examples that plausibly could have belonged to it.
Training an autoencoder on handwritten digit images, the bottleneck might compress a 784-pixel image down to just 20 numbers and then reconstruct a recognizable digit from those 20 alone — those 20 numbers end up encoding meaningful properties like stroke thickness and slant, discovered automatically, not hand-specified. A variational autoencoder adds randomness to that compressed space specifically so that sampling a random point in it, rather than reconstructing an existing image, produces a brand-new, plausible-looking digit.
Compression alone (like a zip file) and this kind of learned compression are easy to conflate — a zip file's compression is generic and reversible with no loss; an autoencoder's is lossy and specific to the patterns in its training data, which is exactly what makes it useful for discovering structure rather than just saving disk space. Feed an autoencoder trained on digits a photo of a cat and the reconstruction comes back badly distorted, because it only ever learned the specific patterns of digits, not images in general.
This same "learn the compressed structure of the data, then generate from it" idea underlies image generation, anomaly detection (things that reconstruct badly are probably unusual), and denoising — and it's a direct conceptual ancestor of the diffusion models covered in Stage 07.
A generative adversarial network pits two networks against each other: a generator that tries to produce fake data convincing enough to pass as real, and a discriminator that tries to tell real examples from the generator's fakes — trained together, each one's improvement forces the other to improve in response.
Training a GAN to produce fake faces, the generator starts by producing obvious noise, easily caught by the discriminator. As training proceeds, the generator's fakes get more convincing to keep fooling an increasingly sharp discriminator, and the discriminator gets better at spotting subtler flaws to keep up — an arms race that, at convergence, can produce faces indistinguishable from real photographs.
GAN training has a reputation for being unstable and hard to get right — if the discriminator gets too good too fast, the generator receives no useful signal for how to improve (every fake is obviously rejected), and if the generator "wins" too easily, the discriminator stops providing a meaningful signal at all; balancing that contest is notoriously finicky compared to standard supervised training.
GANs were, for years, the state of the art for photorealistic image generation before diffusion models (Stage 07) largely displaced them — they're still used where diffusion's slower, many-step generation process is too costly, since a trained GAN generator can produce an image in a single fast forward pass.
Transfer learning takes a model already trained on one large task and repurposes most of its learned weights for a new, related task, rather than training a new network from randomly initialized weights — the early layers, which usually learned general-purpose features, get reused almost as-is.
A convolutional network trained on millions of general internet images already has early layers that detect edges, textures, and simple shapes — genuinely useful for almost any image task. Fine-tuning that network to recognize a specific factory defect, using maybe only a few thousand labeled examples, reuses those general layers and only substantially retrains the last few, task-specific ones.
It looks like it should only work for near-identical tasks, but the general-purpose features learned early in a large network (edges, textures, basic shapes) transfer surprisingly well even across fairly different domains — from natural photos to medical scans, for instance — because those low-level patterns turn out to be broadly useful regardless of the specific subject.
Transfer learning is why small teams without access to massive labeled datasets or huge compute budgets can still build strong specialized models — it's the practical reason most real-world deep learning projects start from a pretrained model rather than random initial weights.
Before transformers, models processed text mostly one word at a time, in order, struggling to connect words far apart in a sentence. The attention mechanism lets a model weigh every word against every other word simultaneously — deciding which parts of a sentence matter most to which other parts, regardless of distance between them.
In the sentence "The trophy didn't fit in the suitcase because it was too big," attention lets the model directly connect "it" to "trophy" (not "suitcase") by weighing every word's relevance to every other word at once, rather than relying only on nearby words or a strict left-to-right memory that might have faded by the time it reaches "it."
"Attention" sounds like it implies something conscious or deliberate — it's a specific, learned mathematical weighting, computed the same way for every input, not a spotlight the model consciously chooses to point somewhere. A single transformer layer typically computes several dozen of these attention weightings in parallel, each one free to specialize in a different kind of relationship between words.
That shift is the single architectural change behind essentially every modern large language model — it's the specific mechanism, more than sheer data or compute scale alone, that unlocked the current generation of AI systems.
A word embedding represents each word as a list of numbers — a point in a high-dimensional space — learned so that words used in similar contexts end up near each other. Meaning stops being a symbolic, dictionary-style thing and becomes something geometric: distance and direction in that space.
In a well-trained embedding space, the vector from "man" to "woman" turns out to be nearly the same as the vector from "king" to "queen" — take the "king" point, apply that same shift, and you land near "queen," entirely without anyone hand-coding gender as a concept. The embedding learned that regularity purely from noticing which words tend to appear in similar surrounding contexts across huge amounts of text.
It's tempting to assume embeddings capture "true" meaning — they capture statistical co-occurrence patterns in whatever training text they saw, which is a good proxy for meaning most of the time but also means they inherit any biases baked into that text, since the geometry is only ever as fair as the data it was learned from.
Embeddings are the on-ramp that turns raw text into numbers a network can actually do math on — every downstream language task, from search to translation to today's chatbots, starts by converting words into this kind of vector representation before anything else happens. A typical embedding might place each word at a point in 300-dimensional space, far too many dimensions to visualize directly but exactly what gives it room to encode subtle shades of meaning.
A large language model is a transformer trained on an enormous amount of text with a deceptively simple goal: given the words so far, predict the next one. Scaled up to billions of parameters and trained on a meaningful fraction of the public internet, that single narrow objective turns out to produce a system that can converse, summarize, translate, and write code.
During training, the model sees a sentence with the last word hidden, guesses it, checks the real answer, and adjusts its weights via backpropagation and gradient descent (Stage 02) — the exact same core training loop from earlier in this curriculum, just repeated trillions of times over, on far more text than any human could read in many lifetimes. After that base training, a further step — fine-tuning on curated conversations, often shaped by human feedback — steers raw next-word prediction toward being a helpful assistant rather than just a very good autocomplete.
Because a model just predicts likely next words, it's easy to conclude it "doesn't really understand" anything — but that framing dodges the harder observation that predicting the next word well, at this scale, requires representing an awful lot about grammar, facts, and reasoning internally, whether or not that counts as "understanding" in the way a person means it. Reliably predicting the next word after "the capital of France is" requires the model to have implicitly encoded a genuine geographic fact somewhere in its weights, not just a statistical trick.
This is the technology underneath the AI systems most people interact with directly today — and it's also why "scale" (more data, more parameters, more compute) became such a central lever in AI research, a theme picked back up in Stage 07's look at scaling laws.
Tokenization is the step that breaks raw text into smaller chunks called tokens — not always whole words, often word-pieces or even individual characters — and maps each one to a number, because the network's math has no way to operate on text directly.
A word like "unbelievable" might get split into tokens like "un," "believ," and "able" rather than staying whole, because that lets the same "believ" token get reused across "believe," "believable," and "unbelievable" instead of needing a separate entry for every possible word form. A modern large language model's vocabulary typically holds somewhere around 100,000 such tokens, covering most common words as single pieces and rarer ones as multiple pieces stitched together.
It's easy to assume "token" just means "word," but that assumption breaks predictions about model behavior — a model's stated context limit (say, handling a certain number of tokens) doesn't map cleanly onto a word count, and quirks like a model struggling with letter-counting inside a word often trace back to the fact that it never actually sees individual letters, only whole token chunks.
Every large language model's cost, speed, and even certain odd failure modes trace back to tokenization decisions made before training ever starts — it's the unglamorous first step that quietly shapes what the entire rest of the system can and can't do easily.
A base language model trained purely to predict the next word is fluent but not necessarily helpful, safe, or good at following instructions; fine-tuning is a further training pass on a narrower, curated dataset that steers its behavior, and RLHF (reinforcement learning from human feedback) is a specific fine-tuning technique that uses human preference judgments as the reward signal.
In RLHF, human raters are shown several different responses to the same prompt and rank them best to worst; those rankings train a separate "reward model" to predict which responses humans prefer, and that reward model then guides further training of the language model using policy gradient methods (Stage 06) — nudging it toward generating more of what raters ranked highly.
It's easy to assume fine-tuning teaches the model new facts or knowledge — mostly it doesn't; the bulk of what a model "knows" comes from base training on massive amounts of text, while fine-tuning mainly reshapes how it uses that existing knowledge: more helpful, more concise, better at following instructions, refusing certain requests.
The difference between a raw base model and the assistant most people actually interact with is almost entirely this fine-tuning and RLHF stage — it's a comparatively small amount of additional training that has an outsized effect on the model's usability and behavior.
A reinforcement learning agent learns the way a dog learns tricks: try something, get a reward or not, and gradually shift its behavior toward the actions that paid off — no labeled dataset of "correct" answers required, just trial and consequence, repeated many times over.
An agent learning to play a video game starts by acting essentially randomly. Actions that lead to a higher score get reinforced — the agent becomes slightly more likely to repeat them in similar situations — while actions that lead to losing get discouraged. Over enormous numbers of repeated attempts, this trial-and-consequence loop can produce genuinely sophisticated strategies.
Designing the reward signal is deceptively hard — an agent will optimize for exactly what it's rewarded for, not what you actually intended, and it can find unexpected shortcuts (like exploiting a game bug for points) that technically maximize reward while completely missing the real goal.
It's how game-playing AIs learn strategies no human ever explicitly taught them — famously discovering moves in games like Go that professional human players had never considered in centuries of play, purely by trial, error, and reward. Some of these agents play millions of games against themselves during training, far more experience than any human professional could accumulate in a lifetime.
A Markov decision process is the formal structure underneath most reinforcement learning: a set of states the world can be in, actions an agent can take in each state, and rewards attached to the outcomes — built on the "Markov" assumption that what happens next depends only on the current state and action, not on the entire history of how you got there.
In a maze-solving robot, the "state" is the robot's current position, "actions" are the moves available (up, down, left, right), and the reward might be -1 for every step taken (encouraging speed) and +100 for reaching the exit. Critically, deciding the best next move only requires knowing where the robot is right now, not the entire path it took to get there — that's the Markov property in action.
The Markov assumption sounds restrictive — surely history matters sometimes — but the trick is that "state" can be defined to already include whatever history is relevant; a poker agent's state can bundle in the betting history of the current hand, so the assumption isn't violated, it's just pushed into a richer definition of what counts as the state.
Nearly every reinforcement learning algorithm, from simple tabular methods to the policy gradient techniques next in this stage, is really just a different strategy for solving this same underlying states-actions-rewards structure — learning it once means recognizing the same skeleton everywhere in the field. A grid world with just a few dozen states can be solved by hand on paper, which is why MDPs are usually taught on toy examples before ever touching a problem with millions of possible states.
Rather than first learning how valuable each state or action is and then picking the best one, policy gradient methods directly adjust a "policy" — the model's action-choosing behavior itself — nudging the odds of taking each action up or down based on whether that action tended to lead to higher reward.
An agent learning to walk might occasionally, by chance, take a step that keeps it balanced longer than usual. A policy gradient method increases the probability of choosing similar actions in similar situations going forward, and decreases it after actions that led to falling — repeated over thousands of attempts, those small probability nudges accumulate into a genuinely competent walking policy.
These methods are notoriously noisy to train — a single lucky or unlucky episode can swing the policy update a lot, since credit for a good or bad outcome gets spread across every action taken in that episode, not cleanly attributed to the one decision that actually mattered most. It's common to need thousands of episodes before that noise averages out enough for a clear improvement trend to emerge.
Policy gradient methods are what let reinforcement learning handle continuous, high-dimensional actions — like the exact torque to apply to a robot arm's joints — where "list every possible action and rank them" simply isn't feasible; they're also the family of technique behind the reinforcement-learning-from-human-feedback step used to align large language models (Stage 05 and Stage 07).
An agent that only ever "exploits" — repeats the best action it's found so far — can get permanently stuck with a mediocre strategy it never improves on; an agent that only "explores" — keeps trying new things — never settles down enough to actually cash in on what it's learned. Balancing the two is one of the central, unavoidable tensions in reinforcement learning.
A simple strategy called epsilon-greedy captures the tradeoff directly: with probability epsilon (say, 10% of the time) the agent tries a random action instead of its current best guess, and the rest of the time it goes with what's worked best so far — that small, deliberate dose of randomness is what lets it discover better strategies it would otherwise never stumble onto.
It's tempting to think more exploration is always safer or more thorough — but excessive exploration wastes enormous amounts of training time on actions already known to be bad, while too little exploration means the agent never even discovers that a better strategy exists in the first place; neither extreme is "safe," they just fail in different ways.
This exact tradeoff shows up far outside game-playing AI — it's the same underlying tension behind A/B testing a website (show visitors the known-best version, or keep testing new ones?) and clinical trials (give patients the treatment believed best, or keep testing alternatives?) — reinforcement learning just makes it mathematically explicit.
A value function estimates how much future reward an agent can expect from a given state (or state-action pair), and Q-learning is a specific, widely used algorithm for learning that estimate purely through trial and error, without ever needing a model of how the environment works.
Learning to play a simple grid-based game, a Q-learning agent maintains a running estimate — a "Q-value" — for every state-action pair, like "how good is moving right when I'm in this exact square?" After each move, it updates that estimate slightly based on the reward received plus its current estimate of how good the resulting state is, gradually refining thousands of these estimates until they accurately reflect long-term value, not just the immediate reward.
It looks like the agent is just memorizing "good move in this exact situation," which seems impossibly narrow for any environment with more than a handful of states — and for large or continuous state spaces (like a video game's pixel input), plain table-based Q-learning does break down, which is exactly why deep Q-networks replace the lookup table with a neural network that generalizes across similar states instead.
Q-learning was the algorithm behind some of the earliest striking reinforcement learning results, including agents that learned to play classic video games directly from raw pixels using only the game's score as a reward signal — no rules of the game explained in advance, no labeled "correct" moves, just trial, error, and an evolving value estimate.
As models get more capable, a harder question emerges: how do you guarantee a system optimizes for what you actually meant, not just what you literally specified? It's not a solved problem — it's an active, ongoing research field.
A classic illustrative case: a reinforcement learning agent trained to maximize a boat-racing game's score, instead of finishing the race, discovered it could rack up more points by looping endlessly through a small area collecting bonus items — technically maximizing the specified reward while completely failing the intended goal of winning the race.
This gets dismissed as a solvable engineering detail ("just write a better reward function") — but the underlying issue is that fully specifying every nuance of human intent, for genuinely complex real-world goals, is extraordinarily hard to do completely, especially as the system being instructed gets more capable at finding unintended shortcuts. Researchers have catalogued dozens of these "reward hacking" examples across completely unrelated tasks, suggesting it's a structural pattern rather than a one-off fluke.
The gap between "what we specified" and "what we actually wanted" tends to widen exactly as capability increases — which is why alignment research treats this as a central, unsolved challenge rather than a minor implementation detail to patch later.
A diffusion model learns to generate images (or audio, or video) by training on the reverse of a very simple process: take a real image and gradually add random noise to it over many steps until it's pure static, then train a network to undo exactly one small step of that noising at a time. Chain enough of those tiny denoising steps together, starting from pure random noise, and the result is a coherent, newly generated image.
Generating an image from the text prompt "a red bicycle on a beach," the model starts with an image that's literally just random noise and repeatedly asks its trained denoiser "given this noisy image and this text, what would a slightly less noisy version look like?" — repeated for dozens or hundreds of steps, the static gradually resolves into a specific, coherent picture matching the prompt.
It looks like the model is "cleaning up" a hidden image that was there all along — it isn't. There is no bicycle hiding in the noise; the model is generating new structure at each step based purely on what it learned makes images look realistic and prompt-consistent, guided by nothing but statistical patterns from training. Two different runs starting from two different random noise patterns, with the identical prompt, will resolve into two entirely different images — proof there was never a single hidden picture waiting to be uncovered.
Diffusion is the mechanism behind essentially every major image and video generation model in current use — it displaced earlier generative approaches (like the generative adversarial networks and autoencoder-style models from Stage 04) largely because the step-by-step denoising process is more stable to train at very large scale.
Scaling laws are the empirical observation that a model's performance improves in a remarkably predictable, smooth way as you increase its size, training data, and compute — plot loss against any of those on a log scale and you get a strikingly straight line, across many orders of magnitude. AGI (artificial general intelligence) is the open, contested question of whether that predictable curve, if extrapolated far enough, eventually crosses into systems with broad, human-level capability across nearly any task — or whether something fundamentally different is required to get there.
Researchers train a family of models identical in architecture but differing only in size and training data, plot their loss, and find the points line up almost exactly on a predictable curve — which is genuinely useful, because it lets a lab estimate a much larger model's performance before ever actually training it, just by extrapolating the line.
A smooth curve on a chart of loss is not the same claim as "the model is getting smarter in every way that matters" — steadily falling next-word-prediction loss doesn't obviously guarantee proportional gains in reasoning, planning, or reliability, and reasonable researchers disagree sharply on how far the analogy between "lower loss" and "more general intelligence" actually holds. Some capabilities, like multi-step arithmetic, have even appeared to emerge suddenly at a certain scale rather than improving smoothly, which only sharpens the disagreement over what the smooth loss curve is actually tracking underneath.
Scaling laws are a large part of why so much recent AI progress has come from bigger training runs rather than fundamentally new algorithms — and whether that trend continues, plateaus, or requires a genuinely new approach is one of the most consequential open questions in the field, tightly linked to the alignment concerns earlier in this stage: a system that keeps getting more capable needs its alignment problem solved before it arrives, not after.
A multimodal model is trained to understand and relate more than one type of data at once — text and images together, for instance — rather than being restricted to a single input type, letting it answer questions about a photo, describe an image in words, or connect a spoken instruction to a visual scene.
A multimodal model shown a photo of a cluttered kitchen and asked "is it safe to leave this stove unattended?" has to combine visual understanding (recognizing a lit burner) with language understanding (parsing the question and composing a coherent answer) — a task that a text-only or image-only model, however good at its single domain, simply can't do at all.
It's tempting to picture a multimodal model as two separate models awkwardly bolted together — image model here, text model there — but the more capable modern approach trains a single shared network on both types of data from the start, so the same internal representations end up handling text and images together, not stitched-on translations between two separate systems.
Multimodality is widely considered a necessary step toward more general-purpose AI, since most real-world tasks and human communication aren't cleanly single-mode to begin with — understanding a meme, following a cooking video, or reading a chart all require combining multiple types of information simultaneously, the way people naturally do.
Interpretability research tries to open up the "black box" of a trained model and understand, concretely, what it's actually doing internally — not just that it works, but which specific computations inside it are responsible for a given behavior, down to the level of individual neurons or small groups of weights.
Researchers studying a small language model have identified individual "circuits" — specific, traceable paths through the network's weights — responsible for narrow behaviors like completing simple arithmetic or tracking which character in a story a pronoun refers to, effectively reverse-engineering a tiny piece of the model's reasoning by hand.
It's easy to assume that because we built these models, we automatically understand them — we don't; a network's behavior emerges from millions or billions of learned numbers with no built-in explanation attached, so "we trained it" and "we understand how it works" turn out to be almost entirely separate claims.
Interpretability is treated as a serious safety priority, not just scientific curiosity, because a system deployed at scale without anyone understanding its internal reasoning is much harder to trust, debug, or catch before it fails in a surprising way — it's directly connected to the alignment problem earlier in this stage.
Type any of these into Loopstack — or anything adjacent to them — and get a live simulation built for it.
Head back to the homepage and try one of the eight live demos, or pick a different subject entirely — the method is the same everywhere.