Introduction

1. Continual Learning
2. Sensorimotor Grounding
3. Sample Efficiency
4. Alignment
5. Resource Budget

The Continual Learning Problem

In a connectionist model with distributed representations […] each connection weight is involved in responding to many different inputs. Thus, adjustment of weights to encode the desired response to a new input pattern will necessarily alter the network's response to other inputs as well (147).

Why care?

1. In the context of LLMs, you'll find they are most useful when you direct them very accurately toward the right problem (prompt engineering). But distilling our preferences, style, and constraints into text is burdensome and difficult to maintain. Perhaps more importantly, text is a lossy medium to transmit feedback, hence why lived experience (not books) is the ultimate teacher for humans. Enabling an LLM to continuously adapt to our feedback, much like a coworker or intern would, transforms these tools from approximate aids to systems we can trust.
2. For other tool-based AI applications, consider deploying an intelligent Mars rover. Imagine NASA sends a rover to Mars with an autonomous driving system that performs perfectly on simulated terrain. As the rover drives around, it encounters something entirely out-of-distribution (perhaps a Martian threatening it?). A human might reason through the situation, simulate possibilities, and adapt. The rover has no such ability. We need to teach this rover (and any other task-focused AI system) to continuously learn to avoid being stuck again.

Summary

Notation

Cheat sheet

• $\theta$ — model parameters at any given time
• $\theta_k^*$ — parameters after training on task $T_k$
• $f_\theta(\cdots)$ — model parameterized by $\theta$
• $T_k$ — the $k$-th task
• $\mathcal{D}_k$ — dataset for task $T_k$
• $\mathcal{L}_k(\theta)$ — loss on task $T_k$
• $R(\theta, \theta_A^*)$ — regularization term measuring deviation from prior solution
• $\lambda$ — stability-plasticity tradeoff coefficient
• $N_k$ — number of samples in dataset $\mathcal{D}_k$
• $D$ — total number of parameters
• $C$ — number of classes

Palliatives for Catastrophic Interference

1. Self-distillation (regularization)
2. Replay buffers (data-centric)
3. Extending context length (infrastructure)
4. Node sharpening (parameter isolation)
5. Progressive neural nets (parameter expansive)
6. Orthogonal gradient descent (regularization)
7. Elastic weight consolidation (regularization)

• Self-distillation restricts the solution space of the newly trained model such that it overlaps with the prior version of itself.
• Replay and rehearsal continually retrain on old information, by cycling in a small, random subset of prior (and, ideally, maximally informative) training data into each new mini-batch.
• Extending the context length is a palliative pursued mainly for LLMs, which provides the model with more memory space to persist across sessions and learnings.
• Node sharpening focuses on turning a highly distributed network into a sparse one, such that new knowledge is slotted into a distinct, non-colliding space rather than overwriting a distributed representation for many inputs. MoE is a modern application of these techniques to LLMs.
• Progressive neural nets add new capacity for each task rather than repurposing existing parameters, sidestepping interference entirely at the cost of parameter growth.
• Orthogonal gradient descent packs more into a single parameter, by optimizing subsequent tasks within a subspace of the loss landscape orthogonal to previous task optimizations.
• Elastic weight consolidation penalizes changes to parameters that were important for earlier tasks, using the Fisher information matrix as a proxy for importance.

Self-Distillation

Algorithm

Code

model_B = swap_head(model_A, task_B_classes) # current model, initialized from A
model_A.eval() # frozen teacher
optimizer = AdamW(model_B.parameters(), lr)

for x, y in batch:
    B_logits = model_B.forward(x)
    with torch.no_grad():
        A_logits = model_A.forward(x) # no gradients needed for teacher

    loss_new = criterion(B_logits, y) # e.g. cross entropy
    loss_distill = kl_div(
        log_softmax(B_logits / T, dim=-1),
        softmax(A_logits / T, dim=-1)
    ) * (T ** 2)

    loss = loss_new + lam * loss_distill
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

Limitations

• Distillation occurs purely in the output space, meaning it has no explicit awareness of which parameters matter most for task A (compared to methods like EWC below), leading to slightly unpredictable retention
• Often practically inferior to replay when memory of previous task data is allowed (which we'll discuss next!)

Recent extensions

Replay Buffers and Rehearsal

Algorithm

1. Maintain a fixed-capacity buffer $\mathcal{B}$
2. On arrival of new task with data $\mathcal{D}_B$: a). For each training step, sample a batch from $\mathcal{D}_B$ and a batch from $\mathcal{B}$. Train $f_\theta$ on their union: $\mathcal{L} = \mathbb{E}_{(x,y) \sim \mathcal{D}_B}[\ell(y, f_\theta(x))] + \mathbb{E}_{(x,y) \sim \mathcal{B}}[\ell(y, f_\theta(x))]$
3. Buffer update: after (or during) training on $\mathcal{D}_B$, select samples from $\mathcal{D}_B$ to add to $\mathcal{B}$. If $\mathcal{B}$ is at capacity, evict existing samples to make room
   • Insertion policy $\pi_{\text{add}}$: which samples from $\mathcal{D}_B$ are stored (Default: all)
   • Eviction policy $\pi_{\text{evict}}$: which samples in $\mathcal{B}$ are removed? (Default: uniform random)

Practical considerations

Limitations

• Overhead from buffer in compute, as (many) additional forward and backward passes are needed to progress on new tasks
• Building replay buffers for LLMs is difficult due to licensing and scale, though training a generator is a common workaround.

Extending Context Length

Limitations

• Attention scales quadratically in context length in both memory and FLOPs
• Context rot demonstrates a nonlinear relationship between tokens in context length and their utility in producing a better response.
• Text is a lossy representation of true guidance, which for humans includes (at least) sensorimotor grounding, social feedback, and temporal context.

Node Sharpening

Code

class Model:
...
	def forward(x):
    h = hidden_layers(x)
    h_sharp = sharpen(h) # sharpen the intermediate activations
    out = output_layers(h_sharp)  
    return out, h, h_sharp # return hidden to compute sharpness loss
    
def sharpen(x, temperature=5.0):
    return torch.softmax(x * temperature, dim=-1)
    
# Train loop with added loss to sharpen
for x, y in batch:
	pred, h, h_sharp = model.forward(x)
	sharpness_loss = ((h - h_sharp) ** 2).sum()
	loss = criterion(pred, y) + lam * sharpness_loss
	optimizer.zero_grad()
	loss.backward()
	optimizer.step()
	...

Limitations

• Sharpening trades generalization abilities for continual learning abilities
• Modern deep representations naturally become sparse

Progressive Neural Networks

Limitations

• Parameter count of PNN grows quadratically with the number of tasks

Orthogonal Gradient Descent

Algorithm

Limitations:

• Storage usage grows with the number of tasks $|S| = \#tasks \times D \times 200$
• $D$ (# params) can be very large (LLMs are over a trillion parameters) and thus even a single gradient vector can be difficult to store
• Rigid orthogonality is theoretically pretty but practically overly conservative

Elastic Weight Consolidation

Understanding Fisher Information

Code

def compute_fisher_diagonal(model, dataset) -> dict:
    """Compute the diagonal Fisher: F_ii = E[(d/d_theta_i log p(y|x))^2]"""
    fisher = {n: torch.zeros_like(p) for n, p in model.named_parameters()}
    N = len(dataset)

    model.eval()
    for input, gt_label in dataset:
        model.zero_grad() # clear gradients
        logits = model(input) # (B, C)
        loss = F.nll_loss(F.log_softmax(logits, dim=1), gt_label) 
        loss.backward() # computes gradients

        for n, p in model.named_parameters():
	        # same shape as params p, e.g. (H, W)
            fisher[n].data += p.grad.data ** 2 / N 

    return fisher

def get_ewc_loss(model, fisher, p_old):
    """Compute the EWC penalty as the sum of Fisher-weighted
    squared parameter shifts"""
    loss = 0
    for n, p in model.named_parameters():
        delta = p - p_old[n] # shape e.g. (H,W)
        loss += (fisher[n] * delta ** 2).sum() # scalar
        
    return loss
    
# USAGE
# 1. Train on task A normally
train(model, dataset_A)
# 2. Snapshot params at theta_A*
p_old = {n: p.data.clone() for n, p in model.named_parameters()}
# 3. Compute Fisher at theta_A* over a subset of D_A
fisher = compute_fisher_diagonal(model, dataset_A_subset) # ~200 samples
# 4. Train on task B with EWC
for x, y in dataset_B:
    pred = model(x)
    loss = criterion(pred, y) + lam * get_ewc_loss(model, fisher, p_old)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

Limitations

• The diagonal Fisher is a crude approximation of the true Fisher information, limiting effectiveness
• $\lambda$ is a new hyperparameter that's hard to balance
• Practically outperformed by a replay buffer of $\sim10-20$ samples per class (above)

Conclusion