The Problem

You’re given 97 weight/bias files (piece_0.pth through piece_96.pth) and a dataset (historical_data.csv with 10,000 rows of 48 input features, plus pred and true columns). The neural network architecture is:

• 48 residual blocks, each consisting of:
  • An “inp” layer: Linear(48 → 96) followed by ReLU
  • An “out” layer: Linear(96 → 48)
  • A residual connection: x = x + out(relu(inp(x)))
• 1 final layer: Linear(48 → 1) producing the prediction

The 97 pieces split into three groups by weight shape:

• 48 pieces with shape (96, 48) — the inp layers
• 48 pieces with shape (48, 96) — the out layers
• 1 piece with shape (1, 48) — the final layer

The solution is a permutation of indices 0–96 specifying which piece goes where. Positions 0,2,4,…,94 hold inp layers, positions 1,3,5,…,95 hold out layers, and position 96 holds the final layer. The solution is verified by SHA-256 hash — there’s exactly one correct answer, no MSE threshold to meet.

This means you need to solve two sub-problems simultaneously:

1. Pairing: Which inp layer goes with which out layer in each block?
2. Ordering: In what sequence do the 48 blocks execute?

The search space is enormous: 48! × 48! ≈ 10¹²¹ possible configurations.

Phase 1: First-Order Approximations (MSE ~0.7)

My first instinct was to exploit the linear structure. If all 48 blocks see roughly the same input X (a first-order approximation), then each block’s contribution is independent, and we can use the Hungarian algorithm to find the optimal pairing.

For each candidate pair (i, j), I computed the block’s effect on the prediction:

h = F.relu(F.linear(X, L1_W[i], L1_B[i]))
delta = F.linear(h, L2_W[j], L2_B[j])  # (N, 48)
pred_delta = (delta * l3_dir).sum(dim=1) * l3_w.norm()

Then built a cost matrix and ran linear_sum_assignment. This got MSE down to ~0.7 — a starting point, but far from correct. The first-order approximation breaks down because blocks modify x sequentially, and the cumulative change is large (~6× the input norm).

Phase 2: Gumbel-Sinkhorn — Differentiable Permutation Learning (MSE ~0.03)

The breakthrough came from treating permutations as differentiable objects using the Gumbel-Sinkhorn framework.

The Key Idea

Instead of searching over discrete permutations, parameterize a continuous relaxation. A 48×48 matrix of learnable logits log_alpha is transformed into a doubly-stochastic matrix (a “soft permutation”) via iterated row/column normalization (Sinkhorn’s algorithm):

def sinkhorn(log_alpha, n_iters=25, tau=1.0):
    log_alpha = log_alpha / tau
    for _ in range(n_iters):
        log_alpha = log_alpha - torch.logsumexp(log_alpha, dim=1, keepdim=True)
        log_alpha = log_alpha - torch.logsumexp(log_alpha, dim=0, keepdim=True)
    return log_alpha.exp()

Adding Gumbel noise before normalization enables exploration, and annealing the temperature tau from high to low gradually sharpens the soft permutation toward a hard one. The MSE loss is fully differentiable through this soft permutation, so we can use Adam to optimize the logits.

Alternating Optimization

Jointly optimizing both the ordering permutation and the pairing permutation is expensive — the forward pass with two soft permutations involves O(48³) operations per position. The key insight was to alternate:

1. Fix pairing, optimize ordering: The soft forward pass weights different block orderings:

   def forward_soft_order(x, pairing, order_weights):
    for pos in range(48):
        # Precompute all block deltas with fixed pairing
        all_deltas = [block_i_j(x) for i,j in pairing]
        # Weighted combination based on soft ordering
        delta = einsum('i,bid->bd', order_weights[pos], all_deltas)
        x = x + delta
   

2. Fix ordering, optimize pairing: Each block position softly selects among all possible out layers:

   def forward_soft_pair(x, order, pair_weights):
    for inp_idx in order:
        h = relu(linear(x, L1_W[inp_idx], L1_B[inp_idx]))
        # Soft-select out layer
        weighted_w = einsum('j,jdo->do', pair_weights[inp_idx], L2_W)
        delta = linear(h, weighted_w, weighted_b)
        x = x + delta
   

Each sub-problem only involves one 48×48 permutation matrix, making it much faster. After optimization, I extract hard permutations using the Hungarian algorithm on the negative logits.

With 5-6 alternations of 500-800 gradient steps each, MSE dropped from 0.8 to ~0.03 — an order of magnitude better than first-order methods.

Why Alternating Works

Alternating optimization works here because the ordering and pairing sub-problems are partially decoupled. Fixing one makes the other a “standard” assignment problem with a smooth loss landscape. The Gumbel noise acts as a form of stochastic exploration, and the temperature annealing provides a natural curriculum from exploration to exploitation.

Phase 3: Local Search — Getting Stuck (MSE ~0.03)

With a good Gumbel-Sinkhorn solution in hand, I tried various local search strategies:

• 2-opt: Swap pairs of positions in the ordering, or pairs of pairings
• 3-opt: Try all triples of positions with all 6 permutations
• Insertion moves: Remove a block and reinsert at every other position
• Coordinate descent: For each position, try all 48×48 possible replacements

None of these could escape the MSE ~0.03 basin. The solution was at a strict local minimum for all single-element and pair-element moves. Multiple random restarts with the Gumbel approach also converged to similar MSE values.

Phase 4: Two Paths to the Solution (MSE 0.008 → 0.0)

From MSE ~0.008, I found two different approaches that both reach MSE = 0. Each reveals something different about the problem structure.

Approach A: Combined 2-opt

The first insight was that standard 2-opt treats order swaps and pairing swaps as independent moves. But the correct solution might require simultaneously changing both the order AND the pairing of two positions.

Combined 2-opt tests all three modifications for each pair of positions (p1, p2):

1. Swap their order positions only
2. Swap their pairings only
3. Swap both order AND pairing simultaneously

for p1 in range(48):
    for p2 in range(p1+1, 48):
        i1, i2 = order[p1], order[p2]
        j1, j2 = pairing[i1], pairing[i2]

        for swap_order, swap_pair in [(True,False), (False,True), (True,True)]:
            if swap_order: order[p1], order[p2] = i2, i1
            if swap_pair: pairing[i1], pairing[i2] = j2, j1
            mse = full_eval(order, pairing)
            if mse < best_mse:
                # Accept improvement
                ...

This is O(48² × 3) = 6,912 evaluations per sweep. Starting from MSE 0.0085, it made 86 consecutive improving swaps in a single pass down to MSE = 0.

The intuition: when two blocks have tangled errors, swapping just their order or just their pairing each makes things worse, but swapping both simultaneously moves between consistent configurations. In optimization terms, the individual moves each increase the loss, but their composition decreases it — a “valley” that requires moving diagonally.

Approach B: Alternating Cycles with Insertions (Simpler, Same Result)

The second approach is simpler but equally effective: cycle through three move types and keep going long after apparent convergence.

The three moves:

1. Pairing swaps: Try all C(48,2) = 1,128 L2 partner exchanges
2. Order swaps: Try all 1,128 position exchanges
3. Block insertions: For each of 48 blocks, remove it and try all 48 positions (2,304 evals)

for round in range(many):
    # Pairing swaps
    for i, j in combinations(range(48), 2):
        swap pairing[i], pairing[j]; accept if improved

    # Order swaps
    for i, j in combinations(range(48), 2):
        swap order[i], order[j]; accept if improved

    # Block insertions
    for i in range(48):
        block = order.pop(i)
        try all 48 insert positions; keep best

What makes this work is patience — continuing to cycle when each individual move type appears converged. The key discovery: pairing corrections trigger cascading order improvements.

Starting from MSE 0.0098 (where standard 2-opt appeared stuck), the trajectory looked like this:

Cycle  5: Pairing fix:    0.008274  ← corrected one L1/L2 pair
          ...18 order swaps...
          Order swap:      0.006588  ← cascade!
          ...7 insertions...
          Block insertion:  0.003861

Cycle  6: Pairing fix:    0.002379  ← biggest single improvement
          ...16 order swaps...
          Order swap:      0.000177  ← nearly there
          Block insertion:  0.000064
          Block insertion:  0.000000  ← EXACT!

Each pairing correction fixed a block that had been paired with the wrong L2 layer. With the wrong partner, no ordering could make that block work correctly — so the optimizer was forced into a compromise. Once the pairing was fixed, a flood of previously-blocked order improvements became available.

Why Insertions Matter

Insert moves find improvements that swaps cannot. A swap exchanges two elements; an insert slides one element to a new position, shifting everything in between. The final three moves to MSE = 0 were all insertions — they refined block positions with a precision that pairwise swaps couldn’t match.

Error Analysis: The Tail Tells the Story

At MSE ~0.01, analyzing the per-row error distribution was revealing:

Percentiles of |error|:
  50th: 0.026    (median row is nearly correct)
  95th: 0.210
  99th: 0.415
  100th: 1.496   (worst row is way off)

Top 100 rows: MSE 0.348  (35x more error per row)
Bottom 9900:  MSE 0.006

The error was concentrated in ~45 extreme rows. This pattern — a mostly-correct solution with a few outliers — is the signature of a few specific misconfigurations rather than a globally wrong solution. It motivated continued cycling over restart.

The Full Pipeline

Both paths share the same initialization and diverge at Phase 4:

1. First-order pairing (200 random restarts + swap optimization) → MSE ~0.7
2. Gumbel-Sinkhorn alternating optimization → MSE ~0.03
3. Standard 2-opt + insertion moves → MSE ~0.008
4. Either:
   • (A) Combined 2-opt → MSE = 0.0 ✓ (single pass, ~7K evals)
   • (B) Alternating pair/order/insert cycles → MSE = 0.0 ✓ (~10 cycles, ~45 min)

Approach A is faster per pass but requires the insight to try simultaneous swaps. Approach B is slower but conceptually simpler — just keep cycling basic moves and let pairing corrections cascade into order improvements.

Total computation: under an hour on a MacBook Pro (M-series, CPU only).

Lessons Learned

Differentiable relaxations are powerful initialization. Gumbel-Sinkhorn took us from a random permutation to within ~1% of the correct answer. Without it, local search would have no hope in a space of 10¹²¹ configurations.

Pairing corrections unlock order improvements. A wrong L1/L2 pairing poisons the ordering — no arrangement of blocks can compensate for a block producing the wrong intermediate values. Each pairing fix unblocked 15-20 order improvements that had been invisible before.

Insert moves find what swaps miss. The final three moves to MSE = 0 were all block insertions. Insertions shift an entire segment of the ordering, exploring a richer neighborhood than pairwise swaps.

Cycle, don’t stop. After apparent convergence, continuing to cycle through move types found improvements for 5+ more rounds. Each round took ~90 seconds, so patience was cheap.

The right neighborhood matters more than the right algorithm. Standard 2-opt, 3-opt, simulated annealing, and coordinate descent all failed at MSE ~0.01. Both solutions came from expanding the move set — either by combining swap types (Approach A) or by adding insertions and being patient (Approach B).

Save incrementally. I learned this the hard way — a script that only saves at the end can lose hours of progress if killed. Every improving move should write to disk immediately.

Exact verification changes the game. The SHA-256 hash means only MSE = 0 is correct. This motivated exhaustive local search: even a tiny MSE improvement matters because there’s no “good enough.”

Dead Ends and Abandoned Approaches

Before finding the two approaches that worked, I tried several others that didn’t pan out:

Simulated annealing. The natural response to getting stuck at a local minimum. I implemented SA with multiple move types (order swaps, pairing swaps, block insertions, segment reversals) and ran it for hundreds of thousands of steps. The problem: each evaluation requires a full sequential forward pass through 48 blocks on thousands of samples (~7ms per eval). At 500K steps, that’s nearly an hour per run — and SA needs many restarts to be effective. Worse, the high-dimensional discrete landscape (two interleaved 48-element permutations) makes it hard to set a temperature schedule that explores enough without wasting time in bad regions. The occasional improvements SA found were always things that deterministic local search could have found faster by just cycling more.

Greedy sequential construction. Rather than optimizing the ordering, build it greedily: at each step, try all remaining blocks and pick the one that minimizes the partial prediction error. This was fast (~1 second per full construction) but gave MSE ~1.8 — worse than the starting point. The problem is myopia: the block that looks best at step k might be terrible for what’s needed at steps k+1 through 47. The residual structure means early blocks fundamentally reshape the input for later blocks, so local greedy choices cascade into globally poor orderings.

3-opt (triple rotations). If 2-opt is stuck, try 3-opt — cyclic rotations of three elements. The cost is O(n³) = 17,296 triples, each tested in two rotation directions, times ~7ms per eval = ~4 minutes per sweep. I ran this on both ordering and pairing. It was too slow to iterate and never found improvements that the simpler approach (cycling 2-opt with insertions) couldn’t find faster. The 3-element moves that matter are better discovered by doing 2-opt after an insertion changes the landscape.

SiLU activation. The puzzle description says ReLU, but in first-order (non-residual) models, SiLU gives much lower MSE (~0.9 vs ~11.0). This was a red herring — SiLU only wins when you ignore the residual connections. In the full sequential model, ReLU gives MSE 0.12 while SiLU gives 4.37. The lesson: test with the full architecture, not a simplified proxy.

Group swaps. Instead of swapping individual blocks, try swapping contiguous groups of 2, 3, 4, or 8 blocks. This occasionally found tiny improvements (~0.001) but was never transformative. The blocks that need to move aren’t in contiguous groups — they’re scattered, and the real bottleneck is fixing pairings, not rearranging chunks.

Lasso/sparse selection. Precompute all 48×48 = 2,304 possible block outputs and use Lasso regression to select a sparse subset of 48. Elegant in theory, but Lasso doesn’t enforce the constraint that each L1 and L2 layer is used exactly once. Post-hoc matching from the Lasso solution didn’t produce better pairings than direct swap optimization.

Training a surrogate model, then matching layers. I trained a fresh neural network with the same architecture on the 10K dataset, hoping to match its learned layers against the puzzle pieces. The results were poor — I suspect 10K samples simply aren’t enough to recover a model similar enough to the target for layer-wise matching to work. The trained model converges to a different local minimum with different internal representations, making piece-to-layer correspondence unreliable.

Training a transformer to predict swaps. The most ambitious attempt: train a transformer model to learn which swaps improve the objective, then let it predict a sequence of moves to solve the puzzle. This ran into a bootstrapping problem — generating training data (pairs of configurations and their MSE changes) required the same expensive forward passes we were trying to avoid, and I couldn’t produce enough samples to train on. The model would need to generalize from a tiny fraction of the 10¹²¹ search space, with no clear inductive bias for this specific combinatorial structure. In hindsight, domain-specific search (exploiting the residual network structure directly) was always going to beat a general-purpose learned search policy for a one-off puzzle like this.

The common thread: the bottleneck was always pairing, not ordering. Approaches that focused on finding better orderings (SA, greedy construction, 3-opt, group swaps) couldn’t overcome wrong pairings. The approaches that worked were the ones that could fix pairings and then let order improvements cascade.

Good luck if you’re attempting this one — it’s a satisfying puzzle to crack.

In this post, I’ll walk through how HRM actually works by tracing the code and architecture step by step. I’ll also cover the important follow-up critiques that question some of these claims.

The Big Idea

Current LLMs reason by writing out their thinking step by step (Chain-of-Thought). This works, but it’s slow, requires huge models, and needs lots of training data. HRM takes a completely different approach: it reasons in latent space — inside the model’s hidden states — through iterative refinement.

The core insight is borrowed from neuroscience: the human brain processes information hierarchically, with slow abstract planning and fast detailed computation happening at different timescales. HRM mimics this with two transformer modules that talk to each other.

The Two-Level Architecture

HRM has two recurrent transformer modules:

H-level (High-level planner) — 4 transformer layers, responsible for slow, abstract reasoning. Think of it as the part that asks: “What strategy should I use?”

L-level (Low-level executor) — 4 transformer layers, responsible for fast, detailed computation. This handles: “What goes in this specific cell?”

They interact in a nested loop:

For each H-cycle (2x):
    For each L-cycle (2x):
        z_L = L_level(z_L, z_H + input_embeddings)
    z_H = H_level(z_H, z_L)

The L-level refines its understanding using the H-level’s guidance plus the raw input. Then the H-level updates its plan based on what L found. Both use non-causal attention — every position can see every other position simultaneously.

One important detail: both modules are ReasoningModule wrappers that add the injection to the hidden state before running through their transformer layers:

def forward(self, hidden_states, input_injection, **kwargs):
    hidden_states = hidden_states + input_injection   # inject
    for layer in self.layers:
        hidden_states = layer(hidden_states=hidden_states, **kwargs)
    return hidden_states

So L doesn’t replace its state — it adds z_H + input to its existing state, then processes. Same for H adding z_L.

Adaptive Computation Time (ACT): The Outer Loop

The H/L cycles above describe what happens within a single step. But HRM can take multiple steps, deciding dynamically how long to think. This is the Adaptive Computation Time (ACT) wrapper.

Each call to model.forward(carry, batch) is one ACT step. The training/evaluation loop calls it repeatedly:

# Evaluation loop
while True:
    carry, _, metrics, preds, all_finish = model(carry, batch)
    if all_finish:
        break

The model can take up to 16 ACT steps (configurable). At each step, it decides: halt or continue?

Here’s how the two levels of looping connect:

ACT Step 1  ──→  H/L cycles (2x2) inside  ──→  logits + Q-values
                                                   │
                                            Q says "continue"
                                                   ↓
ACT Step 2  ──→  H/L cycles (2x2) inside  ──→  logits + Q-values
                 (carry from step 1                │
                  flows in)                  Q says "continue"
                                                   ↓
ACT Step 3  ──→  H/L cycles (2x2) inside  ──→  logits + Q-values
                                                   │
                                            Q says "HALT"
                                                   ↓
                                            Final answer used

With 16 ACT steps, each containing 2 H-cycles x 2 L-cycles, the model can perform up to 64 L-passes + 32 H-passes — massive computational depth from a tiny model, because the same weights are reused every time.

z_H and z_L: The Model’s Working Memory

So what exactly are z_H and z_L? They’re hidden state tensors — the model’s evolving “thoughts” at each level.

Let’s make this concrete with a Sudoku example. A 9x9 puzzle gets flattened into 81 integers:

inputs = [5, 3, 0, 0, 7, 0, 0, 0, 0, 6, 0, 0, ...]
          cell1  cell2  cell3  ...              cell81

Each integer gets embedded into a 512-dimensional vector. Then a puzzle embedding (more on this later) is prepended as position 0. So the final sequence has 82 positions:

position 0:  puzzle embedding    ← 512-dim vector
position 1:  cell 1 embedding   ← 512-dim vector
position 2:  cell 2 embedding   ← 512-dim vector
...
position 81: cell 81 embedding  ← 512-dim vector

Both z_H and z_L have this same shape: (batch_size, 82, 512). Each position holds a 512-dimensional vector representing the model’s current “thoughts” about that cell.

When a sequence starts fresh, both are initialized to learned vectors — H_init and L_init — broadcast across all positions. The model starts with the same state everywhere and must differentiate through the input injection and attention.

After each ACT step, both are detached (gradients cut) and stored in a carry dataclass. The next step picks up where the last left off — but no gradients flow backward between steps. This is what makes the whole thing memory-feasible.

Position 0 is special. Since it holds the puzzle embedding (not a cell value), it acts as a global summary token. Through non-causal attention, it sees all 81 cells. The Q-head reads z_H[:, 0] specifically to make the halt/continue decision:

q_logits = self.q_head(z_H[:, 0])   # position 0 → halt decision

And the final answer is read from the remaining positions:

output = self.lm_head(z_H)[:, puzzle_emb_len:]   # positions 1-81 → predictions

Puzzle Embeddings: Per-Puzzle Identity

Not all puzzle types need this, and the difference is revealing.

Sudoku: every puzzle follows the same rule (fill digits 1-9, no repeats in row/column/box). So puzzle_identifiers = 0 for every example. One universal algorithm.

ARC: every puzzle has a different rule. Puzzle 42 might be “rotate the shape 90°”, puzzle 137 might be “fill enclosed regions with blue”. The model needs to know which puzzle it’s solving.

For ARC, the dataset builder assigns each puzzle a unique integer ID (1 through ~960). The model has a learnable embedding table:

puzzle_emb: shape (961, 512)

Row 0:   [0, 0, ..., 0]            ← blank (unused)
Row 1:   [0.12, -0.34, ..., 0.56]  ← learned embedding for puzzle 1
Row 2:   [-0.78, 0.91, ..., 0.23]  ← learned embedding for puzzle 2
...

Each embedding starts at zero and is trained via SignSGD — a simple optimizer that only uses the sign of the gradient:

w = w * (1 - lr * weight_decay) - lr * sign(gradient)

Every weight goes up by lr or down by lr, regardless of gradient magnitude. Why not Adam? Because puzzle embeddings are extremely sparse — with ~960 puzzles and a batch of 768, most rows get no gradient on any given step. Adam would approximate SignSGD anyway for such sparse updates, but SignSGD is simpler and needs zero optimizer state (no momentum, no second moment to track).

The puzzle embedding is trained with a separate optimizer at 100x the learning rate of the main model (0.01 vs 0.0001) and 10x the weight decay (1.0 vs 0.1). It updates rarely, so it needs to move fast when it does.

The Q-Learning Halting Mechanism

How does the model decide when to stop thinking? Through two Q-values produced by a tiny linear head:

self.q_head = CastedLinear(hidden_size, 2, bias=True)   # 512 → 2 numbers

It reads z_H[:, 0] (the summary token) and outputs:

• q_halt: “how confident am I that my current answer is correct?”
• q_continue: “how confident am I that continuing will lead to a correct answer?”

If q_halt > q_continue, the model halts.

Training q_halt: supervised from ground truth

seq_is_correct = (number_of_correct_cells == total_cells)   # True or False
q_halt_loss = binary_cross_entropy(q_halt_logits, seq_is_correct)

Simple. Did you get every cell right? Push q_halt toward 1. Wrong? Push toward 0.

Training q_continue: bootstrapping from the future

This is the trickier part. There’s no ground truth for “will continuing help?” So the model peeks ahead — it runs one more forward pass from the current carry state:

next_q_halt, next_q_continue = self.inner(new_inner_carry, new_current_data)[-1]

The target for q_continue at step t is: the best outcome achievable from step t+1 onward.

target = sigmoid(
    where(is_last_step,
        next_q_halt,                            # forced to halt next step
        max(next_q_halt, next_q_continue)        # best option at next step
    )
)

This is the Bellman equation from reinforcement learning. If at the next step, halting gives 82% confidence and continuing gives 69%, then the value of continuing now is 82% (you’d halt next step). The target follows whichever future path leads to the best outcome.

The bootstrapping cold start

At the beginning of training, both Q-values are meaningless. The Q-head is initialized with bias = -5, so sigmoid(-5) ≈ 0.007 — the model believes there’s a 0.7% chance of being correct for everything. Since q_halt ≈ q_continue, nobody halts early; everything runs to the maximum 16 steps.

The chain reaction goes:

1. lm_loss gradually teaches the model to produce correct answers
2. q_halt starts learning which answers are correct (grounded in truth)
3. Once q_halt is meaningful at step 16, q_continue at step 15 gets a real target
4. That propagates backward: step 14, 13, 12…
5. Eventually the model learns to halt early for easy puzzles, run longer for hard ones

Exploration

Without exploration, the Q-head can get stuck — if it always halts at step 3, it never discovers that step 8 would give the right answer. So 10% of the time, each batch item gets a random minimum number of steps it must run before halting is allowed:

min_halt_steps = (rand() < 0.1) * randint(2, max_steps + 1)
halted = halted & (steps >= min_halt_steps)

This ensures the model occasionally sees deeper computation and can update its estimates.

Training: Two Optimizers, One Loss

Each training step:

1. Forward pass — puzzle embeddings copied to local buffer, flow through L/H cycles, produce logits + Q-values
2. Single backward pass — gradients flow through everything
3. Two optimizers step:
   • SignSGD for puzzle embeddings (lr=0.01, weight_decay=1.0)
   • Adam for all transformer weights (lr=0.0001, weight_decay=0.1)

The total loss combines three terms:

total_loss = lm_loss + 0.5 * (q_halt_loss + q_continue_loss)

All three losses backpropagate through the entire model. The Q-losses aren’t just training the Q-head — they shape the representations in z_H and z_L throughout, forcing the model to develop internal representations of “how solved is this puzzle.”

The gradient efficiency trick

Within each ACT step, only the final H/L cycle computes gradients. All earlier cycles run in torch.no_grad():

with torch.no_grad():
    # Run H_cycles * L_cycles - 1 warmup iterations
    for H_step in range(H_cycles):
        for L_step in range(L_cycles):
            if not (last H and last L):
                z_L = L_level(z_L, z_H + input)
        if not last H:
            z_H = H_level(z_H, z_L)

# Only this final step has gradients:
z_L = L_level(z_L, z_H + input)
z_H = H_level(z_H, z_L)

The hidden states carry forward information from the no-grad iterations, but only the final refinement contributes to the loss. This dramatically reduces memory usage.

Limitations: No Branching, No Backtracking

HRM’s computation is a single linear path:

carry → step 1 → step 2 → step 3 → ... → answer

As humans, when we solve puzzles, we do something different:

• “What if this cell is 5?” → follow implications → contradiction → backtrack
• “OK, what if it’s 7?” → follow implications → works → keep going

That’s tree search — branching, evaluating, backtracking. HRM can’t do this. If step 2 goes down a wrong path, step 3 builds on that wrong foundation.

The non-causal attention can partially compensate by processing all positions simultaneously (like parallel constraint propagation rather than sequential hypothesis testing). But for tasks that fundamentally require exploring multiple hypotheses — like playing Go, where you need to simulate opponent responses many moves ahead — HRM’s single-path architecture won’t work.

The Follow-Up Critiques

Two important independent analyses appeared after HRM’s release, and they paint a different picture than the original paper.

ARC Prize Team Analysis

The ARC Prize team verified HRM’s results and ran ablation studies. Their key findings:

The hierarchy barely matters. A regular transformer with the same parameter count came within ~5 percentage points of HRM without any hyperparameter tuning. The H/L architectural split isn’t the secret sauce.

The refinement loop is the real driver. Performance jumped +13 percentage points from zero to one refinement iteration. This is the ACT outer loop — but any recurrent architecture could benefit from iterative refinement.

Puzzle embeddings limit generalization. Since each puzzle gets a learned embedding by ID, the model can only work on puzzles it has seen during training. This makes HRM closer to “test-time training” (memorizing each puzzle’s pattern) than genuine reasoning that generalizes to novel puzzles.

Ge, Liao & Poggio Analysis (arXiv 2510.00355)

Researchers from MIT published “Hierarchical Reasoning Models: Perspectives and Misconceptions” with further findings:

A flat model works equally well. An 8-layer L-only model (no H module at all) achieved similar performance and trained faster (1h 48m vs 4h 21m).

The one-step gradient trick isn’t novel. The no-grad warmup + 1-step gradient pattern is mathematically equivalent to how diffusion models and Latent Consistency Models train. It’s a known technique.

ACT doesn’t help at inference. Running for the maximum number of steps always gives the best results. The learned halting policy is never actually useful — the code itself always runs to halt_max_steps during evaluation.

Is it even recurrent? Since only the last cycle has gradients and the carry is detached between ACT steps, the paper questions whether HRM is truly recurrent or just a very deep feedforward model.

What’s Genuinely Interesting

Despite the critiques, HRM points toward ideas worth taking seriously:

Latent-space reasoning works. Instead of generating tokens to “think” (Chain-of-Thought), you can reason inside hidden states. This is fundamentally faster — no autoregressive token generation — and the ARC results show it’s viable even at 27M parameters.

Iterative refinement is powerful. Running the same model multiple times with carried state is a simple idea with outsized impact. The +13pp jump from zero to one refinement iteration shows this clearly.

Small models can do complex reasoning. With the right architecture and training setup, you don’t need billions of parameters for tasks like Sudoku and maze solving. The computational depth comes from recurrence, not model size.

The specific hierarchical architecture may not be essential, and the puzzle embeddings are a significant limitation. But the broader research direction — compact models that reason through iterative latent computation — is one worth watching.

Part 1: BrushNet — The Inpainting Engine

The Problem: Teaching a Model to Fill Holes

Imagine you have a photo of a dog on a beach. You want to replace the dog with a sandcastle. You need a model that:

1. Understands what’s around the hole (beach, sky, waves)
2. Generates something new that matches (a sandcastle)
3. Blends it seamlessly at the edges

The simplest approach? Fine-tune the entire diffusion model for inpainting. But this has a big downside — you break the original model. It can’t do normal image generation anymore, and you can’t swap in a better base model later.

BrushNet’s solution: keep the original model frozen, and add a separate trainable branch alongside it.

The Two-Branch Architecture

BrushNet runs two U-Nets in parallel:

                 ┌─────────────────────────┐
  Text prompt ──→│  Base U-Net (FROZEN)     │──→ Predicted noise
                 │  Has cross-attention     │
                 │  to understand text      │
                 └────────────▲────────────┘
                              │
                         + (add features)
                              │
                 ┌────────────┴────────────┐
  Masked image ─→│  BrushNet (TRAINABLE)    │
  + mask ────────→│  NO cross-attention      │
  + noisy latent →│  Processes spatial info  │
                 └─────────────────────────┘

The Base U-Net does what it always does — denoise an image guided by a text prompt. BrushNet runs alongside it, processing the mask and surrounding context, then injects hints into the Base U-Net at every layer.

What Goes Into BrushNet?

BrushNet takes 3 things, concatenated into a 9-channel input:

┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐
│  Noisy latent    │  │  Masked image    │  │  Binary mask     │
│  (4 channels)    │  │  (4 channels)    │  │  (1 channel)     │
│                  │  │                  │  │                  │
│  Current state   │  │  What's around   │  │  Where the       │
│  of denoising    │  │  the hole        │  │  hole is         │
└──────────────────┘  └──────────────────┘  └──────────────────┘
         │                     │                     │
         └─────────────────────┴─────────────────────┘
                               │
                     Concatenate → 9 channels
                               │
                         ┌─────▼─────┐
                         │ BrushNet  │
                         └───────────┘

Why these 3 inputs? What does each one do?

Each input answers a different question:

1. Noisy latent z_t (4 channels) — “What step are we at?”

This is the current state of the image being denoised. At each timestep during the denoising loop, the image goes from pure noise to clean image. BrushNet needs to see this so it knows how much noise is left and can produce appropriate injection features for the current step.

t=T (start):   z_t = pure noise          → BrushNet: "everything is noisy, give strong guidance"
t=T/2 (mid):   z_t = half noise/half image → BrushNet: "refine the details"
t=0 (end):     z_t = nearly clean         → BrushNet: "just fix edges"

2. Masked image latent z_masked (4 channels) — “What’s around the hole?”

This is the original image with the masked region zeroed out, then VAE-encoded. It tells BrushNet what the surrounding context looks like — colors, textures, edges near the mask boundary.

Original:     [beach][dog][beach]
Mask applied: [beach][ 0 ][beach]    ← dog region zeroed out
VAE encode:   [4-channel latent]     ← this goes to BrushNet

Why 4 channels instead of 3 (RGB)? Because the U-Net operates in VAE latent space, not pixel space. Raw pixels would be mismatched — like feeding English text into a Chinese language model. The VAE encoder translates the image into the same “language” the U-Net understands.

Original image (512×512×3)
        │
   Apply mask (zero out hole region)
        │
   VAE Encoder
        │
Masked image latent (64×64×4)   ← This goes to BrushNet

3. Mask (1 channel) — “Where is the hole?”

A simple binary map: 1 = inpaint here, 0 = keep original. You might think BrushNet could figure this out from the masked image alone (just look for the zeros), but zeroed-out pixels are ambiguous:

Without mask channel:
  z_masked has zeros at (2,3) → Is this black pixels or a hole? 🤷

With mask channel:
  z_masked has zeros at (2,3) + mask=1 at (2,3) → Definitely a hole! ✓

Why all 3 are necessary

Each input answers a different question: when (timestep), what’s around (context), and where (hole location).

The Key Innovation: Zero Convolutions

Here’s the clever part. BrushNet’s features are injected into the Base U-Net through zero convolutions — 1×1 convolutions where all weights start at zero.

At training start:

BrushNet feature ──→ ZeroConv ──→ 0.0 ──→ + Base U-Net feature
                     (all zeros)           (unchanged!)

Why? Because the Base U-Net is a carefully trained model. If you inject random noise into it on day one, you’d destroy its ability to generate images. Starting from zero means:

Training step 0:     BrushNet contributes nothing     (U-Net works normally)
Training step 100:   BrushNet whispers tiny hints      (weights: 0.001)
Training step 10K:   BrushNet provides real guidance   (weights: 0.1)

Concrete Example

Say BrushNet produces a feature value of 0.8 at some position. Here’s what the zero convolution does with it over training:

Step 0:     weight = 0.0    →  0.0 × 0.8 = 0.0    (silent)
Step 1000:  weight = 0.02   →  0.02 × 0.8 = 0.016  (whispering)
Step 10000: weight = 0.25   →  0.25 × 0.8 = 0.2    (contributing)

It’s like slowly turning up the volume from mute. The Base U-Net is never shocked by sudden changes.

Where Are Features Injected?

Unlike ControlNet (which only injects into the decoder), BrushNet injects at every single layer — all encoder blocks, the mid block, and all decoder blocks:

The left column (green) is the trainable BrushNet branch — no cross-attention to text. The right column (blue) is the frozen Base U-Net with text cross-attention. The red arrows are zero-conv injection points where BrushNet features are added element-wise to the Base U-Net.

Each arrow is actually multiple injection points (one per sub-layer), totaling about 25 injection points in total. This dense injection gives BrushNet pixel-level control, which is crucial for inpainting — you need precise boundaries where the generated content meets the original image.

Why No Cross-Attention in BrushNet?

The Base U-Net has cross-attention layers that let it understand text prompts:

Base U-Net block:    ResBlock → CrossAttention("a sunflower") → output
BrushNet block:      ResBlock →                                output
                                   ↑
                             (removed!)

This is by design. BrushNet’s job is purely spatial — “here’s a hole, here’s what’s around it.” The text understanding stays in the Base U-Net. This separation means:

• BrushNet is smaller (~480M vs ~520M params) because it skips attention layers
• It focuses entirely on where to inpaint, not what to generate
• What to generate is handled by the Base U-Net via the text prompt

How Training Works

The training loop is surprisingly simple — it uses the standard diffusion denoising loss:

For each training step:

1. Take a clean image                    "cat on a couch"
2. Generate a RANDOM mask                (random shape, random position)
3. Apply mask to image                   (hole in it)
4. VAE-encode both                       z₀ (clean latent), z_masked (masked latent)
5. Add random noise to clean latent      z_t = mix(z₀, noise, t)
6. Run through both branches:
     BrushNet(z_t, z_masked, mask)       → injection features
     Base_UNet(z_t, text) + features     → predicted noise
7. Loss = ‖ predicted_noise - actual_noise ‖²       (MSE)

Wait — the loss compares noise, not images?

Yes! The model predicts what noise was added, not what the clean image looks like. We know the actual noise because we added it ourselves in step 5. If the model can perfectly predict the noise, we can subtract it to recover the clean image.

We added noise ε to get z_t.
Model predicts ε_θ.
If ε_θ ≈ ε, then z₀ ≈ (z_t - ε_θ) / scale   ← clean image recovered!

No special mask-weighted loss?

Nope. The loss is computed over the entire image, not just the masked region. But the model naturally focuses on the mask because:

• Outside the mask: the frozen Base U-Net already handles this well. BrushNet’s zero-convs learn to stay quiet here (contributing nothing reduces loss just fine).
• Inside the mask: the Base U-Net struggles without context. BrushNet’s features are the only thing that helps here, so gradients push the zero-convs to output useful values.

The mask guides learning implicitly through gradients, not explicitly through loss weighting.

Training data: just clean images

BrushNet doesn’t need paired before/after examples. It’s self-supervised:

Dataset: clean images + text descriptions (same data as Stable Diffusion)
Masks:   generated randomly during training

The model learns to reconstruct whatever was behind a random mask, using the surrounding context and text prompt. At inference, you provide a real mask of what you want to replace.

BrushNet vs. ControlNet vs. Standard Inpainting

Why full U-Net matters for inpainting

ControlNet copies only the encoder half — it injects features into the decoder via the skip connections. This works well for structural guidance (edges, poses) but not for inpainting, where you need fine-grained control at every spatial resolution.

The BrushNet paper showed this clearly:

Full U-Net (BrushNet):  PSNR 19.86  ← best quality
Half U-Net:             PSNR 19.01
ControlNet-style:       PSNR 18.28  ← worst quality

Inpainting needs dense per-pixel control, especially at mask boundaries where generated content must blend seamlessly with the original image.

Inference: Putting It All Together

At inference time, the full pipeline looks like this:

1. User provides: image + mask + text prompt ("a sunflower")

2. Encode:
   masked_image = apply_mask(image, mask)
   z_masked = VAE_encode(masked_image)         [4, 64, 64]
   mask_small = downsample(mask)                [1, 64, 64]

3. Start from pure noise:
   z_T ~ N(0, I)                                [4, 64, 64]

4. Denoise loop (T steps, e.g. 25-50):
   for t in T → 0:
       brushnet_feats = BrushNet(z_t, z_masked, mask_small, t)
       noise_pred = BaseUNet(z_t, t, "a sunflower") + brushnet_feats
       z_{t-1} = scheduler_step(z_t, noise_pred)

5. Decode final latent:
   result = VAE_decode(z_0)                     [3, 512, 512]

6. Blend:
   output = blur_blend(result, original_image, mask)

The final blending step uses a Gaussian-blurred mask to smooth the transition between generated and original pixels, avoiding hard edges.

The Plug-and-Play Promise

Because the Base U-Net is never modified, you can:

• Train one BrushNet and use it with any compatible base model
• Swap in a photorealistic model, an anime model, or a custom fine-tune
• The base model keeps all its original capabilities (text-to-image still works)
• Adjust the conditioning_scale (0.0 to 1.0) to control how much BrushNet influences the output

scale = 0.0  →  Base U-Net only (no inpainting guidance)
scale = 0.5  →  Gentle inpainting hints
scale = 1.0  →  Full BrushNet influence (default)

Model Size

Base U-Net (frozen):     ~520M params
BrushNet (trainable):    ~480M params
  └─ Zero-conv layers:    25 layers, ~20M params
Total at inference:      ~1,000M params (1B)

BrushNet is nearly the same size as the Base U-Net — the only difference is removing cross-attention layers (~40M params saved). The trade-off is clear: 2x memory for plug-and-play flexibility.

BrushNet Summary

BrushNet gives us a powerful inpainting engine. But using it requires you to provide two things manually: a mask (where to edit) and a text prompt (what to generate). For simple cases that’s fine — draw a circle around the dog, type “a sunflower.”

But what if you just want to say “remove the dog” and have the system figure out the rest?

That’s exactly what BrushEdit does. It wraps BrushNet in an intelligent agent pipeline that automates the mask and prompt generation.

Part 2: BrushEdit — From “Remove the Dog” to Edited Image

BrushEdit (arXiv 2412.10316) doesn’t change BrushNet’s architecture at all. Instead, it asks: how do you go from a natural language instruction to a BrushNet-ready mask and prompt?

The answer is an assembly line of 4 AI models:

User: "Remove the dog from the garden"
                │
                ▼
  ┌───────────────────────────┐
  │ 1. MLLM (Qwen2-VL)       │  "What kind of edit? What object?"
  │    Classify + Identify    │  → edit_type = "remove"
  │    + Generate caption     │  → target = "dog"
  └────────────┬──────────────┘  → caption = "garden with flowers"
               ▼
  ┌───────────────────────────┐
  │ 2. GroundingDINO          │  "Where is the dog?"
  │    Text → bounding box    │  → bbox around the dog
  └────────────┬──────────────┘
               ▼
  ┌───────────────────────────┐
  │ 3. SAM                    │  "What's the exact shape?"
  │    Bbox → pixel mask      │  → silhouette of the dog
  └────────────┬──────────────┘
               ▼
  ┌───────────────────────────┐
  │ 4. BrushNet + SD 1.5      │  "Fill the hole"
  │    Mask + caption → image │  → dog replaced with garden
  └───────────────────────────┘

Each model does one thing well. Let’s walk through each step.

Step 1: The MLLM Understands Your Instruction

The MLLM (a vision-language model like Qwen2-VL or GPT-4o) is called three separate times, each with a different question. No fine-tuning — it’s used purely through prompt engineering.

Call 1: “What kind of edit?”

System: "Classify this editing instruction into one of:
         addition, remove, local, global, background.
         Reply with a single word."
User:   "Remove the dog from the garden"

→ "remove"

This classification matters because each edit type needs a different mask strategy:

Call 2: “What object?”

System: "Identify the main object being edited.
         Reply with no more than 5 words, a single noun phrase."
User:   "Remove the dog from the garden"

→ "dog"

This short phrase will be fed to GroundingDINO as a search query. It needs to be concise — just enough to find the right thing in the image.

Call 3: “What should the result look like?”

System: "Describe what the image should look like AFTER the edit.
         Do NOT include elements that are removed or changed."
User:   [source image] + "Remove the dog from the garden"

→ "A peaceful garden path with green grass and flowers"

This becomes the text prompt for BrushNet’s inpainting. Notice: it describes the scene without the dog — because we’re removing it. The MLLM has to understand the instruction well enough to describe the result, not just parrot the input.

Why training-free works here

All three calls use the MLLM off-the-shelf. No fine-tuning. This means you can swap backends freely:

GPT-4o  →  Best quality, requires API key, costs money
Qwen2-VL →  Best open-source, runs locally, ~16 GB VRAM
LLaVA   →  Lighter alternative, ~17 GB VRAM

The paper doesn’t fine-tune any of these models. It just writes good prompts. This is a deliberate design choice — it keeps the system modular and easy to upgrade as better VLMs come out.

Step 2: GroundingDINO Finds the Object

Now we know we’re looking for “dog.” But where in the image is it?

GroundingDINO is an open-vocabulary object detector. Unlike traditional detectors that only recognize a fixed set of classes (like COCO’s 80 categories), it takes any text query and finds matching objects:

Input:  image + "dog"
Output: bounding box (128, 128, 384, 384), confidence 0.89

┌────────────────────────┐
│                        │
│    ┌──────────┐        │
│    │          │        │
│    │   dog    │        │
│    │          │        │
│    └──────────┘        │
│         ↑              │
│    bounding box        │
│    from DINO           │
└────────────────────────┘

This works for any object you can describe in words. “Red car,” “wooden table,” “person in blue shirt” — GroundingDINO handles them all.

Exception: addition edits. If the instruction is “add a cat on the sofa,” there’s no cat to detect yet. In this case, GroundingDINO is skipped entirely. Instead, the MLLM predicts where the new object should go by outputting a bounding box: “given this 512×512 image, the cat should go at [256, 170, 128, 170].”

Step 3: SAM Cuts the Exact Shape

A bounding box is too rough. The box around the dog also includes chunks of grass, maybe a bit of fence. We need the exact silhouette.

SAM (Segment Anything Model) takes the bounding box and produces a pixel-precise mask:

Before (bounding box):          After (SAM mask):

┌────────────────────────┐      ┌────────────────────────┐
│                        │      │                        │
│    ┌──────────┐        │      │      ████████          │
│    │ grass    │        │      │    ████████████        │
│    │   dog    │        │      │    ██████████          │
│    │ grass    │        │      │      ██████            │
│    └──────────┘        │      │        ██              │
│                        │      │                        │
└────────────────────────┘      └────────────────────────┘

Box includes background         Mask follows the dog's
around the dog                   exact silhouette

Edit-type-specific mask adjustments

After SAM produces the mask, BrushEdit adjusts it based on the edit type:

• Remove: Dilate the mask by a few pixels. Fur, hair, and shadows often extend slightly beyond the segmentation boundary. Expanding the mask catches these fuzzy edges.
• Background: Invert the mask. Instead of masking the dog, mask everything except the dog. Now BrushNet will regenerate the entire background while keeping the dog untouched.
• Local: Use the mask as-is. The object is being modified, so we need to cover exactly that region.

Remove (dilated):            Background (inverted):

┌────────────────────────┐   ┌────────────────────────┐
│                        │   │████████████████████████│
│     ██████████         │   │████            ████████│
│   ██████████████       │   │██                ██████│
│   ████████████         │   │████            ████████│
│     ████████           │   │██████        ██████████│
│       ████             │   │████████████████████████│
│                        │   │████████████████████████│
└────────────────────────┘   └────────────────────────┘
Expanded to catch fur/shadow  Everything EXCEPT the dog

Step 4: BrushNet Fills the Hole

Now we have everything BrushNet needs:

This is the exact same BrushNet pipeline we covered in Part 1:

1. masked_image = original × (1 - mask)          ← zero out the dog region
2. z_masked = VAE.encode(masked_image)            ← encode to latent space
3. conditioning = concat(z_masked, mask)          ← 5-channel conditioning
4. Denoising loop (50 steps):
     BrushNet features = BrushNet(z_t, conditioning)
     noise_pred = Base_UNet(z_t, "garden with flowers") + BrushNet features
     z_{t-1} = scheduler.step(z_t, noise_pred)
5. result = VAE.decode(z_0)                       ← back to pixel space
6. output = blur(mask) × result + (1-blur(mask)) × original  ← blend

The blurred mask blending at the end creates a smooth transition at the boundary. Without it, you’d see a hard edge where the generated content meets the original image. This single step accounts for a +10 PSNR improvement in ablation studies.

The Full Pipeline, End to End

Let’s trace through one more example to make sure it’s clear. Instruction: “Change the background to a tropical beach.”

Step 1: MLLM classifies → "background"
        MLLM identifies  → "person" (the foreground object to keep)
        MLLM captions    → "A person standing on a tropical beach with
                            palm trees and turquoise water"

Step 2: GroundingDINO("person") → bounding box around the person

Step 3: SAM(bbox) → pixel mask of the person
        Mask is INVERTED → now covers everything EXCEPT the person
        Coverage: ~75% of the image

Step 4: BrushNet inpaints the masked region (the background)
        using caption "tropical beach with palm trees"
        Person is preserved in the unmasked region
        Blended at edges for seamless transition

The key insight for background edits: GroundingDINO detects the foreground object (the person), SAM segments it, then the mask is inverted. BrushNet never touches the person — it only regenerates the background.

Why Decompose Instead of End-to-End?

You might wonder: why not train one big model that takes “remove the dog” and directly outputs an edited image? That’s what InstructPix2Pix does. BrushEdit’s decomposed approach has three advantages:

1. Transparency. Every intermediate result is visible. You can see the edit classification (“remove”), the detected object (“dog”), the mask, and the caption. If something goes wrong, you know exactly where.

2. User control. You can override any step. Don’t like the auto-generated mask? Draw your own. Want a different caption? Type one. The pipeline doesn’t force you into a black box.

3. No paired training data. InstructPix2Pix needs millions of (instruction, before, after) triples — expensive to create. BrushEdit needs none. The MLLM is used off-the-shelf, GroundingDINO and SAM are pre-trained, and BrushNet trains on standard images with random masks.

The trade-off is complexity. BrushEdit orchestrates 4 separate models totaling ~66 GB of weights. But each model is best-in-class at its job, and you can upgrade any component independently.

How Does It Compare?

vs. Inversion-based methods (DDIM+P2P, Null-Text)

These methods invert the image to noise, then re-denoise with edits. BrushEdit skips inversion entirely — it generates directly in the masked region.

5 PSNR better and 3-40x faster.

vs. Original BrushNet

BrushEdit uses BrushNet internally, but improves on it:

The unified model comes from training on BrushData-v2 — a merged dataset that combines segmentation masks and random masks, plus new removal training pairs where clean-background images are paired with random masks.

BrushEdit’s Limitations

No system is perfect. BrushEdit struggles with:

Irregular masks. Very thin, fragmented, or oddly shaped masks can produce artifacts. The model was trained mostly on blob-like masks and object silhouettes.

Text-mask misalignment. If the caption says “a large elephant” but the mask is tiny, the model can’t fit an elephant in there. The MLLM doesn’t always reason well about spatial constraints.

Base model ceiling. BrushEdit uses Stable Diffusion 1.5 as its backbone. Output quality is bounded by what SD 1.5 can generate. It can’t produce FLUX-quality images because the underlying diffusion model isn’t that capable.

VLM errors cascade. If the MLLM misclassifies the edit type (calling a “remove” a “local edit”), the entire downstream pipeline produces wrong results. There’s no error recovery between steps.

Key Takeaways

BrushNet (Part 1):

1. Dual-branch design: Frozen base model + trainable BrushNet branch. Plug-and-play.
2. 9-channel input: Noisy latent (4) + masked image latent (4) + mask (1).
3. Zero convolutions: Start silent, gradually learn. Stable training.
4. Full U-Net coverage: Encoder + mid + decoder injection. Not just the encoder (ControlNet-style).
5. No cross-attention in BrushNet: Text stays in the Base U-Net. BrushNet handles spatial information only.

BrushEdit (Part 2):

1. 4-model assembly line: MLLM → GroundingDINO → SAM → BrushNet. Each model does one job well.
2. Training-free VLM: The MLLM is used off-the-shelf through prompt engineering. No fine-tuning. Swap backends freely.
3. Edit-type-aware masks: Different edit types get different mask treatments (dilated for removal, inverted for background, bbox for addition).
4. Transparent pipeline: Every intermediate result is visible and overridable by the user.
5. Unified inpainting model: One BrushNet checkpoint handles all mask types, trained on BrushData-v2.

The two papers together tell a clean story: BrushNet solves how to inpaint (the architecture), and BrushEdit solves what to inpaint (the intelligence layer that turns natural language into masks and captions).

This post covers BrushNet (ECCV 2024) and BrushEdit (arXiv 2412.10316). The architecture diagrams come from hands-on experimentation and code analysis of the TencentARC/BrushEdit repository.

What is U-Net?

U-Net is a neural network architecture designed for tasks where you need an image in and an image out of the same size. It was originally created for medical image segmentation in 2015, but has since become the backbone of many modern AI systems, including Stable Diffusion.

The name comes from its shape—when you draw the architecture, it looks like the letter “U”:

Input Image
    │
    ▼
┌─────────────────────────────────────────┐
│  ENCODER (Downsampling)                 │
│  ┌─────┐    ┌─────┐    ┌─────┐         │
│  │64ch │ →  │128ch│ →  │256ch│ → ...   │
│  │128² │    │64²  │    │32²  │         │
│  └──┬──┘    └──┬──┘    └──┬──┘         │
│     │ skip     │ skip     │ skip       │
│     ▼          ▼          ▼            │
│  ┌──┴──┐    ┌──┴──┐    ┌──┴──┐         │
│  │64ch │ ←  │128ch│ ←  │256ch│ ← ...   │
│  │128² │    │64²  │    │32²  │         │
│  └─────┘    └─────┘    └─────┘         │
│  DECODER (Upsampling)                   │
└─────────────────────────────────────────┘
    │
    ▼
Output Image

The Three Key Parts

1. Encoder (The Down Path)

The encoder compresses the image, making it spatially smaller but with more channels:

128×128×3  →  64×64×64  →  32×32×128  →  16×16×256  →  8×8×512
   │              │             │             │            │
   └──────────────┴─────────────┴─────────────┴────────────┘
                    Shrinking spatially
                    Growing in channels

At each step:

• Spatial size halves (128 → 64 → 32 → 16 → 8)
• Channels increase (3 → 64 → 128 → 256 → 512)

This is like summarizing a book—you lose details but capture the main ideas.

2. Bottleneck

The bottleneck is the smallest point in the network:

┌─────────────────────────────────┐
│          8×8×512                │
│                                 │
│  Only 64 spatial positions      │
│  but 512 features each          │
│                                 │
│  "Compressed understanding"     │
└─────────────────────────────────┘

At this point, the network has maximum semantic understanding but minimum spatial detail. It knows “what” is in the image but has lost “where” things are precisely.

3. Decoder (The Up Path)

The decoder expands the image back to full resolution:

8×8×512  →  16×16×256  →  32×32×128  →  64×64×64  →  128×128×3

But here’s the problem: how do you recover the spatial details that were lost?

The Secret Sauce: Skip Connections

This is what makes U-Net special. Skip connections pass information directly from the encoder to the decoder, bypassing the bottleneck:

ENCODER                              DECODER
───────                              ───────
128×128 ─────── skip1 ─────────────→ 128×128
   │                                    ▲
64×64 ───────── skip2 ───────────→ 64×64
   │                                    ▲
32×32 ───────── skip3 ─────────→ 32×32
   │                                    ▲
16×16 ───────── skip4 ───────→ 16×16
   │                                    ▲
   └──→ 8×8 BOTTLENECK ──────────────────┘

Why Are Skip Connections Needed?

Think of it this way:

Visual Example

WITHOUT skip connections:        WITH skip connections:
┌────────────────────┐          ┌────────────────────┐
│                    │          │  ●                 │
│      ◯             │          │   ╲                │
│   (blurry,         │          │    ╲               │
│    wrong spot)     │          │     ●  (sharp,     │
│                    │          │      ╲  correct!)  │
│                    │          │       ●            │
└────────────────────┘          └────────────────────┘

The bottleneck knows “there’s a line somewhere” but lost the exact position. The skip connection says “the line edge is at these exact pixels.” Combined, you get a sharp, accurate output.

The Building Blocks

ConvBlock: The Basic Unit

Every level of the U-Net uses convolutional blocks:

Input
  ↓
Conv 3×3 → BatchNorm → ReLU
  ↓
Conv 3×3 → BatchNorm → ReLU
  ↓
Output

A 3×3 convolution looks at a pixel and its 8 neighbors to compute each output pixel.

Understanding Conv2d

Let’s make this concrete with Conv2d(2, 3, 3) — 2 input channels, 3 output channels, 3×3 kernel.

Key insight: Each output channel has its own filter, and each filter looks at ALL input channels.

INPUT (2 channels)              OUTPUT (3 channels)

┌─────────┐                    ┌─────────┐
│ Ch 0    │──┬─ Filter 0 ─────→│ Ch 0    │
│         │  │                 └─────────┘
└─────────┘  │
             ├─ Filter 1 ─────→┌─────────┐
┌─────────┐  │                 │ Ch 1    │
│ Ch 1    │──┤                 └─────────┘
│         │  │
└─────────┘  └─ Filter 2 ─────→┌─────────┐
                               │ Ch 2    │
                               └─────────┘

Each filter reads ALL input channels to produce ONE output channel.

Concrete Conv2d Example

Input (2 channels, 4×4 each):

Channel 0:              Channel 1:
┌────┬────┬────┬────┐   ┌────┬────┬────┬────┐
│ 10 │ 10 │  0 │  0 │   │  5 │  5 │  5 │  5 │
├────┼────┼────┼────┤   ├────┼────┼────┼────┤
│ 10 │ 10 │  0 │  0 │   │  5 │  5 │  5 │  5 │
├────┼────┼────┼────┤   ├────┼────┼────┼────┤
│ 10 │ 10 │  0 │  0 │   │  5 │  5 │  5 │  5 │
├────┼────┼────┼────┤   ├────┼────┼────┼────┤
│ 10 │ 10 │  0 │  0 │   │  5 │  5 │  5 │  5 │
└────┴────┴────┴────┘   └────┴────┴────┴────┘

Filter 0 (one 3×3 kernel per input channel):

For input ch0:          For input ch1:
┌────┬────┬────┐        ┌────┬────┬────┐
│  1 │  0 │ -1 │        │  0 │  0 │  0 │
├────┼────┼────┤        ├────┼────┼────┤
│  1 │  0 │ -1 │        │  0 │  1 │  0 │
├────┼────┼────┤        ├────┼────┼────┤
│  1 │  0 │ -1 │        │  0 │  0 │  0 │
└────┴────┴────┘        └────┴────┴────┘

To compute output pixel at (row=1, col=1):

From ch0: 10×1 + 10×0 + 0×(-1) + 10×1 + 10×0 + 0×(-1) + 10×1 + 10×0 + 0×(-1) = 30
From ch1: 5×0 + 5×0 + 5×0 + 5×0 + 5×1 + 5×0 + 5×0 + 5×0 + 5×0 = 5
Total: 30 + 5 + bias = 35

DownBlock (Encoder Step)

def forward(self, x):
    features = self.conv(x)     # Process with ConvBlock
    pooled = self.pool(features) # Shrink by half
    return pooled, features      # Return BOTH!

Input: (1, 64, 64, 64)
         │
    ConvBlock
         │
     (1, 128, 64, 64) ──→ SAVED as skip connection
         │
    MaxPool2d (shrink)
         │
Output: (1, 128, 32, 32)

The key: it returns TWO things — the pooled result for the next layer AND the features for the skip connection.

UpBlock (Decoder Step)

def forward(self, x, skip):
    x = self.up(x)              # Grow spatially (ConvTranspose2d)
    x = torch.cat([x, skip], dim=1)  # Concatenate with skip
    x = self.conv(x)            # Process combined features
    return x

Input: (1, 512, 8, 8)    Skip: (1, 512, 16, 16)
         │
  ConvTranspose2d (grow 2×)
         │
     (1, 512, 16, 16)
         │
  Concat with skip (channels add)
         │
     (1, 1024, 16, 16)
         │
  ConvBlock (reduce channels)
         │
Output: (1, 256, 16, 16)

ConvTranspose2d: Growing Images

ConvTranspose2d is the opposite of Conv2d — it makes images bigger:

Conv2d (stride=2):          ConvTranspose2d (stride=2):
  4×4  →  2×2                 2×2  →  4×4
  (shrink)                    (grow)

Each input pixel becomes a 2×2 region:

Input (2×2):          Output (4×4):
┌───┬───┐             ┌───┬───┬───┬───┐
│ 1 │ 2 │             │ 1 │ 1 │ 2 │ 2 │
├───┼───┤      →      ├───┼───┼───┼───┤
│ 3 │ 4 │             │ 1 │ 1 │ 2 │ 2 │
└───┴───┘             ├───┼───┼───┼───┤
                      │ 3 │ 3 │ 4 │ 4 │
                      ├───┼───┼───┼───┤
                      │ 3 │ 3 │ 4 │ 4 │
                      └───┴───┴───┴───┘

Complete Data Flow

Let’s trace through an entire U-Net forward pass:

INPUT:    (1,   3, 128, 128)   "RGB image"

ENCODER:
  enc1:   (1,  64,  64,  64)   → skip1 saved
  enc2:   (1, 128,  32,  32)   → skip2 saved
  enc3:   (1, 256,  16,  16)   → skip3 saved
  enc4:   (1, 512,   8,   8)   → skip4 saved

BOTTLENECK:
          (1, 512,   8,   8)   "Compressed understanding"

DECODER:
  dec4:   (1, 256,  16,  16)   ← uses skip4
  dec3:   (1, 128,  32,  32)   ← uses skip3
  dec2:   (1,  64,  64,  64)   ← uses skip2
  dec1:   (1,  64, 128, 128)   ← uses skip1

OUTPUT:   (1,   3, 128, 128)   "Processed image"

What Can U-Net Do?

U-Net is used for any task requiring pixel-level output:

When NOT to Use Decoder

Not all tasks need a decoder:

Classification (no decoder):
  Image → [shrink, shrink, shrink] → "This is a cat"

U-Net (full decoder):
  Image → [shrink] → [expand] → Processed image

If you only need a label, not a pixel-by-pixel output, skip the decoder.

Summary

U-Net’s power comes from three key ideas:

1. Encoder: Compress spatially, extract “what” is in the image
2. Decoder: Expand back to full resolution
3. Skip connections: Pass “where” information directly from encoder to decoder

This combination allows U-Net to understand both the big picture (global context from bottleneck) and fine details (local information from skips), producing sharp, accurate outputs.

Whether you’re segmenting medical images, generating art with Stable Diffusion, or building your own image editing model, U-Net’s elegant architecture is likely at the core.

This post was created while building a text-conditioned image editing model. The examples and diagrams come from hands-on experimentation with PyTorch.

Try the live demo! - The model runs entirely in your browser using ONNX Runtime Web.

The Architecture: Encoder-Decoder with Cross-Attention

Unlike the decoder-only transformer from my previous experiment, image captioning requires an encoder-decoder architecture. The key insight is that we need to process two different modalities (images and text) and connect them through cross-attention.

The architecture has two parallel paths:

Image Path (Blue): The image goes through patch embedding, then encoder self-attention layers. This produces “image features” — a sequence of patch embeddings that understand spatial relationships.

Text Path (Green): The caption tokens go through token embedding, then decoder layers with both self-attention (causal) and cross-attention to the image features.

The Bridge (Purple): Cross-attention is where the magic happens. It allows each text token to “look at” all image patches and gather relevant visual information.

From Pixels to Patches: The Vision Encoder

The first challenge is converting an image into something a transformer can process. Transformers work on sequences, but images are 2D grids. The solution: split the image into patches.

128x128 image → 16x16 grid of 8x8 patches → 256 patch embeddings

Each 8x8 patch contains 64 pixels × 3 colors = 192 values. A linear layer projects this to 128 dimensions:

class PatchEmbedding(nn.Module):
    def __init__(self, image_size, patch_size, n_embd):
        patch_dim = 3 * patch_size * patch_size  # 192
        self.proj = nn.Linear(patch_dim, n_embd)  # 192 → 128
        self.pos_embd = nn.Parameter(torch.randn(1, n_patches, n_embd))

    def forward(self, x):
        # Split image into patches, flatten, project
        patches = extract_patches(x)  # (B, 256, 192)
        return self.proj(patches) + self.pos_embd  # (B, 256, 128)

Now we have 256 “patch tokens” that can go through self-attention, just like text tokens. The encoder self-attention lets patches learn about each other — a patch showing a dog’s head can attend to patches showing its body and legs, building a coherent understanding of “dog”.

Cross-Attention: The Bridge Between Vision and Language

This is the key difference from text-only transformers. In self-attention, Q, K, and V all come from the same source. In cross-attention:

• Q (Query) comes from the text decoder: “What visual information do I need?”
• K, V (Key, Value) come from the image encoder: “Here’s what each patch contains”

class CrossAttention:
    def forward(self, text_embeddings, image_features):
        Q = text_embeddings @ W_q   # What am I looking for?
        K = image_features @ W_k    # What does each patch contain?
        V = image_features @ W_v    # What info to retrieve?

        scores = Q @ K.T  # (text_len, num_patches)
        weights = softmax(scores)
        return weights @ V  # Weighted sum of patch info

When generating the word “running”, the model learns to attend heavily to patches showing legs in motion. When generating “snow”, it attends to the white ground patches.

Training on Flickr8k

I used the Flickr8k dataset: 8,000 images with 5 human-written captions each. A key insight was using random caption sampling — each epoch, randomly select one of the 5 captions per image. This acts as data augmentation and dramatically reduces overfitting.

The random caption sampling closed the train-val gap from 0.80 to just 0.09.

Results: What the Model Learned

After 30 epochs of training (~17 minutes on M4 Mac), the model generates reasonable captions:

Success case:

Generated: "a black dog is running through the grass ."
Actual:    "A black dog running across green grass ."

Failure case:

Generated: "a man in a blue shirt is standing in the stree"
Actual:    "A crowd of people are enjoying a meal with a view of a mountaintop ."

The model handles simple scenes well (dogs, people, basic actions) but struggles with complex scenes (crowds, multiple objects, subtle context).

Model Statistics

Total parameters: ~980,000 (about 1M)

Breakdown:
- Patch embedding:     32,896 (3%)
- Encoder blocks (2):  395,776 (40%)
- Token embedding:     8,960 (1%)
- Position embedding:  6,144 (1%)
- Decoder blocks (2):  527,616 (54%)
- Output layer:        9,286 (1%)

The decoder is larger than the encoder because each decoder block has both self-attention AND cross-attention.

Key Learnings

1. Patches are the “tokenizer” for images

Just as we split text into tokens, we split images into patches. This converts the 2D spatial structure into a sequence that transformers can process. The same weight matrix processes every patch, learning a universal “patch reader”.

2. Cross-attention is the bridge

The key architectural difference from text-only transformers. It lets the text generation process “see” the image at every step, attending to relevant patches for each word being generated.

3. Data augmentation matters enormously

Using all 5 captions with random sampling was more impactful than doubling the image resolution. The model learns semantic concepts rather than memorizing specific strings.

4. Resolution limits understanding

At 128x128, a tricycle looks like a blob. The model can distinguish dogs from people, but struggles with fine details. Real vision models use 224x224 or higher.

5. This is still a toy model

Production image captioning models use:

• Pretrained vision encoders (CLIP, ViT trained on millions of images)
• Word-level tokenization (shorter sequences)
• Much larger datasets (COCO has 330k images)
• Billions of parameters

Improvement: Using Pretrained CLIP Encoder

After training the from-scratch model, I wanted to see how much a pretrained vision encoder could help. I created a second version that uses CLIP ViT-B/32 as a frozen image encoder, training only the decoder and a projection layer.

Architecture Changes

Instead of learning patch embeddings from scratch:

• CLIP’s pretrained ViT processes the image (224x224 input)
• 50 patch embeddings (768-dim) are projected to the decoder dimension
• Only the decoder (~3.8M params) is trained; CLIP (~87M params) is frozen

class CLIPCaptioningModel(nn.Module):
    def encode_image(self, img):
        # Use CLIP's visual transformer (frozen)
        with torch.no_grad():
            x = clip_model.visual(img)  # (B, 50, 768)
        return self.visual_proj(x)  # Project to decoder dim

Results Comparison

The CLIP-based model achieves 33% lower validation loss with fewer epochs!

Sample Captions

For the same test image (two dogs in snow):

The CLIP-based model produces more natural, concise captions. It benefits from CLIP having been trained on 400 million image-text pairs — it already understands visual concepts like “dogs” and “playing” without needing to learn them from our small 8k image dataset.

Testing on Complex Scenes

I tested both models on the validation set, focusing on complex scenes that the from-scratch model struggled with:

Key observations:

1. Better at groups/crowds — CLIP recognizes “group of people” much better than the from-scratch model which defaults to “a man”
2. Better semantic understanding — Recognizes concepts like “rock climbing”, “mountain”, “boat” that the small model misses entirely
3. Still struggles with fine details — Exact counts (two vs three people), specific activities (ice skating vs standing)
4. More robust to complex scenes — Doesn’t collapse to generic “man in blue shirt” for difficult images

The pretrained visual features give CLIP a huge advantage on scenes requiring real-world knowledge.

Tradeoff: Accuracy vs Size

The improved model is 363MB (vs 4MB), making it impractical for browser deployment. This is the classic accuracy-size tradeoff:

• From-scratch model: Smaller, deployable, but less accurate
• CLIP-based model: More accurate, but requires a large pretrained encoder

For production, you’d typically use the large model on a server, or apply techniques like knowledge distillation to compress it.

Improvement: Word-Level Tokenization

The character-level model processes “a black dog” as 11 tokens (including spaces). Word-level tokenization reduces this to just 3 tokens, making sequences shorter and potentially easier to learn.

Parameter Count Changes

Switching from character-level to word-level tokenization dramatically changes where the parameters live:

The vocabulary explodes from ~70 characters to ~4500 words, but sequences shrink from 48 characters to 20 words. The net effect: 2.2× more parameters, almost entirely in the embedding layers.

Results Comparison

Wait — the word-level loss is higher? This is actually expected:

1. Loss is per-token: Character-level predicts from 70 options; word-level predicts from 4,453 options
2. Different scales: A word-level loss of 2.98 means perplexity ~20 (choosing from 4453 words), while character loss 0.99 means perplexity ~2.7 (choosing from 70 chars)
3. The captions are similar quality despite the different loss values

Sample Caption

For the same test image (two dogs in snow):

The word-level model produces fluent captions but with a smaller effective vocabulary (it saw each word fewer times during training than character-level saw each character).

Key Insight: Vocabulary Size vs Training Data

Word-level tokenization works better when you have lots of training data. With only 8k images:

• Character-level sees each character thousands of times → learns robust patterns
• Word-level sees many words only a few times → harder to learn good embeddings

This is why production models use:

• Subword tokenization (BPE, WordPiece): Best of both worlds
• Much larger datasets: COCO (330k), Conceptual Captions (3M+)
• Pretrained word embeddings: GloVe, Word2Vec, etc.

Improvement: CLIP + GloVe Pretrained Embeddings

Since the word-level model struggled with limited training data, I tried combining the best of both worlds: CLIP’s pretrained vision encoder with GloVe pretrained word embeddings.

The Idea

Instead of learning word embeddings from scratch with only 8k images, why not use GloVe embeddings trained on 6 billion words? This gives the model a head start on understanding word relationships.

class CLIPGloVeCaptioningModel(nn.Module):
    def __init__(self, vocab_size, clip_model, glove_embeddings, ...):
        # Use CLIP for vision (frozen)
        self.clip_model = clip_model

        # Use GloVe for word embeddings (fine-tuned)
        self.token_embed = nn.Embedding(vocab_size, glove_dim)
        self.token_embed.weight.data.copy_(glove_embeddings)

        # Project GloVe dim (100) to decoder dim (256)
        self.glove_proj = nn.Linear(glove_dim, n_embd)

GloVe Coverage

Using GloVe 6B 100d (100-dimensional embeddings trained on 6 billion tokens):

• 4441 out of 4517 words (98.3%) found in GloVe
• Only 76 words missing (mostly rare or domain-specific terms)
• Missing words initialized with small random values

Results

The GloVe embeddings give a 14% improvement in validation loss!

Sample Caption

For the same test image (two dogs in snow):

The GloVe model correctly identifies “two dogs” rather than “a dog”, suggesting the pretrained embeddings help with understanding quantities and relationships.

Key Insight: Transfer Learning Stacks

This experiment shows that transfer learning compounds:

1. CLIP brings pretrained visual understanding (400M image-text pairs)
2. GloVe brings pretrained word relationships (6B tokens)
3. Only the decoder and projection layers need to learn task-specific mappings

Even with just 8k training images, combining two pretrained components achieves significantly better results than training from scratch.

What’s Next

Remaining improvements to explore:

1. Pretrained vision encoder: Use CLIP or ViT instead of learning from scratch ✅ Done!
2. Word-level tokenization: “a black dog” as 3 tokens instead of 11 characters ✅ Done!
3. Pretrained word embeddings: Use GloVe for better word representations ✅ Done!
4. Subword tokenization: Use BPE for better vocab coverage
5. More data: COCO dataset (330k images) instead of Flickr8k (8k)
6. Knowledge distillation: Train a small model to mimic the CLIP-based one

But even the minimal from-scratch implementation demonstrates the core concepts: patch embeddings, encoder-decoder architecture, and cross-attention as the bridge between vision and language.

Code

The complete training script is available in my learn-llm repository as train-image-caption.py.

Preparation: Standing on the Shoulders of Giants

Before diving into code, I spent time building intuition through two excellent resources:

“Build a Large Language Model (From Scratch)” by Sebastian Raschka was my theoretical foundation. The book walks through every component of a transformer with clear explanations and diagrams. Reading it gave me a mental model of how attention, embeddings, and layer normalization fit together — knowledge that proved essential when debugging my own implementation.

Andrej Karpathy’s YouTube series (Neural Networks: Zero to Hero) was equally valuable. His “Let’s build GPT” video demystified the architecture by building it live on screen. Watching someone think through the design decisions — why we use residual connections, how attention matrices work, what LayerNorm actually does — made the concepts stick in a way that reading alone couldn’t. His makemore repository became the dataset and benchmark for my experiments.

With this foundation, I was ready to build.

The Experiment

I incrementally built a character-level transformer for name generation. Each step adds one architectural improvement. All models were trained with batch size 32, AdamW optimizer, and per-name padding with masked loss.

Results - Architecture Comparison (5,000 steps)

Results - Scaling Up

Makemore’s default transformer achieves ~1.92 test loss with N_EMBD=64, 4 heads, 4 layers.

Generated Names

Sample outputs from the final model (N_EMBD=64, 4 heads, 20k steps with all training improvements):

kaelynn, aileigh, elyce, yadi, ovani, derella, nyailee, ranyah, niaa, sett

Key Findings

Depth beats width

Doubling embedding size from 32 to 64 (3x params) gave almost no improvement (2.35 -> 2.34). Adding a second attention head with fewer total params (5,948 vs 8,860) dropped loss by 0.12. Stacking 4 layers was the single biggest improvement, dropping test loss from 2.23 to 2.04. The model benefits far more from multiple layers of processing than from wider representations at a single layer.

Data handling matters most

Before adding per-name padding, our best model achieved 2.36 test loss. After switching to per-name padding with masked loss (same architecture), it dropped to 1.94. This was a larger improvement than all architectural changes combined. The reason: without padding, the model wasted capacity trying to predict across name boundaries — an impossible task that added noise to every gradient update.

MLP adds capacity but needs regularization

Adding the feed-forward network (MLP) to each layer tripled the parameter count (18k -> 52k) but only modestly improved results. It also widened the train-test gap (2.00/2.04 -> 1.97/2.02), suggesting mild overfitting. The MLP lets the model transform representations nonlinearly after attention gathers information, but at this small scale the effect is limited.

LayerNorm and RoPE help incrementally

LayerNorm stabilized training and closed the train-test gap slightly. RoPE (Rotary Position Embeddings) gave the model awareness of character positions without adding any parameters. Neither was dramatic at this scale, but both are essential for larger models — LayerNorm enables training deep networks, and RoPE enables generalization to longer sequences.

GELU vs ReLU is negligible at small scale

Switching from ReLU to GELU activation in the MLP had no measurable effect. The smoother gradient flow matters more when networks are deeper and wider.

Scaling up helps significantly

Doubling N_EMBD to 64 and using 4 heads (matching makemore’s architecture) dropped test loss from 1.94 to 1.92 at 5k steps. With longer training (20k steps), the model reached 1.85 test loss — surpassing makemore’s default.

Dropout trades speed for generalization

Adding 20% dropout increased the train-test gap initially and slowed convergence. At 5k steps, it actually hurt test loss (1.92 -> 2.00). But it prevents overfitting during longer training runs, allowing the model to keep improving past where it would otherwise plateau.

Training improvements compound

Learning rate scheduling (warmup + cosine decay), weight decay (0.01), and gradient clipping (max_norm=1.0) together produced smoother training curves. The cosine decay prevents the learning rate from being too high in later steps when fine-tuning. Weight decay acts as regularization. Gradient clipping prevents instability from occasional large gradients.

Architecture Summary

The final model is a proper transformer decoder:

Input tokens
    -> Token Embedding (28 vocab -> 64 dim)
    -> 4x Transformer Blocks:
        -> LayerNorm -> Multi-Head Attention (4 heads, RoPE, dropout) -> Residual
        -> LayerNorm -> MLP (64 -> 256 -> 64, GELU, dropout) -> Residual
    -> Linear (64 -> 28 vocab)
    -> Cross-entropy loss (masked on PAD tokens)

Training config:

• 20,000 steps
• Batch size 32
• AdamW optimizer with weight decay 0.01
• Learning rate: warmup to 1e-3 over 200 steps, cosine decay to 1e-4
• Gradient clipping: max_norm=1.0
• Dropout: 0.2

What the Loss Means

A loss of 1.86 means the model assigns ~15.6% probability on average to the correct next character (e^(-1.86)). Random guessing over 27 characters would give ~3.7% (loss = 3.30). Perfect prediction is impossible because many positions are genuinely ambiguous — after “ma”, the next character could be r, d, k, x, t, and many others.

Progress through this project:

• Start: 2.35 test loss (~9.5% confidence)
• Final: 1.86 test loss (~15.6% confidence)
• Improvement: ~1.6x more confident on the correct character

Conclusion

Building a transformer incrementally taught me that the magic isn’t in any single component — it’s in how they work together. Data preprocessing had the biggest impact. Depth mattered more than width. And the “modern” improvements (LayerNorm, RoPE, GELU) are less about dramatic gains and more about enabling scale.

I recently went down a rabbit hole reverse-engineering this “protection” mechanism in Guitar Pro 8. What I found was a classic case of “security through obscurity” — and not very deep obscurity at that.

The Problem

Guitar Pro has a feature to “lock” a file. When locked, the file can be opened and played, but the editing features are disabled. If you peek inside the .gp file (which is just a ZIP archive), you’ll see a few interesting things:

1. A file named editLocked.
2. The main content Content/score.gpif is encrypted (it doesn’t have the standard XML header).

Removing editLocked isn’t enough. The app sees it’s missing, but the content remains encrypted and unreadable.

The Breakthrough

As Guitar Pro can open and play the file without ever prompting for a password, it was clear that the key to decrypt the content must be available to the application without user input. This realization led me to investigate how the application handles these files internally.

I analyzed the GuitarPro binary and its libraries, specifically libGPIO.dylib.

1. The Salt

Deep in the binary, I found a reference to a static salt used in the encryption routine. da40cc64900b617a0f72ad4e6ef42f9c

2. The Password

Tracing the assembly code for Score::setLockPwd, I found something surprising. The application reads the entire content of the editLocked file (which contains a salt and a hash of the user’s original password) and sets that string as the internal password for decryption.

So, the “password” to decrypt audio and score data isn’t what you typed. It’s the metadata file itself.

The Solution

Putting it all together, the encryption scheme is:

• Algorithm: AES-256-CBC
• Key Derivation: PBKDF2-HMAC-SHA1 (4096 iterations)
• Password: The content of editLocked (e.g., salt$hash)
• Salt: The static binary salt (da40cc...)

With this information, I wrote a Python script unlock_score.py that fully unlocks these files.

The Script

Here is the core logic of the unlocker:

STATIC_SALT_HEX = "da40cc64900b617a0f72ad4e6ef42f9c"

def decrypt_gpif(encrypted_data, password):
    salt = binascii.unhexlify(STATIC_SALT_HEX)
    # PBKDF2 with 4096 iterations
    key = hashlib.pbkdf2_hmac("sha1", password.encode(), salt, 4096, 32)
    
    iv = encrypted_data[:16]
    ciphertext = encrypted_data[16:]
    
    cipher = Cipher(algorithms.AES(key), modes.CBC(iv), backend=default_backend())
    decryptor = cipher.decryptor()
    decrypted = decryptor.update(ciphertext) + decryptor.finalize()
    
    # Decompress zlib payload
    return zlib.decompress(decrypted)

You can find the full tool on GitHub Gist.

The Role of LLMs in Reverse Engineering

A fascinating part of this project was using an LLM to accelerate the reverse engineering process. While tools like otool and grep provided the raw data, the AI acted as a “force multiplier”:

• Reading Code at Scale: The most daunting part of reverse engineering is the sheer volume of information. A binary dump can contain millions of lines of assembly instructions. For a human, “reading” this to build a mental model of the software’s behavior is a task that takes days or weeks. The LLM, however, could digest these massive text dumps instantly.
• Semantic Understanding: It didn’t just match patterns; it understood the intent of the low-level code. By analyzing the context around function calls (like AES_encrypt or setLockPwd), the AI could infer high-level logic—such as identifying that the password was being sourced from file metadata—without us having to manually trace every register.
• Time Compression: This ability to essentially “read” the binary allowed us to bypass the tedious manual tracing phase entirely. We could ask high-level questions about the software’s behavior and get answers derived from the raw assembly, compressing what would be an “forever” task for a human into a quick conversation.

This collaboration turned what could have been a multi-day debugging session into a targeted, systematic investigation.

Conclusion

This exercise showed that the “lock” feature in Guitar Pro is effectively just a UI flag backed by a fixed-key obfuscation. It prevents casual editing but offers no real security against someone determined to access the data.

Disclaimer: This information is for educational purposes only. Always respect copyright and the wishes of content creators.

Cross Gate (魔力宝贝) was one of the most influential MMORPGs in Taiwan and China during the early 2000s. As someone who spent countless hours collecting pets in this game during my childhood, I recently embarked on a nostalgia-driven project: extracting all the pet sprites from the game files and building a modern web viewer to browse them.

The Challenge

Game resources from the early 2000s are notoriously difficult to work with. Cross Gate uses proprietary binary formats for its graphics and animation data:

• GraphicInfo_*.bin (40 bytes per entry) - Metadata for each graphic including dimensions, offsets, and addresses
• Graphic_*.bin - RLE-compressed 8-bit indexed images with transparency
• AnimeInfo_*.bin (12 bytes per entry) - Animation metadata linking pet IDs to frame sequences
• Anime_*.bin - Animation frame data with actions and directions
• Palette files (.cgp) - 224-color palettes mapping indices 16-239

The compression format is a custom RLE implementation with multiple encoding modes (literal, repeat, transparent) and variable-length counters.

The Solution

Using AI-assisted development (Claude Code and Antigravity), I built a Python extraction pipeline:

1. Parse the binary formats - Read the structured binary files, extracting metadata and addresses
2. Decompress RLE graphics - Implement the full RLE decompression algorithm with all encoding modes
3. Apply palettes - Map 8-bit indexed pixels to RGB colors using the game’s palette files
4. Generate animated GIFs - Combine frames into animated GIFs for each pet’s actions and directions

Each pet has up to 10 actions (Idle, Walk, Attack, Defend, Cast, etc.) and 8 directions, resulting in potentially 80 GIF animations per pet.

The Frontend

I built a Next.js web application to browse the extracted pets:

• Grid view displaying all available pets
• Detail view with interactive controls for actions and directions
• Drag-to-rotate functionality for intuitive direction changes
• Pixel-perfect rendering with image-rendering: pixelated to preserve the retro aesthetic

Lessons Learned

1. Binary format reverse engineering is time-consuming - Even with AI assistance, understanding undocumented binary formats requires careful experimentation and validation
2. Progress persistence is essential - With 1000+ pets to process, the batch generator needed to skip already-processed pets and handle timeouts gracefully
3. Test with edge cases early - Some pets had unusual frame counts or missing animations that caused the initial implementation to fail

References

This project was made possible by the cgg-viewer project, which provided the foundational understanding of Cross Gate’s binary file formats and RLE decompression algorithm. The original Python implementation by the cgg-viewer author was invaluable for understanding how to correctly parse GraphicInfo, AnimeInfo, and palette files.

What’s Next

• Try Tencent Hunyuan 3D to convert 2D sprites into 3D models

You can try it out at https://1203906e.cross-gate-pets.pages.dev/.

The Problem

Evernote had been my digital brain since the late 2000s. But with each passing version, the app became slower, more bloated, and increasingly expensive. Apple Notes, meanwhile, has quietly evolved into a capable, fast, and free alternative that syncs seamlessly across my devices.

The catch? There’s no official migration path. Evernote’s export format (ENEX) doesn’t preserve everything, and Apple Notes doesn’t have any bulk import feature. Manual copy-paste wasn’t an option.

So I built my own migration tool.

What Made This Hard

This wasn’t a simple file conversion:

• Rich text formatting including tables, checklists, and styled text
• Embedded attachments (images, PDFs, documents) referenced by MD5 hashes in Evernote’s proprietary ENML format
• Creation and modification dates that needed to be preserved
• Duplicate detection to allow resumable, interruptible migrations
• Apple Notes’ limitations—no public API, only AppleScript access

Evernote v10 made things even more complicated. Unlike older versions that stored everything in a straightforward SQLite database, v10 uses a hybrid system with:

• A SQLite database for metadata
• Separate .dat files containing rich text content (tables/formatting)
• Protobuf-encoded binary structures
• Server-side attachment storage requiring authenticated downloads

The Solution: A Two-Phase Migration System

I built a Python-based migration pipeline that handles all of this complexity.

Phase 1: Parallel Preparation

The first phase downloads attachments and generates PDFs in parallel using 10 worker threads. For notes with embedded images or files, I render the complete content (HTML + attachments) into a PDF using headless Chrome. This preserves formatting perfectly.

Phase 2: Sequential Import

The second phase imports to Apple Notes via AppleScript—sequentially, because Apple Notes doesn’t handle concurrent modifications well.

Solving the Attachment Problem

Evernote embeds attachments using <en-media> tags with MD5 hashes. To resolve these to actual files, I:

1. Query Evernote’s local database for attachment metadata
2. Download from Evernote’s servers using captured auth tokens
3. Embed them as base64 in generated PDFs
4. Attach the PDF to the Apple Notes entry

Deduplication Done Right

My initial attempt at duplicate detection was fragile—comparing dates via AppleScript often failed. The fix was simple: track Evernote note IDs in a log file. This makes the migration fully resumable.

Bonus: AI-Powered Organization

Once notes were in Apple Notes, I used Gemini AI to automatically categorize them into folders based on content.

Lessons Learned

1. AppleScript is slow but reliable — Building a cache at startup dropped duplicate checks from 0.5s to 0.001s per note.

2. Parallelism for I/O, sequential for mutations — Downloading attachments scales linearly with workers. Writing to Apple Notes must be sequential.

3. Auth tokens expire — Evernote’s tokens last about an hour. I kept Proxyman ready to capture fresh tokens.

4. PDF is a universal container — When your target doesn’t support rich formatting or attachments, bundle everything into a PDF.

The Code

The entire migration toolkit is available on GitHub: apple-notes-toolkit

⚠️ Note: This repo is fully vibe coded. Use with caution.

Final Thoughts

What started as a weekend project turned into a deep dive into Evernote’s internals, Apple’s Scripting Bridge, and the art of data migration. But the result is worth it: my 15 years of notes are now in Apple Notes, fully searchable, syncing across devices, and—most importantly—mine to keep.

If you’re considering leaving Evernote, know that it’s possible. It just takes a bit of engineering.

• 区里提供的馆址是个地下空间，需要在有限的预算内，找到合适的装修公司，把这个地下空间改造成舒适的阅读空间。
• 在图书采购过程中，供应商惯于提供劣质的，滥竽充数的图书，为采购者支付回扣。作者不屑于收受回扣，一心为公，希望图书馆里都是经历了时间检验的好书。
• 为一个图书馆选书，工程浩大，无法仅凭一己之力完成。作者发动自己的人脉，联系了诸多好友帮忙选书。选书缘由，荐者心路，作者缓缓道来，推卷而述，好不痛快。

尽管困难重重，作者心有所往，逆流而上，不畏险阻，最终得偿所愿。主线之余，作者夹叙一年挂职生活所遇的形色人等，有的让人牙关紧咬，有的让人唏嘘感慨，说尽人情冷暖。西安的美食，官场中的本色不改，选书朋友们的人生故事，对弱势群体的关照，五味杂陈，乒乓作响，读者吃到的是酸辣爽口的一餐。

附录里的书单

童书（含漫画）

文学类

人文社科

自然科学

其他系列书

《纳瓦尔宝典》提及书籍与博客索引（含博客链接）

以下列表依照在《The Almanack of Naval Ravikant》中首次出现顺序整理，并补充中文译名及 Naval 的一句话点评。博客及博文已附可点击链接。

I tried Claude Code this week, and instantly felt the empowerment from the tool, and was stunned by how naturally it blends into developer workflows.

It demonstrated how easy the LLM model makers can disrupt the application makers (Cursor in this case). This reminds me of the analogy Andrej Karpathy made in Software Is Changing (Again) presentation that LLM has strong analogies to operating systems. The LLM model makers can easily disrupt app makers like Apple can sherlock other softwares running on top of macOS.

With a similar tool from Google called Gemini CLI released, I begin to question about what is the main complexity Claude Code has, and whether that complexity is challenging enough to support companies relying on building agentic tools.

I found the following video where Boris Cherny (who is the creator of Claude Code) answered my first question:

Audience: I was wondering what was the hardest implementation, like part of the implementation for you of building it?

Boris: I think there’s a lot of tricky parts. I think one part that is tricky is the things that we do to make bash commands safe. Bash is inherently pretty dangerous and it can change system state in unexpected ways. But at the same time, if you have to manually approve every single bash command, it’s super annoying as an engineer.

Boris: … the thing we landed on is there’s commands that are read-only, there’s static analysis that we do in order to figure out which commands can be combined in safe ways, and then we have this pretty complex tiered permission system so that you can allow list and block list commands at different levels.

This highlights a key insight: In agentic systems, safety isn’t an afterthought—it’s the core challenge.

How do we know if a command is safe to run? How can these tools predict the consequences of an action? Currently, the burden is shifted to the developer via permission dialogs. But eventually, developers will expect these tools to act more autonomously—without compromising safety.

For commands that only affect local environments, Docker might offer a partial solution. But many real-world use cases involve remote effects—like modifying a task in Linear or changing a GitHub label. These remote side effects raise thorny questions about trust, auditability, and failure handling.

After exploring Claude Code and Gemini CLI, I’m excited about where this space is headed. The next breakthroughs may come not just from smarter agents—but from safer ones.

– EOF –

为了增加用户活跃度，微信读书团队开发了一个微信小游戏——问答 PK。这是一个双人对决形式的知识问答天梯，题目内容主要基于常识，比如成语填字，古诗词接上下句。

玩了几天后发现，光靠我的知识储备和记忆力，很难持续提升段位。答案在网上一搜就能找到，但是 10 秒钟的答题时间来不及搜索，于是我想到借助 DeepSeek 来自动答题。说干就干，Vide-Coding 了一个 Python 脚本，自动化了整个答题过程，并最终达到了最高等级。本文记录在开发过程中，遇到的问题与一些观察。

技术难点与观察

OCR 错误率导致的复杂度

我首先想到的是将窗口截图转为文字，这一步涉及图片到文字的模态转换：

• macOS 自带的 OCR 中文识别准确率并不完美。有些中文字符在不同帧中会被错误识别为相似字形。
  • 为了判断题目是否更新，程序需要实现较复杂的题目刷新检测逻辑。
  • 在存储与提取已答题目上，也因此增加了额外复杂度。
• 后来想到可以利用 macOS 的 Accessibility API 来获取小程序窗口的文字信息，实现起来就简单多了。
• 结论：
  • 如果可以获取文本内容，应当优先使用文本内容，尽量避免不必要的复杂度。
  • 第一个想到的方法不一定是最好的方法，实现之前可以再多花一点时间比较一下其他方法。

反馈机制的设计

LLM 并不能保证每道题都能准确回答，因此，需要设计一种反馈机制，用于处理错误回答，并逐步提高系统表现：

• 每次答题后，程序会记录实际答案与 LLM 输出是否一致。
• 若识别为错误，会将题目及正确答案保存进本地题库，供后续匹配使用。
• 随着题库积累，LLM 的回答可以逐步退居辅助角色，以“已知题目匹配”为主、生成式回答为辅。
• 在实践中，这种混合策略显著提高了答题准确率，也使系统更加可控。

工具效率与资源消耗

这类依赖模态转换和实时反馈的程序在效率上也面临挑战，尤其当一方发生变化、但未提供明确的推送机制时，工具只能通过“轮询”方式不断查询变化状态：

• 本例中，为了判断题目是否已经刷新，程序只能定期抓取小程序里的文字内容，并比对，轮询带来了显著的资源消耗。这种“拉取式”的检测逻辑效率低下，不适合长期运行。
• 本质上，这类问题的根源在于缺乏变化触发的事件通知机制。如果 macOS 或目标应用能提供“题目变动事件”的观察接口，将显著提高系统效率。期待苹果在接下来的几年持续进化 macOS 来帮助第三方软件加入更多 AI 驱动的功能。
• 实现的过程中用到了 MacPaw 开源的 macapptree 来抓取应用的 Accessibility Tree。估计 MacPaw 团队在开发 Envy 的 actions 也依赖 Accessibility API 来实现各种软件的自动化。
• 结论：在系统设计中，应尽量选择或构建具备事件驱动机制的组件，避免盲目轮询所带来的能耗与复杂度。

Vibe-Coding

作为一个 Weekend Fun Project，没有 Vibe-Coding，我无论如何也无法在两三天里快速迭代实现各种预想中的功能，修复各种 bug，并最终把程序跑起来，自动化整个答题过程的。不得不说，有了 Cursor 以后，没有办法回到一行一行写代码的日子了。Vibe-Coding is fun and the future for everyone。

– EOF –

Sundar Pichai views “moonshot” projects as crucial for several reasons:

• Driving Innovation: He believes that aiming for audacious, seemingly impossible goals, like the original moon landing, forces radical rethinking and leads to breakthroughs that wouldn’t happen with incremental improvements. It’s about finding “10X” improvements rather than “10 percent” improvements.
• Inspiring Talent and Passion: Big, challenging problems ignite both the hearts and minds of people. It’s easier to attract passionate and talented individuals to work on projects that could redefine humanity.
• Societal Impact: Moonshots, even if their initial goal is not fully realized, can lead to numerous technological advancements with real-world applications and inspire future generations. For example, Google considers fighting climate change as a “moonshot” due to its profound societal importance.
• Leveraging Constraints: Pichai has also highlighted that constraints can act as catalysts for innovation. Working within defined limits encourages teams to be more creative and focused, leading to groundbreaking ideas.

To monitor our baby from other rooms, we purchased a Nanit Baby Monitor. Using image recognition, Nanit provides insights into our baby’s nighttime sleep patterns through its app. Each state transition point includes a video for review.

However, the display isn’t very intuitive — the chart doesn’t show the exact timestamps for each transition. For example, the start and end times of the two longer sleep sessions are not clearly marked.

To more intuitively view this information and more flexibly display the baby’s sleep duration and time periods throughout the night, I used Cursor and video-coding to build a Web App:

• Fetch data from Nanit API for any given date
• Render sleep sessions throughout the day
• Plot sleeping trend of most recent dates

Lessons learnt:

• Think through the main features and their designs you want before code generation with Cursor.
  • Although LLM can generate code for you. You would still need to think through what are the features you have in mind, and what things would look like (the design).
  • This reminds me how Firebase Studio is trying to help build a PRD (Product Requirements Document) before beginning to generate code.
  • Remind me apps like https://stitch.withgoogle.com/
• Think about testing if you would like to have some code maintainability.
  • Fully AI generated code without any review and test is not maintainable.
  • As a weekend project to meet myself’s requirements, I didn’t put much effort into how to make it maintainable.
  • I feel the joy of vibe coding goes down slowly when I put more features to it as new changes could break existing features.
    • I probably should add some end-to-end tests to make sure that new changes won’t break existing features. However, I didn’t figure out how to put tests in the iteration loop in Cursor yet.
• Tighter development loop and more agentic behaviors are needed.
  • Cursor stops itself frequently even with agent mode to ask for all kinds of inputs:
    • human input (confirmation, or opinion on design choices)
    • app console output
  • For the human input, I found myself becoming the bottleneck for it to do more useful things. When it’s waiting for some input, I wish it would begin working on other parts which don’t require human input.
  • For the app console output, I wish it has a tighter loop so that I don’t need to copy console output from Chrome DevTools back to Cursor. (Maybe Chrome could provide something to close the loop here?)
• Analyzing images through AI generated code doesn’t work.
  • As Nanit doesn’t provide a way to export data, I was trying to use app screenshots to parse the sleep information (which is challenging for me to code manually), and it turns out that the current AI models cannot do that as well even with dozens of prompts back and forth.
  • I ended up using Proxyman to capture HTTPs requests and responses from the Nanit app to understand the API, and calling that directly from Python.
    • Used some go code from https://github.com/gregory-m/nanit in the prompt to help LLM to implement the authentication part.

世界上大部分的问题悬而未决，
小部分我们以为的答案，
也随着时间的推移不断演变。

观点就像流过身体的水，
并不属于某一个人。

保持质疑一切的态度，
保持开放，
倾听不同的观点。

听到一个观点之后，
不急着相信或者否定，
而是尝试理解观点背后的事实与逻辑，
然后再做出独立的判断。

做好随时修正持有观点的准备，
因为对事实的认知会改变，
行动之后也会得到了更多的事实。

论辩不是为了输赢，
而是共同探索不同观点的根源，
是价值排序的不同，
还是你我看见了不同局部的世界。

放下偏见和自傲，
做一个理智，独立思考的人。

语音输入文字本身并不是什么新鲜的功能，但就像 iPhone 键盘 的诞生一样，它背后仿佛存在着一道无形的界限——在跨越这道界限之前，一切都显得繁琐笨重；而一旦突破，用户才能真正感受到那种 Magic Moment，仿佛一切变得自然、顺畅，甚至有些神奇。

这本书前两章系统地整理了分析问题的逻辑框架和常见的逻辑谬误，对于如何提高思辨能力能有帮助。第三章辩论实战部分讲如何应用在辩论中，对于不直接参与辩论的读者不如前两章实用。

塑造理论的整体结构（第二章的内容）

• MECE（Mutually Exclusive, Collectively Exhaustive）
  • 定义：相互独立、完全穷尽。这些点与点彼此不重合，叫相互独立；它们加在一起能够完整地覆盖对这个问题的分析，叫完全穷尽。

MECE这个概念对我比较有启发，工作中的一些讨论缺乏对问题的总体上的思考。

• 明确定义是讨论的开始
  • 明确定义，达成共识，挖掘更深洞见
• 有标准，才有意义
  • 比较标准是建立论证的关键因素
    • 比较标准的公开是建立共识的前提。选择辩论队员上场的例子。
    • 检视标准是发现分歧、明确重点的方式
    • 比较标准的反驳：有效性、合理性与归谬反驳
  • 明确比较标准：洞悉底层价值，引导决策方向
• 权衡价值与利益的“需根解损”
  • 政策性辩论/价值性辩论
  • 需根解损是政策性辩论的分析框架
  • 概念：
    • 需求：可以是问题导向（空气污染）、利益导向（更好的工作）或目标导向（更文明的社会）。
    • 根属性指的是之所以会存在这个需求，其根本原因是什么。
    • 解决力，也就是这个政策解决问题的效果。包含可行性和效果。
    • 损益比，比比落实这个政策带来的好处和它产生的弊害，划算吗？
      • 量化和补救措施

需根解损这个概念我是在这本书里第一次了解到。工作当中常用的一个决定项目优先级的框架和这个有相似之处：RICE（Rich，Impact，Confidence，Effort）。

• 没有绝对共识，但可比较利弊
  • 为生命提供避风VS助长遗弃之风
  • 利弊比较：让思考完整清晰，但没有绝对真理。利弊比较往往涉及到价值排序，所以因人而异，难以有绝对的共识。
    • 利可否被替代，弊可否被规避
    • 寻找同一标尺，平行比较利弊
    • 看清事件本质，用价值排序判断利弊
• 不说废话，从“决胜点意识”开始
  • 辩论中的决胜点意识：对，但为什么更对
  • 明确目的，达成目的
• 论证观点，检视自己
  • 如何论证论点？
  • 论证的三个部分：逻辑、事实、价值
• 论证强度与论证责任
  • 演绎论证：前提真实，逻辑有效，结果必然
  • 归纳论证：结论超越前提，缺乏绝对有效性
    • 归纳论证的4种方法：摆事实、举数据、讲机理、举例子、引用权威理论。
  • 论证强度：标准不统一，视损益比而定
  • 比例原则：论证强度与对应行为成比例
  • 推定利益与举证责任
• 让道理听得进去　用例子完善逻辑，用故事锦上添花
  • 说服力

常见的逻辑谬误（第一章的内容）

• 相关不等于因果。“错把相关当因果”是我们生活和工作中最常见的逻辑错误之一。
  • 因果倒置
  • C同时带来A和B
  • 如何克服：尝试反向思维，对于观点A带来B，B带来A成立吗；控制变量；
• 实然不能论证应然。
  • 概念理解

    如何理解“落后就要挨打”这句话？第一种理解：落后时更容易有人来欺负我。第二种理解：如果我落后了，别人打我、欺负我无可厚非，落后的人和国家就应该被欺负，甚至被消灭。两种不同的理解分别对应了两个概念——实然和应然。实然，descriptive，是指对现实的描述；应然，normative，讨论的是什么是应该的、好的、对的、值得追求的。这一讲我们就来区分这两个概念。

  • 区分实然还是应然

    门当户对是否过时？实然层面，只需要做问卷调查。应然层面，探讨现代人应不应该还在乎门当户对。

  • 实然不能论证应然
    • 对于存在即合理的误读。“合理”在黑格尔的原意是：凡是现实的都是有原因的、可被归因的、有迹可循的。
  • 实然是对真实世界的确认，属于求真。应然是道德层面上的讨论。
• 行，但不对

  行不行，指的是这些行为能否实现行为实施者的功利性目的；对不对，指的是这些行为在道德上是不是正义的和应该做的。

  • 营救式刑求，行不行？
    • 用一个例子来讲解功利道德观和道德绝对主义道德观之间的交锋。
  • 利弊权衡中隐藏的风险与危机
    • 功利主义更容易被认同。
    • 如何推广这种绝对的道德观？创造某种沉浸式的体验；论证为什么持这样的道德观的世界会更好，这有些类似在功利主义角度去辩驳，杀一也许不能救百。
• 滑坡是谬误，也是合理质疑
  • 滑坡谬误：一环连着一环的不成立
    • 如果A发生，B就会发生；如果B发生，C就会发生。后半部分是真的吗？
  • 关于同性婚姻的滑坡论证：有效自愿与道德原则

    文明社会的两条最基本的行为原则是自愿和对他人无害。

    • 根属性。一件事情发生的根属性是什么？利于艾滋病传播的因素不见得根属于同性性行为，而是根属于未受保护的同性性行为。
• 平等与正义之间隔着一个公平
  • Equity illustration：
    • Equality（平等）：每个人都被给予相同的资源或机会，但由于本身的差异，有人仍然无法受益。
    • Equity（公平）：资源分配考虑到了个人的具体需要，使每个人都能达到同样的成果。
    • Justice（正义）：通过系统性的改革（如移除障碍），让每个人都不再需要额外的帮助就能获得平等的机会。
  • 《平权法案》（Affirmative action）
  • 目标状态，起始状态和过渡状态
• 三段论里的不证自明
  • 引子
    • 大前提：人活着是好事。
    • 小前提：我的伴侣是人。
    • 结论：复活伴侣是好事。
  • 什么是三段论？
    • 大前提，小前提和结论。包含关系。
    • 演绎论证（Deductive Argument）和归纳论证（Inductive Arument）
• 不是所有分歧都叫偷换概念
  • 什么是偷换概念？
    • 偷换概念指的是在同一思维过程中，用一个概念代替另一不同的概念，也就是说，同样的词或短语在同一个论证逻辑中，第一次和第二次出现时表面意思相同，但是实际上却是两个不同的概念，它违反了同一律要求，从而造成逻辑错误。
  • 类比不当不是偷换概念
    • “人不可能伤害自己的孩子，因为虎毒还不食子呢！”你可以反驳我这句话是类比不当，因为人和老虎在对待孩子的恶毒程度上不见得足够相似，或者说人类世界的复杂程度要远远高于动物世界的复杂程度。但这并不是偷换概念。
• 稻草人谬误与红鲱鱼谬误

  在辩论中故意把对方的观点曲解为一个更容易反驳的版本然后对其反驳并觉得自己赢了，这就是稻草人谬误。

  • 稻草人谬误是对观点复杂性的粗暴简化
  • 如何反驳稻草人谬误：忠实原则与宽容原则

    所谓忠实原则，是当对方表达观点后，我们要尽可能按照他的本意去理解、去复述、去反驳，而不是编造出另一个不符合他本意的东西。 所谓宽容原则，是将疑点、利益归于提出观点的人，尽可能使他的论证有说服力。当然这也要在忠于他的原意的前提下。 在这样的前提下，我们反驳的才是这个观点，否则反驳的只是另一个概念，或者我们战胜的只是对方一时没说清的失误而已。

  • 红鲱鱼谬误（Red Herring Fallacy）
    • 比喻那些为了让人分散注意力而提出的不相干的观点甚至是错误信息。
  • 如何反驳红鲱鱼谬误：识别被转移的焦点
• 样本偏误不可信
  • 幸存者偏差：无视“牺牲者”的数据谬误
    • Survivorship bias
  • 选择偏差：具有倾向性的样本无法代表总体全貌
    • Selection bias：选择样本是不是随机的
  • 自选择偏差：主体自我选择带有的特征会影响因果关系的判定
    • Self-selection bias
    • 离婚律师分析离婚
  • 参与偏差（无反应偏差）
    • Non-response bias
  • 条件概率：收集信息，理解自己
    • Conditionial Probability
• 回避论证过程的循环谬误（Begging the question）
  • 好马不吃回头草，因为吃回头草的不是好马。
• 进退两难也许只是假象
  • 什么是虚假两难？虚假两难也称非黑即白，指的是在本来有其他选项的情况下，却要求人们做出非此即彼的选择。
  • 光谱思维
• 人身攻击无法论证观点
  • 人身攻击谬误
  • 诉诸权威谬误
• 充分吗？必要吗？
  • 比较级：面对有限现状，量化最优选项
    • 我最求高尚，但是不追求更高尚。
  • 充分与必要，充分不必要，必要不充分
  • 霸道定义，包山包海：条件不中立，标准不统一

- EOF -

- EOF -