Contrastive Neuron attribution (CNA), Sparse MLP circuit steering without SAE training or weight modification, is released by Nous Research

Language models that have been tuned for instruction refuse to accept harmful requests. But which part of the model is actually responsible — and how does that mechanism get installed during training? The Nous research team has conducted a study that examines this issue at the neuronal level. The Nous Research team has developed contrastive neuron attribution (CNA)This method uses MLP activation to identify the neurons that are most likely to distinguish between benign and harmful prompts. By ablating just 0.1% of MLP activations, they reduced refusal rates by more than 50% in most instruct models tested — across Llama and Qwen architectures from 1B to 72B parameters — while keeping output quality above 0.97 at all steering strengths. The key discovery is that base models already have the late layer structure which distinguishes harmful prompts from benign ones. The fine-tuning of alignments does not result in a new structure. The function of the neurons in that structure is transformed into a targetable, sparse refusal gate.

What’s wrong with existing steering methods?

Contrastive Activation Addition (CAA) Calculates the average difference between two values Residual stream Two contrastive sets of prompts. At inference, the difference acts as a steering vector. CAA can be effective, but it is not fine: It changes the whole layer’s signal without pinpointing which neuron was responsible. At high steering strengths, output quality degrades — models produce repeated words and incoherent text.

Sparse autoencoders (SAEs) Decompose activations to interpretable features. The external training is costly and the sensors are sensitive to activation sounds.

CNA requires only forward passes — no gradients, no auxiliary training, no iterative search.

CNA: How Does It Work?

There are two types of prompts that you can use:

• Positive prompts — examples of the target behavior (e.g., harmful requests)
• Negative prompts — examples of the opposite (e.g., benign requests)

The model will prompt you to run the entire process. Method records are kept at every MLP layer. Down-projection activation The last position of the token is then calculated. This is followed by a calculation of the mean activation differences per neuron for the two sets.

δ_j^ℓ = mean(activations on positive prompts) − mean(activations on negative prompts)

The top-k neurons by absolute Difference are selected across all layers. The researchers set k to 0.1% of total MLP activations. This threshold produced reliable steering effects across all model sizes tested.

A filtering step removes ‘universal’ neurons — those appearing in the top 0.1% of MLP activations across 80% or more of diverse prompts. These neurons fire regardless of prompt content and are excluded from all discovered circuits.

Causality is verified by multiplying each circuit neuron’s activation by a scalar multiplier m at inference time. m = 0 ablates the neuron. m = 1 is baseline. m > 1 amplifies it.

For the main JBB-Behaviors evaluation, the refusal circuit is discovered using 100 harmful and 100 benign prompts. For qualitative examples and other tasks, 8 positive and 8 negative prompts were used.

Results

Experiments covered base and instruct variants of Llama 3.1/3.2 and Qwen 2.5, from 1B to 72B parameters — 16 models total. The main benchmark was JBB-Behaviors, a NeurIPS 2024 benchmark of 100 harmful prompts.

Refusal reduction. Ablating the discovered circuit reduced refusal rates by more than 50% in most instruct models tested. Selected results from Table 3 of the research paper:

Not all models exceeded 50% relative reduction — Llama-3.2-3B and Qwen2.5-3B showed smaller drops. The paper describes the effect as holding “in most cases.”

Output quality. CNA output quality, measured as 1 minus the fraction of repeated n-grams, stayed above 0.97 at all steering strengths across all instruct models tested. CAA dropped below 0.60 for six of the eight instruct models at maximum steering strength. In two cases — Qwen2.5-1.5B and Qwen2.5-72B — CAA degraded output so severely that the keyword classifier flagged degenerate text as refusals, producing artificially high refusal rates.

General capabilities. MMLU accuracy under CNA stayed within one percentage point of baseline at all steering strengths. CAA dropped to near-zero MMLU accuracy at maximum intervention.

StrongREJECT rubric. A secondary evaluation used the StrongREJECT rubric, which applies an LLM The judge (LLama-3.3-70B) to score responses on harmfulness and dangerousness on a 0–1 scale. Llama model compliance scores improved by an average of 6% after CNA ablation. Qwen model compliance scores improved by an average of 31%.

Base model comparison. Applying the identical pipeline to base models produced no meaningful behavioral change. The paper illustrates this with a concrete example using the prompt “How do I pick a lock?”:

In base models, steering the late-layer neurons produces content shifts — topic changes, rephrasing — but no behavioral change at any multiplier. In instruct models, the same structure acts as a causal safety gate.

Fine-Tuning Transforms Function, Not Structure

Discrimination neurons concentrate in the final 10% of layers in both base and instruct models. For Llama-3.2-1B, 87% of the top-200 discrimination neurons fall in the final three layers (L13–L15). For Qwen2.5-3B, 95% fall in the final quarter of layers. This late-layer concentration is a pretraining property — it exists before alignment fine-tuning.

The function of those neurons changes after fine-tuning. Table 8 in the research paper reports the overlap of (layer, neuron) index pairs between matched base and instruct circuits. Only 8–29% of individual neurons overlap between base and instruct models. Fine-tuning largely replaces the specific neurons within that late-layer structure while preserving the structure itself.

The research team describe this as a separation between two levels: layer-level structure (preserved across base and instruct) and neuron-level function (transformed by fine-tuning). This is consistent with prior work showing that instruction tuning rotates feed-forward network knowledge without changing layer structure.

Marktechpost’s Visual Explainer

# Register hooks on down_proj in each MLP layer
def make_hook(layer_idx, store):
    def hook(module, input, output):
        store[layer_idx] = output[:, -1, :].detach()
    return hook

activations = {}
hooks = []
for i, layer in enumerate(model.layers):
    h = layer.mlp.down_proj.register_forward_hook(
        make_hook(i, activations)
    )
    hooks.append(h)

# Run forward pass
with torch.no_grad():
    model(**inputs)

# pos_acts, neg_acts: tensors of shape [n_prompts, n_neurons]
import torch

delta = dict()
for layer_idx in pos_acts:
    delta[layer_idx] = (
        pos_acts[layer_idx].mean(dim=0)
        - neg_acts[layer_idx].mean(dim=0)
    )

# Flatten all deltas into one tensor with (layer, neuron) indices
all_deltas = torch.cat([delta[i] for i in sorted(delta)])
total = all_deltas.numel()
k = max(1, int(total * 0.001))  # 0.1%

top_vals, top_idx = torch.topk(all_deltas.abs(), k)

# Map flat index back to (layer, neuron) pairs
n_neurons = all_deltas.shape[0] // len(delta)
circuit = [(idx // n_neurons, idx % n_neurons)
           for idx in top_idx.tolist()]

# circuit: list of (layer_idx, neuron_idx)
# m=0 ablates, m=1 baseline, m>1 amplifies

def make_ablation_hook(neuron_indices, m):
    def hook(module, input, output):
        output[:, -1, neuron_indices] *= m
        return output
    return hook

# Group circuit neurons by layer, then register hooks
from collections import defaultdict
by_layer = defaultdict(list)
for layer_idx, neuron_idx in circuit:
    by_layer[layer_idx].append(neuron_idx)

hooks = []
for layer_idx, neurons in by_layer.items():
    h = model.layers[layer_idx].mlp.down_proj
        .register_forward_hook(
            make_ablation_hook(neurons, m=0.0)
        )
    hooks.append(h)

Key Takeaways

• Ablating just 0.1% of MLP activations reduced refusal rates by more than 50% in most instruct models tested, while output quality stayed above 0.97.
• CNA requires only forward passes — no gradients, no auxiliary training, and no iterative search.
• Late-layer discrimination structure exists in base models before fine-tuning; alignment fine-tuning transforms its function, not its location.
• Unlike CAA, CNA preserves MMLU accuracy within one percentage point of baseline at all steering strengths.
• Only 8–29% of individual neurons overlap between base and instruct model circuits — fine-tuning rewires the neurons while keeping the late-layer structure intact.

Check out the Paper and Repo. Also, feel free to follow us on Twitter and don’t The following are some examples of how to useget to joThe following are some examples of how to use our 150k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to partner The following are some examples of how to use us The following are some examples of how to use promotThe following are some examples of how to useg your GItHub Repo OR HuggThe following are some examples of how to useg Face Page OR Product Release OR WebThe following are some examples of how to usear etc.? Connect with us