GPT-6 Astra, looped transformers, and latent reasoning

Direct answer

The loop is part of the model's ordinary forward pass, so it normally runs while predicting every token—not only dedicated reasoning tokens. If Astra really uses recurrent depth, the same machinery can process prose, code, tool calls, hidden chain-of-thought, and the final answer. The architecture does not intrinsically know which tokens count as “reasoning.”

A deployment can still vary the work:

• a fixed-depth design applies the same number of loops to every token;
• adaptive halting can give different token positions or decoding steps different numbers of loops;
• a product's reasoning-effort setting can choose a global loop budget, although this is plausible rather than disclosed for Astra;
• explicit chain-of-thought generation can coexist with recurrent depth.

Therefore, “latent reasoning” is a useful description of extra hidden-state computation, but it should not be interpreted as a separate secret reasoning module that switches off when the model writes its answer.

What is known about Astra

The right model is thus: Astra may use constrained recurrence, while the exact topology, loop count, routing, and contribution to its capability remain unknown. Its gains can also come from data, scale, RL, tool/computer-use training, and serving infrastructure.

The mechanics

A conventional causal transformer with distinct layers computes

$$ h^{(0)} = E(x_{1:n}), \qquad h^{(l+1)} = F_l\!\left(h^{(l)}; K^{(l)}_{<n},V^{(l)}_{<n}\right), $$

then projects the final state to next-token logits:

$$ p(x_{n+1}\mid x_{\le n}) = \operatorname{softmax}(W_U h^{(L)}_n). $$

A looped transformer replaces some distinct layers with a shared recurrent core $R_\theta$. A common prelude/core/coda form is

$$ z_0=P(x_{\le n}), \qquad z_{r+1}=R_\theta(z_r, z_0, r), \quad r=0,\ldots,R-1, $$

$$ h=C(z_R), \qquad p(x_{n+1}\mid x_{\le n})=\operatorname{softmax}(W_U h_n). $$

The same parameters $\theta$ are used at every recurrence, but the activations differ because $z_r$ changes. Unrolling the graph makes it look like a deeper network with tied weights.

flowchart LR
    T[Current token prefix] --> E[Embedding / prelude]
    E --> R1[Shared transformer core\npass 1]
    R1 --> R2[Same weights\npass 2]
    R2 --> RN[Same weights\npass R]
    RN --> C[Coda + LM head]
    C --> N[Sample next token]
    N --> T2[Append token and repeat\nthe whole process]

The inner recurrence happens before one next token is selected. The outer autoregressive loop then appends that token and repeats the complete computation. For a response of $N$ tokens with $R$ recurrent passes, the shared core is invoked approximately $N\times R$ times (before batching, caching, and adaptive exits).

Why this counts as latent computation

Chain-of-thought and recurrent depth buy serial computation on different axes:

Explicit token-space reasoning

prompt → forward pass → thought token 1
       → forward pass → thought token 2
       → ...
       → forward pass → answer token

The model writes intermediate state through the narrow vocabulary bottleneck, then can attend to those tokens as a persistent scratchpad. This is long-lived, addressable across later decoding steps, and potentially monitorable as language.

Hidden depth-space refinement

prompt → hidden pass 1 → hidden pass 2 → ... → hidden pass R → one token

The intermediate state stays in continuous residual vectors. It is high-bandwidth and need not be expressible as a sentence, but it usually disappears after the token is emitted except for what survives in the token and per-pass KV cache.

These mechanisms are complementary rather than substitutes. A looped model can use recurrent passes to calculate each token and emit a long textual chain of thought. Huginn's paper calls recurrence “latent reasoning,” but its model can still produce ordinary textual reasoning.

Does every token get the same number of loops?

Not necessarily. Designs occupy a spectrum:

1. Fixed global recurrence: Nanbeige applies its 22-layer stack twice. Every prediction pays approximately the same recurrent cost.
2. Request-level depth: Huginn is trained over sampled recurrence depths, then the server can choose, for example, 8 or 32 passes at inference.
3. Confidence-based stopping: stop when successive next-token distributions change little, such as when $$D_{\mathrm{KL}}(p_{r}\Vert p_{r-1}) < \epsilon.$$
4. Token-level routing: Mixture-of-Recursions sends different sequence positions through different recursion depths. A router reads contextual hidden states, so the same vocabulary token can receive different compute in different sentences.
5. Learned exits: Ouro assigns probabilities to exits after recurrent passes. Its published implementation reportedly computes all four configured passes before selecting an exit, so the conceptual gate does not necessarily yield wall-clock savings in that release.

Causal attention complicates token-level routing: active positions must preserve causality and have compatible keys/values for later positions. Efficient routing therefore needs specialized packing, masking, and cache management; a diagram alone does not guarantee real serving speedups.

How training works

The simplest model is trained from scratch exactly as an autoregressive language model, except the recurrent core is unrolled in the forward graph:

h0 = prelude(token_embeddings)
h = h0
for r in range(R):
    h = recurrent_core(h, injected_input=h0, iteration=r)  # shared weights
logits = lm_head(coda(h))
loss = cross_entropy(logits[:, :-1], tokens[:, 1:])
loss.backward()  # gradients flow through every unrolled application

Because all applications reference the same weights, their gradient contributions accumulate:

$$ \frac{\partial \mathcal L}{\partial \theta} = \sum_{r=1}^{R} \frac{\partial \mathcal L}{\partial z_r} \frac{\partial z_r}{\partial \theta}. $$

This is backpropagation through recurrent depth. Weight memory and optimizer state scale with the number of unique parameters; activation memory and forward/backward FLOPs scale with the number of applications. Activation checkpointing can trade memory for yet more recomputation.

Important training variants include:

• Fixed-depth training: train and deploy at one recurrence count. Nanbeige reports that training recurrence from scratch worked better than retrofitting a pretrained feed-forward model.
• Depth sampling: sample $R$ during training so one checkpoint works at multiple inference budgets. Huginn does this to support test-time depth scaling.
• Intermediate/deep supervision: attach the language-model loss after several loops so useful predictions emerge at multiple exits.
• Halting or routing losses: jointly train a gate, often with a compute penalty, so accuracy competes against expected depth.
• Iteration conditioning: inject an embedding or adaptive normalization based on $r$; otherwise identical weights must infer the iteration only from the changing state.
• Input reinjection: mix the original prelude representation $z_0$ into every pass to prevent drift or loss of source information.
• Residual scaling and normalization: control exploding residual norms, vanishing/exploding gradients, oscillations, and collapse toward unhelpful fixed points.

Random depth alone does not guarantee unlimited extrapolation. A model trained mostly at 4–16 loops may degrade at 64: recurrent dynamics can converge, oscillate, or “overthink” and move away from a correct state.

Compute, parameter memory, and KV cache

Weight sharing saves parameters, not arithmetic.

For a recurrent core containing $B$ transformer blocks and used $R$ times:

• stored core parameters: proportional to $B$;
• effective block applications: $B R$;
• core forward FLOPs: approximately proportional to $B R$;
• serial latency: rises with $R$ because passes depend on earlier passes;
• optimizer state during training: proportional to $B$ unique blocks;
• activation/backprop work: proportional to the unrolled $BR$ applications.

A 22-block stack used twice resembles 44 blocks of computation but stores only 22 blocks' weights. It is not “free thinking.”

KV-cache nuance

Every recurrent visit generally produces different keys and values because its input hidden state is different. To reproduce an unrolled deep transformer faithfully, serving keeps logically separate KV entries for each recurrent application. Therefore, a $B$-block core repeated $R$ times can have roughly the KV-cache footprint of $BR$ untied applications.

Sharing first-pass KV across loops can reduce memory, but changes the computation and has harmed quality in some experiments. Architectures such as Mixture-of-Recursions attempt selective caching, while SMELT changes head dimensions/GQA to match cache budgets explicitly.

This is why looped transformers are principally weight/optimizer-memory efficient, not automatically KV-memory or latency efficient.

What looping buys

More computation without more stored weights

The architecture trades knowledge capacity for algorithmic depth. The 2025 Beyond Parameters experiments found that recurrence barely changed memorization capacity at fixed unique parameter count, while additional unique parameters did. Recurrence did improve multi-step reasoning.

A useful shorthand is:

unique parameters ≈ how much can be stored
effective depth   ≈ how much sequential transformation can be performed

This is not an absolute theorem about all models, but it matches current controlled evidence. Reusing the same “brain” provides another processing pass; it does not install new facts into the weights.

A second test-time-compute axis

A conventional reasoning model can spend more compute by producing more scratchpad tokens. A depth-flexible recurrent model can also increase $R$. The allocation problem becomes two-dimensional:

$$ \text{test-time compute} \approx \text{generated tokens} \times \text{effective depth per token}. $$

For persistent search state, citations, tool observations, and branches that must be revisited much later, token-space scratchpads remain valuable. For local iterative refinement before choosing the next token, latent depth avoids serializing everything through vocabulary tokens.

Possible inductive bias beyond merely “more FLOPs”

Naive comparisons often give looped models more block applications, making it unclear whether recurrence itself helps. SMELT matches non-embedding parameters, per-token FLOPs, and approximately KV cache by narrowing the model, adding sparse experts, and looping the middle layers. Across scales up to 54B non-embedding parameters, its fitted scaling curves report 6.8–18% less training compute to reach the same validation loss. The authors also observe reduced attention sinks and more attention to content on the second visit.

This is evidence for a useful recurrent inductive bias, not proof that arbitrarily many loops keep helping or that Astra's gains come from recurrence.

Is it really “reasoning”?

Calling every hidden transformation reasoning is too strong. The loop mechanically performs iterative representation refinement. Whether it implements search, variable binding, error correction, retrieval, or mundane feature extraction depends on the learned computation and task.

Mechanistic work finds that recurrent blocks can repeat stages resembling those in feed-forward depth and may approach cyclic fixed points. This suggests a continuum:

feature refinement → iterative inference → algorithmic state updates → search-like reasoning

The architecture makes additional latent sequential computation possible; behavior and probes are required to determine what that computation does.

Monitorability: the precise concern

All transformers already perform opaque hidden computation before every token. Recurrence adds more of it and may let the model solve some tasks with fewer explicit scratchpad tokens. That can reduce the amount of natural-language evidence available to a chain-of-thought monitor.

However:

• recurrence does not delete or prohibit textual chain of thought;
• a stronger ordinary feed-forward model can also need fewer reasoning tokens;
• visible CoT was never guaranteed to be a faithful transcript of causal internal computation;
• shorter traces may indicate fewer errors and less backtracking rather than deliberate concealment;
• one 2026 latent-feedback study found shorter MATH-500 traces in a base model, but the effect disappeared after instruction tuning.

The safety question is empirical: at matched task success and compute, does recurrence reduce the sensitivity of monitors to deception, reward hacking, or dangerous plans? That requires controlled recurrent-vs-feed-forward models, causal interventions, probes over hidden states, and behavioral ground truth. Astra's system card establishes a monitorability regression, but public evidence does not identify recurrence as its cause.

Practical mental model

Think of a standard deep transformer as a sequence of different specialists. A looped core is the same small team revisiting a shared working representation several times.

• The team has the same stored expertise on every visit: parameter count is unchanged.
• The draft evolves: activations are different on each pass.
• Every visit costs time and compute.
• Unless a pass is decoded, its work is not a readable note.
• After the team emits one token, the autoregressive process starts again for the next token.

For Astra specifically, the most defensible conclusion is: limited recurrence is plausible, applies as general token-generation machinery if present, and may shift some computation from textual scratchpads into activations; neither its exact design nor a causal link to reduced CoT monitorability is public.

Related

• Compact reasoning traces and reasoning compression
• OpenAI GPT-6 Astra release and access
• Latent Reasoning with Normalizing Flows
• LLM inference engineering

Sources

Main article

• Sebastian Raschka, “GPT-6 Astra, Looped Transformers, and Hidden Reasoning”, 9 September 2026. Local capture: raw/20260920T105532Z-web-article-gpt-6-astra-looped-transformers/.

Primary research

• Dehghani et al., Universal Transformers, 2018.
• Geiping et al., Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach, 2025.
• Saunshi et al., On the Power of Looped Transformers, 2025.
• Zhu et al., Beyond Parameters: Exploring Virtual Logic Depth for Scaling Laws, 2025.
• Bae et al., Mixture-of-Recursions, 2025.
• Zhu et al., Scaling Latent Reasoning via Looped Language Models (Ouro), 2025.
• Nanbeige Lab, Nanbeige4.2-3B, 2026.
• Wang et al., Full-Bandwidth Transformer, 2026.
• Wang et al., SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers, 2026.
• OpenAI, GPT-6 Astra system card and launch materials, 2026.