TRIAGE: Direction-Aware Mismatch Stabilization of Native NVFP4 Reinforcement Learning

Abstract

Low-precision execution can substantially accelerate reinforcement learning (RL) for large language models, but discrepancies between learner and sampler execution can destabilize policy optimization. In this paper, we characterize the interaction between mismatch and the policy-gradient direction, distinguishing locally amplifying from contracting update contributions that mismatch magnitude alone cannot identify. In native NVFP4 runs, we observe an early imbalance between the two amplifying regions, favoring negative-advantage, negative-gap updates. Their tail tokens become concentrated in a small fraction of response segments before mismatch spreads globally. Motivated by these findings, we introduce TRIAGE, a direction-aware stabilization method that uses segment-level diagnosis to selectively rebalance policy-gradient updates and applies bounded repair to residual severe mismatch. TRIAGE modifies the optimization objective while retaining native NVFP4 weight-and-activation 4-bit (W4A4) forward execution on both the sampler and learner. Experiments on Qwen3-4B and Qwen3-30B-A3B show stable optimization throughout the evaluated training horizon and achieve full precision level performance across five mathematical reasoning benchmarks, while native NVFP4 with TRIAGE provides up to 2.3$\times$ higher rollout throughput than BF16.

1. Introduction

Reinforcement learning (RL) has become central to improving the reasoning capabilities of large language models (LLMs). As RL expands toward long-horizon reasoning, coding, and agentic tasks, increasingly long generation trajectories make rollout a dominant wall-clock bottleneck of the training loop, making efficient rollout a primary systems challenge.

NVIDIA Blackwell GPUs provide native weight-and-activation 4-bit (W4A4) Tensor Core operations, offering up to $4\times$ the peak matrix-compute throughput of BF16 and making NVFP4 an attractive precision setting for long-sequence rollout. However, trajectories are sampled by one execution policy and optimized using probabilities recomputed by another. Differences between the rollout and training execution paths already induce learner--sampler probability discrepancies, and low-precision execution can further enlarge this mismatch. This creates a trade-off between hardware efficiency and stable policy optimization.

Existing approaches mitigate learner--sampler mismatch through two main strategies. Quantization-aware training (QAT) attempts to align the learner with the low-precision sampler, but standard QAT retains high-precision arithmetic and therefore does not preserve native W4A4 execution on the learner forward path. Importance-based corrections such as truncated importance sampling (TIS), clipping, and masking intervene according to the magnitude of learner--sampler disagreement. More fundamentally, magnitude alone describes how far the learner and sampler disagree, not whether the subsequent policy-gradient update will reduce or further amplify that disagreement. This raises a more essential question: what determines whether learner--sampler mismatch contracts or amplifies under policy optimization?

Figure 1. Overview of TRIAGE for native NVFP4 RL.

In this work, we show that learner--sampler mismatch must be considered together with the direction of the policy update (Figure 1, left). If the learner already assigns a token a lower probability than the sampler, further reducing that probability can widen the mismatch rather than correct it. In native NVFP4 runs, updates of this kind become increasingly prominent relative to those that increase probabilities already above the sampler's. This imbalance appears before mismatch becomes broadly severe. Tokens with larger discrepancies in this direction also become concentrated within short portions of responses, where they can be hidden by response-level averages. These observations motivate a different stabilization principle: identify and control updates that tend to amplify mismatch, rather than intervening on mismatch magnitude alone.

Motivated by these findings, we propose TRIAGE, a direction-aware stabilization framework for native low-precision RL. As illustrated in Figure 1, TRIAGE uses segment-level statistics to identify locally persistent risk, selectively attenuates only mismatch-amplifying $\mathcal A^-$ (Advantage $<$ 0, mismatch gap $<$ 0) tokens within risky segments, rebalances update mass, and applies a bounded repair term to residual severe mismatch. By acting on the optimization objective rather than replacing low-precision execution with fake-quantized computation, TRIAGE retains native NVFP4 W4A4 forward execution on both the sampler and learner. Across Qwen3-4B and training-sensitive Qwen3-30B-A3B MoE models, NVFP4 with TRIAGE achieves up to 2.3$\times$ the rollout throughput of BF16, while maintaining stable optimization throughout the training horizon and recovering comparable performance on mathematical reasoning benchmarks. Our contributions are summarized as follows:

1. A directional characterization of learner--sampler mismatch. We characterize mismatch risk through its interaction with the policy-gradient direction and identify a directional, locally concentrated precursor to native NVFP4 instability.
2. TRIAGE, a selective stabilization mechanism for low-precision RL. TRIAGE combines segment-level risk diagnosis, direction-selective token gating, response-level update rebalancing, and targeted repair to suppress self-amplifying mismatch while preserving contracting updates.
3. An end-to-end native NVFP4 RL recipe at scale. Across experiments totaling more than 34,500 GPU-hours on B300 GPUs, we validate native W4A4 execution on both sampler and learner forward paths across dense and MoE models, achieving up to 1.3$\times$ end-to-end RL speedup while maintaining the BF16-level performance.

2. Preliminaries

2.1. Group-Relative Policy Optimization

Group-Relative Policy Optimization (GRPO) is widely used in RL for reasoning models. Given a prompt $x$, GRPO samples a group of $G$ responses $\{y_i\}_{i=1}^{G}$ from an old policy $\pi_{\mathrm{old}}$ and evaluates their rewards $\{r_i\}_{i=1}^{G}$. For outcome-based RL, the group-relative advantage is typically $A_i=(r_i-\mu_r)/(\sigma_r+\epsilon)$, where $\mu_r$ and $\sigma_r$ are the mean and standard deviation of group rewards, and is assigned to all valid tokens of response $i$. Let $\rho_{i,t}=\pi_{\theta}(y_{i,t}\mid x,y_{i,<t})/ \pi_{\mathrm{old}}(y_{i,t}\mid x,y_{i,<t})$ donote the token-level importance ratio. Ignoring optional regularization terms, the clipped policy objective can be written as

Locally, a positive advantage increases the probability of sampled tokens, whereas a negative advantage decreases it; this directional property will be central to our analysis in Section 3.

2.2. Decoupled Rollout and Training

Modern LLM RL systems typically decouple rollout generation from gradient computation: a high-throughput inference engine such as SGLang or vLLM acts as the sampler, while a training framework such as Megatron-LM acts as the learner. Let $\pi_{\mathrm{s}}$ and $\pi_{\mathrm{l}}$ denote the sampler and learner policies, respectively. For a sampled token $y_t$ under the same prefix $y_{<t}$, we define their log-probability gap as

Ideally, synchronized policies should yield $\delta_t=0$. In practice, the two execution paths can disagree even when they nominally represent the same model.

This gap admits a useful interpretation in the language of off-policy correction. Because sampled tokens are drawn from the sampler, $y_{i,t}\sim\pi_{\mathrm{s}}(\cdot\mid x,y_{i,<t})$, the importance weight that exact policy evaluation would require is not $\rho_{i,t}=\pi_{\theta}/\pi_{\mathrm{old}}$ but $\pi_{\theta}/\pi_{\mathrm{s}}$, which factors as

Under synchronous weight transfer with one update per rollout, the first factor is one at the start of each iteration, and the entire off-policy error of the update is carried by $\exp(\delta_{i,t})$. Truncated importance sampling, clipping, and masking all intervene on exactly this factor. Crucially, $\exp(\delta_{i,t})$ is a magnitude-only statistic: it records how far the two policies disagree, but not how the ensuing gradient will change that disagreement. This gap between what existing corrections measure and what optimization stability requires motivates our directional analysis in Section 3.

2.3. Low-Precision Rollout with NVFP4

Reduced-precision rollout lowers both matrix-computation cost and model-memory traffic, making it particularly attractive for long-sequence RL. However, quantization introduces a forward perturbation that is qualitatively different from ordinary policy staleness. Even under an identical prefix, quantized weights and activations perturb hidden states and output logits, producing a local probability discrepancy between the sampler and learner. During decoding, such local perturbations can additionally alter token sampling and hence the subsequent prefix, allowing quantization error to accumulate into trajectory-level divergence.

NVFP4 is a Blackwell-supported format combining E2M1 FP4 values, an E4M3 FP8 microscale per 16 values, and an FP32 second-level scale. For an element $x_j$ in a 16-element block $b$, its quantized representation can be abstracted as

where $s_b$ is an E4M3 FP8 microscale shared by 16 consecutive values, $S$ is an FP32 second-level scale shared across the tensor, and $Q_{\mathrm{E2M1}}(\cdot)$ denotes rounding to the FP4 E2M1 grid. The fine-grained E4M3 microscale improves local range utilization, while the tensor-level FP32 scale extends the range representable by the collection of FP8 block scales.

3. Understanding Learner--Sampler Policy Mismatch

We now study how the learner--sampler mismatch introduced in Section 2 evolves under policy optimization. Our analysis reveals a staged failure process. Section 3.1 derives which mismatches are locally amplified by the policy-gradient update. Section 3.2 then identifies when this directional risk becomes visible along training. Finally, Section 3.3 shows where the risk accumulates within long responses and why response-level averages fail to expose it.

3.1. Directional Dynamics of Policy Mismatch

Which mismatches are self-amplifying under an RL policy update?

Let $z_{i,t}(\theta)=\log\pi_\theta(y_{i,t}\mid x,y_{i,<t})$ denote the learner log-probability of rollout token $(i,t)$, $\delta_{i,t}$ is log-probability gap from (2), and $g_{i,t}$ the signed coefficient applied to its policy-gradient term. A generic learner update with step size $\eta > 0$ is

where for vanilla GRPO at the start of a single-update iteration $g_{i,t}=A_i$; importance correction and stabilization mechanisms modify this coefficient. To track whether the mismatch grows or contracts, define the mismatch energy $D=\frac{1}{2}\sum_{i,t}\delta_{i,t}^{2}$. Holding the rollout policy fixed during one update, a first-order expansion gives

where $K$ is the Gram matrix of token log-probability gradients, $\kappa_{i,t}=\|\nabla_\theta z_{i,t}\|_2^2$, and $R_{\mathrm{cross}}$ collects cross-token interactions. Thus $\Delta D>0$ means the current update batch inflates the total mismatch energy. Appendix A derives this expansion for a general preconditioned update with explicit optimizer residuals. Equation 6 is the identity-preconditioner special case.

Writing the amplifying and contracting diagonal masses as

where $[x]_{+}=\max(x,0)$, it becomes

Thus, within the diagonal current-gradient contribution, $\delta_{i,t}g_{i,t}>0$ amplifies the existing mismatch, whereas $\delta_{i,t}g_{i,t}<0$ contracts it. In particular, for vanilla GRPO, the criterion reduces to the sign interaction $A_i\delta_{i,t}$. The two mismatch-amplifying regions are

The remaining two sign combinations are locally mismatch-contracting. Figure 1, left summarizes this directional geometry.

This distinction explains why mismatch magnitude alone is insufficient: a large mismatch may already be self-correcting, while a moderate one grows systematically when the gradient acts along it. Cross-token coupling, optimizer-history terms, and higher-order effects can still alter the realized motion of an individual token, so we use $\delta_{i,t}g_{i,t}$ as a local directional risk indicator rather than an unconditional predictor of the realized sign of $\delta_{i,t}\Delta\delta_{i,t}$. Appendix B.3 audits the scope of this interpretation. Our main use of the decomposition is therefore population-level: it predicts that instability may first appear as an imbalance between the two amplifying regions, even before the marginal gap distribution becomes broadly severe.

3.2. Directional Asymmetry Precedes Negative-Tail Explosion

When does the imbalance become visible?

We characterize native-NVFP4 GRPO trajectories of Qwen3-4B-Base and Qwen3-30B-A3B-Base using four 25-update trailing windows placed at 25%, 50%, 75%, and 100% of each run's characterization horizon. We refer to these windows descriptively as healthy, drift, pre-terminal, and terminal, respectively.

To measure the distribution of tokens between the two amplifying regions in (9), define the advantage-weighted token occupancy

Figure 2. Directional asymmetry precedes terminal negative-tail expansion

Figure 2 reveals a separation between directional drift and magnitude collapse. Across the drift-to-pre-terminal interval, the marginal gap distribution still appears largely benign: the fraction of tokens satisfying $|\delta|<0.05$ remains at least 83.0% on 4B and 91.9% on 30B. Nevertheless, $\rho_{\mathrm{asym}}$ rises from 1.02 to 2.32 by pre-terminal on 4B, and from 1.34 to 1.85 on 30B. The two theoretically symmetric amplifying regions therefore become empirically imbalanced well before a broad magnitude increase is visible.

In the terminal stage, the lower tail expands abruptly. Within the respective mismatch estimands, the 5% gap quantile $q_{0.05}(\delta)$ reaches $-42.75$ on 4B and $-10.94$ on 30B. Directional asymmetry therefore identifies an earlier warning interval, whereas the negative-tail explosion characterizes the terminal regime.

3.3. Localized Accumulation before Global Failure

Where does the amplifying mismatch accumulate?

For each negative-advantage response, we partition it into non-overlapping segments $S$ of $W=64$ tokens. We call tokens satisfying $A_i<0$ and $\delta_{i,t}<\tau_{\mathrm{ana}}$ amplifying-tail tokens, where the model-specific threshold $\tau_{\mathrm{ana}}^{(m)}$ is fixed as the fifth percentile of $\delta$ in the corresponding

Figure 3. Localized mismatch is hidden by response means before terminal. (a) Share of amplifying-tail tokens contained in the top 5% of segments ranked by $f_S^{\mathrm{amp}}$. (b) Fraction of negative-advantage responses containing at least one tail-heavy segment, the fraction with $|\bar\delta_i|<0.05$

healthy window. Their local concentration is measured by

we call a segment tail-heavy when $f_S^{\mathrm{amp}}>0.1$.

From the healthy to the pre-terminal window, their global frequency decreases from $4.8$--$4.9\%$ to $2.8$--$3.6\%$ across the two models. Nevertheless, the top $5\%$ of segments contain approximately one quarter of all amplifying-tail tokens, about five times the mass implied by uniform placement, while adjacent tail-heavy persistence increases from approximately $2.6\times$ to $3.8$--$4.0\times$ the chance rate. The precursor is therefore not an expanding global tail, but a reorganization of a small tail population into a few persistent local regions.

Response averages obscure this structure: at pre-terminal, $97.2\%$ (4B) and $79.9\%$ (30B) of responses containing tail-heavy segments still satisfy $|\bar\delta_i|<0.05$, where $\bar\delta_i=n_i^{-1}\sum_{t\in\mathcal T_i}\delta_{i,t}$. By terminal, roughly half of the eligible segments are tail-heavy and top-segment concentration declines.

These observations motivate segment-level diagnosis with intervention restricted to $\mathcal A^-$ tokens, rather than suppressing every token in an affected response or segment.

4. Method

Section 3 shows that the risk of learner--sampler mismatch depends on its interaction with the policy-update direction: updates with $\delta_{i,t}g_{i,t}>0$ amplify the existing gap, whereas updates with $\delta_{i,t}g_{i,t}<0$ contract it. TRIAGE acts on this distinction through two complementary modifications of the effective update coefficient. Let $g^{\mathrm{base}}_{i,t}$ denote the per-token coefficient of the underlying importance-corrected objective, including its reduction factors. We then write

The two terms partition by advantage sign: $\alpha_{i,t}$ reallocates policy-gradient mass within negative-advantage responses (Section 4.1), while $\psi_{i,t}$ adds bounded corrective pressure on positive-advantage responses (Section 4.2).

4.1. Segment Diagnosis with Direction-Selective Gating

Following Section 3.3, TRIAGE therefore uses short contiguous segments as an intermediate diagnostic scale, a granularity that has also been found useful for avoiding the extremes of token- and response-level importance estimation.

For a diagnostic segment $S$ of response $i$, we compute

which respectively capture persistent displacement and sparse deep-tail mismatch. We map these statistics to a segment gate weight

where a larger negative displacement or a heavier negative tail yields stronger attenuation. The exact operating rule, including eligibility conditions and the fixed operating thresholds, is deferred to Appendix C.

Crucially, a risky segment does not imply that all of its tokens should be attenuated. For a negative-advantage response, tokens with $A_i<0,\ \delta_{i,t}<0$ contribute to mismatch amplification, whereas tokens with $A_i<0,\ \delta_{i,t}>0$ naturally contract the discrepancy. TRIAGE therefore applies

Thus, segment statistics determine whether local intervention is warranted, while the directional criterion from Section 3 determines which token updates are modified. In particular, naturally contracting tokens that coexist in the same risky segment are not directly gated. We do not gate the positive-advantage amplifying region $\{A_i>0,\ \delta_{i,t}>0\}$: the directional imbalance identified in Section 3 concentrates in the negative-advantage region, and attenuating probability increases on reward-positive responses would directly suppress the learning signal.

The weighted policy loss is

where $\ell^{\mathrm{base}}_{i,t}$ denotes the existing importance-corrected token loss. Relative to the original response mean, this induces

The normalization therefore reallocates relative update mass away from the targeted amplifying subset rather than uniformly weakening the entire negative-advantage response, directly modifying the balance between $B_{\mathrm{amp}}$ and $B_{\mathrm{con}}$ identified in Section 3.

4.2. Repair on Positive-Advantage Responses

Direction-selective gating controls the allocation of existing policy gradients, but it is multiplicative: it can attenuate an amplifying update toward zero, yet cannot reverse its sign or directly reduce the remaining learner--sampler gap. We therefore add a repair mechanism on positive-advantage responses. On these responses, the policy gradient on negative-gap tokens is already mismatch-contracting, so raising sampled-token log-probabilities reinforces a motion that agrees with reward-driven learning; applying the same correction to negative-advantage responses would instead oppose their policy gradients, which is why that side is handled by gating alone.

Let $R$ denote a repair segment of response $i(R)$, with $n_R=\sum_{t\in R}m_{i(R),t}$ valid tokens and mean gap $\bar{\delta}_R$. We define

where $q_R$ is the one-sided violation of the repair boundary and $\phi_\beta$ is a pseudo-Huber penalty.

Gating is active throughout training, whereas repair is enabled from a fixed update $k_{\mathrm{rep}}$ onward, with $\lambda_k=\lambda_{\mathrm{rep}}\mathbf{1}[k\geq k_{\mathrm{rep}}]$. All operating thresholds and the repair schedule are fixed before each run. At update $k$, the objective is

Under the ascent-form notation of Equation 12, one gradient-descent step on the repair objective contributes the additive coefficient

All valid tokens within an active segment share the same corrective coefficient. For negative-gap tokens, $\delta_{i,t}\psi_{i,t}<0$, so the repair term contributes to contraction in the diagonal decomposition of Section 3. Positive-gap tokens may also receive the coefficient: the objective repairs negative segment-level displacement rather than enforcing contraction for every token. Appendix C.3 derives the positive-advantage repair coefficient and its bound.

Figure 4. Training dynamics on Qwen3-30B-A3B-Base and Qwen3-4B-Base. From left to right: entropy loss, training reward, and mean absolute train--rollout log-probability gap against optimizer steps.

5. Experiments

5.1. Experimental Setup

Models and task.

We use Qwen3-30B-A3B-Base as the primary model and Qwen3-4B-Base for replication and ablation studies. We perform RL with GRPO on the DAPO dataset. To better understand the quantization error, we use 20,480 as the max response length. Detailed training settings are listed in Appendix D.

Settings.

We compare four training configurations: (i) BF16, the baseline; (ii) NVFP4, native W4A4 execution with Transformer Engine; (iii) NVFP4+TIS, adding token-level importance correction; and (iv) NVFP4+TRIAGE, our full method. All NVFP4 configurations share the same native execution path: W4A4 kernels with per-token second-level activation scaling on both the sampler and the learner forward pass. TIS uses a truncation threshold of $C=\text{2.0}$. TRIAGE operating rules and segment definitions are given in Appendix C; quantization and system settings are described in Appendices D and E.

Evaluation Benchmarks and Metrics.

We evaluate mathematical reasoning on several widely used benchmarks, including GSM8K, MATH-500, AIME 2024/2025, and AMC 2023. We also track training reward, policy entropy, and the mean absolute train--rollout log-probability gap throughout training.

5.2. Experimental Results

Training stability.

Figure 4 shows that naive NVFP4 becomes unstable after approximately 300 steps on Qwen3-4B-Base, with a sharp reward drop and rapid growth in the mean absolute train--rollout log-probability gap. TIS delays this failure but does not prevent it. On Qwen3-30B-A3B-Base, naive NVFP4 also exhibits rapid mismatch growth after roughly 700 steps. TIS does not exhibit a comparable divergence over 1,700 steps, but its reward trajectory remains below BF16 and TRIAGE in later training. In contrast, TRIAGE remains stable throughout the evaluated 600- and 1,700-step schedules, approximately doubling the observed stable training horizon on 4B relative to naive NVFP4.

Benchmark performance.

Table 1 shows that TRIAGE achieves near-BF16 average performance: 58.49% versus 58.26% on 4B, and 70.96% versus 72.41% on 30B. At the shared step-300 checkpoint on 4B, TRIAGE already outperforms naive NVFP4 and NVFP4+TIS by 5.93 and 2.27 percentage points, respectively, showing that its gains are not solely due to longer training. Although NVFP4+TIS does not collapse over the full 1700-step training horizon, it still scores below the pre-collapse naive NVFP4 checkpoint at step 700 (63.18% versus 65.87%).

These observations are consistent with the directional and localized mismatch analysis in Section 3: the global mean does not capture how mismatch interacts with policy updates or where it concentrates within responses, supporting the importance of selective control of mismatch amplification.

Table 1. Performance across five benchmarks. Step denotes the evaluated training checkpoint. Stable runs are evaluated at the final training step, while runs marked with $\dagger$ collapse before the planned training budget and are evaluated at the last saved checkpoint before collapse. Best result among NVFP4 settings for each model is shown in bold.

Table 2. Natural-EOS generation from the base checkpoint. Values are means over ten post-warm-up iterations at two global batch sizes. Initialization, evaluation, and checkpoint saving are excluded. Parentheses report speedups over BF16.

5.3. Rollout Throughput and Learner Overhead

Rollout throughput.

With TRIAGE, native NVFP4 achieves 6,745 and 8,717 output tokens/s at global batch sizes 256 and 512, respectively, corresponding to $2.27\times$ and $2.30\times$ the BF16 throughput. TRIAGE thus retains the more-than-twofold rollout throughput advantage observed for native NVFP4 in these runs.

End-to-end RL efficiency.

Iteration timing includes rollout, learner updates, weight conversion and synchronization, and orchestration. TRIAGE reduces average iteration time from 208.2 to 172.1 s at batch size 256 and from 254.5 to 195.8 s at batch size 512, yielding $1.21\times$ and $1.30\times$ end-to-end speedups, respectively.

TRIAGE overhead.

We separately isolate TRIAGE's incremental learner cost on eight B300 GPUs using five paired repetitions of the same frozen batch. Enabling gating and repair yields an observed median learner-update overhead of 0.84% and a median absolute increase of 0.225 s.

Detailed throughput settings, environment settings, independent configuration-selection results, stress-tests results, and complete timing distributions are provided in Appendix E.

5.4. Ablation Studies

We perform component ablations on Qwen3-4B-Base. All variants branch from the same full-TRIAGE checkpoint at step 300. Full TRIAGE achieves the highest reward. Removing repair keeps the mean log-probability gap low but leaves the advantage-driven update imbalance uncorrected, reducing the reward. TIS alone remains stable when continued from the TRIAGE checkpoint, unlike its failure when used from the beginning in Figure 4. This contrast reflects the stage-dependent mismatch dynamics identified in Section 3. Naive NVFP4 also allows mismatch to grow sharply, leading to a late-stage collapse.

Figure 5. Ablation on Qwen3-4B-Base. Full TRIAGE; Gate+TIS removes repair; TIS additionally removes gating; NVFP4 removes all three components. The dashed line marks the branch point.

6. Related Work

Low-precision training.

Native low-precision training requires careful control of quantization error. NVIDIA's NVFP4 pretraining recipe and Quartet II address this through quantization design and improved gradient estimation. For RL, QeRL combines NVFP4 with LoRA and adaptive quantization noise. FP8-RL stabilizes quantized rollout with importance-based corrections. We focus on the optimization dynamics of the residual learner--sampler mismatch under native NVFP4 W4A4 execution.

Training--inference alignment.

Implementation differences can destabilize RL even with synchronized model weights. QaRL aligns the learner with quantized rollout through low-bit forward computation and introduces sequence-level dual clipping for negative samples. QUADS combines weight-only fake quantization on the learner with rollout-side activation compensation. TRIAGE instead controls how residual mismatch evolves under policy updates, modifying the optimization objective while retaining native W4A4 forward execution on both the learner and sampler.

Directional control.

ACRL uses advantage-guided, four-quadrant probability adjustment, with response-level discrepancy relative to a reference controlling token-wise reweighting. TRIAGE derives a local mismatch-energy decomposition and identifies directional asymmetry and localized negative tails in native NVFP4 runs.

7. Conclusion

We presented TRIAGE, a direction-aware stabilization method for native NVFP4 RL. Our analysis distinguishes locally amplifying and contracting policy-gradient contributions and identifies directional imbalance and segment-localized negative tails before global mismatch growth. TRIAGE translates these observations into selective gating, within-response rebalancing, and bounded repair, while preserving native W4A4 forward execution on both learner and sampler. On Qwen3-4B-Base and Qwen3-30B-A3B-Base, TRIAGE sustains stable optimization over the evaluated horizons and achieves near-BF16 average performance across five mathematical reasoning benchmarks. In our measured configurations, native NVFP4 with TRIAGE achieves up to $2.30\times$ rollout throughput and $1.30\times$ end-to-end speedups over BF16. These results support controlling how mismatch evolves under policy updates, rather than uniformly minimizing its magnitude.

8. Limitations and Future Work

Our study focuses on establishing the importance of direction-aware stabilization for low-precision RL. While truncated importance sampling (TIS) can stabilize the absolute mismatch gap, our 30B experiments show that completing the training trajectory does not necessarily translate into strong downstream performance. In contrast, TRIAGE explicitly accounts for whether learner--sampler mismatch is locally corrected or amplified by the policy update. Nevertheless, our empirical study remains limited in scale. Due to computational constraints, we do not evaluate models at the 100B+ scale or substantially longer training horizons. Our main runs are terminated after the optimization trajectories exhibit clear convergence or stable late-stage behavior. Evaluating whether the same directional dynamics persist over substantially longer training and on larger sparse models is therefore an important direction for future work.

Our current experiments also focus on synchronous RL. The directional criterion underlying TRIAGE is not inherently tied to synchronous sampling, and may be particularly relevant to asynchronous RL, where policy staleness introduces an additional source of learner--sampler mismatch. However, we have not experimentally validated TRIAGE in this setting. Extending the analysis to jointly characterize quantization mismatch and asynchronous policy staleness is an interesting direction for future study.

Finally, we believe that the end-to-end efficiency of native NVFP4 RL remains below the acceleration potential of Blackwell and Vera Rubin Tensor Core computation. Although NVFP4 substantially accelerates rollout generation in our experiments, the overall RL iteration speedup is approximately $1.3\times$. A growing fraction of runtime is instead spent on operations outside the accelerated GEMMs, including parameter synchronization between the trainer and sampler and quantization-related reductions and data transformations such as Amax computation. Future systems could therefore obtain larger gains by co-designing TRIAGE with more tightly integrated low-precision kernels and RL frameworks, including more efficient parameter synchronization and fused quantization paths. On the algorithmic side, our directional analysis also suggests a broader design space beyond the current repair rule: rather than treating all mismatch-amplifying tokens uniformly, future work could develop finer-grained repair mechanisms that adapt to the magnitude, locality, and persistence of directional mismatch.

BibTeX

@misc{li-2026-triage,
      title={TRIAGE: Direction-Aware Mismatch Stabilization of Native NVFP4 Reinforcement Learning},
      author={Zhen Li and Shuai Zhang and Yanggan Gu and Yiming Zhang and Yang Yu and Mingfa Feng and Congkai Xie and Shuang Yu and Junjie Lai and Hongxia Yang},
      year={2026},
      eprint={2610.07043},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2610.07043},
}