November, 2026 | News
Firmus cuts MLPerf training time for GPT-OSS 20B and Llama 3.1 8B
Firmus reports 24% less training time for GPT-OSS 20B and 13.9% for Llama 3.1 8B on eight NVIDIA GB300 GPUs.

17 November 2026 | Firmus Newsroom
Firmus has reduced GPT-OSS 20B training time from 69.60 to 52.90 minutes in its MLPerf® Training v6.1 submission, a reduction of 24%. The team also reduced Llama 3.1 8B training time from 60.88 to 52.40 minutes, or 13.9%. Both workloads used eight NVIDIA GB300 GPUs.
Senior AI Engineers Shaojun Zhang and Lewis Won built on NVIDIA’s Megatron Bridge, Megatron Core and Transformer Engine software. They used GPU profiling and AI coding agents to address bottlenecks in how the GPUs performed calculations and moved data, with verification throughout development.
Developed by MLCommons, MLPerf Training measures how quickly systems train models to defined quality targets. Firmus compared its optimised implementations with its own baselines, each using eight GB300 GPUs across two nodes.
GPT-OSS 20B

Llama 3.1 8B

Why Firmus selected GPT-OSS 20B and Llama 3.1 8B
Firmus selected GPT-OSS 20B and Llama 3.1 8B following an analysis of submissions from previous MLPerf Training rounds. Both were among the most widely represented workloads and continued to attract participation in v6.1.
The eight-GPU configuration was selected on the same basis, providing opportunities to compare Firmus's results with other submissions at that scale.
The two models also represent different architectures.
GPT-OSS 20B uses a mixture-of-experts architecture, which directs computations through selected components of the model, whereas Llama 3.1 8B uses a dense transformer architecture.
Together, the workloads allowed Firmus to evaluate its optimisation methods across two established language model architectures.
Optimising GPU computation
Preparation took several weeks to align the software with the reference implementation and establish baseline performance. Initial configuration tuning provided a starting point for deeper changes to the GPU kernels, the small programs that perform individual calculations.
GPU profiling identified repeated memory access, unnecessary calculations and the overhead of launching many small kernels. Optimisation focused on removing redundant work and combining operations so the same data did not have to be read and written repeatedly.
GPT-OSS 20B presented an additional challenge. The original reference image was implemented in MXFP8, and a naive conversion to NVIDIA’s NVFP4 format resulted in a longer time-to-train than the baseline MXFP8 implementation because of extra work required to prepare data for NVFP4 offset. By reducing conversion overhead and improving memory handling and scheduling, the final GPT-OSS configuration implemented in NVFP4 reduced training time by 16.70 minutes to reach the required quality target.

Llama 3.1 8B retained its baseline numerical precision in NVFP4, training settings and quality target. By combining operations, reducing repeated memory access and changing scheduling, and exchanging weights in the four-bit form already used for calculations to reduce communication cost and avoid repeated conversion work, we reduced the time-to-train by 8.48 minutes.
AI agents generated candidates for engineers to verify
AI coding agents helped the team explore alternative kernel implementations. Drawing on published approaches to automated kernel optimisation, the engineers used a four-stage process to move from a performance bottleneck to a change tested in a complete training run:
1. Profile: Measure kernel performance and prioritise operations with room for improvement.
2. Generate: Use AI coding agents to produce alternative implementations for the selected operations.
3. Verify: Compare candidates with reference implementations for correctness and speed, checking mathematical equivalence where required.
4. Confirm: Put promising kernels into complete model training runs to establish whether the gains carry through to the benchmark’s quality target.
The stages formed an iterative process. Results from testing informed further changes, while engineers chose the tests and assessed what the verification established. A kernel that ran faster on its own still had to improve performance within the complete model.

Faster kernels had to preserve the required calculations

Mathematical equivalence means preserving the required calculation when changing how a kernel is implemented. The engineers checked replacement operations against reference implementations, alongside testing training against the quality target. These were separate requirements: reaching the final quality target alone did not establish that an individual replacement calculation was correct.
Required checks depended on the change. GPT-OSS included numerical-format changes, while Llama retained its existing precision recipe. Llama’s technical documentation records numerical edge cases and the conditions each replacement relies on. Operation-level correctness does not mean that separate training runs produce identical bits throughout.
Case study: combining memory passes reduced optimiser execution time
The GPT-OSS optimiser shows how the team reduced overhead while preserving a calculation. This part of training updates the model’s weights. The engineers changed how working data was stored and combined operations, reducing five passes through memory to two. Reported execution time fell from 31.4 to 12.3 milliseconds per iteration, with bitwise-identical results for this operation.

Firmus is exploring applications beyond the benchmark
Training time affects how quickly developers can test model changes and repeat experiments. For comparable workloads, shorter runs could allow more experiments within a given computing budget. The measured improvements are configurations and workload specific. Performance on other models, hardware and production workloads requires separate validation.
The team at Firmus is investigating how the methods could support model inference and workload-specific optimisation for internal AI systems.
Firmus will share the engineering detail in two forthcoming technical blogs and a video interview with Shaojun Zhang and Lewis Won.
Follow Firmus on LinkedIn for more updates.






