Marin Begins Open Training of a 535B-Parameter MoE Model
Marin has begun an open 535B-parameter MoE training run on 18.75 trillion tokens, with public code, logs, and scaling forecasts.
1. A 535B-Parameter Run Conducted in Public
Marin has started training Marin 535B-A23B, a mixture-of-experts language model with approximately 535 billion total parameters and 23 billion parameters active for each token. The project plans to process 18.75 trillion tokens using 11 NVIDIA GB200 NVL72 systems over roughly three months.
The published plan allocates 80% of the token budget to pretraining and 20% to midtraining. Post-training will follow, but Marin has not yet published a final post-training recipe or timetable. The project estimates the main run will require approximately \(2.7 \times 10^{24}\) floating-point operations.
Each GB200 NVL72 is a rack-scale system containing 72 Blackwell GPUs and 36 Grace CPUs. Eleven such systems represent a hardware footprint of 792 GPUs, although Marin’s expert-parallel implementation report describes a 64-GPU expert-parallel domain within each training rack.
The run is not a model release. As of August 24, training is in progress, so there are no final weights, benchmark results, or post-training evaluations to assess. Its immediate importance is the decision to expose a training effort of this scale while it happens, rather than publishing a technical report only after selecting successful results.
Marin’s public issue for the run was opened on August 18. It contains the operating plan, engineering risks, scaling methodology, context-extension proposal, and contingency procedures. A linked Weights & Biases report provides the project’s live tracking surface.
The announcement specifies 18.75 trillion tokens, while the GitHub issue title abbreviates the run as “18T tokens.” The more precise figure is the one used in the public voyage plan; the project has not presented the shorter title as a revised budget.
2. How the Sparse Model Is Organized
Marin 535B-A23B uses 48 transformer blocks. Every block combines an attention branch with a sparse MoE branch containing 384 routed experts. The router selects eight experts for each token, while two shared experts remain local and process every token independently of the routed path.
The model state is 6,144 values wide. Before routed activations are exchanged between GPUs, a latent projection compresses them to 3,072 values. Marin says this halves the width of the activation traffic passing through the expert-parallel all-to-all operation. The output is projected back to the model width before it is combined with the shared-expert path.
This transport problem is substantial. With 384 routed experts distributed across a 64-GPU expert-parallel domain, each GPU hosts six routed experts. Sending separately sized buffers for every expert would create difficult memory requirements and dynamic communication patterns.
Marin instead developed a fixed pooled-wave all-to-all implementation for JAX and XLA. A sender creates one fixed pool for each destination GPU rather than one buffer for every expert. Transfers occur in three sequential waves with identical array shapes, and expert identifiers are packed into the activation payload. This avoids a separate exchange of token counts or routing metadata.
The implementation uses two capacity limits. A sender capacity factor of 1.10 bounds the amount of traffic sent from one source to one destination, while a receiver capacity factor of 1.15 limits the rows assigned to an individual local expert. Assignments exceeding either fixed buffer are dropped and reported separately.
A 20-step, one-rack gate completed without an out-of-memory failure and recorded median throughput of 250,691 tokens per second from steps two through 19. Marin explicitly warns that this short test does not establish the eventual token-drop rate. It is an engineering qualification result, not a training-quality benchmark or a measurement of the full 11-rack run.
The report also documents failed configurations. A direct fixed expert-cell design produced an XLA memory estimate of 192.65 GiB and failed on a 123.49 GiB CUDA allocation. A six-expert receiver bank also ran out of memory, while splitting work into three waves kept only part of that bank active at once. These negative results are part of the project’s published design record.
3. A Scaling Ladder Preceded the Main Run
Before launching the 535B model, Marin trained a four-rung scaling ladder. It began with a 1.6B-parameter MoE activating 61 million parameters and trained on 48 billion tokens. The largest rung contained 27.7 billion total parameters, activated 1.2 billion parameters, and processed 926 billion tokens.
The ladder serves both as a forecast and as a diagnostic reference. Marin can compare the main run’s loss, gradient norms, token dropping, and evaluation trajectory with patterns observed at smaller scales. A material deviation can trigger an investigation before the full run consumes months of compute.
According to the project, the ladder costs about 1% of the main run’s compute. Marin considers that expense a form of risk control: it tests whether the selected architecture, data mixture, optimizer settings, and training horizon behave consistently as model size increases.
The project says an earlier ladder exposed gradient-norm growth above four as the token horizon expanded. That finding led to the use of logit z-loss. Subsequent ablations reportedly showed that some high-batch configurations could otherwise diverge during training.
The smaller runs do not guarantee that the 535B model will follow their forecast. Extrapolation remains one of the experiment’s central uncertainties. Their practical value is that Marin has published a baseline against which outsiders can evaluate interventions made during the approximately 100-day run.
Marin has also documented a contingency for infrastructure delays or lower-than-expected model FLOP utilization. During roughly the first quarter of the token budget, the default response would be to shorten the token horizon, adjust the data mixture, and retime the linear learning-rate decay so that it still reaches 5% of the peak rate at the revised endpoint. This means the announced schedule remains an operating plan rather than an immutable specification.
4. Long Context Depends on Solving Token Dropping
The model begins pretraining with a 4,096-token sequence length. Marin’s previous large run started at 8K, extended to 65K for one trillion tokens, and was scheduled for a later 262K extension. Returning to 4K gives each batch twice as many individual sequences as 8K, which should distribute tokens more evenly among experts.
That choice addresses a weakness in the current expert-parallel implementation. In earlier tests, token dropping increased from approximately 7% at 4K context to about 40% at 65K. Marin reports that its newer pooled-wave design drops roughly 3% at 4K, but it expects the rate at 65K could still be excessive.
Dropped assignments do not necessarily mean that the entire token disappears. The two shared experts continue processing every token, and the eight selected routed experts are in addition to that shared path. Marin says the shared experts provide a denser backbone when routed assignments exceed available capacity, but high dropping can still reduce the benefit of the sparse experts and change training behavior.
The project plans a one-to-two-day early cooldown approximately 10 to 20 days into the run. This branch is intended to supply a full-scale checkpoint for reinforcement-learning experiments and test how a longer context affects routing without altering the main training trajectory.
If that experiment is stable, the tentative context schedule moves from 4K to 8K halfway through training, from 8K to 65K at approximately 95%, and then to a targeted 262K phase near the end. These are conditional targets, not confirmed capabilities of a completed model.
Marin lists three alternatives if token dropping remains too high: adopt a dropless ragged all-to-all implementation, increase the capacity factor and accept the accompanying memory cost, or introduce sequence-level balancing. The last option could force experts to respecialize during training, so the project currently treats it as a fallback.
5. What Makes the Run an Open-Development Milestone
Marin distinguishes open development from releasing weights after training. Its standard workflow begins with a GitHub issue that records an experiment’s hypotheses and goals. The implementation is submitted as reviewable code, execution is linked to public telemetry, and analysis—including failed attempts—is returned to the issue.
For the 535B run, the public record already includes the hero-run issue, the live tracking report, the source repository, and a detailed account of the expert-transport implementation. The transport report separates measured results from engineering judgments and identifies which conclusions rest only on short profiling runs.
That record gives researchers a way to evaluate whether the scaling forecast survives contact with the full model. It also exposes decisions that final model cards often compress into a few lines: why the run starts at 4K context, where token assignments are dropped, how memory constraints changed the transport design, and what the team plans to do if training falls behind schedule.
Open documentation does not make the main experiment inexpensive to reproduce. Replicating an 11-rack run remains beyond the resources of most independent researchers. The more accessible artifacts are the code, smaller scaling runs, failure analyses, configuration choices, and live measurements that can be inspected or tested at reduced scale.
The project’s status must therefore be described narrowly: Marin has begun an openly documented frontier-scale training run. It has not yet demonstrated the final model’s quality, long-context behavior, post-training performance, or eventual artifact release.
Frequently Asked Questions
What does “535B-A23B” mean?
The model has approximately 535 billion total parameters, while about 23 billion are active for each token. Sparse routing selects eight of 384 routed experts in each transformer block.
Can the finished model be downloaded now?
No. Training is still underway, and Marin has not released final weights or post-training evaluations for this run.
Why does training begin with only 4K context?
A 4K sequence length places more separate sequences in each batch, improving expert balancing. Marin is testing longer contexts because earlier implementations experienced sharply higher token dropping at 65K.
How can researchers follow the run?
Marin provides a public GitHub issue, source repository, expert-parallel design report, and a linked Weights & Biases tracking page.
Is the 262K context length guaranteed?
No. The proposed extensions to 8K, 65K, and 262K depend on results from early cooldown and token-dropping experiments.
Sources
Share