Final Architecture

1. Executive Architecture Summary

W-1.1-125M is an English-only, decoder-only autoregressive Transformer designed for pretraining from random initialization. It contains 123,688,704 trainable parameters and uses a 32,000-token byte-level BPE vocabulary, 768-dimensional hidden states, 14 Transformer blocks, 12 attention heads, RoPE positional encoding, RMSNorm, SwiGLU feed-forward networks and tied input/output embeddings.

The base model learns next-token prediction from 2.5 billion English training tokens. It is trained in PyTorch using BF16 on an NVIDIA A100 40 GB GPU. Pretraining produces a base completion model; instruction following and conversational behavior are added later through a separately versioned supervised instruction-tuning stage.

2. Frozen Model Identity

3. End-to-End Architecture Flow

Figure 1. Token flow through the decoder-only W-1.1-125M architecture.

For an input sequence x₁…xₜ, the model predicts the distribution of xₜ₊₁ using only the current and earlier tokens. Strict causal masking prevents every position from attending to future tokens. Generation repeats this process one token at a time, using a key/value cache for efficiency.

4. Exact Architecture Configuration

5. Token Embedding and Input Representation

Each integer token ID indexes a learned embedding matrix with shape 32,000 × 768. The result is a sequence of 768-dimensional vectors. There is no learned absolute-position embedding table; positions are incorporated inside attention using rotary positional encoding.

5.1 Weight Tying

The input embedding and final language-model head share the same weight matrix. Weight tying saves 24,576,000 parameters, reduces memory use and ensures that input and output token representations remain linked. Checkpoint loading must preserve this shared relationship.

6. Decoder Block × 14

Each of the fourteen blocks uses pre-normalization and two residual sublayers in this exact order:

• RMSNorm on the residual stream.
• Causal multi-head self-attention with RoPE on queries and keys.
• Attention output projection followed by residual addition.
• Second RMSNorm.
• SwiGLU feed-forward transformation.
• MLP down projection followed by residual addition.

Residual-stream additions should remain numerically stable; main matrix multiplications use BF16, while accumulation and sensitive reductions may use FP32 where PyTorch kernels provide it.

7. Causal Multi-Head Self-Attention

The attention score for a query and key is scaled by 1/√64 = 1/8 before the softmax. RoPE rotates query and key coordinates according to token position. Values are not rotated. During inference, keys and values from previous tokens are stored in a per-layer cache.

8. Rotary Positional Encoding

9. RMSNorm

Every block contains two RMSNorm layers and the network ends with one final RMSNorm. RMSNorm has one learned scale parameter per hidden dimension and no bias. Epsilon is fixed at 1e-5. Pre-normalization keeps the residual path clean and supports stable deep optimization.

10. SwiGLU Feed-Forward Network

The feed-forward sublayer uses two parallel 768 → 2,048 projections. One branch is passed through SiLU and multiplied elementwise by the other branch. A 2,048 → 768 down projection returns the result to the residual width. All three linear transformations are bias-free.

11. Final Normalization and Output

After block 14, the model applies final RMSNorm and multiplies the normalized hidden states by the transposed token-embedding matrix. The result has shape batch × sequence × 32,000. Cross-entropy loss compares these logits with the next-token targets. Padding or deliberately masked positions must use ignore_index and must not contribute to the loss.

12. Exact Parameter Budget

13. Parameter Allocation Chart

Figure 2. Exact parameter allocation. Norm scales are too small to appear prominently beside the matrix weights.

The model name “125M” is a size-class label. The exact unique trainable parameter count is 123,688,704. Distorting dimensions merely to reach exactly 125,000,000 would provide no architectural benefit.

14. English Tokenizer Specification

Tokenizer QA must measure round-trip decoding, average tokens per English word, code and punctuation handling, number/date splitting, whitespace behavior, special-token collisions and stability across long documents.

15. Pretraining Dataset Architecture

Figure 3. Planned English-only mixture for the 2.5-billion-token training split.

15.1 Data Quality and Governance

• Use only sources with a documented license or permitted usage basis.
• Record source name, snapshot/version, retrieval date, document ID and processing history.
• Normalize encoding and remove corrupt, spam, boilerplate and extremely low-quality records.
• Remove exact duplicates and perform document-level near-duplicate detection before splitting.
• Filter credential dumps, secrets and sensitive personal data where feasible.
• Create train, validation and test splits at document level before token packing.
• Insert EOS or DOC_SEP between documents; never concatenate boundaries silently.
• Save shard manifests, counts, checksums and deterministic processing configuration.

16. Data Storage and Packing

The 32,000-token vocabulary fits in uint16 storage. A raw 2.5-billion-token stream occupies about 5 GB as uint16 before indexes, document metadata, validation/test sets and backups. Shards should be sized for sequential reading and fast checkpoint-resume positioning.

17. Training Hardware and Precision

18. Full Training Configuration

At 524,288 non-padding tokens per update, 4,769 optimizer updates process 2,500,329,472 tokens, slightly exceeding the 2.5-billion target. The final partial batch may instead be truncated if the sampler supports exact token accounting.

19. Pilot and Go/No-Go Gate

20. Checkpoint and Reproducibility Contract

• Save model weights in SafeTensors for distribution.
• Save optimizer, scheduler, scaler state if used, global step, tokens seen and RNG states for training resume.
• Store the exact model configuration and tokenizer beside every checkpoint.
• Record dataset-manifest version, shard hashes, source proportions, code commit and package versions.
• Maintain best-validation, latest-resumable and permanent milestone checkpoints.
• Verify exact resume with a deterministic short test before full pretraining.

21. Validation and Base-Model Evaluation

Benchmark targets are not fabricated in advance. The pilot and early checkpoints establish credible baselines; release gates then require measurable improvement and no unexplained late-run regression.

22. Instruction Tuning and Chat Model

The pretrained checkpoint is a base next-token model, not yet a reliable assistant. Production conversational behavior is created in a separate stage so the original base model remains preserved.

• Train on a carefully licensed English instruction/conversation dataset.
• Define and version one chat template with explicit system, user and assistant boundaries.
• Reserve or reuse control tokens consistently; never remap pretrained token IDs.
• Evaluate instruction following, summarization, question answering, refusal behavior and hallucination.
• Compare general-language loss before and after tuning to detect catastrophic degradation.
• Optionally run preference optimization only when reliable preference data and evaluation are available.

23. Production Inference Architecture

24. Reliability, Safety and Security

• Validate all request fields and reject oversized or malformed inputs.
• Apply authentication, authorization, rate limiting, timeouts and concurrency limits at the API layer.
• Do not log sensitive prompt content by default; define retention and redaction policy.
• Test prompt injection, unsafe requests, memorization, data leakage and denial-of-service patterns.
• Document that a 125M model has limited knowledge and reasoning and can hallucinate.
• Keep dependency versions pinned, scan container images and restrict checkpoint write permissions.
• Release model and data cards covering intended use, excluded use, data categories, evaluation and limitations.

25. Production-Ready Acceptance Gates

26. Implementation Module Layout

27. Frozen Configuration Contract

28. Final Frozen Blueprint

29. Research Basis

The design is informed by the following primary research. These references justify mechanisms and scaling choices; they do not guarantee a particular W-1.1 benchmark score.

• Vaswani et al. (2017), “Attention Is All You Need.”
• Radford et al. (2019), “Language Models are Unsupervised Multitask Learners.”
• Su et al. (2021), “RoFormer: Enhanced Transformer with Rotary Position Embedding.”
• Zhang and Sennrich (2019), “Root Mean Square Layer Normalization.”
• Shazeer (2020), “GLU Variants Improve Transformer.”
• Dao et al. (2022), “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness.”
• Hoffmann et al. (2022), “Training Compute-Optimal Large Language Models.”

Final Conclusion

W-1.1-125M v1.0 is now defined as an English-only, decoder-only PyTorch language model with 123,688,704 trainable parameters and a 2.5-billion-token pretraining budget. Its architecture combines RoPE, RMSNorm, SwiGLU, tied embeddings, causal attention and KV-cached inference. Training uses BF16 on an NVIDIA A100 40 GB, with a mandatory pilot before the full run. Production readiness is achieved only after data governance, reproducible pretraining, evaluation, instruction tuning, safety testing, optimization, API hardening and complete documentation have all passed their acceptance gates.