Report field | Value |
|---|---|
Project | W-1.1-project-spectrum |
Experiment phase | Phase 4 |
Training status | Complete |
Training steps | 20,000 optimizer steps |
Final train loss | 3.69 |
Final validation loss | 3.71 |
Author | Wasif |
Report date | 17 August 2026 |
1. Executive Summary
W-1.1 Project Spectrum completed a 20,000-step pretraining experiment using a compact decoder-only Transformer and Spectrum’s own processed dataset and tokenizer. The reported loss began near 9.00 and finished at 3.69 for training and 3.71 for validation.
The final train–validation gap is only 0.02. This is a strong sign that the run did not exhibit severe overfitting at the final recorded point. However, the generated samples still show repetition, topic drift, dated source style and unreliable factual details. Loss convergence therefore did not equal assistant-quality generation.
Phase outcome: COMPLETE. Spectrum successfully produced readable English and supplied important evidence for Phase 5 and the later design of W-1.1-125M.
2. Experiment Objective
- Train a broader-domain experimental model after the TinyStories-focused Spark phase.
- Measure optimization stability over 20,000 updates on an NVIDIA T4 GPU.
- Compare final training and validation behavior for signs of overfitting.
- Test whether the model can generate readable completions across education, science, government and narrative prompts.
- Extract practical lessons before planning the final W-1.1-125M model.
3. Confirmed Architecture and Training Configuration
Component | Spectrum setting |
|---|---|
Model type | Decoder-only causal Transformer |
Vocabulary size | 8,000 tokens |
Context length | 256 tokens |
Embedding width | 384 |
Transformer blocks | 8 |
Attention heads | 6 |
MLP expansion | 4 × d_model |
Dropout | 0.10 |
Micro-batch size | 16 sequences |
Gradient accumulation | 8 |
Effective tokens/update | 32,768 |
Maximum steps | 20,000 |
Peak learning rate | 3 × 10⁻⁴ |
Minimum learning rate | 3 × 10⁻⁵ |
Warm-up | 500 steps |
Optimizer | AdamW |
Weight decay | 0.10 |
Evaluation interval | 250 steps |
Evaluation batches | 50 |
Precision | FP16 mixed precision |
At 32,768 training tokens per optimizer update, a full 20,000-step run samples approximately 655.36 million token positions. Because batches are randomly sampled from memory-mapped data, this is processed-token exposure rather than a guarantee of 655.36 million unique tokens.
4. Loss Results
Metric | Recorded value | Interpretation |
|---|---|---|
Starting loss | ≈ 9.00 | Initial untrained-model uncertainty |
Final training loss | 3.69 | Model fitted the training distribution substantially |
Final validation loss | 3.71 | Validation remained close to training |
Absolute loss reduction | 5.31 | 59.0% reduction from the reported start |
Final generalization gap | 0.02 | Small final train–validation difference |
Train perplexity | 40.04 | Derived as exp(3.69) |
Validation perplexity | 40.85 | Derived as exp(3.71) |
Figure 1. Recorded loss milestones. The connecting line summarizes direction only; intermediate checkpoint values were not supplied.
Figure 2. Final training and validation loss differ by 0.02.
5. Perplexity and Generalization
Perplexity converts cross-entropy loss into a more intuitive scale using e^loss. Spectrum’s final estimated train perplexity is 40.04 and validation perplexity is 40.85. Lower is better.
Figure 3. Perplexity derived from the supplied final losses; values are rounded to two decimals.
The small validation gap suggests good final numerical generalization. It does not prove factual accuracy or long-range coherence. The generation samples show that those capabilities remain limited even when validation loss is close to training loss.
6. Text Generation Tests
Test prompt | Initial behavior | Main weakness |
|---|---|---|
Education is important because | Readable encyclopedic prose | Drifts into invented biography and dates |
Scientists study the natural world by | Grammatical religious/academic style | Repeats “book”, “read” and “workmanship” |
The government announced today that | Sustained dialogue and punctuation | Repetitive “state of things” dialogue |
In the quiet village, a traveler discovered | Coherent narrative opening | Drifts into town and architectural description |
7. Qualitative Generation Evaluation
The four generated samples were reviewed for surface grammar, prompt relevance, coherence, repetition and factual reliability. The scores below are a first qualitative assessment for Phase 5, not a standardized benchmark.
Figure 4. Initial qualitative review on a five-point scale.
Dimension | Finding |
|---|---|
Grammar and punctuation | Generally readable sentences, dialogue marks, names and dates. |
Prompt relevance | Usually follows the prompt initially, then moves to loosely related material. |
Long-range coherence | Meaning becomes unstable over longer completions. |
Repetition | Repeated nouns and phrases appear in science and government samples. |
Factual reliability | Names, dates, titles and claims are plausible-looking but cannot be trusted. |
Dataset style | Outputs resemble older encyclopedic, historical, literary and religious sources. |
8. Overfitting Analysis
Final numerical evidence: no severe overfitting is visible from the supplied endpoint values. Validation loss is only 0.02 above training loss, approximately 0.54% relative to the training loss.
Limitation: a reliable overfitting diagnosis requires the complete train-loss and validation-loss history. If validation loss fell and then rose while training loss continued downward, that pattern could reveal late-stage overfitting even though the final gap is small. The JSONL training log should be preserved for the combined Phase 5 report.
9. What Spectrum Successfully Learned
- English word formation, sentence boundaries and conventional punctuation.
- Longer prose structure across several prompt types.
- Dialogue formatting and formal expository style.
- Names, dates, abbreviations and citation-like patterns present in the dataset.
- A substantially better tokenizer–checkpoint match than the earlier corrupted local test.
10. Remaining Limitations
- Topic drift after the first few sentences.
- Phrase and concept repetition.
- Weak semantic consistency over long generations.
- Hallucinated biographical and historical details.
- Strong imitation of dated source material.
- No instruction tuning or reliable question-answering behavior.
- Short 256-token training context limits long-range modeling.
11. Lessons for the Final W-1.1 Model
Observation | Design implication |
|---|---|
Small final loss gap | Retain validation monitoring and best-checkpoint selection. |
Readable but drifting prose | Increase model capacity and context length; improve document boundaries. |
Dated/encyclopedic style | Rebalance and quality-filter the final dataset mixture. |
Repetition | Improve data quality and evaluate decoding separately from training quality. |
Unreliable facts | Treat the base model as a completion model; later add instruction tuning and retrieval where appropriate. |
Tokenizer dependence | Archive tokenizer, configuration and checkpoint as one inseparable model package. |
12. Phase 5 Evidence to Preserve
- logs/training_log.jsonl
- config/spectrum_config.json
- data/processed/meta.json
- tokenizer/spectrum_tokenizer.json
- checkpoints/spectrum_best.safetensors
- checkpoints/spectrum_latest.safetensors
- samples/spectrum_generation_samples.json
- GPU name, peak memory, total training time and throughput output from the notebook.
Final Conclusion
W-1.1 Project Spectrum completed its experimental purpose. Loss decreased from approximately 9.00 to 3.69 for training, while validation finished at 3.71. The 0.02 final gap indicates stable numerical generalization, and the model now generates readable English. The generation review also exposes clear limits in factuality, repetition, semantic continuity and topic control. These results should now be combined with Tiny Prototype, Tazi and Spark in the complete Phase 5 observation document before the final W-1.1-125M plan is designed.
Data Integrity Note
This report uses the confirmed Spectrum notebook configuration, the four supplied generation samples, and the supplied loss values. No intermediate loss checkpoints were fabricated. Once the full JSONL log is available, Figure 1 can be upgraded to the exact train-versus-validation curve across all evaluation steps.