Project Insight

1. Executive Summary

Project Insight consolidates the evidence from four W-1.1 experiments: the Tiny Prototype, Project Tazi, Project Spark and Project Spectrum. It separates engineering success from language quality and uses only recorded measurements; unavailable values remain explicitly marked as not recorded.

Spectrum provides the strongest raw evidence because its JSONL log contains 80 unique evaluation steps after duplicate early records are resolved. Its final loss was 3.5749 training and 3.6465 validation. Its best validation loss was 3.6282 at step 19,000.

Across the programme, data quality and domain focus affected output quality more visibly than small differences in parameter count. Tazi learned broad English patterns from noisy web text but generated weak semantics. Spark achieved the best fluency inside its narrow story domain. Spectrum broadened domain coverage but produced dated encyclopedic style, repetition and factual-looking hallucinations.

2. Evidence and Method

Spectrum log cleaning rule: the file contains 106 lines but only 80 unique steps. Steps 250–6,500 appear twice because a restarted run appended to the same log. Project Insight keeps the later occurrence for each duplicate step and does not average two different runs.

3. Programme Progression

Figure 1. Experimental parameter progression. Tazi and Spectrum use 17,366,784 parameters; Spark is reported as approximately 17.34M.

4. Architecture Comparison

5. Dataset and Compute Comparison

5.1 Key Dataset Finding

Tazi and Spark were similar in model size, context and intended training length, yet their language quality differed sharply. Spark’s focused and simpler TinyStories distribution produced much lower validation loss and more coherent generation. This demonstrates that validation loss must be interpreted within each dataset rather than as a universal score across unrelated corpora.

Spectrum broadened the data distribution and consequently faced a harder prediction task. Its higher loss than Spark does not automatically mean that Spectrum is a worse general model; Spark’s very low loss is partly a consequence of its narrow, repetitive story domain.

6. Phase 1 — Tiny Prototype Findings

7. Phase 2 — Project Tazi Evaluation

Tazi established the first reproducible ~17M-parameter broad-web baseline. The preprocessing pipeline successfully encoded 500 MB without exhausting memory, and checkpoint resume recovered correctly after an interruption at step 15,500.

Figure 2. Tazi train and validation losses at the five milestones preserved in the Phase 2 report.

7.1 Tazi Generation Finding

Tazi learned spelling, punctuation and sentence-like structure, but prompt relevance and long-range meaning remained weak. Repetition and invented words appeared, especially in technical prompts. The experiment was technically successful but not ready for instruction tuning or production use.

8. Phase 3 — Project Spark Evaluation

Spark kept a similar compact architecture but moved to a clean, focused TinyStories corpus and a Tesla T4. It completed 20,000 updates and achieved the best recorded validation loss among the experiments, 1.4872.

Figure 3. Spark validation milestones reported in the Phase 3 document. Intermediate values are approximate statements from the report, not a reconstructed raw log.

Spark’s overall qualitative rating was approximately 7/10 inside the TinyStories domain. This must not be interpreted as a score for general knowledge, coding, factuality or instruction following.

9. Phase 4 — Project Spectrum Evaluation

The exact cleaned Spectrum history covers 80 evaluation points from step 250 through 20,000. The final train loss was 3.574862; final validation loss was 3.646481; final perplexity was 38.3395.

Figure 4. Exact Spectrum loss curves after resolving duplicated early steps by retaining the later occurrence.

10. Spectrum Optimization Diagnostics

Figure 5. Exact Spectrum learning-rate schedule and recorded gradient norm across the cleaned run.

The learning rate warmed up and then decayed toward approximately 3×10⁻⁵. Gradient norms remained finite throughout the preserved records. The curve contains normal fluctuations but no NaN, infinity or catastrophic gradient explosion.

10.1 Generalization and Late-Stage Behavior

The best validation result occurred at step 19,000, while the final validation loss at step 20,000 was higher by 0.0183. This is a small late-stage regression, not severe overfitting. Best-checkpoint selection correctly preserved the stronger step-19,000 model.

11. Cross-Experiment Numerical Comparison

Figure 6. Best recorded validation losses for the three BPE experiments. Values are not directly comparable across different datasets.

Figure 7. Perplexity derived from each experiment’s best recorded validation loss. Dataset difficulty strongly affects these values.

Comparison warning: lower loss does not automatically mean broader capability. Spark’s domain was simpler and more repetitive. Cross-domain model selection must combine held-out loss with fixed-prompt generation tests and domain-specific evaluation.

12. Generation Quality Comparison

Figure 8. Only Spark’s report preserved an overall numeric generation rating. Missing scores are shown as not recorded, not estimated.

12.1 Central Interpretation

The experiments show a progression from engineering validity to surface fluency, but none demonstrates reliable reasoning, factual accuracy or instruction following. The final model must therefore be evaluated as a base language model first; chat behavior should be added only after pretraining quality is established.

13. Overfitting, Underfitting and Stability

Underfitting was the larger programme-wide limitation. The ~17M models learned distributional style but lacked capacity and/or sufficient high-quality token exposure for stable semantics, factual knowledge and long-context behavior.

14. What Worked

• A complete tokenizer → preprocessing → training → checkpoint → generation pipeline.
• Memory-mapped uint16 token arrays for large processed datasets.
• Warm-up, cosine learning-rate decay, AdamW, mixed precision and gradient clipping.
• Best and latest checkpoints with interruption recovery.
• Focused clean data for rapidly improving domain-specific fluency.
• Fixed-prompt qualitative testing alongside numerical validation loss.
• Preserving the tokenizer and configuration with each checkpoint.

15. What Did Not Work or Remains Weak

• Judging capability from loss alone; low domain loss did not guarantee reasoning or broad knowledge.
• Noisy broad data at small scale produced grammatical but semantically weak completions.
• Missing explicit document boundaries and EOS stopping caused continuation into new documents or stories.
• Long generations amplified repetition, topic drift and entity confusion.
• Early experiment records did not always preserve complete raw logs, timing and GPU-memory metrics.
• Generated names, dates and facts looked plausible but were unreliable.
• The 256-token context restricted long-range learning and evaluation.

16. Requirements Derived for the Final Model

17. Decisions for Phase 6 — Project Blueprint

Phase 6 should now plan and freeze the final W-1.1 model. It must not automatically inherit conflicting numbers from older planning documents. In particular, the attached architecture draft specifies 32,000 vocabulary tokens and a 5B-token target, while later project discussion considered 40,000 vocabulary tokens and 2B training tokens. Project Blueprint must resolve this conflict using tokenizer tests, compute estimates and data availability.

18. Final Phase 5 Verdict

Final Conclusion

W-1.1 Project Insight completes Phase 5. The four experiments established a functioning research pipeline and revealed the decisive roles of data quality, domain breadth, model capacity, context length, checkpoint discipline and honest evaluation. Spark delivered the strongest narrow-domain fluency; Spectrum provided the richest exact training evidence; Tazi proved broad-data stability; and the Tiny Prototype verified the original pipeline. None of the experiments should be presented as a factual or instruction-following assistant. Their combined value is the evidence they provide for a disciplined final-model design.

The programme may now proceed to Phase 6 — W-1.1 Project Blueprint, where the final architecture, tokenizer, context, data mixture, token budget, compute plan and evaluation gates will be selected and frozen.

Appendix A — Data Integrity Record

Appendix B — Source Documents

• w-1.1-testing(2)(1).docx — Tiny Prototype guide and observations
• W-1.1-project-tazi(3).docx — Phase 2 training report
• W-1.1-spark(4).docx — Phase 3 completion report
• w-1.1-project-spectrum(5).docx — Phase 4 report
• w-1.1-training_log.jsonl — exact Spectrum training records
• w-1.1-projects(0).docx — earlier roadmap
• W-1.1-125M-Architecture(1).docx — preliminary final-model architecture draft