Current position: Phase 3 — w-1.1-project-spark | 17M-parameter improvement model training on 500 MB TinyStories. |
Complete Phase Directory
Phase | Project name | Main purpose | Status |
|---|---|---|---|
1 | w-1.1-tiny-prototype | Build and validate the first small Transformer prototype, training pipeline, tokenizer workflow, checkpointing, and text generation. | Completed |
2 | w-1.1-project-tazi | Scale the prototype to about 17M parameters and study training behavior on a larger OpenWebText dataset. | Completed |
3 | w-1.1-project-spark | Improve the 17M model on 500 MB TinyStories for stronger coherence, stability, and reduced repetition. | Current |
4 | w-1.1-project-spectrum | Train and evaluate the 17M experimental model on balanced education, code, web, news, and research text. | Planned |
5 | w-1.1-project-reason | Experiment with instruction following, dialogue, reasoning behavior, and generation-quality improvements. | Planned |
6 | w-1.1-project-blueprint | Compare all experiments and freeze the final tokenizer, 125M architecture, and training configuration. | Planned |
7 | w-1.1-project-foundry | Collect, clean, deduplicate, tokenize, and validate the large final pretraining dataset. | Planned |
8 | w-1.1-project-forge | Pretrain a new W-1.1-125M model from random initialization using the finalized design and dataset. | Planned |
9 | w-1.1-project-align | Instruction-tune the model and improve helpfulness, dialogue quality, reliability, and safety. | Planned |
10 | w-1.1-project-launch | Complete evaluation, optimization, quantization, packaging, documentation, inference API, and release. | Planned |
Project Progression
Tiny Prototype → Tazi → Spark → Spectrum → Reason → Blueprint → Foundry → Forge → Align → Launch
Important: The 17M model is an experimental research model. The final W-1.1-125M model will be newly initialized in Phase 8 and will use the architecture, tokenizer, dataset, and training settings finalized in earlier phases.