1. Executive Summary
Phase 2 established a reproducible baseline for W-1.1 by training a compact decoder-only Transformer on a 500 MB subset of OpenWebText. The run demonstrated stable optimization, strong reduction in training and validation loss, and no clear overfitting. The model learned spelling, punctuation, paragraph structure, and broad English patterns, but its free-running text remains semantically weak and vulnerable to repetition.
Phase 2 verdict Training was technically successful. The resulting checkpoint is a valid research baseline, but it is not yet a reliable assistant or production language model. |
2. Model Identity and Objective
Item | Phase 2 specification |
|---|---|
Model name | W-1.1 |
Project folder | W-1.1-project-tazi |
Model type | Decoder-only causal language model |
Training objective | Next-token prediction |
Parameters | 17,366,784 |
Context length | 256 tokens |
Vocabulary | 8,000 BPE tokens |
Target hardware | NVIDIA GeForce GTX 1650, 4 GB VRAM |
3. Phase 2 Architecture
W-1.1 uses a GPT-style decoder stack. It converts token IDs into embeddings, applies repeated causal self-attention and feed-forward transformations, then predicts a probability distribution over the 8,000-token vocabulary.
Component | Configuration / role |
|---|---|
Token embedding | Maps each token ID to a learned vector |
Position embedding | Represents each position within the 256-token context |
Transformer depth | 8 decoder blocks |
Self-attention | Causal attention prevents access to future tokens |
MLP | Expands and projects hidden representations inside each block |
Residual connections | Preserve signal and stabilize optimization |
Layer normalization | Normalizes activations before attention and MLP operations |
Language-model head | Projects final hidden states to 8,000 token logits |
4. Dataset Preparation
The original preparation script failed because it attempted to hold all encoded tokens in memory. The replacement pipeline used bounded-memory chunks and two passes: the first counted tokens, and the second wrote train and validation binaries safely.
Dataset field | Recorded value |
|---|---|
Source | OpenWebText subset |
Source size | 500,000,000 bytes (500 MB) |
Total tokens | 133,878,233 |
Training tokens | 120,490,409 |
Validation tokens | 13,387,824 |
Storage dtype | uint16 |
Tokenizer vocabulary | 8,000 |
Fingerprint | 6d7d83f3f2764ff33cd204f57380033d15b12069169156e45e73b923d8c22c62 |
Prepared files
- data\processed\train.bin
- data\processed\val.bin
- data\processed\tokenizer.json
- data\processed\meta.json
5. Training Configuration
Setting | Value | Purpose |
|---|---|---|
Target steps | 20,000 | Meaningful first run covering about 34% of the training tokens |
Physical batch size | 1 | Fits 4 GB VRAM |
Gradient accumulation | 8 | Creates an effective update over 2,048 tokens |
Peak learning rate | 0.0002 | Stable from-scratch learning rate for the prototype |
Schedule | Warmup + decay | Starts gently and makes smaller updates later |
Precision | CUDA AMP | Reduces memory use and improves speed |
Checkpoints | Best + latest | Protects best validation state and allows resume |
python train.py --preset phase2 --steps 20000 --batch-size 1 --grad-accum 8 --learning-rate 0.0002 --device cuda
6. Training Results
Checkpoint in log | Train loss | Validation loss | Perplexity |
|---|---|---|---|
Step 1 | 9.0489 | 9.0438 | 8,466 |
Step 5,000 | 5.1967 | 5.3124 | 202.8 |
Step 10,500 | 4.7656 | 4.6842 | 108.2 |
Step 15,000 | 4.5154 | 4.4548 | 86.0 |
Step 19,200 | 4.4489 | 4.3268 | 75.7 |
Latest reported result At step 19,200, validation loss reached 4.3268, corresponding to perplexity ≈ 75.7. This was the best result supplied for this report; the exact step-20,000 result was not provided. |
7. Training Stability Analysis
- Training loss fell from 9.0489 to 4.4489 in the latest reported log.
- Validation loss fell from 9.0438 to 4.3268 and continued reaching new lows.
- Training and validation losses remained close, so there was no clear sign of overfitting.
- Gradient norms were finite and generally stable; no NaN or infinity failure appeared.
- The learning rate decayed correctly from its peak toward smaller late-stage updates.
- A manual KeyboardInterrupt at step 15,500 caused no damage; training resumed correctly at step 15,501 from phase2_latest.pt.
8. Text Generation Evaluation
The model learned surface-level English structure, but it did not yet learn reliable long-range meaning. Lower temperature and narrow top-k sampling sometimes intensified repetition because too few alternatives remained available.
Prompt | Observed behaviour | Assessment |
|---|---|---|
Artificial intelligence can | Produced sentence-like political/news text but drifted from AI | Partly readable; weak relevance |
Scientists discovered that | Repeated U.S. and government phrases | Grammatical fragments; repetition loop |
Python is a high-level programming… | Invented words and repeated short fragments | Poor technical understanding |
Once upon a time | Produced story-like sentences but inconsistent events | Some narrative structure; weak coherence |
Hello | Generated paragraphs, then repeated “pytical” | Surface fluency followed by collapse |
Representative generated excerpt
Model output “Artificial intelligence can be made on the government for our own freedom and the political position of the world...” The text is syntactically sentence-like, but the concepts are loosely connected and the prompt is not followed precisely. |
9. Why Perplexity Improved but Generation Remained Weak
Perplexity measures average next-token uncertainty under teacher-forced validation. Generation is harder: every sampled token becomes context for the next prediction, so small mistakes compound. Therefore, a validation perplexity near 75.7 can coexist with poor free-running text.
- The model has only 17.37M parameters, limiting knowledge and sequence modelling capacity.
- Only about 40.96M tokens are processed in 20,000 optimizer steps—roughly 34% of the training split.
- OpenWebText is varied and noisy, which is difficult for a small model.
- The 256-token context limits long-range information.
- Generation initially lacked top-p sampling, repetition penalties, n-gram blocking, and clean end-of-text stopping.
10. Phase 2 Conclusion
Area | Final judgment |
|---|---|
Data pipeline | Successful: 500 MB encoded safely without exhausting RAM |
Optimization | Successful: stable loss reduction and learning-rate decay |
Generalization | Successful baseline: validation improved with no clear overfitting |
Language structure | Partially successful: punctuation and sentence shapes learned |
Coherence | Not yet sufficient |
Repetition control | Not yet sufficient |
Instruction tuning readiness | Not ready; base generation should be improved first |
Final conclusion Phase 2 achieved its research objective: it proved that the W-1.1 training pipeline can prepare a real dataset, train stably on the GTX 1650, recover from interruption, and learn measurable English patterns. The saved phase2_best.pt checkpoint should be preserved as the OpenWebText baseline for future comparison. |
11. Recommended Next Work
Archive phase2_best.pt as the OpenWebText baseline. Improve decoding with top-p sampling, repetition penalties, three-token n-gram blocking, and end-of-text stopping. Then continue pretraining on clean, licensed human-written text while preserving the same tokenizer and architecture. Begin instruction tuning only after short completions are coherent and repetition is controlled.
Protected artifact Keep checkpoints\phase2_best.pt, tokenizer.json, meta.json, and this report together. They define the reproducible Phase 2 baseline. |