Outcome Instruction tuning completed successfully. The model is demonstrably more instruction-following, but benchmark and qualitative evidence show it remains an experimental small language model—not yet a generally reliable production assistant. |
Prepared 21 August 2026
Jhelum Labs / W-1.1 research program
Executive summary
W-1.1 Project Instruct converted the pretrained W-1.1-125M causal language model into an instruction-following checkpoint using 300,000 question–answer and task examples. The run completed on an NVIDIA H100 80GB GPU in approximately 60 minutes and produced a final validation loss of 1.951 (perplexity 7.03).
Indicator | Result | Interpretation |
|---|---|---|
Base model | 123,688,704 parameters | GPT-style decoder-only causal LM |
Instruction data | 300,000 examples | 285k train / 7.5k validation / 7.5k test |
Training | 8,906 optimizer steps, 2 epochs | Completed and checkpointed |
Final validation | Loss 1.951; perplexity 7.03 | Strong fit to held-out instruction format |
Full benchmark mean | 41.68% preferred accuracy | Mixed capability; weak reasoning consistency |
Deployment status | Research preview | Needs stronger evals, safety, serving, and monitoring |
Bottom line The training pipeline worked. The model learned the answer format and improved fluency, but its 125M-parameter capacity and data quality limit factual reliability, multi-step math, and robust reasoning. Use it for controlled demonstrations and research, not high-stakes answers. |
Report map
Section | Contents |
|---|---|
1 | Model and experiment definition |
2 | Dataset construction and governance |
3 | Instruction-training configuration |
4 | Training results and diagnostics |
5 | Qualitative generation tests |
6 | Benchmark evaluation |
7–8 | Risk assessment and recommended next phase |
Appendices | Artifacts, full metrics, and conclusion |
1. Model and experiment definition
The project starts from W-1.1-125M, a pretrained next-token predictor. Instruction tuning does not replace next-token prediction: it teaches the same objective on structured prompts where the desired continuation is an answer. At inference time, the model still predicts one token at a time, conditioned on the instruction and all previously generated tokens.
Component | Specification |
|---|---|
Architecture | Decoder-only Transformer / causal language model |
Parameters | 123,688,704 |
Vocabulary | 32,000 BPE tokens |
Context window | 2,048 tokens (instruction sequences prepared at 1,024) |
Hidden width | 768 |
Transformer blocks | 14 |
Attention heads | 12 |
Feed-forward width | 2,048 |
Core layers | RMSNorm, RoPE, SwiGLU, tied token embedding / LM head |
Pretraining corpus consumed | 2,825,248,272 tokens |
Base checkpoint | best.safetensors (~471.8 MB shown in Drive) |
Experiment objective
The objective was to produce an instruction-following variant capable of answering general questions, science prompts, basic mathematics, coding requests, classification, summarization, and creative tasks while retaining the pretrained language knowledge.
Pipeline
Stage | Action | Primary compute |
|---|---|---|
1 | Prepare and split 300k instruction examples | Local CPU |
2 | Stage dataset and base checkpoint from Google Drive to a persistent Modal Volume | CPU / network |
3 | Tokenize to fixed 1,024-token input and label binaries | CPU |
4 | Fine-tune the pretrained model with supervised next-token loss | NVIDIA H100 |
5 | Validate, checkpoint, and sync final outputs to Google Drive | GPU + CPU / network |
6 | Run qualitative generation and standardized benchmarks | GPU |
2. Dataset construction and governance
The final instruction corpus contains 300,000 examples. A deterministic split reserved 285,000 examples for training and 7,500 each for validation and testing. The mixture is dominated by OpenOrca, with targeted additions for general instruction following, code, science, and grade-school mathematics.
Source: dataset statistics.json produced by the W-1.1 instruction-data preparation pipeline.
Source | Examples | Share |
|---|---|---|
OpenOrca | 198,081 | 66.0% |
Alpaca Cleaned | 50,000 | 16.7% |
Python Instructions | 18,000 | 6.0% |
Dolly 15k | 14,774 | 4.9% |
SciQ | 11,672 | 3.9% |
GSM8K | 7,473 | 2.5% |
Category coverage
Category | Examples | Category | Examples |
|---|---|---|---|
Mixed | 198,081 | Science | 11,672 |
General | 50,000 | Math | 7,473 |
Code | 18,000 | Open QA | 3,673 |
Classification | 2,131 | General QA | 2,090 |
Brainstorming | 1,755 | Closed QA | 1,753 |
Information extraction | 1,500 | Summarization | 1,178 |
Creative writing | 694 |
Data-quality and contamination cautions
- OpenOrca supplies 66.0% of the corpus, so its style and any duplicated or benchmark-derived content can dominate learned behavior.
- GSM8K and SciQ were directly included in training; they should not be reported as clean independent benchmarks without a decontaminated holdout.
- Public instruction datasets vary in factuality, formatting, licensing, safety coverage, and duplication. A production release needs documented filtering and provenance checks.
3. Instruction-training configuration
Hyperparameter | Value | Purpose |
|---|---|---|
Sequence length | 1,024 | Fixed instruction/response training window |
Microbatch | 8 sequences | Per-forward-pass batch |
Gradient accumulation | 8 | Effective batch: 64 sequences / up to 65,536 tokens |
Epochs | 2 | Two passes through the 285k-example training split |
Peak learning rate | 1.0 × 10⁻⁵ | Conservative adaptation of pretrained weights |
Minimum learning rate | 1.0 × 10⁻⁶ | Cosine-decay floor |
Warmup | 3% | Stabilizes early optimization |
Weight decay | 0.05 | Regularization |
Gradient clipping | 1.0 | Limits unstable updates |
Validation / save interval | 250 steps | Frequent monitoring and resumability |
Evaluation batches | 50 | Held-out estimate per validation event |
Seed | 42 | Reproducibility control |
Precision / hardware | CUDA on NVIDIA H100 80GB HBM3 | High-throughput fine-tuning |
The run targeted 8,908 optimizer steps and completed 8,906. The two-step difference is operationally negligible and is consistent with sequence-count rounding or end-of-epoch batching. Checkpoints were committed throughout the run, enabling recovery.
Compute separation The notebook UI could remain on a small CPU kernel while the detached Modal function trained on H100. The notebook submitted and monitored the remote job; the GPU lived inside the training container. |
4. Training results and diagnostics
Source: Pasted instruction-training history and final completion metadata.
Validation loss declined smoothly from approximately 2.38 at step 250 toward 2.00 by step 8,750, followed by a final reported evaluation of 1.951. The overall trend is healthy: the model adapted steadily without a sustained validation reversal. Training loss fluctuated batch-to-batch, as expected for a heterogeneous mixture.
Training outcome | Recorded value |
|---|---|
Completed steps | 8,906 |
Elapsed time | ~60 minutes |
Final validation loss | 1.9508458743 |
Final perplexity | 7.0346354822 |
Best periodic validation metric | 2.0000909448 at step 8,750 |
Final checkpoint | Jhelum Labs 125M-instruct-v1.safetensors |
Final checkpoint size | 593,068,584 bytes (~565.6 MiB) |
Training status | Completed and synchronized to Google Drive |
Perplexity interpretation
Perplexity is exp(loss). For the final loss, exp(1.9508459) ≈ 7.03. This means the model’s average uncertainty on the held-out instruction tokens is equivalent to choosing among roughly seven plausible next tokens. It is a useful optimization signal, but it does not directly measure truthfulness, safety, or question-answer accuracy.
Operational incidents resolved
Incident | Cause | Resolution |
|---|---|---|
Google Drive token not found | Lightning path was reused inside Modal | Upload token.json to /root and point the script to that location |
Function disk request out of bounds | Modal disk value below current allowed minimum | Increase ephemeral disk request to the platform-supported range |
CUDA Graph tensor overwrite | torch.compile CUDA Graph interaction across accumulated steps | Disable CUDA Graphs / use safe compile configuration |
torch.compile options conflict | mode and options supplied together | Specify only one configuration path |
Checkpoint key mismatch | Instruction model used abbreviated norm names | Restore original attn_norm / ffn_norm module names and tied-head handling |
5. Qualitative generation tests
Post-training prompts showed clear paragraph-level fluency and instruction-style formatting. The model generated topical continuations for artificial intelligence, education, and narrative prompts. However, the samples also exposed hallmark small-model weaknesses: unsupported citations, repetition, semantic drift, and unreliable arithmetic.
Test area | Observed behavior | Assessment |
|---|---|---|
General expository text | Maintained topic and article-like structure; invented MIT/Nature-style claims | Fluent, not reliably factual |
Education prompt | Produced headings and bullet-like advice; grammar and redundancy remained | Useful formatting, weak editing |
Creative continuation | Produced a coherent scientist narrative before drifting into unrelated drug-making language | Local coherence, poor long-range control |
Discount arithmetic | Repeated the price, confused percentages, and failed to complete the calculation | Unreliable multi-step reasoning |
Instruction format | Generally returned an answer continuation rather than raw web-style prose | Instruction tuning worked |
Qualitative verdict W-1.1 can demonstrate an answer-oriented interface, but outputs require external verification. It should not be positioned as “fully accurate” or equivalent to a modern production assistant. |
6. Benchmark evaluation
Two evaluation passes were observed. The first used small samples and produced wide uncertainty. The final run evaluated the complete benchmark sets and is the authoritative scorecard. Preferred accuracy uses normalized accuracy where provided, otherwise raw accuracy.
Source: W-1.1 smoke benchmark summary and full lm-evaluation-harness result file.
Source: Full benchmark scores; reference baselines are random choice except BoolQ majority-class baseline (approx. 62.2%).
Benchmark | Raw acc. | Preferred | Std. error | Reading |
|---|---|---|---|---|
ARC Easy | 44.78% | 40.70% | ±1.01 pp | Above 25% chance |
ARC Challenge | 20.22% | 24.40% | ±1.26 pp | Near 25% chance |
HellaSwag | 27.45% | 28.33% | ±0.45 pp | Slightly above 25% chance |
PIQA | 60.17% | 58.98% | ±1.15 pp | Above 50% chance |
WinoGrande | 50.43% | 50.43% | ±1.41 pp | Near 50% chance |
BoolQ | 59.30% | 59.30% | ±0.86 pp | Below ~62.2% majority baseline |
OpenBookQA | 17.60% | 29.60% | ±2.04 pp | Above 25% chance |
The unweighted mean preferred score is 41.68%. This average is a convenient report-card summary, not an official aggregate benchmark because the tasks have different sizes, baselines, and difficulty.
What the scores say
- Strongest relative result: PIQA at 58.98%, indicating some practical commonsense discrimination.
- Moderate result: ARC Easy at 40.70%, showing partial elementary science knowledge.
- Weakest areas: ARC Challenge, HellaSwag, WinoGrande, and OpenBookQA remain close to simple baselines.
- BoolQ accuracy is below the dataset’s approximate majority baseline, so the raw percentage should not be interpreted as strong yes/no reasoning.
- The full-run results are lower than several smoke estimates, demonstrating why small-sample scores should not be used for release claims.
7. Risk and production-readiness assessment
Dimension | Status | Evidence / gap |
|---|---|---|
Training pipeline | Green | Resumable run completed; artifacts synchronized |
Instruction following | Amber | Answer-oriented behavior learned, but inconsistent |
Language fluency | Amber | Readable short-form output with repetition and drift |
Factual reliability | Red | Hallucinated sources and claims observed |
Math / reasoning | Red | Basic discount problem failed |
Benchmark strength | Amber–Red | 41.68% mean; several tasks near baseline |
Safety alignment | Red | No documented safety preference tuning or red-team suite |
Data governance | Amber–Red | Public mixture; provenance, licensing, and decontamination incomplete |
Serving stack | Amber | Weights exist; API, batching, quotas, monitoring still required |
High-stakes use | Red | Not suitable without retrieval, guardrails, and human review |
Production definition
A model becomes usable by people only after the weights are wrapped in a reliable product system. For W-1.1 this means a versioned inference service, tokenizer/config packaging, prompt template, output controls, authentication, rate limits, logging, latency tests, abuse safeguards, evaluation gates, rollback, and clear user-facing limitations.
Recommended release label W-1.1 Instruct v1 — experimental research preview. Do not claim GPT-2 parity or general-answer accuracy until direct, reproducible comparisons and safety evaluations are complete. |
8. Recommended next phase
Priority | Action | Success criterion |
|---|---|---|
P0 | Freeze and hash the final checkpoint, tokenizer, config, metrics, and dataset manifest | Reproducible release bundle with SHA-256 checksums |
P0 | Create a decontaminated evaluation set not represented in training | Reliable generalization estimate |
P0 | Build a 200–500 prompt human evaluation across factuality, math, code, safety, and refusal | Scored rubric with error taxonomy |
P1 | Clean and rebalance instruction data; reduce low-quality or duplicated OpenOrca dominance | Higher validation quality and fewer repeated patterns |
P1 | Add targeted supervised data for arithmetic, concise QA, grounding, and refusals | Improvement on held-out targeted tests |
P1 | Run preference tuning only after SFT data quality is stable | Better instruction adherence without benchmark regression |
P2 | Package an inference API with deterministic mode, sampling controls, monitoring, and quotas | Stable pilot endpoint |
P2 | Quantize and profile on target hardware | Measured memory, latency, and quality trade-off |
P2 | Publish a model card and known-limitations statement | Responsible research release |
Suggested acceptance gates
- No regression on the seven full benchmarks; statistically meaningful improvement on at least three.
- ≥90% format compliance on a held-out instruction suite.
- Substantial reduction in arithmetic and factual error rates in human review.
- Safety and abuse tests documented, including refusal behavior and leakage checks.
- End-to-end serving tests cover load, latency, failures, logging, rollback, and cost.
Appendix A. Artifact inventory
Artifact | Purpose | Location / status |
|---|---|---|
Jhelum Labs 125M-instruct-v1.safetensors | Final instruction-tuned weights | Google Drive project output; ~565.6 MiB |
best.safetensors | Best periodic checkpoint | Instruction output folder |
config.json | Architecture and tokenizer configuration | Packaged with checkpoints |
tokenizer.json | 32k BPE tokenizer | Base project / instruction bundle |
instruct-metrics.json | Training and validation history | Instruction output folder |
_TRAINING_COMPLETE.json | Completion marker and final metadata | Instruction output folder |
train/validation/test JSONL | Raw 300k instruction corpus splits | W-1.1-project-instruct/dataset |
tokenized binaries | 1,024-token inputs and response-masked labels | Persistent Modal Volume |
jhelum-labs-1.1-instruct-full-results.json | Full benchmark output | /vol/evaluation/ |
Appendix B. Full benchmark record
Task | Raw acc. | Raw SE | Norm. acc. | Norm. SE |
|---|---|---|---|---|
ARC Easy | 0.447811 | 0.010204 | 0.406987 | 0.010081 |
ARC Challenge | 0.202218 | 0.011737 | 0.244027 | 0.012551 |
HellaSwag | 0.274547 | 0.004454 | 0.283310 | 0.004497 |
PIQA | 0.601741 | 0.011422 | 0.589771 | 0.011476 |
WinoGrande | 0.504341 | 0.014052 | — | — |
BoolQ | 0.592966 | 0.008593 | — | — |
OpenBookQA | 0.176000 | 0.017048 | 0.296000 | 0.020435 |
Benchmark runtime: 2.47 minutes. Results file: /vol/evaluation/jhelum-labs-1.1-instruct-full-results.json.
Appendix C. Research conclusion
The W-1.1 instruction experiment is technically successful: a custom 123.7M-parameter pretrained model was converted into an answer-oriented checkpoint through a reproducible H100 workflow, and the final artifacts were preserved. The evidence does not support calling it a fully accurate or general production GPT. Its main value today is as a functioning research platform for data curation, fine-tuning, evaluation, quantization, and serving experiments. The most productive next investment is not simply more epochs; it is cleaner targeted data, uncontaminated evaluation, and a guarded inference system.
Source: User-supplied W-1.1 preprocessing, pretraining, instruction-training, generation, and benchmark logs dated 19–21 August 2026. All numerical claims in this report are derived from those project records.