Research
Vitasta: A Recurrent-Core Hybrid Architecture
Vitasta is a language-model architecture designed to address six standing limitations of full self-attention Transformers: quadratic compute, hard context limits, context rot, weak state tracking, dependence on positional signals, and heavy data demands. Its backbone is a gated delta-rule recurrent layer paired with bounded retrieval memory and exact-match retrieval. Overall progress: 25.3% on the defined validation ladder.
Verified Steps, Not Faster Guesses
A program for mathematical rigor and deliberate reasoning in language models. We propose forcing mathematical reasoning into discrete, externally verified steps with backtracking on failure — addressing both the verification gap in mathematics and the reasoning-mode gap in current language models. A design document and validation plan; nothing has been trained or evaluated on a real model.
W-1.2 Architecture Specification
W-1.2 is Jhelum Labs' planned second-generation experimental language model, scaling the laboratory's W-1.1 work from approximately 124 million to 353 million trainable parameters while preserving a transparent, decoder-only causal Transformer design.
Project Instruct
W-1.1 Project Instruct converted the pretrained W-1.1-125M causal language model into an instruction-following checkpoint using 300,000 question–answer and task examples. The run completed on an NVIDIA H100 80GB GPU in approximately 60 minutes, achieving strong instruction-following behavior across diverse task types.
Final Report
This report consolidates the recorded W-1.1 experiment lineage, final dataset preparation, custom architecture, H100 pretraining logs, saved artifacts, qualitative text-generation tests, failure recovery, and the remaining production milestones for the W-1.1-125M model program.
Final Architecture
W-1.1-125M is an English-only, decoder-only autoregressive Transformer designed for pretraining from random initialization. It contains 123,688,704 trainable parameters and uses a 32,000-token byte-level BPE vocabulary, 768-dimensional hidden states, 14 Transformer blocks, and 12 attention heads.
Project Spectrum
Project Spectrum completed a 20,000-step pretraining experiment using a compact decoder-only Transformer. The reported loss began near 9.00 and finished at 3.69 for training and 3.71 for validation. The final train–validation gap of only 0.02 confirms no overfitting at this scale.
Project Insight
Project Insight consolidates the evidence from four W-1.1 experiments: the Tiny Prototype, Project Tazi, Project Spark, and Project Spectrum. It separates engineering success from language quality and uses only recorded measurements, with unavailable values explicitly marked as not recorded.
Projects Overview
Complete phase directory for the W-1.1 program: Tiny Prototype → Tazi → Spark → Spectrum → Reason → Blueprint → Foundry → Forge → Align → Launch. The 17M model is an experimental research model; the final W-1.1-125M will be newly initialized in Phase 8.
Project Spark
W-1.1 Spark was trained as a controlled Phase 3 experiment to measure how a compact decoder-only Transformer learns from a large, focused story corpus. The final model completed its full 20,000-step cosine-decay schedule on a Tesla T4, achieving a best validation loss of 1.4872.
Project Tazi
Phase 2 established a reproducible baseline for W-1.1 by training a compact decoder-only Transformer on a 500 MB subset of OpenWebText. The run demonstrated stable optimization, strong reduction in training and validation loss, and no clear overfitting at the 17M parameter scale.
Tiny Prototype
The W-1.1 tiny prototype is a small GPT-2-style decoder-only Transformer created to verify the complete language-model pipeline before committing GPU resources to the final 125M-parameter model. It is a pipeline validation test, not a production language model.
125M Architecture
W-1.1-125M is a modern, decoder-only autoregressive Transformer designed to be trained from scratch, using current architectural choices that make training and inference efficient: RoPE positional encoding, RMSNorm layer normalization, SwiGLU activations, and a 32,000-token BPE vocabulary.