ML · 2025
Implementing GPT-2 from Scratch
A decoder-only Transformer built in the GPT-2 style, adapted across sentiment analysis, paraphrase detection, and sonnet generation — comparing full fine-tuning to LoRA, ReFT, and DPO.
Role: Team of 3 — CS224N
Overview
For CS224N (NLP with Deep Learning), our team implemented a decoder-only Transformer in the GPT-2 style — token and positional embeddings, causal multi-head self-attention, pre-LayerNorm blocks with residual connections, and output/embedding weight tying — with support for loading pretrained GPT-2 weights. We study how one backbone adapts to three downstream tasks (sentiment analysis, paraphrase detection, and conditional sonnet generation) and compare standard fine-tuning against parameter-efficient methods (LoRA, ReFT) and preference alignment (DPO), measuring the tradeoff between task performance and trainable-parameter efficiency.
Highlights
- Implemented causal multi-head self-attention, pre-LayerNorm Transformer blocks, and embedding weight tying from scratch.
- Built end-to-end training and evaluation pipelines for sentiment and paraphrase classification on the GPT-2 backbone.
- Compared full fine-tuning against LoRA / ReFT and DPO alignment, analyzing the performance-vs-efficiency tradeoff.
Tech & topics
- NLP
- Transformers
- PyTorch
- LoRA
- DPO
- LLMs