Skip to content
All projects

ML · NLP · 2025

Parameter-Efficient Adaptation of GPT-2

Implemented a GPT-2-style Transformer and compared lightweight adaptation methods across sentiment classification, paraphrase detection, and conditional sonnet generation.

LoRA and ReFT approached or exceeded full fine-tuning on short-form sentiment while updating a fraction of the parameters, but the advantage did not transfer uniformly across longer classification inputs and generation tasks.

Role
Machine Learning Engineer
Team
3
Backbone
GPT-2 small
Tasks
3
Report (PDF)
Bar chart comparing peak GPU memory usage across fine-tuning methods on SST
InputFrozen backboneSelected interventionTask output
One pretrained backbone, multiple intervention strategies, and three downstream tasks.
0.0178%

Trainable parameters, highlighted SST ReFT run

0.526

Best individual ReFT run on SST

61.9%

Memory reduction under the selected CFIMDB configuration

43.11

Development chrF, not classification accuracy

3

Downstream NLP tasks

Metrics come from different task-specific configurations and should not be interpreted as one combined model result.

Overview

How much of a language model actually needs to change?

  1. Frozen encoder

    Last linear layer

  2. Input adaptation

    Soft prompts

  3. Representation adaptation

    ReFT

  4. Weight adaptation

    LoRA

  5. Full-model adaptation

    Full fine-tuning

We compared whether task behavior could be changed by modifying only the classifier, learned inputs, hidden representations, low-rank attention updates, or the entire model.

For generation, can inference-time decoding decisions matter more than additional training?

The central question was not whether GPT-2 could be adapted, but where adaptation should occur — and how much of the model actually needed to change.

Backbone

What we implemented

The team implemented this decoder-only architecture and loaded pretrained GPT-2 weights into it — the pretrained language model itself was not trained from scratch.

  1. Token + positional embeddings

  2. Transformer block × 12

    Pre-LayerNorm → causal multi-head self-attention → residual → Pre-LayerNorm → feed-forward → residual

  3. Final LayerNorm

  4. Autoregressive LM head

    Tied to the input embedding weights

Architecture
Decoder-only Transformer
Attention
Causal multi-head self-attention
Normalization
Pre-LayerNorm
Embeddings
Token and positional embeddings
Output
Tied input/output embedding weights
Classification representation
Final non-padding token
Generation
Autoregressive next-token prediction
Optimizer
AdamW
Model selection
Best development checkpoint
Random seed
11711 for primary experiments

Methods

Where does each method modify the model?

Input embeddings
Attention projections (× 12 layers)
Hidden representations (× 12 layers)
Task classifier / output head
Output-time candidate selection

Full fine-tuning

Where
All model parameters
Mechanism
Update the entire pretrained backbone and task head.
Strength
Maximum adaptation capacity.
Cost
Highest trainable-parameter count and GPU-memory use.

Tasks

One backbone, three different adaptation problems

Sentiment classification

Datasets

SST, CFIMDB

Objective

  • 5-class sentence sentiment
  • Binary movie-review sentiment

Compared

  • Last linear layer
  • Full fine-tuning
  • LoRA
  • ReFT

Metric

Development accuracy

Paraphrase detection

Dataset

Quora question pairs

Objective

  • Binary paraphrase classification

Compared

  • Full fine-tuning
  • LoRA
  • ReFT
  • Soft prompt tuning

Metric

Development accuracy and final full-model test accuracy

Parameter-efficient experiments used a 25% training subset; the full-model result used the complete training set.

Conditional sonnet generation

Dataset

Shakespeare sonnets

Objective

  • Continue a sonnet from its first three lines

Compared

  • Full fine-tuning
  • LoRA and ReFT experiments
  • DPO
  • Hyperparameter tuning
  • Sampling and reranking

Metric

chrF

SST — 5-class sentiment
Train: 8,544Dev: 1,101Test: 2,210
CFIMDB — Binary sentiment
Train: 1,701Dev: 245Test: 488
Quora — Paraphrase detection
Train: 283,003Dev: 40,429Test: 80,858
Sonnets — Conditional generation
Train: 131Dev: 12Test: 12

Experimental workflow

  1. Source dataset
  2. Tokenize & batch
  3. Load shared pretrained backbone
  4. Apply adaptation method
  5. Train for up to 10 epochs
  6. Select best development checkpoint
  7. Evaluate performance
  8. Record trainable parameters & GPU memory

Classification

  1. 1.Final-token representation
  2. 2.Linear head
  3. 3.Accuracy

Generation

  1. 1.Next-token loss
  2. 2.Autoregressive decoding
  3. 3.chrF

DPO

  1. 1.Construct synthetic preferred/dispreferred pairs
  2. 2.Preference loss
  3. 3.Generate on held-out prompts
  4. 4.chrF

Test data was not used for hyperparameter selection.

Efficiency

Sentiment classification: performance versus efficiency

Representative best-run configurations, not five-seed means (see the multi-seed analysis below).

Accuracy versus trainable parameters on SSTLast linear layer: 0.0031% trainable parameters, 0.467 accuracy, 681 MB peak GPU memory. Full fine-tuning: 100% trainable parameters, 0.520 accuracy, 4866 MB peak GPU memory. ReFT: 0.0178% trainable parameters, 0.526 accuracy, 2398 MB peak GPU memory. LoRA: 0.1209% trainable parameters, 0.519 accuracy, 2862 MB peak GPU memory.0.4470.4720.4970.5210.5460.001%0.01%0.1%1%10%100%Trainable parameters (%, log scale)Last linear layerFull fine-tuningReFTLoRA

Point size represents peak GPU memory — larger points used more memory.

Last linear layer
Trainable parameters: 0.0031%Peak GPU memory: 681.36 MBAccuracy: 0.467
Full fine-tuning
Trainable parameters: 100%Peak GPU memory: 4865.96 MBAccuracy: 0.520
ReFT
Trainable parameters: 0.0178%Peak GPU memory: 2398.49 MBAccuracy: 0.526
LoRA
Trainable parameters: 0.1209%Peak GPU memory: 2861.55 MBAccuracy: 0.519
  • On SST, the highlighted ReFT run slightly exceeded the full-model baseline while updating 0.0178% of parameters.
  • On CFIMDB, full fine-tuning retained the highest accuracy.
  • ReFT reduced CFIMDB peak GPU memory from 2,710.30 MB to 1,032.63 MB.
  • The last-linear-layer baseline was cheapest but lost substantial accuracy.

Layers

Where you intervene mattered more than how large the intervention was

Early — layer 2
SST: 0.507CFIMDB: 0.976
Middle — layer 5
SST: 0.504CFIMDB: 0.967
Late — layer 11
SST: 0.480CFIMDB: 0.873
Distributed — layers 1, 6, 11
SST: 0.526CFIMDB: 0.971
Late cluster — layers 8, 9, 10
SST: 0.507CFIMDB: 0.976
  • Distributed interventions produced the strongest individual SST result.
  • A single final-layer intervention performed poorly, especially on CFIMDB.
  • Placement produced larger changes than modest rank adjustments.
  • The result suggests — but doesn't prove — that sentiment information is developed across multiple stages of the network.

More adaptation capacity was not always better

Rank 8, α = 16

SST

0.517

CFIMDB

0.939

Rank 4, α = 8

SST

0.519

CFIMDB

0.971

Rank 8, α = 32

SST

0.509

CFIMDB

0.820

Rank 16, α = 32

SST

0.515

CFIMDB

0.898

Moderate rank and scaling produced the most stable performance, while larger update magnitudes degraded CFIMDB accuracy.

Paraphrase

Parameter efficiency did not transfer equally to every task

The full-model baseline and parameter-efficient methods were not trained on equal amounts of data, so this is not a perfectly controlled head-to-head comparison.

Full fine-tuning
Full Quora training set
Dev: 0.898Test: 0.859
LoRA
25% training subset
Dev: 0.856Test: Not reported
ReFT
25% training subset
Dev: 0.817Test: Not reported
Soft prompt tuning
25% training subset
Dev: 0.743Test: Not reported
  • Full-model fine-tuning produced the strongest result.
  • LoRA retained more performance than ReFT or soft prompts in the reduced-data experiment.
  • Input-level adaptation alone was insufficient under the tested soft-prompt configuration.
  • Longer or pairwise reasoning tasks may require greater adaptation capacity or better tuning — a hypothesis, not a settled conclusion.

Generation

For generation, inference strategy mattered more than another training objective

01

Train

  • Fine-tune GPT-2 on complete sonnets
  • Condition evaluation on the first three lines
  • Predict the continuation autoregressively
02

Sample

  • Temperature
  • Top-p
  • Candidate-pool size
  • Multiple continuations
03

Rerank

  • Repetition penalty
  • Incorrect line-count penalty
  • Structural-issue penalty
≈ 43.11

Best development chrF

42.220

Final test chrF

Best-of-10

Best candidate strategy

Best-of-5 and best-of-15 performed worse.

How reranking picks a continuation

Illustrative diagram of the mechanism — not real project output. Descriptors are representative structural scores, not recorded generations.

PromptCandidate A(14/14 lines · low repetition)Candidate B(14/14 lines · high repetition)Candidate C(12/14 lines · structural issue)Selected: Candidate A

When preference optimization made generation worse

Line shuffling after truncation
chrF: 41.30
Partial continuation replacement
chrF: 41.04
Full continuation replacement
chrF: 41.27
  • Synthetic corrupted continuations were used because the dataset did not contain human preference pairs.
  • None of the DPO configurations exceeded the best reranked full-model result.
  • Qualitative inspection found genre drift, repetition, prose-like narration, and text corruption.
  • The preference pairs likely did not encode the qualities most important for sonnet continuation.

A preference objective is only as useful as the preference data used to define it.

Statistics

Single best runs were not the whole story

ReFT, layer 2
SST: 0.5124 ± 0.0046CFIMDB: 0.9696 ± 0.0040
ReFT, layers 1/6/11
SST: 0.5148 ± 0.0078CFIMDB: 0.9670 ± 0.0057
LoRA
SST: 0.5216 ± 0.0053CFIMDB: 0.9666 ± 0.0135
Full fine-tuning
SST: 0.5080 ± 0.0105CFIMDB: 0.9808 ± 0.0018
Last linear layer
SST: 0.4630 ± 0.0060CFIMDB: 0.8586 ± 0.0108

SST significance (vs. full fine-tuning)

  • LoRA vs. full fine-tuningp = 0.0411
  • ReFT layer 2 vs. fullp = 0.4910
  • ReFT distributed vs. fullp = 0.2353

CFIMDB significance (vs. full fine-tuning)

  • LoRA vs. fullp = 0.0653
  • ReFT layer 2 vs. fullp = 0.0070
  • ReFT distributed vs. fullp = 0.0041
  • LoRA's SST mean was significantly higher than the full-model mean under the five tested seeds.
  • ReFT was statistically indistinguishable from full fine-tuning on SST.
  • Full fine-tuning retained a statistically significant advantage over ReFT on CFIMDB.
  • LoRA's CFIMDB difference did not meet the p < 0.05 threshold.
  • Five seeds are not enough for a highly robust statistical conclusion.

My role

My focus: paraphrase detection and generation

This was a three-person team project. ReFT, LoRA, and DPO were primarily implemented by other members of the team — the contributions below describe what I personally worked on.

Paraphrase detection

  • Implemented the GPT-2-based paraphrase-detection pipeline
  • Encoded question pairs for binary classification
  • Built and evaluated full-model and parameter-efficient experiments

Soft prompt tuning

  • Implemented learned continuous prompt embeddings
  • Integrated prompt positions with the frozen GPT-2 backbone
  • Evaluated soft prompting on Quora paraphrase detection

Sonnet generation

  • Implemented and improved the conditional sonnet-generation pipeline
  • Tuned training and decoding hyperparameters
  • Explored candidate generation and best-of-N reranking
  • Helped analyze why decoding improvements outperformed the tested DPO configurations

Communication

  • Contributed to experiment interpretation and the final technical report

Engineering judgment

Technical challenges and decisions

Challenge 1

One model had to support classification and generation

Decision

Use a shared decoder-only GPT-2 backbone and task-specific heads or objectives.

Lesson

A shared architecture makes method comparisons easier, but task behavior still differs substantially.

Challenge 2

Efficiency has multiple dimensions

Decision

Measure accuracy, percentage of trainable parameters, and peak GPU memory.

Lesson

Low parameter count does not automatically imply proportionally low memory use because activations and optimizer behavior still matter.

Challenge 3

ReFT required choosing intervention locations

Decision

Sweep rank and layer placement separately.

Lesson

Layer placement affected performance more strongly than modest rank changes.

Challenge 4

Likelihood training did not guarantee strong poetic structure

Decision

Tune sampling and rerank multiple candidates using lightweight structural penalties.

Lesson

For open-ended generation, inference-time selection can be as important as the training objective.

What worked and what did not

What worked

  • LoRA and ReFT offered strong SST performance with very small trainable parameter budgets
  • ReFT substantially reduced CFIMDB GPU-memory use
  • Distributed ReFT interventions performed well on SST
  • Moderate LoRA rank and scaling were more stable than aggressive configurations
  • Best-of-10 reranking produced the strongest sonnet-generation result
  • Multi-seed analysis revealed differences hidden by individual runs

What did not consistently work

  • The last-linear-layer baseline lost substantial accuracy
  • Soft prompting underperformed on the tested paraphrase configuration
  • Late-only ReFT intervention degraded CFIMDB performance
  • Larger LoRA scaling did not improve results
  • PEFT did not match full fine-tuning on the reduced-data paraphrase setting
  • Synthetic-corruption DPO did not outperform decoding-based reranking

What this experiment does — and does not — establish

Limitations

  • Experiments used GPT-2 small
  • Compute limited the number of random seeds
  • Hyperparameter sweeps were not exhaustive
  • Some task comparisons used different training-data amounts
  • Sonnet evaluation relied heavily on chrF
  • Synthetic DPO negatives may not reflect genuine human preferences
  • Results may not transfer directly to larger models
  • Development metrics were used for model selection

Next steps

  • Repeat experiments with more seeds
  • Equalize training data across paraphrase methods
  • Explore additional ReFT layer combinations
  • Tune LoRA rank and scaling more systematically
  • Compare wall-clock training time and energy use
  • Evaluate larger backbones
  • Use human evaluation for generation quality
  • Create preference pairs that explicitly encode meter, rhyme, structure, and semantic coherence
  • Compare learned rerankers with hand-designed penalties
  • Analyze whether task complexity predicts the best adaptation location

Tech & topics

  • Python
  • PyTorch
  • Transformers
  • GPT-2
  • LoRA
  • ReFT
  • Soft Prompt Tuning
  • DPO
  • Hyperparameter Tuning
  • Autoregressive Decoding
  • Statistical Analysis
PreviousBiosurveillance “Digital Immune System”NextVirtual Stenting in the Fontan Circulation