← All projects
Research · In progress · 2026 · … min read

Llama 3.1 8B — SFT → DPO Pipeline

A full supervised fine-tuning → direct preference optimization loop run on a homelab RTX 4060 Ti, with rigorous evaluation at every step.

PyTorch Unsloth QLoRA MLflow W&B Claude Sonnet judge

The question

Most fine-tuning tutorials run SFT on a dataset, eyeball a few outputs, and call it done. That's not enough to say what fine-tuning actually does to a model. I wanted to run the full SFT → DPO loop on a model I could actually feel, and build an evaluation rig honest enough to catch regressions.

Setup

  • Model: Llama 3.1 8B Instruct, 4-bit NF4 via Unsloth.
  • Hardware: Single RTX 4060 Ti (16 GB) on my homelab.
  • SFT data: Curated Dolly-15k subset, filtered for instruction clarity.
  • DPO data: UltraFeedback preference pairs.
  • Tracking: MLflow for experiment metadata, Weights & Biases for loss curves and eval metrics.

Evaluation harness

This is the part I spent the most time on. An SFT that looks good on training loss can easily regress on everything a user cares about.

  • Held-out perplexity on a shard never seen during training.
  • Claude Sonnet as pairwise LLM-judge with explicit A/B position swap to control for position bias.
  • Mini-IFEval — a slimmed instruction-following benchmark.
  • 30-prompt qualitative probe covering refusal, reasoning, tone, safety, and edge cases.
  • Cross-checkpoint weight-delta diagnostics — what actually changed, layer by layer.

What happened

SFT phase regressed against the base RLHF-tuned Instruct model on the judge metric. The SFT model got more confident on Dolly-style tasks but traded off calibration and safety behavior from the base RLHF pass. This is consistent with what the literature says — RLHF gives you things that a narrow SFT distribution tends to erode — but it's a completely different thing to see the regression on your own curves.

I documented the regression honestly rather than hiding it. The DPO phase is queued next; the hypothesis is that preference data will restore some of what SFT traded away.

Why this matters for real work

A/B swap
Position-bias correction in LLM-judge
5
Independent eval signals per checkpoint
Honest
Documented regression, not hidden
The most valuable output of this project isn't a better Llama. It's a repeatable evaluation rig that tells the truth about what a fine-tune did — and the habit of writing down the regression when it happens.

Repo

Private on GitHub — happy to share on request. Walks through the full SFT run, logs, and eval outputs.