Distill yourself onto a small model

RLHF distills the crowd into big models. This distills you — your taste, your trade-offs, your moral instincts — into a small model that runs on hardware you own.

Play Twin Builder Eval suite on GitHub

The loop

Play. 60 rapid-fire this-or-that choices in the Twin Builder game (~5 minutes). The twin predicts each answer live.

Export. Download your answers as training data — SFT and DPO JSONL in the eval's exact format.

Fine-tune. Train a LoRA adapter on a small base model, on your machine or a rented GPU. Minutes, not hours.

Evaluate. Run the eval suite against base vs tuned. If tuning internalized your taste, the tuned model should match or beat the base model's best in-context score on held-out items.

Step 1 — Play the game

Open Twin Builder, pick a starting prior (your MBTI type, or blank slate — it's just a starting point), and tap through 60 choices. Answer fast and honestly; there are no right answers. A few questions repeat later on purpose — that measures how consistent you are with yourself, which is the ceiling your twin is scored against. Want more training data? Hit "keep going" for endless mode.

Step 2 — Export your data

The end screen offers three downloads. Know which is which:

twin-profile.json — the measurement: your fitted 10-trait vector plus stats. Good for inspection, not for training.
twin-sft.jsonl — the training data: one line per answer, {"messages": [{"role":"user","content": prompt}, {"role":"assistant","content":"A"}]}. Train on this.
twin-dpo.jsonl — same answers as preference pairs, {"prompt","chosen","rejected"}, for a DPO second stage.

Prompts are byte-identical to the eval suite's forced-choice format, so fine-tuned agreement is directly comparable to in-context scores. Consistency retests are excluded from training files. Your answers never leave your device until you download them.

Step 3 — Fine-tune

You're training a LoRA adapter (a small set of extra weights on a frozen base model), not the full model. With 60–500 short examples this is a minutes-scale job. Pick your platform:

A. Your Mac (Apple Silicon) — recommended

Private, free, no uploads. MLX is the native path.

pip install mlx-lm

# split into train/valid (hold out ~10 for validation)
mkdir -p twin-data
head -n 50 twin-sft.jsonl > twin-data/train.jsonl
tail -n 10 twin-sft.jsonl > twin-data/valid.jsonl

python -m mlx_lm.lora --model mlx-community/Qwen3-4B-Instruct-2507-4bit \
  --train --data twin-data --adapter-path ./twin-adapter \
  --iters 300 --batch-size 4
# run it:
python -m mlx_lm.generate --model mlx-community/Qwen3-4B-Instruct-2507-4bit \
  --adapter-path ./twin-adapter --prompt "Scenario: ..." --max-tokens 5

Check mlx_lm.lora --help for current flags. Any small instruct model with an MLX 4-bit build works (Qwen3, Llama 3.2, Gemma 3 in the 1–8B range).

B. Google Colab (free) — Unsloth + QLoRA

No Mac or no Apple Silicon? A free Colab T4 handles ≤8B models in 4-bit via Unsloth's turnkey notebooks:

New notebook → Runtime → Change runtime type → T4 GPU.
pip install per Unsloth's Colab instructions, load a 4-bit base (e.g. an Unsloth Qwen3/Llama 3 4-bit build), attach LoRA adapters.
Upload twin-sft.jsonl, format with the model's chat template, train with TRL's SFTTrainer, download the adapter.

Start from Unsloth's official Colab notebooks — they track library changes better than any static guide can.

C. Rented GPU — RunPod, Modal, Lambda Labs

For bigger models or faster turnaround, rent an A100/H100 by the hour and run the same Unsloth/TRL script you would on Colab. A short LoRA run like this typically costs a few dollars at current rates — check pricing before you start.

D. Fine-tuning APIs

Services like Together AI and Fireworks offer hosted fine-tuning: upload the JSONL, get back a model endpoint. Least setup, least control, and your data leaves your machine — the trade-off to weigh.

A note on Replicate. Replicate's turnkey fine-tuning is built around image models (LoRA trainers for Stable Diffusion / Flux). For LLM LoRA it isn't the push-button path — use B, C, or D above instead. Replicate is useful afterward if you want to host your tuned model behind an API.

Rough time estimates

For this workload specifically — 60–500 short examples, LoRA rank 8–16. Dataset size and step count dominate; these are ballparks, not benchmarks.

Setup	Rough time
MacBook M1–M3 (MLX), 3–8B model	~5–30 min
Mac Studio (MLX), 8B+	~5–20 min
Colab free T4 (Unsloth QLoRA)	~15–45 min
Colab Pro A100	~5–15 min
Rented A100 / H100	~5–15 min + setup

Step 4 — Evaluate

Clone the eval suite and compare base vs tuned on held-out items:

export OPENAI_BASE_URL=http://localhost:1234/v1
export OPENAI_API_KEY=lm-studio
# in-context baseline (no tuning): how well does the base model predict you from K examples?
inspect eval src/inspect_evals/tasks/daily_decisions.py@daily_decisions_predict \
  --model openai/<base-model-id> -T profile=twin -T k=30
# then the same task against your fine-tuned model -- agreement should meet or beat it

The eval items are held out from the game by design — the game draws from a separate 504-item capture pool, so a tuned model can't win by memorization. Score twins against your self-agreement rate (the retest ceiling), not against 100%.

Step 5 — Keep it learning (feedback mode)

Your twin doesn't freeze after the first fine-tune. On the game's end screen, Test your twin starts feedback mode: the twin shows its prediction first, and you agree or disagree with each one.

Agree — confirms the twin; it keeps learning from the answer.
Disagree — you pick the right answer. That correction becomes a DPO pair for free: the twin's wrong pick is the rejected response, yours the chosen one.

The end screen reports your twin's feedback accuracy, and the DPO download now includes your corrections alongside the base pairs. Re-tune on the accumulated file every so often — that's the continual loop: play, tune, judge, correct, re-tune.

Notes

Privacy: the local path (Mac + MLX, or LM Studio) never sends your answers anywhere. Your twin can live entirely on your hardware.
Data size: 60 answers is a start. Endless mode in the game grows the set; a few hundred diverse answers beats dozens.
DPO: optional second stage after SFT, using twin-dpo.jsonl.
Eval vs train: never train on the 240-item eval set — it's the held-out test.

Part of digital-twin-evals — eval suites for moral dilemmas, preferences, and individual digital twins.