LoRA Fine-Tuning Pipeline

A practical QLoRA fine-tuning pipeline for instruction-following language models, with deployment to Ollama via GGUF export.

Built for learning — every design decision is documented and the code is meant to be readable.

---

Table of Contents

• Overview
• Requirements
• Project Structure
• Quick Start
• Configuration Reference
• Data Preparation
• Training
• Evaluation
• Export to Ollama
• Key Concepts
• Troubleshooting

---

Overview

This project implements the full LoRA fine-tuning lifecycle:

Raw data  →  Prepare  →  Train (QLoRA)  →  Evaluate  →  Export (GGUF)  →  Ollama

Stack

Component	Library	Role
Training	Unsloth + TRL	~2× faster, ~50% less VRAM than vanilla PEFT
LoRA adapters	PEFT	Adapter management and inference
Models	HuggingFace Transformers	Model loading and tokenization
Quantization	bitsandbytes	4-bit NF4 quantization (QLoRA)
GGUF conversion	llama.cpp	Export to Ollama-compatible format
Experiment tracking	Weights & Biases	Optional — off by default

---

Requirements

Hardware

• NVIDIA GPU with at least 6 GB VRAM (tested on RTX 5070 Laptop, 8 GB)
• 16 GB system RAM recommended for the export step (CPU merge)

Software

• Python 3.10+
• CUDA 12.4+ (Blackwell GPUs require 12.4 minimum)
• nvidia-smi accessible in PATH

---

Project Structure

LoRA/
├── .env.example                # environment variable reference — copy to .env
├── .gitignore
├── CLAUDE.md                   # development guidelines and architecture notes
├── Makefile                    # workflow shortcuts
├── setup.sh                    # one-shot environment bootstrap
│
├── configs/
│   ├── train.yaml              # base config — single source of truth
│   └── exp_<name>.yaml         # one file per experiment (never overwrite)
│
├── data/
│   ├── raw/                    # original data files (never modify after creation)
│   │   └── sample.json         # invoice extraction demo dataset
│   ├── processed/              # generated train JSONL (git-ignored)
│   └── eval/
│       ├── eval.jsonl          # standard eval set (git-ignored, generated)
│       └── eval_hard.jsonl     # hand-crafted hard cases (committed)
│
├── src/
│   ├── prepare_data.py         # converts raw data to ShareGPT JSONL
│   ├── train.py                # QLoRA training entry point
│   ├── eval.py                 # base vs fine-tuned side-by-side comparison
│   ├── metrics.py              # parse rate, exact match, per-field accuracy
│   └── export.py               # adapter → merged model → GGUF → Modelfile
│
└── outputs/                    # checkpoints, adapters, GGUF files (git-ignored)
    └── exp_<name>/
        ├── adapter/                       # LoRA adapter weights (PEFT format)
        ├── merged/                        # merged base + adapter (HuggingFace format)
        ├── model.gguf                     # full-precision GGUF (intermediate)
        ├── Modelfile                      # Ollama model definition
        ├── eval_results.jsonl             # standard eval output
        └── eval_results_eval_hard.jsonl   # hard eval output

---

Quick Start

1. Bootstrap the environment

bash setup.sh
source .venv/bin/activate

This installs PyTorch (CUDA 12.8), Unsloth, TRL, PEFT, bitsandbytes, and all dependencies into a .venv virtual environment.

2. (Optional) Configure environment variables

cp .env.example .env
# Edit .env — all variables are optional, see comments inside

3. Run the demo pipeline

make demo          # prepare sample data → train → evaluate

4. Full workflow with your own data

make prepare INPUT=data/raw/my_dataset.json FORMAT=alpaca
make train   CONFIG=configs/train.yaml
make eval    SAMPLES=20
make export  QUANT=q4_k_m

---

Configuration Reference

All training configuration lives in configs/train.yaml. Create a copy per experiment — never overwrite a config that produced a good result.

model:
  name: "unsloth/Qwen2.5-3B-Instruct"  # HuggingFace model ID
  max_seq_length: 2048                  # max tokens per example
  load_in_4bit: true                    # QLoRA — required for 8 GB VRAM

lora:
  r: 16             # adapter rank — start here, increase only if underfitting
  alpha: 16         # scaling factor — keep equal to r (scale = 1.0)
  dropout: 0.0      # regularization — add 0.05 only if overfitting
  target_modules:   # which weight matrices to adapt
    - q_proj
    - k_proj
    - v_proj
    - o_proj
    - gate_proj
    - up_proj
    - down_proj

training:
  output_dir: "outputs/run_001"
  num_epochs: 3
  batch_size: 2
  gradient_accumulation_steps: 4  # effective batch = batch_size × accum_steps = 8
  learning_rate: 2.0e-4
  lr_scheduler: "cosine"
  warmup_ratio: 0.05
  weight_decay: 0.01
  seed: 42
  bf16: true          # Blackwell (RTX 5070+) supports bf16 natively
  logging_steps: 10
  save_steps: 100
  eval_steps: 100

data:
  train_file: "data/processed/train.jsonl"
  eval_file: "data/eval/eval.jsonl"
  format: "sharegpt"

wandb:
  enabled: false      # set to true and add WANDB_API_KEY to .env
  project: "lora-finetuning"
  run_name: "run_001"

Model Selection by VRAM

VRAM	Recommended model	Notes
6 GB	unsloth/Llama-3.2-1B-Instruct	Very fast iterations
8 GB	unsloth/Qwen2.5-3B-Instruct	Default — best balance at this VRAM
8 GB	unsloth/Phi-3.5-mini-instruct	Strong reasoning alternative
16 GB	unsloth/Meta-Llama-3.1-8B-Instruct	Significantly more capable
24 GB	unsloth/Meta-Llama-3.1-13B-Instruct	Near frontier quality

All unsloth/* models are pre-patched for optimal Unsloth performance.

---

Data Preparation

make prepare                                      # uses data/raw/sample.json
make prepare INPUT=data/raw/my_data.json FORMAT=alpaca

Input Formats

Alpaca — single-turn instruction tasks:

[{"instruction": "Summarize this text.", "input": "The quick brown fox...", "output": "A fox jumps over a dog."}]

input is optional — leave it empty "" for tasks without additional context.

ShareGPT — multi-turn conversations:

[{"conversations": [{"from": "human", "value": "What is 2+2?"}, {"from": "gpt", "value": "4"}]}]

CSV — tabular data with columns instruction, input (optional), output.

All formats are normalized to ShareGPT internally before training.

Data Quality Rules

• Lock the eval set before the first run — never add to data/eval/ manually after splitting.
• Prefer 500 clean examples over 5,000 noisy ones.
• For structured outputs (JSON, SQL), validate that every output is parseable.
• Remove near-duplicates — they inflate train loss without improving generalization.

---

Training

make train                                  # uses configs/train.yaml
make train CONFIG=configs/exp_01.yaml       # custom config

Running Experiments

cp configs/train.yaml configs/exp_high_rank.yaml
# Edit: lora.r → 32, training.output_dir → "outputs/exp_high_rank"
make train CONFIG=configs/exp_high_rank.yaml

The adapter is saved to outputs/<run>/adapter/ in PEFT format.

---

Evaluation

Standard eval

make eval                                      # 20 examples, configs/train.yaml
make eval CONFIG=configs/exp_01.yaml SAMPLES=50

Results are saved to outputs/<run>/eval_results.jsonl with fields: prompt, reference, base, finetuned.

Hard eval set

data/eval/eval_hard.jsonl contains hand-crafted challenging cases that expose model weaknesses not visible in standard eval:

Category	What it tests
Recipient is a natural person	No S.L. / S.A. suffix
Freelancer with role in parentheses	Pedro Ruiz (Diseñador Freelance)
Base + VAT broken out separately	Must return the grand total, not the base
Large amount without thousands separator	125000 EUR
Abbreviated month in English	Jun 10, 2024
English date and amount format	EUR 102,450.75, November 1, 2024
OCR-corrupted text	Em1sor, lmprenta, T0tal
Long institutional recipient name	Full government department names
Dense commercial preamble	Invoice data buried in a paragraph
No field labels — positional only	Pipe-separated with no keys
Two-digit year	12-11-24
Near-identical issuer and recipient	Same group, North vs. South entity
Minimal text, no labels	Comma-separated fields only
Female freelancer as issuer	Ambiguous role assignment
Discount + base + VAT	Must return the payable total after discount

make eval-hard                                 # configs/train.yaml
make eval-hard CONFIG=configs/exp_01.yaml

Metrics

python src/metrics.py outputs/exp_01/eval_results.jsonl
python src/metrics.py outputs/exp_01/eval_results_eval_hard.jsonl

Sample output:

  Parse rate  fine-tuned : 100.0%  (15/15)
  Parse rate  base model : 100.0%  (15/15)

  Exact match (full JSON): 100.0%  (15/15)

  Precisión por campo:
                        base model    fine-tuned       delta
    ----------------  ------------  ------------  ----------
    emisor                  73.3%       100.0%      +26.7%
    receptor                73.3%       100.0%      +26.7%
    importe_total            0.0%       100.0%     +100.0%
    fecha                    0.0%       100.0%     +100.0%

What to watch:

Signal	Meaning
importe_total / fecha delta near +100%	Model learned normalization — the core fine-tuning goal
Parse rate < 100% for fine-tuned	Output format not learned — check training data format
Exact match stalls across experiments	Failing example needs targeted training data, not more generic data

---

Export to Ollama

Prerequisites

llama.cpp must be available — it does not need to be inside the project directory. export.py searches in order:

Priority	Location
1	$LLAMA_CPP_PATH — set in .env for non-standard paths
2	./llama.cpp — project root (default clone location)
3	../llama.cpp — one level up
4	~/llama.cpp — home directory

git clone https://github.com/ggerganov/llama.cpp --depth=1
pip install -r llama.cpp/requirements.txt

Export

make export                                    # q4_k_m, configs/train.yaml
make export CONFIG=configs/exp_01.yaml QUANT=q5_k_m

If the llama-quantize binary is not compiled (cmake not available), use Ollama's built-in quantization instead:

# Generate the merged model and Modelfile (skips quantize step on error)
make export QUANT=q4_k_m

# Let Ollama quantize on import
ollama create my-model --quantize q4_K_M -f outputs/exp_01/Modelfile

Quantization options

Format	Size (3B model)	Quality	Recommendation
q4_k_m	~2.5 GB	Minimal loss	Default — best balance
q5_k_m	~3.0 GB	Very low loss	When output quality matters more
q8_0	~5.0 GB	Near-lossless	Benchmarking
f16	~6.2 GB	No loss	Intermediate step only

Import and run

ollama create my-model -f outputs/exp_01/Modelfile
ollama run my-model

# Or via REST API
curl http://localhost:11434/api/generate -d '{
  "model": "my-model",
  "prompt": "Your prompt here",
  "stream": false
}'

Note: Ollama cannot load raw LoRA adapters. The export step permanently merges the adapter into the base model weights. The original adapter in outputs/<run>/adapter/ is not affected.

---

Key Concepts

What is LoRA?

LoRA (Low-Rank Adaptation) adds small trainable weight matrices alongside the frozen pre-trained weights. Instead of updating all ~3B parameters, LoRA trains only ~1–3% of them by decomposing weight updates into two low-rank matrices A (random init) and B (zero init), so the adapter output starts at zero:

Output = W₀x + ΔWx = W₀x + BAx
         frozen      trainable (rank r << d)

What is QLoRA?

QLoRA combines LoRA with 4-bit NF4 quantization. The base model loads in 4-bit (~2.5 GB for 3B), LoRA adapters stay in bf16. This makes 7–13B models trainable on consumer GPUs.

Why Unsloth?

Unsloth reimplements attention kernels in Triton/CUDA, achieving ~2× training throughput and ~50% less VRAM than vanilla PEFT — with identical outputs. Training uses the standard SFTTrainer API.

The targeted data lesson

More data does not always fix a specific failure. In the included invoice extraction task, going from 9 to 55 training examples left one OCR pattern (l vs I confusion) unsolved. Adding 5 targeted examples for that specific pattern fixed it completely. When a model consistently fails one category: diagnose first, then add targeted examples for that category.

---

Troubleshooting

CUDA out of memory during training

• Reduce batch_size to 1, increase gradient_accumulation_steps to 8
• Reduce max_seq_length to 1024
• Remove MLP modules (gate_proj, up_proj, down_proj) from target_modules
• Try a smaller base model

CUDA out of memory during export

• The merge runs on CPU — ensure system RAM ≥ 2× model size in fp32 (~24 GB for 3B)

torchao / trl import error at startup

• Cause: transformers v5+ installs torchao, which requires torch >= 2.7.
• Fix: pip install "torch==2.11.0+cu128" torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128

peft import fails with BloomPreTrainedModel error

• Cause: llama.cpp/requirements.txt may downgrade transformers to a version incompatible with peft.
• Fix: if the merged model already exists at outputs/<run>/merged/, run the GGUF conversion directly — the merge does not need to repeat. Otherwise, upgrade peft after installing llama.cpp requirements.

RTX 5070 not listed in PyTorch CUDA capabilities

• You installed a cu124 or cu126 build — compiled without Blackwell (sm_120) support.
• Fix: use the cu128 wheel (see setup.sh).

unsloth-zoo requires torch<2.11 warning

• Outdated metadata constraint. Safe to ignore.

Fine-tuned model is worse than base

• Check the eval set is not contaminated with training examples
• Reduce learning_rate to 5e-5
• Reduce num_epochs — likely overfitting
• Inspect training data for inconsistent outputs or near-duplicates

Model consistently fails one category despite adding more data

• Adding more data of the same type will not help. Identify the exact failure with python src/metrics.py, add 3–5 targeted examples covering that specific pattern, and retrain.

convert_hf_to_gguf.py not found

• Clone llama.cpp: git clone https://github.com/ggerganov/llama.cpp --depth=1
• Or set LLAMA_CPP_PATH=/path/to/llama.cpp in .env

W&B not logging

• Set wandb.enabled: true in the config YAML
• Add WANDB_API_KEY to .env and run wandb login