How to Fine-Tune a Large Language Model

Step 1: Set Up Your Environment

Install the core libraries. The Hugging Face ecosystem provides everything you need for QLoRA fine-tuning. The key packages are: transformers for model loading, peft for LoRA adapter configuration, bitsandbytes for 4-bit quantization, trl for the supervised fine-tuning trainer, and datasets for data loading.

Create a virtual environment and install dependencies with pip: pip install torch transformers peft bitsandbytes trl datasets accelerate. Verify your GPU is accessible by running python -c "import torch; print(torch.cuda.is_available())", which should print True.

If you are using a cloud GPU service (Lambda Labs, RunPod, Vast.ai), the CUDA drivers are typically pre-installed. On your own hardware, make sure your NVIDIA driver version supports CUDA 12.x. Check with nvidia-smi, which also shows your available VRAM.

For larger models (70B+) or multi-GPU setups, you will also need deepspeed for distributed training. For 8B models on a single GPU, the basic stack above is sufficient.

Step 2: Prepare Your Training Data

Your training data needs to be formatted as conversations in the chat template format. Each example is a list of messages with roles (system, user, assistant) and content. The model learns to produce the assistant messages given the system and user messages as context.

A single training example looks like this in JSONL format:

{"messages": [{"role": "system", "content": "You are a technical support agent for Acme Software."}, {"role": "user", "content": "My dashboard is showing error code 5012."}, {"role": "assistant", "content": "Error 5012 indicates a database connection timeout. First, check that your database server is running. Then verify the connection string in Settings > Database > Connection URL. If the server is running and the connection string is correct, the issue is likely a firewall rule blocking port 5432."}]}

Aim for 500 to 2,000 examples for your initial fine-tune. Each example should be a complete, correct demonstration of the behavior you want. Remove personally identifiable information and fix any errors in the reference outputs before training. Inconsistencies in your training data become inconsistencies in your model.

Split your dataset into training (90%) and validation (10%) sets. The validation set must contain examples the model will never see during training so you can evaluate whether the model is generalizing or just memorizing. Save both as JSONL files where each line is one training example.

Diversity in your training data is critical. Include a range of input lengths, difficulty levels, and topic areas within your domain. If all your examples are similar, the model will only work well on inputs that look like your training data and fail on anything outside that narrow distribution.

Step 3: Load the Base Model with Quantization

QLoRA works by loading the base model in 4-bit precision, which reduces the memory footprint by approximately 75%. An 8B model that normally requires 16GB of VRAM in half precision fits in roughly 5GB when quantized to 4 bits, leaving plenty of room for the LoRA adapters and training state on a 24GB GPU.

The quantization configuration specifies 4-bit NormalFloat (nf4) data type, which is optimized for the weight distributions found in neural networks. Enable double quantization (quantizing the quantization constants themselves) for additional memory savings with negligible quality impact. Set the compute dtype to bfloat16 for the forward and backward passes.

Load the model from Hugging Face Hub using the model ID (e.g., "meta-llama/Llama-3.1-8B-Instruct") with your quantization config. This downloads the model weights and loads them directly in 4-bit format. The first load downloads several gigabytes, but subsequent loads use the cached weights.

Also load the tokenizer for your model. The tokenizer converts text to token IDs and back. Set the padding token to the end-of-sequence token if the model does not have a dedicated padding token, which is common for Llama models. Set padding side to "right" for training compatibility.

Step 4: Configure LoRA Parameters

LoRA injects small trainable matrices into specific layers of the model. The key parameters are:

Rank (r): Controls the size of the trainable matrices. Higher rank captures more complex adaptations but uses more memory and trains slower. Start with r=16, which works well for most tasks. Increase to 32 or 64 only if you find that r=16 does not capture the complexity of your task.

Alpha: A scaling factor that controls how much the LoRA weights influence the output. The effective scaling is alpha/rank. A common setting is alpha = 2 * rank (so alpha=32 with r=16), which provides a learning rate multiplier of 2.0. This makes the LoRA updates impactful enough to change behavior without being so large that they destabilize training.

Target modules: Which layers in the model get LoRA adapters. For Llama and similar architectures, target the query, key, value, and output projection layers in the attention mechanism (q_proj, k_proj, v_proj, o_proj). Some practitioners also include the MLP layers (gate_proj, up_proj, down_proj) for more thorough adaptation, at the cost of more trainable parameters.

Dropout: A regularization parameter that randomly zeros out LoRA weights during training to prevent overfitting. Set to 0.05 for datasets over 1,000 examples, or 0.1 for smaller datasets where overfitting is a greater risk.

With r=16 targeting attention layers, you are training approximately 20-40 million parameters out of 8 billion, which is roughly 0.3% of the model. This is enough to meaningfully change the model's behavior while keeping training fast and memory-efficient.

Step 5: Train the Model

Configure the training arguments. The most important settings are:

Learning rate: Start with 2e-4, which is the standard for LoRA fine-tuning. If training loss oscillates wildly, reduce to 1e-4. If it barely decreases, increase to 5e-4. Use a cosine learning rate scheduler that gradually reduces the rate during training.

Batch size: Set per-device batch size to 4 with gradient accumulation steps of 4, giving an effective batch size of 16. Adjust the per-device batch size down if you run out of VRAM. The effective batch size (per-device * accumulation steps) should stay in the 8-32 range for stable training.

Epochs: Train for 2-3 epochs on your dataset. One epoch means the model sees every training example once. More than 3 epochs risks overfitting, where the model memorizes training examples rather than learning general patterns. Watch validation loss: if it starts increasing while training loss continues decreasing, you are overfitting.

Max sequence length: Set to the maximum length of your training examples plus some margin. For most tasks, 2048 tokens is sufficient. Longer sequences use more VRAM per example, so keep this as short as your data allows.

Launch the training with the SFTTrainer from the trl library. It handles the training loop, logging, checkpointing, and evaluation automatically. Training loss is logged at each step and should show a steady downward trend. A typical training run on 2,000 examples with 3 epochs takes 2-4 hours on a single A100 or H100, and 4-8 hours on an RTX 4090.

Save checkpoints periodically (every 100-200 steps) so you can resume if training is interrupted or compare different checkpoints to find the optimal stopping point. The best checkpoint is not always the last one, sometimes stopping earlier produces better generalization.

Step 6: Evaluate and Deploy

Load the fine-tuned model (base model + LoRA adapter) and run it against your validation set. Compare outputs to the base model on the same inputs. Look for three things: does the model follow your formatting consistently? Does it use domain terminology correctly? Does it handle edge cases better than the base model?

Quantitative evaluation depends on your task. For classification, measure accuracy and F1 score. For generation, use BLEU, ROUGE, or task-specific metrics. For structured output, measure parse success rate (what percentage of outputs are valid JSON/XML). For subjective tasks, do a blind comparison where human evaluators rate outputs from both the fine-tuned and base model without knowing which is which.

Check for regression on general tasks. Run a few general-purpose questions through both the fine-tuned and base model. Some regression is normal and expected, but if the model cannot hold a basic conversation or produces gibberish on off-topic queries, you have overfit. Reduce the number of epochs, increase data diversity, or reduce the LoRA rank.

For deployment, you have two options. You can merge the LoRA adapter into the base model weights, producing a single model file that runs without any special loading. Or you can keep them separate and load the adapter at serving time, which lets you swap different fine-tunes in and out of the same base model. The merged approach is simpler for single-purpose deployments. The adapter approach is better if you maintain multiple fine-tuned variants.

Serve the model using vLLM, Ollama, or text-generation-inference for production deployments. These inference servers handle batching, quantization, and efficient GPU utilization automatically. For testing and small-scale usage, the Hugging Face transformers pipeline works directly but is not optimized for throughput.