Fine-Tuning Open Source Models: Llama, Mistral, and Qwen

Llama 3.1 (Meta)

Llama 3.1 is the most widely used open-source model family for fine-tuning. Available in 8B, 70B, and 405B parameter sizes, it offers a clear upgrade path from prototyping (8B) through production (70B) to frontier-competitive (405B). The 8B Instruct variant is the most common starting point for fine-tuning because it balances quality with resource requirements.

Strengths for fine-tuning: The Llama ecosystem is the largest of any open-source model. Every major fine-tuning library (PEFT, Unsloth, Axolotl) has first-class Llama support. The community has published thousands of fine-tuned Llama variants on Hugging Face, which means troubleshooting is easy because someone has likely hit and solved your problem already. Meta's licensing (Llama Community License) allows commercial use for organizations under 700 million monthly active users, which covers virtually every company.

Performance profile: Llama 3.1 8B scores competitively on MMLU (68.4), HumanEval (72.6), and GSM8K (84.5). It handles English tasks well and has decent multilingual support for major European languages. The 128k context window gives you room for long-context fine-tuning if needed, though most fine-tuning datasets use much shorter sequences.

Fine-tuning considerations: Llama 3.1 uses a standard transformer architecture with grouped-query attention, which is well-supported by all QLoRA implementations. The tokenizer uses a BPE vocabulary of 128,256 tokens. One specific detail: Llama's chat template uses <|begin_of_text|> and <|eot_id|> special tokens. Your training data formatter must apply this template correctly, or the model will not learn the expected conversation structure. The trl library handles this automatically when you specify the model name.

Mistral 7B (Mistral AI)

Mistral 7B v0.3 is a strong alternative to Llama 8B, particularly for tasks that require precise instruction following and structured output generation. Mistral models consistently outperform their parameter-count peers on instruction adherence benchmarks, which makes them excellent base models for fine-tuning agent tool-calling and structured data extraction tasks.

Strengths for fine-tuning: Mistral models use sliding window attention, which makes them more memory-efficient during inference on long sequences. The model's native function-calling support means fine-tuning for tool use builds on an already-strong foundation rather than teaching the capability from scratch. Mistral also supports European languages (French, German, Spanish, Italian) at a level that exceeds Llama's non-English performance.

Performance profile: Mistral 7B v0.3 matches or exceeds Llama 3.1 8B on most benchmarks despite having fewer parameters. It is particularly strong on MMLU (64.0 for 7B, which is impressive for the parameter count), and its instruction-following accuracy is consistently 2-3 percentage points above Llama 8B in head-to-head tests. The 32k context window is shorter than Llama's 128k but sufficient for the vast majority of fine-tuning use cases.

Fine-tuning considerations: Mistral's architecture includes sliding window attention and grouped-query attention, both supported by PEFT/QLoRA. The tokenizer uses a 32,768 vocabulary. Mistral's Apache 2.0 license is more permissive than Llama's, with no usage thresholds or restrictions. This matters for companies that want the simplest possible licensing terms.

Mistral also offers Mixtral 8x7B, a mixture-of-experts model that activates only 12.9B of its 46.7B total parameters per forward pass. Fine-tuning Mixtral requires targeting the expert layers, which is supported by PEFT but requires more memory than a dense 7B model. The quality gains over dense 7B are meaningful for complex reasoning tasks.

Qwen 2.5 (Alibaba Cloud)

Qwen 2.5 has emerged as a serious contender, particularly in the 7B and 14B sizes. Its standout feature is multilingual performance: Qwen handles Chinese, Japanese, Korean, Arabic, and dozens of other languages at a quality level that neither Llama nor Mistral matches. For any application serving a global or multilingual user base, Qwen is often the best choice.

Strengths for fine-tuning: Qwen 2.5 leads the open-source field in code generation benchmarks, scoring 65.5 on HumanEval for the 7B variant compared to Llama's 72.6 for 8B (close, with Qwen 14B pulling clearly ahead at 79.9). The 14B variant is a particularly appealing base model because it fits in 24GB VRAM with QLoRA while offering a meaningful quality jump over the 7B class. Qwen's Apache 2.0 license has no usage restrictions.

Performance profile: Qwen 2.5 7B scores 74.2 on MMLU, 65.5 on HumanEval, and 85.4 on GSM8K, competitive with or ahead of both Llama 8B and Mistral 7B. The 14B variant (MMLU 79.9, HumanEval 79.9) approaches the quality of Llama 70B on many tasks while requiring a fraction of the hardware. For teams that need more capability than an 8B model but cannot afford 70B inference costs, Qwen 14B is the sweet spot.

Fine-tuning considerations: Qwen uses a standard transformer architecture with RoPE positional embeddings and grouped-query attention. The tokenizer vocabulary is 151,936 tokens, larger than Llama or Mistral, which means Qwen is more efficient on multilingual text (fewer tokens per word in non-English languages). PEFT and QLoRA work the same way as with Llama. The Qwen community on Hugging Face is growing rapidly but is still smaller than Llama's, so you may find fewer community resources for troubleshooting.

Other Models Worth Considering

Gemma 2 (Google): Available in 2B, 9B, and 27B sizes. The 9B variant is competitive with Llama 8B on most benchmarks. Gemma's advantage is its smaller 2B variant, which is useful for edge deployment where even 8B is too large. Fine-tuning the 2B model produces a tiny specialist that runs on mobile devices and embedded systems.

Phi-3 and Phi-4 (Microsoft): Small models (3.8B for Phi-3, 14B for Phi-4) that punch above their weight on reasoning and code tasks. Phi-4 14B is comparable to Llama 70B on several benchmarks while requiring far less compute. However, Phi's training data composition means it sometimes struggles with creative writing and open-ended generation tasks. Best for highly structured, reasoning-heavy applications.

DeepSeek V3 (DeepSeek): A mixture-of-experts model with 671B total parameters but only 37B active per forward pass. Competitive with frontier closed-source models on many benchmarks. Fine-tuning requires expert-layer targeting and significant GPU memory, but the results can be exceptional for research and complex reasoning tasks.

How to Choose Your Base Model

Default choice: Llama 3.1 8B Instruct. Largest ecosystem, broadest tool support, good all-around performance. Choose this unless you have a specific reason to use something else.

For instruction following and tool use: Mistral 7B v0.3 Instruct. Its native function-calling support gives you a head start on agent fine-tuning. The stronger instruction adherence means less training data is needed to achieve reliable tool selection.

For multilingual applications: Qwen 2.5 7B or 14B. No other open-source model matches Qwen's breadth and depth of language support. If your users interact in Chinese, Japanese, Korean, Arabic, or other non-European languages, Qwen is the clear winner.

For maximum quality on a single GPU: Qwen 2.5 14B. It fits in 24GB with QLoRA and outperforms all 7B-8B models on most benchmarks. The trade-off is slower inference (roughly 2x compared to an 8B model), which matters for latency-sensitive agent applications.

For edge deployment: Gemma 2 2B. Fine-tune for your specific task and deploy on devices where larger models cannot run. The quality is lower than 8B models, but for narrow, well-defined tasks, a fine-tuned 2B model can match the performance of a prompted 8B model.

Test before committing: Run your task with 2-3 base models using prompt engineering before choosing one for fine-tuning. The model that performs best with prompts alone will almost certainly perform best after fine-tuning. This quick test costs nothing and prevents the expensive mistake of fine-tuning the wrong base model.