Running Multiple Local Models Simultaneously

Why Run Multiple Models

No single model excels at everything. A 70B general-purpose model handles complex reasoning well but is slow for simple tasks. A 3B model responds instantly for classification and routing but lacks depth for analysis. A dedicated coding model like Codestral outperforms general models on programming tasks. An embedding model like nomic-embed-text generates vector representations for search but cannot generate text.

Running multiple models lets you route each task to the optimal model. This approach, sometimes called a model router or model cascade, can deliver better overall quality than using a single large model for everything, while also reducing average response latency by sending simple queries to small, fast models.

Memory Considerations

The primary constraint is memory. Each loaded model occupies memory proportional to its parameter count and quantization level. A 7B Q4 model uses roughly 5GB. A 70B Q4 model uses roughly 40GB. An embedding model might use 1-2GB. If your system has 64GB of available memory (GPU VRAM or unified memory on Apple Silicon), you need to plan which models can coexist.

Two strategies handle memory allocation. Concurrent loading keeps all models in memory simultaneously. This provides instant response from any model but requires enough total memory for all models plus overhead. A system with 64GB could keep a 30B Q4 model (18GB), a 7B Q4 model (5GB), and an embedding model (1.5GB) all loaded, using about 25GB total.

Dynamic loading keeps the most recently used model in memory and swaps others in on demand. This works with less total memory but introduces a loading delay (5-30 seconds depending on model size) when switching between models. Ollama uses this approach by default, with a configurable keep-alive timer that unloads inactive models after a set period.

Multi-Model with Ollama

Ollama handles multi-model serving natively. You can pull multiple models and request any of them through the API by specifying the model name in each request. Ollama manages loading and unloading automatically based on available memory and the keep-alive setting.

For concurrent loading, set the OLLAMA_MAX_LOADED_MODELS environment variable to the number of models you want kept in memory. Set OLLAMA_KEEP_ALIVE to -1 to prevent automatic unloading. This configuration works well for systems with sufficient memory to hold all required models.

A common Ollama multi-model setup uses three models: a small model (3-7B) for fast responses and classification, a large model (30-70B) for complex tasks, and an embedding model for RAG retrieval. Your application logic routes requests to the appropriate model based on task type.

Multi-Model with vLLM

vLLM takes a different approach. Each vLLM instance serves a single model. To serve multiple models, you run multiple vLLM instances on different ports, each configured with its own model. A reverse proxy or application-level router directs requests to the appropriate instance.

This approach is more complex to set up but offers better isolation and performance characteristics. Each model has dedicated GPU resources, preventing one model from affecting another performance. For multi-GPU systems, you can assign different GPUs to different models.

Model Routing Strategies

Task-based routing assigns models by task type: coding queries go to Codestral, general questions go to Llama 70B, embeddings go to nomic-embed-text. This is the simplest approach and works well when task types are clearly defined.

Complexity-based routing uses a small classifier model to assess query complexity, then routes simple queries to a fast small model and complex queries to a large model. This reduces average latency and cost while maintaining quality on hard queries.

Fallback routing tries the small model first, evaluates confidence, and falls back to the large model if confidence is low. This works well for applications where most queries are simple but some require deep reasoning.