Self-Hosted AIOps Tools and Platforms

Why Self-Host AIOps

Three factors drive organizations toward self-hosted AIOps rather than vendor platforms. The first is data sensitivity. Operational data, including logs, metrics, configuration files, and incident details, often contains sensitive information: database connection strings, API keys, customer data fragments in error messages, internal architecture details, and business-specific metrics. Sending this data to a vendor's SaaS platform for AI analysis means trusting the vendor's security, data handling, and retention policies. For organizations in regulated industries (healthcare, finance, government), self-hosting may be a compliance requirement rather than a preference. For all organizations, self-hosting eliminates the risk of a vendor breach exposing operational intelligence.

The second factor is cost at scale. Vendor AIOps platforms charge per host, per GB of data ingested, or per seat, and costs grow linearly or super-linearly with infrastructure size. A Datadog bill for a mid-size infrastructure can reach $50,000 to $200,000 per year. Self-hosted alternatives using open source monitoring, local LLM inference, and custom agent logic have a higher upfront engineering cost but a much flatter cost curve as infrastructure scales. For organizations with more than a few hundred monitored hosts, the total cost of ownership for self-hosted AIOps often breaks even with vendor platforms within the first year.

The third factor is customizability. Vendor platforms provide general-purpose AI capabilities that work across many customers but cannot encode the operational knowledge specific to your infrastructure. A self-hosted agent can be trained on your runbooks, post-mortem reports, and architecture documentation. It can integrate with proprietary internal tools that no vendor platform supports. It can follow your organization's specific incident response procedures, compliance requirements, and escalation policies. This deep customization is what makes the difference between an AI assistant that provides generic advice and one that provides genuinely useful, contextually appropriate operational support.

Open Source AIOps Building Blocks

K8sGPT is the most widely adopted open source AI tool for Kubernetes operations. It scans a Kubernetes cluster for issues, analyzing pods, deployments, services, network policies, and other resources, and produces natural language explanations of any problems it finds. K8sGPT integrates with multiple LLM backends including OpenAI, Azure OpenAI, and local models via Ollama, making it a flexible choice for both cloud-connected and air-gapped environments. The tool can run as a CLI for on-demand diagnostics or as an in-cluster operator that continuously scans and reports issues. K8sGPT is most useful as a diagnostic aid for Kubernetes-specific issues, but it does not cover broader operational concerns like CI/CD, cloud cost, or cross-service incident management.

OpenTelemetry provides the telemetry collection layer that feeds data to AI agents. The OpenTelemetry Collector receives metrics, logs, and traces from instrumented applications and infrastructure, processes and transforms them, and exports to any supported backend. For a self-hosted AIOps stack, the Collector feeds data to local storage, such as Prometheus for metrics, Loki for logs, and Jaeger or Tempo for traces, which the AI agent then queries. OpenTelemetry's vendor-neutral approach means you can switch between AI backends, add new data sources, or change storage backends without re-instrumenting your applications. The Collector itself is lightweight and can run as a sidecar, DaemonSet, or standalone deployment.

Prometheus and Grafana form the monitoring foundation of most self-hosted operations stacks. Prometheus collects and stores time-series metrics, and Grafana visualizes them. Grafana has added AI-powered features including intelligent alerting and anomaly detection, and its plugin ecosystem supports custom AI integrations. An AI agent that queries Prometheus via PromQL and Grafana via its HTTP API has access to the full metrics history of your infrastructure without sending any data externally. For organizations already running Prometheus and Grafana, adding an AI agent layer on top requires minimal infrastructure changes.

Loki is Grafana's log aggregation system designed for cost-effective operation. Unlike Elasticsearch, which indexes every field in every log entry, Loki indexes only the metadata labels and stores log content as compressed chunks, resulting in dramatically lower storage costs and simpler operation. An AI agent that queries Loki via LogQL can search and aggregate logs efficiently, and the lower storage cost means you can retain logs for longer periods, giving the agent more historical data for baseline learning and pattern analysis.

Building a Custom AIOps Agent Stack

A practical self-hosted AIOps stack combines telemetry collection, local or cloud LLM inference, an agent framework for orchestration, and a set of operational tools that the agent can call. The architecture looks like this:

The telemetry layer consists of OpenTelemetry Collectors feeding Prometheus (metrics), Loki (logs), and Tempo (traces). These three backends provide the data the agent needs for monitoring, diagnosis, and historical analysis. They run on your infrastructure, store data locally, and expose query APIs that the agent accesses directly.

The LLM layer provides the reasoning capability. For fully self-hosted deployments, Ollama running models like Llama 3.1 70B or Qwen2.5 72B on GPU hardware provides strong reasoning without any external API calls. For environments where cloud API calls are acceptable, Claude or GPT-4 class models through their respective APIs provide the best diagnostic reasoning. A hybrid approach uses a local model for high-volume, low-stakes analysis like log pre-processing and anomaly detection, and a cloud model for complex diagnostic reasoning during incidents. Our self-hosted LLM guide covers hardware requirements and model selection, and server requirements details the GPU configurations that work well for inference at operational scale.

The agent framework orchestrates the LLM's reasoning with tool calls and workflow logic. LangGraph is the strongest choice for DevOps agents because its stateful graph execution model maps naturally onto multi-step diagnostic workflows where the agent needs to branch based on findings, retry failed queries, and accumulate context across multiple tool calls. CrewAI is better suited for architectures where you want multiple specialized agents, like a monitoring agent, a diagnosis agent, and a remediation agent, that collaborate on incidents. For simpler workflows, n8n provides a visual workflow builder with AI nodes that can be configured without writing code.

The tool layer is the set of functions the agent can call to interact with your infrastructure. These tools wrap your existing operational interfaces: kubectl commands for Kubernetes, AWS CLI calls for cloud resources, Terraform operations for infrastructure changes, Ansible playbooks for configuration management, and custom scripts for organization-specific operations. Each tool should be implemented as a function with typed inputs and outputs, clear permission boundaries, and audit logging. The Model Context Protocol (MCP) provides a standardized way to expose these tools to the agent, and the tool calling guide covers the patterns for building reliable tool interfaces.

Hardware and Infrastructure Requirements

The infrastructure requirements for self-hosted AIOps depend primarily on whether you run the LLM locally or use cloud APIs. The telemetry layer (Prometheus, Loki, Tempo) and agent framework run comfortably on standard server hardware without GPUs. The LLM layer, if self-hosted, requires GPU hardware whose specifications depend on the model size.

For a 7B to 13B parameter model suitable for log pre-processing and simple diagnostics, a single NVIDIA RTX 4090 (24GB VRAM) or A10 (24GB VRAM) is sufficient, costing $1,500 to $2,500 for the GPU. For a 70B parameter model that provides stronger reasoning for complex diagnostic tasks, you need 40GB to 80GB of VRAM, which means either an A100 80GB ($15,000 to $20,000), two A6000s ($9,000 to $12,000 total), or renting cloud GPU instances from providers like Vast.ai where an A100 80GB costs approximately $0.80 to $1.50 per hour. For organizations that do not want to manage GPU hardware, the cloud API approach with Claude or GPT-4 costs approximately $0.01 to $0.10 per diagnostic interaction depending on context size, which is economical for most operational workloads.

Storage requirements for the telemetry layer depend on data volume and retention period. A reasonable starting point for a mid-size infrastructure is 500GB to 1TB for Prometheus (90-day metric retention), 1TB to 5TB for Loki (30-day log retention), and 200GB to 500GB for Tempo (7-day trace retention). SSD storage is recommended for Prometheus and query-heavy Loki workloads; HDD is acceptable for long-term Loki storage and Tempo. These numbers scale linearly with the number of monitored services and the volume of telemetry they produce.

For deployment, Docker Compose provides the simplest path to running the full stack on a single server. Prometheus, Loki, Tempo, Grafana, Ollama, and the agent framework each run in their own container with shared networking and persistent volumes. For production deployments that need high availability, Kubernetes with Helm charts for each component provides the scaling and resilience needed for continuous operation. The key operational requirement is that the AIOps stack itself must be monitored independently of the infrastructure it monitors, otherwise a failure in the monitoring infrastructure goes undetected, defeating the purpose.