A Multi-Model Gateway That Replaces Shadow AI
Shadow AI appears when staff paste company data into external chat services and no system records it. The ESTAYA AI Platform puts a single gateway in front of that, built on LiteLLM. Every call passes through one OpenAI-compatible API. The gateway handles routing across models, failover when a backend goes down, and cost tracking per team and use case. The result is an auditable record instead of scattered private accounts.
We run open-weight models such as Llama, Mistral, Qwen and Teuken ourselves with vLLM on our own GPU systems in Germany. This keeps you model-independent: a new model is swapped behind the gateway without changing your applications. No US cloud sits in the path, and your data stays inside your environment.
The remaining layers build on that gateway. RAG returns answers with source citations rather than free-form text. AI agents use tool-calling and the Model Context Protocol (MCP) to reach line-of-business systems. Guardrails inspect inputs and outputs for personal data and prompt injection before anything reaches a model.
- Governance: roles, quotas, audit logging and model allowlists per department
- Observability: latency, token usage and error rates visible per model
- Deployment: SaaS, a dedicated VPC, or air-gapped on-premises
- In production as the ESTAYA AI Platform and a major German regional newspaper AI archive
- The audit log supplies the records the EU AI Act expects for high-risk systems