Use caseCustom softwareKnowledge and search
A private ChatGPT for every employee, behind single sign-on, with redaction and team budgets
A private AI gateway that gives staff ChatGPT-style chat behind single sign-on, redacts client data before it leaves, routes to EU models and caps team spend.
A blueprint, not a client story. The business described is illustrative; the architecture, integrations and trade-offs are real, and this is how I would build it. By Ergini, .
The short version
A private AI gateway gives every employee a ChatGPT-style assistant under the company's own sign-on, instead of personal accounts. Before a prompt leaves, it replaces client names, personal data and secrets with placeholders, then routes the request to an approved model, on an EU endpoint or self-hosted, within the team's budget. Usage is logged under rules agreed with staff, and department assistants add approved firm context. People set the policy and review what it flags.
- Best for
- Law firms, financial firms and other confidential businesses of 100 to 1,000 people whose staff already use personal AI accounts for work.
- Connects to
- Microsoft Entra ID or Google Workspace, Slack or Teams, OpenAI, Anthropic and Mistral, Open models on EU infrastructure, Open WebUI, LibreChat or a custom interface, Gateway layer, Practice management system
- The AI does
- Answers and drafts through the approved models, while a small entity-recognition model inside your network spots names that fixed rules miss.
- People do
- Set what may be sent where, own team budgets, review flagged events, and agree the logging rules with staff representatives.
- Built as
- AI Integration, usually $3.5K - $15K
Three hundred people, one policy memo, and ChatGPT on every phone
Picture a 300-person law firm with offices in Frankfurt and Munich: 140 lawyers, the rest paralegals, assistants, finance and IT. In 2024 the managing partners sent a memo banning public AI tools for client work. Associates still summarize judgments on their phones, assistants polish English letters in a free chatbot, and at least one partner pays for a personal subscription. Nobody knows what has been pasted, or where it went.
That is the normal state of things, not a discipline problem. KPMG and the University of Melbourne found that 48% of employees had uploaded sensitive company information to public AI tools, and that at organizations banning generative AI, 67% of employees still reported uploading sensitive data or similar behavior. IBM's Cost of a Data Breach report for 2025 found that one in five breached organizations it studied had a breach involving shadow AI, and that heavy shadow AI use added about $670,000 to the average cost of a breach.
Another memo will not change that. A sanctioned tool that is at least as good as the one on the phone will: current models, long documents, file upload, the firm's own templates, no copy-paste. Behind it sit the controls the partners care about: who may send what, to which model, in which region, at what cost, with what record. If the firm's tool is slower or worse, people go back to their phones.
The path every prompt takes through the gateway
One step uses a small model that runs inside the firm, and the answer comes from whichever model the rules allow. Every decision about where data may go is code.
01 Trigger · Chat interface, Teams bot
A lawyer sends a prompt
In the firm's chat app or the Teams bot, often with a document attached and a matter number picked from the practice management system.
02 Plain code · Entra ID over OIDC
Identify the person and their team
Single sign-on supplies the user and their groups, which decide the team, the budget, the allowed models and which department assistants appear. Leavers lose access the moment Entra ID disables them.
03 Plain code · Pattern rules, matter dictionary
Apply the fixed rules
Patterns catch IBANs, tax and ID numbers, phone numbers and email addresses, and a nightly dictionary from the practice management system catches client and matter names. API keys, passwords and private keys stop the request outright instead of being redacted.
04 AI model · Presidio with a local NER model
Catch the names the rules miss
A small entity-recognition model on the firm's own servers flags remaining person and organization names. Each hit becomes a consistent placeholder, so CLIENT_1 stays CLIENT_1 for the whole conversation and the model can still reason about it.
05 Decision
Where may this request go?
A policy table decides, per team and per matter flag. The model never chooses its own route.
- Placeholders only, ordinary work, budget left then the team's default frontier model on an EU endpoint
- The matter is flagged confidential, or the team's policy keeps data in-house then the self-hosted open model on EU infrastructure
- A secret was found, or the team's monthly budget is spent then blocked, or moved to the cheaper model, with a plain message saying why
- The chosen provider is failing then the next approved model with the same data terms, never an unapproved region
06 System · EU provider endpoints, vLLM
Call the model
Streamed through the gateway with a timeout and one retry on a provider error. The budget is reserved before the call and settled after it, so parallel requests cannot all spend the same last euro.
07 Plain code
Put the names back
Placeholders in the reply are swapped back inside the firm, so the lawyer reads a normal answer and the provider never saw the names.
08 Person
People review what the rules flag
The security team sees blocked secrets and repeated overrides, pseudonymized until an investigation is opened under the agreed procedure. Team leads get a budget alert at 80%.
09 Result · Gateway log, SIEM
Record only what the policy allows
Metadata for every request: pseudonymous user, team, matter number, model, tokens, cost and redaction counts. Prompt text is kept for 30 days for security investigations, or not at all, as agreed with staff.
One request, and what the provider actually received
The associate sees an ordinary chat. These calls are what the gateway did before and after any model saw the request.
Firm chat, Wednesday 16:05
Associate · Firm chat
Summarize the attached judgment and draft two paragraphs for our letter to Dr. Hofmann on what it means for his purchase of the Munich bakery chain. Opposing counsel is Dr. Brandt. Escrow IBAN: DE89 3704 0044 0532 0130 00.
- resolve_user(sso)a.keller / Corporate / matter 2026-0412 selected / budget 61% used / allowed: frontier-eu, local
- redact_rules(text, attachments: 1)matter dictionary: Hofmann -> CLIENT_1 (4 hits) / IBAN -> IBAN_1 / secrets: none
- redact_ner(text)Dr. Brandt -> PERSON_1 / 'the Munich bakery chain' not flagged: a description, not a name
- route(team: "corporate", matter: "2026-0412")matter not flagged confidential / frontier-eu allowed under DPA / about 9,800 tokens, within budget
- call_model(model: "frontier-eu", stream: true)200 OK / 1,240 tokens out / EUR 0.06 charged to Corporate
- restore(answer)CLIENT_1 -> Hofmann (3) / PERSON_1 -> Dr. Brandt (1) / IBAN_1 not used
Firm assistant · Firm chat
Here is a five-point summary of the judgment, followed by two draft paragraphs for your letter to Dr. Hofmann. I have left the escrow IBAN out of the draft; add it from the file if the letter needs it.
- What left the building: the judgment, and the question with CLIENT_1, PERSON_1 and IBAN_1 in place of the names and the account number. 'The Munich bakery chain' went through unchanged.
Personal accounts versus the firm's own gateway
Personal AI accounts
- Consumer terms the firm never signed, with settings each person chooses
- Client names, account numbers and draft clauses sent exactly as typed
- No record of what was sent, and the history leaves with the employee
- Whatever model the person happens to pay for
- Costs on personal cards and expense claims, invisible to finance
The firm's gateway
- Business terms and a data processing agreement with each provider
- Names, numbers and secrets replaced before anything leaves
- A record by matter, under rules staff were told about, and history that stays with the firm
- The right model per task, including a self-hosted one for confidential matters
- A budget per team, with alerts before it runs out
Where a gateway like this fails, and the design choice for each
Redaction that misses a name
Descriptions, nicknames, misspellings and names inside scanned PDFs slip past both rules and models, and no redaction catches everything. So redaction reduces exposure, routing is the control, and the policy covers what software cannot: describing a client is still disclosing one. Misses found in sampled reviews become new rules.
Logging prompts is monitoring too
A prompt log shows what people work on, how fast and how well. In Germany a system able to monitor employee behavior or performance needs the works council's agreement under § 87(1) no. 6 BetrVG, and GDPR requires telling staff what is kept and why. So the gateway logs metadata by default, pseudonymizes people and keeps prompt text briefly or not at all. Partners who want a record of AI use per matter still get one, because requests are tagged by matter, not ranked by person.
A budget gone by the twelfth
A few heavy users summarizing long documents with the largest model can spend a team's month in a week. Per-team budgets, per-person daily caps and a cheaper default model keep spend predictable; the large model is one click away, not the default. Budget owners see spend daily, not on the invoice.
The provider goes down during a deadline
Health checks move traffic to the next approved model with the same data terms, and the self-hosted model is the last resort. Users see which model answered. Fallback never crosses a data boundary: a confidential matter waits rather than going to an unapproved region.
The gateway itself gets compromised
In March 2026, versions of LiteLLM, a widely used open-source AI gateway, were published to PyPI with a backdoor. A gateway holds every provider key, so dependencies are pinned by hash, installed from a reviewed mirror and upgraded deliberately. The gateway runs in its own network segment, can reach only approved endpoints, and its keys can be rotated in minutes.
People drift back to their phones
If the gateway is slower, runs older models or feels policed, use returns to personal accounts. New models go live within days of release, file upload works, and department assistants offer what public tools cannot: the firm's templates, clause library and know-how.
Switching people over without another ban
- Measure first, anonymously: a few weeks of aggregated traffic counts to public AI sites, and a short survey on what people use them for.
- Agree the rules before launch: what may go to which model, what is logged and for how long, signed off by the partners, the data protection officer and, where there is one, the works council.
- Pilot with one practice group and one support team, about 30 people, and fix whatever is slower or clumsier than the public tools.
- Add department assistants: drafting with the firm's templates, translation with its glossary, and answers from its know-how through an internal knowledge assistant.
- Open it to everyone, then point public AI sites to a notice that links to the firm's tool: a warning at first, a block only where the risk demands it.
The order matters. Blocking before there is a better alternative is exactly the situation the KPMG figures describe.
The layers, and what I would pick for a firm this size
Most of these parts exist as products or open-source projects. The work is choosing, hardening and connecting them, not writing them from scratch.
| Layer | Options | What I would pick here |
|---|---|---|
| Sign-in | Entra ID or Google Workspace over OIDC, groups synced by SCIM | Entra ID, since the firm already runs Microsoft 365 |
| Chat interface | Open WebUI, LibreChat, a custom app, a Teams bot | An open-source interface plus the Teams bot, after checking the interface's license terms at 300 users |
| Gateway | LiteLLM, Portkey, a cloud provider's gateway, or a thin custom service | A thin custom service, or a pinned and reviewed open-source gateway in its own network segment |
| Redaction | Pattern rules, Microsoft Presidio, a dictionary of client and matter names | All three, rules first |
| Models | OpenAI, Anthropic and Mistral on EU endpoints; open models such as Mistral Small 4, Gemma 4 or Qwen3.5 | One frontier model as the default, one self-hosted open model for confidential matters |
| Logs and spend | The gateway's own logs, Langfuse, the firm's SIEM | Metadata to the SIEM, and spend per team on a dashboard finance can read |
ChatGPT Enterprise, Copilot Chat, or a gateway of your own?
If one vendor is enough, buy. ChatGPT Enterprise, Claude's Team and Enterprise plans, Microsoft 365 Copilot Chat and Mistral's Le Chat Enterprise all offer single sign-on, admin controls and business terms under which your data is not used for training, and Copilot Chat comes with many Microsoft 365 business subscriptions. For a lot of 300-person firms the right move is to license one, write a clear policy and stop there. I will tell you if that is you.
A custom or self-hosted gateway wins when you want in one place what no single vendor offers: several models chosen per task, including a self-hosted model on EU infrastructure for the most sensitive work, EU-only routing you control, redaction before anything leaves, and per-team budgets with chargeback. It also wins when only part of the firm uses AI daily and per-seat licenses for everyone make little sense.
The hybrid is common: license one vendor's chat product for most staff, and put the gateway in front of everything that calls models through an API, such as department assistants, the knowledge assistant and internal tools. If the bill rather than the risk is the problem, start with an AI cost audit.
How you would know it is working
A blueprint has no results to report, so here is what I would measure from the first week instead, on your own data.
- Share of AI use through the gateway
- Aggregated counts of gateway sessions against traffic to public AI sites, never per person. It shows whether shadow use is shrinking.
- Weekly active users by team
- Low use in one team usually means a missing feature or assistant, not a lack of interest.
- Redaction hits and misses
- Hits per thousand prompts by type, and misses found in sampled reviews under the agreed procedure.
- Spend against budget
- Per team and per model, with cost per active user, reviewed monthly with finance.
- Fallbacks and errors
- How often the default model failed, and what answered instead.
What a build like this costs
This is built as AI Integration, which runs $3.5K - $45K overall. A build like this one usually lands in the single AI feature tier: $3.5K - $15K, 1-3 weeks. The first working version runs on your real data well before the end of that window.
What it costs to run
Model usage is paid per token and usually driven by a small group of heavy users, which per-team budgets make visible. A self-hosted open model on a rented EU GPU is a fixed monthly cost that pays off only once usage is steady. The gateway, interface and logs are a modest hosting bill.
What moves the price
- Whether an open model is self-hosted: GPU hosting, serving and updates are the biggest single decision
- Redaction depth: patterns only, or a matter dictionary synced from the practice management system plus a local entity model
- How many department assistants, and whether each needs its own approved documents
- Logging and retention design where a works council agreement or a DPIA is needed
- Which chat surfaces: an open-source web interface, a Teams or Slack bot, or a custom app
Who this is for
- Law firms where lawyers already use personal ChatGPT accounts and the ban on them is quietly ignored
- Banks, asset managers and insurers that need DORA-compliant contracts with AI providers and EU-only routing
- Firms with a works council that must know exactly what is logged before anything launches
- Companies paying for three different AI subscriptions across teams with no view of the total
Questions people ask about this
How do we get past confidentiality fears about using AI with client files?
Start with where the data goes, not with the software. Keep client work on EU endpoints under a data processing agreement and business terms that exclude training, or on a self-hosted model for the most sensitive matters, and redact names and numbers before anything leaves. Partners are usually persuaded by a record of AI use per matter. German lawyers also need providers bound to confidentiality, as § 43e BRAO requires.
Can we self-host a ChatGPT alternative for our company?
Yes. An open-source interface such as Open WebUI or LibreChat in front of open models like Mistral Small 4, Gemma 4 or Qwen3.5 on rented EU GPUs gives a private assistant that never leaves your infrastructure. It is not free: GPUs, serving and updates cost money and attention. Most firms self-host only for the most sensitive work and use frontier models on EU endpoints for the rest.
What is an AI gateway, and do we need one?
It is a single endpoint between your staff and apps and every model provider. It checks who is asking, redacts what should not leave, picks the approved model, enforces budgets, falls back when a provider fails and records usage. You need one once several teams or tools call models, because a policy change or a new model then becomes one change in one place.
Is it legal to log employees' AI prompts in Germany?
It can be, with conditions. GDPR requires a clear purpose and telling staff what is kept and why, and where a works council exists, a system able to monitor behavior or performance needs its agreement under § 87(1) no. 6 BetrVG. I design logging so it can be agreed: metadata by default, pseudonymized, short retention for prompt text, access only under a documented procedure. Your lawyers and data protection officer decide.
How much does a private AI gateway cost?
A gateway on an existing open-source interface, with one or two providers and pattern-based redaction, is a small build in the lowest AI integration tier shown on this page. A self-hosted model, a matter dictionary, department assistants and a negotiated logging design move it up. Running costs are tokens plus hosting, and the team budgets make both visible.
Sources