AI Gateway

Connect your organization's model providers in AI Gateway, manage the model catalog and fallback model groups, and monitor Gateway traffic and issues.

AI Gateway is where your organization centrally manages model access. Connect your organization's provider credentials (such as OpenAI or Amazon Bedrock) or a self-hosted OpenAI-compatible endpoint to create custom models, then use fallback model groups to specify which models to try in order when the primary model fails. Connected models can be used by AI assistants, MaiGPT, or applications integrated through an OpenAI-compatible API. This page explains the administrative interface for organization administrators and IT staff responsible for model governance. For application integration details, see AI Gateway: OpenAI-Compatible API Integration in the technical guide.

AI Gateway must be enabled for your organization before you can use it (an Enterprise feature, currently in Beta). Until it is enabled, AI Gateway will not appear in the left menu, and it will be grayed out in the “Advanced Features” section of Organization Overview. To enable it, contact your sales representative (see Advanced Features).

Who Can Use It

Action	Required Permission
See AI Gateway in the left menu and view its tabs	The AI Gateway Permission role permission (organization owners have it by default; for other roles, see Enable AI Gateway for a Specific Role)
Add, edit, test, or delete provider credentials, custom models, and fallback model groups	Organization owner
Toggle Available to Organization for managed models	Organization owner (the same setting as on the Model Access Permissions page)
Create API Keys for calling the Gateway	API Key Management Permission

Members who are not organization owners can view each tab, but the system rejects attempts to add, modify, delete, or test resources. Buttons such as Add Provider, Add Model, and Add Fallback Model Group are disabled. Hovering over them displays “Only organization owners can manage provider credentials,” “Only organization owners can manage custom models,” or “Only organization owners can manage fallback rules.”

How to Access It

Click AI Gateway in the left menu. It opens the Overview tab by default. There are five tabs:

Tab	Purpose
Overview	Gateway traffic, spending, latency, and issues that need attention
Model Providers	Manage your organization's provider credentials
Models	The model catalog: managed and custom models
Fallback Model Groups	Fallback chains used in order when the primary model fails
Integration Examples	Code examples for calling the OpenAI-compatible API

If your organization has no providers yet, Overview displays “Welcome to AI Gateway” with three steps: Add Provider → enable the models you want in the Model Catalog → Add API Key. After you complete the first step, the overview becomes a live health dashboard.

Overview

The overview summarizes your organization's Gateway traffic, including OpenAI-compatible API calls and AI assistant conversations using custom models. At the top right, select Last 1 Hour, Last 24 Hours, Last 7 Days, or Last 30 Days, or specify start and end times with the date picker (custom ranges can span up to 92 days).

Section	Contents
Status summary	Displays “All systems normal · No pending items” or “N items need attention/action”; Access Diagnostics is on the right, and you can jump to the “Needs Attention” card with one click when there are pending items
Metric cards	Requests, Spending (credits), Tokens, Error Rate, Slowest Latency, and Fallback Triggers, compared with the preceding period of the same length
Providers	Each provider has a status label: Healthy, High Rate Limiting (upstream error rate above 5% over the last 24 hours), or Setup Required (credentials disabled). Click to see the upstream error rate over the last 24 hours and suggested actions, and to Test Connection or Manage Credentials; Create Fallback Rule is also available for High Rate Limiting
Usage Trends, Latency	Trend charts for request count or cost; Time to First Token (TTFT), Total Latency, and Upstream Latency each show Typical (p50) and Slowest (p95) values
Usage Breakdown	Lists requests, fallback counts, and spending by Provider or Model (providers also show error rates); expand a model to see the API Keys calling it
Needs Attention	A provider's upstream error rate exceeds 5% over the last 24 hours; a member bound to an API Key has used at least 90% of their quota, or a disabled key has received requests in the last 24 hours; a fallback rule has triggered at least twice as often this month as during the same period last month

Use Access Diagnostics to investigate cases where someone cannot access a model. Select an API Key (and optionally a model), then click Diagnose. The system checks that the key has not been revoked, is enabled, and has not expired; that the member and user are active; that the organization has Gateway enabled and can use the model; and that the key has access to the model. It highlights the first check that fails.

Add a Provider

In Model Providers, click Add Provider and follow the four-step wizard. You can add multiple entries for the same provider type (for example, Bedrock in different regions).

1

Choose a Provider

Choose a provider type. Providers whose cards display Automatic Discovery Supported can automatically list available models after creation. For those labeled Manual Model Addition Required, you will need to add models individually in the model catalog later.

2

Connection Credentials

Enter a Display Name (for example, “Bedrock US East”) and the provider's credential fields, then click Create and Discover. Credentials are saved at this step, so closing the wizard afterward will not discard them.

Provider	Required Fields	Optional Fields	Automatic Model Discovery
OpenAI	API Key	—	Supported
Anthropic	API Key	API Base (optional; leave blank to use the official endpoint)	Supported
Amazon Bedrock	AWS Access Key ID, AWS Secret Access Key, AWS Region	—	Manual addition required
Google Vertex AI	Service Account JSON (paste the entire service account JSON)	—	Manual addition required
vLLM	API Base (the URL of your self-hosted vLLM service, such as https://vllm.example.com/v1)	API Key (optional)	Supported
OpenAI Compatible	API Base (the URL of a self-hosted or third-party OpenAI-compatible endpoint)	API Key (optional)	Supported

3

Discover Models

The system queries the provider for available models and selects all of them by default. Models already in the catalog are labeled Already Exists and cannot be selected. After reviewing the selection, click “Import N Models.” Each selected model becomes a custom model. You can click Stop during import. If you do not need discovery, click Skip and Add Manually.

4

Finish

The screen displays Provider Connected and the import results. The imported models use default context and capability settings, which you can adjust individually in the Model Catalog.

Each row in the provider list shows the status, masked credentials, and an “N Models” link (click it to open the model catalog filtered to that provider). On the right, you can Test, Edit, or Delete the provider. You cannot change the provider type when editing. Leaving a secret field blank keeps its existing value.

Manage Models

The Models tab lists two model types in one catalog:

• Managed models: Models provided by MaiAgent, with the source labeled “MaiAgent Managed.”

• Custom models: Models created with your organization's own provider credentials, with the source labeled Custom. The Pricing column shows the traffic fee rate as “Total tokens × N%.”

Catalog columns include Name, Available to Organization, Referenced by Fallbacks, Source, Provider, Model ID, Pricing, and Context. At the top, filter by capability (Chat, Vision, Tool Calling, Reasoning), provider, or keyword. Click any row to open Model Details, where you can copy the Gateway model ID, view its provider, and see Which Fallback Chains Use It.

Available to Organization Toggle

The Available to Organization toggle for managed models uses the same setting as the Model Access Permissions page. Turning it off here also removes the model from AI assistants' model menus. Before disabling it, a “Disable this model?” confirmation lists how many assistants and fallback chains will be affected. A lock icon means MaiAgent has not enabled the model for the organization, or the platform has disabled Gateway access to it; the organization cannot enable it itself. Custom models always display Custom · Automatically Available and are not controlled by this toggle.

Add a Custom Model

1. Click Add Model, then select provider credentials in the Choose Provider step (if you have no providers yet, click Add Provider directly).

2. In Model Settings, fill in:

   • Name: The name displayed in the catalog and model menus.

   • Family: Anthropic, GPT, Gemini, Grok, Qwen, DeepSeek, Mistral, or Other.

   • Model ID: The model ID expected by the provider (for example, gpt-4.1). It must be unique under the same provider credentials.

   • Context and Maximum Output Tokens: Defaults are 128,000 and 8,000; adjust them to the model's specifications.

   • Capabilities: Vision, Tool Calling, Streaming, and Reasoning.

3. Click Create. The completion page displays the automatically generated Gateway model ID. Click Test Connection to confirm that you can call it.

The Gateway model ID is generated from the provider's Slug and the model name, and does not change if the model is renamed. Use this ID in application calls. On the right of each custom model row, you can Test, Edit, or Delete the model. After creating a model or changing its model ID or provider, the system detects in the background whether it rejects parameters such as temperature, top_p, and top_k. The results appear under Request Parameter Support in the edit dialog, where you can also click Detect Again. Not all providers support detection. If detection is unavailable, the system displays “Cannot detect this model (valid credentials and a supported provider are required).”

Fallback Model Groups

Fallback model groups let the Gateway try the next model in a fallback chain when the primary model fails. Chains can span models and providers.

1. In Fallback Model Groups, click Add Fallback Model Group.

2. Enter a Group Name (for example, “Claude High Availability”).

3. Select the primary model in the first row of the Fallback Chain, then click Add Next Priority to add fallback models. You can choose managed or custom models, and must select at least one. Drag rows or use Move Up and Move Down to reorder them.

4. Click Save. The system automatically generates a group Slug. Use the Slug in the model field of your application calls to invoke the entire fallback chain.

Trigger Conditions are fixed by the system and cannot be customized: Timeout, Connection Error, 429 Rate Limit, and 5xx Error. Each group card displays the trigger count for this month and the same period last month. A red warning appears when this month's count is at least twice that of the same period last month. Groups labeled Platform Level · Read-Only are shared platform rules; organizations can only View them.

Integration Examples

The Integration Examples tab explains the three steps for calling the Gateway through an OpenAI-compatible interface (obtain a Gateway key, specify a model name, and use automatic forwarding and fallback) and provides cURL, Python, and Node.js examples. After you select a model or group from Choose Model or Fallback Group, the model field in the examples changes to the corresponding Gateway model ID or fallback group Slug. Click Copy to use the code directly.

An administrator with API Key Management Permission creates API Keys for calls under Organization Settings → API Key Management. Each key is bound to a member. You can further restrict Enabled Models within the models available to the organization, and set Requests per Minute (RPM) and Tokens per Minute (TPM). For authentication, error codes, and other details, see AI Gateway: OpenAI-Compatible API Integration and AI Gateway Security.

Use with AI Assistants and MaiGPT

• AI assistants: After creation, custom models automatically appear in the LLM Model menu in AI Assistant Settings, labeled Custom. When a member with AI Gateway Permission edits an AI assistant, the menu also includes a Fallback Model Groups section for selecting the organization's fallback groups directly. Thinking settings are not displayed when a fallback group is selected.

• MaiGPT: MaiGPT only lists models with the Tool Calling capability, so enable Tool Calling for a custom model to make it appear. If the member's role restricts the models available in MaiGPT, you must also select the model in MaiGPT Role Permissions.

Before deleting custom models or fallback model groups, switch the AI assistants using them to other models, then delete them to avoid interrupting live services.

Billing

When you use custom models (with your own credentials or self-hosted endpoints), MaiAgent charges a traffic fee based on total tokens instead of the platform's model rates. Managed models retain their existing rates. For how to view rates and usage records, see AI Gateway Custom Model Traffic Fees.

FAQ

Q: Why is AI Gateway missing from the left menu?

Check two things: whether AI Gateway is enabled for your organization (under “Advanced Features” in Organization Overview), and whether your role has AI Gateway Permission selected.

Q: Why does testing an OpenAI Compatible provider report that there is no default test model?

The Test action in the provider row sends a test request using the provider's default test model. OpenAI-compatible endpoints have no default model available. Create a custom model in the model catalog first, then use Test in that model's row to check the connection.

Q: Why can't I delete a provider?

The system rejects deletion if custom models still use the provider credentials. Use the “N Models” link in the provider row to find and delete those custom models first, then delete the provider.

Q: Why are fallback model groups missing from an AI assistant's model menu?

The member editing the AI assistant must have AI Gateway Permission for the menu to list the organization's fallback model groups. Read-only platform-level groups do not appear in AI assistants' menus.