AI Gateway: OpenAI-Compatible API Integration

Integrate with MaiAgent AI Gateway using the OpenAI SDK without changing your application logic：endpoints and base URL, Bearer authentication, model identifiers (aliases, custom models, fallback group

This page is for engineers who want to call language models through MaiAgent AI Gateway, especially independent software vendors (ISVs) with applications already built using the OpenAI SDK. Gateway provides an OpenAI-compatible interface. Change the SDK's base_url to MaiAgent and api_key to a MaiAgent API Key to access models from different providers available to your organization through a single entry point, without changing your calling logic.

For the safeguards Gateway provides across sources, models, data, responsibilities, and records, see AI Gateway Security; this page covers integration only.

MaiAgent must enable AI Gateway for your organization. If it is not enabled, any key calling an inference endpoint receives the same 401 as an invalid key.

Endpoint Overview

Method	Path	Authentication	Purpose
POST	/api/v1/chat/completions	Required	Chat completions, with streaming and non-streaming support
POST	/api/v1/embeddings	Required	Text embeddings
GET	/api/v1/models	Optional	List callable model identifiers and pricing

Base URL (the OpenAI SDK's base_url):

https://api.maiagent.ai/api/v1

The SDK automatically appends /chat/completions, /embeddings, or /models. Do not add a trailing slash to the path.

Authentication

Inference endpoints use Bearer authentication, matching the headers the OpenAI SDK sends by default:

Authorization: Bearer <GATEWAY_API_KEY>
Content-Type: application/json

• The key is a MaiAgent API Key, but Gateway endpoints accept only the Bearer prefix; the Api-Key <key> format used by other MaiAgent APIs returns 401 here.

• All of the following return the same 401 invalid_api_key without revealing which check failed: the key does not exist, has been revoked, disabled, or expired; the associated member or user is disabled; or AI Gateway is not enabled for the organization.

• Request bodies must be JSON.

Get a Gateway Key

In the MaiAgent admin console, go to Organization Settings → API Key Management → Create API Key. The creation wizard has three steps: select a member, configure access and limits, and generate the key.

• Each key is associated with one member. Calls are attributed to that member and are subject to their Credits quota.

• In “Access and Limits,” configure which models the key can call (you can only narrow the organization's available models; all are enabled by default), requests per minute (RPM), and tokens per minute (TPM).

• The full key is displayed only once, at creation. Save it to your secret management system immediately.

• A member can hold multiple valid keys at the same time. For uninterrupted rotation, issue a new key first, deploy it, and then revoke the old key.

• The API Key Management page requires the “API Key Management” permission. Members without this permission can create their own keys under the avatar in the upper-right corner → Account → API Keys (these also work with Gateway); administrators configure model access and RPM/TPM on the API Key Management page.

In the left menu of the console, AI Gateway → Integration Examples generates copyable cURL/Python/Node.js examples for the selected model. If a call fails, go to AI Gateway → Overview and run Access Diagnostics to check the key, member, organization, and model access status in sequence. Your role needs AI Gateway permission to see the AI Gateway menu; see Enable AI Gateway for a Specific Role in the user manual for setup instructions.

Minimal Examples

Store the key in an environment variable first; do not hardcode it:

export GATEWAY_API_KEY="<GATEWAY_API_KEY>"

OpenAI Python SDK

import os

from openai import OpenAI

client = OpenAI(
    api_key=os.environ["GATEWAY_API_KEY"],
    base_url="https://api.maiagent.ai/api/v1",
)

resp = client.chat.completions.create(
    model="<vendor>/<model>",  # id from GET /models, or a fallback group slug
    messages=[{"role": "user", "content": "Hello!"}],
)
print(resp.choices[0].message.content)

curl

curl https://api.maiagent.ai/api/v1/chat/completions \
  -H "Authorization: Bearer $GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<vendor>/<model>",
    "messages": [
      { "role": "user", "content": "Hello!" }
    ]
  }'

Model Identifiers

The model field takes a Gateway model identifier, not a provider's native ID. All identifiers contain /; values without / immediately return 400.

What you want to call	What to put in model	Where to get it
MaiAgent managed model	{vendor}/{model} (or an alias configured by the platform)	The id returned by GET /models
Organization custom model (BYOK/self-hosted)	A system-generated Gateway model ID, in the form {provider slug}/{model name}-{random code}	“Gateway Model ID” on the console's AI Gateway → Models page
Fallback group	A system-generated group slug, in the form maiagent/fallbacks-{random code}	AI Gateway → Fallback Model Groups in the console

Resolution order: platform aliases first, then {vendor}/{model}, followed by organization custom models, and finally fallback groups. Platform models (aliases and {vendor}/{model}) are resolved only within the intersection of “available to the organization ∩ open to Gateway ∩ enabled for this key.” Custom models and fallback groups must belong to the organization and are restricted by the key's access list. Models outside this scope and nonexistent models return the same 404 model_not_found.

List Callable Models with /models

for m in client.models.list():
    print(m.id)

When called with a Gateway key, the list contains only models the key can actually call (including the organization's blocking settings and the key's access list). Without authentication, it returns the platform's public catalog without organization filtering. In addition to the OpenAI standard fields id, object, created, and owned_by, each entry includes:

Field	Description
name	Display name
type	Currently always language
tags	Capability tags：tool-use, multimodal, reasoning, prompt-caching, streaming; reasoning means the model accepts reasoning_effort, see “Reasoning Levels”
pricing	Managed models：input/output, in credits_per_1k_tokens (includes the organization's Gateway service fee when called with a Gateway key; the unauthenticated public catalog shows base prices before the service fee); custom models：traffic_fee_percent, in percent_of_total_tokens
providers	List of backend providers that can serve the model; each entry has a slug that you can specify using provider.order below
context_window, max_tokens	Present only when configured

Integration Behavior When an Organization Hides Chinese Models

When an organization enables “Hide Chinese Models,” Chinese platform models are excluded based on the model origins registered by the platform, including models added in the future. This restriction applies to both the Gateway model catalog and actual calls:

• Querying the catalog with a key: GET /models excludes hidden platform models; the unauthenticated public catalog has no organization information and does not apply this restriction.

• Calling a platform model directly: Hidden models cannot be called through aliases or {vendor}/{model} and return 404 model_not_found. Calling the same platform model with your own key does not bypass the organization's hiding setting.

• Calling a fallback group: Steps using hidden platform models are excluded first, and the remaining steps are used. If none remain, the call returns 404 model_not_found, and the group is also excluded from the catalog retrieved with a key.

• Organization custom models: Gateway models created by the organization are not subject to this origin-based hiding setting, but must still meet the existing key access rules.

For example, an enterprise integration engineer may have set a Chinese platform model as the default. After the organization enables hiding, the original identifier returns 404. The engineer should retrieve /models again using the key actually used by the service, change the default to an available model in that list, and then resume calls. Do not use the unauthenticated public catalog to determine whether the organization is authorized to use a model.

The hiding setting does not clear existing per-model block lists. After hiding is disabled, access still depends on the original organization enablement settings and the key's access scope.

Specify a Backend Provider

Multiple providers may serve the same model identifier (for example, a model available from both its original vendor and a cloud platform). To route through a specific provider, add provider.order to the request body using a providers[].slug returned by /models:

resp = client.chat.completions.create(
    model="<vendor>/<model>",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_body={"provider": {"order": ["<provider-slug>"]}},
)

Currently, only the first value in the list is used. If that provider does not have the model, the call returns 404 model_not_found. When provider.order is specified, resolution does not fall through to fallback groups.

Bring Your Own Key (BYOK)

Organization owners can register their organization's provider credentials in the console under AI Gateway → Model Providers. Once registered:

• When calling a managed model, if the organization has valid credentials for the same provider, Gateway uses those credentials for the upstream call (if there are multiple credentials for the provider, it uses the most recently updated entry). The caller does not need to change model.

• You can also use these credentials to create “custom models” and call them with system-generated Gateway model IDs.

• If the upstream provider rejects the organization's credentials with 401/403, Gateway returns 502 upstream_auth_error. The message indicates that the organization's BYOK credentials were rejected; update the credentials in AI Gateway settings.

Partially Update Provider Credentials

Integration applications can use PATCH /api/v1/gateway/credentials/{id}/ to update their own organization's provider credentials. This management interface requires login, AI Gateway enabled for the organization, AI Gateway access permission, and an organization owner as the writer. Its authentication serves a different purpose from the Gateway Bearer key used for inference endpoints.

Requests can include name, isActive, or credentials; fields inside credentials use camelCase names such as apiBase and apiKey. Each credential field is handled as follows:

Value sent	Update result
Field omitted	Keep the original value
Empty string ""	Keep the original value; indicates no change
Nonempty new value	Replace the original value, then validate according to the provider's rules
null	Clear an optional field; required fields cannot be cleared and return 400

For example, apiKey for vLLM and OpenAI-compatible providers can be cleared with null, but apiBase is required. Anthropic's optional apiBase can be cleared with null to remove the custom endpoint from the credentials; the API Key remains required.

If an API Key is already stored, changing apiBase to another nonempty endpoint requires you to re-enter apiKey at the same time, or explicitly clear the key with null if the provider allows it. Omitting the key or sending an empty string returns 400, with the error at credentials.apiKey, and leaves the original settings unchanged. Changing only the name or resubmitting the same endpoint does not require re-entering the key.

Read responses provide only the masked preview credentialsPreview, never the original credentials. See Model Provider Settings for console instructions; this section covers only update semantics for integration applications.

Fallback Groups

A fallback group is an ordered model list created by an organization owner in the console. The entire fallback chain is enabled only when model is set to the group slug; calls to a single model do not switch automatically.

• Switching conditions (fixed by the system): timeouts, connection failures, or upstream responses of 408/429/500/502/503/504/529. Client-side errors such as invalid parameters or invalid keys do not trigger a switch; the error is returned directly.

• Streaming: Gateway retrieves the first content chunk from upstream before starting the response, so connection errors and rate limits can trigger a switch before the first byte is sent. Once content starts being returned, switching stops.

• The response's model field remains the group slug you supplied. The actual model serving the request and whether fallback was triggered are recorded in the console's usage and request logs.

• Steps whose model data no longer exists, or whose organization custom model has been deleted (disabled), are skipped before the call. The primary model also becomes the first still-callable step in the list. This skip rule does not apply when a platform model is merely disabled.

• If the entire chain has no callable steps, inference returns 400 invalid_request_error with error code fallback_group_unavailable. Edit the group to add available models; do not retry it as a temporary upstream error.

• When reading a group with GET /api/v1/gateway/fallback-groups/{id}/, these invalid steps return modelId: null, modelName: "", managedModel: null, and customModel: null. Reading the management interface requires login, AI Gateway enabled for the organization, and AI Gateway access permission; only visible platform groups and the organization's own groups can be read.

• Organization-owned fallback groups appear in /models only when they still have callable steps. This rule does not mean that every platform virtual model disappears from the model catalog.

Chat Completion Parameters

Fields accepted by POST /chat/completions:

Field	Description
model	Required; see the previous section
messages	Required and nonempty. role can be system, user, assistant, or tool; content can be a string or a multimodal array of text/image_url; supports tool_calls, tool_call_id, and name
stream	Defaults to false
stream_options.include_usage	Defaults to false; see “Streaming”
temperature	Numeric value
max_tokens	Minimum 1
top_p	0 to 1
stop	String or string array
n	1 to 8; defaults to 1
tools	Only type: "function" is supported; function names can be up to 64 characters
tool_choice	auto, none, required, or a specified function object
parallel_tool_calls	Defaults to true
response_format	type is text, json_object, or json_schema; json_schema requires a json_schema object
reasoning_effort	none, minimal, low, medium, high, xhigh, max; applies only to models tagged reasoning in /models; see “Reasoning Levels”
provider	MaiAgent extension field; see “Specify a Backend Provider”

Notes:

• If the target model does not support top_p or stop, these parameters are skipped rather than returning an error.

• tools and response_format: json_schema cannot be used together; doing so returns 400.

• In streaming json_schema mode, the complete JSON is sent at once in a single delta, rather than streamed character by character.

• json_object mode adds an instruction to the system message to “output only valid JSON.”

• If a json_object response consists entirely of a single Markdown json code block with valid JSON inside, Gateway removes the block markers before returning it. This applies to both regular and streaming responses.

• If there is explanatory text outside the block, the block content is not valid JSON, or truncation means the response no longer forms a complete valid JSON block, Gateway preserves the model's original text. Integrations must still parse and handle invalid JSON themselves; json_object alone cannot guarantee a parseable object every time.

• If a streaming json_object response starts with a backtick, Gateway waits until the entire response finishes before sending the content; other responses are sent chunk by chunk as usual. If upstream fails mid-response, content already received but not yet sent is sent unchanged, followed by the error.

The response is a standard chat.completion object, with an id in the form chatcmpl-... and model set to the identifier you supplied. The response header x-request-id matches id; include it when reporting issues.

Reasoning Levels (reasoning_effort)

reasoning_effort specifies how deeply a model should reason using a set of levels shared across providers, from weakest to strongest:

none < minimal < low < medium < high < xhigh < max

• Accepted values: Only the seven values above or null are accepted. Other values (including off and numeric token budgets) return 400 invalid_request. none means reasoning is turned off.

• Omitted or null: Gateway does not specify a level. If the platform has configured a default reasoning level for the model, it uses that level; otherwise, it sends no reasoning parameter and lets the provider's default apply. The exception is a small number of OpenAI models (such as the GPT-5.6 series) that do not accept tool definitions without a level; the platform sends none for these models.

• Models without the reasoning tag: These models do not have reasoning controls; reasoning_effort is skipped without an error.

• Conversion: The level is first mapped to a level supported by the target model, then converted to the provider's native parameters (such as OpenAI's reasoning_effort, Claude's effort or reasoning token budget, or Gemini's reasoning budget). The platform maintains the levels each model supports by model family. /models currently provides only the reasoning tag, without listing each model's supported levels.

If a model does not support the level you send, Gateway adjusts it according to the following rules without returning an error:

Situation	What is actually sent
The model supports this level	Sent unchanged
The level is unsupported, but a weaker level exists	Use the nearest level no stronger than requested (for example, if the model supports up to high, sending max results in high); the adjustment does not make it more expensive
The requested level is below the model's minimum	Use the model's minimum level
Reasoning is requested (anything other than none)	It will not be adjusted to off
none is sent, but the model cannot disable reasoning	Send no reasoning parameter and let the provider's default apply; these models still reason and are billed for actual tokens

The same level represents different amounts of actual reasoning across models. Reassess latency and token usage when switching models; do not assume the same level costs the same.

Streaming

With stream: true, the response uses Server-Sent Events (Content-Type: text/event-stream). Each event is a line of data: {chat.completion.chunk JSON}, and a normal completion ends with data: [DONE]. You can iterate directly with the OpenAI SDK:

stream = client.chat.completions.create(
    model="<vendor>/<model>",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True,
    stream_options={"include_usage": True},
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

• With include_usage: true, an additional final event has an empty choices array and includes usage. This option only controls whether usage is returned, not billing; Gateway records and bills usage whether or not it is enabled.

• Errors before streaming starts (authentication, quota, nonexistent model, upstream connection failure) return ordinary HTTP error responses, not SSE.

• If upstream fails during streaming, the HTTP status is already 200. Gateway sends a data: {"error": {...}} event and then ends without sending [DONE]. Handle this event in your client.

Embeddings

POST /embeddings works the same as OpenAI:

emb = client.embeddings.create(
    model="<embedding-model>",
    input=["First text passage", "Second text passage"],
)
print(len(emb.data), len(emb.data[0].embedding))

• input: A nonempty string or a nonempty string array.

• encoding_format: float (default) or base64.

• model: Matches identifiers of embedding models the platform has opened to Gateway (then matches names). Embedding models are not in the /models list and are not restricted by the API Key's model access list; contact MaiAgent to confirm available embedding model identifiers.

• Gateway estimates usage.prompt_tokens using the cl100k_base tokenizer; this value is also the basis for billing.

Error Format

All errors use the OpenAI format, and the OpenAI SDK converts them to the corresponding exceptions:

{
  "error": {
    "message": "Model 'vendor/model' not found or not enabled",
    "type": "invalid_request_error",
    "code": "model_not_found",
    "param": null
  }
}

HTTP	type	code	Common cause	Recommended handling
400	invalid_request_error	invalid_request	Field validation failure, model without /, or using tools with json_schema	Fix the request; do not retry
400	invalid_request_error	upstream_invalid_request	Upstream rejects the request with 400/413/415/422 (parameters, tool definitions, or input size exceed the model's limits)	Adjust the request to the target model's limits
400	invalid_request_error	fallback_group_unavailable	No callable steps in the fallback group	Edit the group to add available models
401	authentication_error	invalid_api_key	Invalid, disabled, expired, or revoked key, or Gateway not enabled for the organization	Check the key; use console access diagnostics to identify the issue
402	billing_error	insufficient_credits	Insufficient organization Credits, or exhausted member/group quota; blocked before calling upstream	Add Credits or adjust the quota
404	invalid_request_error	model_not_found	Model nonexistent, unavailable to the organization, not enabled for this key, or unavailable from the specified provider	Confirm available IDs with /models
429	rate_limit_error	rate_limit_exceeded	Key's RPM or TPM limit exceeded	Retry with backoff
502	server_error	upstream_auth_error	Upstream rejects credentials (BYOK or platform credentials)	BYOK：update credentials; platform credentials：contact MaiAgent
502	server_error	upstream_error	Upstream timeout, rate limit, 5xx, or other error; the original upstream error message is not returned	Retry with backoff, or use a fallback group

Usage and Credits Billing

Each successful call deducts Credits from the organization. Chat completions use the token count reported by upstream, while embeddings use the token count estimated by Gateway (see “Embeddings”):

• Managed models: Billed at the model's input/output token rates (including prompt caching), plus the organization's Gateway service fee percentage. When /models is called with a Gateway key, pricing already includes the service fee; the unauthenticated public catalog shows base prices before the service fee.

• Custom models and BYOK credentials: Your company pays upstream charges directly to the provider. MaiAgent charges only a traffic fee, calculated as the call's total tokens (input + output + prompt cache writes and reads) × the organization's traffic fee rate, without LLM token pricing. AI Gateway Custom Model Traffic Fees in the user manual explains how to verify this in the console.

• Embeddings: Billed by input token count, also with the Gateway service fee added.

• Interrupted streaming: Tokens already generated are still billed if the client disconnects or upstream fails mid-response.

• If organization Credits are insufficient or the member's quota is exhausted, the request returns 402 before calling upstream.

View usage by model, provider, and API Key under AI Gateway → Overview in the console. Deduction details are in Credits consumption records.

Limits

• RPM: The key's requests-per-minute limit applies to all MaiAgent APIs callable with this key, not just Gateway.

• TPM: The tokens-per-minute limit applies only to Gateway chat completions and embeddings. Token counts are known only after the upstream response, so a single request may exceed the limit once in that minute. Subsequent requests in that minute return 429; the limit resets the next minute.

• No rate limit applies when RPM/TPM is not configured.

• Only the three OpenAI-compatible endpoints in the table above are currently available. OpenAI Responses, Assistants, image, audio, and other endpoints are outside the scope of Gateway.

Related Documentation

• AI Gateway Security: Endpoint sources, model controls, data protection, and audit logs

• Quickstart: Authentication for MaiAgent platform APIs (Agents, conversations, knowledge bases)

• User manual Credits: Credits balance, billing, and consumption records