> For the complete documentation index, see [llms.txt](https://docs.maiagent.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.maiagent.ai/tech/en/platform-development/onprem-model-services.md).
# On-Premises Model Service Integration
Connecting models in private cloud and on-premises deployments: the local model catalog included in the deployment package (vLLM), existing OpenAI-compatible model services (including URL /v1 rules),
MaiAgent needs three types of models to operate: a large language model (LLM), Embedding, and Reranker. Private cloud and on-premises deployments can obtain these models from three sources and combine them, for example by using a cloud LLM while running Embedding and Reranker on the host. This page is for customer IT teams and SI partners planning models and GPUs.
{% hint style="info" %}
Models are specified in the deployment configuration during deployment. This page explains the rules and limitations of each connection option. The actual configuration fields are documented in the installation manual included with the deployment package; contact your MaiAgent representative to obtain it.
{% endhint %}
## Three Connection Options
| Option | Description | Suitable for |
| ------------------------------- | --------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
| Run on the host (local models) | The deployment package starts vLLM model services on the GPU host, using models selected from its model catalog | Data must stay in the data center; GPUs are already available |
| Connect existing model services | Enter the URL and key for the customer's existing OpenAI-compatible model services | An existing self-hosted inference platform or model relay service |
| Connect cloud model services | The customer supplies cloud provider keys, and MaiAgent calls the cloud APIs directly | Data can be sent to cloud providers; proprietary models are desired |
An offline installation can also connect to external or cloud model services: offline means that installation materials are not downloaded from the internet. The firewall only needs to allow connections from the host to the model services (the tested environment allowed only port 443 to the model services).
## Local Model Catalog
The deployment package's model catalog offers the following models, served with vLLM. Weights are not included in the deployment package: online installations download them automatically from Hugging Face; offline installations require downloading them when preparing installation materials.
| Model | Type | Highlights | Default context |
| ---------------------------------- | --------- | ------------------------------------------------------------------ | --------------- |
| gpt-oss-20b | LLM | Supports tool calling | 128K |
| gpt-oss-120b | LLM | Supports tool calling | 128K |
| Gemma 3 4B | LLM | Supports image input; **does not support tool calling** | 128K |
| Gemma 4 26B A4B | LLM | MoE architecture; supports tool calling and image input | 128K |
| Gemma 4 31B | LLM | Supports tool calling and image input | 64K |
| Qwen3.6 27B（FP8） | LLM | Supports tool calling | 32K |
| Qwen3-VL 32B | LLM | Supports tool calling and image input (up to 5 images per request) | 32K |
| jina-embeddings-v3 | Embedding | — | — |
| jina-reranker-v2-base-multilingual | Reranker | Multilingual | — |
| bge-reranker-v2-m3 | Reranker | — | — |
* For each model's weight size and recommended GPUs, see [GPU Hardware Planning](/tech/en/platform-development/gpu.md). Embedding and Reranker use only a small portion of GPU memory by default and can share a GPU with the LLM.
* To use tools with an Agent (Agent mode), choose an LLM that supports tool calling; Gemma 3 4B is unsuitable.
* The context values in the table are deployment template defaults, configured according to GPU memory. Contact MaiAgent to assess adjustments.
* Contact MaiAgent to assess open-weight models outside the catalog.
## Connecting Existing OpenAI-Compatible Model Services
Each model can be switched to an external service by entering its URL and key. If the model service does not require a key, leave the field blank to use the default value.
**URL rules** (the most common configuration error):
| Model type | Should the URL include `/v1`? | Example |
| ---------- | ----------------------------- | -------------------------- |
| LLM | Yes | `http://10.0.0.20:8001/v1` |
| Embedding | Yes | `http://10.0.0.20:8001/v1` |
| Reranker | **No** | `http://10.0.0.20:8001` |
The system appends `/v1/rerank` when calling Reranker. If the Reranker URL ends in `…/v1`, requests go to `…/v1/v1/rerank`, causing all knowledge base and FAQ searches to fail.
Other rules:
* **Model name**: If the name on the model service differs from the default name in the deployment configuration, explicitly specify the service's model name in the deployment configuration.
* **Changing settings after installation**: After changing a model URL or key, reload the model configuration (about 1 minute in testing); reinstallation is unnecessary. A normal reinstallation skips model configurations that have already been loaded, so changes will not take effect.
## Connecting Cloud Model Services (Your Own Keys)
Private cloud and on-premises deployments use the customer's own cloud accounts and keys when calling cloud models. The deployment configuration accepts credentials for these providers: OpenAI, Anthropic, Azure, AWS Bedrock, Cohere, Voyage, Gemini, Groq, and Google Vertex AI. Each provider supports different model types—LLM, Embedding, or Reranker. See [Cloud Model Inference API Services](/tech/en/platform-development/cloud-model-inference-api-service.md) for the comparison.
* The host must be able to connect to the cloud service.
* **AWS Bedrock**: The IAM identity associated with the keys needs model invocation permissions (including streaming and cross-region inference profiles; Claude models can only be called through inference profiles in most regions). If using Bedrock Guardrails or Bedrock Reranker, add the corresponding permissions. IAM policy examples are included with the deployment package.
* **Bedrock and backend versions**: Backend versions from `2026.Q3-08.24` through `2026.Q3-09.14` cannot use Bedrock keys. Use `2026.Q3-09.15` or later.
* **AI Gateway**: AI Gateway in on-premises deployments can only use self-hosted models. See [Deployment Types and Feature Comparison](/tech/en/platform-development/onprem-deployment-types.md#feature-availability).
### Hybrid Cloud: Local Platform, Cloud Models
Installing the platform in the customer's data center or cloud and using cloud provider models uses the cloud model connection option above, with no additional deployment type or component. Configure it in this order:
{% stepper %}
{% step %}
### Choose a Source for Each Model Type
Choose cloud or host separately for LLM, Embedding, and Reranker. A common combination uses a cloud LLM and runs Embedding and Reranker on the host (keeping that data within the internal network and avoiding the need for a large GPU). Fewer cloud providers support Embedding and Reranker; see [Cloud Model Inference API Services](/tech/en/platform-development/cloud-model-inference-api-service.md).
{% endstep %}
{% step %}
### Allow Host Connections to Provider Endpoints
Only outbound connections from the host to each provider's API endpoints (port 443) are required; providers do not connect back to the host. If a proxy or allowlist is required, inform MaiAgent before deployment. Offline installation environments can also connect this way; see the explanation at the start of this page.
{% endstep %}
{% step %}
### Enter Keys in the Deployment Configuration
Enter the provider credentials in the deployment configuration (for AWS Bedrock, first prepare an IAM identity with model invocation permissions). The deployment configuration contains keys; store it according to company policy. To change keys or models after installation, reload the model configuration; reinstallation is unnecessary.
{% endstep %}
{% step %}
### Verify
After installation or reloading the configuration, run functional acceptance tests to confirm that conversations and knowledge base searches work. Then check [Model Access Permissions](https://docs.maiagent.ai/org/model-access) in the console to confirm that the organization can see these models.
{% endstep %}
{% endstepper %}
{% hint style="warning" %}
* **No automatic switching**: If a cloud model times out or its provider goes down, the platform does not automatically switch to models on the host. Contact MaiAgent during planning if failover is needed.
* **Switching back to on-premises compute**: LLM and Reranker can be switched to models on the host at any time. Once a knowledge base uses an Embedding model, it cannot be changed. To change Embedding, create a new knowledge base and import the documents again.
* **Data flow**: Conversation content is sent to the cloud provider for processing. Only model calls leave the internal network; conversation records, knowledge base files, and vectors remain on the host.
{% endhint %}
## Limitations for Your Own Embedding and Reranker Services
| Limitation | Description |
| ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| The Embedding service must accept two parameters | MaiAgent sends `encoding_format: base64` and `truncate_prompt_tokens`. Services that do not accept them reject the requests, causing all knowledge base files to fail parsing. Confirm support with the model service provider before connecting |
| The Embedding name contains a colon (`:`) | Specify a separate index name using only lowercase English letters, digits, and hyphens. Without it, the knowledge base cannot create a search index |
| Reranker URL | Do not append `/v1`; see the URL rules above |
| Supported provider types | Embedding: vLLM, Ollama, OpenAI, Cohere, Voyage, AWS Bedrock; Reranker: Cohere, Voyage, AWS Bedrock, FlagEmbedding, vLLM. Azure and Google Vertex AI do not support Embedding or Reranker |
Tested example: When accessed through the same OpenAI-compatible model relay service, Cohere Embed v4 worked. Cohere Embed Multilingual v3 did not accept `encoding_format: base64`, and Amazon Titan Text Embeddings V2 did not accept `truncate_prompt_tokens`; neither could be used as Embedding in that environment.
## Frequently Asked Questions
**Knowledge base and FAQ searches fail, but other features work.**\
First check whether the Reranker URL has an extra `/v1`. Correct it, reload the model configuration, and rerun functional acceptance tests to verify.
**All knowledge base files show “Parsing failed” after connecting an external model service.**\
This is usually an Embedding issue: confirm that the service accepts the two parameters listed above and that an index name is specified if the model name contains a colon.
**Different questions receive the same response.**\
The model service is not responding correctly, often because the URL or model name is incorrect.
## Related Pages
* [GPU Hardware Planning](/tech/en/platform-development/gpu.md)
* [Cloud Model Inference API Services](/tech/en/platform-development/cloud-model-inference-api-service.md)
* [System Requirements](/tech/en/platform-development/system-requirements.md)
* User manual: [Model Access Permissions](https://docs.maiagent.ai/org/model-access)
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.
Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:
```
GET https://docs.maiagent.ai/tech/en/platform-development/onprem-model-services.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.