> For the complete documentation index, see [llms.txt](https://docs.maiagent.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.maiagent.ai/tech/en/platform-development/gpu.md).
# GPU Compute Hardware Planning
GPU hardware planning: recommended GPUs and VRAM, hardware configurations, and NVIDIA driver requirements for self-hosted LLM, Embedding, Reranker, or offline speech recognition in a private cloud or
The MaiAgent platform itself does not require a GPU. GPUs are only needed when you self-host model services (LLM, Embedding, Reranker) or offline speech recognition on private cloud or on-premises servers. You can also connect model services to existing inference services or cloud models. See [On-Premises Model Service Integration](/tech/en/platform-development/onprem-model-services.md).
## Models and Recommended GPUs
The following table is based on the deployment package's model catalog. Models run with vLLM. The recommended GPUs come from the deployment templates' settings and notes. For models marked “Contact MaiAgent for an assessment,” the deployment templates do not specify a recommended card.
| Model | Type | Approximate Weight Download Size | Recommended GPU |
| ---------------------------------- | --------- | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Gemma 3 4B | LLM | 10 GB | A single 24 GB-class card, such as L4, A10G, or RTX 4090 |
| Gemma 4 26B A4B | LLM | 52 GB | A single card with at least 80 GB, such as H100, H200, A100 80GB, or RTX PRO 6000 Blackwell (96 GB); cards with 48 GB or less require multi-GPU parallelism. Contact MaiAgent for an assessment |
| Gemma 4 31B | LLM | 63 GB | A dedicated card with at least 80 GB that cannot be shared with other models. An 80 GB card has a context limit of 64K; a 96 GB card can be configured for 128K |
| Qwen3-VL 32B | LLM | 65 GB | A single H200 (141 GB) |
| Qwen3.6 27B (FP8) | LLM | 30 GB | Contact MaiAgent for an assessment. Uses FP8 KV cache by default, requiring Ada Lovelace, Hopper, or Blackwell generation GPUs |
| gpt-oss-20b | LLM | 40 GB | Contact MaiAgent for an assessment. Does not use FP8 KV cache by default and can also run on Ampere generation GPUs (such as A10G and A100) |
| gpt-oss-120b | LLM | 240 GB | Contact MaiAgent for an assessment. Uses FP8 KV cache by default, requiring Ada Lovelace, Hopper, or Blackwell generation GPUs |
| jina-embeddings-v3 | Embedding | 2 GB | Shares a card with the LLM; uses 20% of GPU memory by default |
| jina-reranker-v2-base-multilingual | Reranker | 1 GB | Shares a card with the LLM; uses 12% of GPU memory by default |
| bge-reranker-v2-m3 | Reranker | 2 GB | Shares a card with the LLM; uses 15% of GPU memory by default |
* **Weight download size** is an estimate from the deployment templates for planning disk space and transferring offline deployment materials. It is not the GPU memory used at runtime.
* Runtime GPU memory also includes the KV cache for conversation content. Longer contexts use more memory; the context limits in the table are defaults set in the deployment templates according to card capacity.
* Tested configuration: Gemma 4 26B A4B, Embedding, and Reranker can share a single RTX PRO 6000 Blackwell (96 GB).
* For each model's capabilities (tool calling and image input), see [On-Premises Model Service Integration: Local Model Catalog](/tech/en/platform-development/onprem-model-services.md#local-model-catalog).
## Multiple GPUs and Models Outside the Catalog
* **Models that do not fit on a single card require multi-GPU parallelism**: deployment templates use vLLM tensor parallelism to split a model across multiple cards. The parallel GPU count must match the number of GPUs actually allocated to the model. For example, gpt-oss-120b does not fit on a single card, so the deployment templates provide a multi-GPU configuration example. To determine the number and type of cards needed, provide MaiAgent with your machine's GPU models and quantities for an assessment.
* **Open-weight models outside the catalog** (such as the Llama family) are not included in the deployment package's model catalog and have no ready-made deployment configuration. Contact MaiAgent for an assessment if needed. After assessment, they may be integrated as external OpenAI-compatible model services. See [On-Premises Model Service Integration: Connecting an Existing OpenAI-Compatible Model Service](/tech/en/platform-development/onprem-model-services.md#external-openai-compatible).
## Information Needed Before Estimating GPU Quantities
The number of requests a single card can serve concurrently depends on the model, quantization method, context length, input and output length per request, and acceptable response latency. There is no universal conversion formula, and this guide does not provide a “users per card” figure. Provide the following information for an assessment so MaiAgent can recommend GPU models and quantities:
| Item | Description |
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Number of users and concurrent online ratio | Recommended platform server specifications assume approximately 20% of users are online concurrently. See [System Requirements: Recommended Specifications by User Count](/tech/en/platform-development/system-requirements.md#sizing-by-users); GPUs are sized separately based on concurrent conversations |
| Models to run | Select from the model catalog above; specify whether image input or tool calling is required |
| Use cases | General Q\&A, long-document summaries, or Agent mode (tool calling increases the number of model calls per conversation) |
| Acceptable response time | For example, the wait time until the first character appears |
| Existing GPUs | GPU models and quantities already purchased or planned, and whether they will be shared with Embedding and Reranker |
## Recommended Hardware Configurations
| GPU | VRAM | Applicable Models (Based on Deployment Templates) |
| ------------------------------------------ | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| NVIDIA H200 | 141 GB | Qwen3-VL 32B, Gemma 4 family |
| NVIDIA RTX PRO 6000 Blackwell | 96 GB | Gemma 4 family; Gemma 4 26B A4B can share the card with Embedding and Reranker (tested) |
| NVIDIA H100 | 80 GB | Gemma 4 family |
| NVIDIA A100 | 80 GB | Gemma 4 family; does not support FP8 KV cache |
| NVIDIA RTX 6000 Ada | 48 GB | Gemma 3 4B; larger models require multiple cards. Contact MaiAgent for an assessment |
| NVIDIA L40S | 48 GB | Gemma 3 4B; larger models require multiple cards. Contact MaiAgent for an assessment |
| NVIDIA GB10 (Dell Pro Max GB10, DGX Spark) | 128 GB (shared with the system) | gpt-oss-20b has been run with jina Embedding and Reranker; memory allocation ratios must be adjusted. See the notes below |
## NVIDIA Drivers and Compute Capability
| Item | Requirement |
| ------------------------ | ------------------------------------------------ |
| GPU | NVIDIA GPU with compute capability 7.0 or higher |
| NVIDIA Driver | 525.60 or higher |
| NVIDIA Container Toolkit | Installed automatically during deployment |
* **FP8**: model configurations using FP8 KV cache require Ada Lovelace, Hopper, or Blackwell generation GPUs. Ampere generation GPUs (such as A100) do not support it.
* **GB10**: the GPU and system share 128 GB of memory. A different GPU mounting method is required, and the combined memory allocation ratios for all model services on the same host should not exceed 0.92 (a configuration that has been run uses LLM 0.60, Embedding 0.20, and Reranker 0.12). Inform MaiAgent during deployment.
* Deployment templates support NVIDIA GPUs only. Contact MaiAgent for an assessment of other accelerators.
* For other host specifications (CPU, memory, and operating system), see [System Requirements](/tech/en/platform-development/system-requirements.md).
## Other Components That Use GPUs
* **Offline speech recognition**: speech-to-text services deployed on the host require a GPU.
* **Offline document parsing**: parsing itself does not require a GPU. To describe images in documents, connect an OpenAI-compatible LLM; you can use a local model from the table above.
For more details, contact MaiAgent's professional consultants at
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.
Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:
```
GET https://docs.maiagent.ai/tech/en/platform-development/gpu.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.