> For the complete documentation index, see [llms.txt](https://docs.maiagent.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.maiagent.ai/tech/en/platform-development/celery-periodic-tasks.md).

# Celery Periodic Task Configuration and OAuth Token Refresh

> This document explains how the MaiAgent platform uses the Celery Beat scheduler to run periodic tasks, particularly the implementation of automatic OAuth Token refresh.

## 1. The Role of Celery in MaiAgent

Celery is a distributed task queue system that plays a key role in the MaiAgent platform:

| Task Type              | Description                                                                      | Execution Method                                      |
| ---------------------- | -------------------------------------------------------------------------------- | ----------------------------------------------------- |
| **Asynchronous tasks** | Time-consuming background processing, such as document parsing and vectorization | Celery Workers receive and execute tasks              |
| **Periodic tasks**     | Scheduled maintenance, such as Token refresh and data cleanup                    | Celery Beat triggers tasks according to a schedule    |
| **Delayed tasks**      | Tasks that must run at a specific time, such as scheduled notifications          | Set the countdown or eta parameter to delay execution |
| **Task chains**        | Complex workflows that require multiple steps to run sequentially                | Use Celery Chain to combine multiple tasks            |

Common use cases:

* **Knowledge base processing**: Time-consuming operations such as parsing, chunking, and vectorizing uploaded documents
* **Periodic maintenance**: OAuth Token refresh, expired data cleanup, and system health checks
* **Report generation**: Periodically generate reports such as usage statistics and conversation quality analyses
* **Notification delivery**: Send email notifications and Webhook callbacks in batches

***

## 2. Celery Architecture and Configuration

```mermaid
flowchart LR
    App["Django Application"]
    Beat["Celery Beat<br/>(Scheduler)"]
    Broker["Message Broker<br/>(Redis/RabbitMQ)"]
    Worker1["Worker 1"]
    Worker2["Worker 2"]
    WorkerN["Worker N"]
    DB[("Database")]
    
    App -- "Submit task" --> Broker
    Beat -- "Scheduled trigger" --> Broker
    Broker -- "Dispatch task" --> Worker1
    Broker -- "Dispatch task" --> Worker2
    Broker -- "Dispatch task" --> WorkerN
    Worker1 -- "Read/write data" --> DB
    Worker2 -- "Read/write data" --> DB
    WorkerN -- "Read/write data" --> DB
```

### 2.1 Core Components

* **Celery Beat (Scheduler)**: Triggers periodic tasks according to the configured schedule
* **Message Broker**: Uses Redis or RabbitMQ as the task queue
* **Celery Workers**: Processes that execute tasks and can scale horizontally
* **Result Backend**: Stores task execution results, typically using Redis or a database

### 2.2 Periodic Task Configuration

MaiAgent uses Django Celery Beat to manage periodic tasks:

| Task Name           | Schedule                       | Description                                                 |
| ------------------- | ------------------------------ | ----------------------------------------------------------- |
| OAuth Token refresh | Hourly                         | Automatically refresh OAuth Tokens that are about to expire |
| Session cleanup     | Every day in the early morning | Remove expired user Sessions                                |

**Schedule expression types**:

* **crontab**: Unix cron-like time expressions that support minutes, hours, days of the week, months, and more
* **timedelta**: Runs at a fixed interval, such as every 30 minutes
* **solar**: Schedules tasks based on sunrise and sunset times

***

## 3. Automatic OAuth Token Refresh Task

### 3.1 Task Execution Flow

```mermaid
sequenceDiagram
    participant Beat as Celery Beat
    participant Broker as Redis Queue
    participant Worker as Celery Worker
    participant DB as Database
    participant OAuth as OAuth Provider
    
    Note over Beat: Triggered once per hour
    Beat->>Broker: Send refresh task
    Broker->>Worker: Dispatch task to an available Worker
    
    Worker->>DB: Query Tokens that are about to expire<br/>(valid for less than 1 hour)
    DB-->>Worker: Return list of Tokens to refresh
    
    loop Process each Token
        Worker->>DB: Check whether the Token has client credentials
        alt Complete credentials available
            Worker->>OAuth: Request a new Token using the Refresh Token
            OAuth-->>Worker: Return a new Access Token
            Worker->>DB: Update Token information
        else Credentials missing
            Worker->>DB: Mark as a legacy Token and exclude from processing
        end
    end
    
    Worker->>Broker: Report task completion
```

### 3.2 Query Optimization Strategies

MaiAgent implements several query optimizations to improve Token refresh efficiency:

* **Filter legacy Tokens**: Exclude legacy Tokens without a client\_id and client\_secret
* **Time-window query**: Query only Tokens that will expire within the next hour
* **Batch processing**: Query multiple Tokens awaiting refresh at once to reduce the number of database queries
* **Error handling**: Record detailed error information for Tokens that fail to refresh to simplify troubleshooting

### 3.3 Error Handling

The following errors may occur during Token refresh:

| Error Type                  | Possible Cause                                         | Handling Method                                                       |
| --------------------------- | ------------------------------------------------------ | --------------------------------------------------------------------- |
| **Invalid Refresh Token**   | The user revoked authorization or the Token expired    | Mark as requiring reauthorization and notify the user                 |
| **Network timeout**         | The OAuth service provider is temporarily unavailable  | Automatically retry three times using exponential backoff             |
| **Rate limit**              | The OAuth service request rate limit has been exceeded | Delay the retry to avoid being blocked                                |
| **Client credential error** | The client\_id or client\_secret is incorrect          | Log the error and notify the administrator to check the configuration |

***

## 4. Task Monitoring and Debugging

### 4.1 Task Status Monitoring

MaiAgent provides several ways to monitor Celery task status:

* **Flower monitoring interface**: A Web interface for viewing task execution status and Worker health in real time
* **Logging**: Detailed records of each task's execution time, parameters, and results
* **Metrics collection**: Prometheus integration to collect task execution metrics such as success rates and execution time
* **Alerting**: Automatically sends alerts when tasks fail repeatedly or take an abnormal amount of time to execute

### 4.2 Performance Optimization Recommendations

**Worker pool configuration**:

* **Concurrency adjustment**: Adjust Worker concurrency based on the number of CPU cores and task types
* **Queue separation**: Assign urgent and non-urgent tasks to separate queues
* **Task priority**: Set a higher priority for critical tasks

**Task design principles**:

* **Idempotency**: Ensure tasks can be retried safely without side effects
* **Timeliness**: Set reasonable task expiration times to avoid executing stale tasks
* **Batch processing**: Process large volumes of data in batches to avoid running out of memory

***

## 5. Technical Advantages of MaiAgent's Celery Configuration

### 5.1 Reliability and Stability

* **Task persistence**: Task information is stored in the Message Broker so it is not lost when the system restarts
* **Automatic retries**: Supports automatically retrying failed tasks
* **Graceful shutdown**: Workers finish running tasks before shutting down
* **Health checks**: Regularly checks the operating status of Workers and Beat

### 5.2 Scalability and Performance

* **Horizontal scaling**: Easily add Workers to accommodate traffic growth
* **Task routing**: Assign different task types to dedicated Worker pools
* **Asynchronous execution**: Avoid blocking the main application to improve system responsiveness
* **Batch optimization**: Intelligent batch processing reduces the number of database queries

### 5.3 Operational Convenience

* **Visual monitoring**: Flower provides an intuitive monitoring interface
* **Detailed logs**: Records the complete task execution process to simplify troubleshooting
* **Dynamic configuration**: Adjust schedules without restarting the system
* **Alert integration**: Integrates with enterprise monitoring systems to detect anomalies promptly

***

## 6. Related Technical Documentation

* [OAuth 2.0 Integration and Automatic Token Refresh](/tech/en/advanced-genai-tech/oauth-integration.md) - Learn more about the OAuth Token refresh logic
* [Deployment Architecture](/tech/en/platform-development/architecture.md) - Learn where Celery fits into the overall system architecture

### References

* [Celery Documentation](https://docs.celeryq.dev/)
* [Django Celery Beat](https://django-celery-beat.readthedocs.io/)
* [Flower - Celery Monitoring Tool](https://flower.readthedocs.io/)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.maiagent.ai/tech/en/platform-development/celery-periodic-tasks.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.