Long Running Jobs

In Coherence, you can run long-running jobs that require more time to complete than a typical request. Long-running jobs are useful for tasks such as processing large files, generating reports, or sending emails.

For example, you can specify a worker process to run a long-running job separately from the main application process. This allows the main application process to continue serving requests while the worker process runs in the background.

With Coherence you can also queue long-running jobs to be processed by a worker process and scale the number of worker processes to handle the load.

Creating a long-running job

Let's create a simple Celery worker that processes a request and sleeps for a specified amount of time. The worker will then return the result of the processing. We'll use Fastapi to create a simple API to trigger the worker process. We'll also use Redis as the message broker to queue the tasks for the worker process.

You can follow along by forking the example repository.

Create an application

Follow the steps in the Getting Started guide to create a new application, collection and environment.

Define your services

In your environment's dashboard, navigate to the Services tab.

Add a backend service

Click the New service button.

• Select FastAPI under Container.
• Enter "backend" under Name.
• Enter the start command uvicorn app.main:app --port 8080 --host 0.0.0.0 --reload.
• Enter "/" under URL path.
• Choose Repository under Container source.
• Click Add one under Repo.
• Paste in the link to the GitHub repository you forked at the start of this guide.
• Click Add repository.
• Enter "/" under Context.
• Enter "Dockerfile" under Dockerfile.
• Select main under Track branch.

Add a worker service

Now, add a worker service to process the long-running job. Under Workers tab, click the Add Row button.

• Enter the name for your worker service. For example, "celery_worker".
• Enter the command to start the worker process. Here we'll use celery -A tasks worker --loglevel=info.

Click Save to create your services.

Add a Redis service

Click the New service button and click Redis.

• Enter redis_queue under Name.
• Enter the version of Redis you want to use. We'll use 6
• Enter the engine for the service. For example, redis.

Click Save to create the Redis service.

Provision, build and deploy your application

Follow the steps in the Getting Started guide to provision, build and deploy your application.