Building a Custom Background Task Queue in ASP.NET Core Without Hangfire

Improve Application Performance by Offloading Long-Running Tasks to Background Services Using QueueBackgroundWorkItem

Introduction

When building scalable web applications in ASP.NET Core, it’s often necessary to perform time-consuming tasks, such as sending emails, data processing, or calling external APIs, without blocking the main request-response flow. Running those operations in the background can significantly enhance application performance.

Here, we will learn how to create a custom background task queue and processor without the usage of libraries like Hangfire. We’ll demonstrate how to pass jobs to a background service using a QueueBackgroundWorkItem approach and trigger the background task from an API controller, including sending emails as an example.

Why Use Background Jobs?

Background jobs are essential for tasks that don’t need to block the user’s interaction with the application. For example:

• Email notifications: Sending emails after a user action.
• Long-running processes: Executing data-heavy operations.
• Third-party API calls: Non-blocking interactions with external services.

By queuing these tasks to run in the background, we free up the server to address different requests, enhancing the general efficiency of the application.

Understanding Background Task Queue

ASP.NET Core’s BackgroundService provides a way to implement long-running background tasks. To make this more adaptable, we can set up a background task queue that lets us add tasks to be processed later. The queued tasks will be processed asynchronously by a background worker.

Setting Up the Task Queue

We’ll start by defining an interface for our background task queue:

public interface IBackgroundTaskQueue
{
    void QueueBackgroundWorkItem(Func<IServiceProvider, CancellationToken, Task> workItem);
    Task<Func<IServiceProvider, CancellationToken, Task>> DequeueAsync(CancellationToken cancellationToken);
}

The QueueBackgroundWorkItem method allows queuing tasks, and the DequeueAsync method retrieves them for processing.

Task Queue Implementation

Next, we implement this interface using a ConcurrentQueue and a SemaphoreSlim to signal when a new task is available:

public class BackgroundTaskQueue : IBackgroundTaskQueue
{
    private readonly SemaphoreSlim _signal = new(0);
    private readonly ConcurrentQueue<Func<IServiceProvider, CancellationToken, Task>> _workItems = new();

    public void QueueBackgroundWorkItem(Func<IServiceProvider, CancellationToken, Task> workItem)
    {
        if (workItem == null) throw new ArgumentNullException(nameof(workItem));

        _workItems.Enqueue(workItem);
        _signal.Release(); // Signal that a new item is available
    }

    public async Task<Func<IServiceProvider, CancellationToken, Task>> DequeueAsync(CancellationToken cancellationToken)
    {
        await _signal.WaitAsync(cancellationToken);
        _workItems.TryDequeue(out var workItem);

        return workItem!;
    }
}

This class allows us to enqueue tasks in a thread-safe manner using ConcurrentQueue and signals the background service to start processing when a task is added.

Creating the Background Processor

With the queue in place, we need a background service to process the queued tasks. ASP.NET Core’s BackgroundService is the ideal candidate for this:

public class QueuedProcessorBackgroundService : BackgroundService
{
    private readonly IBackgroundTaskQueue _taskQueue;
    private readonly IServiceProvider _serviceProvider;
    private readonly ILogger _logger;

    public QueuedProcessorBackgroundService(IBackgroundTaskQueue taskQueue,
        IServiceProvider serviceProvider,
        ILoggerFactory loggerFactory)
    {
        _taskQueue = taskQueue;
        _serviceProvider = serviceProvider;
        _logger = loggerFactory.CreateLogger<QueuedProcessorBackgroundService>();
    }

    protected override async Task ExecuteAsync(CancellationToken cancellationToken)
    {
        _logger.LogInformation("Queued Processor Background Service is starting.");

        while (!cancellationToken.IsCancellationRequested)
        {
            var workItem = await _taskQueue.DequeueAsync(cancellationToken);

            try
            {
                await workItem(_serviceProvider, cancellationToken);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, $"Error occurred executing {nameof(workItem)}.");
            }
        }

        _logger.LogInformation("Queued Processor Background Service is stopping.");
    }
}

This service continuously checks the task queue and processes tasks as they are dequeued. If the service is stopped, the cancellation token ensures a graceful shutdown of any ongoing tasks.

Enqueueing Jobs from an API

We can now create an API endpoint that enqueues a job for background processing. The job will resolve required services like IEmailService from the IServiceProvider and process them asynchronously.

[ApiController]
[Route("api/[controller]")]
public class JobController : ControllerBase
{
    private readonly IBackgroundTaskQueue _taskQueue;

    public JobController(IBackgroundTaskQueue taskQueue)
    {
        _taskQueue = taskQueue;
    }

    [HttpPost("enqueue-email")]
    public IActionResult EnqueueEmailJob([FromBody] EmailRequest emailRequest)
    {
        _taskQueue.QueueBackgroundWorkItem(async (serviceProvider, token) =>
        {
            var logger = serviceProvider.GetRequiredService<ILogger<JobController>>();
            var emailService = serviceProvider.GetRequiredService<IEmailService>();

            logger.LogInformation("Email job started");

            try
            {
                // Send the email
                await emailService.SendEmailAsync(emailRequest.To, emailRequest.Subject, emailRequest.Body, token);
                logger.LogInformation("Email job completed successfully.");
            }
            catch (Exception ex)
            {
                logger.LogError(ex, "Error occurred while sending email.");
            }
        });

        return Ok("Email job has been queued.");
    }
}

This API endpoint receives an email request and queues the email-sending job using the task queue.

Complete Example: Sending an Email in Background

To send an email in the background, we’ll define a model EmailRequest to handle the incoming email data and an email service to simulate sending emails.

EmailRequest Model

public class EmailRequest
{
    public string To { get; set; }
    public string Subject { get; set; }
    public string Body { get; set; }
}

IEmailService and EmailService Implementation

public interface IEmailService
{
    Task SendEmailAsync(string to, string subject, string body, CancellationToken token);
}

public class EmailService : IEmailService
{
    private readonly ILogger<EmailService> _logger;

    public EmailService(ILogger<EmailService> logger)
    {
        _logger = logger;
    }

    public async Task SendEmailAsync(string to, string subject, string body, CancellationToken token)
    {
        _logger.LogInformation($"Sending email to {to} with subject {subject}.");
        
        // Simulate email sending delay
        await Task.Delay(2000, token);

        _logger.LogInformation($"Email to {to} sent successfully.");
    }
}

This service simulates sending an email with a small delay. In a real-world scenario, this would involve integrating with an SMTP server or a third-party email provider like SendGrid.

Registering Services in Startup.cs

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddSingleton<IBackgroundTaskQueue, BackgroundTaskQueue>();
        services.AddHostedService<QueuedProcessorBackgroundService>();
        services.AddSingleton<IEmailService, EmailService>();
        services.AddControllers();
        services.AddLogging();
    }

    public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
    {
        if (env.IsDevelopment())
        {
            app.UseDeveloperExceptionPage();
        }

        app.UseRouting();
        app.UseEndpoints(endpoints => { endpoints.MapControllers(); });
    }
}

Testing the API
You can now trigger the email job by sending a POST request to the /api/job/enqueue-email endpoint:

POST /api/job/enqueue-email
Content-Type: application/json

{
    "to": "example@example.com",
    "subject": "Test Email",
    "body": "This is a test email body."
}

This will enqueue the email job, and the background service will process it without blocking the API response.

Best Practices

Respect Cancellation Tokens: Always ensure your background tasks respect the CancellationToken to allow graceful shutdown of tasks.

Error Handling: Implement proper error handling in background jobs to handle any failures and provide appropriate logging.

Dependency Resolution: Use IServiceProvider correctly within QueueBackgroundWorkItem to ensure proper service lifetimes (e.g., scoped services).

Monitoring: Consider logging or monitoring tools to keep track of queued and processed tasks.

Conclusion

Here we built a lightweight solution for running background jobs in ASP.NET Core without relying on external libraries like Hangfire. We created a background service to process tasks, and showed a way to enqueue a task from the queue and send an email. This method helps you to manipulate time-consuming tasks well at the same time as maintaining your app responsive.