How to make sure your Celery Beat Tasks are working

Hugo Bessa
August 28, 2017
<p>Celery is a great tool to run asynchronous tasks. It handles situations where you don't want to lock web requests with time consuming operations or when you want things to happen after some time or even in specific date/time in the future. Another great feature of Celery are periodic tasks. They allow you to create scheduled recurring tasks.</p><p>You probably already know Celery. It's pretty famous in the python community and it's a very well documented lib. Running tasks is somewhat easy. Making sure they are working properly on the other hand is sometimes a more challenging task (pun intended).</p><p>The first thing that comes mind are unit tests. Asynchronous tasks are normally nothing more than functions (or maybe some simple classes if you need a little more control). In our test environment we can call these functions we want to test with test arguments, and see how they behave, eg.: what do they return? Do they update data correctly? Do they create files correctly?</p><p>But how do we know if this tasks are being called correctly? One approach is to use mocks to replace Celery's <code>delay</code> and <code>apply_async</code> methods. By doing this we can check if they are being called correctly and with the right arguments. Also, the <code>always_eager</code> setting allows tasks to run synchronously and it's very handy for debugging and testing.</p><p>[newsletter widget]</p><p>Still, there's another thing we need to worry about when we want to be sure everything is ok with our tasks: are things correctly configured? Celery runs independently of your application and makes use of brokers and results backend to communicate with it. Poorly designed and configured architecture may result in performance issues, bugs and even failure in tasks execution. To check if everything is ok we really need run the whole environment and make integration and stress tests. This is not a very simple job.</p><p>Periodic tasks are particularly hard to automatically test. Just like delayed tasks, periodic tasks are functions and classes, so they are just as hard as normal tasks to unit test. A harder job is to check if the <code>celerybeat</code> configuration was done correctly. Imagine we have a task that is supposed to run every Monday, but we wrongfully configured it to run only in the first day of the month. We would take at least one week to realise the mistake.</p><p>After some hard time dealing with this we at Vinta started thinking: integration testing Celery is hard, specially for periodic tasks; it would be nice to know at least the next execution time of a task for a minimum sanity check. This isn't an automated test, but it is an easy check, that could save lives and contracts. To achieve that we implemented a Django package called <a href="https://github.com/vintasoftware/django-celerybeat-status/">django-celerybeat-status</a>. It integrates with the admin interface and provides some info about periodic tasks such as: the next execution date/time, which function/class it is going to call and its arguments. It is an open source project and contributions are very welcome. All you need to do to start using is to follow these very simple three steps:</p><p>Install the package with <code>pip install django-celerybeat-status</code></p><p>Add <code>'celerybeat_status'</code> to you INSTALLED_APPS in django settings</p><p>Add <code>url(r'^admin/statuscheck/', include('celerybeat_status.urls'))</code> to your Django urls</p><p>You're all set. Just access the Django admin and you will find a link to the magical celerybeat screen in the sidebar.</p><p>That's it for now folks! You can now rest with a little more confidence that  periodic tasks are working like a charm! ;)</p><p>What you people think about the lib? Was it useful? Is there a new feature that you think would help a lot? Let us know in the comments or <a href="https://github.com/vintasoftware/django-celerybeat-status/issues">open an issue in the GitHub repo</a>!</p><p><strong>More on Django and tests</strong><br><a href="https://www.vinta.com.br/blog/2017/dont-forget-stamps-testing-email-content-django/">Don't forget the stamps: testing email content in Django</a><br><a href="https://www.vinta.com.br/blog/2017/how-i-test-my-drf-serializers/">How I test my DRF serializers</a></p>

Related articles

Browse all articles

Two Django Talks: secrets to reliable Celery integration + strategies for API development

Controlling access: a Django permission apps comparison

How to Use Django REST Framework Serializers Effectively

We use Cookies to promote our services. Settings and Privacy.
Accept
Close Cookie Preference Manager
Cookie Settings
By clicking “Accept all”, you agree to the storing of cookies on your device to enhance site navigation, analyze site usage and assist in our marketing efforts. Check our privacy policies.