Logging in Celery and Django

Celery can work with django, it's very simple.

from __future__ import absolute_import
import os
from celery import Celery
from django.conf import settings

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'your_app.settings')
app = Celery('you_app')

# Using a string here means the worker will not have to
# pickle the object when using Windows.
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

Then write some task files named tasks.py under your apps directory, celery will auto discovery and run these tasks.

We can set celery options in django settings file, all the support settings can be founded here .

By default, celery will setup it's own logger, even if you have setuped in django. You can use the setup_logging signal to disable this behaviour.

from celery import signals

@signals.setup_logging.connect
def on_celery_setup_logging(**kwargs):
    pass

And there is also an option named CELERYD_HIJACK_ROOT_LOGGER , I don't know exactly what is this option for, but maybe you can set this to true if the logging still not working properly.

Logs generated by celery are all use celery logger as parent. So we can set a handler named celery to handle these logs.

        'celery': {
            'handlers': ['celery'],
            'level': 'INFO',
            'propagate': False
        },

If you need to setup your logger use a handler can auto rotate log files, you must pay attention here.

The log settings configured in django will be used by django , celery-worker , celery-beat , these are offen run as diffent processes, if you run them on the same node, all of these process will try to write logs to the smae files. And also will try to rotate the log files. So if the file is rotated by an other process, the logs generated by this process will no longer write into the file, logs will be lost.

You can use lsof /logs/celery.log to check if the file is opened by different processes.

To avoid this problem, we should setup loger for django and celery seperatlly. We can use signals.setup_logging to setup logger for celery.

@signals.setup_logging.connect
def on_celery_setup_logging(**kwargs):
    config = {
        'version': 1,
        'disable_existing_loggers': False,
        'formatters': {
            'default': {
                'format': '%(asctime)s
%(process)d/%(thread)d
%(name)s
%(funcName)s %(lineno)s
%(levelname)s
%(message)s',
                'datefmt': "%Y/%m/%d %H:%M:%S"
            }
        },
        'handlers': {
            'celery': {
                'level': 'INFO',
                'class': 'logging.FileHandler',
                'filename': '/logs/celery.log',
                'formatter': 'default'
            },
            'default': {
                'level': 'DEBUG',
                'class': 'logging.StreamHandler',
                'formatter': 'default'
            }
        },
        'loggers': {
            'celery': {
                'handlers': ['celery'],
                'level': 'INFO',
                'propagate': False
            },
        },
        'root': {
            'handlers': ['default'],
            'level': 'DEBUG'
        },
    }

    logging.config.dictConfig(config)

Please notice that you didn't have disable_existing_loggers set to True in django settings files, or it will be disalbe celery logging before your setup.

And you also need to pay more attention here, if your celery-worker and celery-beat are still runing on the same server speartly, they still will try to access the same logging files. Celery provide an option the run worker and beat at same time, just user command like celery -A gafly worker -l info --beat -s /logs/celerybeat-schedule , the worker and the beat will run at the sametime.

Celery worker is working in a multi-process mode, so If you can't use logging.handlers.TimedRotatingFileHandler to auto rotate the logs, otherwise the log file will be rotated by multi-process, there will be a problem.

You can use logrotate provided by the system to rotate the logs, or maybe your celery is running by supervisord, you can just set celery output logs to stdout, and let supervisord to write, manage and rotate your logs.