Get celery work with django
Celery can work with django, it's very simple.
Then write some task files named
tasks.py under your apps directory, celery will auto discovery and run these tasks.
We can set celery options in django settings file, all the support settings can be founded here .
Logging for celery
By default, celery will setup it's own logger, even if you have setuped in django. You can use the
setup_logging signal to disable this behaviour.
And there is also an option named
CELERYD_HIJACK_ROOT_LOGGER , I don't know exactly what is this option for, but maybe you can set this to true if the logging still not working properly.
Logs generated by celery are all use
celery logger as parent. So we can set a handler named
celery to handle these logs.
If you need to setup your logger use a handler can auto rotate log files, you must pay attention here.
The log settings configured in django will be used by
celery-beat , these are offen run as diffent processes, if you run them on the same node, all of these process will try to write logs to the smae files. And also will try to rotate the log files. So if the file is rotated by an other process, the logs generated by this process will no longer write into the file, logs will be lost.
You can use
lsof /logs/celery.log to check if the file is opened by different processes.
To avoid this problem, we should setup loger for django and celery seperatlly. We can use
signals.setup_logging to setup logger for celery.
Please notice that you didn't have
disable_existing_loggers set to
True in django settings files, or it will be disalbe celery logging before your setup.
And you also need to pay more attention here, if your
celery-beat are still runing on the same server speartly, they still will try to access the same logging files. Celery provide an option the run worker and beat at same time, just user command like
celery -A gafly worker -l info --beat -s /logs/celerybeat-schedule , the worker and the beat will run at the sametime.
Celery worker is working in a multi-process mode, so If you can't use
logging.handlers.TimedRotatingFileHandler to auto rotate the logs, otherwise the log file will be rotated by multi-process, there will be a problem.
You can use
logrotate provided by the system to rotate the logs, or maybe your celery is running by
supervisord, you can just set celery output logs to stdout, and let supervisord to write, manage and rotate your logs.