How to schedule a Jupyter notebook to run every day, week, or month
As your team grows, you will likely need to run several notebooks on a regular basis. It can be a self-updated dashboard, a web scraper, or automation of email sending. Conveniently, there are existing solutions that let you very quickly schedule a notebook — potentially saving you a lot of time as you don’t need to come and manually run the notebook.
1. Papermill (local)
Papermill is a tool for parameterizing, executing, and analyzing Jupyter Notebooks. This means that you can add parameters to your notebook and change them in your scheduled runs.
It’s important to note that Papermill doesn’t do scheduling on its own. You can think of it as a tool for executing notebooks that’s easy to pass information into, but the scheduling part needs to be handled by something like crontab.
To install papermill:
pip install papermillTo execute a notebook through papermill:
papermill notebook.ipynb result.ipynbTo schedule a notebook execution, type crontab -e and then:
0 0 * * 1 papermill notebook.ipynb result.ipynbThis will schedule the notebook to run at 00:00 every Monday. Use crontab.guru if you need help with cron expressions. You can explore your listed cron jobs by typing crontab -l.
2. Deepnote (cloud)
Note: The author of this article is affiliated with Deepnote
Deepnote is a free cloud platform that can run your Jupyter notebooks. You can execute your notebooks on a daily or a weekly basis. With Papermill, you need to have your personal computer always on, so the scheduled job can execute. Since Deepnote runs in the cloud, you can simply set it to run at the given interval and close the window, the notebook will be executed at the given time without any extra actions needed from you.
In Deepnote, you can also select to be notified through email if your notebook finishes running.
Check out Deepnote’s docs to find out about notebook scheduling or jump into a live project and try it yourself.
