Run parts of a ipython notebook in a loop / with different input parameter
ipython, jupyter, jupyter-lab, jupyter-notebook, python
Solution
`papermill.execute_notebook` was designed (by Netflix) to let us run notebooks as a function (which aids in code reuse, experiments reproducibility and memory efficiency - it runs notebooks in batch mode unlike standard python REPL):
import papermill as pm
pm.execute_notebook(
'path/to/input.ipynb',
'path/to/output.ipynb',
parameters=dict(alpha=0.6, ratio=0.1)
)
More info: docs | example.
Problem
I have written a ipython notebook, which analyses a dataset. Now I want to use this code to loop over different datasets. The code is split into about 50 cells (including comments, markdown explanations,...). Is there a way to run parts of a notebook in a loop or running a whole notebook with different input parameters? I don't want to merge all cells into one function or download the code as a python script, as I really like to run (and experimenting with) parts of the analysis by executing only certain cells. Basically its refactoring parts of a script into a function and calling the function in a loop, just that the "parts of the script" are notebook cells.