A Case Against Jupyter Notebooks

Summary

You may not need Jupyter notebooks. As an Emacs user, IPython and Jupytext is all I want.

Observations

After a couple of weeks of daily Jupyter notebook use (yes, I did learn most keyboard shortcuts—why do you ask?), I really grew frustrated about the UX:

• Enter/Return logic inverted: contrary to many chat apps, in notebooks, Shift-Return executes the cell, Return moves the cursor to the next line.
• Weird selection
  • Selecting text, with or without mouse, does not auto-copy to clipboard. We need to explicitly Ctrl-c.
  • Mouse-selecting may or may not edit a cell. Markdown cells need re-execution.
• Navigation
  • ArrowUp inside a cell at the top moves the cursor to the previous cell. Same for ArrowDown at the bottom. Markdown cells need re-execution.
  • Weird scrolling. I commonly struggle to scroll and display the exact portion I want.
  • The viewport doesn’t center around the currently edited cell. Adding a cell after the current cell may place it half outside the viewport. The edited cell may also be completely outside the viewport.
  • If I’m viewing a cell, while a distant one is inadvertently left edited, I cannot select nor edit the currently viewed cells. I believe I need to exit edit mode first.
• A inserts a cell Before, and B After. grep has it the other way around¹.
• The kernel legendarily needs occasional restarting—for example when the page goes blank 😳—leaving the user unsure about the notebook’s state. I typically re-run all cells above 🕙.

I recognize I may have missed some important, non-default Jupyter options—there is maybe an auto-center option after all.

So all this was really clashing with my usual workflow. Things like copy-pasting from a notebook to Emacs and vice-versa, or simply editing Python code in a rudimentary web interface.

Maybe for mouse users most of this is irrelevant. 🤷

Alternatives

I thus looked for ways to maybe edit code in Emacs and have it auto-magically synced to Jupyter. I did find some but they were either semi-abandoned or too slow for my taste:

• emacs-jupyter
• jupyter-ascending
• ein

I also stumbled upon jupytext, which is “just” an ipynb/py converter. But I couldn’t figure out how to make it play nice with notebooks.

Well, turns out, you don’t.

I then found code-cells for Emacs, which supports Jupytext’s py:percent format, and finally understood:

• code-cells directly interacts with an IPython process.
• Jupyter is a layer on top of IPython². 💡

So all I needed really is an Emacs interface to IPython. It turns out it exists and is pretty mature actually.

Results

My new (yet unpolished) setup looks like this:

• Install pyqt5 (with your favorite package manager or pip install pyqt5)
• Update Emacs’ config:

(setq
  python-shell-interpreter "ipython3"
  python-shell-interpreter-args "--simple-prompt --no-color-info") ; -i

(use-package code-cells
  :ensure t
  :config
  (add-to-list 'code-cells-eval-region-commands '(python-base-mode . python-shell-send-region))
  (let ((map code-cells-mode-map))
  :bind
  (:map code-cells-mode-map
        ("M-p"          . code-cells-backward-cell)
        ("M-n"          . code-cells-forward-cell)
        ("C-<return>"   . code-cells-eval)
        ("C-S-<return>" . code-cells-eval-and-step)
        ("M-<return>"   . outline-insert-heading)
        ("C-S-<tab>"    . outline-cycle-buffer)
        ("C-<backtab>"  . outline-cycle-buffer))
  :hook
  (python-base-mode . code-cells-mode-maybe))

• Add IPython initialization:

# ~/.ipython/profile_default/startup/00-startup.py
import os
import numpy as np
import pandas as pd
# set matplotlib backend if desired
# %matplotlib qt  # note: magic won't run from plain .py; use programmatic backend below
get_ipython().run_line_magic("matplotlib", "qt")

from IPython.display import HTML
from sklearn.utils import estimator_html_repr

import tempfile
import webbrowser
from pathlib import Path

def display_estimator(pipe):
    """Display estimator/pipeline/transformer in browser"""
    html = HTML(estimator_html_repr(pipe))

    with tempfile.NamedTemporaryFile('w', suffix='.html', delete=False, encoding='utf-8') as f:
        f.write(html.data)
        path = Path(f.name).as_uri()

    webbrowser.open(path)
    print(f"Estimator saved to: {f.name}")

print("IPython startup: loaded numpy, pandas; matplotlib set to Qt5Agg")

My workflow so far looks like this:

• Convert existing notebook: jupytext --set-formats ipynb,py:percent notebook.ipynb.
• Open resulting percent-formatted notebook.py in Emacs.
• Open *Python* buffer with run-python (C-c C-p).
• Enjoy editing and fast eval in IPython 😎. Quick back-and-forth between script and ipython, copy-paste as usual, browse documentation with usual copy-paste, auto-complete, etc. ❤️
• Convert back to .ipynb. TODO no need so far.

Plots are displayed in external Qt windows. Estimators in the browser via display_estimator(). To render LaTex formulas, I copy-paste them to my markdown notes. Emacs markdown-mode supports math/latex. I just auto-convert to html for live-view with pandoc:

watch:
	ls $(DOC).md | entr -r pandoc --standalone --mathjax -o $(DOC).html $(DOC).md

Other pros and cons I see:

• (+) Notebooks, Python scripts really, feel much less hairy. I get the idea of notebooks as scratch pads, but they become difficult to navigate fast. Yes, we can discipline ourselves in structuring them with foldable headings. But again, keyboard navigation is… weird.
• (+) Notebook diffs are possible. 🎉
• (-) The obvious downside is that artifacts (plots, tables with data excerpts, rendered markdown—math formulas) are not captured. At least not automatically captured and displayed.

Conclusion

I realize I’m not alone, and Jupyter notebooks present a number of additional challenges.

I’m so happy about my new setup ☀️. I can work with my beloved editor and finally focus on the work 🚀.

As a comparison, VS Code mates enjoy built-in support for .ipynb files. There’s also a Google Colab plugin to run your code on Colab GPUs without ever leaving your IDE 🎉. Which I haven’t been able to replicate in Emacs yet 😏.