Configuring Python in Emacs

Table of Contents

import re

A literate programming file for configuring Python.

Introduction

The critical part of Python integration with Emacs is running LSP in Python using . And the question to ask is if the Python we run it in Docker or in a virtual environment.

While Emacs supplies a Python editing environment, we’ll still use use-package to grab the latest: 

(use-package python
  :after flycheck
  :mode (((rx ".flake8" eol) . conf-mode)
         ((rx "Pipfile" eol) . conf-mode)
         ((rx ".wsgi" eol)   . python-mode))
  :init
  (setq python-indent-guess-indent-offset-verbose nil
        flycheck-flake8-maximum-line-length 120)
  :config
  (setq python-shell-interpreter (or (executable-find "ipython") "python"))

  (flycheck-add-next-checker 'python-pylint 'python-pycompile 'append))

Keybindings

Instead of memorizing all the Emacs-specific keybindings, we use major-mode-hydra defined for python-mode: 

(use-package major-mode-hydra
  :after python
  :config
  (defvar ha-python-eval-title (font-icons 'mdicon "run" :title "Python Evaluation"))
  (defvar ha-python-goto-title (font-icons 'faicon "python" :title "Python Symbol References"))
  (defvar ha-python-refactor-title (font-icons 'faicon "recycle" :title "Python Refactoring"))

  (pretty-hydra-define python-evaluate (:color blue :quit-key "C-g"
                                        :title ha-python-eval-title)
    ("Section"
     (("f" python-shell-send-defun "Function/class")
      ("e" python-shell-send-statement "Line")
      (";" python-shell-send-string "Expression"))
     "Entirety"
     (("f" python-shell-send-file "File")
      ("b" python-shell-send-buffer "Buffer")
      ("r" elpy-shell-send-region-or-buffer "Region"))))

  (pretty-hydra-define python-refactor (:color blue :quit-key "C-g"
                                        :title ha-python-refactor-title)
    ("Simple"
     (("r" iedit-mode "Rename"))
     "Imports"
     (("A" python-add-import "Add Import")
      ("a" python-import-symbol-at-point "Import Symbol")
      ("F" python-fix-imports "Fix Imports")
      ("S" python-sort-imports "Sort Imports"))))

  (pretty-hydra-define python-goto (:color pink :quit-key "C-g"
                                    :title ha-python-goto-title)
    ("Statements"
     (("s" xref-find-apropos "Find Symbol" :color blue)
      ("j" python-nav-forward-statement "Next")
      ("k" python-nav-backward-statement "Previous"))
     "Functions"
     (("F" imenu "Jump Function" :color blue)
      ("f" python-nav-forward-defun "Forward")
      ("d" python-nav-backward-defun "Backward")
      ("e" python-nav-end-of-defun "End of" :color blue))
     "Blocks"
     (("u" python-nav-up-list "Up" :color blue)
      (">" python-nav-forward-block "Forward")
      ("<" python-nav-backward-block "Backward"))))

  (major-mode-hydra-define python-mode (:quit-key "C-g" :color blue)
    ("Server"
     (("S" run-python "Start Server")
      ("s" python-shell-switch-to-shell "Go to Server"))
     "Edit"
     (("r" python-refactor/body "Refactor...")
      (">" python-indent-shift-left "Shift Left")
      ("<" python-indent-shift-right "Shift Right"))
     "Navigate/Eval"
     (("e" python-evaluate/body "Eval...")
      ("g" python-goto/body "Go to..."))
     "Docs"
     (("d" python-eldoc-at-point "Docs on Symbol")
      ("D" python-describe-at-point "Describe Symbol")))))

Sections below can add to this with major-mode-hydra-define+.

Note: Install the following packages globally for Emacs: 

pip install flake8 flake8-bugbear pylint pyright mypy pycompile black ruff ipython

But certainly add those to each project’s requirements-dev.txt file.

iPython has a feature of loading code on startup per profile. First, create it with: 

ipython profile create

Next, after reading David Vujic’s Are We There Yet essay, I took a look at his Python configuration, and added the auto reloading feature to the iPython profile configuration: 

c = get_config()  #noqa

%load_ext autoreload
%autoreload 2

# c.InteractiveShellApp.extensions = ['autoreload']
# c.InteractiveShellApp.exec_lines = ['%autoreload 2']

Isolated Python Environments

While the Python community (and my work at my company) had difficulty transitioning from Python 2 to 3, I often run into issues needing a particular Python version and modules. After playing around with different approaches, I’m finding:

  • Docker environments are nicely isolated, but annoying to work from outside the container
  • The Builtin venv is works well for different library modules, but not for different versions
  • The pyenv deals with different Python versions, but is overkill for library isolation

While the auto-virtualenv project attempts to resolve this, I’m using the direnv project abstraction for situations where I need project-specific isolation in more than just Python.

Virtual Environments

Use the built-in module, venv, to create isolated Python environments for specific projects, enabling you to manage dependencies separately.

Create a virtual environment, either in the project’s directory, or in a global spot: 

python3 -m venv .venv
#
python3 -m venv ~/.venv/my_project/

And then activate it: 

source ~/.venv/my_project/bin/activate

Or add that to a projects’ .envrc.

Now, do what you need to do with this isolation: 

pip install -r test-requirements.txt

Managing Python Versions

Pyenv is a tool for managing multiple versions of Python on your machine, allowing you to switch between them easily (see this essay). On a Mac, installed it via Homebrew: 

brew install readline xz
brew install pyenv pyenv-virtualenv

Or on other systems, use the system Python to install pyenv globally: 

pip install pyenv

Make sure we load this in the Zsh profile: 

export PATH="$HOME/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"

Install the python versions you need, for instance: 

pyenv install 3.9.23

Run pyenv versions to see what you have installed.

In any particular project directory, use a version you installed by creating a .python-version file, or call: 

pyenv local 3.9.23

And have this in your .envrc file for use with direnv: 

use python 3.7.1

Also, you need the following in your ~/.config/direnv/direnvrc file (which I have): 

use_python() {
  local python_root=$(pyenv root)/versions/$1
  load_prefix "$python_root"
  if [[ -x "$python_root/bin/python" ]]; then
    layout python "$python_root/bin/python"
  else
    echo "Error: $python_root/bin/python can't be executed."
    exit
  fi
}

Tell Emacs about pyenv-mode: 

(use-package pyenv-mode
  :config
  (defun setup-pyenv ()
    "Pyenv."
    (setenv "WORKON_HOME" "~/.pyenv/versions")
    (pyenv-mode +1)))

Now specify the pyenv Python version by calling pyenv-mode-set: 

M-x pyenv-mode-set

When you run inferior Python processes (like run-python), the process will start inside the specified Python installation. You can unset the current version with: 

M-x pyenv-mode-unset

Or, we can do it automatically when we get into a project (if the project has a .python-version file): 

(use-package pyenv-mode
  :config
  (defun project-pyenv-mode-set (&rest _)
    "Set pyenv version matching project name."
    (ignore-errors
      (let* ((filename (thread-first
                         (project-current)
                         (project-root)
                         (file-name-concat ".python-version")))
             (version (when (file-exists-p filename)
                        (with-temp-buffer
                          (insert-file-contents filename)
                          (buffer-string)))))
        (when version
          (pyenv-mode-set version)
          (pyenv-mode-unset)))))

  ;; Either set/unset the pyenv version whenever changing tabs:
  (add-hook 'tab-bar-tab-post-select-functions 'project-pyenv-mode-set))

Docker Environment

Docker allows you to isolate your project’s environment. The downside is that you are using Docker and probably a bloated container. On my work laptop, a Mac, this creates a behemoth virtual machine that immediately spins the fans like a wind tunnel.

But, but… think of the dependencies!

Enough of the rant (I go back and forth), after getting Docker installed and running (ooo Podman … shiny), and you’ve created a Dockerfile for your project, let’s install container-env.

Your project’s .envrc file would contain something like: 

CONTAINER_NAME=my-docker-container
CONTAINER_WRAPPERS=(python3 pip3 yamllint)
CONTAINER_EXTRA_ARGS="--env SOME_ENV_VAR=${SOME_ENV_VAR}"

container_layout

Editing Python Code

Let’s integrate this Python support for evil-text-object project: 

(when (fboundp 'evil-define-text-object)
  (use-package evil-text-object-python
    :hook (python-mode . evil-text-object-python-add-bindings)))

This allows me to delete a Python “block” using dal.

Unit Tests

(use-package python-pytest
  :after python
  :commands python-pytest-dispatch
  :init
  (use-package major-mode-hydra
    :config
    (defvar ha-python-tests-title (font-icons 'devicon "pytest" :title "Python Test Framework"))
    (pretty-hydra-define python-tests (:color blue :quit-key "q"
                                              :title ha-python-tests-title)
      ("Suite"
       (("a" python-pytest "All")
        ("f" python-pytest-file-dwim "File DWIM")
        ("F" python-pytest-file "File"))
       "Specific"
       (("d" python-pytest-function-dwim "Function DWIM")
        ("D" python-pytest-function "Function"))
       "Again"
       (("r" python-pytest-repeat "Repeat tests")
        ("p" python-pytest-dispatch "Dispatch"))))

    (major-mode-hydra-define+ python-mode (:quit-key "q" :color blue)
      ("Misc"
       (("t" python-tests/body "Tests..."))))))

Elpy

The Elpy Project expands on the python-mode. 

(use-package elpy
  :init
  (advice-add 'python-mode :before 'elpy-enable)
  :config
  ;; (elpy-enable)
  (setq elpy-test-runner 'elpy-test-pytest-runner)
  (setq elpy-formatter 'black)
  (setq elpy-shell-echo-input nil)
  (setq elpy-modules (delq 'elpy-module-flymake elpy-modules)))

After we’ve loaded the Company section, we can add jedi to the list of completions: 

(use-package elpy
  :after company-mode
  :config (add-to-list 'company-backends 'company-jedi))

Let’s expand our major-mode-hydra with some extras: 

(use-package major-mode-hydra
  :after elpy
  :config

  (pretty-hydra-define python-evaluate (:color blue :quit-key "q"
                                        :title ha-python-eval-title)
    ("Section"
     (("F" elpy-shell-send-defun "Function")
      ("E" elpy-shell-send-statement "Statement")
      (";" python-shell-send-string "Expression"))
     "Entirety"
     (("B" elpy-shell-send-buffer "Buffer")
      ("r" elpy-shell-send-region-or-buffer "region"))
     "And Step..."
     (("f" elpy-shell-send-defun-and-step "Function" :color pink)
      ("e" elpy-shell-send-statement-and-step "Statement" :color pink))))

  (pretty-hydra-define+ python-refactor nil
    ("Elpy"
     (("r" elpy-refactor-rename "Rename")
      ("i" elpy-refactor-inline "Inline var")
      ("v" elpy-refactor-extract-variable "To variable")
      ("f" elpy-refactor-extract-function "To function")
      ("a" elpy-refactor-mode "All..."))))

  (major-mode-hydra-define+ python-mode (:quit-key "q" :color blue)
    ("Server"
     (("s" elpy-shell-switch-to-shell "Go to Server")
      ("C" elpy-config "Config Elpy"))
     "Edit"
     (("f" elpy-black-fix-code "Fix/format code"))
     "Docs"
     (("d" elpy-eldoc-documentation "Describe Symbol")
      ("D" elpy-doc "Docs Symbol")))))

Anaconda

The anaconda-mode project seems as good as Elpy, but also include Evil keybindings. 

(use-mode anaconda-mode
  :hook ((python-mode . anaconda-mode)
         (python-mode ../ anaconda-eldoc-mode)))

Since we are using

LSP Integration of Python

Dependencies

Each Python project’s requirements-dev.txt file would reference the python-lsp-server (not the unmaintained project, python-language-server): 

python-lsp-server[all]

Note: This does mean, you would have a tox.ini with this line: 

[tox]
minversion = 1.6
skipsdist = True
envlist = linters
ignore_basepython_conflict = True

[testenv]
basepython = python3
install_command = pip install {opts} {packages}
deps = -r{toxinidir}/test-requirements.txt
commands = stestr run {posargs}
           stestr slowest
# ...

Pyright

I’m using the Microsoft-supported pyright package instead. Adding this to my requirements.txt files: 

pyright

The pyright package works with LSP. 

(use-package lsp-pyright
    :hook (python-mode . (lambda () (require 'lsp-pyright)))
    :init (when (executable-find "python3")
              (setq lsp-pyright-python-executable-cmd "python3")))

Keybindings

Now that the LSP Integration is complete, we can stitch the two projects together, by calling lsp. I oscillate between automatically turning on LSP mode with every Python file, but I sometimes run into issues when starting, so I conditionally turn it on. 

(defvar ha-python-lsp-title (font-icons 'faicon "python" :title "Python LSP"))

(defun ha-setup-python-lsp ()
  "Configure the keybindings for LSP in Python."
  (interactive)

  (pretty-hydra-define python-lsp (:color blue :quit-key "q"
                                   :title ha-python-lsp-title)
    ("Server"
     (("D" lsp-disconnect "Disconnect")
      ("R" lsp-workspace-restart "Restart")
      ("S" lsp-workspace-shutdown "Shutdown")
      ("?" lsp-describe-session "Describe"))
     "Refactoring"
     (("a" lsp-execute-code-action "Code Actions")
      ("o" lsp-organize-imports "Organize Imports")
      ("l" lsp-avy-lens "Avy Lens"))
     "Toggles"
     (("b" lsp-headerline-breadcrumb-mode "Breadcrumbs")
      ("d" lsp-ui-doc-mode "Documentation Popups")
      ("m" lsp-modeline-diagnostics-mode "Modeline Diagnostics")
      ("s" lsp-ui-sideline-mode "Sideline Mode"))
     ""
     (("t" lsp-toggle-on-type-formatting "Type Formatting")
      ("h" lsp-toggle-symbol-highlight "Symbol Highlighting")
      ("L" lsp-toggle-trace-io "Log I/O"))))

  (pretty-hydra-define+ python-goto (:quit-key "q")
    ("LSP"
     (("g" lsp-find-definition "Definition")
      ("d" lsp-find-declaration "Declaration")
      ("r" lsp-find-references "References")
      ("t" lsp-find-type-definition "Type Definition"))
     "Peek"
     (("D" lsp-ui-peek-find-definitions "Definitions")
      ("I" lsp-ui-peek-find-implementation "Implementations")
      ("R" lsp-ui-peek-find-references "References")
      ("S" lsp-ui-peek-find-workspace-symbol "Symbols"))
     "LSP+"
     (("u" lsp-ui-imenu "UI Menu")
      ("i" lsp-find-implementation "Implementations")
      ("h" lsp-treemacs-call-hierarchy "Hierarchy")
      ("E" lsp-treemacs-errors-list "Error List"))))

  (major-mode-hydra-define+ python-mode nil
    ("Server"
     (("l" python-lsp/body "LSP..."))
     "Edit"
     (("=" lsp-format-region "Format"))
     "Navigate"
     (("A" lsp-workspace-folders-add "Add Folder")
      ("R" lsp-workspace-folders-remove "Remove Folder"))
     "Docs"
     (("D" lsp-describe-thing-at-point "Describe LSP Symbol")
      ("h" lsp-ui-doc-glance "Glance Help")
      ("H" lsp-document-highlight "Highlight"))))

  (call-interactively 'lsp))

(use-package lsp-mode
  :config
  (major-mode-hydra-define+ python-mode (:quit-key "q")
    ("Server"
     (("L" ha-setup-python-lsp "Start LSP Server")))))

;; ----------------------------------------------------------------------
;; Missing Symbols to be integrated?
;; "0" '("treemacs" . lsp-treemacs-symbols)
;; "/" '("complete" . completion-at-point)
;; "k" '("check code" . python-check)
;; "Fb" '("un-blacklist folder" . lsp-workspace-blacklist-remove)
;; "hs" '("signature help" . lsp-signature-activate)
;; "tT" '("toggle treemacs integration" . lsp-treemacs-sync-mode)
;; "ta" '("toggle modeline code actions" . lsp-modeline-code-actions-mode)
;; "th" '("toggle highlighting" . lsp-toggle-symbol-highlight)
;; "tl" '("toggle lenses" . lsp-lens-mode)
;; "ts" '("toggle signature" . lsp-toggle-signature-auto-activate)

Project Configuration

I work with a lot of projects with my team where I need to configure the project such that LSP and my Emacs setup works. Let’s suppose I could point a function at a project directory, and have it set it up: 

(defun ha-python-configure-project (proj-directory)
  "Configure PROJ-DIRECTORY for LSP and Python."
  (interactive "DPython Project: ")

  (let ((default-directory proj-directory))
    (unless (f-exists? ".envrc")
      (message "Configuring direnv")
      (with-temp-file ".envrc"
        ;; (insert "use_python 3.7.4\n")
        (insert "layout_python3\n"))
      (direnv-allow))

    (unless (f-exists? ".pip.conf")
      (message "Configuring pip")
      (with-temp-file ".pip.conf"
        (insert "[global]\n")
        (insert "index-url = https://pypi.python.org/simple\n"))
      (shell-command "pipconf --local")
      (shell-command "pip install --upgrade pip"))

    (message "Configuring pip for LSP")
    (with-temp-file "requirements-dev.txt"
      (insert "python-lsp-server[all]\n")

      ;; Let's install these extra packages individually ...
      (insert "pyls-flake8\n")
      ;; (insert "pylsp-mypy")
      ;; (insert "pyls-isort")
      ;; (insert "python-lsp-black")
      ;; (insert "pyls-memestra")
      (insert "pylsp-rope\n"))
    (shell-command "pip install -r requirements-dev.txt")))

Major Mode Hydra