#+title:  Configuring Python in Emacs
#+author: Howard X. Abrams
#+date:   2021-11-16
#+tags: emacs python programming

import re

A literate programming file for configuring Python.

#+begin_src emacs-lisp :exports none
  ;;; ha-programming-python --- Python configuration. -*- lexical-binding: t; -*-
  ;;
  ;; © 2021-2023 Howard X. Abrams
  ;;   Licensed under a Creative Commons Attribution 4.0 International License.
  ;;   See http://creativecommons.org/licenses/by/4.0/
  ;;
  ;; Author: Howard X. Abrams <http://gitlab.com/howardabrams>
  ;; Maintainer: Howard X. Abrams
  ;; Created: November 16, 2021
  ;;
  ;; This file is not part of GNU Emacs.
  ;;
  ;; *NB:* Do not edit this file. Instead, edit the original literate file at:
  ;;            ~/src/hamacs/ha-programming-python.org
  ;;       And tangle the file to recreate this one.
  ;;
  ;;; Code:
  #+end_src
* Introduction
The critical part of Python integration with Emacs is running LSP in Python using [[file:ha-programming.org::*direnv][direnv]]. And the question to ask is if the Python we run it in Docker or in a virtual environment.

While Emacs supplies a Python editing environment, we’ll still use =use-package= to grab the latest:
#+begin_src emacs-lisp
  (use-package python
    :after flycheck
    :mode (((rx ".flake8" eol) . conf-mode)
           ((rx "Pipfile" eol) . conf-mode)
           ((rx ".wsgi" eol)   . python-mode))
    :init
    (setq python-indent-guess-indent-offset-verbose nil
          flycheck-flake8-maximum-line-length 120)
    :config
    (setq python-shell-interpreter (or (executable-find "ipython") "python"))

    (flycheck-add-next-checker 'python-pylint 'python-pycompile 'append))
#+end_src


** Keybindings
Instead of memorizing all the Emacs-specific keybindings, we use [[https://github.com/jerrypnz/major-mode-hydra.el][major-mode-hydra]] defined for =python-mode=:

#+BEGIN_SRC emacs-lisp
  (use-package major-mode-hydra
    :after python
    :config
    (defvar ha-python-eval-title (font-icons 'mdicon "run" :title "Python Evaluation"))
    (defvar ha-python-goto-title (font-icons 'faicon "python" :title "Python Symbol References"))
    (defvar ha-python-refactor-title (font-icons 'faicon "recycle" :title "Python Refactoring"))

    (pretty-hydra-define python-evaluate (:color blue :quit-key "C-g"
                                          :title ha-python-eval-title)
      ("Section"
       (("f" python-shell-send-defun "Function/class")
        ("e" python-shell-send-statement "Line")
        (";" python-shell-send-string "Expression"))
       "Entirety"
       (("f" python-shell-send-file "File")
        ("b" python-shell-send-buffer "Buffer")
        ("r" elpy-shell-send-region-or-buffer "Region"))))

    (pretty-hydra-define python-refactor (:color blue :quit-key "C-g"
                                          :title ha-python-refactor-title)
      ("Simple"
       (("r" iedit-mode "Rename"))
       "Imports"
       (("A" python-add-import "Add Import")
        ("a" python-import-symbol-at-point "Import Symbol")
        ("F" python-fix-imports "Fix Imports")
        ("S" python-sort-imports "Sort Imports"))))

    (pretty-hydra-define python-goto (:color pink :quit-key "C-g"
                                      :title ha-python-goto-title)
      ("Statements"
       (("s" xref-find-apropos "Find Symbol" :color blue)
        ("j" python-nav-forward-statement "Next")
        ("k" python-nav-backward-statement "Previous"))
       "Functions"
       (("F" imenu "Jump Function" :color blue)
        ("f" python-nav-forward-defun "Forward")
        ("d" python-nav-backward-defun "Backward")
        ("e" python-nav-end-of-defun "End of" :color blue))
       "Blocks"
       (("u" python-nav-up-list "Up" :color blue)
        (">" python-nav-forward-block "Forward")
        ("<" python-nav-backward-block "Backward"))))

    (major-mode-hydra-define python-mode (:quit-key "C-g" :color blue)
      ("Server"
       (("S" run-python "Start Server")
        ("s" python-shell-switch-to-shell "Go to Server"))
       "Edit"
       (("r" python-refactor/body "Refactor...")
        (">" python-indent-shift-left "Shift Left")
        ("<" python-indent-shift-right "Shift Right"))
       "Navigate/Eval"
       (("e" python-evaluate/body "Eval...")
        ("g" python-goto/body "Go to..."))
       "Docs"
       (("d" python-eldoc-at-point "Docs on Symbol")
        ("D" python-describe-at-point "Describe Symbol")))))
#+end_src

Sections below can add to this with =major-mode-hydra-define+=.

Note: Install the following packages /globally/ for Emacs:
#+begin_src sh
  pip install flake8 flake8-bugbear pylint pyright mypy pycompile black ruff ipython
#+end_src

But certainly add those to each project’s =requirements-dev.txt= file.

iPython has a feature of [[https://ipython.readthedocs.io/en/stable/config/intro.html#python-configuration-files][loading code on startup]] /per profile/. First, create it with:

#+BEGIN_SRC sh
  ipython profile create
#+END_SRC

Next, after reading David Vujic’s [[https://davidvujic.blogspot.com/2025/03/are-we-there-yet.html][Are We There Yet]] essay, I took a look at [[https://github.com/DavidVujic/my-emacs-config?tab=readme-ov-file#python-shell][his Python configuration]], and added the /auto reloading/ feature to the iPython /profile configuration/:

#+BEGIN_SRC python :tangle ~/.ipython/profile_default/ipython_config.py
  c = get_config()  #noqa

  %load_ext autoreload
  %autoreload 2

  # c.InteractiveShellApp.extensions = ['autoreload']
  # c.InteractiveShellApp.exec_lines = ['%autoreload 2']
#+END_SRC

** Isolated Python Environments
While the Python community (and my work at my company) had difficulty transitioning from Python 2 to 3, I often run into issues needing a particular Python version and modules. After playing around with different approaches, I’m finding:
  * Docker environments are nicely isolated, but annoying to work from outside the container
  * The Builtin =venv= is works well for different library modules, but not for different versions
  * The =pyenv= deals with different Python versions, but is overkill for library isolation

While the [[https://github.com/marcwebbie/auto-virtualenv][auto-virtualenv]] project attempts to resolve this, I’m using the [[file:ha-programming.org::*Virtual Environments with direnv][direnv project]] abstraction for situations where I need project-specific isolation in more than just Python.
*** Virtual Environments
Use the built-in module, venv, to create isolated Python environments for specific projects, enabling you to manage dependencies separately.

Create a virtual environment, either in the project’s directory, or in a global spot:

#+BEGIN_SRC sh
  python3 -m venv .venv
  #
  python3 -m venv ~/.venv/my_project/
#+END_SRC

And then activate it:

#+BEGIN_SRC sh
  source ~/.venv/my_project/bin/activate
#+END_SRC

Or add that to a projects' =.envrc=.

Now, do what you need to do with this isolation:

#+BEGIN_SRC sh :tangle no
  pip install -r test-requirements.txt
#+END_SRC
*** Managing Python Versions
[[https://github.com/pyenv/pyenv][Pyenv]] is a tool for managing multiple versions of Python on your machine, allowing you to switch between them easily (see [[https://realpython.com/intro-to-pyenv/][this essay]]). On a Mac, installed it via Homebrew:

#+BEGIN_SRC sh
  brew install readline xz
  brew install pyenv pyenv-virtualenv
#+END_SRC

Or on other systems, use the /system/ Python to install [[https://github.com/pyenv/pyenv][pyenv]] globally:
#+begin_src sh
  pip install pyenv
#+end_src

Make sure we load this in [[file:zshell.org::*Python][the Zsh profile]]:

#+BEGIN_SRC sh
  export PATH="$HOME/.pyenv/bin:$PATH"
  eval "$(pyenv init -)"
  eval "$(pyenv virtualenv-init -)"
#+END_SRC

Install the python versions you need, for instance:

#+BEGIN_SRC sh
  pyenv install 3.9.23
#+END_SRC

Run =pyenv versions= to see what you have installed.

In any particular project directory, use a version you installed by creating a =.python-version= file, or call:

#+BEGIN_SRC sh
  pyenv local 3.9.23
#+END_SRC

And have this in your =.envrc= file for use with [[file:ha-programming.org::*Virtual Environments with direnv][direnv]]:
#+begin_src conf
  use python 3.7.1
#+end_src

Also, you need the following in your =~/.config/direnv/direnvrc= file (which I have):
#+begin_src shell
  use_python() {
    local python_root=$(pyenv root)/versions/$1
    load_prefix "$python_root"
    if [[ -x "$python_root/bin/python" ]]; then
      layout python "$python_root/bin/python"
    else
      echo "Error: $python_root/bin/python can't be executed."
      exit
    fi
  }
#+end_src

Tell Emacs about [[https://github.com/pythonic-emacs/pyenv-mode][pyenv-mode]]:

#+BEGIN_SRC emacs-lisp
  (use-package pyenv-mode
    :config
    (defun setup-pyenv ()
      "Pyenv."
      (setenv "WORKON_HOME" "~/.pyenv/versions")
      (pyenv-mode +1)))
#+END_SRC

Now specify the =pyenv= Python version by calling [[help:pyenv-mode-set][pyenv-mode-set]]:

#+begin_example
  M-x pyenv-mode-set
#+end_example

When you run inferior Python processes (like =run-python=), the process will start inside the specified Python installation. You can unset the current version with:

#+begin_example
  M-x pyenv-mode-unset
#+end_example

Or, we can do it automatically when we get into a project (if the project has a =.python-version= file):

#+BEGIN_SRC emacs-lisp
  (use-package pyenv-mode
    :config
    (defun project-pyenv-mode-set (&rest _)
      "Set pyenv version matching project name."
      (ignore-errors
        (let* ((filename (thread-first
                           (project-current)
                           (project-root)
                           (file-name-concat ".python-version")))
               (version (when (file-exists-p filename)
                          (with-temp-buffer
                            (insert-file-contents filename)
                            (buffer-string)))))
          (when version
            (pyenv-mode-set version)
            (pyenv-mode-unset)))))

    ;; Either set/unset the pyenv version whenever changing tabs:
    (add-hook 'tab-bar-tab-post-select-functions 'project-pyenv-mode-set))
#+END_SRC

*** Docker Environment
Docker allows you to isolate your project's environment. The downside is that you are using Docker and probably a bloated container. On my work laptop, a Mac, this creates a behemoth virtual machine that immediately spins the fans like a wind tunnel.

But, but... think of the dependencies!

Enough of the rant (I go back and forth), after getting Docker installed and running (ooo Podman ... shiny), and you've created a =Dockerfile= for your project, let's install [[https://github.com/snbuback/container-env][container-env]].

Your project's =.envrc= file would contain something like:
#+begin_src shell
  CONTAINER_NAME=my-docker-container
  CONTAINER_WRAPPERS=(python3 pip3 yamllint)
  CONTAINER_EXTRA_ARGS="--env SOME_ENV_VAR=${SOME_ENV_VAR}"

  container_layout
#+end_src
** Editing Python Code
Let’s integrate this [[https://github.com/wbolster/evil-text-object-python][Python support for evil-text-object]] project:
#+begin_src emacs-lisp
  (when (fboundp 'evil-define-text-object)
    (use-package evil-text-object-python
      :hook (python-mode . evil-text-object-python-add-bindings)))
#+end_src
This allows me to delete a Python “block” using ~dal~.
** Unit Tests
#+begin_src emacs-lisp
  (use-package python-pytest
    :after python
    :commands python-pytest-dispatch
    :init
    (use-package major-mode-hydra
      :config
      (defvar ha-python-tests-title (font-icons 'devicon "pytest" :title "Python Test Framework"))
      (pretty-hydra-define python-tests (:color blue :quit-key "q"
                                                :title ha-python-tests-title)
        ("Suite"
         (("a" python-pytest "All")
          ("f" python-pytest-file-dwim "File DWIM")
          ("F" python-pytest-file "File"))
         "Specific"
         (("d" python-pytest-function-dwim "Function DWIM")
          ("D" python-pytest-function "Function"))
         "Again"
         (("r" python-pytest-repeat "Repeat tests")
          ("p" python-pytest-dispatch "Dispatch"))))

      (major-mode-hydra-define+ python-mode (:quit-key "q" :color blue)
        ("Misc"
         (("t" python-tests/body "Tests..."))))))
#+end_src
* LSP Integration of Python
** Dependencies
Each Python project's =requirements-dev.txt= file would reference the [[https://pypi.org/project/python-lsp-server/][python-lsp-server]] (not the /unmaintained/ project, =python-language-server=):

#+begin_src conf :tangle no
  python-lsp-server[all]
#+end_src

*Note:* This does mean, you would have a =tox.ini= with this line:
#+begin_src conf
  [tox]
  minversion = 1.6
  skipsdist = True
  envlist = linters
  ignore_basepython_conflict = True

  [testenv]
  basepython = python3
  install_command = pip install {opts} {packages}
  deps = -r{toxinidir}/test-requirements.txt
  commands = stestr run {posargs}
             stestr slowest
  # ...
#+end_src
** Pyright
I’m using the Microsoft-supported [[https://github.com/Microsoft/pyright][pyright]] package instead, adding it to the global Python scope.

#+begin_src conf :tangle no
  pyright
#+end_src

While the [[https://github.com/emacs-lsp/lsp-pyright][pyright package]] works with LSP, this isn’t needed, as =eglot= works directly with it.
** eglot
The built-in Eglot package natively supports [[https://github.com/Microsoft/pyright][pyright]] via =eglot-server-programs= (no extra package needed):

#+begin_src emacs-lisp
  (use-package eglot
    :straight nil
    :after python
    :defer t
    :hook (python-mode . eglot-ensure)
    :config
    (add-to-list 'eglot-server-programs
                 '((python-mode python-ts-mode) . ("pyright-langserver" "--stdio"))))
#+end_src

Keybindings for the eglot-based LSP sub-menu, wired into the existing =python-mode= hydra:

#+begin_src emacs-lisp
  (defvar ha-python-lsp-title (font-icons 'faicon "python" :title "Python LSP"))

  (use-package eglot
    :straight nil
    :after major-mode-hydra
    :config
    (pretty-hydra-define python-lsp (:color blue :quit-key "q"
                                     :title ha-python-lsp-title)
      ("Server"
       (("D" eglot-shutdown "Disconnect")
        ("R" eglot-reconnect "Reconnect")
        ("?" eglot-show-workspace-configuration "Describe"))
       "Refactoring"
       (("a" eglot-code-actions "Code Actions")
        ("o" eglot-code-action-organize-imports "Organize Imports")
        ("r" eglot-rename "Rename"))
       "Navigation"
       (("." xref-find-definitions "Definition")
        ("/" xref-find-references "References")
        ("t" eglot-find-typeDefinition "Type Definition")
        ("i" eglot-find-implementation "Implementation"))
       "Docs"
       (("d" eldoc-doc-buffer "Eldoc Buffer")
        ("h" eglot-inlay-hints-mode "Inlay Hints")
        ("=" eglot-format-buffer "Format Buffer"))))

    (major-mode-hydra-define+ python-mode (:quit-key "q")
      ("Server"
       ( ;; ("L" eglot-ensure "Start eglot") --> Not needed with eglot-ensure
        ("l" python-lsp/body "LSP...")))))
#+end_src
* Project Configuration
I work with a lot of projects with my team where I need to /configure/ the project such that LSP and my Emacs setup works. Let's suppose I could point a function at a project directory, and have it /set it up/:

#+begin_src emacs-lisp
  (defun ha-python-configure-project (proj-directory)
    "Configure PROJ-DIRECTORY for LSP and Python."
    (interactive "DPython Project: ")

    (let ((default-directory proj-directory))
      (unless (f-exists? ".envrc")
        (message "Configuring direnv")
        (with-temp-file ".envrc"
          ;; (insert "use_python 3.7.4\n")
          (insert "layout_python3\n"))
        (direnv-allow))

      (unless (f-exists? ".pip.conf")
        (message "Configuring pip")
        (with-temp-file ".pip.conf"
          (insert "[global]\n")
          (insert "index-url = https://pypi.python.org/simple\n"))
        (shell-command "pipconf --local")
        (shell-command "pip install --upgrade pip"))

      (message "Configuring pip for eglot/pyright")
      (with-temp-file "requirements-dev.txt"
        (insert "pyright\n"))
      (shell-command "pip install -r requirements-dev.txt")))
#+end_src
* Major Mode Hydra
* Technical Artifacts                                :noexport:
Let's =provide= a name so we can =require= this file:

#+begin_src emacs-lisp :exports none
  (provide 'ha-programming-python)
  ;;; ha-programming-python.el ends here
  #+end_src

#+description: A literate programming file for configuring Python.

#+property:    header-args:sh :tangle no
#+property:    header-args:emacs-lisp  :tangle yes
#+property:    header-args    :results none :eval no-export :comments no mkdirp yes

#+options:     num:nil toc:t todo:nil tasks:nil tags:nil date:nil
#+options:     skip:nil author:nil email:nil creator:nil timestamp:nil
#+infojs_opt:  view:nil toc:t ltoc:t mouse:underline buttons:0 path:http://orgmode.org/org-info.js
