#+title:  Literate Programming with Org
#+author: Howard Abrams
#+date:   2024-07-07
#+filetags: emacs hamacs
#+lastmod: [2025-02-20 Thu]

A literate programming file for literate programming in Emacs Org Files.

#+begin_src emacs-lisp :exports none
  ;;; ha-org-literate --- literate programming helpers -*- lexical-binding: t; -*-
  ;;
  ;; © 2024 Howard Abrams
  ;;   Licensed under a Creative Commons Attribution 4.0 International License.
  ;;   See http://creativecommons.org/licenses/by/4.0/
  ;;
  ;; Author: Howard Abrams <http://gitlab.com/howardabrams>
  ;; Maintainer: Howard Abrams
  ;; Created: July  7, 2024
  ;;
  ;; While obvious, GNU Emacs does not include this file or project.
  ;;
  ;;; Commentary:
  ;;
  ;; This file contains a collection of functions to easy some of the
  ;; sharp edges when doing literate programming in Org files.
  ;;
  ;; *NB:* Do not edit this file. Instead, edit the original
  ;; literate file at:
  ;;            ~/src/hamacs/ha-org-literate.org
  ;; And tangle the file to recreate this one.
  ;;
  ;;; Code:
#+end_src
* Introduction
I do a lot of /literate programming/ using capabilities found in the Org project. Over the years, I’ve smoothed some of the rough edges by writing supporting functions, collected below.

What are the advantage of [[https://en.wikipedia.org/wiki/Literate_programming][literate programming]]? I listed some in my essay and video about my [[https://howardism.org/Technical/Emacs/literate-devops.html][literate devops]] ideas, but a brief recap:

  - ambiguous and/or complicated ideas can be reasoned about in prose before code
  - links to ideas, code snippets and supporting projects are more accessible than in comments
  - diagrams written in [[file:ha-org.org::*PlantUML][PlantUML]], [[file:ha-org.org::*Graphviz][Graphviz]], or [[file:ha-org.org::*Pikchr][Pikchr]] maintained and displayed with code
  - full access to org features, like clocks, task lists and agendas
  - mixing languages in the same file, e.g. shell instructions for installing supporting modules before the code that uses that module

If literate programming is so great, why doesn’t everyone do it? Most of the /advantages/ listed above are advantages only because we are using Emacs and Org Mode, and getting teams to adopt this requires changing editors and workflows. Therefore, literate programming is a solo endeavor.

Since we are using Emacs, the downsides of using a literate approach (even for personal projects) can be minimized.

*Note:* What follows is advanced LP usage. I would recommend checking out my [[https://www.howardism.org/Technical/Emacs/literate-programming-tutorial.html][Introduction to LP in Org]] before venturing down this essay.
* Snippets
** Surround with Org Block
This simple change allows me to highlight any text, and surround it in an org block. We

#+BEGIN_SRC snippet :tangle ~/.emacs.d/snippets/org-mode/surround-org-block
  # name: surround-org-block
  # --
  ,#+BEGIN_${1:SRC} ${2:emacs-lisp} $0
  `yas-selected-text`
  ,#+END_$1
#+END_SRC

Without a key (that is intentional), I can call the snippet with ~C-c & C-s~ (ugh).

** Split an Org Block
What is the /language/ or /major-mode/ associated with the “current” block?

#+BEGIN_SRC emacs-lisp
  (defun ha-org-block-language ()
    "Return the language associated with the current block."
    (save-excursion
      (re-search-backward (rx bol (zero-or-more space)
                              "#+begin_src"
                              (one-or-more space)
                              (group (one-or-more (not space)))))
      (match-string 1)))
#+END_SRC

Actually, I probably want all the parameters (if any) to come along for the ride:

#+BEGIN_SRC emacs-lisp
   (defun ha-org-block-language-and ()
     "Return language and parameters of the current block."
     (save-excursion
       (re-search-backward (rx bol (zero-or-more space)
                               "#+begin_src"
                               (one-or-more space)
                               (group (one-or-more any))))
       (match-string 1)))
    #+END_SRC

This allows us to now split a block into two parts:

#+BEGIN_SRC snippet :tangle ~/.emacs.d/snippets/org-mode/split-org-block
  ,# -*- mode: snippet -*-
  ,# name: split-org-block
  ,# key: #split
  ,# --
  ,#+END_SRC
  `yas-selected-text`
  ,#+BEGIN_SRC `(ha-org-block-language-and)`

#+END_SRC

* Visit Tangled File
After tangling a file, you might want to take a look at it. But a single org file can tangle out to more than one file, so how to choose? Each block could have its one file, as would the particular language-specific blocks in a section or the entire file. I don’t care to be too pendantic here, because I often know the name of the file.

Perhaps a little function to map a /language/ to a file name:

#+BEGIN_SRC emacs-lisp
  (defun ha-org-babel-tangle-default (language)
    "Return the default tangled file based on LANGUAGE."
    (format "%s.%s"
            (file-name-sans-extension (buffer-file-name))
            (pcase language
              ("emacs-lisp" "el")
              ("elisp" "el")
              ("python" "py")
              ("ruby" "rb")
              ;; ...
              ("bash" "sh")
              ("sh" "sh")
              ("shell" "sh"))))
    #+END_SRC

This function returns the name of the tangled file based on either the block or a default name based on the file name:

#+BEGIN_SRC emacs-lisp
  (defun ha-org-babel-tangle-file-name ()
    "Return _potential_ file name or Org source.
  If nearest block has a `:tangle' entry, return that.
  If block doesn't have `:tangle' (or set to `yes'),
  return the filename based on the block's language."
    (save-excursion
      (org-previous-block 1)
      (when (looking-at
             (rx "#+begin_src"
                 (one-or-more space)
                 (group (one-or-more (not space)))
                 (one-or-more space) (zero-or-more any)
                 (optional
                  (group ":tangle" (one-or-more space)
                         (optional "\"")
                         (group (one-or-more any))
                         (optional "\"")))))
        (let ((language (match-string 1))
              (filename (match-string 3)))
          (if (or (null filename) (equal filename "yes") (equal filename "true"))
              (ha-org-babel-tangle-default language)
            filename)))))
    #+END_SRC

And this function calls =find-file= on the tangled file (if found):

#+BEGIN_SRC emacs-lisp
  (defun ha-org-babel-tangle-visit-file ()
    "Attempt to call `find-file' on Org's tangled file."
    (interactive)
    (let ((tangled-file (ha-org-babel-tangle-file-name)))
      (if (file-exists-p tangled-file)
          (find-file tangled-file)
        (message "Looking for %s, which doesn't exist." tangled-file))))
#+END_SRC

*Note:* You can /return/ to the original Org file with =org-babel-tangle-jump-to-org=, if you set the tangle =comments= to =link=, as in this global property:

#+begin_example
  ,#+PROPERTY: header-args:emacs-lisp :tangle yes :comments link
#+end_example

* Navigating Code Blocks
:PROPERTIES:
:ID:       3230b1f4-0d2d-47c7-9f3d-fa53083f8c8d
:END:
I’ve been using Oleh Krehel’s (abo-abo) [[https://github.com/abo-abo/avy][Avy project]] to jump around the screen for years, and I just learned that I can wrap the =avy-jump= function to provide either/or regular expression and action to perform.

For instance, the following function can be used to quickly select a source code block, and jump to it:

#+begin_src emacs-lisp
  (defun avy-jump-org-block ()
    "Jump to org block using Avy subsystem."
    (interactive)
    (avy-jump (rx line-start (zero-or-more blank) "#+begin_src")
              :action 'goto-char)
    ;; Jump _into_ the block:
    (forward-line))
#+end_src

I need to take advantage of this feature more.
* Evaluating Code
Hitting ~C-c C-c~ in a source code block /evaluates/ the code. Simple, sure, but the following enhancements make this more accessible.
** Evaluating a Block
:PROPERTIES:
:ID:       93a9695c-67be-448a-b068-9727cd0aa9b0
:END:
At times I would like to jump to a particular block, evaluate the code, and jump back. This seems like a great job for the [[https://github.com/abo-abo/avy][avy project]]. The =avy-jump= function takes a regular expression of text /in the frame/ (which means you can specify text in other windows), and highlights each match. Normally, selecting a match moves the cursor to that match, the =avy-jump= accepts a function to execute instead:

#+begin_src emacs-lisp
  (defun org-babel-execute-src-block-at-point (&optional point)
    "Call `org-babel-execute-src-block' at POINT."
    (save-excursion
      (goto-char point)
      (org-babel-execute-src-block)))

  (defun avy-org-babel-execute-src-block ()
    "Call `org-babel-execute-src-block' on block given by Avy.
  Use Avy subsystem to select a visible Org source code block,
  e.g. `#+begin_src', and then executes the code without moving
  the point."
    (interactive)
    (avy-jump (rx line-start (zero-or-more blank) "#+begin_src")
              :action 'org-babel-execute-src-block-at-point))
#+end_src

In this case, =avy-org-babel-execute-src-block= highlights all /visible blocks/ on the frame, with a letter on each. Selecting the letter, evaluates that block without moving the cursor.

TODO Screenshot of multiple highlighted blocks.
** Evaluating a Section
:PROPERTIES:
:ID:       188e378c-bed4-463c-98d4-d22be1845bc2
:END:
A trick to =org-babel-tangle=, is that it tangles /what Emacs shows/, that is, it tangles /visible/ code blocks after narrowing to the current org section. This means, we can call =org-narrow-to-subtree= to temporary hide everything in the org file except the current heading, evaluate all blocks in the “now visible” buffer, and then widen:

#+begin_src emacs-lisp :results silent
  (defun org-babel-execute-subtree (prefix)
    "Execute all Org source blocks in current subtree.
  The PREFIX is passed to `org-babel-execute-buffer'."
    (interactive "P")
    (save-excursion
      (org-narrow-to-subtree)
      (org-babel-execute-buffer prefix)
      (widen)))
#+end_src
** Editing a Block
:PROPERTIES:
:ID:       f143bbd6-fb4d-45b8-bcfa-196c7a26ed34
:END:
Why navigate to a block, solely to focus on that block in a dedicated buffer, when we can take advantage of the =avy-jump= and edit any visible block?

#+begin_src emacs-lisp
  (defun org-babel-edit-src-block-at-point (&optional point)
    "Call `org-babel-execute-src-block' at POINT."
    (save-excursion
      (goto-char point)
      (org-edit-src-code)))

  (defun avy-org-babel-edit-src-block ()
    "Call `org-edit-src-code' on block given by Avy.
  Use Avy subsystem to select a visible Org source code block,
  e.g. `#+begin_src', and then executes the code without moving
  the point."
    (interactive)
    (avy-jump (rx line-start (zero-or-more blank) "#+begin_src")
              :action
              'org-babel-edit-src-block-at-point))
#+end_src

* Finding Code
One of the issues with literate programming is not being able to use the same interface for moving around code when the source code is in org files.

** XRef Interface
:PROPERTIES:
:ID:       4dc771a4-b974-4b0d-9cc3-a943108c9d3a
:END:
The Emacs interface for jumping to function definitions and variable declarations is called xref (see [[https://www.ackerleytng.com/posts/emacs-xref/][this great article]] for an overview of the interface). I think it would be great to be able, even within the prose of an org file, to jump to the definition of a function that is defined in an org file.

  - [[*Definitions][Definitions]] :: To jump to the line where a macro, function or variable is defined.
  - [[*References][References]] ::  To get a list of all /calls/ or usage of a symbol, but only within code blocks.
  - [[*Apropos][Apropos]] :: To get a list of all references, even within org-mode prose.

In a normal source code file, you know the language, so you have way of figuring out what a symbol is and how it could be defined in that language. In org files, however, one can use multiple languages, even in the same file.

In the code that follows, I’ve made an assumption that I will primarily use this xref interface for Emacs Lisp code, however, it wouldn’t take much (a single regular expression) to convert to another language.

Taking a cue from [[https://github.com/jacktasia/dumb-jump][dumb-jump]], I’ve decided to not attempt to build any sort of [[https://github.com/dedi/gxref/][tag interaction]], but instead, call [[https://github.com/BurntSushi/ripgrep/blob/master/GUIDE.md][ripgrep]]. I love that its =–-json= option outputs much more parseable text.
***  Symbols
I wrote the =ha-literate-symbol-at-point= function as an attempt at being clever with figuring out what sort of symbol references we would want from an org file. I assume that a symbol may be written surrounded by =~= or ~=~ characters (for code and verbatim text), as well as in quotes or braces, etc.

While the goal is Emacs Lisp (and it mostly works for that), it will probably work for other languages as well.

#+begin_src emacs-lisp
  (defun ha-literate-symbol-at-point ()
    "Return an alphanumeric sequence at point.
  Assuming the sequence can be surrounded by typical
  punctuation found in `org-mode' and markdown files."
    (save-excursion
      ;; Position point at the first alnum character of the symbol:
      (cond ((looking-at (rx (any "=~({<\"'“`") alnum))
             (forward-char))
            ;; Otherwise go back to get "inside" a symbol:
            ((not (looking-at (rx alnum)))
             (re-search-backward (rx alnum))))

      ;; Move point to start and end of the symbol:
      (let ((start (progn (skip-chars-backward "a-zA-Z0-9_-") (point)))
            (end   (progn (skip-chars-forward "?a-zA-Z0-9_-") (point))))
        (buffer-substring-no-properties start end))))
#+end_src

Examples of references in an Org file that should work:
  - =ha-literate-symbol-at-point=
  - “ha-literate-symbol-at-point”
  - `ha-literate-symbol-at-point`

This magical incantation connects our function to Xref with an =org= backend:

#+begin_src emacs-lisp
  (cl-defmethod xref-backend-identifier-at-point ((_backend (eql org)))
    (ha-literate-symbol-at-point))
#+end_src
*** Calling ripgrep
This helper function does the work of calling =ripgrep=, parsing its output, and filtering only the /matches/ line. Yes, an interesting feature of =rg= is that it spits out a /sequence/ of JSON-formatted text, so we can use =seq-filter= to grab lines that represent a match, and =seq-map= to “do the work”. Since we have a couple of ways of /doing the work/, we pass in a function, =processor=, which, along with transforming the results, could spit out =nulls=, so the =seq-filter= with the =identity= function eliminates that.

#+begin_src emacs-lisp
  (defun ha-literate--ripgrep-matches (processor regex &optional symb)
    "Return list of running PROCESSOR of `rg' matches from REGEX.
  PROCESSOR is called with an assoc-list of the JSON output from
  the call to ripgrep."
    (let* ((default-directory (if (project-current)
                                  (project-root (project-current))
                                default-directory))
           (search-str (rxt-elisp-to-pcre regex))
           (command (format "rg --json --type org '%s'" search-str)))

      ;; (message "Literate xref Calling: %s" command)
      (message "Searching `%s' %s..." default-directory
               (if symb (format "for `%s' " symb) ""))
      (thread-last command
                   (shell-command-to-list)
                   (seq-map 'ha-literate--parse-rg-line)
                   (seq-filter 'ha-literate--only-matches)
                   (seq-map processor)
                   ;; Remove any nulls from the list:
                   (seq-filter 'identity))))
#+end_src

Note: the =processor= function creates an =xref= object, described below. See =ha-literate--process-rg-line=.

The output from =ripgrep= goes through a couple of transformation functions listed here:

#+begin_src emacs-lisp
  (defun ha-literate--parse-rg-line (line)
    "Process LINE as a JSON object with `json-parse-string'."
    (json-parse-string line :object-type 'alist
                            :array-type 'list))

  (defun ha-literate--only-matches (json-data)
    "Return non-nil if JSON-DATA is an alist with key `type' and value `match'."
    (string-equal "match" (alist-get 'type json-data)))
#+end_src

Since our =ripgrep= searches from the /project root/, but xref wants to make file references relative to the buffer that is calling it, we need to make some changes:

#+BEGIN_SRC emacs-lisp
  (defun ha-literate-make-xref-file (filepath)
    "Return FILEPATH relative to current buffer's file."
    (let ((abspath (expand-file-name filepath
                                     (if (project-current)
                                         (project-root (project-current))
                                       default-directory)))
          (relative-to (file-name-parent-directory (buffer-file-name))))
      (file-relative-name abspath relative-to)))
#+END_SRC

Let’s test this function:
#+BEGIN_SRC emacs-lisp :tangle no
  (ert-deftest ha-literate-make-xref-file-test ()
    ;; Current directory
    (should (equal (ha-literate-make-xref-file "ha-display.org")
                   "ha-display.org"))
    ;; Subdirectory
    (should (equal (ha-literate-make-xref-file "elisp/beep.el")
                   "elisp/beep.el"))
    (should (equal (ha-literate-make-xref-file "~/foo/bar.org")
                   "../../foo/bar.org")))
    #+END_SRC

***  Definitions
As mentioned above,  let’s assume we can use =ripgrep= to search for /definitions/ in Lisp. I choose that because most of my literate programming is in Emacs Lisp. This regular expression should work with things like =defun= and =defvar=, etc. as well as =use-package=, allowing me to search for the /definition/ of an Emacs package:

#+begin_src emacs-lisp
  (defun ha-literate-definition-rx (symb)
    "Return a regular expression to search for definition of SYMB."
    (rx "("
        (or "use-package"
            (seq ; Match both defun and cl-defun:
             (optional "cl-")
             "def" (1+ (not space))))
        (one-or-more space)
        (literal symb)
        word-boundary))

  (defun ha-literate-definition (symb)
    "Return list of `xref' objects of SYMB location in org files.
  The location is based on a regular expression starting with
  `(defxyz SYMB' where this can be `defun' or `defvar', etc."
    (ha-literate--ripgrep-matches 'ha-literate--process-rg-line
                                  (ha-literate-definition-rx symb)
                                  symb))
#+end_src

The work of processing a match for the =ha-literate-definition= function. It calls =xref-make= to create an object for the Xref system. This takes two parameters, the text and the location. We create a location with =xref-make-file-location=.

#+begin_src emacs-lisp
  (defun ha-literate--process-rg-line (rg-data-line)
    "Return an `xref' structure based on the contents of RG-DATA-LINE.
  The RG-DATA-LINE is a convert JSON data object from ripgrep.
  The return data comes from `xref-make' and `xref-make-file-location'."
    (when rg-data-line
      (let-alist rg-data-line
        ;; (message "xref-make %s" .data.path.text)
        (xref-make .data.lines.text
                   (xref-make-file-location
                    ;; Relative filename:
                    (ha-literate-make-xref-file .data.path.text)
                    ;; Line number:
                    .data.line_number
                    ;; Column: Icky to parse:
                    (thread-last
                      (first .data.submatches)
                      (alist-get 'start)))))))
#+end_src

I like the use of =let-alist= where I can access the /parsed/ output from JSON via /variables/, like =.data.path.text=.

We connect this function to the =xref-backend-definitions= list, so that it can be called when we type something like ~M-.~:

#+begin_src emacs-lisp
  (cl-defmethod xref-backend-definitions ((_backend (eql org)) symbol)
    (ha-literate-definition symbol))
#+end_src
***  Apropos
The /apropos/ approach is anything, so the regular expression here is just the symbol, and we can re-use our processor:

#+begin_src emacs-lisp
  (defun ha-literate-apropos (symb)
    "Return an `xref' object for SYMB location in org files.
  The location is based on a regular expression starting with
  `(defxyz SYMB' where this can be `defun' or `defvar', etc."
    (ha-literate--ripgrep-matches 'ha-literate--process-rg-line
                                  (rx word-boundary
                                      (literal symb)
                                      word-boundary)))
#+end_src

And this to /hook it up/:

#+begin_src emacs-lisp
  (cl-defmethod xref-backend-apropos ((_backend (eql org)) symbol)
    (ha-literate-apropos symbol))
#+end_src
***  References
While traditionally, =-apropos= can reference symbols in comments and documentation, searching for /references/ tend to be /calls/ and whatnot. What does that mean in the context of an org file? I’ve decided that references should only show symbols /within org blocks/.

How do we know we are /inside/ an org block?

I call =ripgrep= twice, once to get all the =begin_= and =end_src= lines and their line numbers.
The second =ripgrep= call gets the references.

#+begin_src emacs-lisp
  (defun ha-literate-references (symb)
    "Return list of `xref' objects for SYMB location in org files.
  The location is limited only references in org blocks."
    ;; First, get and store the block line numbers:
    (ha-literate--block-line-numbers)
    ;; Second, call `rg' again to get all matches of SYMB:
    (ha-literate--ripgrep-matches 'ha-literate--process-rg-block
                                  (rx word-boundary
                                      (literal symb)
                                      word-boundary)))
#+end_src

Notice for this function, we need a new processor that limits the results to only matches between the beginning and ending of a block, which I’ll describe later.

The =ha-literate--block-line-numbers= returns a hash where the keys are files, and the value is a series of begin/end line numbers. It calls =ripgrep=, but has a new processor.

#+begin_src emacs-lisp
  (defun ha-literate--block-line-numbers ()
    "Call `ripgrep' for org blocks and store results in a hash table.
  See `ha-literate--process-src-refs'."
    (clrhash ha-literate--process-src-refs)
    (ha-literate--ripgrep-matches 'ha-literate--process-src-blocks
        (rx line-start (zero-or-more blank)
                            "#+" (or "begin" "end") "_src")))
#+end_src

And the function to process the output simply attempts to connect the =begin_src= with the =end_src= lines. In true Emacs Lisp fashion (where we can’t easily, lexically nest functions), we use a global variable:

#+begin_src emacs-lisp
  (defvar ha-literate--process-src-refs
    (make-hash-table :test 'equal)
    "Globabl variable storing org-mode's block line numbers.
  The key in this table is a file name, and the value is a list of
  line numbers marking #+begin_src and #+end_src.")

  (defvar ha-literate--process-begin-src nil
    "Global variable last `#+begin_src' line number.")

  (defun ha-literate--process-src-blocks (rg-data-line)
    "Return nil if RG-DATA-LINE contains a begin_src entry.
  Otherwise return a list of previous begin_src, and the
  current end_src line numbers."
    (let-alist rg-data-line
      (puthash .data.path.text ; filename is the key
               (append
                (gethash .data.path.text ha-literate--process-src-refs)
                (list .data.line_number))
               ha-literate--process-src-refs)))
#+end_src

With a collection of line numbers for all org-blocks in all org files in our project, we can process a particular match from =ripgrep= to see if the match is /within/ a block. Since the key is a file, and =.data.path.text= is the filename, that part is done, but we need a helper to walk down the list.

#+begin_src emacs-lisp
  (defun ha-literate--process-rg-block (rg-data-line)
    "Return an `xref' structure from the contents of RG-DATA-LINE.
  Return nil if the match is _not_ with org source blocks.
  Note that the line numbers of source blocks should be filled
  in the hashmap, `ha-literate--process-src-refs'."
    (let-alist rg-data-line
      (let ((line-nums (thread-first .data.path.text
                                     (gethash ha-literate--process-src-refs)
                                     ;; Turn list into series of tuples
                                     (seq-partition 2))))
        (when (ha-literate--process-in-block .data.line_number line-nums)
          (ha-literate--process-rg-line rg-data-line)))))

  (defun ha-literate--process-in-block (line-number line-numbers)
    "Return non-nil if LINE-NUMBER is inclusive in LINE-NUMBERS.
  The LINE-NUMBERS is a list of two element lists where the first
  element is the starting line number of a block, and the second
  is the ending line number."
    (when line-numbers
      (let ((block-lines (car line-numbers)))
        (if (and (> line-number (car block-lines))
                 (< line-number (cadr block-lines)))
            (car block-lines)
          (ha-literate--process-in-block line-number (cdr line-numbers))))))
#+end_src

The helper function, =ha-literate--process-in-block= is a /recursive/ function that takes each tuple and sees if =line-number= is between them. If it isn’t between any tuple, and the list is empty, then we return =nil= to filter that out later.

Let’s connect the plumbing:

#+begin_src emacs-lisp
  (cl-defmethod xref-backend-references ((_backend (eql org)) symbol)
    (ha-literate-references symbol))
#+end_src

Whew! This is pretty cool to jump out my literate code base as if it were actual =.el= files.
***  Identifier Completion Table
Need the completion table before we can find the references. It actually doesn’t even need to return anything purposeful:

#+begin_src emacs-lisp
  (defun ha-literate-completion-table ())
#+end_src

Now we /hook this up/ to the rest of the system, and the =xref= is now complete:

#+begin_src emacs-lisp
  (cl-defmethod xref-backend-identifier-completion-table ((_backend (eql org)))
    (ha-literate-completion-table))
#+end_src
*** Activation of my Literate Searching
To finish the connections, we need to create a /hook/ that I only allow to turn on with org files:

#+begin_src emacs-lisp
  (defun ha-literate-xref-activate ()
    "Function to activate org-based literate backend.
  Add this function to `xref-backend-functions' hook."
    (when (eq major-mode 'org-mode)
      'org))

  ;; Add this hook to the beginning, as we want to call our
  ;; backend reference before dumb-jump:
  (add-hook 'xref-backend-functions #'ha-literate-xref-activate -100)
#+end_src

At this point, we can jump to functions and variables that I define in my org file, or even references to standard symbols like =xref-make= or =xref-backend-functions=.

I can jump around my literate code as if they were =.el= files. I may want to think about expanding the definitions to figure out the language of the destination.
** Noweb References
A noweb definition, e.g. =<<something-something>>= should /jump/ to the =#name= definition.

Since [[https://github.com/BurntSushi/ripgrep][ripgrep]] is pretty fast, I’ll call it instead of attempting to build a [[https://stackoverflow.com/questions/41933837/understanding-the-ctags-file-format][CTAGS]] table. Oooh, the =rg= takes a =—json= option, which makes it easier to parse.

#+begin_src emacs-lisp :noweb no
  (defun ha-org-noweb-block-jump (str pos)
    "Go to a literate org file containing a symbol, STR.
  The POS is ignored."
    ;; Sometimes I wrap a function name in `=' characters, and these should be removed:
    (when (string-match (rx "<<" (group (one-or-more any)) ">>") str)
      (setq str (match-string 1 str)))
    (ignore-errors
      (let* ((default-directory (project-root (project-current)))
             (command (format "rg --ignore-case --json '#\\+name: +%s' *.org" str))

             (results (thread-last command
                                   shell-command-to-list
                                   second
                                   json-parse-string))
             (file    (thread-last results
                                   (gethash "data")
                                   (gethash "path")
                                   (gethash "text")))
             (line    (thread-last results
                                   (gethash "data")
                                   (gethash "line_number"))))
        (find-file file)
        (goto-line line))))

  (when (fboundp 'evil-goto-definition-functions)
    (add-to-list 'evil-goto-definition-functions 'ha-org-noweb-block-jump))
#+end_src

** Searching by Header
:PROPERTIES:
:ID:       de536693-f0b0-48d0-9b13-c29d7a8caa62
:END:
As large literate programming projects grow, I refine, re-organize and refactor content. I don’t always remember where I put particular code. For instance, in my Emacs configuration, did I configure /eww/, in [[file:ha-config.org][my default config]] file, or did I move it somewhere?  Originally, after loading the file, I  could issue a call to [[file:ha-general.org::*Consult][consult-imenu]] or [[help:org-consult-heading][org-consult-heading]] to get to the right location, but that assumes I have the correct file loaded.

The following section shows some code to use the fuzzy matching features of [[file:ha-config.org::*Orderless][Orderless]], to choose a headline in any of my Org files in a project, and then load that file and jump to that headline. The interface is =ha-hamacs-edit-file-heading=, and the supporting functions begin with =ha-hamacs-edit-=:

#+begin_src emacs-lisp
  (defun ha-hamacs-edit-file-heading (&optional project-root)
    "Edit a file based on a particular heading.
  After presenting list of headings from all Org files,
  it loads the file, and jumps to the line number where
  the heading is located."
    (interactive)
    (let* ((default-directory (or project-root (project-root (project-current))))
           (file-headings (ha-hamacs-edit--file-heading-list))
           (file-choice   (completing-read "Edit Heading: " file-headings))
           (file-tuple    (alist-get file-choice file-headings
                                     nil nil 'string-equal)))
      (find-file (first file-tuple))
      (goto-line (second file-tuple))))
#+end_src

This function  collects all possible headers by issuing a call to =ripgrep=, which returns something like:

#+begin_example
ha-applications.org:29:* Git and Magit
ha-applications.org:85:** Git Gutter
ha-applications.org:110:** Git Delta
ha-applications.org:136:** Git with Difftastic
...
"ha-applications.org:385:* Web Browsing
ha-applications.org:386:** EWW
...
#+end_example

We then filter out non-useful headers (with =ha-hamcs-edit—filter-heading=), and convert the headlines with =ha-hamcs-edit—process-entry= to be more presentable:

#+begin_src emacs-lisp
  (defun ha-hamacs-edit--file-heading-list ()
    "Return list of lists of headlines and file locations.
  This is found by calling `ripgrep' in the `default-directory'.
  Using the output from the shell command, `ha-hamacs-edit-ripgrep-headers',
  it parses and returns something like:

  '((\"Applications∷ Git and Magit\" \"ha-applications.org\" 29)
    (\"Applications∷ Git and Magit ﹥ Git Gutter\" \"ha-applications.org\" 85)
    (\"Applications∷ Git and Magit ﹥ Git Delta\" \"ha-applications.org\" 110)
    (\"Applications∷ Git and Magit ﹥ Time Machine\" \"ha-applications.org\" 265)
    ...)"
    (thread-last ha-hamacs-edit-ripgrep-headers
                 (shell-command-to-list)
                 ;; Let's remove non-helpful, duplicate headings,
                 ;; like Introduction:
                 (seq-remove 'ha-hamacs-edit--filter-heading)
                 ;; Convert the results into both a displayable
                 ;; string as well as the file and line structure:
                 (seq-map 'ha-hamacs-edit--process-entry)))
#+end_src

As the above function’s documentation string claims, I create a list that contains the data structure necessary for =completing-read= as well as the information I need to load/jump to a position in the file. This is a three-element list of the /headline/, /filename/ and /line number/ for each entry:

#+begin_src emacs-lisp :tangle no
  '(("Applications∷ Git and Magit" "ha-applications.org" 29)
    ("Applications∷ Git and Magit ﹥ Git Gutter" "ha-applications.org" 85)
    ("Applications∷ Git and Magit ﹥ Git Delta" "ha-applications.org" 110)
    ("Applications∷ Git and Magit ﹥ Time Machine" "ha-applications.org" 265)
    ("Applications∷ Git and Magit ﹥ Gist" "ha-applications.org" 272)
    ("Applications∷ Git and Magit ﹥ Forge" "ha-applications.org" 296)
    ("Applications∷ Git and Magit ﹥ Pushing is Bad" "ha-applications.org" 334)
    ("Applications∷ Git and Magit ﹥ Github Search?" "ha-applications.org" 347)
    ("Applications∷ ediff" "ha-applications.org" 360)
    ("Applications∷ Web Browsing" "ha-applications.org" 385)
    ("Applications∷ Web Browsing ﹥ EWW" "ha-applications.org" 386)
    ;; ...
    )
#+end_src

We’ll use this shell command to call  =ripgrep= to search my collection of org files:

#+begin_src emacs-lisp
  (defvar ha-hamacs-edit-ripgrep-headers
    (concat "rg"
            " --no-heading"
            " --line-number"
            ;; " --max-depth 1"
            " -e '^\\*+ '"
            " *.org")
    "A ripgrep shell call to search my headers.")
#+end_src

Not every header should be a destination, as many of my org files have duplicate headlines, like *Introduction* and *Technical Artifacts*, so I can create a regular expression to remove or flush entries:

#+begin_src emacs-lisp
  (defvar ha-hamacs-edit-flush-headers
    (rx "*" (one-or-more space)
        (or "Introduction"
            "Install"
            "Overview"
            "Summary"
            "Technical Artifacts"))
    "Regular expression matching headers to purge.")
#+end_src

Note: This variable should be set in the =.dir-locals.el= for a particular project, as in:

#+begin_src emacs-lisp :tangle no
((org-mode . ((ha-hamacs-edit-flush-headers .
               "\\*[[:space:]]+\\(?:Background\\|Summary\\)"))))
#+end_src

And this next function is callable by the filter function, it uses the regular expression and returns true (well, non-nil) if the line entry given, =rg-input=, should be removed:

#+begin_src emacs-lisp
  (defun ha-hamacs-edit--filter-heading (rg-input)
    "Return non-nil if we should remove RG-INPUT.
  These are headings with typical, non-unique entries,
  like Introduction and Summary."
    (string-match ha-hamacs-edit-flush-headers rg-input))
#+end_src

The =seq-map= needs to take each line from the =ripgrep= call and convert it to a list that I can use for the =completing-read= prompt. I love the combination of =seq-let= and =s-match= from Magnar’s [[https://github.com/magnars/s.el][String library]]. The built-in function, =string-match= returns the index in the string where the match occurs, and this is useful for positioning a prompt, in this case, I want the /contents/ of the matches, and =s-match= returns each /grouping/.

#+begin_src emacs-lisp
  (defun ha-hamacs-edit--process-entry (rg-input)
    "Return list of heading, file and line number.
  Parses the line entry, RG-INPUT, from a call to `rg',
  using the regular expression, `ha-hamacs-edit-rx-ripgrep'.
  Returns something like:

    (\"Some Heading\" \"some-file.org\" 42)"
    (seq-let (_ file line level head)
        (s-match ha-hamacs-edit-rx-ripgrep rg-input)
      (list (ha-hamacs-edit--new-heading file head (length level))
            file
            (string-to-number line))))
#+end_src

Before we dive into the implementation of this function, let’s write a test to validate (and explain) what we expect to return:

#+begin_src emacs-lisp :tangle no
  (ert-deftest ha-hamacs-edit--process-entry-test ()
    (setq ha-hamacs-edit-prev-head-list '())
    (should (equal
             (ha-hamacs-edit--process-entry
              "ha-somefile.org:42:* A Nice Headline  :ignored:")
             '("Somefile∷ A Nice Headline  " "ha-somefile.org" 42)))

    ;; For second-level headlines, we need to keep track of its parent,
    ;; and for this, we use a global variable, which we can set for the
    ;; purposes of this test:
    (setq ha-hamacs-edit-prev-head-list '("Parent"))
    (should (equal
             (ha-hamacs-edit--process-entry
              "ha-somefile.org:73:** Another Headline")
             '("Somefile∷ Parent﹥ Another Headline"
               "ha-somefile.org" 73)))

    (setq ha-hamacs-edit-prev-head-list '("Parent" "Subparent"))
    (should (equal
             (ha-hamacs-edit--process-entry
              "ha-somefile.org:73:*** Deep Heading")
             '("Somefile∷ Parent﹥ Subparent﹥ Deep Heading"
               "ha-somefile.org" 73)))

    (setq ha-hamacs-edit-prev-head-list '("Parent" "Subparent"
                                          "Subby" "Deepsubby"))
    (should (equal
             (ha-hamacs-edit--process-entry
              "ha-somefile.org:73:***** Deepest Heading")
             '("Somefile∷ ... Deepest Heading"
               "ha-somefile.org" 73))))
#+end_src

We next need a regular expression to pass to =s-match= to parse the output:
#+begin_src emacs-lisp
  (defvar ha-hamacs-edit-rx-ripgrep
    (rx (group (one-or-more (not ":"))) ":"   ; filename
        (group (one-or-more digit)) ":"       ; line number
        (group (one-or-more "*"))             ; header asterisks
        (one-or-more space)
        (group (one-or-more (not ":"))))      ; headline without tags
    "Regular expression of ripgrep default output with groups.")
#+end_src

The =—new-heading= function will /prepend/ the name of the file and its parent headlines (if any) to the headline to be more useful in both understanding the relative context of the headline, as well as better to search using fuzzy matching.

This /context/ is especially important as =completing-read= will place the most recent choices at the top.

I found the use of =setf= to be quite helpful in manipulating the list of parents. Remember a =list= in a Lisp, is a /linked list/, and we can easily replace one or more parts, by pointing to an new list. This is my first iteration of this function, and I might come back and simplify it.

Essentially, if we get to a top-level headline, we set the list of parents to a list containing that new headline. If we get a second-level headine, =B=, and our parent list is =A=, we create a list =’(A B)= by setting the =cdr= of =’(A)= to the list =’(B)=. The advantage of this approach is that if the parent list is =’(A C D)=, the =setf= works the same, and the dangled /sublist/, =’(C D)= gets garbage collected.

#+begin_src emacs-lisp
  (defun ha-hamacs-edit--new-heading (file head level)
    "Return readable entry from FILE and org headline, HEAD.
  The HEAD headline is, when LEVEL is greater than 1,
  to include parent headlines. This is done by storing
  the list of parents in `ha-hamacs-edit-prev-head-list'."
    ;; Reset the parent list to include the new HEAD:
    (cond
      ((= level 1)
       (setq ha-hamacs-edit-prev-head-list (list head)))
      ((= level 2)
       (setf (cdr ha-hamacs-edit-prev-head-list) (list head)))
      ((= level 3)
       (setf (cddr ha-hamacs-edit-prev-head-list) (list head)))
      ((= level 4)
       (setf (cdddr ha-hamacs-edit-prev-head-list) (list head)))
      ((= level 5)
       (setf (cddddr ha-hamacs-edit-prev-head-list) (list head))))
      ;; Let's never go any deeper than this...

    (format "%s∷ %s"
     (ha-hamacs-edit--file-title file)
     (s-join "﹥ " ha-hamacs-edit-prev-head-list)))
#+end_src

The following test should pass some mustard and explain how this function works:
#+begin_src emacs-lisp :tangle no
  (ert-deftest ha-hamacs-edit--new-heading-test ()
    (should (equal
             (ha-hamacs-edit--new-heading "ha-foobar.org" "Apples" 1)
             "Foobar∷ Apples"))
    (setq ha-hamacs-edit-prev-head-list '("Apples"))
    (should (equal
             (ha-hamacs-edit--new-heading "ha-foobar.org" "Oranges" 2)
             "Foobar∷ Apples﹥ Oranges"))
    (setq ha-hamacs-edit-prev-head-list '("Apples" "Oranges"))
    (should (equal
             (ha-hamacs-edit--new-heading "ha-foobar.org" "Bananas" 3)
             "Foobar∷ Apples﹥ Oranges﹥ Bananas"))
    (setq ha-hamacs-edit-prev-head-list '("Apples" "Oranges" "Bananas"))
    (should (equal
             (ha-hamacs-edit--new-heading "ha-foobar.org" "Cantaloupe" 4)
             "Foobar∷ Apples﹥ Oranges﹥ Bananas﹥ Cantaloupe")))
#+end_src

I store the current list of parents, in the following (/gasp/) /global variable/:

#+begin_src emacs-lisp
  (defvar ha-hamacs-edit-prev-head-list '("" "")
    "The current parents of headlines as a list.")
#+end_src

I would like to make the /filename/ more readable, I use the =s-match= again, to get the groups of a regular expression, remove all the dashes, and use =s-titleize= to capitalize each word:

#+begin_src emacs-lisp
  (defun ha-hamacs-edit--file-title (file)
    "Return a more readable string from FILE."
    (s-with file
      (s-match ha-hamacs-edit-file-to-title)
      (second)
      (s-replace "-" " ")
      (s-titleize)))

  (defvar ha-hamacs-edit-file-to-title
    (rx (optional (or "README-" "ha-"))
        (group (one-or-more any)) ".org")
    "Extract the part of a file to use as a title.")
#+end_src

So the following tests should pass:

#+begin_src emacs-lisp :tangle no
  (ert-deftest ha-hamacs-edit-file-title-test ()
    (should (equal (ha-hamacs-edit-file-title "ha-apples.org") "Apples"))
    (should (equal (ha-hamacs-edit-file-title "apples.org") "Apples"))
    (should (equal (ha-hamacs-edit-file-title "README-apples.org") "Apples"))
    (should (equal (ha-hamacs-edit-file-title "README.org") "Readme")))
#+end_src

* Keybindings
:PROPERTIES:
:ID:       2412ef3b-b5d0-43a3-bd01-764fd92b0c3c
:END:
With a lovely collection of functions, we need to have a way to easily call them. I’ve been using the =pretty-hydra= feature of [[https://github.com/jerrypnz/major-mode-hydra.el][major-mode-hydra]]:

#+begin_src emacs-lisp
  (defvar org-babel--title (font-icons 'faicon "pencil_square_o" :space 2
                                       :title "Literate Programming Support"))
  (pretty-hydra-define org-babel
    (:title org-babel--title :color blue :quit-key "q")
    ("Code Blocks"
     (("g" avy-jump-org-block "Goto ")
      ("j" org-next-block "Previous" :color pink)
      ("k" org-previous-block "Next" :color pink))
     "Evaluate"
     (("o" avy-org-babel-execute-src-block "Block ")
      ("h" org-babel-execute-subtree "Section")
      ("b" org-babel-execute-buffer "Buffer"))
     "Tangle"
     (("t" org-babel-tangle "to Default")
      ("f" org-babel-tangle-file "choose File")
      ("T" org-babel-detangle "from File"))
     "Misc"
     (("e" avy-org-babel-edit-src-block "Edit Block ")
      ("s" org-babel-pop-to-session-maybe "Session REPL")
      ("v" ha-org-babel-tangle-visit-file "Visit Tangled"))))
#+end_src

And tie this hydra into the existing leader system:
#+begin_src emacs-lisp
  (ha-leader :keymaps 'org-mode-map "o s" '("babel" . org-babel/body))
#+end_src
* Technical Artifacts                           :noexport:
Let's =provide= a name so we can =require= this file:

#+begin_src emacs-lisp :exports none
  (provide 'ha-org-literate)
  ;;; ha-org-literate.el ends here
#+end_src

#+DESCRIPTION: literate programming in Emacs Org Files.

#+PROPERTY:    header-args:sh :tangle no
#+PROPERTY:    header-args:emacs-lisp :tangle yes :comments link
#+PROPERTY:    header-args    :results none :eval no-export :comments no mkdirp yes

#+OPTIONS:     num:nil toc:t todo:nil tasks:nil tags:nil date:nil
#+OPTIONS:     skip:nil author:nil email:nil creator:nil timestamp:nil
#+INFOJS_OPT:  view:nil toc:nil ltoc:t mouse:underline buttons:0 path:http://orgmode.org/org-info.js

# Local Variables:
# jinx-local-words: "parseable"
# End:
