emacs

Fast refiling in org-mode with hydras

Org-mode is a great tool for organizing your todo lists, ideas, and notes. One of the pain points for me, though, is keeping all of my notes in the proper categories.

An unordered list of tasks

Ideally, I want these tasks to be sorted.

Tasks refiled into proper headings

Normal refiling can be pretty slow. In this post, I'll show how to use the hydra package with simple macros and elisp functions to achieve fast refiling (and jumping) with just one or two keystrokes. The end result (code here) looks like this:

Refiling with a hydra

Why use hydras to refile?

Using hydras allow you to quickly repeat commands. Using nested hydras you can make most refile targets available in one or two keystrokes. When you're refiling a bunch of items, this goes from typing

C-c C-w Tasks RET Refile to tasks
C-c C-w Boo RET Refile to "Books to read" which appears as a completion candidate
C-c C-w RET Refile to last location

to

f9 r My hydra keybinding
t Refile to "Tasks"
mb In my "maybe.org" file, refile to "Books to read"
mb (Same as above)

How to refile with hydra

Refiling to a specified location

If you look at the documentation for org-refile, you'll see there's an optional argument RFLOC that allows you to programmatically specify a refile location. It doesn't mention what format it should be in, but a quick search brings up a Stack Exchange answer that shows the format and gives the following example of how to use it:

1
2
3
4
5
6
(defun my/refile (file headline &optional arg)
  (let ((pos (save-excursion
               (find-file file)
               (org-find-exact-headline-in-buffer headline))))
    (org-refile arg nil (list headline file nil pos)))
  (switch-to-buffer (current-buffer)))

This is great! We can use this to bind keys to expressions like (my/refile "someday-maybe.org" "Someday/Maybe"). One problem with this is that we'd have to keep pressing the hotkeys over and over again to refile a bunch of tasks in a row.

Using Hydra

So let's use a hydra to make refiling much faster.

1
2
3
4
5
6
7
8
9
10
11
(defhydra josh/org-refile-hydra (:foreign-keys run) 
  "Refile"
  ("g" (my/refile "shopping.org" "Grocery store") "Refile to Grocery store")
  ("o" (my/refile "shopping.org" "Office supplies") "Refile to Office supplies")
  ("e" (my/refile "tasks.org" "Email tasks") "Email tasks")
  ("r" (my/refile "tasks.org" "Research tasks") "Research tasks")
  ("j" org-refile-goto-last-stored "Jump to last refile")
  ("q" nil "cancel"))
;; Or whatever you want your keybinding to be
(global-set-key (kbd "<f9> r") 'josh/org-refile-hydra/body)

For each keybinding, we provide:

  1. The key (e.g. "g")
  2. The command which refiles to a specific location
  3. A hint, which describes what the key does (e.g. "Refile to Grocery store")

I also use the :foreign-keys key to continue running the hydra, so I can move to a headline using C-n and C-p without exiting the hydra and having to call it again.

With just these commands, you can already begin to refile quickly.

Refiling with a simple hydra

Nested hydras

But what happens when you have a lot of headlines to refile to? If you keep adding headlines, you might get a giant hydra like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
(defhydra josh/org-refile-hydra (:foreign-keys run) 
  "Refile"
  ("g" (my/refile "shopping.org" "Grocery store") "Grocery store")
  ("o" (my/refile "shopping.org" "Office supplies") "Office supplies")
  ("a" (my/refile "shopping.org" "Buy on Amazon") "Buy on Amazon")
  ("e" (my/refile "tasks.org" "Email tasks") "Email tasks")
  ("r" (my/refile "tasks.org" "Research tasks") "Research tasks")
  ("c" (my/refile "tasks.org" "Calendar") "Calendar")
  ("p" (my/refile "tasks.org" "Projects") "Projects")
  ("s" (my/refile "someday-maybe.org" "Someday/Maybe") "Someday/Maybe")
  ("b" (my/refile "media.org" "Books to read") "Books to read")
  ("m" (my/refile "media.org" "Movies to watch") "Movies to watch")
  ("E" (my/refile "ideas.org" "Emacs ideas") "Emacs ideas")
  ("J" (my/refile "ideas.org" "Jokes") "Jokes")
  ("j" org-refile-goto-last-stored "Jump to last refile")
  ("q" nil "cancel"))

You might find that you run out of keybindings fairly quickly. Here we had to start using uppercase letters because lowercase keys were overlapping.

By using nested hydras we can break refiling into two keystrokes: The first for the file and the second for the headline. This moves us from using 26 keys to $26^2 = 676$ possible locations to refile to (and if you use uppercase it's 2704!).

Here's how you can do a simple nested hydra, for refiling into three files (File A, B, and C), each with two headlines (roughly named "1" and "2").

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
(defhydra josh/org-refile-hydra-file-a
 (:color blue :after-exit (josh/org-refile-hydra/body))
 "File A"
 ("1" (my/refile "file-a.org" "Headline 1") "Headline 1")
 ("2" (my/refile "file-a.org" "Headline 2") "Headline 2")
 ("q" nil "cancel"))
(defhydra josh/org-refile-hydra-file-b
 (:color blue :after-exit (josh/org-refile-hydra/body))
 "File B"
 ("1" (my/refile "file-b.org" "One") "One")
 ("2" (my/refile "file-b.org" "Two") "Two")
 ("q" nil "cancel"))
(defhydra josh/org-refile-hydra-file-c
 (:color blue :after-exit (josh/org-refile-hydra/body))
 "File C"
 ("1" (my/refile "file-c.org" "1") "1")
 ("2" (my/refile "file-c.org" "2") "2")
 ("q" nil "cancel"))
(defhydra josh/org-refile-hydra (:foreign-keys run) 
  "Refile"
  ("a" josh/org-refile-hydra-file-a/body "File A" :exit t)
  ("b" josh/org-refile-hydra-file-b/body "File B" :exit t)
  ("c" josh/org-refile-hydra-file-c/body "File C" :exit t)
  ("q" nil "cancel"))

I find the easiest way to construct a nested hydra is to use the :after-exit keyword to pop back to the parent hydra.

We can see that refiling again is pretty straightforward.

Refiling with a nested hydra

Using macros to make hydras

One problem with writing hydras this way is that it gets ugly pretty quickly, with a lot of redundant code. Notice how we repeatedly have to specify the file in each file's hydra, even though it doesn't change within it. Also note that the hydra hint is the same as the headline.

We can write an Elisp macro, to do the heavy lifting in creating our hydras, so we only have to specify a file once:

1
2
3
4
5
6
7
8
(defmacro josh/make-org-refile-hydra (hydraname file keyandheadline)
  "Make a hydra named HYDRANAME with refile targets to FILE.
KEYANDHEADLINE should be a list of cons cells of the form (\"key\" . \"headline\")"
  `(defhydra ,hydraname (:color blue :after-exit (josh/org-refile-hydra/body))
     ,file
     ,@(cl-loop for kv in keyandheadline
		collect (list (car kv) (list 'my/refile file (cdr kv)) (cdr kv)))
     ("q" nil "cancel")))

This makes defining our file hydras way simpler. Now our previous code simplifies to:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
(josh/make-org-refile-hydra josh/org-refile-hydra-file-a
			    "file-a.org"
			    (("1" . "Headline 1")
			     ("2" . "Headline 2")))
(josh/make-org-refile-hydra josh/org-refile-hydra-file-b
			    "file-b.org"
			    (("1" . "One")
			     ("2" . "Two")))
(josh/make-org-refile-hydra josh/org-refile-hydra-file-c
			    "file-c.org"
			    (("1" . "1")
			     ("2" . "2")))
(defhydra josh/org-refile-hydra (:foreign-keys run) 
  "Refile"
  ("a" josh/org-refile-hydra-file-a/body "File A" :exit t)
  ("b" josh/org-refile-hydra-file-b/body "File B" :exit t)
  ("c" josh/org-refile-hydra-file-c/body "File C" :exit t)
  ("q" nil "cancel"))

Great! Now our code is a bit cleaner, and adding new refile targets is as easy as adding a ("Key" . "Headline") pair to a list.

Extra goodies

Using the previous code allows you to do most of your refiling tasks quickly and easily. I've made some modifications to the code to have the following features.

  1. Be able to jump (without refiling) to a headline by using a C-u prefix
    • And quit the hydra if we jump, so we don't accidentally refile things on the next key press.
  2. Be able to refile while you're using org-capture
  3. Be able to refile in org-agenda
  4. Deal with some bugs:
    • Don't refile when a headline doesn't exist
    • Use with-current-buffer so we don't accidentally switch to the buffer we're refiling to

All the code for this is available here, as a gist.

Quickly jump with our hydra

Note: Problems with current methods

Why did I use hydras to refile in the first place? There were a few problems I was having with org-refile. Some of these can be solved with some customizations that I wasn't aware of before.

  • Selecting refile targets can be slow

    (length (org-map-entries 1 nil 'agenda)) tells me I have 5208 entries in my agenda files at this time of writing. Even using a narrowing framework like helm, it takes a long time to narrow down to the headline I want. And generally there are specific categories I do most of my refiling to. (Note: You could use the :tag setting in org-refile-targets to limit refile targets to specific headlines. This might work well)

    You can limit the :maxlevel of org-refile-targets, but unless I set it to "1" this doesn't speed things up much since most of my entries are level 1 or 2. Also, I like the ability to refile to any location when I need to.

    One option is to set org-refile-cache, so you don't have to generate all refile targets every time you refile.

    You can also use org-refile-target-verify-function to limit locations you refile to, like "non-DONE tasks", "only projects", "headlines with children", etc. This can become slow as the function needs to be run on all possible candidates. (But in combination with org-refile-cache, this can be less painful.)

  • Refiling many headlines can be really slow

    If you follow Bernt Hansen's org-mode guide, you might have a "refile.org" capture bucket of things to refile later. For me, sequential headlines aren't necessarily related and can be refiled to many places. Picking places, again, takes several keystrokes, which is kind of annoying.

  • Why not just use org-capture to directly refile to headlines?

    Normally, when using org-capture, I actually do capture directly to the headline I want. A problem with this, though, is that I need to have capture templates not just for each headline I want to capture to, but for each style/template I'd like to capture. I'd like to have different templates for things like "events", "events mentioned in emails" (so org-capture can automatically add the email link), "emails I need to reply to" and so forth.

    You can refile from org-capture, but again this runs into my first problem.

  • Why not batch refile with org-agenda?

    I do this sometimes too. However, sometimes getting things I want to refile into the agenda in the first place is really cumbersome, and I'd like to navigate the tree structure of org mode (rather than flattening things with org-agenda).

I still use normal org-refile somewhat frequently, but refiling and jumping using hydras saves me a fair amount of keystrokes