Fast refiling in org-mode with hydras

Org-mode is a great tool for organizing your todo lists, ideas, and notes. One of the pain points for me, though, is keeping all of my notes in the proper categories.

An unordered list of tasks

Ideally, I want these tasks to be sorted.

Tasks refiled into proper headings

Normal refiling can be pretty slow. In this post, I'll show how to use the hydra package with simple macros and elisp functions to achieve fast refiling (and jumping) with just one or two keystrokes. The end result (code here) looks like this:

Refiling with a hydra

Why use hydras to refile?

Using hydras allow you to quickly repeat commands. Using nested hydras you can make most refile targets available in one or two keystrokes. When you're refiling a bunch of items, this goes from typing

C-c C-w Tasks RET Refile to tasks
C-c C-w Boo RET Refile to "Books to read" which appears as a completion candidate
C-c C-w RET Refile to last location

to

f9 r My hydra keybinding
t Refile to "Tasks"
mb In my "maybe.org" file, refile to "Books to read"
mb (Same as above)

How to refile with hydra

Refiling to a specified location

If you look at the documentation for org-refile, you'll see there's an optional argument RFLOC that allows you to programmatically specify a refile location. It doesn't mention what format it should be in, but a quick search brings up a Stack Exchange answer that shows the format and gives the following example of how to use it:

1
2
3
4
5
6
(defun my/refile (file headline &optional arg)
(let ((pos (save-excursion
(find-file file)
(org-find-exact-headline-in-buffer headline))))
(org-refile arg nil (list headline file nil pos)))
(switch-to-buffer (current-buffer)))

This is great! We can use this to bind keys to expressions like (my/refile "someday-maybe.org" "Someday/Maybe"). One problem with this is that we'd have to keep pressing the hotkeys over and over again to refile a bunch of tasks in a row.

Using Hydra

So let's use a hydra to make refiling much faster.

1
2
3
4
5
6
7
8
9
10
11
(defhydra josh/org-refile-hydra (:foreign-keys run)
"Refile"
("g" (my/refile "shopping.org" "Grocery store") "Refile to Grocery store")
("o" (my/refile "shopping.org" "Office supplies") "Refile to Office supplies")
("e" (my/refile "tasks.org" "Email tasks") "Email tasks")
("r" (my/refile "tasks.org" "Research tasks") "Research tasks")
("j" org-refile-goto-last-stored "Jump to last refile")
("q" nil "cancel"))
;; Or whatever you want your keybinding to be
(global-set-key (kbd "<f9> r") 'josh/org-refile-hydra/body)

For each keybinding, we provide:

  1. The key (e.g. "g")
  2. The command which refiles to a specific location
  3. A hint, which describes what the key does (e.g. "Refile to Grocery store")

I also use the :foreign-keys key to continue running the hydra, so I can move to a headline using C-n and C-p without exiting the hydra and having to call it again.

With just these commands, you can already begin to refile quickly.

Refiling with a simple hydra

Nested hydras

But what happens when you have a lot of headlines to refile to? If you keep adding headlines, you might get a giant hydra like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
(defhydra josh/org-refile-hydra (:foreign-keys run)
"Refile"
("g" (my/refile "shopping.org" "Grocery store") "Grocery store")
("o" (my/refile "shopping.org" "Office supplies") "Office supplies")
("a" (my/refile "shopping.org" "Buy on Amazon") "Buy on Amazon")
("e" (my/refile "tasks.org" "Email tasks") "Email tasks")
("r" (my/refile "tasks.org" "Research tasks") "Research tasks")
("c" (my/refile "tasks.org" "Calendar") "Calendar")
("p" (my/refile "tasks.org" "Projects") "Projects")
("s" (my/refile "someday-maybe.org" "Someday/Maybe") "Someday/Maybe")
("b" (my/refile "media.org" "Books to read") "Books to read")
("m" (my/refile "media.org" "Movies to watch") "Movies to watch")
("E" (my/refile "ideas.org" "Emacs ideas") "Emacs ideas")
("J" (my/refile "ideas.org" "Jokes") "Jokes")
("j" org-refile-goto-last-stored "Jump to last refile")
("q" nil "cancel"))

You might find that you run out of keybindings fairly quickly. Here we had to start using uppercase letters because lowercase keys were overlapping.

By using nested hydras we can break refiling into two keystrokes: The first for the file and the second for the headline. This moves us from using 26 keys to $26^2 = 676$ possible locations to refile to (and if you use uppercase it's 2704!).

Here's how you can do a simple nested hydra, for refiling into three files (File A, B, and C), each with two headlines (roughly named "1" and "2").

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
(defhydra josh/org-refile-hydra-file-a
(:color blue :after-exit (josh/org-refile-hydra/body))
"File A"
("1" (my/refile "file-a.org" "Headline 1") "Headline 1")
("2" (my/refile "file-a.org" "Headline 2") "Headline 2")
("q" nil "cancel"))
(defhydra josh/org-refile-hydra-file-b
(:color blue :after-exit (josh/org-refile-hydra/body))
"File B"
("1" (my/refile "file-b.org" "One") "One")
("2" (my/refile "file-b.org" "Two") "Two")
("q" nil "cancel"))
(defhydra josh/org-refile-hydra-file-c
(:color blue :after-exit (josh/org-refile-hydra/body))
"File C"
("1" (my/refile "file-c.org" "1") "1")
("2" (my/refile "file-c.org" "2") "2")
("q" nil "cancel"))
(defhydra josh/org-refile-hydra (:foreign-keys run)
"Refile"
("a" josh/org-refile-hydra-file-a/body "File A" :exit t)
("b" josh/org-refile-hydra-file-b/body "File B" :exit t)
("c" josh/org-refile-hydra-file-c/body "File C" :exit t)
("q" nil "cancel"))

I find the easiest way to construct a nested hydra is to use the :after-exit keyword to pop back to the parent hydra.

We can see that refiling again is pretty straightforward.

Refiling with a nested hydra

Using macros to make hydras

One problem with writing hydras this way is that it gets ugly pretty quickly, with a lot of redundant code. Notice how we repeatedly have to specify the file in each file's hydra, even though it doesn't change within it. Also note that the hydra hint is the same as the headline.

We can write an Elisp macro, to do the heavy lifting in creating our hydras, so we only have to specify a file once:

1
2
3
4
5
6
7
8
(defmacro josh/make-org-refile-hydra (hydraname file keyandheadline)
"Make a hydra named HYDRANAME with refile targets to FILE.
KEYANDHEADLINE should be a list of cons cells of the form (\"key\" . \"headline\")"
`(defhydra ,hydraname (:color blue :after-exit (josh/org-refile-hydra/body))
,file
,@(cl-loop for kv in keyandheadline
collect (list (car kv) (list 'my/refile file (cdr kv)) (cdr kv)))
("q" nil "cancel")))

This makes defining our file hydras way simpler. Now our previous code simplifies to:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
(josh/make-org-refile-hydra josh/org-refile-hydra-file-a
"file-a.org"
(("1" . "Headline 1")
("2" . "Headline 2")))
(josh/make-org-refile-hydra josh/org-refile-hydra-file-b
"file-b.org"
(("1" . "One")
("2" . "Two")))
(josh/make-org-refile-hydra josh/org-refile-hydra-file-c
"file-c.org"
(("1" . "1")
("2" . "2")))
(defhydra josh/org-refile-hydra (:foreign-keys run)
"Refile"
("a" josh/org-refile-hydra-file-a/body "File A" :exit t)
("b" josh/org-refile-hydra-file-b/body "File B" :exit t)
("c" josh/org-refile-hydra-file-c/body "File C" :exit t)
("q" nil "cancel"))

Great! Now our code is a bit cleaner, and adding new refile targets is as easy as adding a ("Key" . "Headline") pair to a list.

Extra goodies

Using the previous code allows you to do most of your refiling tasks quickly and easily. I've made some modifications to the code to have the following features.

  1. Be able to jump (without refiling) to a headline by using a C-u prefix
    • And quit the hydra if we jump, so we don't accidentally refile things on the next key press.
  2. Be able to refile while you're using org-capture
  3. Be able to refile in org-agenda
  4. Deal with some bugs:
    • Don't refile when a headline doesn't exist
    • Use with-current-buffer so we don't accidentally switch to the buffer we're refiling to

All the code for this is available here, as a gist.

Quickly jump with our hydra

Note: Problems with current methods

Why did I use hydras to refile in the first place? There were a few problems I was having with org-refile. Some of these can be solved with some customizations that I wasn't aware of before.

  • Selecting refile targets can be slow

    (length (org-map-entries 1 nil 'agenda)) tells me I have 5208 entries in my agenda files at this time of writing. Even using a narrowing framework like helm, it takes a long time to narrow down to the headline I want. And generally there are specific categories I do most of my refiling to. (Note: You could use the :tag setting in org-refile-targets to limit refile targets to specific headlines. This might work well)

    You can limit the :maxlevel of org-refile-targets, but unless I set it to "1" this doesn't speed things up much since most of my entries are level 1 or 2. Also, I like the ability to refile to any location when I need to.

    One option is to set org-refile-cache, so you don't have to generate all refile targets every time you refile.

    You can also use org-refile-target-verify-function to limit locations you refile to, like "non-DONE tasks", "only projects", "headlines with children", etc. This can become slow as the function needs to be run on all possible candidates. (But in combination with org-refile-cache, this can be less painful.)

  • Refiling many headlines can be really slow

    If you follow Bernt Hansen's org-mode guide, you might have a "refile.org" capture bucket of things to refile later. For me, sequential headlines aren't necessarily related and can be refiled to many places. Picking places, again, takes several keystrokes, which is kind of annoying.

  • Why not just use org-capture to directly refile to headlines?

    Normally, when using org-capture, I actually do capture directly to the headline I want. A problem with this, though, is that I need to have capture templates not just for each headline I want to capture to, but for each style/template I'd like to capture. I'd like to have different templates for things like "events", "events mentioned in emails" (so org-capture can automatically add the email link), "emails I need to reply to" and so forth.

    You can refile from org-capture, but again this runs into my first problem.

  • Why not batch refile with org-agenda?

    I do this sometimes too. However, sometimes getting things I want to refile into the agenda in the first place is really cumbersome, and I'd like to navigate the tree structure of org mode (rather than flattening things with org-agenda).

I still use normal org-refile somewhat frequently, but refiling and jumping using hydras saves me a fair amount of keystrokes

ace-mc - Add multiple cursors using ace-jump

Happy leap day!

To celebrate this leap day, I just released my first Emacs package called ace-mc which allows you to quickly and easily add as well as remove multiple-cursors mode cursors using ace-jump-mode.

It's available on MELPA now! So installing it is as easy as M-x package-install RET ace-mc

Documentation is available on the GitHub page, but here are a couple screencasts:

Adding cursors with ace-mc Removing cursors with ace-mc

The main reason I made this package is because adding cursors with mc/mark-next-like-this or mc/mark-all-like-this-dwim doesn't work super well if you have a lot of potential matches. For example, if I'm trying to rename a variable "i", there's often a bunch of i's in other words that I don't want to touch. While multiple-cursors does make it possible to add multiple cursors using the mouse, this often is a bit of a hassle for me.

It seems like some people are already using it. In the time it's been out, one person already reported a bug (which is now fixed) and a couple people have already asked if I'll be adding avy support.

I eventually plan to add avy support. I first off used ace-jump-mode as it's the package I use currently. I understand, though, that many people have switched to avy. I've already messed around with adding a "add cursor" action to avy-dispatch-alist. The tricky, thing, though is how multiple-cursors deals with read-prompts. And since avy provides many types of jumping styles in separate commands, I'm not sure how best to add ace-mc support for all of them. But as I keep playing around with avy, I plan to finally switch to it.

Anyways, give ace-mc a try, and if you have any improvements or suggestions, let me know!

Getting IBus working with Emacs

Emacs comes with a lot of Chinese input methods like pinyin, four-corner method, and various forms of Cangjie among others (listed quite handily here). For basic usage, it actually does fairly well. I’ve been able to use the four corner method to look up characters of which I don’t know the pronunciation. However, Emacs’s 4corner and Cangjie methods are limited in that they only use traditional characters and can’t look up simplified characters. So if I tried to look up 龙 (“dragon”), which looks like 4corner “43040” to me, I wouldn’t be able to, since it’s a simplified character. I’d only be able to look up the traditional form of dragon: “龍” (which is “01211”). So I looked for other input methods that might support both traditional and simplified, one of which is Wubi. Wubi isn’t available for Emacs, but can be installed via IBus.

I installed IBus and tried it out. It’s input is pretty good, and better than Emacs’s pinyin in that it has phrase matching. So if I wanted to enter in “lǎoshī” (“teacher”, “老师”) in Emacs, it would get “lao -> 老” correct, but would guess that “shi” is “是”, since shì (是) is more common than shī (师). IBus’s pinyin is smart enough to recognize “laoshi” as “老师”, among other words and phrases.

IBus worked out of the box for applications like Chromium and even xterm, but for some reason it seemed to have no effect whatsoever in Emacs. I thought this had something to do with not having ibus-el installed, so I installed it via apt. Even with correct setup I still had problems. Nothing was showing up. When I tried ibus-toggle I got the error 'IBusELInputContext' object has no attribute 'enable'. It turns out that IBus 1.5 no longer works with ibus-el, and that ibus-el pretty much doesn’t work anymore (see this discussion). But some seemed to be able to get IBus working without ibus-el. Since Emacs has XIM support, it should be able to support it automatically. But whenever I entered text, only English characters appeared, without the IBus character selection dialog popup. I tried adding

export GTK_IM_MODULE=ibus
export XMODIFIERS=@im=ibus
export QT_IM_MODULE=ibus

to my ~/.zshrc (it turns out you probably don’t need to, as GTK_IM_MODULE and XMODIFIERS were already set to these values).

I found someone mention the workaround of using LC_CTYPE="zh_CN.UTF-8" emacs to start Emacs. It turns out that this somewhat works. I started to see the IBus character selection dialog popup, but I wasn’t able to enter any characters. I tracked the problem to Gtk-WARNING **: Locale not supported by C library, which suggested that I didn’t actually have “zh_CN.UTF-8” installed. So I installed it via sudo dpkg-reconfigure locales and selected the appropriate option. Now if I start emacs using

LC_CTYPE="zh_CN.UTF-8" emacs

it can accept input through Wubi, Pinyin, Cangjie5, and others. Cool!

There’s still some problems with using IBus on Emacs without ibus-el. It’s hard to do commands like C-x k (kill-buffer) without the k being read as something else. Usually you have to switch to temporary English mode using Shift, or switch back to the US keyboard. Maybe someday ibus-el will work with IBus again, but the API conflicts seem to suggest this won’t happen anytime soon.

Anyways, that’s it. Hopefully this helps if you with Emacs and IBus if you were tearing your hair out like I was. 再见!

Emacs is great for sysadmins, too

I work as a Unix Systems Administrator for UC Berkeley’s Rescomp and it occasionally comes up that sysadmins generally prefer vim while programmers prefer Emacs. The reasoning for this is that vim or vi is generally more available on servers and generally has a more consistent interface across servers. That is, if you use Emacs, you generally have a hefty .emacs file, and using an unconfigured Emacs is painful.

I think it’s no longer the case that Emacs isn’t installed by default. I’ve only ever had to use vim a handful of times, and the only thing I really needed to know was how to

  1. Insert text (i)
  2. Save & Exit (Esc : wq ENTER)

However, I’m a sysadmin that prefers Emacs, and there are a number of reasons why using Emacs is very helpful for sysadminning.

Dired

Dired mode is Emacs’s visual “directory editor”, and it makes navigating and operating on files much easier than just using the command line.

Using marks

One task that’s very easy in Dired that’s really cumbersome to do elsewhere is repeated grepping. Say, for example, that I want to find files with “hello” in them. In Dired I do this by pressing % g and entering the string.

A number of files displayed in Dired.

And what I get is a number of marked files (in orange), that I can easily, among other things:

  • copy (C)
  • move/rename (R) (even to another server with Tramp!)
  • change the mode of (M)
  • run a shell command on (!)

Highlighting files to perform actions on them.

Now I can filter out files that don’t match by pressing t k (which toggles, then kills lines).

Filtering out a file by "killing" lines.

Now say I forgot that I also need the files to contain “world” somewhere in them. I just repeat the process by pressing % g again and entering “world” to get a list of marked files that contain both “hello” and “world”.

Searching with dired highlights files.

And now it’s really easy to do any operations on them.

In bash, however, it feels a little more clumsy for me. It’s possible to search by doing:

grep -l "hello" .

But if I remember later that it also has to contain “world”, I have to go edit the last command to be:

grep -lr hello . | xargs grep -l world

And now I just get a list of files. Say now that I want to copy these files somewhere. I have to again tack on another command, like so:

grep -lr hello . | xargs grep -l world | xargs -n1 -i cp {} /some/directory

It gets really cumbersome, and it requires you to remember how to use substitute arguments like {} in xargs. And you might also have to hope your file names don’t contain whitespace. With Dired, you really don’t have to worry about these kinds of things. Dired’s marking system makes a bunch of operations super convenient.

Edit Dired

“Edit Dired” mode also just makes it so much easier to rename files in bulk. Instead of having to think of a regexp or sed expression to use for rename or whatever, I can just use C-x C-q, define a macro (or use query-replace) , and save the buffer. Dired automatically does all the renaming for you.

Make the Dired editable by pressing C-x C-q:

Dired allows direct editing of file names.

Create a macro to rename files (or use query-replace):

Typing new file names in Dired.

Apply to all files, then save the buffer:

Using a macro to rename files.

Dired X

Dired X is also very useful. You load it by putting

(require 'dired-x)

in your .emacs

One of the cool things it can do is automatically guess the shell command you want to perform on a file. Say that I’ve forgotten the command to extract a .tar.gz file. Well, Dired X will remember for me!

Dired-X shows useful command suggestions for a ".tar.gz" file.

As you can see, it correctly suggests tar zxvf. Quite handy, huh?

Tramp

Tramp mode, combined with Dired, also just makes it really easy to move files around. I can browse directories on a remote server and say to myself “I’d like to have that locally” and copy it very quickly to my computer, without having to type scp and enter in the entire path. Another situation where this is useful is copying a file between two servers that have a firewall between each other. And this has actually happened for me on several occasions. Normally what I have to do is something like:

scp server1:/path/to/file .
scp file server2:/path/to/file
rm file

But with Tramp mode I can just copy it, quickly changing the server name in /ssh:server1:/path/to/file to /ssh:server2:/path/to/file

Tramp also makes it really easy to view images and PDFs on remote servers that don’t have X11, since Emacs can display images and PDFs.

It’s even possible to remotely edit files as root using /sudo:server:/path/to/file, although this doesn’t work out of the box. You’ll need to add this to your .emacs

(add-to-list 'tramp-default-proxies-alist 
     '((and (string-match system-name 
                  (tramp-file-name-host (car target-alist)))
            "THISSHOULDNEVERMATCH")
       "\\`root\\'" "/ssh:%h:"))

This allows you to sudo into remote servers, but also prevents it from interfering with sudoing locally.

I can also use M-x ediff to compare two files on different servers, and selectively merge differences.


So these are just a few reasons why Emacs can come in handy for a sysadmin, or any normal user for that matter. Tramp in conjunction with Dired make it extremely easy to handle files on a number of servers.