Biff Barks: Polling Gmail with Gnus

“Biffing” usually refers to polling for new email and giving some sort of indication that new mail has arrived, i.e. our dog “Biff” will “bark” when the mailman comes. Over time, Emacs has had various code for this. So what’s new?

A lot, actually, when using IMAP and especially Gmail.

Emacs doesn’t have threads and many operations are blocking. If you’re using Gnus to fetch email via IMAP and you’re not using a local storage scheme, polling for email can take an annoying number of seconds, during which you can’t do anything else. Sure, you can keep the number of groups that you poll small, and that gives you some speed-up. Or you can ditch the idea of polling the source, and run something that polls separately and brings new mail into a local store.

I didn’t want to do the latter. I want to keep mail synced across something like seven devices, and local storage isn’t really an option. There is too much to try to coordinate. I’ve got to go right to the source, which for me is Gmail.

The simplest idea is to have a run-with-timer process, or maybe a run-with-idle-timer process, to poll mail every so often. If you do run-with-timer you’re back to the blocking problem, and the timer tends to go off at the most inconvenient moments. If you run-with-idle-timer, that’s nice, but you don’t get regular polling, and you can’t set the idle timer too short or blocking will again be an issue.

I found some really clever stuff online. I saw one that runs asynchronous processes with a process sentinel, but it relied on at least limited local storage.

So, naturally, I came up with my own idea. It’s not a complete solution, as you’ll see, but it’s doing what I want.

Flash update: New ideas at the end of the document.

This is what I want.

1. I want to poll for mail only if the network is up, and I want to do it at regular intervals.
2. I want biff to bark; that is, I want an audible warning if I have new unread email. I don’t want biff to bark if I have unread email about which I’ve been previously notified (biff has already barked).
3. I don’t need the email to be fetched; I just need to know it’s there and I can go fetch whenever I wish. (This is what may be viewed by some as lacking — you have to go fetch yourself at a time when you don’t care that it’s a blocking operation.)

Here’s all the code at once, with comments. You will have to change a few things, like putting in your own userid and password for Gmail, and defining the path and names of the .wav files used when biff barks. Finally, be sure the INBOX name is exactly the same as the one you use with gnus.

You need to have ’curl’ and ’sed’ installed, which will be the case for most Linux systems.

;; Reintroduce biff.

;; Biff will bark when an interval (timer-run) mail check comes up with
;; new inbox email. Biff does not bark on manual checks (with 'g').

;; This is a little tricky. We have to track the number of new emails
;; so that biff will only bark when that number increases. When we
;; exit the inbox after reading mail, we remember the number of unread
;; mails for the same reason.

;; Initialize.
(defvar rjn-gnus-inbox-saved-count 0)
;; Modify to suit.
(defvar rjn-gnus-inbox-name "nnimap+gmail:INBOX")

;; Function that counts remaining unread mail when leaving INBOX.
(defun rjn-gnus-leave-group ()
  "Function run when leaving gnus group"
  (if (string= rjn-gnus-inbox-name gnus-newsgroup-name)
      (progn
;	(message "leaving INBOX")
	(setq rjn-gnus-inbox-saved-count (gnus-group-unread rjn-gnus-inbox-name)))))
;; Set up the function to run in the group exit hook.
(add-hook 'gnus-exit-group-hook 'rjn-gnus-leave-group)

;; Array of biff bark wav file locations.

(defvar biffbark-wavs [
"/home/bnewell/data/elisp/dog_bark_x.wav"
"/home/bnewell/data/elisp/rooster.wav"
"/home/bnewell/data/elisp/elephant.wav"
"/home/bnewell/data/elisp/pig.wav"
"/home/bnewell/data/elisp/cow1.wav"
])

;; IMAP fetching hangs emacs while running.  The latest bright idea is
;; to combine an external program with biff-barks. We check and notify
;; if there was new mail, but we do /not/ do a fetch. The customer
;; does that at his leisure.

(defun gmailchk ()
 "Check single gmail inbox for new mail, return integer count of
  new mail"
 (interactive)
 ;; Get the gmail count this clever way found on the 'net, using
 ;; curl. Clean up with sed. BUT FIRST CHECK NETWORK as we don't want
 ;; to wait around for timeouts. The port test is very fast.
(if (outbound-port-test "imap.gmail.com" 993)
 (let* ((mail-count (string-to-int (shell-command-to-string
 "curl -u youruserid\@gmail.com:yourpassword --silent 'https://mail.google.com/mail/feed/atom' | sed -e 's/^[^[]*<fullcount>//' | sed -e 's/<\\/fullcount>.*//'"))))
   ;; See if the new mail count has increased.
   ;; If it has, biff barks and we issue a message. We also
   ;; save the new (higher) count.
       (if (> mail-count rjn-gnus-inbox-saved-count)
	   (progn
	     (setq rjn-gnus-inbox-saved-count mail-count)
   ;; Play random sound to announce new email
	     (play-sound-file (aref biffbark-wavs (random (length biffbark-wavs))))
             (message "New Gmail --- Biff says, go fetch")))
       mail-count)
    0)
)

;; Let's check every 15 minutes.
(run-with-timer 900 900 'gmailchk)

This may seem complex, but it really isn’t, even though there are a number of pieces. The clever curl/sed idea, modified from something I found on the web, is blocking, but very fast, and gives you the unread email count. This is compared with a stored value to see if new things have arrived. When you read your mail with Gnus, and leave the INBOX, the unread mail count is updated (it will be 0 if you’ve read everything in the INBOX). When curl/sed find new mail, biff barks and you get a message. You go fetch whenever you wish.

Why the network check? Because if you’re offline, you don’t want to wait around for curl to time out. The network check is very fast. Here’s the code for it. You must have ’nmap’ installed.

(defun outbound-port-test (host port)
"Test if outgoing TCP port is open"

;; nmap must be installed 
;; returns t if port ok, nil if not
;; "--system-dns" and "tcp open" help avoid false positives, but now
;; this is good for tcp only.

(if (string-match "tcp open" 
       (shell-command-to-string 
         (concat "nmap --system-dns -PN -p" 
             (number-to-string port) " " host)))
    t
    nil)
)

And that’s it. My Gmail inbox is checked every 15 minutes and I’m notified if there’s something that’s truly new. The process is fast with very minimal blocking. No offline IMAP is necessary.

If you want to check mailboxes other than INBOX, that can be done, but you’ll have to make some modifications. I haven’t delved into this as I have no need for it at this time.

Well, all of the above does what I want, but then I added additional IMAP mail sources[1], and in some flaky internet environments I still ran into blocking issues.

I found a way to deal with all of that but of course it makes everything much more complex. I’ll publish the code eventually, when I get into a form that someone other than myself can make sense of (I’m not even sure about myself).

I wanted zero blocking operations that have to do with the internet, at least for those things which run periodically. (I’ll accept blocking when I run something myself that I am aware will block; I just don’t want blocking to occur when I’m in the middle of typing something, for instance.)

To get this, I developed a bash script that I launch asyncronously from my Emacs startup. (I add a hook to kill the script when Emacs shuts down.) The script does all the network checks and stay-alive stuff; it also does the biffing. I can check as many sources as I want without blocking because the script runs apart from Emacs. Biff barks from the script, not Emacs (barking blocked as well, and that got annoying, the blocking, that is).

Now all is not so simple. To do correct biffing and barking (remember the stuff above that tracks emails that have been barked about but not opened?), the script has to communicate with Emacs. I do this through a drop file. Code in my elisp routines updates the drop file after I read some previously unread mail.

The drop file has a nice advantage. It can be polled and the number of unread emails shown in the mode line.

I also don’t want multiple inboxes, and I don’t like virtual groups. So I have some elisp code that moves mail from the “extra” inboxes into my one desired inbox (gmail in my case, at least for now). This turned out to be not so simple for multiple IMAP servers. Moving mail from one IMAP server to another takes some effort.

But it’s all done, and working. I have zero asynchronous blocking, all my mail from multiple sources goes to my inbox of choice, biff barks if and only if he’s supposed to, and the mode line tells me how many unread emails are in that inbox.

This took hours to get right, which included tracing some anomalies in Gnus. Was it worth it? Uh, well. That’s a definite “maybe.” I had many higher priorities let-me-tell-ya, but this was the holiday season and I really didn’t want to do very much “real” work.

[1] I got myself an AOL account. No, I am not joking. Sometimes you want to send email from an account that says to the world “I am not computer literate.” There are sometime advantages to this; think about it. And the irony of sending AOL email from Gnus is absolutely priceless.