To add a todo you just use a simple terminal command:

$ todo example blog todo

To complete a todo:

$ todone 3

It was pretty easy to port the system from Mac to Linux (I did it on Ubuntu Unity, but it should work for pretty much any Linux distro with a few minor changes) by following the instructions in the post and follow-up comments, but I’ll repeat them here to save others effort. Firstly, create the following shell scripts:


$ cat /usr/local/bin/todo
#!/bin/bash
if [ $# == "0" ]; then cat $TODO; else n=$(($(tail -1 $TODO | cut -d ' ' -f 1) +1)); echo "$n ⇾ $@" >> $TODO; fi

$ cat /usr/local/bin/todone
#!/bin/bash
sed -i -e "/^$*/d" $TODO;


UPDATE: Improvements to these scripts can be found in the comments. Or, even better, try Todo .txt, which has a similarly minimalist interface but some more advanced features. There is also a page on setting it up with conky, similar to what is done below. (I found that I had to rename todo.cfg to config to get Todo .txt to work correctly though). Thanks to Fred for suggesting Todo .txt.

Remember to set them to be executable (chmod +x /usr/local/bin/todo). You then need to set a TODO environment variable to wherever you want to keep your TODO file (i.e. add something like export TODO=~/todo.txt to your .profile). If you want to sync your TODOs over several machines, you could point it to a dropbox folder or similar.

For the next step, you need to install conky, a utility for displaying info (more normally system monitor stuff) on the desktop (you should be able to grab this from your package manager). To get the same set-up as me, use the .conkyrc in this gist – I had to play around with the basic config to get it to display properly in Unity, you may face different problems in other window managers.

Finally, you should set conky to run on startup – in Ubuntu just open Startup Applications and add it to the list.

I can’t take credit for any of this stuff, all I did was put together a bunch of hints from the internet.

For example:

Document testDoc = createDocument(
"<a>aa<b attr='test'>b<!-- comment -->c<c/></b>d</a>");

System.out.println(XPathGen.getXPath(aa));
//Should print "/node()[1]/node()[1]"


I pulled this code out of diffxml and re-licensed it under the Apache licence, so that it can be used more freely.

(This was mainly prompted by this StackOverflow question).

Grab it from github if you’re interested.

A while ago I wrote a post berating the lack of full-blown IDEs available for the various Lisp languages (cue the sound of Emacs fanboys sharpening their knives in the background ;) ). To some degree I regret that post, as although its popularity proved it hit something of nerve, it was basically a whine that lacked helpful or insightful comment.

Whilst my views haven’t substantially changed, in this post I thought I’d try to put things right by compiling some advice to people who want to edit Lisp for the first time.

In the beginning was the REPL

To begin with forget about the editor or IDE. Just use whatever simple text editor you are used to in conjunction with the REPL, whether this is Notepad, gedit, nano or Vim (or, yes, Emacs). Don’t even worry about syntax highlighting etc if you are just trying out a few simple examples. Most Lisps come with a pretty decent REPL, especially CLISP and to a lesser degree SBCL and Clojure¹.

Get used to typing in simple functions and testing things out directly in the REPL. Once you start wanting to save your work, just learn your Lisp’s command to load files (e.g. (load 'filename.lsp') in SBCL or (load-file 'filename.clj') in Clojure) and use this to in-directly interface with your text editor. This should actually provide you with quite a friendly environment, where it is quick and easy to build your code up and try out changes in the REPL.

Next Steps

The advantages a more integrated Lisp editor will bring you are things like syntax highlighting, code formatting and an integrated REPL with the ability to evaluate single expressions at a time rather than whole files. More advanced environments (notably Emacs) will include debuggers and other tools.

Unfortunately, there a few options to pick from here and they are not all equal. If you already know (and like?) Emacs, your decision is easy and you just need to look at some documentation on getting slime and paredit set-up. Emacs is probably the most common and perhaps the most powerful setup. The disadvantage is needing to learn Emacs, although there are some packages designed to make this a little easier, such as lispbox and aquamacs (Mac only).

If you know Vi or Vim, the slimv plug-in has progressed very nicely and provides most of the functionality of its Emacs cousin, although I find the modal editing of Vi can make using the REPL a little awkward.

If you’re used to IDEs such as Visual Studio or Eclipse and would rather not leave their cosy warmth, you are in for a rougher ride. The best I have found so far is counterclockwise, a Clojure plug-in for Eclipse. There are Eclipse plug-ins for other Lisps, but all the ones I found were abandoned and/or buggy (this has possibly since changed but my experience was bad enough that I don’t want to revisit it). There is also LispWorks, which looks good, but note that there are significant restrictions on the free (as in beer) version (mainly that you can only use for up to 5 hours a day per session). In most cases it seems you are best advised to bite the bullet and learn Emacs.

Racket

A final honourable mention needs to go to Racket, which includes its own IDE, DrRacket. This seems to be a really nice environment for learning in. If you just want to play around with functional programming, this might be exactly what you are looking for — it comes with a great tutorial which gets the user playing with graphics primitives directly in the REPL (circle, rectangle etc). However, if you want to use a different type of Lisp, you will need to consider the above choices.

I believe that this covers the main choices, although there were some other suggestions in the comments to the previous blog. Again, the main take-away should be not to worry about IDEs at first. Just get started with the REPL and a text editor and put off the IDE question till later.

1. If you’re getting started with Clojure, I recommend using leiningen to install Clojure and launch the REPL, but at least make sure you install JLine or rlwrap to improve the basic REPL. There also seems to be some work on a more advanced Clojure REPL called nREPL.

TL;DR: Clojure is very close to Common Lisp but has several minor syntactical differences, some bigger differences in macros, support for more intrinsic types, Java integration and, perhaps most significantly, a programming model that encourages a more functional style of development and is designed to support concurrency.

“Evolution” attempts to simulate a very simple world where animals roam in directions dictated by their genes, eating food and reproducing asexually. Reproductions cause random changes to genes and hence the movement of animals. The center of the game world has more food resources than the rest, which means we can expect different types of animal to evolve in the middle than the rest of the world. It’s apparently based on article by A.K. Dewdney: “Simulated evolution: wherein bugs learn to hunt bacteria” (searching for this will find you some interesting resources). Full source code for the Common Lisp version is in the book as well as on the Land of Lisp website, the Clojure version is available on github.

UPDATE: the above github link is to the latest version of the code, the first version is available here. Both the post and code have been updated to reflect changes suggested in comments.

We start off by defining some parameters for the game world, which is almost identical in both versions. For example, defining the *width* parameter in Common Lisp:

(defparameter *width* 100)

And in Clojure:

(def *width* 100)

However, we almost immediately stumble upon a more important difference when we try to create a hash-set to store the location of plants in the game-world. In Common Lisp, hash-sets are faked using hash-tables where the value stored is unimportant (we are only interested if the key exists in the table, not what its value is). The Evolution example uses the following code to add plants to the game world at a random location:

(defparameter *plants* (make-hash-table :test #'equal))

(defun random-plant (left top width height)
   (let ( (pos (cons (+ left (random width))
                     (+ top (random height)))))
        (setf (gethash pos *plants*) t)))

The position of the plant in the game world is the key and true (t in Common Lisp) is used as the value. Note the use of the equal method for testing equality rather than eq (so that the contents of the con cells are compared rather than testing if they are the same cons cell). In Clojure we can achieve something similar like this:

(def *plants* (ref #{}))

(defn random-plant [left top width height]
  (let [pos (list (+ left (rand-int width))
                  (+ top (rand-int height)))]
    (dosync (alter *plants* conj pos)))))

In Clojure, if you want to modify something, you instead point to it using a ref which can be changed to point at a different “thing” (refs are the one of the few things in Clojure that are allowed to mutate). All modification of refs must be synchronised in “transactions” such as the dosync method. This marks a fundamental difference between the languages; Clojure programmers are forced to think up-front about parallelism and are almost ushered down a more functional path. The #{} is just shorthand for an empty hash-set (Clojure has a intrinsic support for more data structures than just lists). Also, in this case, we don’t need to worry about the equals method in Clojure — the default does what we need. Other minor differences are the use of square brackets for function arguments (which helps make things more readable), random becoming rand-int and using list to create a new list rather cons (which appears trivial, but points towards a slightly bigger difference in how lists are constructed in Common Lisp and Clojure).

Creating the list of animals and populating with the first animal also has a few differences. In Common Lisp:

(defstruct animal x y energy dir genes)

(defparameter *animals*
    (list (make-animal :x      (ash *width*  -1)
                       :y      (ash *height* -1)
                       :energy 1000
                       :dir    0
                       :genes  (loop repeat 8
                                     collecting (1+ (random 10))))))

And in my Clojure version:

(defstruct animal :x :y :energy :dir :genes)

(def *animals*
  (list (struct animal
          (round (/ *width* 2)) ; x
          (round (/ *height* 2)) ; y
          1000 ; energy
          0 ; dir
          (loop [genes [] i 8]
            (if (zero? i)
              genes
              (recur (conj genes (+ 1 (rand-int 10))) (dec i)))))))

Note that Conrad uses the ash function to do division. Personally, I would only use ash when I was doing bitwise operations or if it provided a large performance increase on a time-sensitive section of code, as I feel it obscures the meaning of the code slightly. Clojure does have the bit-shift-left and bit-shift-right operations to provide similar functionality but I decided to use the standard division operator instead. (Conrad later told me that he used ash to avoid using round which would have required him to explain about returning multiple values from a single Lisp function).

More interestingly, the loop construct is considerably different. The loop macro is a lot simpler in Clojure (it’s little more than a recur binding), so we need a little more coding to achieve the same effect as Common Lisp (in particular, Clojure has no collecting keyword). However, comments on this post showed me that this could be better written as:

(def *animals*
  (list {:x (round (/ *width* 2))
         :y (round (/ *height* 2))
         :energy 1000
         :dir 0
         :genes (vec (repeatedly 8 #(inc (rand-int 10))))}))

Which gets rid of the unnecessary use of structs (plain maps are easier here) and generates the random genes in a cleaner, more concise way.

At the next point, my Clojure code takes a significant departure from the Common Lisp. The Common Lisp function for moving animals in the game world looks like:

(defun move (animal)
  (let ( (dir (animal-dir animal))
        (x (animal-x animal))
        (y (animal-y animal)))
    (setf (animal-x animal)
        (mod (+ x
                    (cond ( (and (>= dir 2) (< dir 5)) 1)
                           ( (or (= dir 1) (= dir 5)) 0)
                          (t -1))
                     *width*)
                     *width*))
    (setf (animal-y animal)
        (mod (+ y
                    (cond ( (and (>= dir 0) (< dir 3)) -1)
                           ( (and (>= dir 4) (< dir 7)) 1)
                           (t 0))
                    *height*)
                    *height*))
    (decf (animal-energy animal))))

And my Clojure looks like:

(defn move [{:keys [x y dir] :as animal}]
    {:x (mod (+ x
             (cond
               (and (>= dir 2) (< dir 5)) 1
               (or (= dir 1) (= dir 5)) 0
               :else -1)) ; note clojure has else form and less ()s
        *width*)
      :y (mod (+ y
             (cond (and (>= dir 0) (< dir 3)) -1
               (and (>= dir 4) (< dir 7)) 1
               :else 0))
        *height*)
      :energy (dec (:energy animal))
      :dir (:dir animal)
      :genes (:genes animal)}) ; return new animal rather than change given

Movement is calculated by using an animals “dir” which is integer representing a direction (0 is up-left, 1 is up, 2 is up-right etc).

The fundamental difference between the versions is that I decided not to directly change the animal in the function argument, but instead to return a new animal representing the animals location. This meant I could avoid worrying about refs and also let me play with programming in a more pure functional manner. Some other minor points:

• Clojure can use destructuring to bind symbols to parts of parameter lists in functions or let bindings. This allows us to use x, y and dir directly rather than have to extract it from the map each time.
• When we do want to pull bits out of structs/maps, it works a little differently; in Clojure you can use the key directly as a method (e.g. (:energy orig) whereas Common Lisp generates methods like animal-energy
• The Clojure version of the cond macro has less brackets than the Common Lisp and an else keyword.
• I removed the unneeded addition of *width* and *height* before calling mod (I think this intended to avoided negative results from mod, but both the Clojure and Common Lisp version of mod return the sign of the divisor not the dividend i.e. (mod -1 4) is 3. (This was pointed out to me by a commenter on reddit).

The eat method, which simulates an animal eating any plants in its current location, also contains a few interesting differences. In Common Lisp we have:

(defun eat (animal)
  (let ( (pos (cons (animal-x animal) (animal-y animal))))
    (when (gethash pos *plants*)
      (incf (animal-energy animal) *plant-energy*)
      (remhash pos *plants*))))

Whereas my Clojure version looks like:

(defn eat [{:keys [x y energy] :as animal}]
  (let [pos (list x y)]
    (dosync
      (if (contains? @*plants* pos)
        (do
          ; get rid of the plant first
          (alter *plants* disj pos)
          ; so we can return the updated animal
          (assoc animal :energy (+ energy *plant-energy*)))
        animal))))

(Or at least it does after I fixed the concurrency error pointed out by cgrand).

The code for checking for existence of a plant at the animal’s location is fairly similar, but note that Clojure needs to dereference the hash-map using @ and uses the contains? predicate rather than getting the value in the hash-table. In Clojure I needed to remove the plant first (which is a side-effect) before returning the updated animal. Removing the plant in Clojure is a little more complicated — rather than just calling remhash, I need to set the *plants* reference to a new hash-map which is the “disjoint” of the old hash-map and the position of the eaten plant. Whilst the Common Lisp version can just increment the energy of the animal (another side-effect), the Clojure version creates a new animal which is the same as the old animal, except with its energy increased, thorugh use of the assoc function.

The reproduce function (which I haven’t reproduced for brevity) was modified in a similar way, most notably to return a list of all animals including the new animal rather than inserting directly into animal list (which isn’t mutable in my version). The update-world function was also modified to be more functional, taking an animal list as input and returning an updated list. The Clojure version uses # as shorthand for an anonymous function rather than lambda. flatten is needed so that we keep all the animals in a single list rather have lists of lists.

The draw-world method (also not reproduced) doesn’t contain any interesting new differences, but was the source of a bug for a long time — turns out if you do print "\n" instead of println in Clojure, the output buffer won’t be flushed properly. Also worth noting is that Clojure uses first and rest instead of car and cdr, as you can see if you look at the code for the turn function.

The final function we’ll look at is the one that holds it all together and runs the simulation in response to user input. The idea is that the user can enter a number of time-steps to process or just hit enter to process a single step. The original Common Lisp code looks like this:

(defun evolution ()
  (draw-world)
  (fresh-line)
  (let ( (str (read-line)))
    (cond ( (equal str "quit") ())
          (t (let ( (x (parse-integer str :junk-allowed t)))
               (if x
                   (loop for i
                      below x
                      do (update-world)
                      if (zerop (mod i 1000))
                      do (princ #\.))
                   (update-world))
               (evolution))))))

And my, not quite equivalent, Clojure code looks like this:

(defn parseInput [x]
  (try (Integer/parseInt x) (catch Exception e 0)))

(defn evolution [animals]
  (do
    (draw-world animals @*plants*)
    (println) ; don't do 'print "\n"' or 'newline' as won't flush
    (let [line (read-line)] ;str is a function in clojure
      (cond (= line "quit") ()
        (= line "p") (do (println animals) (evolution animals))
        :else (evolution
                (let [x (parseInput line)]
                  (nth (iterate update-world animals) (inc x))))))))

Again, there are variations caused by differences in the loop macro (here we use iterate instead) and my decision to keep the animals list immutable. Also, I didn’t implement outputting a “.” every 1000 steps. However, the most interesting change is in the way input is parsed. Clojure doesn’t have the parse-integer function, but I was quickly able to implement something that did the job by calling out to the Java parseInt method. If the parseInt method throws an exception, we just return 0 instead which will run the simulation for 1 time step. It’s worth noting how concise this bit of code is (how many lines would the equivalent Java be?).

Out of all of these differences, the most important ones are the Java integration and the support for concurrency, with intrinsic support for more types of collections somewhere slightly behind. The Java integration wasn’t forefront in this article, but I was able to use it to quickly create a parse method that did what I needed. Having a large, tested and useful library is one of the most important requirements for a general purpose language if it is to gain uptake with the average developer. The concurrency support also didn’t really come out in this article (although you were introduced to dosync and refs), but it seems to offer a relatively simple path to scalability; I hope to play more with Clojure’s concurrency model and see if this really is the case and see what sort of speed-up is possible at what cost to simplicity (possibly I’ll try to write an agent based version of the simulation à la Rich Hickey’s ants demo).

Not being an Emacs user, I tried a few different possibilities. I tried to get the impressive sounding slimv Vim plug-in working, but had no joy. I think the issue was that I was using Windows and needed to install some more python (yes, python) libs, but the lack of error messages made me wonder if I was wasting my time. This plug-in also won’t appeal to anyone who doesn’t want to invest in learning vi.

After giving up on slimv, I tried the CUSP plug-in for Eclipse. This plug-in pleasantly surprised me, given that the web pages for it don’t inspire confidence. It worked straight away on my Windows box, providing a decent REPL/editor integration. However, it does have a few issues:

There is another Eclipse plug-in called Dandelion which claims to work with both CLISP and SBCL, but I never managed to get it to connect to the REPL and had problems with the text-editor hanging.

A few more options are hinted at by this stackoverflow question, but none of them look very promising, with the exception of Emacs.

This leaves newcomers to Lisp with the following choices:

Which isn’t the greatest set of choices. Why is this the case for a programming language with such a long and illustrious history?

If you’re considering learning Lisp, this may well present a considerable dilemma to you. If you don’t want to learn Emacs (and why should you have to?), I suggest trying CUSP. I also just found the Lispdev fork of CUSP, which I’m hoping may solve some of the issues with CUSP.

I think I saw the programming equivalent today – proving that a GUID is not unique.

The poor programmer does get a bit much stick, but he has just tried to write a loop that will take 10790283070806014188970 years to compute. He then asks if using multiple threads will help solve his problem…

• Why you might want to consider learning/using Clojure is concisely covered by Rich Hickey here.
• One of the first decisions you will face is which editor to use. Unfortunately, this isn’t necessarily an easy decision. Clojure and Lisp programmers typically use a REPL to interact with their programs as they are writing them. You want your editor to have some level of integration with a Clojure REPL. This Getting Started resource covers most of the common choices. It’s probably worth saying that Emacs offers one of the most mature and powerful environments and is a good choice if you are already comfortable with the editor. If you’re on Windows, Clojure Box offers a quick way to get started with Clojure and Emacs. Personally, I’ve tried  the Eclipse plug-in counter-clockwise, but had some problems with it and I currently use Vim set-up according to this guide at write-quit. (I see counter-clockwise has had a new release since I tested it, so it may well be worth revisiting).
• Once you have your environment set up, you will probably want some sort of quick start guide. One of the most commonly referenced tutorials is R. Mark Volkmann’s at ociweb. This is certainly a very valuable and authoritative resource, but I found it rather hard-going for a first contact. The Intro to Clojure video series was highly rated by this stackoverflow question, but I can’t say I want to use videos to learn a language. Satish Talim suggested Clojure Notes, which looks like a good introduction and may be the best place to start.
• Regarding books, Stuart Halloway has ported some of the Practical Common Lisp book (original available free here) to Clojure. This leads me onto his Programming Clojure book, which is the best resource I have found so far (but not free). There is also a wikibook available, but it seems incomplete.
• Planet Clojure is your best reference for new and interesting Clojure posts (they describe themselves as a “meta-blog” of various Clojure hackers and contributors).

Hope this helps someone!

In case you don’t know, Premiere Pro is “professional” video editing software, in a similar category to Avid Xpress and Apple’s Final Cut Pro. And to be fair, Premiere certainly seems to be fully featured. It’s just that it doesn’t work. It randomly crashes. Constantly. And when it’s not randomly crashing, it’s repeatedly crashing at the same point.

The first thing that irritated me is that I’m running it on a split new laptop with the maximum amount of memory (which is admittedly constrained to < 4 GB as I’m using 32-bit Windows XP) and yet it gives me messages such as “Adobe is running low on system memory. Please save your project and proceed with caution.”. Quickly you realise that it’s best to give Adobe full run of the machine and close any other programs which use memory (like Web browsers).

The next thing that perplexed me was that every time I close Premiere, I get a message telling me that Premiere unexpectedly quit and offering to send Adobe the crash report. No, Adobe, the quit wasn’t unexpected — I normally exited the application — but apparently you shouldn’t do that in Adobe land. This started to bug me, so I filled in the crash report and attempted to send it. Except I can’t, because my e-mail address isn’t valid according to Adobe¹.

Quite often, Premiere doesn’t want to import certain media. Sometimes it tells you this, other times it imports the footage but only displays a blank screen or inverted colours. This is irritating, but can normally be worked around by using free software such as ffmpeg or Virtual Dub to re-encode the footage. (Yes, these free programs seem to handle various video formats better than Adobe’s £750 payware. Adobe even suggest using VirtualDub themselves (see Solution 6)).

However, these issues pale into insignificance once you face the beast that goes by the name of “Export Media”. This launches what is sort of an associated application known as “Adobe Media Encoder”, which renders your project into your final move file (e.g. .mov, .avi or .mpg). Sometimes this works flawlessly.  Sometimes it will stop at a seemingly random point in the encoding and the logfile will contain such revelations as:

...
 - Encoding Time: 00:01:07

2009-10-28 14:18:47 : Encoding Failed

-----------------------------------------

Error compiling movie.

 Unknown error.

-----------------------------------------

Hmm, “Unknown Error”. Doesn’t really help, does it? Restarting the export will result in it stopping at exactly the same point. This forces you to come up with random hypotheses as to what caused it crash, testing them in order until you either fix the problem or give up. Sometimes re-encoding the original video footage can fix things, which would suggest the video is corrupted (although it will play fine in Premiere and other video software).

I had thought “Unknown Error” was the least useful error message I’d seen. I was wrong. Sometimes the encoding fails part-way, yet Media Encoder reports “Encoding completed successfully”. I don’t want to take personal offence from a bit of software, but it stopped with the progress bar halfway, created half a video file then declared it was happily done. Surely that’s simply rude?

Of course, you can attempt to find and follow the official Adobe advice. The first “note” here tells you to restart your computer “to refresh its memory” (read: not only is our software full of bugs, its also full of memory leaks). Solution 1 is to turn various effects off (read: lots of our effects have bugs). Solution 2 is to change the encoding (read: our codecs have bugs). The third solution annoys me even more: “Quit any applications that may be interfering with the render process”. Why? Surely Adobe should be able to share memory and resources with other processes? Solution 4 is “Use a different codec for your export” (read: our export function is full of bugs). Solution 5 is “Disable Auto Save” (read: our basic functionality breaks our other functions). I tried all of these; no dice.

In the interest of of ending on a positive, helpful note, I have the following suggestions for how Adobe can make things better:

• Replace “Unkown Error” wherever it occurs in the codebase. Handle exceptions properly – you’re meant to be a professional company and this is an very sloppy coding practice.
• Test the software. I’m using a split new machine from a vary large vendor. Why on earth doesn’t it work? There must be huge flaws in the testing process at Adobe.
• Remove support for features that don’t work. A particular encoding doesn’t work a given percentage of the time? Either fix it, or don’t pretend to support it.
• Look at your own advice. Most of these “solutions” indicate deficiencies in the underlying software.
• Provide some sort of cuddly toy with each purchase of Premiere Pro. Although this won’t make the software any better, it will reduce stress by giving users a representation of Adobe that can be tortured during periods of stress brought on by attempting to use the software.

1. I think this must be because my work address contains too many dots for Adobe. I’ve redacted my address as this post does not represent the views of my employer, just me. But trust me, it’s valid.

UPDATE 4/11/2009: This article has now been syndicated (if that’s the right word!) on HPCWire.