Archive for the ‘Uncategorized’ Category

Using Docker on Early Release

Monday, April 27th, 2015

I can officially announce that my book Using Docker: Developing and Deploying Software with Containers is available on early release from O’Reilly.

using_docker

The book is split into three parts:

  1. Background and Basics. The book starts off by looking at what Docker is, why it’s so important and how to get started using it, including a tutorial chapter designed to bring all readers up-to-speed with the basics of Docker. The last chapter of part 1 describes how Docker works in more detail and the underpinning concepts you need to understand in order to use Docker effectively.

  2. The Software Lifecycle with Docker. The central part of the book walks through using containers from development, through testing and into production. A small web app is developed throughout these chapters, serving as an illustrative example and discussion point. Topics discussed cover both development issues (how to write and test code with containers) and operations issues (how to run containers in production, as well as monitoring and logging).

  3. Tools and Techniques. The final section goes into advanced details and the tools and techniques needed to run clusters of Docker containers safely and reliably. If you are already using Docker and need to understand how to scale-up or solve networking and security issues, this is for you.

The early release currently has rough, unedited versions of all of Part 1 and Part 2 up to (but not including) monitoring and logging. Part 3 is currently in the works. Be warned that the book is still in a very early stage. The disadvantage of this is that parts are missing and the prose still needs work, but the advantage is that you can — if willing! — provide feedback and help shape the book.

The cover of the book is a bowhead whale (possibly the world’s longest lived mammal). Some people have complained that she or he looks a little "unwell". This is probably quite accurate; O’Reilly bases its covers on historical engravings, so it’s likely the original artist based the image on a whale that had been beached or washed up. Personally, I like my whale, though I haven’t managed to come up with a name yet. If you have a good suggestion, let me know — there’s a free copy of the book in it if I decide to use it!

This is proving to be a difficult book to write, not least because new versions of Docker and supporting tools mean I constantly have to re-write and re-evaluate content. I really appreciate all the feedback and support I get (positive and negative!), and a particular shout-out needs to go to Container Solutions who let me rant at them and publish my ramblings and CloudSoft who generously let me use their office in Edinburgh when I need to.

If you feel brave enough to take a look at the early release, you should be able to find a discount code on my twitter page. If you already have a copy, please consider sending me feedback or joining the google group discussion; the only way I can write the book you’re looking for is if you tell me what you’re looking for!

P.S. There have been some complaints about the cost of international shipping of the dead-tree version. I’m reliably informed that this will come down once the book is out and should be around 8 euros. Of course, you can also get it from Amazon.

Docker Posts at Container Solutions

Friday, February 6th, 2015

Recently I’ve been doing a lot writing on Docker over on the Container Solutions blog.

Here’s a collection of my articles for Container Solutions to date (last updated 27 April):

Intro to Docker Talk

Monday, May 26th, 2014

As part of my work with UglyDuckling, I gave this Introduction to Docker talk at the Edinburgh techmeetup.
(more…)

Book Review: Clojure for Domain-specific Languages

Sunday, January 26th, 2014

I’m clinging to the excuse that I had only read the title of the book before volunteering to review it. If I had even got as far the sub-title, I would have thought twice.
(more…)

(How To Do) XML Schema Validation

Thursday, November 21st, 2013

Judging by the popularity of this question on StackOverflow (and my answer), it seems that a lot of people struggle to check the validity of an XML file against an XML Schema. It’s a shame that what should be a trivial task has wasted hours of developer’s lives. In this article I’ll try to offer a few alternatives for various platforms and hopefully make things a bit simpler.
(more…)

Network Audio Players and Linux Revisited

Thursday, January 24th, 2013

In my previous post Network Audio Players and Linux, I described some ways in which to stream music from a Linux server to network audio player (primarily the Denon DNP-720AE). In the comments to that post, mjheagle suggested I try out MPD, which I’d missed when trying out various solutions. This post describes the benefits of the MPD solution (which is by far the best one I’ve tried so far) and how to get it going for yourself.

(more…)

Clojure and Google App Engine

Thursday, August 16th, 2012

I recently finished an MVP version of Edinburgh Festival Roulette, the first web app I’ve written using Google App Engine and Clojure, and I thought I’d share some of my thoughts and experiences.

Clojure and Compojure were definitely a good choice; I was able to get something up and running quickly despite having limited experience with both. For HTML I used Hiccup, but in retrospect Enlive would probably would have been a better fit (although I guess the two aren’t mutually exclusive).

I’m a bit more conflicted about GAE. It definitely seems responsive enough and I certainly can’t complain about the cost. However, there are a few gotchas that Clojure developers should be aware of:

  • Leiningen with the Ring plug-in will produce an “uber war” for you that you can deploy straight into a Java servlet container such as Tomcat. But GAE isn’t quite as straightforward. To deploy to GAE you need to use their command line tool which expects a directory rather than a war, and needs one or more special files depending on your needs (everyone needs appengine-web.xml and I use a cron.xml as well). To get around this I wrote a simple shell script which unzips the directory and copies the files across.
  • GAE manages all the scaling, which sounds like a good thing – and in many ways is – but means you lose control of things you might not have expected. You need to understand how GAE will respond to latency in your app and under which conditions it will spin up a new instance.
  • It’s extremely important to be aware of which libraries and APIs you can use. There is a whitelist which contains the list of JRE classes the app will have access to. Whilst you can add your own libraries, you need to know if they are dependent on a JRE class not in the list. This hit me, as I used clj-http which doesn’t obey this rule (or rather the Apache lib it wraps doesn’t). I ended up rolling my own as a wrapper around java.net.HttpURLConnection. I don’t think you could even use the deprecated http.agent library as although it uses HttpURLConnection it falls foul of the next point…
  • You can’t do your own threading. This is a major issue. It is understandable, as Google want to have control over any background work, but it creates serious problems when working with Clojure. In my case it meant that I had to drop my idiomatic (well, reasonably idiomatic) Clojure code that used agents to update a cache, as agents are of course entirely dependent on threads. One of the aims of Clojure is to make concurrent programming as painless as possible, but GAE takes a lot of this of control away from you. Worse, you can’t use any libraries (such as http.agent) that use agents, futures, or other thread based constructs. But how do you know if a library uses threading or a non-whitelisted class? I’m not convinced GAE can be considered a safe platform for Clojure when it is so hard to tell if your code will run, or if an update to a third party library will break everything.

If you still want to use Clojure and GAE, be aware that are some nice looking libs for working with GAE and Clojure e.g. appengine-magic. I haven’t tried any of them though.

Finally, please have a look at the Edinburgh Festival Roulette site and tweet if you like it. It might not make much sense to you unless you’ve been to Edinburgh during festival time though!

Network Audio Players and Linux

Wednesday, May 30th, 2012

This is a short guide to using network audio players (sometimes called media streamers) with Linux. It mainly pertains to the Denon DNP-720AE, as this is the network audio player I use. However, the techniques and issues are likely to be similar for most network audio players.

There are two things you probably want to be able to do; browse music stored on your Linux box from the audio player and control what music is played from your Linux box.

Browsing Music

In order to browse music stored on a PC from the player, what you need to do is set-up UPnP (often called DLNA, but as far as I can tell this is the organisation behind UPnP, not the protocol itself). There are a few different software options for this, you should be able to find most of them in your distro’s repository:

XBMCXBMC logo

Perhaps the biggest and most developed software is XBMC, which provides a complete media environment. This is probably an excellent option if you want to set up a dedicated box for handling your media, but leave it alone otherwise – it doesn’t work very well as a desktop app and seemed to be fairly resource intensive. Also I didn’t quite get it working with the Denon; I never managed to get past a “Not Authorised” problem.

GMediaServer

If you’re happy with (or looking for) a simple console app that runs in the background, try GMediaServer. It needs almost no configuration – just point it at your media directory and go (e.g. gmediaserver /media/audio/mp3/). The major drawback of GMediaServer is that the only index it provides is directory based – you are stuck with whatever ordering your directory structure gives you (so you won’t be able to browse by artist or genre for instance). Also GMediaServer has no direct support for flac files, but you may be able to work around this by telling it to serve “unknown” file types. The website mentions that because all tag info is stored in memory, there may be issue with large collections, but in reality it’s a very light application – System Monitor tells me it is only using 11.5 MiB despite handling a collection with thousands of mp3s.

MediaTomb

MediaTomb logo If you want something that provides multiple indexes, or want to serve video files as well as audio, try MediaTomb. It’s not quite as simple to get started with as GMediaServer; Ubuntu users may want to follow this installation guide. Another advantage of MediaTomb is that it supports on-the-fly transcoding, so you could serve your ogg files as mp3s for example. Be aware that if you install MediaTomb on Ubuntu it immediately sets itself to launch on boot, which might not be what you want.

Streaming Music

UPDATE: See my newer post which describes how to use MPD to stream music. This is my recommended solution.

In order to directly control what the Denon plays, there are a few options. Apple airplay is supposedly supported, but I couldn’t get any of the Linux apps to work with it. In the end I had to set up a stream (effectively a local internet radio station). The main option for doing this seems to be:

IceCast 2

Be warned – the documentation for IceCast seems to be pretty sparse, and it’s a slightly convoluted process to get going.

Once you’ve installed the software (again it should be your distro’s repository), edit the config file (/etc/icecast2/icecast.xml on Ubuntu) to set the passwords and then set in ENABLE=true in /etc/default/icecast2. You should now be able to launch IceCast via /etc/init.d/icecast2 start, and get to the web interface at http://localhost:8000/ (assuming you didn’t modify the port in icecast.xml).

Now that IceCast is working, you need to add a stream, either corresponding to a playlist or just echoing a soundcard. The easiest solution I found was to use EzStream. To get playlist streaming working, copy the example config file /usr/share/doc/ezstream/examples/ezstream_mp3.xml somewhere and edit the following elements:

  • url – this needs to be the same as your IceCast server plus whatever stream name you want.
  • sourcepassword – same as IceCast config
  • filename – where your playlist lives

Once this is done, launch it with ezstream -c ezstream_mp3.xml. You should now be able to open the URL you provided in a music player such as banshee, or try pointing your web browser at it.

At this point you might think we are home clear, but that’s not quite the case – we still need to point the audio player at the stream. In the case of the Denon, there is no way to type in a given URL, so you have to set-up a RadioDenon account (hopefully other players give you a means to manually enter URLs). At RadioDenon you can enter the URLs of any stations you like and they will be picked up by the player. You will need to create a stream pointing to the local network address of the PC hosting the stream e.g. http://192.168.0.1/mystream (you can find your IP address via the ifconfig utility). Be aware that you might want to configure your router to always assign your PC the same address if you’re using DHCP.

Finally, you should now be able to access the RadioDenon option on the player and select your local stream.

Minimal TODOs for Linux

Saturday, January 7th, 2012

Up until now I’ve just used bits of paper for my TODO system. This has served me quite well, but the problem is that bits of paper are too easily lost or mislaid. So, after reading Minimally Awesome TODOs I wanted to set something similar up for my Ubuntu box. Minimally Awesome TODOs describes a very simple TODO system for the Mac that displays your TODOs on the desktop. With a few tweaks, I got a very similar system working on Linux, which looks like this:

To add a todo you just use a simple terminal command:

$ todo example blog todo


To complete a todo:

$ todone 3

It was pretty easy to port the system from Mac to Linux (I did it on Ubuntu Unity, but it should work for pretty much any Linux distro with a few minor changes) by following the instructions in the post and follow-up comments, but I’ll repeat them here to save others effort. Firstly, create the following shell scripts:


$ cat /usr/local/bin/todo
#!/bin/bash
if [ $# == "0" ]; then cat $TODO; else n=$(($(tail -1 $TODO | cut -d ' ' -f 1) +1)); echo "$n ⇾ $@" >> $TODO; fi

$ cat /usr/local/bin/todone
#!/bin/bash
sed -i -e "/^$*/d" $TODO;

UPDATE: Improvements to these scripts can be found in the comments. Or, even better, try Todo .txt, which has a similarly minimalist interface but some more advanced features. There is also a page on setting it up with conky, similar to what is done below. (I found that I had to rename todo.cfg to config to get Todo .txt to work correctly though). Thanks to Fred for suggesting Todo .txt.

Remember to set them to be executable (chmod +x /usr/local/bin/todo). You then need to set a TODO environment variable to wherever you want to keep your TODO file (i.e. add something like export TODO=~/todo.txt to your .profile). If you want to sync your TODOs over several machines, you could point it to a dropbox folder or similar.

For the next step, you need to install conky, a utility for displaying info (more normally system monitor stuff) on the desktop (you should be able to grab this from your package manager). To get the same set-up as me, use the .conkyrc in this gist – I had to play around with the basic config to get it to display properly in Unity, you may face different problems in other window managers.

Finally, you should set conky to run on startup – in Ubuntu just open Startup Applications and add it to the list. UPDATE: I had a problem with the conky window staying on top of all other windows. To fix this I needed to launch conky from a script which first did sleep 10 to give the window manager time to finish starting up.

I can’t take credit for any of this stuff, all I did was put together a bunch of hints from the internet.

XPathGen Released

Thursday, January 5th, 2012

I’ve just released XPathGen, a simple utility class for generating XPath paths that uniquely identify DOM Nodes in Java.

For example:

Document testDoc = createDocument(
"<a>aa<b attr='test'>b<!-- comment -->c<c/></b>d</a>");

//Grab text node "aa"
Node aa = testDoc.getDocumentElement().getFirstChild();

System.out.println(XPathGen.getXPath(aa));
//Should print "/node()[1]/node()[1]"

I pulled this code out of diffxml and re-licensed it under the Apache licence, so that it can be used more freely.

(This was mainly prompted by this StackOverflow question).

Grab it from github if you’re interested.