My Hacking Journal

Comment on previous post

2010-02-14T15:14:00.004+10:00

I received a comment from a past work colleague of mine via Buzz in response to my previous post about my experiences with dynamically typed languages. Unfortunately, what I wanted to write didn't fit in the blog comments, hence the new post:

Murray Spork - So if I comment here I guess the comment only appears in Buzz - or does buzz know how to post this to your blog?

Looks like it's not that smart yet...

My thoughts:

1. I think you are correct in identifying the main problem of dynamic languages - what has been call "contract obfuscation"

I think it's more the fact that it is not a system that allows contracts to be automatically checked.

2. Test driven dev is essential with dynamic languages (well - I would say it is essential even when using static languages ;-). I feel your pain regarding maintaining the existing Rails app - but if they had better test coverage that would have gone a long way to solve that problem

It depends how coverage is determined. As I mentioned in the previous post, this particular code base had over 98% coverage, but it didn't take into account multiple code paths on a single line.

It's also not as effective as it's made out to be. Imagine I had some sort of address property on a User. I write unit tests around the User class based on the internal meaning of the property. Then, I write tests around all code that accesses the address property. I do that using a mock framework:


  mockUser.expects(address).andReturns("1 Park Road")
  proxy = new Proxy(mockUser)
  assertThat(proxy.address).equals("1 Park Road")

Now, three months down the track, we find out that a user can have multiple addresses, and we store them as a map (:residential -> "1 Park Road", :postal -> "PO Box 132"), under a new property called addresses.

So the new programmer on the team goes in and updates the User class (or should I say, updates the tests around the User class first, then updates the User) so that it uses the new "addresses" property. And hey presto, all the tests pass. Time to move on, right? Well, no, there's still that bit of test code above - the one that still works, because the proxy still calls the "address" method, and the mock object is setup to expect a call to address.

This is a trivial example of a problem I came across many times during maintenance. Of course, some people would say that integration testing would catch the problem, but isn't what we really want to test the fact that the property exists and returns something of a particular type? Doesn't that sound like a good job for a compiler?

So much unit/integration testing I've seen on dynamically typed code is based on testing the type contracts, not the business logic. It seems like such a wasted effort to me.

3. "in 5 years time static compilation will be seen as a weak form of unit testing" - forget who said that - provocative but some truth to it

I would put it the other way - test-driven development is a weak form of typing. Look at the Scala code in the previous post. What test can I possibly write to ensure that resetSystem must be called with a User that has already been authorised?

Or another example; what tests and defensive coding practices would you use around a list data structure that was assumed to always have at least one element in it? Using a static type system, I can write a data structure (or even better, use one from the standard library) that enforces this constraint. It would just not be possible to create an instance of this list with no elements in it (compiler error), and so any client code that ended up with an instance of one of these lists can call get(0) without the need to worry about the list being empty.

4. metaprogramming should of course be used with utmost caution - rails is horrible in this respect - they went crazy with "monkey patching" for the sake of it - even when there we must simpler solutions. Pitty that Merb got merged into Rails - they were doing a much better job in this respect.

Monkey patching is a great concept that has a terrible name because of dynamically typed languages. Imagine if you could do it in a scoped way that is guaranteed by the compiler not to conflict with other "patching" on the same class. Scala has exactly that feature.

4. did you know that the automated refactoring came from the smalltalk world (Ralph Johnson)? Don't confuse immaturity of ruby tooling for some fundamental language weakness.

And was the refactoring 100% automated and effective, or did it require some human intervention or produce any false positives? It's been too long since I've programmed in SmallTalk, but I'm pretty sure I could think of cases that would be missed.

5. IMO the only argument in favor of static languages is performance. However I'm a self-confessed dynamic language bigot - and as one static language bigot confessed to me: ultimately it is probably just a matter of style - something to do with how our brains are wired differently as to which we prefer.

Before you consider what I'm about to say, bear in mind that I'm not talking about Java or C#. Both of those languages have a terrible implementation of static typing that more often that not gets in the way of what you're trying to do.

So what do you consider the advantages of dynamic languages? Is it terseness? If so, I'll show you some Haskell. It certainly can't be correctness, as static typing is an encoding of a mathematical theorem; something that you can use to prove correctness (actual proofs, not failure to disprove, aka unit testing). Given the amount of additional effort of writing tests to reproduce what the compiler can do, I can't imagine that efficiency would be a benefit. Lower barrier of entry? Well, yes, I think that's the big advantage of dynamic languages. But is that something you really want to strive for?

Using the type system for discoverability and enforcing constraints

2009-04-27T17:52:00.022+10:00

Now, anyone from a Ruby or other dynamically typed language background who read my last post might be thinking "big deal, I've been able to do that for ages". And it's true; some of these languages with support for first-class functions make writing functional code just as easy to write as it is in Scala and the like. So why would I want to use a strongly statically typed language over a dynamically typed one?

Well, I'm glad you asked. Before I really got stuck into understanding functional programming and strongly statically typed languages, I spent some time working on a Rails app. Unlike most of the experiences I'd heard from people moving to Ruby/Rails though, I joined this project during the maintenance phase. What followed for me was an up-hill battle of trying to understand the code and gain the implicit knowledge of my other colleagues who worked on the code before me. The codebase left a lot to be desired in terms of discoverability. Due to the magic of meta-programming, I couldn't even rely on grep to find all references to a particular method.

One example that remains burned into my memory was the frustration surrounding something as seemingly simple as:


tags = item.tags

What is the type of tags? Is it an array of Tag objects, an array of string tag names, or a comma separated string of tag names?

The answer in my case was all of the above! What it was at runtime depended on the code path that lead to the tags property being set on the item. By coincidence/implicit design/whatever, it just so happened that the way it was used was consistent across the code paths that invoked it, so it never showed up as a problem until I came in one day to re-use the data in a different context. Not even the tests (which covered > 98% of the code base) picked this up for me, because the tests were written upfront for the particular context that was being implemented and so it was always set to what was expected.

It got to the point where I was so paranoid about releasing anything because I wasn't sure of all of the knock-on effects my changes would have. I was bitten too many times by not executing all of the code paths that lead to the point of my changes, because I couldn't find them all up front. I was paranoid about deleting code that I didn't think was being used any more, and even worried about factoring common-looking code out because at least on one of each of those occasions, a bug turned up either late in testing or actually in production.

My flirt with mid-size scale projects in dynamically typed languages wasn't all doom and gloom though. The second rails app I worked on went a lot smoother. I put this more down to the fact that it was a green-fields project rather than because I was more familiar with the language/framework. In developing the project from the start, I had all of the assumptions and implicit knowledge about the types in my head. As a result, using the Rails framework was a breeze, and we managed to get the project completed on time and on budget.

Yes, it could be said that my first experience was just a one-off; that the previous developers didn't always stick to the conventions (whatever they're supposed to be), or were doing things in ways I wasn't aware of. But it's exactly some of these implicit assumptions and conventions that can be enforced by the compiler in a strongly statically typed language. There's no need to write your all assumptions as comments in code; many can be encoded in the language you're writing in.

Consider this code:


# user.rb
 class User
   def authorised?
     ...
   end
   ...
 end

# admin.rb
 class Admin
   def Admin.reset_system(user)
     check_authorisation(user)
     ...
     send_admin_email(user, "System was reset")
   end

   def Admin.send_admin_email(user, msg)
     check_authorisation(user)
     ...
   end

   def Admin.check_authorisation(user)
     raise "Unauthorised" unless user.authorised?
   end
 end

Without looking at the implementation, the tests or the documentation (assuming it exists), there is no way to indicate that clients calling the resetSystem and sendAdminEmail functions must pass in an authorised user (never mind the fact that you don't know for sure if the methods accept a User instance, a user identifier, or something else). Also note the authorisation check duplication. You could probably do some meta-programming to do the check before the methods are called, thus not having to worry about remembering to add the check for each authorised method in the future. However, the check is now removed from the context of the method, and it would be easy to circumvent the check if you needed to do a quick hack to get something working. Yes, the tests you (are supposed to) write should catch these problems, but they don't give you any guarantee over future changes (and from my own personal experience, the tests are a PITA to maintain).

Now, imagine not only being able to get around the problems above, but to do so in a way where it is practically not possible to get the code past the compiler without it conforming to your assumptions (unless you subvert the type system, in which case you're defeating its purpose). This is the situation I've become used to in a strongly statically typed language.


// User.scala
  class User(...) {
    def authorised_? = ...
  }

// AuthorisedUser.scala
  sealed abstract class AuthorisedUser(val user:User)

  private class AuthorisedUserImpl(user:User) extends AuthorisedUser(user)

  object AuthorisedUser {
    def authorise(user:User):Option[AuthorisedUser] =
        if (user.authorised_?) Some(new AuthorisedUserImpl(user)) else None
  }

// Admin.scala
  object Admin {
    def resetSystem(user:AuthorisedUser) = {
      ...
      sendAdminEmail(user, "System was reset")
    }
    def sendAdminEmail(user:AuthorisedUser, msg:String) = ...
  }

A quick note about the Scala syntax again:

Scala makes the distinction between objects and classes, where the former declares a singleton and only defines static functions (eg. authorise), and the latter can be instantiated and defines instance methods (eg. authorised_?). Java does not make this distinction; a class can define both static functions and instance methods.

The sealed keyword denotes a class that cannot be extended from outside of the file it is defined in. In this case, as a client, you cannot circumvent the authorisation check by creating another subclass of AuthorisedUser to pass in to the restricted functions.

Some and None are subclasses of the Option type in Scala. In this case, it allows an optional AuthorisedUser to be returned from the authorise function depending on whether or not the underlying user is authorised.

From the above code, it is just not practically possible to write client code that calls the resetSystem and sendAdminEmail functions without first calling the authorise function. If you try, your code won't compile. None of this finding out at test-time/run-time - it can't run, because it won't compile.

Not only that, but the compiler actually removes the need to test authorisation checks for the Admin functions. There are no tests to write that would make sense; either the user is authorised, or the code doesn't compile!

As you can see, in this case the statically typed language allows more assumptions to be made explicit in the code. If I decide to do a one-off hack, the compiler can be used to keep me honest, and I've found that it generally makes me think more about the problem I'm trying to solve. Yes, it can be frustrating trying to get your code to pass the compiler (especially when you think it's right), but more often than not, when it does compile, I am a lot more confident that it is correct in terms of being consistent with my assumptions. The type system can also be seen as the ultimate defensive coding practice. With a type system, discussions like these tend to become moot.

The most common objection I've come across to using a statically typed language over a dynamically typed language is that the former is more verbose and requires more code to achieve the same functionality in the latter. This is only true of languages with a poor type system implementation, and has already been addressed many times before. I wasn't going to mention it, but for those who didn't notice, the strongly-statically typed version above actually has fewer lines of code than the dynamically typed version. It also requires fewer tests (because there's no way to call the admin functions with an unauthorised user) and has all the other benefits of statically compiled code. Win-win-win.

Whilst there will be more examples of some of the advantages you can get from the type system, I don't plan to write too much more on the specific topic of Dynamic vs. Static types as it has already been covered many times before. I am very open to criticisms of the above comparison, as long as it focuses on the topic at hand, not the actual implementation (it is one of the smaller examples I could come up with to try get my point across, and as such, it is somewhat contrived).

Also note that you can get these benefits in a language with a weaker type system, such as Java. However, the ceremony involved in declaring the types usually makes it infeasible to do for the trivial cases and painful for the complex cases. Below you can see the previous Scala code that has been ported to Java, making use of the FunctionalJava library:


// User.java
  public class User {
    public boolean isAuthorised() {
      ...
    }
  }

// AuthorisedUser.java
  public abstract class AuthorisedUser {
    final User user;
 
    private AuthorisedUser(final User user) {
      this.user = user;
    }
    
    private static class AuthorisedUserImpl extends AuthorisedUser {
      public AuthorisedUserImpl(final User user) {
        super(user);
      }
    }
    
    public static Option<AuthorisedUser> authorise(final User user) {
      return user.isAuthorised()
          ? Option.<AuthorisedUser>some(new AuthorisedUserImpl(user))
          : Option.<AuthorisedUser>none();
    }
  }

// Admin.java
  public class Admin {
 
    public static void resetSystem(final AuthorisedUser user) {
      ...
      sendAdminEmail(user);
    }
 
    public static void sendAdminEmail(final AuthorisedUser user) {
      ...
    }
  }

Filtering a list

2009-03-13T16:28:00.016+10:00

Ok, let's start off with an easy one. How many times have you written something like the following:


  public Iterable<User> getManagers(Iterable<User> users) {
    List<User> managers = new ArrayList<User>();
    for (User user : users) {
      if (user.isManager()) {
        managers.add(user);
      }
    }
    return managers;
  }

I know I've written a lot of code like that in the past. Each time, there's something slightly different. Either the types are different or the test in the if statement is different. For example, imagine a similar requirement to get the list of active users. There doesn't appear to be much more you can do aside from:


  public Iterable<User> getActiveUsers(Iterable<User> users) {
    List<User> activeUsers = new ArrayList<User>();
    for (User user : users) {
      if (user.isActive()) {
        activeUsers.add(user);
      }
    }
    return activeUsers;
  }

Well, there is some stuff we can factor out if we really wanted to:


  private static interface Condition {
    boolean check(User user);
  }

  private Iterable<User> filterUsers(Iterable<User> users, Condition cond) {
    List<User> filteredUsers = new ArrayList<User>();
    for (User user : users) {
      if (cond.check(user)) {
        filteredUsers.add(user);
      }
    }
    return filteredUsers;
  }

  public Iterable<User> getManagers(Iterable<User> users) {
    return filterUsers(users, new Condition() {
      public boolean check(User user) {
        return user.isManager();
      }
    });
  }

  public Iterable<User> getActiveUsers(Iterable<User> users) {
    return filterUsers(users, new Condition() {
      public boolean check(User user) {
        return user.isActive();
      }
    });
  }

However, given that it's more code than the two original methods, I probably wouldn't actually do that unless I had a lot of similar looking methods. In any case, there's still duplication in the creation of the conditions. Without resorting to reflection (which would instantly open myself up to a whole new class of bugs), I can't think of much more that can be done with this code in Java.

In terms of the bounds of readability I mentioned in my dilemma, I would have left it at the first two methods and moved on.

Now take a look at the same two methods written in Scala; a language that supports first class functions:


  def getManagers(users:List[User]):List[User] = {
    return users.filter(isUserManager);
  }

  def getActiveUsers(users:List[User]):List[User] = {
    return users.filter((user:User) => user.isActive());
  }

  def isUserManager(user:User):Boolean = {
    return user.isManager();
  }

First, some quick notes about the syntax:

The def keyword is used to declare a function or a method

In Scala, type annotations come at the end of the parameter and function declarations (after the ':' character)

Generics in Scala are expressed using square brackets as opposed to Java's angle brackets

On to the semantics of the code. The filter function on Scala's List class is a higher-order function, which in this case it means that it is a function that takes another function as a parameter. In the first case, you can see how the parameter to the filter function is another named function that is defined later on. In the second case, the parameter is defined in-line as an anonymous (unnamed) function.

In this example, the type of the parameter that List.filter expects is:

A function, which takes a single User parameter, and returns a Boolean value.

In the example, the List.filter function returns a (possibly empty) List of User objects.

I intentionally wrote out the Scala code in a verbose manner to try to make it easier read if you're coming from Java. However, in most cases there are a lot of aspects of the Scala syntax that are optional, such as semicolons, the return keyword, braces for single statements, and return types when they can be inferred. As someone who is more comfortable with the language, I find it easier to write it and at least as easy to read it as:


  def getManagers(users:List[User]) = users.filter(isUserManager)
  def getActiveUsers(users:List[User]) = users.filter((user:User) => user.isActive())
  def isUserManager(user:User) = user.isManager()

Actually, I would probably inline the isUserManager function because it's almost like an alias now, and because the user parameter in the anonymous function is only used once and the type of it can be inferred, the last function can be shortened further still:


  def getManagers(users:List[User]) = users.filter(_.isManager())
  def getActiveUsers(users:List[User]) = users.filter(_.isActive())

Now, imagine getting used to writing code like that and then coming back to this:


  public Iterable<User> getManagers(Iterable<User> users) {
    List<User> managers = new ArrayList<User>();
    for (User user : users) {
      if (user.isManager()) {
        managers.add(user);
      }
    }
    return managers;
  }

  public Iterable<User> getActiveUsers(Iterable<User> users) {
    List<User> activeUsers = new ArrayList<User>();
    for (User user : users) {
      if (user.isActive()) {
        activeUsers.add(user);
      }
    }
    return activeUsers;
  }

And, if you'll allow me to mix my metaphors, this is barely scratching the tip of the iceberg.

Moral dilemma

2009-03-04T20:30:00.006+10:00

I have found myself in the unfortunate position of being asked to dumb-down the work that I do so it is understandable to an "average" programmer (whatever that means). I understand where the request is coming from, ie, there is a fear that if it's not dumbed down, there will be a difficulty in finding someone to understand and maintain the work in the future. However, for me to comply with the request would mean to knowingly write code that is less correct, contain more duplication (both in terms of the effort and the resulting code), and have more known potential traps for future maintenance - something that the "average" programmer would tell you is a "bad thing".

How did I find myself in this position?

In the past, I would try to adhere to the principles of DRY, but on a number of occasions, I would get to the point where in order to avoid repetition, I had to bend the language that I used in ways it wasn't designed for. The resulting code would end up being verbose and very difficult to understand after time, and I still wasn't achieving the re-use that I thought should be possible. In the end, I learned roughly where some of those boundaries of readability lay, and resigned myself to the belief that the duplication was a fact of life.¹

Then, about a year ago, I was lucky enough to spend a bit of time learning about Functional Programming and Strong, Static Type Systems with someone who is a self confessed fundamentalist functional programmer. Over many months, I was able to change the way I thought about programming. My progress was very slow to begin with, and I found that a lot of the discussions and reading I was doing on the subject were out of reach of my understanding. However, through continual encouragement, and when I finally started doing the excercises and writing small programs in this new way, things started to sink in and make sense. I didn't change my way of thinking in a single Eureka! moment, but more in a series of small "A ha, now I think I get it" moments.

Now I can write far more concise code using very powerful abstractions, with the added benefit that a large class of potential bugs don't even get past the compiler.

What am I going to do about it?

Back to my dilemma. Obviously, if I want to do something about it, I need to present my argument in a convincing way. I don't have a way of showing the whole picture in a single, simple, succinct way. In fact, I'm very doubtful that such a way exists. There was a lot of unlearning I had to do before I could really begin to see the benefits of functional programming.

I had planned to write up my experiences along the way while I was learning, but I didn't think I could do any better than what other people have already put out there; people who have come to the same conclusions and made the same observations that I have. Having been forced back into the position of writing imperative code though has given me a good reason to get my thoughts down in writing.

So, with that in mind, I'm planning to write up some examples of situations I've come across, where I have relented and written repetitive, less correct, or more error prone code in order to fulfill the contradictory requirements that lead to my dilemma. Using these examples, I hope to build a case for using languages that support functional programming and strong, static type systems even if these languages aren't popular or in the mainstream.

From the personal experience of someone who heavily bought into TDD, OOAD, Design Patterns, etc, I can predict that some of the examples will look wrong, or completely opposite to what is regarded as "best practice" in the mainstream. I will try to identify those areas and drill down into them when I get a chance. In the meantime, you should at least be prepared to question your current knowledge.

This is not going to be another burrito tutorial, and I can almost guarantee that anyone who reads only what I plan to write will not come away with an understanding of Functional Programming. The aim is to show some of the possibilities and maybe encourage people to look further into the context that makes it possible.

Finding large files

2008-10-28T16:57:00.001+10:00

Yet another thing I keep forgetting:


  find . -size +10000k

Finds files over roughly 10Mb

Proxy instantiation problem from Hibernate Session load

2008-10-24T08:45:00.007+10:00

This one had me puzzled for a while yesterday. I was able to retrieve objects from a Session via a query, but I was getting the following error when I was using the load method:


org.hibernate.HibernateException: Javassist Enhancement failed: Thing
        at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.getProxy(JavassistLazyInitializer.java:142)
        at org.hibernate.proxy.pojo.javassist.JavassistProxyFactory.getProxy(JavassistProxyFactory.java:72)
        at org.hibernate.tuple.entity.AbstractEntityTuplizer.createProxy(AbstractEntityTuplizer.java:402)
        at org.hibernate.persister.entity.AbstractEntityPersister.createProxy(AbstractEntityPersister.java:3483)
        at org.hibernate.event.def.DefaultLoadEventListener.createProxyIfNecessary(DefaultLoadEventListener.java:298)
        at org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:219)
        at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:126)
        at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:905)
        at org.hibernate.impl.SessionImpl.load(SessionImpl.java:822)
        at org.hibernate.impl.SessionImpl.load(SessionImpl.java:815)
...
Caused by: java.lang.InstantiationException: Thing_$$_javassist_0
        at java.lang.Class.newInstance0(Class.java:335)
        at java.lang.Class.newInstance(Class.java:303)
        at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.getProxy(JavassistLazyInitializer.java:139)
        ... 49 more

When I changed the code from


  session.load(classOf[Thing], id).asInstanceOf[Thing]


  session.createQuery("from Thing where id = " + id).list().asInstanceOf[java.util.List[Thing]].toList(1)

everything worked as expected.

I found a few results from the web, but nothing to indicate what my specific problem was.

The code I was working on was loosely based on another Scala project, which uses an older version of Hibernate. I changed the libraries over, and got a similar problem with CGLIB:


org.hibernate.HibernateException: CGLIB Enhancement failed: Thing
        at org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy(CGLIBLazyInitializer.java:96)
        at org.hibernate.proxy.pojo.cglib.CGLIBProxyFactory.getProxy(CGLIBProxyFactory.java:49)
        at org.hibernate.tuple.entity.AbstractEntityTuplizer.createProxy(AbstractEntityTuplizer.java:379)
        at org.hibernate.persister.entity.AbstractEntityPersister.createProxy(AbstractEntityPersister.java:3455)
        at org.hibernate.event.def.DefaultLoadEventListener.createProxyIfNecessary(DefaultLoadEventListener.java:257)
        at org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:191)
        at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:103)
        at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:878)
        at org.hibernate.impl.SessionImpl.load(SessionImpl.java:795)
        at org.hibernate.impl.SessionImpl.load(SessionImpl.java:788)
...
Caused by: java.lang.InstantiationException: Thing$$EnhancerByCGLIB$$e29dbda1
        at java.lang.Class.newInstance0(Class.java:335)
        at java.lang.Class.newInstance(Class.java:303)
        at org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxyInstance(CGLIBLazyInitializer.java:107)
        at org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy(CGLIBLazyInitializer.java:93)
        ... 49 more

This time I was luckier with my search results, which eventually lead me to these two pages. Indeed, the problem was due to having a private no-args constructor in my Thing object. As soon as I changed the constructor to be visable, the calls to load() started working as expected.

I was thrown a bit by the InstantiationException, which according to the JavaDoc is

Thrown when an application tries to create an instance of a class using the newInstance method in class Class, but the specified class object cannot be instantiated because it is an interface or is an abstract class.

While I can see that it's related, I don't really think the private constructor issue strictly falls under that description. In any case, I'm sure that a better message, or a different exception, or an update to the documentation, would have helped me isolate the problem a lot quicker.

Oddly, the code from the other project also has a private no-args constructor, yet it seems to work. That part remains a mystery to me, but I thought I'd post my other findings in case it helps anyone else who runs into a similar issue.

Home directory version control notification

2008-10-16T20:55:00.011+10:00

I spent some time in admin mode recently, seeing if I could improve the way my home directory is versioned. Previously, I had a few Mercurial repositories scattered around the place to track my documents, scripts, config and miscellaneous files.

Most of my work-related directories have their own Subversion repositories attached to them, but there were also some scratch projects that weren't under any VCS. My config files (eg, .bashrc) were sym-linked to a separate directory that had its own repository.

The main problems I had with the different repositories is that I didn't have an easy way to determine if I had any outstanding changes that need to be committed (my documents directory was shockingly out of date), and there were things that I wanted versioned (for example, application preferences) that weren't being versioned.

My first thought was to add a single Mercurial repository at the root of my home directory so I only had one repository to remember, and could easily track anything that gets added over time (for example, config files for new programs). Once I started thinking about all of the entries in my ignore file though, and how I wanted to share some stuff between work and home but not others, this option became less and less appealing.

So, back to the multiple repositories it was. What I ended up was repositories for the following directories:


  ~/Documents
  ~/Library
  ~/home
  ~/projects

The ~/Library directory is the standard directory on OSX for storing application support files, preferences, etc, and while I'm not particularly interested in the day-to-day happenings of this directory, it is helpful to have snap-shots that I can revert to at a later date.

I'm using the home repository to track utility scripts and dot files that live directly under ~ (mostly run config and history files). Almost all dot files in my ~ directory are sym-linked to somewhere in my home repository, and I have a bootstrap script to take care of this if I need to setup another home directory on a different machine in the future.

Finding out how to merge my old scripts and config repositories (whilst retaining their histories) was not a simple exercise. I thought I might be able to use Mercurial's import and export commands to merge one into the other, but there doesn't appear to be a way of specifying a relative path to import a repository at, so when I tried, I ended up with everything in the root of the repository. In hindsight, I realise I could have created a new repository and imported both repositories there, making sure to move the contents straight after importing, but there was another similar solution that I eventually found and got to work.

Now that I was happy with my repositories layout, I wanted a way to make sure they were kept up to date. At first I thought about adding a cron job that adds, removes, and commits changes automatically at given periods. Then I thought that might lead to stuff being committed that I didn't want (for example, if I generated a large amount of data that slipped through my ignore file), and having a repository log full of static messages. So instead, I decided to go with a notification approach instead.

Finding uncommitted changes was pretty straight forward with the following command:


  find . -name .hg -not -path "./Library/.hg" -exec /opt/local/bin/hg st {}/.. \;

(I took the Library repository out of the results because I didn't want to be bugged about those changes throughout the day; instead, I just take a snapshot of that at the end of each day).

This has the added benefit that if I add more Mercurial repositories in the future, they'll automatically be included in the notification without me having to change anything.

Next up, notification. I find Growl on the Mac to be useful for receiving reasonably unobtrusive notifications throughout the day. It ships with a command-line interface (that you need to install separately), which was perfect for what I needed:


  /usr/local/bin/growlnotify "Uncommitted Changes" <$CHANGES_FILE

$CHANGES_FILE holds the output of the previous find command. I wanted it stored so I could also copy the changes to my desktop, giving me a secondary visual cue that I have changes outstanding in case I missed the Growl notification.

I ended up with the following bit of script (after adding a few finishing touches, such as using a temp file for the intermediate results, only notifying if the size of the changes file is greater than zero, and displaying the notification with a nice Mercurial icon):


  TMP_FILE=`mktemp /tmp/uncommitted_changes.XXXXXX` || exit 1
  
  cd $HOME
  find . -name .hg -not -path "./Library/.hg" \
      -exec /opt/local/bin/hg st {}/.. \; >$TMP_FILE
  
  if [[ -s $TMP_FILE ]]; then
    CHANGES_FILE=$HOME/Desktop/uncommitted_changes.txt
    mv $TMP_FILE $CHANGES_FILE
    /usr/local/bin/growlnotify \
        --image $HOME/bin/img/hg-icon-50.png \
        "Uncommitted Changes" <$CHANGES_FILE > /dev/null 2>&1
  else
    rm $TMP_FILE
  fi

And here it is in action:

One thing to note is that the output of growlnotify is being ignored. This is because of a known issue of invoking growlnotify from cron, whereby the notification still works but the following warning is mailed to you:


growlnotify[1592:10b] could not find local GrowlApplicationBridgePathway, falling back to NSDNC

I tried using the network settings described here, but this resulted in my notifications not showing up at all. For now, I'm just ignoring the output of growlnotify. I figure if something else goes wrong with it, it'll most likely result in notifications not showing up, which I think I will notice once I start getting used to them.

One final thing to mention is that I had intermittent trouble with using the --image flag when explicitly running the script. It sounds related to this problem, in that the notification would only be displayed intermittently but no other error or warning was emitted. Interestingly, I don't seem to get the problem when the script runs from cron, so I've left the flag in for now.

So there you go; notifications for uncommitted changes under my home directory, and a great reminder for me to go back and look at whether I want to keep configuration changes that are usually made when my mind is on other things. I've currently got it set to run every 15 minutes, which seems to be a reasonable balance between being too annoying and having too many outstanding changes.

Replacement in Bash parameter expansions

2008-10-03T16:31:00.008+10:00

I have a few utility scripts to make working on the command line easier. One of them is a shortcut for temporarily getting a file or directory out of the way, by renaming it with a .bak suffix. So instead of typing

mv tmp/foo.txt tmp/foo.txt.bak

I can type


bk tmp/foo.txt

and achieve the same thing.

The script to do this was this simple one-liner

mv $1 "$1.bak"

Unfortunately, due to using tab-completion for file names, I'd often get the following error

bk tmp/bar/
mv: rename tmp/bar/ to tmp/bar/.bak: Invalid argument

which is caused by the trailing slash that is included when you tab-complete the directory name. So, ideally, what I wanted to do was check if there was a trailing slash and remove it for the second argument to mv.

Conveniently, this is easily achieved using a feature of Bash parameter expansions:

${parameter/pattern/string}
The pattern is expanded to produce a pattern just as in filename expansion. Parameter is expanded and the longest match of pattern against its value is replaced with string. If pattern begins with ‘/’, all matches of pattern are replaced with string. Normally only the first match is replaced. If pattern begins with ‘#’, it must match at the beginning of the expanded value of parameter. If pattern begins with ‘%’, it must match at the end of the expanded value of parameter. If string is null, matches of pattern are deleted and the / following pattern may be omitted. If parameter is ‘@’ or ‘*’, the substitution operation is applied to each positional parameter in turn, and the expansion is the resultant list. If parameter is an array variable subscripted with ‘@’ or ‘*’, the substitution operation is applied to each member of the array in turn, and the expansion is the resultant list.

A quick change to my script and all is well:

mv $1 "${1/%\//}.bak"

Incidentally, calling it with the wrong directory (something like sudo bk /usr/lib) can be a bit disastrous, but that's a different story.

Updating MacPorts

2008-09-19T10:59:00.009+10:00

Hopefully the last time I forget this:


  sudo port selfupdate

I've found that this is the most common fix for a failed port install. For example, today, I had the following error when trying to upgrade Subversion:


> sudo port upgrade subversion
--->  Building sqlite3 with target all
Error: Target org.macports.build returned: shell command " cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_databases_sqlite3/work/sqlite-3.6.1" && gnumake all " returned error 1
Command output: sh: line 0: cd: /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_databases_sqlite3/work/sqlite-3.6.1: No such file or directory

Sometimes a port clean is also required.

Fix for "access denied" problem when submitting background file upload

2008-09-18T13:02:00.009+10:00

Since Scoodi started to rise in popularity, we've been getting feedback from some Internet Explorer 6 users saying that their photos weren't being uploaded during the process of creating an item. Scoodi uses a common browser hack feature to achieve uploading files in the "background", similar to adding attachments in GMail, so you can continue to fill in the form while photos are uploading. The trick involves setting the target of the file upload form to a hidden frame, so that when the form is submitted, the result is sent to the hidden frame instead of having whole page re-displayed with the result.

Luckily, we were able to reproduce the problem on one of our test machines (though not on another, which made things more frustrating interesting). The difference ended up being in the security settings, specifically the "allow sub-frames to navigate across different domains" setting.

To make a long story short, our issue was to do with how we were setting the src attribute on the hidden iframe. When we enabled Asset Hosts, our source attribute was an absolute path to one of the asset hosts:


  <iframe ... src="http://asset2.scoodi.com/path/blank.html">

Going back to the security setting, in our case, it was saying you can't submit a form served from www.scoodi.com to a target frame served from asset2.scoodi.com. When the setting was enforced, it raised an "access denied" error from the line of JavaScript that submits the form.

Once we worked out the problem, the fix was simple: make sure the iframe src attribute uses the same domain as the page source. In our case, it was as easy as using a path relative to the server:


  <iframe ... src="/path/blank.html">

It was no coincidence that the bug appeared with the increasing number of users to the site, though it wasn't because we were getting a wider range of browsers and configuration permutations. We enabled Asset Hosts because of the increase in users, and when we did that, those IE6 browsers with higher security settings stopped working.

Scoodi on TV

2008-08-18T19:48:00.005+10:00

A couple of weeks ago, I was involved the filming of a "news-fo-tainment" segment where we got to talk about and promote Scoodi, the project I've been involved in for the past few months:

The impact that the segment had on the website traffic was a mixed blessing, and it would probably be fair to say that it caught us a little off guard. We've since fixed one of the main bottle necks, and by all accounts, the site is quite stable once again.

Installing GHC on OSX

2008-08-07T18:15:00.001+10:00

During my quest to better understand Functors & Monads, it was suggested that I take a detour and learn Haskell first, as I was struggling a bit with Scala's type annotations. So, off to MacPorts I went:


  ~> sudo port install ghc
  --->  Fetching ghc
  --->  Verifying checksum(s) for ghc
  --->  Extracting ghc
  --->  Applying patches to ghc
  --->  Configuring ghc
  --->  Building ghc with target all

Half an hour later, I was still waiting for it to build. Something was not quite right. I killed the process and tried again (after issuing a sudo port clean ghc, because it seemed to be in an inconsistent state after killing it).

Same problem - it got up to building, then seemed to hang. Looking at top, I could see ghc-6.8.2 running intermittently. After a while, ghc-6.8.3 was also added to the list of processes (running intermittently), but still no progress on the build.

I gave it about 45 minutes before killing it again. Brad has been down this path before, and he put me on to the pre-packaged installer available from the http://haskell.org/ghc download page. As per the instructions, I installed XCode off the Leopard disc, then ran the installer, and hey presto, I was ghc hello_world.hs'ing it up!

Installing/upgrading Scala

2008-08-04T01:45:00.001+10:00

Because I keep forgetting:


cd /opt
sudo tar xzf ~/install/scala-2.7.1.final.tgz
sudo rm scala
sudo ln -s scala-2.7.1.final scala
cd scala
sudo bin/sbaz install scala-devel-docs

This cryptic message when installing the docs indicates lack of privileges, ie, not running with sudo:


planning to install: scala-devel-docs/2.7.1.final
Installing...
java.io.FileNotFoundException: /opt/scala-2.7.1.final/meta/cache/scala-devel-docs-2.7.1.final.zip.tmp (No such file or directory)

Splitting up large files

2008-06-30T20:52:00.001+10:00

This is one of those things that I need to do occasionally, but not often enough that I've been able to commit it to memory.


 split -b 4000m large_file large_file.

This will split a the large_file file into 4Gb files with filenames large_file.aa, large_file.ab, etc. Before doing this, it really helps to make sure you've got enough space for the new files - ie, at least as much free space as the file you are splitting!

To get the parts back together:


 cat large_file.* > large_file

Solving The Tangled Working Copy Problem

2008-06-17T17:36:00.001+10:00

I just read this article, which goes into great (as in good) detail about one way to solve the problem where you have more than one conceptual change in your work-space, and you want to physically separate them into different groups.

When I've encountered this problem previously, I've generally just backed up the working copy, reverted the unwanted changes in the working copy, submitted it, then pulled back the additional changes form the backup. But the way described in the article is a much better way!

Apparently it is supported in Mercurial, so I'll have to give it a go next time. Unfortunately, it doesn't help when I'm working with Subversion, as is currently the case at work. However, the article also mentions a way of do a similar thing using the patch program, which is still better than my manual way around it.