find . -size +10000k
Finds files over roughly 10Mb
Tuesday, October 28, 2008
Finding large files
Yet another thing I keep forgetting:
Finds files over roughly 10Mb
Finds files over roughly 10Mb
Friday, October 24, 2008
Proxy instantiation problem from Hibernate Session load
This one had me puzzled for a while yesterday. I was able to retrieve objects from a Session via a query, but I was getting the following error when I was using the load method:
When I changed the code from
I found a few results from the web, but nothing to indicate what my specific problem was.
The code I was working on was loosely based on another Scala project, which uses an older version of Hibernate. I changed the libraries over, and got a similar problem with CGLIB:
This time I was luckier with my search results, which eventually lead me to these two pages. Indeed, the problem was due to having a private no-args constructor in my Thing object. As soon as I changed the constructor to be visable, the calls to load() started working as expected.
I was thrown a bit by the InstantiationException, which according to the JavaDoc is
While I can see that it's related, I don't really think the private constructor issue strictly falls under that description. In any case, I'm sure that a better message, or a different exception, or an update to the documentation, would have helped me isolate the problem a lot quicker.
Oddly, the code from the other project also has a private no-args constructor, yet it seems to work. That part remains a mystery to me, but I thought I'd post my other findings in case it helps anyone else who runs into a similar issue.
org.hibernate.HibernateException: Javassist Enhancement failed: Thing
at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.getProxy(JavassistLazyInitializer.java:142)
at org.hibernate.proxy.pojo.javassist.JavassistProxyFactory.getProxy(JavassistProxyFactory.java:72)
at org.hibernate.tuple.entity.AbstractEntityTuplizer.createProxy(AbstractEntityTuplizer.java:402)
at org.hibernate.persister.entity.AbstractEntityPersister.createProxy(AbstractEntityPersister.java:3483)
at org.hibernate.event.def.DefaultLoadEventListener.createProxyIfNecessary(DefaultLoadEventListener.java:298)
at org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:219)
at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:126)
at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:905)
at org.hibernate.impl.SessionImpl.load(SessionImpl.java:822)
at org.hibernate.impl.SessionImpl.load(SessionImpl.java:815)
...
Caused by: java.lang.InstantiationException: Thing_$$_javassist_0
at java.lang.Class.newInstance0(Class.java:335)
at java.lang.Class.newInstance(Class.java:303)
at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.getProxy(JavassistLazyInitializer.java:139)
... 49 more
When I changed the code from
to
session.load(classOf[Thing], id).asInstanceOf[Thing]
everything worked as expected.
session.createQuery("from Thing where id = " + id).list().asInstanceOf[java.util.List[Thing]].toList(1)
I found a few results from the web, but nothing to indicate what my specific problem was.
The code I was working on was loosely based on another Scala project, which uses an older version of Hibernate. I changed the libraries over, and got a similar problem with CGLIB:
org.hibernate.HibernateException: CGLIB Enhancement failed: Thing
at org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy(CGLIBLazyInitializer.java:96)
at org.hibernate.proxy.pojo.cglib.CGLIBProxyFactory.getProxy(CGLIBProxyFactory.java:49)
at org.hibernate.tuple.entity.AbstractEntityTuplizer.createProxy(AbstractEntityTuplizer.java:379)
at org.hibernate.persister.entity.AbstractEntityPersister.createProxy(AbstractEntityPersister.java:3455)
at org.hibernate.event.def.DefaultLoadEventListener.createProxyIfNecessary(DefaultLoadEventListener.java:257)
at org.hibernate.event.def.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:191)
at org.hibernate.event.def.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:103)
at org.hibernate.impl.SessionImpl.fireLoad(SessionImpl.java:878)
at org.hibernate.impl.SessionImpl.load(SessionImpl.java:795)
at org.hibernate.impl.SessionImpl.load(SessionImpl.java:788)
...
Caused by: java.lang.InstantiationException: Thing$$EnhancerByCGLIB$$e29dbda1
at java.lang.Class.newInstance0(Class.java:335)
at java.lang.Class.newInstance(Class.java:303)
at org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxyInstance(CGLIBLazyInitializer.java:107)
at org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy(CGLIBLazyInitializer.java:93)
... 49 more
This time I was luckier with my search results, which eventually lead me to these two pages. Indeed, the problem was due to having a private no-args constructor in my Thing object. As soon as I changed the constructor to be visable, the calls to load() started working as expected.
I was thrown a bit by the InstantiationException, which according to the JavaDoc is
Thrown when an application tries to create an instance of a class using the newInstance method in class Class, but the specified class object cannot be instantiated because it is an interface or is an abstract class.
While I can see that it's related, I don't really think the private constructor issue strictly falls under that description. In any case, I'm sure that a better message, or a different exception, or an update to the documentation, would have helped me isolate the problem a lot quicker.
Oddly, the code from the other project also has a private no-args constructor, yet it seems to work. That part remains a mystery to me, but I thought I'd post my other findings in case it helps anyone else who runs into a similar issue.
Thursday, October 16, 2008
Home directory version control notification
I spent some time in admin mode recently, seeing if I could improve the way my home directory is versioned. Previously, I had a few Mercurial repositories scattered around the place to track my documents, scripts, config and miscellaneous files.
Most of my work-related directories have their own Subversion repositories attached to them, but there were also some scratch projects that weren't under any VCS. My config files (eg, .bashrc) were sym-linked to a separate directory that had its own repository.
The main problems I had with the different repositories is that I didn't have an easy way to determine if I had any outstanding changes that need to be committed (my documents directory was shockingly out of date), and there were things that I wanted versioned (for example, application preferences) that weren't being versioned.
My first thought was to add a single Mercurial repository at the root of my home directory so I only had one repository to remember, and could easily track anything that gets added over time (for example, config files for new programs). Once I started thinking about all of the entries in my ignore file though, and how I wanted to share some stuff between work and home but not others, this option became less and less appealing.
So, back to the multiple repositories it was. What I ended up was repositories for the following directories:
The ~/Library directory is the standard directory on OSX for storing application support files, preferences, etc, and while I'm not particularly interested in the day-to-day happenings of this directory, it is helpful to have snap-shots that I can revert to at a later date.
I'm using the home repository to track utility scripts and dot files that live directly under ~ (mostly run config and history files). Almost all dot files in my ~ directory are sym-linked to somewhere in my home repository, and I have a bootstrap script to take care of this if I need to setup another home directory on a different machine in the future.
Finding out how to merge my old scripts and config repositories (whilst retaining their histories) was not a simple exercise. I thought I might be able to use Mercurial's import and export commands to merge one into the other, but there doesn't appear to be a way of specifying a relative path to import a repository at, so when I tried, I ended up with everything in the root of the repository. In hindsight, I realise I could have created a new repository and imported both repositories there, making sure to move the contents straight after importing, but there was another similar solution that I eventually found and got to work.
Now that I was happy with my repositories layout, I wanted a way to make sure they were kept up to date. At first I thought about adding a cron job that adds, removes, and commits changes automatically at given periods. Then I thought that might lead to stuff being committed that I didn't want (for example, if I generated a large amount of data that slipped through my ignore file), and having a repository log full of static messages. So instead, I decided to go with a notification approach instead.
Finding uncommitted changes was pretty straight forward with the following command:
(I took the Library repository out of the results because I didn't want to be bugged about those changes throughout the day; instead, I just take a snapshot of that at the end of each day).
This has the added benefit that if I add more Mercurial repositories in the future, they'll automatically be included in the notification without me having to change anything.
Next up, notification. I find Growl on the Mac to be useful for receiving reasonably unobtrusive notifications throughout the day. It ships with a command-line interface (that you need to install separately), which was perfect for what I needed:
$CHANGES_FILE holds the output of the previous find command. I wanted it stored so I could also copy the changes to my desktop, giving me a secondary visual cue that I have changes outstanding in case I missed the Growl notification.
I ended up with the following bit of script (after adding a few finishing touches, such as using a temp file for the intermediate results, only notifying if the size of the changes file is greater than zero, and displaying the notification with a nice Mercurial icon):
And here it is in action:
One thing to note is that the output of growlnotify is being ignored. This is because of a known issue of invoking growlnotify from cron, whereby the notification still works but the following warning is mailed to you:
I tried using the network settings described here, but this resulted in my notifications not showing up at all. For now, I'm just ignoring the output of growlnotify. I figure if something else goes wrong with it, it'll most likely result in notifications not showing up, which I think I will notice once I start getting used to them.
One final thing to mention is that I had intermittent trouble with using the --image flag when explicitly running the script. It sounds related to this problem, in that the notification would only be displayed intermittently but no other error or warning was emitted. Interestingly, I don't seem to get the problem when the script runs from cron, so I've left the flag in for now.
So there you go; notifications for uncommitted changes under my home directory, and a great reminder for me to go back and look at whether I want to keep configuration changes that are usually made when my mind is on other things. I've currently got it set to run every 15 minutes, which seems to be a reasonable balance between being too annoying and having too many outstanding changes.
Most of my work-related directories have their own Subversion repositories attached to them, but there were also some scratch projects that weren't under any VCS. My config files (eg, .bashrc) were sym-linked to a separate directory that had its own repository.
The main problems I had with the different repositories is that I didn't have an easy way to determine if I had any outstanding changes that need to be committed (my documents directory was shockingly out of date), and there were things that I wanted versioned (for example, application preferences) that weren't being versioned.
My first thought was to add a single Mercurial repository at the root of my home directory so I only had one repository to remember, and could easily track anything that gets added over time (for example, config files for new programs). Once I started thinking about all of the entries in my ignore file though, and how I wanted to share some stuff between work and home but not others, this option became less and less appealing.
So, back to the multiple repositories it was. What I ended up was repositories for the following directories:
~/Documents
~/Library
~/home
~/projects
The ~/Library directory is the standard directory on OSX for storing application support files, preferences, etc, and while I'm not particularly interested in the day-to-day happenings of this directory, it is helpful to have snap-shots that I can revert to at a later date.
I'm using the home repository to track utility scripts and dot files that live directly under ~ (mostly run config and history files). Almost all dot files in my ~ directory are sym-linked to somewhere in my home repository, and I have a bootstrap script to take care of this if I need to setup another home directory on a different machine in the future.
Finding out how to merge my old scripts and config repositories (whilst retaining their histories) was not a simple exercise. I thought I might be able to use Mercurial's import and export commands to merge one into the other, but there doesn't appear to be a way of specifying a relative path to import a repository at, so when I tried, I ended up with everything in the root of the repository. In hindsight, I realise I could have created a new repository and imported both repositories there, making sure to move the contents straight after importing, but there was another similar solution that I eventually found and got to work.
Now that I was happy with my repositories layout, I wanted a way to make sure they were kept up to date. At first I thought about adding a cron job that adds, removes, and commits changes automatically at given periods. Then I thought that might lead to stuff being committed that I didn't want (for example, if I generated a large amount of data that slipped through my ignore file), and having a repository log full of static messages. So instead, I decided to go with a notification approach instead.
Finding uncommitted changes was pretty straight forward with the following command:
find . -name .hg -not -path "./Library/.hg" -exec /opt/local/bin/hg st {}/.. \;
(I took the Library repository out of the results because I didn't want to be bugged about those changes throughout the day; instead, I just take a snapshot of that at the end of each day).
This has the added benefit that if I add more Mercurial repositories in the future, they'll automatically be included in the notification without me having to change anything.
Next up, notification. I find Growl on the Mac to be useful for receiving reasonably unobtrusive notifications throughout the day. It ships with a command-line interface (that you need to install separately), which was perfect for what I needed:
/usr/local/bin/growlnotify "Uncommitted Changes" <$CHANGES_FILE
$CHANGES_FILE holds the output of the previous find command. I wanted it stored so I could also copy the changes to my desktop, giving me a secondary visual cue that I have changes outstanding in case I missed the Growl notification.
I ended up with the following bit of script (after adding a few finishing touches, such as using a temp file for the intermediate results, only notifying if the size of the changes file is greater than zero, and displaying the notification with a nice Mercurial icon):
TMP_FILE=`mktemp /tmp/uncommitted_changes.XXXXXX` || exit 1
cd $HOME
find . -name .hg -not -path "./Library/.hg" \
-exec /opt/local/bin/hg st {}/.. \; >$TMP_FILE
if [[ -s $TMP_FILE ]]; then
CHANGES_FILE=$HOME/Desktop/uncommitted_changes.txt
mv $TMP_FILE $CHANGES_FILE
/usr/local/bin/growlnotify \
--image $HOME/bin/img/hg-icon-50.png \
"Uncommitted Changes" <$CHANGES_FILE > /dev/null 2>&1
else
rm $TMP_FILE
fi
And here it is in action:
One thing to note is that the output of growlnotify is being ignored. This is because of a known issue of invoking growlnotify from cron, whereby the notification still works but the following warning is mailed to you:
growlnotify[1592:10b] could not find local GrowlApplicationBridgePathway, falling back to NSDNC
I tried using the network settings described here, but this resulted in my notifications not showing up at all. For now, I'm just ignoring the output of growlnotify. I figure if something else goes wrong with it, it'll most likely result in notifications not showing up, which I think I will notice once I start getting used to them.
One final thing to mention is that I had intermittent trouble with using the --image flag when explicitly running the script. It sounds related to this problem, in that the notification would only be displayed intermittently but no other error or warning was emitted. Interestingly, I don't seem to get the problem when the script runs from cron, so I've left the flag in for now.
So there you go; notifications for uncommitted changes under my home directory, and a great reminder for me to go back and look at whether I want to keep configuration changes that are usually made when my mind is on other things. I've currently got it set to run every 15 minutes, which seems to be a reasonable balance between being too annoying and having too many outstanding changes.
Friday, October 3, 2008
Replacement in Bash parameter expansions
I have a few utility scripts to make working on the command line easier. One of them is a shortcut for temporarily getting a file or directory out of the way, by renaming it with a .bak suffix. So instead of typing
The script to do this was this simple one-liner
Unfortunately, due to using tab-completion for file names, I'd often get the following error
Conveniently, this is easily achieved using a feature of Bash parameter expansions:
A quick change to my script and all is well:
Incidentally, calling it with the wrong directory (something like sudo bk /usr/lib) can be a bit disastrous, but that's a different story.
I can typemv tmp/foo.txt tmp/foo.txt.bak
and achieve the same thing.
bk tmp/foo.txt
The script to do this was this simple one-liner
mv $1 "$1.bak"
Unfortunately, due to using tab-completion for file names, I'd often get the following error
which is caused by the trailing slash that is included when you tab-complete the directory name. So, ideally, what I wanted to do was check if there was a trailing slash and remove it for the second argument to mv.bk tmp/bar/
mv: rename tmp/bar/ to tmp/bar/.bak: Invalid argument
Conveniently, this is easily achieved using a feature of Bash parameter expansions:
${parameter/pattern/string}
The pattern is expanded to produce a pattern just as in filename expansion. Parameter is expanded and the longest match of pattern against its value is replaced with string. If pattern begins with ‘/’, all matches of pattern are replaced with string. Normally only the first match is replaced. If pattern begins with ‘#’, it must match at the beginning of the expanded value of parameter. If pattern begins with ‘%’, it must match at the end of the expanded value of parameter. If string is null, matches of pattern are deleted and the / following pattern may be omitted. If parameter is ‘@’ or ‘*’, the substitution operation is applied to each positional parameter in turn, and the expansion is the resultant list. If parameter is an array variable subscripted with ‘@’ or ‘*’, the substitution operation is applied to each member of the array in turn, and the expansion is the resultant list.
A quick change to my script and all is well:
mv $1 "${1/%\//}.bak"Incidentally, calling it with the wrong directory (something like sudo bk /usr/lib) can be a bit disastrous, but that's a different story.
Friday, September 19, 2008
Updating MacPorts
Hopefully the last time I forget this:
I've found that this is the most common fix for a failed port install. For example, today, I had the following error when trying to upgrade Subversion:
Sometimes a port clean is also required.
sudo port selfupdate
I've found that this is the most common fix for a failed port install. For example, today, I had the following error when trying to upgrade Subversion:
> sudo port upgrade subversion
---> Building sqlite3 with target all
Error: Target org.macports.build returned: shell command " cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_databases_sqlite3/work/sqlite-3.6.1" && gnumake all " returned error 1
Command output: sh: line 0: cd: /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_ports_databases_sqlite3/work/sqlite-3.6.1: No such file or directory
Sometimes a port clean is also required.
Thursday, September 18, 2008
Fix for "access denied" problem when submitting background file upload
Since Scoodi started to rise in popularity, we've been getting feedback from some Internet Explorer 6 users saying that their photos weren't being uploaded during the process of creating an item. Scoodi uses a common browser hack feature to achieve uploading files in the "background", similar to adding attachments in GMail, so you can continue to fill in the form while photos are uploading. The trick involves setting the target of the file upload form to a hidden frame, so that when the form is submitted, the result is sent to the hidden frame instead of having whole page re-displayed with the result.
Luckily, we were able to reproduce the problem on one of our test machines (though not on another, which made things more frustrating interesting). The difference ended up being in the security settings, specifically the "allow sub-frames to navigate across different domains" setting.
To make a long story short, our issue was to do with how we were setting the src attribute on the hidden iframe. When we enabled Asset Hosts, our source attribute was an absolute path to one of the asset hosts:
Going back to the security setting, in our case, it was saying you can't submit a form served from www.scoodi.com to a target frame served from asset2.scoodi.com. When the setting was enforced, it raised an "access denied" error from the line of JavaScript that submits the form.
Once we worked out the problem, the fix was simple: make sure the iframe src attribute uses the same domain as the page source. In our case, it was as easy as using a path relative to the server:
It was no coincidence that the bug appeared with the increasing number of users to the site, though it wasn't because we were getting a wider range of browsers and configuration permutations. We enabled Asset Hosts because of the increase in users, and when we did that, those IE6 browsers with higher security settings stopped working.
Luckily, we were able to reproduce the problem on one of our test machines (though not on another, which made things more frustrating interesting). The difference ended up being in the security settings, specifically the "allow sub-frames to navigate across different domains" setting.
To make a long story short, our issue was to do with how we were setting the src attribute on the hidden iframe. When we enabled Asset Hosts, our source attribute was an absolute path to one of the asset hosts:
<iframe ... src="http://asset2.scoodi.com/path/blank.html">
Going back to the security setting, in our case, it was saying you can't submit a form served from www.scoodi.com to a target frame served from asset2.scoodi.com. When the setting was enforced, it raised an "access denied" error from the line of JavaScript that submits the form.
Once we worked out the problem, the fix was simple: make sure the iframe src attribute uses the same domain as the page source. In our case, it was as easy as using a path relative to the server:
<iframe ... src="/path/blank.html">
It was no coincidence that the bug appeared with the increasing number of users to the site, though it wasn't because we were getting a wider range of browsers and configuration permutations. We enabled Asset Hosts because of the increase in users, and when we did that, those IE6 browsers with higher security settings stopped working.
Monday, August 18, 2008
Scoodi on TV
A couple of weeks ago, I was involved the filming of a "news-fo-tainment" segment where we got to talk about and promote Scoodi, the project I've been involved in for the past few months:
The impact that the segment had on the website traffic was a mixed blessing, and it would probably be fair to say that it caught us a little off guard. We've since fixed one of the main bottle necks, and by all accounts, the site is quite stable once again.
The impact that the segment had on the website traffic was a mixed blessing, and it would probably be fair to say that it caught us a little off guard. We've since fixed one of the main bottle necks, and by all accounts, the site is quite stable once again.
Thursday, August 7, 2008
Installing GHC on OSX
During my quest to better understand Functors & Monads, it was suggested that I take a detour and learn Haskell first, as I was struggling a bit with Scala's type annotations. So, off to MacPorts I went:
Half an hour later, I was still waiting for it to build. Something was not quite right. I killed the process and tried again (after issuing a
Same problem - it got up to building, then seemed to hang. Looking at
I gave it about 45 minutes before killing it again. Brad has been down this path before, and he put me on to the pre-packaged installer available from the http://haskell.org/ghc download page. As per the instructions, I installed XCode off the Leopard disc, then ran the installer, and hey presto, I was
~> sudo port install ghc
---> Fetching ghc
---> Verifying checksum(s) for ghc
---> Extracting ghc
---> Applying patches to ghc
---> Configuring ghc
---> Building ghc with target all
Half an hour later, I was still waiting for it to build. Something was not quite right. I killed the process and tried again (after issuing a
sudo port clean ghc, because it seemed to be in an inconsistent state after killing it). Same problem - it got up to building, then seemed to hang. Looking at
top, I could see ghc-6.8.2 running intermittently. After a while, ghc-6.8.3 was also added to the list of processes (running intermittently), but still no progress on the build.I gave it about 45 minutes before killing it again. Brad has been down this path before, and he put me on to the pre-packaged installer available from the http://haskell.org/ghc download page. As per the instructions, I installed XCode off the Leopard disc, then ran the installer, and hey presto, I was
ghc hello_world.hs'ing it up!Monday, August 4, 2008
Installing/upgrading Scala
Because I keep forgetting:
This cryptic message when installing the docs indicates lack of privileges, ie, not running with sudo:
cd /opt
sudo tar xzf ~/install/scala-2.7.1.final.tgz
sudo rm scala
sudo ln -s scala-2.7.1.final scala
cd scala
sudo bin/sbaz install scala-devel-docs
This cryptic message when installing the docs indicates lack of privileges, ie, not running with sudo:
planning to install: scala-devel-docs/2.7.1.final
Installing...
java.io.FileNotFoundException: /opt/scala-2.7.1.final/meta/cache/scala-devel-docs-2.7.1.final.zip.tmp (No such file or directory)
Monday, June 30, 2008
Splitting up large files
This is one of those things that I need to do occasionally, but not often enough that I've been able to commit it to memory.
This will split a the large_file file into 4Gb files with filenames
To get the parts back together:
split -b 4000m large_file large_file.
This will split a the large_file file into 4Gb files with filenames
large_file.aa, large_file.ab, etc. Before doing this, it really helps to make sure you've got enough space for the new files - ie, at least as much free space as the file you are splitting!To get the parts back together:
cat large_file.* > large_file
Tuesday, June 17, 2008
Solving The Tangled Working Copy Problem
I just read this article, which goes into great (as in good) detail about one way to solve the problem where you have more than one conceptual change in your work-space, and you want to physically separate them into different groups.
When I've encountered this problem previously, I've generally just backed up the working copy, reverted the unwanted changes in the working copy, submitted it, then pulled back the additional changes form the backup. But the way described in the article is a much better way!
Apparently it is supported in Mercurial, so I'll have to give it a go next time. Unfortunately, it doesn't help when I'm working with Subversion, as is currently the case at work. However, the article also mentions a way of do a similar thing using the patch program, which is still better than my manual way around it.
When I've encountered this problem previously, I've generally just backed up the working copy, reverted the unwanted changes in the working copy, submitted it, then pulled back the additional changes form the backup. But the way described in the article is a much better way!
Apparently it is supported in Mercurial, so I'll have to give it a go next time. Unfortunately, it doesn't help when I'm working with Subversion, as is currently the case at work. However, the article also mentions a way of do a similar thing using the patch program, which is still better than my manual way around it.
Subscribe to:
Posts (Atom)