Helpful online tools

These are a bunch of online tools which I find indispensable when I need to do some quick validation, encoding, or formatting/indenting.

1. XML escaping - http://escapehtmlforxml.com/
   
2. XML indenting (critical when trying to read ugly XML) - http://xmlindent.com/
3. JSlint (javascript code checker) - http://www.jslint.com/
4. JSONlint (JSON validator and formatter) - http://www.jsonlint.com/
5. W3C HTML validator - http://validator.w3.org/
6. W3C CSS validator - http://jigsaw.w3.org/css-validator/
7. WDG tools (web validators) - http://htmlhelp.com/tools/
8. XHTML/CSS page validator - http://xhtml-css.com/
9. shell tools (xml, base64, md5/sha1, and more) - http://www.shell-tools.net/

Hope this list of tools is helpful!

Easy license headers with maven

If you are like me you probably hate trying to maintain license headers on your source code files. It has to be done for pretty much all of my projects (since I deal in open source 99% of the time) but it is pure drudgery. I found a great plugin for maven 2 which makes this a piece of cake (very easy). The maven-license-plugin can (optionally) check your source files for headers (you control which ones or just use the defaults) and add in or replace the headers for you. Forget about doing this manually anymore; those days are over. You just specify a license header template like this (i.e. create a file, I use LICENSE_HEADER):

Copyright (C) ${year} ${holder} <${contact}>

This file is part of ${name}.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

 http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

Then add something like this to your project pom.xml (in the build section under plugins):

<plugin>
<groupId>com.google.code.maven-license-plugin</groupId>
<artifactId>maven-license-plugin</artifactId>
<configuration>
   <header>${basedir}/LICENSE_HEADER</header>
   <excludes>
       <exclude>target/**</exclude>
       <exclude>m2-target/**</exclude>
       <exclude>**/*.properties</exclude>
   </excludes>
   <properties>
       <name>${project.name}</name>
       <year>${project.inceptionYear}</year>
       <holder>Aaron Zeckoski</holder>
       <contact>azeckoski@gmail.com</contact>
   </properties>
   <encoding>UTF-8</encoding>
</configuration>
<executions>
   <execution>
       <goals>
           <goal>check</goal>
       </goals>
   </execution>
</executions>
</plugin>

You need to add in the plugin repo in the pluginRepositories section:

<pluginRepository>
<id>mc-release</id>
<url>http://mc-repo.googlecode.com/svn/maven2/releases</url>
</pluginRepository>

That config will cause the check to run on every build (ignoring properties files is a good idea since the plugin has trouble with them). Files with a missing license header will cause the build to fail ensuring you remember to run the command to format them. The properties you set there will fill in the ${field} vars in the license header template.

Now run the maven command to check for license headers:

mvn license:check

or simply do a build (which will also run the check):

mvn clean install

You should get a report about the files missing license headers.
Run this command and all the license headers will be added or updated to match your template:

mvn license:format

One final note, you can remove all the license headers using "mvn license:remove". Very cool.

Keys to High Performance Web Applications

I know web application performance has been discussed about 100 times before, but it bears repeating (if only briefly and mostly with links) since it is such an important topic.

If you have never tried to ensure your web application will run well then my rule #1 is to not change your application architecture. Program performance is a separate issue that rarely is a problem compared to network latency and server request overhead. I am not saying it is never a problem but you should try things that are much easier first before diving into a restructuring or a rewrite of your app (in most cases buying more hardware is cheaper and safer). As Donald Knuth says, "Premature optimization is the root of all evil (or at least most of it) in programming"

Now that you have done nothing to start (so far so good right?) it is time to do something. Get the YSlow analyzer for Firebug and run it against your web application. It will provide you with a report which you can use to identify possible performance issue. The Firebug network monitor and to a lesser extent the Safari Web Inspector are also good tools for profiling the requests on a page.
Here is a list of a few more performance apps from the RazorSpeed blog and around the web:

• HttpWatch - browser plugin for IE and Firfox for profiling page loads
  
• PageSpeed - browser plugin for Firefox for profiling page loads
• Hammerhead - an addon for Firebug for profiling page loads
• RockStar Optimizer - an eclipse plugin (update site) for optimizing HTML/CSS/JS/IMG
• Rockstar Profiler - a robust profiler application which uses proxies

No discussion of web app performance would be complete without including a link to Steve Souders' blog. While you are there check out compare. Some of the results are surprising and others not so much.

Many tuning option are in the hands of your system administrator so if that is not you then you can relax a little bit. However, as a web application developer (frontend/web developer or backend engineer), you should at least know where the common problems lie and this is where the bible of web application performance (Yahoo performance rules) comes in. It is a list of best practices which can be roughly summarized as reduce requests, spread the load, utilize caching and compression, and structure pages for efficiency. If you want the shorter list then check out 14 Rules for Faster-Loading Web Sites (it is just a list of rules taken from the bible with samples). If you prefer an alternative list then try the PageSpeed rules.

If you are lucky you are using an environment that has performance tuning built in (like the grails ui performance plugin or the RockStarApps eclipse/aptana plugin) which will do most of what the performance rules suggest automatically (what can be done in the app anyway). Most web servers provide support for compression so that usually is best handled at that layer anyway. For the rest of the best practices, you will just have to learn and apply the performance rules best practices. In most cases it will be well worth your time.

JavaForge requires authn to access SVN

I went to setup an account on JavaForge for Steeple today. Everything went pretty smoothly with the initial setup. It was easy to create an account and setup a new project. The site allows for fine-grained permissions which are easy to configure and has a very nice wiki. It also included code analysis and build tools (which are why I decided to try it out in the first place).

I hit the first bump after creating the SVN repository. I could not find the URL to the respository anywhere. After searching around for ahile I figured out that the URL was:
http://svn.javaforge.com/svn/steeple

The next issue, which ended up being insurmountable, was related to access to the SVN respository. Try as I might there was no way to allow public access to it. Anyone trying to access the public URL will receive a basicauth challenge. Just to view the respository a user has to enter in the username of "anonymous" with a password of "anon". As a result I had to drop javaforge and go with my backup of google code for now.

I did post a question on the javaforge forums about this but from reading the other forum messages I think it is just not possible.

My first week with Groovy and Grails

I spent time over the last week learning about Groovy (a dynamic language for the JVM) and Grails (a code by convention web application framework built on Groovy) so I thought I would write up my impressions and some of the fun things I learned.
So you have a sense of where I am coming from, I am a long time Web applications and Java/PHP/Javascript/Perl developer. I am somewhat newer to Python and Ruby but I prefer Python. I am a REST and Open Source advocate when I am in the right mood.

If you are totally unfamiliar with Groovy then I recommend you take a look at this post as it lays out the reasons why you might want to learn more about it:
http://codetojoy.blogspot.com/2009/06/case-for-groovy.html

If you know you are going to be writing web-apps then just skip Groovy and go straight for Grails. If you are looking to do some JSR-223 (Java Scripting) stuff (with Groovy) then Groovy is the place to focus on. Either way, you will need to get familiar with the basics of Groovy so look at these:
http://groovy.codehaus.org/Quick+Start
http://groovy.codehaus.org/Collections
Feel free to checkout some sample Groovy scripts I made which illustrate many of the key concepts.

The Getting Started Guide for Groovy is huge and not really a very good place to try to get started unfortunately. That said, the Beginners Tutorial is pretty good, especially the section on closures.

If you want to get going with Grails then it is a little bit easier since it mostly builds on Groovy. Grails borrows heavily from Ruby on Rails so if you are familiar with it then things will come to you quickly. This is the best place to start (not surprisingly):
http://www.grails.org/Quick+Start
I really liked the screencasts (which are oddly located here also). They provided a nice introduction to Grails without much effort. When you are ready for a little more the tutorials are a good next step.

Things I learned in no particular order:

• Maven and Grails do not get along - There is some really weak maven integration available but it does not work very well. The structure of grails (e.g. src/groovy) does not match the maven standard structure (e.g. src/main/groovy). Mostly maven just allows you to run the grails build commands which is easier to do with grails itself. I struggled with this for awhile before just giving up on using maven. The Grails team recommends using Ivy if you want to add dependency management (or Grails plugins which are preferred if they are available).
  The grails and groovy artifacts are available in maven repositories which is nice.
  
• Grails and Eclipse don't easily integrate - The Groovy plugin is pretty good (not great) but the build integration is pretty poor and requires you to jump through hoops. It seems like the integration with IDEA is a lot better and recommended by the Grails team.
• Groovy supports closures - The closure support in groovy is great and very easy to use. I found myself writing closures like crazy (even more than in Javascript) and it made the code very clean.
  NOTE: I ran across one weird bug where passing in a String[] to a closure causes it to be misinterpreted as a collection of separate arguments for each array entry. There are hacks to get around this but be aware that it may bite you.
• Grails has a great plugin system - Grails has a pretty powerful plugin system which allows easy extension of a grails app. The plugins seems to be very easy to install and fairly easy to write. There is a complete guide if you are interested in developing your own plugins.
  
• Grails app creation puts in too much stuff - The structure generated by grails create-app has a lot of stuff in it which you will probably want to cleanup (like the hibernate plugin by default for example). There is no uninstall for plugins so just remove the dir of the plugin to get rid of it. Be careful to not leave in a lot of things you are not going to use and clean out the sample stuff under web-app as well.
  
• Grails convention is not very flexible - Grails prides itself on "code by convention" and "convention over configuration" and it does a good job of establishing a lot of conventions. It takes a little while to get used to them but if you follow them then things are pretty easy. Unfortunately, this implies that it is possible to override the convention using configuration if needed and in many cases it is not. I have been bitten a few times already when I tried to do things that are not "on the rail".
  
• Grails uses prototype.js by default - The built in javascript engine in Grails is the portal/multi-framework unfriendly prototype.js. I can't use it so I am playing around with using jQuery instead (so far this is proving to be manageable). There is a jQuery plugin which helps make this easier.
  

I am still getitng used to things but I am not sure what Groovy/Grails gains me over using Jython/. It seems like Jython does everything Groovy does plus it has the massive Python community for support.
As far as scripting languages go I think I prefer PHP, Javascript, Python, and Perl (in that order) over Groovy but this may just due to a lack of familiarity on my part.

Sakai AppBuilder Plugin updated to 0.8.7

The Sakai AppBuilder Eclipse Plugin is updated to a new version (0.8.80.8.7) which includes updates for Sakai K1 and support for Wicket. Many thanks to Steve Swinsburg who did all the heavy lifting on this update. You can install the plugin using instructions here or update it to the new version from within eclipse if you have installed it before.

The Sakai AppBuilder is a RAD tool that allows you to quickly create Sakai webapp projects in Eclipse that will work in the Sakai Framework. Use these as a basis for the projects that you want to make without all the busy work of creating the structures and adding in all the dependencies. You can choose various UI layer options and implementation types to get you started quickly.

NOTE: Updated for the 0.8.8 release (minor fix from 0.8.7)

Aptana Studio 1.2 crash and upgrading to 1.3

I recently upgraded Java to version 1.6 (build 1.6.0_13-b03-211) on my macbook pro running OSX 10.5.7 (leopard). It was a bit of a chore but it mostly sped things up and allowed me to run some of the newer apps that require Java 6.

I had a major casualty though when Aptana Studio stopped working. It would simply crash without even giving a decent error message and the logs were not helpful.I normally run the standalone version of Aptana Studio (1.2.7) which is built on eclipse 3.2. This seems to no longer run on OSX and Java 6 so I went in search of a fix. After lots of forum browsing, tweaking configurations, and reinstalling I ended up retiring version 1.2 and trying out version 1.3 (still in beta). It was hard to find the 1.3 downloads page so here is a link.

It installs quite easily and ran which was a major improvement. However, when I tried to install the features (plugins) I am used to, they all indicated that they were incompatible and would not install. This seemingly hopeless situation was actually easily fixed by updating Aptana Studio (Help -> Software Updates). Once that was done (definitely restart here) I installed the features (php, pydev, git) and plugins (epic) that I like and everything seems to work fine again.

Hopefully this will save someone a little pain.

PHP dash in class and method names

I ran into what seems like a common issue when working with PHP and SimpleXML today. Parsing XML is normally pretty easy:

<?php
header('Content-type: text/plain');

$xmlData = <<<XML
<?xml version='1.0'?>
<trees>
 <fruit>
   <apple name='apple' type='Deciduous' has-fruit='Y' />
   <pear name='pear' type='Deciduous' has-fruit='Y' />
 </fruit>
 <pine>
   <white name='whitepine' type='Coniferous' has-fruit='N' />
 </pine>
</trees>
XML;

$xml = simplexml_load_string($xmlData);

echo "Testing SimpleXml";
echo "\n".$xmlData;
echo "\nName:".$xml->fruit->apple->getName();
echo " Type:".$xml->fruit->apple->attributes()->type;
?>

Output:
Name:apple Type:Deciduous

However, if you decide to include a hyphen or a dash in the name of your attribute things get a bit more interesting. The code has to be adjusted since the name of a class method cannot contain "-". To make it work, the attribute name has to include braces and single quotes (e.g. "{'name'}").

echo "\n".$xmlData;
echo "\nName:".$xml->fruit->apple->getName();
echo " Type:".$xml->fruit->apple->attributes()->type;
echo "\nFruit?:".$xml->fruit->apple->attributes()->{'has-fruit'};

Output:
Name:apple Type:Deciduous
Fruit?:Y

Tricky SOLR schema issue with StrField

I have been setting up SOLR (version 1.3) as a search index for the Darwin Correspondence project. While making a few changes I ran into a really annoying issue today related to the way the schema configuration works. The SOLR schema (schema.xml) allows you to setup Analyzers and Filters which allow control of how terms are indexed and searches are executed.

I needed to make it so we could match names when the case is not exact and when the chars are special (i.e. "u" needs to match a name with "ü"). The field started out like this:

<fieldType name="name" class="solr.StrField" sortMissingLast="true" omitNorms="true" compressed="false" indexed="true" stored="true">

For my first attempt I added an analyzer to the field like so:

<fieldType name="name" class="solr.StrField" sortMissingLast="true" omitNorms="true" compressed="false" indexed="true" stored="true">
<analyzer type="index">
  <tokenizer class="solr.HTMLStripStandardTokenizerFactory"/>
  <filter class="solr.StandardFilterFactory"/>
  <filter class="solr.ISOLatin1AccentFilterFactory"/>
  <filter class="solr.LowerCaseFilterFactory"/>
  <filter class="solr.TrimFilterFactory" />
</analyzer>
<analyzer type="query">
  <tokenizer class="solr.StandardTokenizerFactory"/>
  <filter class="solr.LowerCaseFilterFactory"/>
  <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>
</fieldType>

I loaded data into SOLR and tried out some searches and go no results. I was getting exact matches only (as if I had no analyzers). When I checked the solr admin analysis page it indicated that the filters were working and the tests there even seemed to show that things were ok. Unfortuantely, I found out that SOLR does not actually execute the analyzers if the field class is set to solr.StrField. It doesn't fail or indicate errors in the logs but your searches will not work the way you expect them to. Changing the field over to class solr.TextField fixed the problem.
The correct configuration for the field is this:

<fieldType name="name" class="solr.TextField" sortMissingLast="true" omitNorms="true" compressed="false" indexed="true" stored="true">
<analyzer type="index">
  <tokenizer class="solr.HTMLStripStandardTokenizerFactory"/>
  <filter class="solr.StandardFilterFactory"/>
  <filter class="solr.ISOLatin1AccentFilterFactory"/>
  <filter class="solr.LowerCaseFilterFactory"/>
  <filter class="solr.TrimFilterFactory" />
</analyzer>
<analyzer type="query">
  <tokenizer class="solr.StandardTokenizerFactory"/>
  <filter class="solr.LowerCaseFilterFactory"/>
  <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>
</fieldType>

I spent a few hours figuring this out so I hope that this saves someone a little time.

PHP CodeSniffer tips

I really love tools which help my code be more correct and more readable. I have referred to tools like JSLint (JS) and FindBugs (Java) in previous posts and now I am going to write some tips about using PHP CodeSniffer (PHP) (a.k.a. phpcs). It is probably the most aggressive of the three and can be especially tricky on the requirements it puts on your file headers.

Here is a sample file header:

<?php
/**
* Presto - a lightweight REST framework for PHP
*
* Presto is a simple to use and very lightweight REST framework for PHP,
* it will help you to handle rest routing and input/output of data without
* getting in your way
*
* PHP Version 5
*
* LICENSE:
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
* @category  File
* @package   Presto
* @author    Aaron Zeckoski <azeckoski@vt.edu>
* @copyright 2009 Aaron Zeckoski
* @license   http://www.apache.org/licenses/LICENSE-2.0 Apache License, Version 2.0
* @version   SVN: $Id:$
* @link      https://link/to/your/project/site
* @since     inception
*/

A few comments about the header:

• The copyright cannot have a comma after the year but the year can be a range. "2002-2009 AZ" is ok, "2009, AZ" is not
• The license should appear inside the header like shown, not above it (this would cause an ERROR in phpcs)
• The alignment of the data after the tags (e.g. @license, @version) is not optional, misaligned data causes an ERROR

Sample class:

/**
* My class which does some stuff
*
* @category Class
* @package  Presto
* @author   Aaron Zeckoski <azeckoski@vt.edu>
* @license  http://www.apache.org/licenses/LICENSE-2.0 Apache License, Version 2.0
* @link     https://link/to/the/project/again
*/
class RestController
{
 const DEFAULT_RESOURCES_DIR = 'resources';

 /**
  * This is a method in my class
  *
  * @param object $_resourcesPath [optional] the resource path
  *
  * @return void
  */
 protected function loadResources($_resourcesPath = self::DEFAULT_RESOURCES_DIR)
 {

There are also a few of the rules about classes that caught me out as well:

• Class methods must use camelCase. myMethodName is good, my_method_name is not
• Classes MUST have a comment on them and it has to include a lot of the fields from the header. The ones I list in the sample above are the minimum (seriously).
• The space between @params and @return is not optional
  
• Just a note on constants in PHP, you use const inside classes and define outside

Take a look at the sample file and class headers here for more details: http://pear.php.net/manual/en/standards.sample.php