MSBuild, YuiCompressor and making CSS and Javascript titchy

Both Google (via PageSpeed) and Yahoo (via YSlow and their Developer Network guidelines) (among many others) tell us that how quickly our page loads and how optimised the site is for fast download is important – Google announced that the speed your page loads is important, and Yahoo have a number of guidelines highlighting the same thing.

I’ve done a lot of work in other areas (distributed caching to avoid hitting the DB, output caching of pages to avoid any unecessary parsing, etc.) but I thought I’d start to focus on the performance on the web side of things.

After almost 2 hours googling and a final chat on twitter, I thought I’d have a look at YUICompressor first.  I quite like the ‘being part of the build’ nature of it all, and it fits in with my desire to learn a little more about MSBuild to help in the ongoing mission to ensure we have a good Continuous Integration process within the workplace.

The documentation for YUI isn’t bad so long as you want to piggyback onto the Pre and Post-build events (found from the properties window in your web project/build tab), but this has never really felt clean to me, so I thought I’d piggyback onto the ‘AfterBuild’ target within the .csproj file instead.

Here’s what I did.

Download and Setup YUICompressor

YUICompressor is available from here, and comes as a zip with a couple of DLLs in it.

In order to tie in with the CI process, I keep a ‘lib’ folder within my solution folder for our project for all external dependencies and these get checked into source control along with your solution – one of the early goals of CI is repeatability, and including (and referencing) local resources allows you to build the project on any clean machine.

Tie into your MSBuild file for your project

We use a ‘Web.Resources’ project, which acts as a pseudo-cdn, so all static resources (scripts, css, images, flash) go into this and keep the core web solution a little cleaner.  It’s another task that assists in speeding up your site too as some older browsers have a limit of 2 concurrent requests per domain – splitting your static resources into another domain (even a sub domain) increases the concurrency of downloads and hence speeds up load times.

In visual studio, right click on your project with the CSS/Javascript in, unload project, and then right click and ‘edit’.  You’ll be presented with the .csproj file (which as anyone reading this will already know is also an MSBuild file).

Towards the end of the file, you will see a section like this:

<!-- To modify your build process, add your task inside one of the targets below and uncomment it. 
  
     Other similar extension points exist, see Microsoft.Common.targets.
<Target Name="BeforeBuild">
</Target>
<Target Name="AfterBuild">
</Target>
-->

We’re going to move the ‘Afterbuild’ target outside of the comment, and replace it with the following:

  <UsingTask TaskName="CompressorTask" AssemblyFile="../lib/Yahoo.Yui.Compressor/Yahoo.Yui.Compressor.dll" />
  
  <Target Name="AfterBuild">
    <PropertyGroup>
      <MainSiteCssOutputFile Condition=" '$(MainSiteCssOutputFile)'=='' ">tombola/css/tombola.compiled.css</MainSiteCssOutputFile>
      <MicroSiteCssOutputFile Condition=" '$(MicroSiteCssOutputFile)'=='' ">cinco/css/cinco.compiled.css</MicroSiteCssOutputFile>
      <!--<JavaScriptOutputFile Condition=" '$(JavaScriptOutputFile)'=='' ">JavaScriptFinal.js</JavaScriptOutputFile>-->
    </PropertyGroup>
    <ItemGroup>
      <!-- Single files, listed in order of dependency -->
      <MainSiteCssFiles Include="$(SourceLocation)css/core/reset.css" />
      <MainSiteCssFiles Include="$(SourceLocation)css/core/site.css" />
      <MainSiteCssFiles Include="$(SourceLocation)tombola/css/site.css" />
      <MicroSiteCssFiles Include="$(SourceLocation)css/core/reset.css" />
      <MicroSiteCssFiles Include="$(SourceLocation)css/core/site.css" />
      <MicroSiteCssFiles Include="$(SourceLocation)cinco/css/site.css" />
      <!--<JavaScriptFiles Include="$(SourceLocation)jquery-1.3.2.js"/>-->
      <!-- All the files. They will be handled (I assume) in alphabetically. -->
      <!-- <CssFiles Include="$(SourceLocation)*.css" />
            <JavaScriptFiles Include="$(SourceLocation)*.js" />
            -->
      <!--
          JavaScriptFiles="@(JavaScriptFiles)"
          JavaScriptOutputFile="$(JavaScriptOutputFile)"
          ObfuscateJavaScript="True"
          DeleteJavaScriptFiles="false"
-->
    </ItemGroup>
    <CompressorTask CssFiles="@(MainSiteCssFiles)" DeleteCssFiles="false" CssOutputFile="$(MainSiteCssOutputFile)" CssCompressionType="YuiStockCompression" PreserveAllSemicolons="True" DisableOptimizations="Nope" EncodingType="Default" LineBreakPosition="-1" LoggingType="ALittleBit" ThreadCulture="en-gb" IsEvalIgnored="false" />
    <CompressorTask CssFiles="@(MicroSiteCssFiles)" DeleteCssFiles="false" CssOutputFile="$(MicroSiteCssOutputFile)" CssCompressionType="YuiStockCompression" PreserveAllSemicolons="True" DisableOptimizations="Nope" EncodingType="Default" LineBreakPosition="-1" LoggingType="ALittleBit" ThreadCulture="en-gb" IsEvalIgnored="false" />
  </Target>

You’ll see there’s a lot going on there, so lets break it down.  Firstly, you will see I’ve commented out the javascript minification and concatenation – you should (after working through the CSS stuff) be able to come up with your own use case for javascript on your site.

So:

  <UsingTask TaskName="CompressorTask" AssemblyFile="../lib/Yahoo.Yui.Compressor/Yahoo.Yui.Compressor.dll" />

This is basically creating a ‘task’ by pointing to the functionality within an assembly (in this case, our local Yahoo.Yui.Compressor install. In my own head I have this down as a ‘using’ statement for MSBuild (in a similar way to Import, though I know you can do a lot more with the UsingTask msbuild command that I haven’t yet delved into.

  <Target Name="AfterBuild">
  
    <PropertyGroup>
      <MainSiteCssOutputFile Condition=" '$(MainSiteCssOutputFile)'=='' ">mainsite/css/tombola.compiled.css</MainSiteCssOutputFile>
      <MicroSiteCssOutputFile Condition=" '$(MicroSiteCssOutputFile)'=='' ">microsite/css/cinco.compiled.css</MicrositeCssOutputFile>
    </PropertyGroup>

This basically builds up the variables $(MainSiteCssOutputFile) and $(MicroSiteCssOutputFile). I could have just as easily called them $(Bob) and $(Fred), though I’m a fan of self documenting variables ;)

 
    <ItemGroup>
      <!-- Single files, listed in order of dependency -->
      <MainSiteCssFiles Include="$(SourceLocation)css/core/reset.css" />
      <MainSiteCssFiles Include="$(SourceLocation)css/core/site.css" />
      <MainSiteCssFiles Include="$(SourceLocation)tombola/css/site.css" />
      <MicroSiteCssFiles Include="$(SourceLocation)css/core/reset.css" />
      <MicroSiteCssFiles Include="$(SourceLocation)css/core/site.css" />
      <MicroSiteCssFiles Include="$(SourceLocation)cinco/css/site.css" />
    </ItemGroup>

A few arrays of items to be included in the compression

  
    <CompressorTask CssFiles="@(MainSiteCssFiles)" DeleteCssFiles="false" CssOutputFile="$(MainSiteCssOutputFile)" CssCompressionType="YuiStockCompression" PreserveAllSemicolons="True" DisableOptimizations="Nope" EncodingType="Default" LineBreakPosition="-1" LoggingType="ALittleBit" ThreadCulture="en-gb" IsEvalIgnored="false" />
    <CompressorTask CssFiles="@(MicroSiteCssFiles)" DeleteCssFiles="false" CssOutputFile="$(MicroSiteCssOutputFile)" CssCompressionType="YuiStockCompression" PreserveAllSemicolons="True" DisableOptimizations="Nope" EncodingType="Default" LineBreakPosition="-1" LoggingType="ALittleBit" ThreadCulture="en-gb" IsEvalIgnored="false" />

The clever stuff :) This is where the CompressorTask picks up the CSS files to compress/join @(MainSiteCssFiles) and compresses them down into the CssOutputFile $(MainSiteCssoutputFile).

The options on this CompressorTask are numerous, and I’d recommend referring to the main YUICompressor site to get the settings correct for your environment – I’ve stuck with the defaults, but you can increase the amount of compression, delete original files, etc. etc. etc.

What about the Javascript?

I’ve left the javascript parts of the CompressorTask commented out in the main post above, and I’m about to go and play with those now, though it seems like it’ll be pretty much an identical process to that above.

Does it make any difference?

I’ve moved from a 85 rating to an 87 rating on YSlow – wow you say, was it really worth it?  When we know that even an extra second on load speed can significantly affect revenue for companies (god, I wish I could find decent resources to back that up after seeing people show them in talks on this sort of thing) it very much is an ‘Every little helps’ approach.  The jump of 2 was without the concat or join of javascript, so I hope to achieve perhaps another 1 point there too.  From there, smushing of images, perhaps spriting up images that can be, and generally just trying to eek out every last ounce of performance without any additional hardware costs.

What’s next?

After chatting to a few of those ‘clever people™’ that I follow on twitter, in particular @red_square and @stack72, they both recommended Chirpy which looks very interesting, and I really like the idea of .LESS and the concept of variables within the CSS, so I may well look at that next (will need more buyin from the team as we’ll all have to install it, but that’s never been a problem).

At least for now, our build is automated with the optimised versions of our static resources.

TeamCity – Install and Setup (Basics)

Been a while since I posted and I thought that the past few days warranted getting my thoughts down as we’ve just setup our first foray into Continuous Integration/Build Automation with TeamCity.  We’re in the process of rewriting the corporate site from classic asp/asp.net into an MVC2 front end with some solid (though not always SOLID) design behind it.  We’ve written a lot of unit tests (though many more to go), and thought it was about time we looked at the whole CI/Build side of things.  I’d hasten to add, the following post will remain at a fairly basic level, as that’s where I’m at at the moment – hopefully something in here will be useful, though it’s as much about documenting the steps for the team I work with and whenever I write something like this down it always helps solidify it in the grey matter.

Why Continuous Integration/Build Automation?

The answers for us fit pretty much into the standard reasons behind CI – primarily ensuring quality, though easing the deployment burden was certainly a part of it.  CI completes the circle really – you’ve written your quality code, you’ve written your unit tests (and any other tests, integration, UI, etc.), so why not have an easy way to get all of that validated across your whole team, making sure that the quality remains and that you don’t have the manual task of pushing the code out to your dev servers? 

Continuous Integration helps with all of this, and a whole lot more, though the ‘more’ part is something that will come in time for us I think – we now have a working checkin build (I’ll detail the steps I went through) so that at least gives us ongoing feedback.

TeamCity was the immediate choice for us as we don’t really qualify for a TFS setup and CruiseControl.net seemed to have a higher learning curve (I may be mis-representing it here mind).

Before going through the detail of the install, a quick shout out to Paul Stack (@stack72), the fantastic Continuous Integration book from Manning, and the as yet unread MS Build book from Microsoft – these as well as the blog posts from many have helped massively in getting this setup.

Team City 6 – Install

Generally, the defaults in the setup were fine.  I made sure that all options were enabled with regards to the service etc. – I can’t see the use case when you wouldn’t want this, but it’s worth stating.

image

I changed the path to the build server configuration to a more general location – it initially suggested my own windows account user area, though I was unsure (and couldn’t find easy documentation) on whether this would make the configuration available to others, so I defaulted to a different path.

image

With regards to server port (for the TeamCity web administration tool) I changed the default port too.  Although it’s recommended that the build server remains as a single purpose box, I felt uncomfortable going with the default port 80 just in case we ever chose to put a web server on there for any other purpose.

image

I also chose to stick with the default and ran the service under the SYSTEM account – it doesn’t seem to have affected anything adversely and I’d rather do that than have to create a dedicated account.

Team City – Initial Setup

Initially you are asked to create an administrator account – do so, though if you’re in a team of people, there is an option later to create user accounts for each user – far better to do that and leave the admin account separate.  In the free version you can have up to 20 users, so it’s ideal for small teams.

Create a Project

The first steps in linking up your project to TeamCity is to create your project.

image

Here, you can give the project any name and description – it can (though doesn’t have to) match the project name in Visual Studio.

image

TeamCity from this point on holds your hand fairly effectively.

image

oh, ok – thanks Smile <click>

The build configuration page has a lot of options, but some of the pertinent ones (at least early doors – once you have more experience, which I don’t, then the others will certainly come into play).

Name your build – I named ours ‘checkin build’ as I intend for it to happen after every checkin… does what it says on the tin kinda thing.

Build number format – I left this as the default ‘{0}’ – it may be prudent to tie it in later on with the Subversion version number, but for now, we want to get a working CI process.

Artifact Paths – very much steered clear of this at the moment – it seems there’s a lot of power in these, though I haven’t touched on them enough.

Fail Build If – I went with the defaults plus a couple of others – ‘build process exit code is not zero’, ‘at least one test failed’, ‘an error message is logged by the build runner’.

Other than that, I pretty much stuck with the defaults.

Version Control Settings

image

I deliberately selected to checkout to the agent as I suspect this’ll give me more scalability in future – the build server can have multiple build agents on other machines from what I understand (kinda the distributed computing model?) and those agents can handle the load if there are very regular/large team checkins.  I think there are limitations on build agents in the free version, but again – if we use this solidly, and need more, then the commercial license isn’t too badly priced.

I also chose a different checkout directory to the default, just because – no solid reason here other than I have a lot of space on the D: drive.

Our project is significant (24 VS projects at last count, a lot of them testing projects (1 unit and 1 integration per main project), and initially I experimented with ‘clean all files before build’ but the overall build was taking approximately 8mins (delete all, checkout all, build, unit test) – I’m going to try to not clean the files and do a ‘revert’ instead but at present, I don’t have any experience on which is better – certainly cleaning all files worked well, but 8mins seemed a while.

Attach to a VCS Root

The important part – linking up your project to your source control (subversion in our case).

image

Click ‘Create and attach…’  Most of the settings in here are defaults, but you will notice further down the page it defaults to subversion 1.5 – we’re using 1.6, so double check your own setup.

image

I also experimented with enabling ‘revert’ on the agent ala:

image

With an aim to bringing down the overall build time – I haven’t played enough to warrant feedback yet, though I suspect the revert will work better than a full clean and checkout.

Build Steps

The CI build will be broken into a number of steps, but firstly we need to get the core project building on the agent.  There will be a lot more to learn on this one, but for now, what worked well for us was the following:

image

Our Solution file contains all the information we need to work out what needs to be built, and TeamCity supports it so jobs a good un.  As I extend the base build then this method will still just work as I’ll be modifying the .csproj files belonging to the solution anyway.

Build Step 2

This one was slightly more convoluted, but basically, giving relative paths to the DLLs that contain the unit tests is the way forward here.

image

Make sure you target the right framework version (I didn’t initially, though the error messages from TeamCity are pretty good in letting you figure it out).

Build Triggering

We want this all to trigger whenever we checkin to our source control system (in our case, subversion), so when we click on ‘build triggering’ and ‘add trigger’, selecting ‘VCS Trigger’ will get you everything you need:

image

Are we there yet?

Well, just about – you will see the admin interface has a ‘run’ button against this configuration (top right of browser), lets do an initial run and see what the problems are (if any).  You can monitor the build by clicking on the ‘agents’ link at the top of the page and then clicking on the ‘running’ link under the current build.

Should you get the message:

… Microsoft.WebApplication.targets" was not found…

This basically happens because you don’t have web deployment projects (or indeed VS2010) installed on the build server.  The path of least resistance is to copy the C:\Program Files\MSBuild folder over to the build machine’s Program Files folder (if x64, make sure you put it in the x86 one).  You should find the build just works after that.

Ok, Build is working – Tell me about it!

Notifications were the last thing I setup (make sure you’ve setup a user account for yourself before you do this, the admin account shouldn’t necessarily have notifications switched on).  Click on ‘My Settings & Tools’ at the top and then ‘Notification Rules’.

I’ve setup an email notifier (which will apparently tell me of the first failed build, but only the first after a successful), and I’ve downloaded the windows tray notifier (My Settings & Tools, General, right hand side of the page) which is setup likewise.

Next Steps?

There are a lot of other tasks I want to get out of this, not just from a CI point of view.  I’ve deliberately (as @stack72 suggested) kept the initial setup ‘simple’ – getting a working setup was far more important than getting an all encompassing setup that does everything I want from the off.  I can now see the guys doing their checkins and the tests passing, I’m now far more aware if someone has broken the build (and lets face it, we’ll all deliberately break it to see that little tray icon turn red), and I know there’s so much more that I can do.

Next priorities are:

  1. Learn MSBuild so that I can perform tasks more efficiently in the default build – e.g. I want to concatenate and minify all CSS on the site, I want to minify our own Javascript, etc.
  2. Setup deployment on the checkin build – I suspect this will use Web Deployment Projects (which themselves generate MSBuild files so are fully extensible) to get our checked in code across to our dev servers.
  3. Setup a nightly build that runs more tests.  As you can see above, we build and run unit tests for our checkin build – I want to run nightlies that perform both unit and integration tests – I want the nightly to deploy to dev also, but to promote the files necessary to our staging server (not publish them) so that we can at any point promote a nightly out to our staging and then (gulp) live servers.

I’d urge anyone working on projects where deployment is a regular and pain in the arse task, or if there are a few of you and you’ve taken unit testing and TDD (be that test first or just good solid functionality coverage), my view now is that Continuous Integration is the tool you need. 

It’s the new Santa Claus – It knows when you’ve been coding, it knows when you’re asleep, it knows if you’ve been hacking or not, so code good for goodness sake!

As per all of my other posts, the above is from a novice CI person – any feedback that anyone can give, any little nuggets of advice, any help at all – I’ll soak it up like a sponge – this has been a lot of fun, and there’s definitely a warm glow knowing it’s now in place, but there’s a long way to go – feedback *very* welcome!

MSBuild – the voyage of the noob

I’ve been determined to have a play with build automation/continuous integration for a while now and just have always found something more fun to play with (ORM, MVC, etc.), though I know as the team where I work move forward, there needs to be some control and some vision on how all of our work should hang together.  With that in mind, this weekend I started to read up on MSBuild (yup, I know there are other build managers out there, but I thought I’d start with that as my learning platform and move on from there).

Why do I need to modify the default build?

Why does anyone really I suppose, but I like what we get from it.  As we move forward, the following I think will be useful to us:

  • automating unit test runs on successful builds
  • auto-deploying to our development server
  • minifying and concating javascript and css
  • ensuring coding style rules are followed (once I setup a set of company rules for us)
  • other things I haven’t imagined… there will be lots!

So where do I learn?

This was my first stumbling block.  There are a lot of resources on MSBuild, and trudging through them to find the one that was right for my learning style and approach was a nightmare.  I though to start out with the task that was at the forefront of my mind (concat/minify JS/CSS), but I just didn’t find any resources that were straightforward (my failing more than the resources available I’m sure!)

I’ve grabbed a few useful ones on my delicious bookmarks, and in particular, a significant thanks must go to the Hashimi brothers for their fantastic series on dnrTV and finishing up with some Stack Overflow discussion.

So what did I learn?

Firstly, a quick look at a post by Roy Osherove highlighted to me some of the tools available.  I found two other visual build tools: MS Build Sidekick and MSBuild Explorer, both of which I found very useful in actually *seeing* the build process, but after a watch through those dnrTV vids, I though I’d try something straight forward – concating CSS files into a ‘deploy’ css.

Get into the .csproj file

Unload your project, right click on it, and select ‘edit <projectname.csproj>’

image

MSBuild projects seem to be broken down quite succinctly into Targets, Tasks, Items, and Properties.  For my particular need, I needed to look at Items and Targets.

The schema in MSBuild is incredibly rich – you get intellisense for the most part, but because you can define your own schema elements, you are never going to get 100% intellisense.

You have a number of different ‘DependsOn’ items (mostly defined in Microsoft.CSharp.targets file), so you can create tasks that hang onto some of these like so:


    
			ConcatenateCSS;
			$(BuildDependsOn);
		
  

This is telling the build process that I have a target called ‘ConcatenateCSS’ that should happen before the ‘BuildDependsOn’ target (roughly speaking!)

I then created that target with the following:


	  
		  
	  
    
    
    
      
    
    

Which to me, looks bloody complex! I had to find some help on this one naturally.  But essentially, we have created a target called ‘ConcatenateCSS’ which is going to execute before the build.  We create an ItemGroup (and this is where the intellisense falls over) called ‘InFiles’, and we tell it to include everything ending in .css under the _assets\css folder (it seems the **\\*.css is the wildcard for recursion too, though I may be wrong on this!), and we want to exclude _assets\css\site.css (more on this in a sec).

I then send a message (which will be seen on ‘output’ during build which tells us it’s happening, and then use the combination of ‘ReadLinesFromFile’ and ‘WriteLinesToFile’.  The %(InFiles.Identity) in the ReadLinesFromFile essentially turns this into a foreach loop, and Identity is one of the MSBuild defaults.  So this is essentially, foreach of the files we’ve identified, output the contents to the ‘Lines’ variable/parameter.  We then Write the whole lot back to our file using the @(Lines) variable.

Now, on each build, we generate a single css file (site.css) that our site can reference, but all edits go in via the broken files.  Yes, there are more elegant ways to do this, and yes, I will likely do that in time, but I’ve made a start!

Where next?

I’d be lying if I said I could do the above without some solid examples and help, so the next steps for me are creating a solid understanding of the core concepts, playing with the tools, and looking to solve some of our core business issues as we move forward in order to take some of the human elements out of the build process.  Obviously I have to investigate continuous integration and see where that all fits in too, but I’m happy with the start I’ve made.