Performance Budgets with Phantomas and Jenkins

As we all know, there is more to web performance testing than your server response time. Number of requests, image sizes, cache headers all play their part. RWD has raised the profile of these factors once again due to limited bandwidth on mobile devices.

While it has been (relativley) easy to measure your server performance in the past, the other factors have often relied on manual checks using google page speed.

Performance Budget

One technique to acheive your goals is to set your project a performance budget. The budget sets targets for page weight, number of requests etc. If you're doing RWD you could set a different budget for mobile, tablet and desktop.

Setting a Budget

At the start involves a little guesswork, but as you project evolves you can understand your product better and set better values for your budget.

I'm going to start my budget with the aim that we want our site to load within 4 seconds on Mobile - here's why.

Page Weight

Say are users are using 3G with a speed of 2Mbps. This equates to a maximum page weight of 1MB

Number of Requests

However, download speed is not the bottleneck. The bottleneck is the network latency, smartphone’s memory and CPU. Even if the phone can download 1MB in 4 seconds, the website will take longer to load because the phone needs to receive and process the code and images.

This can take a little guesswork, but lets start with the goal that we will have only a single css file & a single js file and less than 20 requests in total.

HTML load time

One of the most important metrics is perceived page load - this is often regarded as the time taken to load & render the HTML. Images etc can load after but the site will still be useable by the user. Here we can set a more aggressive target of 1s

Automated checks with Phantomas

Phantomas is a PhantomJS-based modular web performance metrics collector, and provides the key measurements we need to validate our performance budget in a scriptable repeatable way.

getting up and running is easy

npm install --global phantomas  

then

phantomas https://github.com/macbre/phantomas --verbose  

Adding our Metrics

To ensure our budget is met - we'll create a configuration file that contains our metrics config.json.

{
    "asserts": {
        "bodySize": 1024,
        "requests": 20,
        "cssCount": 1,
        "jsCount: 1,
        "timeToLastByte": 1000
    }
}

we can then run our tests using

phantomas https://github.com/macbre/phantomas --verbose --config=config.json  

RWD

The first valuable feature of phantomas is the ability to run the tests agains a viewport size, using this we can check metrics for mobile --phone vs tablet --tablet vs desktop.

More...

Phantomas provides a wealth of stats and measures - I'm not going to go into any more detail here but consider looking through the options and fine tune your tests.

Tracking in Jenkins

The final step is to have this test running as part of a CI. It is quite simple to add the script steps above to fail a build, but I want to do slightly more and graph the stats over time to identify how the metrics for our site have changed.

To acheive this we will use the TAP, Plot & NodeJs plugins for Jenkins.

Ensure the plugins have been installed, from Jenkins global configuration add phantomas to the Global npm packages to install section.

We want to generate a TAP file to archive our test result, but we also want to generate a CSV to plot the trends over time. Create your build and add the following build step

phantomas https://github.com/macbre/phantomas --verbose --config=config.json --phone --format tap > phone.tap  
phantomas https://github.com/macbre/phantomas --verbose --config=config.json --phone --format csv > phone.csv  

Add a post build step to Publish TAP results and then another to Plot build data. The build data requires a little more setup as you'll have to include the specific CSV columns that you want to track.

Once you've done all of this you should have some pretty sophisticated metrics to measure your performance against - and some pretty graphs.

Request Graph

Comments