Querying the PageSpeed Insights API in Node.js

Now at some point or other we’re all going to need some sweet, sweet page speed metrics, whether it be to highlight how well your competitors are doing, or instead to shame your dev / product teams into fixing the unholy mess that is your site’s performance.

While this can be achieved in a number of ways, by far the easiest is using something like PageSpeed Insights to grab that information - as this acts as a good proxy for how Google will be viewing the performance - or lack thereof - of your pages, and allow you to benchmark yourself against others in your space.

The difficulty often comes in needing to grab this data en masse, and where I’ve tended to find that, unless you’ve got competitor benchmarking tools already in place like Calibre or SpeedCurve, then it can be tricky to gather all the metrics you need at once.

Of course there are numerous services such as Screaming Frog and URL Profiler that will do this for you; and while bulk usage of the API has been written about numerous times, namely here and here - I personally quite like this approach, as it’s both quick, and works in the format that I like: i.e. uploading a big ol’ list of pages in a text file, and outputting it to a CSV, no frills attached.

Note: if you want to query different metrics (and lawd knows there’s plenty of them) you can query the docs, to experiment with what data you want to return, and sub in / out different metrics.

const axios = require('axios');
const fs = require('fs');
const ObjectsToCsv = require('objects-to-csv');

let data = [];
let promises = [];

// 1. Import URLS
const urls = fs.readFileSync('./urls.txt', 'utf8').split('\r\n');

// 2. Build Promises array to fetch data
urls.map(function(url){
  const psi = `https://www.googleapis.com/pagespeedonline/v5/runPagespeed?url=${url}&key=${API_KEY}`;
  promises.push(axios.get(psi))
});

// 3. Loop through all the promises and return the response
axios.all(promises).then(function(results) {
    results.map(function(res) {
      // Lab metrics
      try {
        const {
          metrics,
          categories,
          audits
        } = res.data.lighthouseResult;

        // Build data object
        data.push({
          'URL': res.data.id,
          'Performance Score': Math.ceil(categories['performance'].score * 100),
          'First Contentful Paint': audits['first-contentful-paint'].displayValue,
          'Largest Contentful Paint': audits['largest-contentful-paint'].displayValue,
          'Speed Index': audits['speed-index'].displayValue,
          'Time to Interactive': audits['interactive'].displayValue,
          'First Input Delay': audits['estimated-input-latency'].displayValue,
          'Cumulative Layout Shift': audits['cumulative-layout-shift'].displayValue,
        });
      }
      catch(e) {
        console.log(e);
      }
    })

    // 4. Write data to CSV
    // console.log(data);
    const csv = new ObjectsToCsv(data)
    csv.toDisk('./page-speed-data.csv')
});

The nuts and bolts of this is that it reads the pages into an array, then it uses the awesome Axios package to queue up a list of URLs we want to fetch data from the API on as a list of promises.

We then use the axios.all() method to take that array of Promises, and map over the response of each one, extracting the given metrics that we want, and pushing it to the main data array at the top.

We then take this array of objects and then write the resulting data to a CSV using the aptly named ObjectsToCsv package.

And we’re done! As many URLs as you want delivered direct to your spreadsheet, in a memory efficient, asynchronous manner.