Testing on Blacksmith

Every change we make to our website builder has to pass a suite of automated tests before it goes live. There are 466 of them, and in each one Playwright drives a browser through the site the way a person would. The tests run on GitHub Actions, split into four parallel parts, and each part first builds the whole site in its production form before testing it. One round takes about twenty minutes.

Since September these tests haven't run on GitHub's machines but on Blacksmith. The switch took a change to a single line of configuration. In October Blacksmith accepted us into their startup program, so we decided it was time to measure what the move had actually given us. Until then we had never compared it directly with GitHub's machines.

How we measured

We pushed the same code twice, 25 seconds apart. One run went to Blacksmith, the other to GitHub's standard runners – the only difference was the line that decides where the tests run. To keep the comparison fair, both machines were the same size, 2 virtual CPUs.

Both runs passed with the same result: 395 tests passed, 71 were skipped and none failed. It was one run on each side, so read the numbers below as one specific measurement, not an average.

Results

• From push to the last job finished: 21.0 min on Blacksmith, 33.3 min on GitHub.
• Slowest part: 20.6 min against 33.1 min.
• Billed minutes: 71 against 106.
• Cost of one round at list price: $0.28 against $0.64.

Billed minutes are the sum of all jobs, each rounded up to a whole minute, which is how both providers bill. Blacksmith's list price is $0.004 per minute; GitHub charges $0.006 for a comparable Linux machine.

Where the extra 12 minutes came from

At first glance it looks like a huge difference. Most of it, though, came from one unexpected event. Before the tests start, every machine has to install system packages for the Chromium browser. On GitHub, two of the four parts hit a stalled Ubuntu package mirror that was sending fonts at 35 kB/s, so the install took about 11 minutes instead of the usual 0.3. The other two parts, and all four on Blacksmith, finished this step in 0.3 minutes.

That one stall accounts for 11 of the 12 minutes. We saw it in this single run and haven't measured how often it happens on GitHub.

And the raw performance?

With the setup taken out, the gap gets smaller but stays on Blacksmith's side. Building the site (next build) took 1.78 minutes on average against 2.15 on GitHub, 1.2× faster. The Playwright tests themselves took 14.3 minutes against 16.4, 1.15× faster.

Blacksmith also had a small head start because it restored our installed dependencies from its cache, while GitHub installed them again on this first run. That only cost GitHub about 10 seconds per part.

What we take from it

On a machine of the same size, building and testing run roughly 15 to 20 % faster on Blacksmith. A minute also costs a third less and this run used 35 fewer of them, so one round of tests came out at less than half the price. The longest wait in this measurement wasn't caused by the CPU but by a stalled package mirror, and that didn't happen on Blacksmith.

Thank you to Blacksmith for accepting us into their startup program.