New research from Soluto claims that “if you’re looking for top reliability … MacBook Pro is the best Windows PC on the market”. The study is based on long term analysis of 150,000 laptops over three months in Q1 2013.
However, the message seems confused – they say Apple’s Macbook Pro 13 is the “best performing” Windows laptop but also say that it is “best on the market” for reliability.
Bold claims but does the actual research support them? Best performing is not usually the same as most reliable…
As the results below show, the top 3 (lower score is better) are a long way clear of the rest of the field so I’ll compare those only.
As always, the devil is in the detail – each laptop has a subset of 5 individual scores, of which 3 are given arbitrary weightings to give them more prominence. Soluto admit those weighting decisions may seriously impact the overall scores and they invite people to look at the actual numbers.
Challenge accepted! Let’s look at the individual scores of the top 3 to see if Mac Book Pro really is the best Windows laptop (Acer total score is slightly different due to rounding because the report figures only go to 2 decimal places for each factor):
1. Crashes per week is a fair criteria to include and receives no weighting. Acer is slightly better than Mac whilst Dell lags well behind.
2. Hangs (non responsive events) is also reasonable and receives no weighting. Acer scores exceptionally well but Mac and Dell average more than 1 hang per week.
3. BSOD (Blue Screen of Death) crashes are more serious and are therefore weighted – multiplied by 10 i.e. one BSOD is counted the same as 10 application crashes. Although Acer scored better than the Mac on crashes and hangs, it is worse for BSODs – and as Soluto weight them so heavily they give Mac a 0.5 point advantage over Acer. This 0.5 point difference alone easily propels Apple’s baby into top spot.
Of course Acer would easily overtake Mac overall if we applied a reduced weighting to BSODs but, although I disagree with Soluto’s weighting of 10, let’s leave it in for the moment – there is no need to quibble as the next 2 criteria quickly dispel the notion that Mac Book Pros are the most reliable/best performing Windows laptops…
4. ‘Average running processes’ is also heavily weighted and results in the Mac gaining a 0.25 point advantage over the Acer and 0.5 over the Dell. In my opinion, not only should this factor not be weighted, but it actually has no place at all in a report covering ‘best performance’. Sure, fewer running processes may help Windows perform better in theory but, in practice, it all depends on what they are – just one runaway process sucking up CPU and RAM can bring a system to its knees.
Without any info on how much resource each process typically uses it is impossible to derive any performance or reliability data – there is nothing to gain from including this factor, much less apply a significant weighting to it. Here’s the same chart without running processes included (all scores are higher because processes applied a negative score before).
Now the top three positions are reversed, moving the Dell to the top. But we’re not finished yet.
5. The last criteria is ‘average boot time’ and this is heavily weighted too. As a result, the Dell gains a whopping 1.4 points over the others because it is relatively (*) fast to boot – it uses an SSD boot drive whereas the others have standard, slower, hard drives.
(*) Note: the Dell’s boot time is ‘relatively’ fast as 67 seconds for a $1000 SSD equipped PC is still stone age stuff – my own recent sub-$1000 PC build fully boots in less than 20 seconds. However, the average boot time for all 10 laptops was a glacial 167 seconds, close to 3 minutes…
Whilst boot time obviously matters in terms of performance, the weighting is so huge that it completely overrides the reliability scores – to ascertain pure reliability, here’s the same chart without boot times or average processes included (all scores are higher because boot times also applied a negative score before).
This leaves Acer in top place (even with the very high 10x BSOD weighting) and Mac a creditable second whilst Dell performs worst due to its very high numbers of crashes and hangs.
Performance vs Reliability
For pure performance, there’s little to choose between the top 10 specs in terms of RAM and CPU – the only real difference is whether an SSD or hard drive is used. This is why the Dell can perform even worse than its Latitude E5530 cousin (in 8th place) for every reliability criteria yet still be bumped up to third place – just because it uses a fast SSD and benefits from much quicker boot times.
For pure reliability, including average processes and boot times doesn’t make sense – omitting those puts Acer firmly in top spot, some way ahead of the Mac.
Soluto emphasize the fact that every Windows installation on the MacBook is clean (i.e. no crapware from OEM manufacturers) and suggest that this accounts for its relatively good reliability scores. However, this ignores the fact that the MacBook Retina in 6th place will also benefit from clean Windows installations yet suffers the worst number of crashes of all ten laptops – a massive 2.05 per week…
Conclusion
A fast SSD transforms Windows performance – it seems ridiculous to claim that the $1199 MacBook is the ‘best performing’ Windows laptop without one. As with all statistics, you can manipulate them in as many different ways as you have views to prove.
However, of the top three, my pick for the best Windows laptop would clearly be the Acer – I could buy 2 of them for the price of the Dell or 3 of them for the price of the MacBook Pro (which needs an extra $100-$200 for a copy of Windows).
Alternatively, I’d just buy one plus an SSD ($100) and install Windows cleanly to make it the best performer as well, not just the most reliable. At a total cost of $529 I’d be happy to call that the ‘best performing’ and ‘most reliable’ Windows laptop (of those tested in the study) – and even after adding a new SSD it would still be the second cheapest of all ten.
