Huawei has been caught optimizing some of its top smartphones to over-perform on benchmark tests. On Tuesday, AnandTech discovered that Huawei’s P20 had been programmed to maximize performance specifically when running 3DMark, a popular benchmarking app. Today, the company behind 3DMark followed up with a statement saying that it had confirmed the findings and would delist the P20, as well as three other Huawei phones with similar behavior, from its benchmark leaderboards.
The delisted phones include the P20, P20 Pro, Nova 3, and Honor Play. Huawei admitted to this behavior in a statement given to Android Authority, saying that its phones are designed to adjust their performance based on the app that’s running.
But the way that Huawei implemented that behavior isn’t allowed. While phones can adjust their performance as part of their typical behavior under high workloads, they can’t be hard coded to maximize their behavior just because a specific benchmark app is running. That’s what Huawei seems to have done, according to UL, which is behind the 3DMark software.
When UL ran an internal version of 3DMark, which Huawei’s phones couldn’t recognize the name of, the phones performed worse in the test. That indicated that the phones weren’t actually smart enough to identify high performance demands on their own, which meant the benchmark score wasn’t an accurate reflection of how the phone would handle a typical app without special attention from Huawei.
As punishment, 3DMark has removed these phones’ rankings from its leaderboard and adorned their listings on its website with a note that the phone’s “manufacturer has not complied with UL benchmark rules.” Many of their results have been removed as well.
Huawei is far from the first company to get caught toying with benchmark results. Samsung got busted for the same behavior on its flagship phones in 2013, and just last year, OnePlus was found to have done the same. This timing is particularly unfortunate for Huawei, since just weeks ago it got caught trying to pass off a DSLR photo as a photo from one of its phones.
What’s funny about all of this is that benchmarks don’t really matter that much. Tweaking a phone to optimize benchmark apps might produce some numbers that make a small subset of nerds drool, but those numbers don’t correlate to the actual experience of using the phone. They might speak to how well the phone performs under heavy stress, like while gaming, but a better test is to just play a game with it and find out what happens.
Huawei even admits this. In its statement, the company said it “always prioritizes the user experience rather than pursuing high benchmark scores — especially since there isn’t a direct connection between smartphone benchmarks and user experiences.” And yet, it still coded its phone to deliver higher performance when running a benchmark test.
But Huawei also claims that its phones’ include AI that’s smart enough to optimize performance based on whatever app is running, and clearly that’s not the case. If it were, then boosting performance for benchmarks would be fair game — but as it is, it’s not a true reflection of how the phone performs in demanding situations.
The delisted phones include the P20, P20 Pro, Nova 3, and Honor Play. Huawei admitted to this behavior in a statement given to Android Authority, saying that its phones are designed to adjust their performance based on the app that’s running.
But the way that Huawei implemented that behavior isn’t allowed. While phones can adjust their performance as part of their typical behavior under high workloads, they can’t be hard coded to maximize their behavior just because a specific benchmark app is running. That’s what Huawei seems to have done, according to UL, which is behind the 3DMark software.
When UL ran an internal version of 3DMark, which Huawei’s phones couldn’t recognize the name of, the phones performed worse in the test. That indicated that the phones weren’t actually smart enough to identify high performance demands on their own, which meant the benchmark score wasn’t an accurate reflection of how the phone would handle a typical app without special attention from Huawei.
As punishment, 3DMark has removed these phones’ rankings from its leaderboard and adorned their listings on its website with a note that the phone’s “manufacturer has not complied with UL benchmark rules.” Many of their results have been removed as well.
Huawei is far from the first company to get caught toying with benchmark results. Samsung got busted for the same behavior on its flagship phones in 2013, and just last year, OnePlus was found to have done the same. This timing is particularly unfortunate for Huawei, since just weeks ago it got caught trying to pass off a DSLR photo as a photo from one of its phones.
What’s funny about all of this is that benchmarks don’t really matter that much. Tweaking a phone to optimize benchmark apps might produce some numbers that make a small subset of nerds drool, but those numbers don’t correlate to the actual experience of using the phone. They might speak to how well the phone performs under heavy stress, like while gaming, but a better test is to just play a game with it and find out what happens.
Huawei even admits this. In its statement, the company said it “always prioritizes the user experience rather than pursuing high benchmark scores — especially since there isn’t a direct connection between smartphone benchmarks and user experiences.” And yet, it still coded its phone to deliver higher performance when running a benchmark test.
But Huawei also claims that its phones’ include AI that’s smart enough to optimize performance based on whatever app is running, and clearly that’s not the case. If it were, then boosting performance for benchmarks would be fair game — but as it is, it’s not a true reflection of how the phone performs in demanding situations.
