Thursday, March 5, 2015

A deeper look at graphics benchmark results, including GFXBench 3.1 and Basemark X

In this post I will take a closer at graphics benchmark results for different SoCs. I will look beyond just GFXBench (for which a new version has appeared), because the workload tested by well-known GFXBench tests such as T-Rex and Manhattan is not necessarily reflective of the actual gaming experience. Alternative benchmarks exist, such as Basemark X which uses the Unity engine that is commonly used in games.

GFXBench 3.1 released for OpenGL ES 3.1, Snapdragon 805 does well


Kishonti recently released a new version of GFXBench, GFXBench 3.1 for OpenGL ES 3.1, that includes tests for the OpenGL ES 3.1 API standard supported by many recent devices. A few results from the new benchmark tests are already available, with the Adreno 420 GPU inside Snapdragon 805 closing most of the performance gap with the Mali-T760 MP6/MP8 in Samsung's Exynos SoCs in the Manhattan 3.1 test.

                                                      Offscreen Manhattan Manhattan
Device               SoC             GPU              T-Rex        3.0       3.1

NVIDIA Shield Tablet NVIDIA K1-32    Tegra K1 GPU        3692     1979      1443  
HTC One M9           Snapdragon 810  Adreno 430          2732     1413
Galaxy S6 Edge       Exynos 7420     Mali-T760 MP8?      3312     1607       793
Sams. Galaxy Note 4  Snapdragon 805  Adreno 420          2386     1153       773
Samsung Galaxy S6    Exynos 7420     Mali-T760 MP8?      3314     1609       634
Sams. Galaxy Note 4  Exynos 5433     Mali-T760 MP6       2163     1110       436
HTC One M8           Snapdragon 801  Adreno 330          1608      768
Teclast X98 Air      Atom Z3736F     Intel HD            1014      564       307
Google Nexus 10      Exynos 5250     Mali-T604 MP4        818      351       185

NVIDIA's Tegra 32-bit version of Tegra K1 leads (the 64-bit Denver-based version of Tegra K1, and Tegra X1, have not yet been tested). Performance of Snapdragon 805 as implemented in certain models of the Samsung Galaxy Note 4 holds up better in the Manhattan 3.1 test than Samsung's Exynos SoCs with Mali-T760 MP6/MP8. Whereas Exynos 7420 (used in the Galaxy S6) has a clear advantage in existing benchmarks (1609 vs 1153 for Manhattan and 3314 vs 2386 for T-Rex), it loses that advantage in the new Manhattan 3.1 test (although the Galaxy S6 Edge benchmarks result suggests it is still slightly superior). Intel's Baytrail SoCs seem to hold up relatively well looking at the result for an Atom Z3736F-based tablet, albeit at a lower performance level.

GFXBench 3.1 results for Snapdragon 801 and the new Snapdragon 810 are not yet available. However, given the fact that GFXBench appears to generally do well on Snapdragon SoCs, they can be expected to score fairly highly. I'll say more about the apparent advantage for Qualcomm's SoC in GFXBench in the final section of this article.

Basemark X is a useful alternative to GFXBench


Basemark X is a gaming benchmark that utilizes the Unity engine that is commonly used in games, and developer Rightware claims that it actually reflects practical performance in games. Although it does include an on-screen demo, the actual benchmark scores appear to be derived from off-screen rendering at a fixed resolution, so that benchmark results can be compared objectively between different devices.

Previous generation SoCs: MT6582 beats Snapdragon 400 in Basemark X


Taking a look at previous-generation cost-sensitive SoCs, while MediaTek's ubiquitous quad-core 3G SoC MT6582 (which supports Open GL ES 2.0 only, through its Mali-400 MP2 GPU) scores lower than Snapdragon 400 in GFXBench's OpenGL ES 2.0-based T-Rex test (about 230 vs 330), in Basemark X MT6582-based devices score higher than Snapdragon 400 based devices. This is despite the fact that Snapdragon was/is often employed in devices with a considerably higher selling price than MT6582-based devices.

Device               SoC             GPU                 Display*   Medium   High

Samsung SM-G800F     Exynos 3470     Mali-400 MP4        1280x720    7527    2712
Vodafone 985N        MT6582          Mali-400 MP2         960x540    4950    1717
Acer E53             MT6582          Mali-400 MP2        1280x720    4870    1694
Wiko Rainbow         MT6582          Mali-400 MP2        1280x720    4826
Galaxy S3 Neo        Snapdragon 400T Adreno 305          1280x720    4540    1551
Moto G (XT1032)      Snapdragon 400  Adreno 305          1280x720    4440
HTC Desire 816d      Snapdragon 400T Adreno 405          1280x720    4354    1441
Samsung SM-A500F     Snapdragon 410  Adreno 306          1280x720    4132    1900
Samsung SM-A300F     Snapdragon 410  Adreno 306           960x540    4076    1892
Samsung SM-G530H     Snapdragon 410  Adreno 306           960x540    3987    1690
Samsung SM-G800A     Snapdragon 400  Adreno 305          1280x720    3946    1362
HTC Desire 820q      Snapdragon 410  Adreno 306          1280x720    3786

* While Basemark X is independent of display resolution in terms of rendering, the
memory bandwidth used for screen refresh has some impact, giving lower-resolution
devices a small advantage.
Notes: Samsung SM-G800F is the Galaxy S5 Mini (Exynos version), while SM-G800A is a Snapdragon 400 running at the non-standard maximum clock speed of 1.4 GHz; Vodafone 985N is the Vodafone Smart 4 Power; Acer E53 is the Acer Liquid E700; Galaxy S3 Neo runs the Snapdragon 400 SoC at a non-standard maximum speed of 1.4 GHz; HTC Desire 816d runs the Snapdragon 400 SoC at 1.6 GHz; SM-A500F is the Galaxy A5, while SM-A300F is the Galaxy A3; SM-G530H is the Galaxy Grand Prime.

For both the medium detail and high detail settings, MT6582-based devices consistently score higher in Basemark X than Snapdragon 400 and also Snapdragon 410-based devices for the medium detail test, which gives a different picture than the one you get from just looking at GFXBench's T-Rex benchmark

Snapdragon 410 performs worse than Snapdragon 400 in Basemark X medium-detail


Also notable is that Snapdragon 410, which is the successor of the Snapdragon 400 and would normally be expected to improve performance, actually has lower performance in practice as judged by the Basemark X medium detail benchmark. This matches earlier findings of performance flaws in Snapdragon 410. When running the high detail Basemark X benchmark, Snapdragon 410 does better and beats Snapdragon 400.

Mid-range SoCs: Snapdragon 615 and MT6752 closely matched


When running GFXBench, Snapdragon 615 and MT6752 are closely matched, with Snapdragon 615 scoring about 830 to 850 in T-Rex while MT6752 scores just above 870. For T-Rex, devices using MediaTek's prior-generation octa-core MT6592 score in the range 650 to 750. In the OpenGL ES 3.0 API-based Manhattan benchmark, Snapdragon 615 and MT6752 are very closely matched, both scoring around 360. We will also take a look at Basemark X results.

The following table shows Basemark X results for the new competing mid-range SoCs Snapdragon 615, MT6752 and HiSilicon's octa-core Hi6210 (Kirin 620), as well as for the prior-generation octa-core MT6592 from MediaTek.

Device               SoC             GPU                 Display*   Medium   High

Lenovo P70-A         MT6752          Mali-T760 MP2       1280x720   11311 
Meizu M1 Note        MT6752          Mali-T760 MP2       1920x1080  11168    4636
HTC Desire 816G      MT6592          Mali-450 MP4        1280x720   10984
Huawei CHE2-TL00     Hi6210          Mali-450 MP4        1280x720   10546    3439
Oppo R8106           Snapdragon 615  Adreno 405          1920x1080  10277    4846 
HTC Desire 820       Snapdragon 615  Adreno 405          1280x720   10133    4814
Samsung SM-A700FD    Snapdragon 615  Adreno 405          1920x1080  10052    4757
Archos 50C Oxygen    MT6592          Mali-450 MP4        1280x720    9867    3702
HTC Desire 616d      MT6592M         Mali-450 MP4        1280x720    7976    3045

* While Basemark X is independent of display resolution in terms of rendering, the
memory bandwidth used for screen refresh has some impact, giving lower-resolution
devices a small advantage.
Notes: SM-A700FD is the Galaxy A7; Huawei CHE2-TL00 is a new version of the Honor 4X.

When running the standard medium-detail version of Basemark X, MediaTek's MT6752 has  a moderate advantange over Snapdragon 615, while at the high detail setting Snapdragon 615 has a small advantage. Huawei's Kirin 620 performs adequately and just ahead of Snapdragon 615 in the medium detail setting.

MediaTek's prior-generation octa-core MT6592 with Mali-450 MP4 GPU keeps up relatively well in Basemark X,  with certain models (e.g. HTC Desire 816G) actually beating Snapdragon 615 in the medium detail setting.

Performance-oriented SoCs with Basemark X


The following table shows Basemark X results for several performance-oriented mobile SoCs.

Device               SoC             GPU                 Display*   Medium   High

Samsung Galaxy S6    Exynos 7420     Mali-T760 MP6       2560x1440  36017
Galaxy S5 LTE-A      Snapdragon 805  Adreno 420          1920x1080  32685   18334
Google Nexus 6       Snapdragon 805  Adreno 420          2560x1440  30362   20265
Sams. Galaxy Note 4  Snapdragon 805  Adreno 420          2560x1440  31963   21152
Sams. Galaxy Note 4  Exynos 5433     Mali-T760 MP6       2560x1440  29335   19019 

Apple iPad Air 2     Apple A8X       PowerVR Series 6    2048x1536  41700   29239
Google Nexus 9       NVIDIA K1-64    Tegra-K1 GPU        2048x1536  37939   28646
Apple iPad Mini 3    Apple A7        PowerVR Series 6    2048x1536  26499   14780
Teclast X98 Air      Atom Z3736F     Intel HD            2048x1536  14825    7160
Teclast P90HD        Rockchip RK3288 Mali-T764           2048x1536  13053    5645
Onda V989 Core8      Allwinner A80   PowerVR G6230       2048x1536  11004    5724

Meizu MX4 Pro        Exynos 5430     Mali-T628 MP6       1920x1200  25547   12674
Samsung SM-G900A     Snapdragon 801  Adreno 330          1920x1080  25178   11930
Samsung SM-G850F     Exynos 5430     Mali-T628 MP6       1280x720   21872   10666
Meizu MX4            MT6595          PowerVR G6200       1920x1200  17038    7817
Huawei MT7-TL10      Kirin 925       Mali-T624 MP4       1920x1080  15973    6802

* While Basemark X is independent of display resolution in terms of rendering, the
memory bandwidth used for screen refresh has some impact, giving lower-resolution
devices a small advantage.
Notes: SM-G900A is the Samsung Galaxy S5 (US version), Huawei MT7-TL10 is the Huawei Mate 7.

Looking at the ultra-high-end smartphone segment (mostly with a display resolution of 2560x1440), Exynos 7420 provides superior performance in Basemark X. Snapdragon 805 follows, a small distance ahead of Exynos 5433 as used in the Samsung Galaxy Note 4.

In the high-end tablet segment, Apple's iPad Air 2 with the Apple A8X leads, but the Nexus 9 with NVIDIA's Tegra K1 (64-bit version) comes fairly close. Apple's prior generation SoCs also delivers good performance, while Intel's current Baytrail SoC for the tablet market outperforms two high-end chips from established Chinese players in the tablet SoC market, Rockchip's RK3288 and Allwinner A80 Octa.

Mainwhile, in the mainstream performance smartphone segment, Snapdragon 801 (in the past the performance leader in the market) still provides good performance, but is actually just beaten by the 32-bit Exynos 5430 in the Meizu MX4 Pro. The chip is also used in the Galaxy Alpha (for which it provides higher-than-necessary performance given its relatively low screen resolution), while the performance of MediaTek's MT6595 SoC, while not bad, falls short of most other high-end solutions. HiSilicon's Kirin 925 as implemented in the Huawei Mate 7 is just behind.

Conclusion


It appears that just concentrating on GFXBench may give a misleading picture with regard to 3D graphics performance of mobile SoCs. In particular it is apparent that Qualcomm's Snapdragon SoCs consistently do better in GFXBench than in other benchmarks such as Basemark X. This is particularly true for the lower-end Snapdragon 400 and higher-end Snapdragon 800 series; for Snapdragon 615, results are more consistent across different benchmarks.

Basemark X, which utilizes the Unity game engine commonly used in mobile games, may more accurately reflect real-world performance.

Sources: Rightware Power Board (Basemark X benchmark results), GFXBench results database

Updated 5 March 2015: Add Galaxy S6 Edge result for GFXBench 3.1.
Updated 15 March 2015.

No comments: