Patch: ATi und nVidia cheaten im 3DMark03

Bereits mit dem Release des neuen Detonator 44.03 wurden Stimmen laut, nach denen nVidia im 3DMark03 künstlich die Reihenfolge des Renderings umändert, um so Rechenleistung sparen zu können. Auch Futuremark, die Entwicklerschmiede hinter 3DMark, hat sich das Verhalten der nVidia Treiber näher angesehen.

Hierbei konnte man in der Tat eine Manipulation feststellen und hat aus diesem Grund heute ein neues 3DMark03 Patch auf Version 3.3.0 veröffentlicht, das diesem Cheaten Einhalt gebieten soll.

In our testing, all identified detection mechanisms stopped working when we altered the benchmark code just trivially and without changing any of the actual benchmark workload. With this altered benchmark, NVIDIA“s certain products had a performance drop of as much as 24.1% while competition“s products performance drop stayed within the margin of error of 3%. To our knowledge, all drivers with these detection mechanisms were published only after the launch of 3DMark03. According to industry“s terminology, this type of driver design is defined as ‘driver cheats“.

Insgesamt geht es mit 24,1 Prozent sehr deutlich in der Leistung des GeForce 5900 Ultra nach Unten. Das kommt aber nicht von ungefähr, schließlich konnte Futuremark acht verschiedene Optimierungen im neuesten nVidia entdecken:

Futuremark“s audit revealed cheats in NVIDIA Detonator FX 44.03 and 43.51 WHQL drivers. Earlier GeForceFX drivers include only some of the cheats listed below.

• The loading screen of the 3DMark03 test is detected by the driver. This is used by the driver to disregard the back buffer clear command that 3DMark03 gives. This incorrectly reduces the workload. However, if the loading screen is rendered in a different manner, the driver seems to fail to detect 3DMark03, and performs the back buffer clear command as instructed.
• A vertex shader used in game test 2 (P_Pointsprite.vsh) is detected by the driver. In this case the driver uses instructions contained in the driver to determine when to obey the back buffer clear command and when not to. If the back buffer would not be cleared at all in game test 2, the stars in the view of outer space in some cameras would appear smeared as have been reported in the articles mentioned earlier. Back buffer clearing is turned off and on again so that the back buffer is cleared only when the default benchmark cameras show outer space. In free camera mode one can keep the camera outside the spaceship through the entire test, and see how the sky smearing is turned on and off.
• A vertex shader used in game test 4 (M_HDRsky.vsh) is detected. In this case the driver adds two static clipping planes to reduce the workload. The clipping planes are placed so that the sky is cut out just beyond what is visible in the default camera angles. Again, using the free camera one can look at the sky to see it abruptly cut off. Screenshot of this view was also reported in the ExtremeTech and Beyond3D articles. This cheat was introduced in the 43.51 drivers as far as we know.
• In game test 4, the water pixel shader (M_Water.psh) is detected. The driver uses this detection to artificially achieve a large performance boost - more than doubling the early frame rate on some systems. In our inspection we noticed a difference in the rendering when compared either to the DirectX reference rasterizer or to those of other hardware. It appears the water shader is being totally discarded and replaced with an alternative more efficient shader implemented in the drivers themselves. The drivers produce a similar looking rendering, but not an identical one.
• In game test 4 there is detection of a pixel shader (m_HDRSky.psh). Again it appears the shader is being totally discarded and replaced with an alternative more efficient shader in a similar fashion to the water pixel shader above. The rendering looks similar, but it is not identical.
• A vertex shader (G_MetalCubeLit.vsh) is detected in game test 1. Preventing this detection proved to reduce the frame rate with these drivers, but we have not yet determined the cause.
• A vertex shader in game test 3 (G_PaintBaked.vsh) is detected, and preventing this detection drops the scores with these drivers. This cheat causes the back buffer clearing to be disregarded; we are not yet aware of any other cheats.
• The vertex and pixel shaders used in the 3DMark03 feature tests are also detected by the driver. When we prevented this detection, the performance dropped by more than a factor of two in the 2.0 pixel shader test.

Doch es ist nicht nur nVidia, bei denen man Optimierungen feststellen konnte. Auch beim neuen ATi-Treiber geht es in der Leistung mit dem Futuremark Patch nach Unten. Allerdings hat ATi nur kleine Treiberoptimierungen für den vierten Gametest vorgenommen: Hier sinkt die Performance um 8,2 Prozent und wirkt sich auf das Endergebnis mit nur 2 Prozent weniger negativ aus.

A test system with GeForceFX 5900 Ultra and the 44.03 drivers gets 5806 3DMarks with 3DMark03 build 320. The new build 330 of 3DMark03 in which 44.03 drivers cannot identify 3DMark03 or the tests in that build gets 4679 3DMarks - a 24.1% drop.

Our investigations reveal that some drivers from ATI also produce a slightly lower total score on this new build of 3DMark03. The drop in performance on the same test system with a Radeon 9800 Pro using the Catalyst 3.4 drivers is 1.9%. This performance drop is almost entirely due to 8.2% difference in the game test 4 result, which means that the test was also detected and somehow altered by the ATI drivers. We are currently investigating this further.

Nicht vergessen sollte man allerdings, dass die GeForce FX5900 in einer Reihe an Spielen bewiesen hat, dass sie mit der Radeon 9800 Pro mithalten und sie teilweise gar überflügeln kann. Dass der Patch im 3DMark03 nun einen derartigen Einbruch bedeutet, hat mit der Performance in Spielen nichts zu tun. Hier gehören beide Karten weiterhin zur absoluten Leistungsspitze. Im Grunde genommen muss man somit zumindest indirekt einem Großteil der Käufer eine gewisse Mitschuld an dieser Misere in zuschreiben. Solange Benchmarks wie der 3DMark als beinahe einziges Kaufkriterium gedultet werden, obwohl sie mit der realen Performance wenig gemeinsam haben, solange werden Firmen darauf angewiesen sein, sich die Ergebnisse "zurecht zu rücken", um ihr anfürsich gutes Produkte durch eine vollkommen falsche Interpretation einer einzigen Anwendung nicht ins Abseits manövrieren zu lassen. Gerechtfertig ist ein solches Verhalten dadurch jedoch noch lange nicht.

Ein Download des letzten 3DMark03 Patch ist bei Futuremark oder direkt bei uns möglich.