Nearly 18 months ago Futuremark released 3DMark03 - the latest in the line of 3D performance benchmarks and their first to make use of DirectX9 technologies. While, internally, they probably realised that not all was happy in in the Beta members (the group of members that steer the benchmark - which includes the IHV's who's products will be tested by the benchmark) camp, its doubtful that anyone would have expected the rough ride that was to ensue over disagreements with what the benchmark tests and overzealous optimisations coded into drivers.

After some time all parties patched up the majority of their differences and the work on the next release of 3DMark was allowed to progress relatively unfettered. Over the past few months Futuremark have been busy putting together the 3DMark04 and they are giving us some small insights as to what we may see when the benchmark is ultimately released.

We've fired a bunch of questions at Futuremark and Nicklas Renqvist, Patric Ojala, and Tero Sarkkinen gives us some answers...

3DMark is billed as a gamers benchmark, and is designed to offer a look into the future of the types of  3D hardware feature use and applications of those features. With 3DMark2001 and the use of the Max Payne engine it was easy for people to see a correlation between the benchmark and an actual application, however with 3DMark03 this was obfuscated without any direct links to engines. How do you decided what features to use and how they should be applied - is this all done in-house, or through canvassing of other developers? What specific research has been taken for 3DMark04?

Nick: Before we start to develop a new version of 3DMark, we have done a lot of research in advance. Our research consists of talking to other developers, IHVs (our Benchmark Development Program members), studying available tech-papers, looking & testing new and innovative tech-demos and so on. All this combined gives us a pretty good picture of where the real-time 3D is heading, and what features are a “must” in the next generation engines. This is exactly we did in 3DMark03. Our research told that the stencil-shadow technique was going to be used in future games, so we implemented that feature into 3DMark03. So far, there are only a handful of games using stencil-shadows, but there will be more and big titles coming out using about the same shadow technique.

When we are satisfied with the new ideas we would like to use, we start planning the game-tests and which features we could use in the various tests. Some features never make it into the engine, but they aren’t forgotten. If they are good enough, they will find their way into the next version of the benchmark. There is only so much we can squeeze into the benchmark while keeping an acceptable development time.

The research for the next 3DMark has been in the works ever since 3DMark03 was released in February 2003. We are always open for discussion with game developers on future innovations and what techniques will be used in next generation game-engines. Of course our BDP members play a very big role when we plan our next 3DMark versions, as do various tech-papers and forward looking tech-demos. BDP members, for example, share their technology roadmaps with us and that helps us determine which types of tests are the most valid during the benchmark’s life.

Patric: Commenting on the ‘gamers benchmark’ slogan is really a marketing issue, and maybe Tero could add something about that. From maybe a less PR perspective, I think the slogan still fits very nicely. We do indeed aim to measure game performance, but our focus is on the 3D performance of games. This is an informed decision, since graphics performance tends to increase faster than CPU performance, and all benchmarks in time become system (meaning usually CPU) bound anyway. Game benchmarks are in general heavily system bound and/or present current or past generation graphics technology. So there clearly is a gap in the range of available benchmarks. No other benchmark measures 3D performance of future games, and that’s where 3DMark is aimed.

As for the engine choice, MAX-FX was an excellent engine as long as 3DMark just measured fixed function multi-texturing performance (’99, 2000 and mostly 2001). The introduction of shader technology made such massive middleware blocks unneeded from 3D performance measurement point of view. The middleware blocks are needed only for legacy hardware code paths. Since 3DMark is only forward looking without legacy code paths, we had no need for MAX-FX type technology anymore in 3DMark03. Game test 1 indeed still does multi-texturing, but only two code paths were needed because of the content – dual texturing and quad texturing paths. So there was no reason to add a whole engine just to get two multi-texturing code paths.

The new 3DMark engine is more of a ‘game engine’ than the one in 3DMark03. That engine was a first generation shader engine for a smaller variety of shaders. It just loaded pre-compiled shaders in shader language and the artwork. The new 3DMark engine is a next generation shader engine, with a wide variety of shaders dynamically built and runtime compiled. This is how we believe future game engines will work, and some developers have already confirmed it.

Tero: The Gamers’ Benchmark term has been associated with 3DMark from the very beginning and also continues to be so. The reason has not changed over the time: 3DMark is a tool we make for gamers with their needs on our mind. It helps gamers to objectively assess today how today’s technology will perform in games of tomorrow, i.e. to get important data points of how ‘future proof’ their investment is.