The Nightmare That Is Reviewing

Here at Beyond3D, we had long ago (from its inception, actually) decided that we must find a way of determining if the evangelisms by the IHVs are relevant at all, especially when we're talking about the latest technologies that won't be featured in games for at least a year after the debut of the latest video chip. We feel that it is our duty to see if the "wow" factor usually associated with such evangelisms is justified or not. Taking the current state of the industry as an example, both ATI and NVIDIA have made much noise about the DirectX9 (DX9) features of their respective DX9 hardware (apologies if that sounded rather silly). "High Precision for cinematic quality", "Thousands of vertex and pixel shader instructions", etc. etc. are claimed. Are such claims deceiving? How would the hardware perform with 32-bit floating point or a multitude of shader instructions? More importantly, what do we use to investigate performance?

The author of this article, after feeling a bit depressed about the state of the hardware reviewing industry lately, decided that while most of the public are appear to be generally satisfied with the information provided in the majority of reviews by various hardware review outlets (including this one), what do the people behind the games, the developers, think of video card reviews? Surely reviews matter to them because reviews influence the sales of hardware and what hardware gets sold the most influence the way they develop their games.

Instead of going on and on, the reason for this short article can be explained by an email this reviewer sent to a few developers recently, with the following content :

As you may well know, the hardware (primarily video card) reviewing industry is more or less caught in a conundrum. Actually, it's been this way for the longest time, probably originating with the introduction of the original GeForce. I mention the GeForce as the prime example of this caught-between-a-rock-and-a-hard-place because it was the first affordably priced video card to expose hardware T&L and hence all the evangelism by NVIDIA that followed it and all its successors. What's the tough situation that the hardware reviewing industry faces? That of commending the IHVs for coming out with the latest-and-greatest while trying to lend some relevance of this latest technology to current games.

One of the things John Carmack said in CGW's very recent article made this all the more succinct. "First generation hardware really isn't for consumers, it's for the developers.", he said.

So, anyway... I guess that it all comes down to what a particular hardware review outlet focusses on. The majority do tend to review a video card using what's available (games that are available, i.e.) because, well, what can they do? My site tend to focus a bit more on the technologies that a particular IHV evangelize or promote. Evangelism that targets not the developers, but the public. For what better way to inform the public than to tell them if what is being promoted is relevant for that particular video card or not? Of course, my site will still use existing games to provide information/benchmark because it still needs to be done, but we also think that various specific synthetic benchmarks also play an extremely important role in educating the public about whether to believe the hype or not, and many hardware review outlets do not have the same view as ours (they lambast the use of synthetic benchmarks as they claim they are not relevant).

But I'm getting more and more frustrated by the limitations that are forced on the hardware reviewing industry. Getting to test/benchmark upcoming games is a improbable possibility, although it should be that much better than testing currently available games on the latest hardware.

So... I'd appreciate it if you could advise me if my site is going about the right way, providing a good mix of synthetic benchmarks that investigates the performance of the latest video card in specific feature tests in conjunction with using existing games to see if various games gets some performance boost (especially with any anti-aliasing and/or anisotropic filtering improvements, both features that can be appreciated by gamers "out-of-the-box").

Is there a better way of reviewing video cards? What should the reviewing focus be on -- how well the latest video cards run existing games (that almost always do not feature technologies found in the latest video cards), or how well the latest video cards perform in the very features/technologies its creators evangelize?

The next page tells us what developers think should be "The Way It's Meant To Be Reviewed" (okay, so we couldn't resist the play on NVIDIA's marketing slogan), and not exactly about whether Beyond3D is writing reviews the right way.