NT beats Linux ... maybe

NT beats Linux ... maybe

Study finds NT is faster than Linux as a Web server...at least according to MS-sponsored tests.

By Steven J. Vaughan-Nichols
ZDNet

April 15, 1999

It's so simple -- at least on the face of it. Mindcraft Inc., in a new study commissioned by Microsoft Corp., found that "Microsoft Windows NT Server 4.0 is 2.5 times faster than Linux as a File Server and 3.7 times faster as a Web Server."

In reality, however, it's really not simple at all.

Mindcraft, an independent test lab, pitted the two network operating systems against each other on a Dell PowerEdge 6300/400 to test both NT 4.0 SP 4 and Red Hat Linux 5.2 upgraded to the Linux 2.2.2 kernel. This quad 400MHz Xeon symmetric multiprocessor system (SMP) had a whopping 4 gigabyte of memory (a gigabyte of which was used in the test) and a PowerEdge RAID II Adapter, 32 MB cache set to RAID 0. The tests used ZD's own NetBench and WebBench benchmark tests for file and Web serving tests.

As tested, NT beat Linux like a drum. But, that's not the whole story.

Singing the right tune

Mindcraft tested NT with NT tuning, benchmarking and technical support from Microsoft, and Internet Information Server (IIS) 4.0 tuning information from the Standard Performance Evaluation Corp. In the case of Linux, though, Mindcraft's President, Bruce Weiner was "surprised that Red Hat Software Inc. didn't help us in tuning the OS."

Red Hat spokesperson Melissa London explains that while, "There was a query to our contract technical support but they didn't know what to do with Mindcraft's request. Had the question came through the right channels, the public relations or the engineering staff, we're sure Linux would have, at least, have had a better representation."

In the future, Red Hat "would love to repeat the tests with our input, support, and involvement," London adds.

In addition, Mindcraft says it found "little real tuning information in the Linux knowledge bases." What information they did find was scattered hither and yon over the Web. And, for Mindcraft, at least, the "word of mouth" Linux support of newsgroups, mailing lists and online conference sites didn't work.

That said, Jeremy Allison, a leading member of the Samba Team -- the open-source group that developed a product that lets Unix boxes look like NT file servers -- checked his records and says he can find no trace of anyone from. Indeed, after looking over the Mindcraft's published Linux parameters, he declares, "They [Mindcraft] didn't do any tuning on the Linux side," says Allison.

Consequently, Mindcraft has found itself buried in Linux tuning "suggestions." Many of these are just flames from outraged Linux users, but many of them, as Weiner admits, are well founded.

For example, Mindcraft turned the Samba parameter, 'widelinks' off. But, according to Andrew Trigell, the Samba Team leader, by setting widelinks off, this "lower(s) the performance enormously. It adds 3 chdir() and 3 getwd() calls to every filename lookup. That will especially hurt any SMP [symmetric multiprocessing] system."

Mindcraft's Weiner agrees: "Clearly, we wouldn't make that tune if we were doing that test today."

The result? Highly tuned NT beats a barely tuned, if that, Linux. But, there's more to it than that.

For one thing, there's a useful lesson here for Linux resellers and integrators. Linux tuning information is hard to gather up. Yes, it's there. But, it really requires an expert Linux user to find it. If Linux is to win commercially, information known only to Linux gurus and hidden away in a dozen Web sites needs to be gathered together and made more readable and approachable. For new users, learning advanced Linux is still much too difficult.

Take Apache, for example. There are several files, off the Apache Project's Web site on how to tune Apache and Linux. Even if you know they're there, though, these files aren't easy to find. On the flip side, finding information on tuning Internet Information Server 4.0 is easy at Microsoft's knowledgebase.

Beyond Tuning: Incompatibilities

For Linux, SMP is a new option, incorporated in the 2.2 release. This is not the current base OS release that Red Hat is shipping today or that was used in the test. Instead, the system as upgraded to 2.2 and SMP activated by inexperienced Linux administrators using early Red Hat documentation. Moreover, the Linux RAID driver was a beta release and not well tested with the "so new the shiny hasn't worn off" 2.2 kernel.

As a source close to Ziff-Davis Benchmark Operations, the ZD benchmark's authoring agency, said of Mindcraft's testing, "A more accurate headline might have been Linux does not scale up" instead of "Windows NT Server Outperforms Linux."

Sm@rt Reseller's own tests, using the same benchmarks, on low-end, uniprocessor machines, has shown that Linux combined with Apache and Samba, beats NT and IIS handily. (See Linux Up Close: Time To Switch and Samba 2.0: A License to Kill NT).

That said, it's likely that a highly tuned NT, on the Mindcraft SMP platform, would prove a bear for even a tuned Linux to defeat. For NT, this kind of machine is a home field. For Linux, at this point in its development, it's an away field, far, far from home.

On single processor systems, Linux wins. On SMP/RAID systems, Linux is just now getting in the game. Even so, even at the high end of stress testing, Linux's numbers would have been much closer to NT, if not higher, had Linux been as well tuned as NT.