Ubuntu Hardy vs Windows XP SP3: Benchmarks

En español, tomado de:

Ubuntu 8.04 LTS vs. Windows XP SP3: Application Performance Benchmark

Two months ago, I wanted to compare Windows XP to Ubuntu Linux in terms of applications performance. I thought that since most of Linux programs are cross-platform and available for Windows, it could be a good idea to see how only platform change can affect the performance of a particular application. One of my reasons was also to verify whether or not Linux is capable of getting the most out of new hardware technologies. However, I only had a Pentium 4 HT machine, and even though I went ahead with the test, I knew it was not going to answer that.

A lot has happened since; Ubuntu 8.04 came out, and I was happy with it. Microsoft finalized their third service pack for Windows XP, and best of all I upgraded my PC with an AMD Athlon64 X2 5600+ CPU, 2GB DDR2-800 RAM and a GeForce 8600GT VGA card.

I decided to test again, especially after I read in the news that Microsoft has made clear its intention to take on Linux in the low-end computers market. Microsoft will exclusively rely on Windows XP for that purpose, cutting down its price by more than a half, and building on the extreme popularity of the 7-years-old operating system.

Recently, Microsoft invaded Asus Eee MIDs and the OLPC project which once thought to be Linux best chance of world-wide popularity. This gives us an idea that Microsoft is well aware of Linux threat and has started to take actions to protect its monopoly.

In fact, I see a true threat because Windows XP is very popular, has the best compatibility of any OS, mature enough to be reliable and also known to be fast and very solid in both gaming and productive domains.

Of course, Linux has many advantages. It more secure, stable and efficient and it is capable of bringing cutting-edge technologies to low-end machines, but there will be much less incentive to switch to Linux from a cheap Windows XP than from an expensive and complicated Windows Vista.

All of this adds another purpose of my test. I’ll try to discover how a 2008 edition of Ubuntu Linux will perform against the 2001-born Windows XP fortified with 7 years of bug-fixing and 3 Service Packs.

Before I start with the results, I’d like to say that even though I tried my best to be accurate and academic, I don’t consider my test conclusive. There are several things that had to be better and plenty of other factors to consider. However, I can safely say that the test results are indicative and grossly reproducible.

Let’s start with some command line programs. RAR is the backbone of the popular WinRAR, and it is also available for many platforms. I used the command line version because it is exactly the same for both Linux and Windows.

My test showed that Windows XP SP3 was tangibly faster in creating RAR compressed archives (by about 25%) while extracting compressed data was much faster on Ubuntu 8.04. While I am pretty sure that RAR compression is more about the CPU and RAM than the hard drive performance, I am not sure if the faster decompression on Ubuntu was because of faster EXT3 write times or a faster algorithm execution.

Next was ClamAV anti-virus scanner. It’s free and relatively popular on Linux (although most Linux users don’t use any AV software), but it is also available for Windows. There are several GUI projects for ClamAV, but I used the original command line version.

Ubuntu had a small advantage here, but I am sure that it resulted from the difference between NTFS and EXT3 file systems.

GIMP photo-editing program was next, as the first of three GUI-fitted applications in my test.

Two month ago, I noticed that the Windows version of GIMP could be very slow especially when the script required excessive screen re-drawing (refreshing). This time I tried to evade this problem by keeping GIMP windows minimized while executing my test script. However, I couldn’t make GIMP perform faster on Windows XP. It was 25% slower than its Ubuntu counterpart.

Blender is one of the most important 3D-modeling software around. It’s very powerful and absolutely free. In my test, I used 3 sample projects downloaded from the application’s website, and measured the time needed to render 25 frames of each one.

Windows XP SP3 was a comfortable winner here, beating Ubuntu by just over 20%. This matched the results from my old test, and since 3D rendering is mainly about the CPU (system RAM is also involved but mainly in its capacity rather than speed), I began to develop suspicions about Ubuntu’s ability to take advantage of modern CPU’s capabilities (like SSE3, SSE4). It would explain RAR compression performance and also the next test which was video encoding.

For Video Encoding, I chose Avidemux. With the right media codecs installed, it can pretty much encode to any video format. I chose x264 as a target (the same video format of Hi-Def videos) and both MPEG2 (standard DV) and MJPEG as sources (The most common standard-definition formats).

Again, Windows XP SP3 won that one with ease, with just about the same 20% speed margin over Ubuntu 8.04. This consolidated a clear advantage in multi-media processing for Windows XP. One possible justification for this is that Ubuntu 8.04 Desktop Edition ships with a generic kernel that can work on any x86 platform. Linux kernel has many incarnations which are more specific and can perform better on a particular hardware. However, changing the default kernel can cause stability problems. Many Linux experts recommend compiling a custom kernel for every machine to take full advantage for its potentials, but this requires technical knowledge and, if you are new to Linux, a brave heart.

The next test was an important one. It was the multi-tasking performance test. I wrote a script to execute the above-mentioned tests (some partially) simultaneously. Of course, they would finish in different times, and therefore, the shorter tests suffered the most while Blender, for example, stayed alone for more than 6 minutes after everything else was done. However, most of the tests were close in duration, so despite the lack of accuracy, the test had a very good value in showing how well each operating system could handle multi-tasking.

Apparently, there was no contest at all between Windows XP and Ubuntu when it came to multi-tasking. The average increase of test duration between mono- and multi-tasking was about 300% with Ubuntu 8.04, while it was more than seven folds for Windows XP. It seems as if the CFS (completely fair scheduler) of the newer Linux kernel has paid off.

I always say that multi-tasking tests are of high value because they represent real-life situations. Of course, my test was an extreme example and it was more of a stress test, but even with lighter workloads, and especially with such a huge difference between Windows XP and Ubuntu, I think that Ubuntu should be the real-life performance leader. Many people talk about perceived performance, in contrast to benchmark performance, and some of them stress that multi-tasking capacity is the most important factor in perceived performance.

Here we take a look on some details about mono- vs. multi-tasking tests. The graph shows the test-duration changes from single to multiple operations. In Windows XP, Avidemux and LAME MP3 encoding tests were run on a higher priority and thus suffered the least. Meanwhile, decompressing RAR archives and scanning disk folder with ClamAV were severely affected due to extreme hard disk activity, highlighting the NTFS inferiority to EXT3.

On the Linux side, UnRAR decompression was also the biggest loser (but not nearly as Windows) for the same reason; while ClamAV did much better (I couldn’t understand why). LAME and Avidemux were run with the exact same parameters as in the Windows test, but for some reason, Ubuntu wouldn’t give them as much CPU time as Windows XP did. They ended up losing as much time as any other program.

To sum-up, I’d say that Windows XP SP3 is a very solid performer and it beats the default Ubuntu in anything related to multi-media processing. This only adds to the misery of Linux in this domain as it still struggles with proprietary codecs and lacks a proper HD support. I’d like to further investigate whether Linux kernel has anything to do with this performance deficit.

On the other hand, Ubuntu is a big-time winner in multi-tasking which reflects how well it handles system resources. It also out-performs Windows in hard disk performance thanks to its support for EXT3 file system.

I’ll try to keep you updated about this subject. Thank you for taking the time to read my article.

Test Computer:

• CPU: AMD Athlon64 X2 5600+
• Board: MSI K9N Neo V3 nForce560
• RAM: 2GB Dual Channel DDR2-6400 (800)
• HDD (Windows XP): Western Digital, WD1600JD, Capacity:160GB, Cache: 8 MB, SATA150, 7200rpm.
• HDD (Ubuntu 8.04): Maxtor DiamondMax 21, STM3160215A, Capacity:160GB, Cache: 2MB, ATA100, 7200rpm.
• VGA: GeForce 8600GT
• I also disabled RAM swapping on both Windows XP and Ubuntu.
• Windows XP Professional SP2 (32bit) with SP3 on top and all updates up to 18-05-2008
• Ubuntu 8.04 LTS 32bit (hardy heron) with all security and recommended updates up to 18-05-2008

Test technical details:

• ClamAV:
  • Engine version: 0.92.1
  • Known viruses: 288170
  • Single Task:
    • Scanned directories: 1035
    • Scanned files: 16297
    • Data scanned: 3708.96 MB
  • Multi-Task:
    • Scanned directories: 15
    • Scanned files: 171
    • Data scanned: 1387.55 MB
• Avidemux 2.4.1
  • Test1: Converting 03:17 DV (MPEG2) clip to MP4 (x264 (2-pass, 1024kbps) /AAC)
  • Test2: Converting 05:32 MJPEG clip to MP4 (x264 (2-pass, 1024kbps) /AAC)
• LAME 3.97
  • Encoding Test: 44:58 495MB WAV to VBR MP3
    • lame -V 0 -m j -q 0 -s 48 –vbr-new 1.wav 1.mp3
  • Decoding Test: 44:58 72MB MP3 256Kbps VBR to WAV
    • lame –decode b.mp3 decode.wav
  • Transcoding Test: 44:58 72MB MP3 256Kbps VBR to standard MP3
    • lame -h b.mp3 b2.mp3
• RAR 3.71
  • Compression: RAR 1,516 items, totaling 565.2 MB
  • Decompression: UnRAR 481.7 MB archive
• Blender 2.45:
  • Single Task: Rendering 25 frames of each one of sample files:
    • dolphin.blend
    • group-duplicate.blend
    • FPSTemplateLightMap.blend
  • Multi Task: Rendering 25 frames of sample file FPSTemplateLightMap.blend
• GIMP 2.4.5:
  • Source Image: 2816×2112 JPEG image.
  • Single Task: Applying these filters with the default settings:
    • Filters>render>nature>IFS Fractal
    • Filters>render>Line Nova
    • Filters>render>Spyrogimp
  • Multi-Task: Applying this filter with the default settings:
    • Filters>render>nature>IFS Fractal
• Misc:
  • Tests were run on fresh installs of Ubuntu 8.04 and Windows XP SP2 with SP3 on top with only latest updates, media codecs, hardware drivers and test programs installed.
  • Desktop effects on both Windows XP and Ubuntu were ON.
  • nVIDIA-supplied VGA drivers on both (v175.16 for Windows XP and v169.12 on Ubuntu)
  • Disabled Paging/Swapping on both Windows and Linux to limit disk IO effect on CPU/RAM tests (RAR compression, GIMP, Blender, MP3 Encoding)
  • Data=Writeback Journaling on the EXT3 hard drive.
  • Disabled NTFS last-access time stamp on the NTFS hard drive.
  • All test read/wrote from/to the same hard disk partition: EXT3 for Linux and NTFS for Windows
  • Each test was repeated 3 times and the average time was calculated.