Performance Analysis/Benchmarks

As the author of Barebones CMS, I'm deeply concerned with the performance of the product. It should, first and foremost, be capable of withstanding Slashdot, Digg, Reddit, etc. "attacks" where numerous visitors are hitting the site every second.

I could care less how Barebones CMS stacks up against other CMS products. Since it is drastically different from other CMS products at the very core, a comparison chart between this and any other CMS product would be foolish for a number of reasons:

This comparison, therefore, consists of five simple tests comparing response times of content from Barebones CMS to straight-up HTML. Obviously, no serious product in PHP can compare to the lightning fast speed of plain-ol' HTML, so this should be a relatively fair comparison.

Hardware and Software

The hardware in use is a Intel Q9550 Core2 Quad @ 2.83GHz (each core runs at something like 333MHz according to the BIOS), 4GB RAM @ 1GHz (in theory but might be @ 800MHz), and Western Digital Caviar 640GB SATA 3.0Gb/s hard drive @ 7200RPM.

The software in use is Windows XP Professional SP3, Apache 2.2.13, ApacheBench 2.3, and PHP 5.2.13 as an Apache module. This system does not use any PHP caching modules (e.g. APC, Zend Optimizer, etc).

Basically, a decent system for personal use in 2010. I'm sure we'll laugh at these specs in a few years.

Tests

Every test is done on 'localhost' to eliminate the Internet aspect from the picture.

The ApacheBench command used for all tests is: 'ab -n 50000 -t 30 -c 10 URL' (without the quotes).

Test: Serving up content that is permanently cached

For this test, the end of the fourth video here is used as the content. It is a page with a Content widget with some basic content including one image (not used for this benchmark) and a detached Layout widget.

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 5000 requests
Completed 10000 requests
Completed 15000 requests
Completed 20000 requests
Completed 25000 requests
Completed 30000 requests
Completed 35000 requests
Completed 40000 requests
Completed 45000 requests
Completed 50000 requests
Finished 50000 requests


Server Software:        Apache/2.2.13
Server Hostname:        localhost
Server Port:            80

Document Path:          /test/
Document Length:        1073 bytes

Concurrency Level:      10
Time taken for tests:   27.000 seconds
Complete requests:      50000
Failed requests:        0
Write errors:           0
Total transferred:      66200000 bytes
HTML transferred:       53650000 bytes
Requests per second:    1851.85 [#/sec] (mean)
Time per request:       5.400 [ms] (mean)
Time per request:       0.540 [ms] (mean, across all concurrent requests)
Transfer rate:          2394.39 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   1.4      0      16
Processing:     0    5   7.4      0      31
Waiting:        0    5   7.3      0      31
Total:          0    5   7.4      0      31

Percentage of the requests served within a certain time (ms)
  50%      0
  66%     16
  75%     16
  80%     16
  90%     16
  95%     16
  98%     16
  99%     16
 100%     31 (longest request)

This test reveals that Barebones CMS can serve about 1,800 requests per second. I also ran a quick test to see what 100 concurrent connections would look like and got 1713.98 requests per second. The caching system in Barebones CMS is really robust. But now let's blow your mind with the next test.

Test: Serving up the same content as plain HTML

This next test is a raw performance comparison to serving up the exact same content as through Barebones CMS. The Barebones CMS caching system dumps the output HTML into the same directory as the PHP files, which makes this comparison super easy to do - just point at the HTML file instead of the PHP file.

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 5000 requests
Completed 10000 requests
Completed 15000 requests
Completed 20000 requests
Completed 25000 requests
Completed 30000 requests
Completed 35000 requests
Completed 40000 requests
Completed 45000 requests
Completed 50000 requests
Finished 50000 requests


Server Software:        Apache/2.2.13
Server Hostname:        localhost
Server Port:            80

Document Path:          /test/index_en_us.html
Document Length:        1073 bytes

Concurrency Level:      10
Time taken for tests:   9.281 seconds
Complete requests:      50000
Failed requests:        0
Write errors:           0
Total transferred:      67350000 bytes
HTML transferred:       53650000 bytes
Requests per second:    5387.21 [#/sec] (mean)
Time per request:       1.856 [ms] (mean)
Time per request:       0.186 [ms] (mean, across all concurrent requests)
Transfer rate:          7086.49 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   1.2      0      16
Processing:     0    2   4.9      0      16
Waiting:        0    2   4.8      0      16
Total:          0    2   5.0      0      16

Percentage of the requests served within a certain time (ms)
  50%      0
  66%      0
  75%      0
  80%      0
  90%     16
  95%     16
  98%     16
  99%     16
 100%     16 (longest request)

This test reveals that Apache can serve approximately 5,300 requests per second. I ran the same quick tests against 100 concurrent connections and got 4610.95 requests per second. Basically, out-of-the-box Barebones CMS is almost three times slower than plain HTML.

Test: An empty PHP file that does nothing

Of course, this brings up the interesting question of - is it Barebones CMS or PHP that causes the slowness? This test uses a completely empty PHP file (0 bytes) to find out.

This is ApacheBench, Version 2.3 <$Revision: 1554214 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 5000 requests
Completed 10000 requests
Completed 15000 requests
Completed 20000 requests
Completed 25000 requests
Completed 30000 requests
Completed 35000 requests
Completed 40000 requests
Completed 45000 requests
Completed 50000 requests
Finished 50000 requests


Server Software:        Apache/2.4.9
Server Hostname:        localhost
Server Port:            80

Document Path:          /test/empty.php
Document Length:        1 bytes

Concurrency Level:      10
Time taken for tests:   9.800 seconds
Complete requests:      50000
Failed requests:        0
Total transferred:      9350000 bytes
HTML transferred:       50000 bytes
Requests per second:    5102.03 [#/sec] (mean)
Time per request:       1.960 [ms] (mean)
Time per request:       0.196 [ms] (mean, across all concurrent requests)
Transfer rate:          931.72 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   1.0      0      10
Processing:     0    2   3.9      0      30
Waiting:        0    1   3.5      0      20
Total:          0    2   3.9      0      30

Percentage of the requests served within a certain time (ms)
  50%      0
  66%      0
  75%      0
  80%      0
  90%     10
  95%     10
  98%     10
  99%     10
 100%     30 (longest request)

So, yes, Barebones CMS is slower. Keep in mind though that the CMS is doing a lot of stuff though to determine if the cached file needs to be updated or the cache is being bypassed and what language/translation of the content to serve up. Also keep in mind that 1,700 pages per second under heavy load is still fairly impressive. Also keep in mind that all of this is being done without any PHP optimizers.

Test: Serving up content that is never cached

For the last two tests, there are some people who will want to know what sort of performance hit there might be on content that is never cached or the cache gets refreshed frequently. This test is run against the same content but without caching it.

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 5000 requests
Finished 7287 requests


Server Software:        Apache/2.2.13
Server Hostname:        localhost
Server Port:            80

Document Path:          /test/
Document Length:        1073 bytes

Concurrency Level:      10
Time taken for tests:   30.000 seconds
Complete requests:      7287
Failed requests:        0
Write errors:           0
Total transferred:      9203481 bytes
HTML transferred:       7818951 bytes
Requests per second:    242.90 [#/sec] (mean)
Time per request:       41.169 [ms] (mean)
Time per request:       4.117 [ms] (mean, across all concurrent requests)
Transfer rate:          299.59 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   2.6      0      31
Processing:    16   40  12.2     47     125
Waiting:       16   40  12.0     47     125
Total:         16   41  12.1     47     125

Percentage of the requests served within a certain time (ms)
  50%     47
  66%     47
  75%     47
  80%     47
  90%     47
  95%     63
  98%     63
  99%     78
 100%    125 (longest request)

This test reveals how important the Barebones CMS caching system is. The performance drops like a rock to 242 requests per second and doesn't complete the test within the 30 second time frame.

Test: Serving up content on a 5 second cache

This test keeps the content in a five second cache.

This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 5000 requests
Completed 10000 requests
Completed 15000 requests
Completed 20000 requests
Completed 25000 requests
Completed 30000 requests
Completed 35000 requests
Completed 40000 requests
Completed 45000 requests
Completed 50000 requests
Finished 50000 requests


Server Software:        Apache/2.2.13
Server Hostname:        localhost
Server Port:            80

Document Path:          /test/
Document Length:        1073 bytes

Concurrency Level:      10
Time taken for tests:   26.938 seconds
Complete requests:      50000
Failed requests:        7
   (Connect: 0, Receive: 0, Length: 7, Exceptions: 0)
Write errors:           0
Total transferred:      66192089 bytes
HTML transferred:       53645093 bytes
Requests per second:    1856.15 [#/sec] (mean)
Time per request:       5.388 [ms] (mean)
Time per request:       0.539 [ms] (mean, across all concurrent requests)
Transfer rate:          2399.66 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   1.4      0      16
Processing:     0    5   7.5      0      63
Waiting:        0    5   7.4      0      63
Total:          0    5   7.5      0      63

Percentage of the requests served within a certain time (ms)
  50%      0
  66%      0
  75%     16
  80%     16
  90%     16
  95%     16
  98%     16
  99%     16
 100%     63 (longest request)

This test reveals that even a small five second cache can result in a huge performance boost. Of all the tests, this one had a few failed replies. This is probably due to the fact that there are requests coming in for the cached file while a new cached file is being written out to the file system while the system is under load. So the rule of thumb should be to use a reasonable cache time for the page.

Test Results

Barebones CMS without caching is rather sluggish but still performs reasonably well. Even with caching, it is still three times slower than both HTML and an empty PHP file. It is also likely slower than very basic PHP. However, it seems to be quite capable of weathering a storm of users hitting the site. Handling up to 1,700 requests per second under load on a single box without doing any optimizations is fairly impressive.

I leave any additional comparisons (e.g. against other CMS products, PHP optimizers, etc) up to you. My goal was to show how big of a performance hit the out-of-the-box Barebones CMS product is on most systems to let you make informed decisions about your website development strategy.

© CubicleSoft