Lies, damned lies and benchmarks

Posted:

Ye Olde slashdot recently ran an article about the tiny web server Boa. That lead me to look at its obligatory benchmark page were it compares favorably to apache. The issue of web server benchmarks as become of interest to me as I work on my game State Secrets, which is has a single-threaded all Perl web server. I haven’t been bowled over by the performance of SS as a web server (I didn’t build it for speed) and some pages seem to take far too long to render.

Enter apache’s benchmarking tool ab.

I present the results of two benchmarking runs. The benchmarking software was run on the same machine as the server (I realize there are inherent problems with this). The first set of numbers represents static content service. The second set of number is for dynamic content. There is an order of magnatude difference between these numbers, as one would expect.

What, if anything, do you make of these numbers? To me, it seems the server is performing pretty well. Of course, my expectations are pretty low. I don’t expect to support more than 4 simultaneous users.

static content

invocation: ab http://marian:8080/ -n 1000 -c 12 Server Software: State-Secrets-server/0.1 Server Hostname: marian Server Port: 8080

Document Path: / Document Length: 2752 bytes

Concurrency Level: 12 Time taken for tests: 15.477 seconds Complete requests: 1000 Failed requests: 0 Broken pipe errors: 0 Total transferred: 2935000 bytes HTML transferred: 2752000 bytes Requests per second: 64.61 [#/sec] (mean) Time per request: 185.72 [ms] (mean) Time per request: 15.48 [ms] (mean, across all concurrent requests) Transfer rate: 189.64 [Kbytes/sec] received

Connnection Times (ms) min mean[+/-sd] median max Connect: 0 1 6.7 0 85 Processing: 61 183 29.6 197 330 Waiting: 7 183 30.1 196 330 Total: 61 184 27.3 197 330

Percentage of the requests served within a certain time (ms) 50% 197 66% 199 75% 200 80% 201 90% 203 95% 232 98% 235 99% 245 100% 330 (last request)

dynamic content

invocation: ab 'http://marian:8080/game?function=hq&tid=ff46fa784a60d86029abca19d204ed17' -n 1000 -c 12 Server Software: State-Secrets-server/0.1 Server Hostname: marian Server Port: 8080

Document Path: /game?function=hq&tid=ff46fa784a60d86029abca19d204ed17 Document Length: 3019 bytes

Concurrency Level: 12 Time taken for tests: 35.151 seconds Complete requests: 1000 Failed requests: 0 Broken pipe errors: 0 Total transferred: 3131000 bytes HTML transferred: 3019000 bytes Requests per second: 28.45 [#/sec] (mean) Time per request: 421.81 [ms] (mean) Time per request: 35.15 [ms] (mean, across all concurrent requests) Transfer rate: 89.07 [Kbytes/sec] received

Connnection Times (ms) min mean[+/-sd] median max Connect: 0 1 4.9 0 64 Processing: 46 419 35.7 427 499 Waiting: 1 418 36.5 426 499 Total: 46 420 32.7 427 499

Percentage of the requests served within a certain time (ms) 50% 427 66% 428 75% 429 80% 429 90% 432 95% 445 98% 465 99% 468 100% 499 (last request)

[Original use.perl.org post and comments.]