Zbr's days.

About :: TODO :: Blog :: RSS :: Old blog :: Projects :: GIT :: Gallery :: Notes

Sat, 21 Jan 2006

FreeBSD kqueue benchmark.

FreeBSD is a good system, but either it's network stack behaves worse than Linux or, more likely, I have not configured it right, but under load FreeBSD kqueue based server behaves noticebly worse than Linux kevent one.

This set of tests was designed to compare FreeBSD kqueue latency with Linux' epoll() and kevent subsytems.

Hardware and software.
Server: Xeon 2.4 GHZ, HT disabled, 1Gb RAM, 1Gb Intel e1000 adapter. FreeBSD 6.0-RELEASE. No firewall.
Client: AMD64 3500+ 2.2 Ghz, 1Gb RealTek 8169 adapter. Linux FC4. httperf-0.8.
D-Link DGS-1216T gigabit switch.

Test description.
evserver_kqueue.c is a kqueue based simple static web server, which uses read()/send() for data transfer in FreeBSD, since it's sendfile() interface is horrible. It runs with root privileges to eliminate various user's limits in *nix systems.

First test.
Run 10k connection one-by-one:

httperf --client=0/1 --server=freebsd --port=80 --uri=/ --send-buffer=4096 --recv-buffer=16384 --num-conns=10000 --num-calls=1
Maximum connect burst length: 1

Total: connections 10000 requests 10000 replies 10000 test-duration 9.667 s

Connection rate: 1034.4 conn/s (1.0 ms/conn, <=1 concurrent connections)
Connection time [ms]: min 0.3 avg 1.0 max 220.6 median 0.5 stddev 3.8
Connection time [ms]: connect 0.1
Connection length [replies/conn]: 1.000

Request rate: 1034.4 req/s (1.0 ms/req)
Request size [B]: 64.0

Reply rate [replies/s]: min 1013.9 avg 1013.9 max 1013.9 stddev 0.0 (1 samples)
Reply time [ms]: response 0.4 transfer 0.4
Reply size [B]: header 198.0 content 3931.0 footer 0.0 (total 4129.0)
Reply status: 1xx=0 2xx=0 3xx=0 4xx=10000 5xx=0

CPU time [s]: user 2.30 system 7.06 (user 23.8% system 73.1% total 96.9%)
Net I/O: 4235.6 KB/s (34.7*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

Request rate is 1034 requests per second, when Linux in the same hardware (it had only 512 Mb of RAM but it does not matter) and server software (kevent based server used read()/send() syscalls) showed 1700 requests per second. Strange thing is that even Apache on FC4 run faster.


Second test.
Run several simultaneous connections with different burst size.

1. 10k connections with maximum 1k connections in a burst with 1 second timeout.
httperf --timeout=1 --client=0/1 --server=freebsd --port=80 --uri=/ --rate=1000 --send-buffer=4096 --recv-buffer=16384 --num-conns=10000 --num-calls=1
Maximum connect burst length: 62

Total: connections 10000 requests 10000 replies 9733 test-duration 10.000 s

Connection rate: 1000.0 conn/s (1.0 ms/conn, <=64 concurrent connections)
Connection time [ms]: min 0.1 avg 1.0 max 15.7 median 0.5 stddev 0.9
Connection time [ms]: connect 0.2
Connection length [replies/conn]: 1.000

Request rate: 1000.0 req/s (1.0 ms/req)
Request size [B]: 64.0

Reply rate [replies/s]: min 970.3 avg 970.3 max 970.3 stddev 0.0 (1 samples)
Reply time [ms]: response 0.4 transfer 0.4
Reply size [B]: header 198.0 content 3931.0 footer 0.0 (total 4129.0)
Reply status: 1xx=0 2xx=0 3xx=0 4xx=9733 5xx=0

CPU time [s]: user 2.46 system 7.16 (user 24.6% system 71.6% total 96.2%)
Net I/O: 3987.1 KB/s (32.7*10^6 bps)

Errors: total 267 client-timo 0 socket-timo 0 connrefused 0 connreset 267
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

As we see here, request rate is higher for FreeBSD (1000 vs 910), but number of errors is higher too.

2. 20k connections with maximum 2k connections in a burst with 1 second timeout.
httperf --timeout=1 --client=0/1 --server=freebsd --port=80 --uri=/ --rate=2000 --send-buffer=4096 --recv-buffer=16384 --num-conns=20000 --num-calls=1
Maximum connect burst length: 4

Total: connections 20000 requests 20000 replies 14371 test-duration 10.000 s

Connection rate: 2000.0 conn/s (0.5 ms/conn, <=6 concurrent connections)
Connection time [ms]: min 0.2 avg 1.0 max 2.5 median 0.5 stddev 0.1
Connection time [ms]: connect 0.2
Connection length [replies/conn]: 1.000

Request rate: 2000.0 req/s (0.5 ms/req)
Request size [B]: 64.0

Reply rate [replies/s]: min 1534.9 avg 1534.9 max 1534.9 stddev 0.0 (1 samples)
Reply time [ms]: response 0.4 transfer 0.4
Reply size [B]: header 198.0 content 3931.0 footer 0.0 (total 4129.0)
Reply status: 1xx=0 2xx=0 3xx=0 4xx=14371 5xx=0

CPU time [s]: user 2.32 system 7.68 (user 23.2% system 76.8% total 100.0%)
Net I/O: 5919.8 KB/s (48.5*10^6 bps)

Errors: total 5629 client-timo 0 socket-timo 0 connrefused 0 connreset 5629
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

Very good request rate for FreeBSD, but very big number of "connection reset" errors.

3. 30k connections with maximum 3k connections in a burst with 1 second timeout.
httperf --timeout=1 --client=0/1 --server=freebsd --port=80 --uri=/ --rate=3000 --send-buffer=4096 --recv-buffer=16384 --num-conns=30000 --num-calls=1
Maximum connect burst length: 285

Total: connections 30000 requests 29964 replies 18608 test-duration 10.000 s

Connection rate: 2999.9 conn/s (0.3 ms/conn, <=329 concurrent connections)
Connection time [ms]: min 0.4 avg 1.5 max 105.2 median 1.5 stddev 0.9
Connection time [ms]: connect 0.4
Connection length [replies/conn]: 1.000

Request rate: 2996.3 req/s (0.3 ms/req)
Request size [B]: 64.0

Reply rate [replies/s]: min 1709.9 avg 1860.6 max 2011.3 stddev 213.2 (2 samples)
Reply time [ms]: response 0.6 transfer 0.6
Reply size [B]: header 198.0 content 3931.0 footer 0.0 (total 4129.0)
Reply status: 1xx=0 2xx=0 3xx=0 4xx=18608 5xx=0

CPU time [s]: user 1.53 system 8.38 (user 15.3% system 83.8% total 99.1%)
Net I/O: 7690.1 KB/s (63.0*10^6 bps)

Errors: total 11392 client-timo 40 socket-timo 0 connrefused 0 connreset 11352
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

Excellent request rate for FreeBSD (3000 vs. 2500 in Linux with sendfile()), but enormous number of errors, and only 62% of connection requests were successfully finished.


Conclusion.
After various sysctls have been changed (sysctl -a output is available here) things become slightly better (btw, default FreeBSD installation does not allow such tests at all due to default network parameters), but number of "connection reset" errors is still very high.
FreeBSD drops too many connections due to either misconfiguration or lack of resources.

According to FreeBSD and Linux comparison, in Linux number of connection errors is much smaller than in FreeBSD with comparable or bigger requests rate.

:: Link / Comments ()