Network asynchronous IO.
About TODO Blog RSS Old blog Projects Gallery Notes

Network asynchronous IO.
This project was evolved from receiving zero-copy support.

This project is closed, true AIO will be developed instead. Expect starting after New Year vacations.

Network AIO was moved in kevent as AIO (async sendfile).

Network AIO is based on kevent and works as usual kevent storage on top of inode.
When new socket is created it is associated with that inode and when some activity is detected appropriate notifications are generated and kevent_naio_callback() is called.
When new kevent is being registered, network AIO ->enqueue() callback simply marks itself like usual socket event watcher. It also locks physical userspace pages in memory and stores appropriate pointers in private kevent structure.
Network AIO callback gets pointers to userspace pages and tries to copy data from receiving skb queue into them using protocol specific callback. This callback is very similar to ->recvmsg(), so they could share a lot in future.

Three system calls can be used as long as ioctl() based method:
  • aio_send()
  • aio_recv()
  • aio_sendfile()


I've run several benchmarks of asynchronous receiving versus stock recv().

Hardware.
Receiving side: Xeon 2.4 Ghz, HT disabled, 1Gb RAM, 1Gbps Intel 8254IPI (PCI-X 133Mhz slot) e1000 adapter.
Sending side: AMD64 3500+ 2.2 Ghz, 1Gb RAM, 1Gbps RealTek 8169 adapter integrated into nVidia nForce3 chipset (MSI K8N Neo2).
Connection: D-Link DGS-1216T gigabit switch.

Receiving software (naio_recv.c) can be found in archive. Sending software is a simple sendfile() based server.
Receiving side runs 2.6.15-rc7-event FC3 system. Default settings.
Sending side runs 2.6.15-1.1830_FC4 FC4 system. Default settings.

Results.
Client receives 1Gb of data on each of 8 runs (4 asynchronous receiving and 4 synchronous).
Each part of 4 graphs contains speed of both types and CPU usage during test.
Performance reported by netperf-2.3 is about 400Mbit/sec, graphs have Mbytes/sec vertical axis for speed test and CPU usage in percents for CPU test.


Network asynchronous IO vs. synchronous recv()

Kevent and network AIO have been announced in netdev@ maillist [ inroduction, kevent and network AIO].


Zero-copy asynchronous sending support.

Network asynchronous sending support is implemented in a zero-copy way similar to how sendpage() works.
Here is benchmark of system with 10 concurent sockets which are used for sending. Receiving server and synchronous sender use epoll() to select active socket.
Benchmark results.
Network asynchronous IO vs. synchronous send()

Patches and userspace utility (naio_send.c) are available in archive.

It was announced in netdev@ maillist.

AIO senfile() is supported.

Initial commit can be found in archive.
Patch is called aio_sendfile.1 and depends on full kevent patchset kevent_full.diff.2, which was recenly sent to netdev@.
Patch is fairly trivial - just use file->f_op->sendpage() for page sending, all asynchronous mechanism lives in page propagation into VFS cache.