The netperf project is working to enhance the performance of the FreeBSD network stack. This work grew out of the SMPng Project, which moved the FreeBSD kernel from a "Giant Lock" to more fine-grained locking and multi-threading. SMPng offered both performance improvement and degradation for the network stack, improving parallelism and preemption, but substantially increasing per-packet processing costs. The netperf project is primarily focussed on further improving parallelism in network processing while reducing the SMP synchronization overhead. This in turn will lead to higher processing throughput and lower processing latency. Tasks include completing the locking work, optimizing locking strategies, amortizing locking costs, introducing new synchronization primitives, adopting non-locking synchronization strategies, and improving opportunities for parallelism through additional threading.

Between July, 2004, and December, 2004, the Netperf project did a great deal of work, for which there is room only to include limited information. Much more information is available by visiting the URLS above, including information on a variety of on-going activities. Accomplishments include:

July, 2004: A variety of improvements to PCB locking in the IPv6 implementation; locking for the if_xl driver; socket locking for the NFS client; cleanup of the soreceive() code path including structural improvements, assertions, and locking fixes; cleanup of the IPX/SPX code in preparation for locking; additional locking and locking assertions for the TCP implementation; bug fixes for locking and memory allocation in raw IP; netatalk cleanup and locking merged to FreeBSD CVS; locking for many netgraph nodes merged to FreeBSD CVS; SLIP structural improvements; experimental locking for netatalk ifaddrs; BPF locking optimizations (merged); Giant assertions for VFS to check VFS/network stack boundaries; UNIX domain socket locking optimizations; expansion of lock order documentation in WITNESS, additional NFS server code running MPSAFE; pipe locking optimizations to improve pipe allocation performance; Giant no longer required for fstat on sockets and pipes (merged); Giant no longer required for socket and pipe file descriptor closes (merged); IFF_NEEDSGIANT interface flag added to support compatibility operation for unlocked device drivers (merged); merged accept filter locking to FreeBSD CVS; documented uidinfo locking strategy (merged); Giant use reduced in fcntl().

August, 2004: UMA KTR tracing (merged); UDP broadcast receive locking optimizations (merged); TCP locking cleanup and documentation; IPv6 inpcb locking, cleanup, and structural improvements; IPv6 inpcb locking merged to FreeBSD CVS; KTR for systems calls added to i386; substantial optimizations of entropy harvesting synchronization (merged); callout(9) sampling converted to KTR (merged); inpcb socket option locking (merged); GIANT_REQUIRED removed from netatalk in FreeBSD CVS; merged ADAPTIVE_GIANT to FreeBSD CVS, resulting in substantial performance improvements in many kernel IPC-intensive benchmarks; prepend room for link layer headers to the UDP header mbuf to avoid one allocation per UDP send (merged); a variety of UDP bug fixes (merged); additional network interfaces marked MPSAFE; UNIX domain socket locking reformulated to protect so_pcb pointers; MP_WATCHDOG, a facility to dedicate additional HTT logical CPUs as watchdog CPUs developed (merged); annotation of UNIX domain socket locking merged to FreeBSD CVS; kqueue locking developed and merged by John-Mark Gurney; task list for netinet6 locking created; conditional locking relating to kqueues and socket buffers eliminated (merged); NFS server locking bugfixes (merged); in6_prefix code remoted from netinet6 by George Neville-Neil, lowering the work load for netinet6 (merged); unused random tick code in netinet6 removed (merged); ng_tty, IPX, KAME IPSEC now declare dependence on Giant using compile-time declaration NET_NEEDS_GIANT("component") permitting the kernel to detect unsafe components and automatically acquire the Giant lock over network stack operation if needed (merged) ; additional locking optimizations for entropy code (merged); Giant disabled by default in the netperf development branch (merged).

September, 2004: bugs fixed relating to Netgraph's use of the kernel linker while not holding Giant (merged); merged removal of Giant over the network stack by default to FreeBSD CVS; races relating to netinet6 and if_afdata corrected (merged); annotation of possible races in the BPF code; BPF code converted to queue(3) (merged); race in sopoll() corrected (merged).

October, 2004: IPv6 netisr marked as MPSAFE; TCP timers locked, annotated, and asserted (merged); IP socket option locking and cleanup (merged); Netgraph ISR marked MPSAFE; netatalk ISR marked MPSAFE (merged); some interface list locking cleanup (merged); use after free bug relating to entropy harvesting and ethernet fixed (merged); soclose()/sofree() race fixed (merged); IFF_LOCKGIANT() and IFF_UNLOCKGIANT() added to acquire Giant as needed when entering the ioctls of non-MPSAFE network interfaces.

November, 2004: cleanup of UDPv6 static global variables (merged); FreeBSD 5.3 released! First release of FreeBSD with an MPSAFE and Giant-free network stack as the default configuration!; additional TCP locking documentation and cleanup (merged); optimization to use file descriptor reference counts instead of socket reference counts for frequent operations results in substantial performance optimizations for high-volume send/receive (merged); an accept bug is fixed (merged) experimental network polling locking introduced; substantial measurement and optimization of mutex and locking primitives (merged); experimental modifications to UMA to use critical sections to protect per-CPU caches instead of mutexes yield substantial micro-benchmark benefits when combined with experimental critical section optimizations; FreeBSD Project Netperf page launched; performance micro-benchmarks benchmarks reveal IP forwarding latency in 5.x is measurably better than 4.x on UP when combined with optional network stack direct dispatch; several NFS server locking bugfixes (merged); development of new mbufqueue primitives and substantial experimentation with them permits development of amortized cost locking APIs for handoff between the network stack and network device drivers (work in collaboration with Sandvine, Inc); Linux TCP_INFO API added to allow user-space monitoring of TCP state (merged); SMPng task list updated; UDP static/global fixes merged to RELENG_5.

December, 2004: UDP static/global fixes developed for multi-threaded in-bound UDP processing (merged); socket buffer locking fixes for urgent TCP input processing (merged); lockless read optimizations for IF_DEQUEUE() and IF_DRAIN(); Giant-free close for sockets/pipes/... merged to FreeBSD CVS; optimize mass-dequeues of mbuf chains in netisr processing; netrate tool merged to RELENG_5; TCP locking fixes merged to RELENG_5; "show alllocks" added to DDB (merged); IPX locking bugfixes (merged); IPX/SPX __packed fixes (merged); IPX/SPX moved to queue(9) (merged); TCP locking fixes and annotations merged to FreeBSD CVS; IPX/SPX globals and pcb locking (merged); IPX/SPX marked MPSAFE (merged); IP socket options locking merged to FreeBSD; SPPP locked by Roman Kurakin (merged); UNIX domain socket locking fixes by Alan Cox (merged).

On-going work continues with regard to locking down network stack components, including additional netinet6 locking, mbuf queue facilities and operations; benchmarking; moving to critical sections or per-CPU mutexes for UMA per-CPU caches; moving to critical sections or per-CPU mutexes for malloc(9) statistics; elimination of separate mbuf allocator statistics; additional interface locking; a broad variety of cleanups and documentation of locking; a broad range of optimizations.