Not too long ago Qing Li committed ECMP (equal-cost multi-path) support to FreeBSD. One particular application of this is to use FreeBSD as a load-balancer by setting it up as a gateway between a large number of internet clients and a rack of a backend servers. This works fine ... until one adds or deletes a route to the backend servers. The route selected for a given source / destination pair is the nth entry from a list, where n = (source ip xor destination ip) mod #routes. This means that when the number of routes changes the selected backend route will change as well, causing any existing connections to be torn down.
I've chosen to address this by creating a hash table to cache the flow to route mapping. Which works fine, provided there are fewer than a few million simultaneous connections being forwarded by the machine.
It occurred to me that this could also address the route locking issue that FreeBSD currently has in ip_output. When called from tcp_output or udp_output in_rtalloc_ign looks up the rtentry in the routing table locks it, increments the reference count, and then returns it unlocked. The mbuf is then (in most cases) passed down to ether_output. ether_output calls arpresolve to look up the destination L2 address. arpresolve then calls rt_check which locks the rtentry, checks if it has a gateway rtentry, and then returns one of them locked so that arpresolve can safely look at its attached arp information. arpresolve then unlocks the rtentry. After ether_output returns to ip_output, RTFREE is called on the rtentry which locks it *again* in order to decrement its refcount. On a system with 8 cores and one default route, this obviously doesn't scale very well. I am experimenting with using per cpu flow tables (the code which was initially developed for stateful ECMP) to locklessly cache references to rtentrys.
I've chosen to address this by creating a hash table to cache the flow to route mapping. Which works fine, provided there are fewer than a few million simultaneous connections being forwarded by the machine.
It occurred to me that this could also address the route locking issue that FreeBSD currently has in ip_output. When called from tcp_output or udp_output in_rtalloc_ign looks up the rtentry in the routing table locks it, increments the reference count, and then returns it unlocked. The mbuf is then (in most cases) passed down to ether_output. ether_output calls arpresolve to look up the destination L2 address. arpresolve then calls rt_check which locks the rtentry, checks if it has a gateway rtentry, and then returns one of them locked so that arpresolve can safely look at its attached arp information. arpresolve then unlocks the rtentry. After ether_output returns to ip_output, RTFREE is called on the rtentry which locks it *again* in order to decrement its refcount. On a system with 8 cores and one default route, this obviously doesn't scale very well. I am experimenting with using per cpu flow tables (the code which was initially developed for stateful ECMP) to locklessly cache references to rtentrys.
