This is just a quick note on some experiences using an external USB drive with OpenWRT 8.09 on an Asus WL-500gP.

Mount options not honoured

It seems that mount options set in /etc/fstab aren’t honoured properly. I’ve filed a bug report here with a fix. If you have a running OpenWRT installation, you can grab the /sbin/usb-storage replacement from there and copy it onto your router.

A particular badness of this bug is if you’re relying noatime/nodiratime to stop your disk spinning up, or your USB stick wearing out(!)

USB startup timing

This isn’t a bug in OpenWRT, just an annoyance.

On the Asus WL-500gP (at least), USB devices don’t come online until 10-20 seconds after startup. This means that USB disks don’t mount until some time during the boot process.

This causes a problem if you have any init scripts that rely on an external disk (for example, if you’ve added Optware packages and symlinked in some Optware init scripts.)

The (half-assed) solution I have come up with is to put in an init script that waits for usb-storage to come online, and then sleeps while the disks are mounted. Make sure this script runs before any other init scripts that require the external disk.

  1. Download the script from here
  2. Copy the script to /etc/init.d on the router.
  3. Enable it with /etc/init.d/usb-storage enable
  4. You will now have /etc/rc.d/S50usb-storage. Make sure any other init scripts that rely on external disks come after ’50’

There are a variety of other hacks to get all this stuff working, this is just my take on it. It looks like external storage has been dramatically overhauled in the upcoming OpenWRT release, so hopefully it will no longer be an issue.

EDIT: The setup described below worked fine for me for six months, with only the annoyances described below (mostly not being able to traverse DHCP requests across the bridge.) However, I recently upgraded to OpenWRT Backfire and took the opportunity to replace my proprietary ADSL router with a Fonera. This allowed me to switch over to WDS as described in this post. So this setup is no longer in use at my house.

Renting sucks when you’re a nerd. When you move, you have to re-network all your bits and pieces, without the benefits of laying cable. Wireless is great, but not every network device has it. I have a VoIP phone and an XBMC XBox, each without wireless and each in a separate room.

I need a way to connect the phone and the Xbox to my existing wireless access point.

Here’s a rough diagram of my network:

Wired & Unwired Diagram

Rough diagram of my network

Options

I want a flat network, single broadcast domain, for all of the (wired and unwired) devices in the diagram above.

I already have two Linux-compatible wireless routers, an Asus WL-500G and a Fonera. Both running OpenWRT.

If my routers had Broadcom wireless chipsets then this task would be easy. Thanks to driver support, OpenWRT can automatically bridge network segments through a wireless client (at least on 2.4 kernels, not sure about 2.6.) Neither of my routers have Broadcom. The Asus used to have it, but I swapped in an Atheros card. Oops.

WDS is a possibility, using each router as a repeater station. However, each repeater costs half the available bandwidth. I also can’t get WDS+WPA to work with my access point. YMMV.

You can set up separate networks, one for each segment, and the routers can route(!) between them using iptables. I’ve tried this and I don’t like it. It’s much easier when you have one single network with a single broadcast domain: simple DHCP, simple NetBIOS lookups, no routing tables to maintain, etc.

The final option, and the one I settled on, is using ARP-NAT.

What is ARPNAT?

ARPNAT is a clever hack. It’s the same idea as NAT for IP networks, except it works one layer deeper. Instead of translating IP network addresses, the router translates between the MAC hardware addresses on each side of it.

If something on the wired side of the router makes an ARP request for the MAC address of an IP on the wireless side, then the router forwards the request as if it came from the router. When the response comes back, it mangles that too. Instead of passing back the real MAC (which lives on the wireless network), the router gives its own wired MAC address. Then, when it receives frames for IP addresses on the wireless network, it forwards them through. It does this to both sides of the bridge.

Here’s an example, taken from my network:

  1. XBMC Xbox asks “What is the MAC address of 10.0.0.1? Tell me: 10.0.0.5.” This is a broadcast ARP request over the wired ethernet.
  2. Asus router sees the ARP request, and forwards the broadcast onto the wireless network, as “What is the MAC address of 10.0.0.1? Tell me: 10.0.0.4.”
  3. Access point (at 10.0.0.1) gets the ARP request from the wireless, and replies with its MAC address: AA:AA:AA:AA:AA:AA.
  4. Asus router gets the ARP response, sent to 10.0.0.4. It mangles it and forwards it back to the Xbox as “10.0.0.1 is at BB:BB:BB:BB:BB:BB”, where BB:BB:BB:BB:BB:BB is the wired address of the Asus itself.
  5. Now, any time the Xbox wants to talk to 10.0.0.1, it will talk to BB:BB:BB:BB:BB:BB, and the Asus will forward that frame onto the wireless network, to the real destination AA:AA:AA:AA:AA:AA.

On Linux, this is done using a patched version of the ethernet filtering program ebtables.

How to do it?

Out of the box, OpenWRT does not support either ebtables or ARPNAT. The developers have expressed opinions that ARPNAT is a hack, and ebtables is too slow (see here & here.) We’ll get to that issue, later.

However, there are some forum posts that show how to do it. In this post, a developer of the OpenWRT fork Gargoyle links to their patch for ARPNAT. So, one option is to just use Gargoyle. Gargoyle apparently has a very nice web interface, and is good for intermediate users. I prefer flat config files, so I’m going to stick with OpenWRT.

Building OpenWRT with ARPNAT Support

If you’ve never compiled OpenWRT before, start here. Alternatively, I have some prebuilt 8.09.2 images for Atheros devices (like the Fonera), here.

To build, you’ll need two patches. One is from Gargoyle. The other is a small bugfix to allow a DHCP client to run on the router. The patches can be found here. These patches are against the 8.09 branch of OpenWRT SVN, I don’t think either will apply to the trunk.

EDIT: I’ve just posted a slightly different patch, 04-arpnat-brcm47xx.patch, for brcm47xxx devices like the Asus WL-500g. It’s different because these devices have an older kernel version in 8.09, and also because the arpnat kernel module doesn’t seem to load the first time around. Also, on this architecture DHCP responses don’t seem to pass back correctly to devices on the other side of the bridge (while on my Atheros-based Fonera they are passed back.) I’m just using a static IP on those devices, for now. Things will also work fine if the DHCP server is on the bridge router, itself.

You’ll need to enable ‘ebtables’ in your configuration (ie when you make menuconfig.) You’ll probably want wpa-supplicant as well. The .config that I used for the Fonera can be found at the link above, as well.

Once up and running, /etc/config/wireless only needs one tweak. Here’s mine:

config wifi-device  wifi0
	option type     atheros
	option channel  auto

config wifi-iface
	option device	wifi0
	option network	lan
	option mode	sta
	option client_bridge 1
	option ssid	<your ssid here>
	option encryption psk
	option key <your key here>

The only difference is “option client_bridge 1“, which indicates that you want ARPNAT enabled.

/etc/config/network is simple, too. Here’s the bridge section of mine:

config interface lan
	option ifname	eth0
	option type 	bridge
	option macaddr  CC:CC:CC:CC:CC:CC
	option proto	dhcp

The only difference here is option macaddr. You will need this only if you want to assign the router’s IP address dynamically via DHCP. If you do this, “option macaddr” must be set to the MAC address of the interface where the DHCP server lives (ie wired side or wireless side.) The DHCP request will only go out on a single interface, so you need to make sure it is the correct one.

If you don’t need DHCP then you don’t need to manually assign any MAC address. “option macaddr” can be totally removed.

That’s it! When the router boots up, it will connect to the wireless network, and any devices on the wired side will automatically be bridged onto the wireless (and vice-versa.) To them, it just looks like one big seamless network.

Bear in mind that not all routers support failsafe mode (Fonera for one), so if these values are wrong then you may end up locked out of your router, especially if you run the router with a DHCP-assigned address. If you’re not certain, I recommend running with a static IP first, so you can always plug in a network cable and still connect to the router.

Thanks again to Eric Bishop and the Gargoyle project, because it’s really his patch (building on other people’s work for ARPNAT) that makes all this possible.

Problems with ARPNAT

The OpenWRT developers will not consider ARPNAT, or even include ebtables, in the default package set of OpenWRT. They say this is because of performance impact. Every ethernet frame that hits the router needs to be inspected in software, so it’s understandable that there will be a cost.

Some cost is OK by me, in exchange for convenience. So I did some tests:

The test methodology was to ping flood the Fonera from a PC:

Performance of ebtables & ARPNAT (longer is better)

  • “Ethernet, no ebtables” is a vanilla Openwrt 8.09.2 install
  • “Ethernet, ebtables installed” is with patches and ebtables installed, but no ARPNAT enabled.
  • “Ethernet, ARPNAT enabled” is with ARPNAT turned on, but no traversal (the pings still go just to the router.)
  • “Direct WiFi” is pinging the router directly over the wireless network. It’s slower, but only because WiFi is slower than ethernet.
  • “WiFi via ARPNAT” is pinging a client on the wireless side, from a client on the wired side. So the ping is going through ARPNAT translation on the router.

Under all of these tests the CPU on the router was pegged at >95%.

The results seem to show that ebtables & ARPNAT do not drastically affect the performance of the Fonera. Turning on ARPNAT causes direct traffic to the router to be about 17% more costly. Actually bridging packets through ARPNET on the router is only 32% more costly than directly accessing the router itself.

I’m surprised these figures aren’t worse, given the doom & gloom spouted by the OpenWRT devs. It’s possible there’s another circumstance under which ARPNAT falls over, or possibly it performs worse on certain architectures. I’ll post back if I learn anything interesting.

EDIT: One more discovery, obvious in retrospect, is that IPv6 won’t get ARPNATted by this config. If you have any devices that use IPv6 (like iTunes talking to an Apple Airport Express), you’ll either need to disable IPv6 or tweak the ebtables scripts somehow to translate IPv6 as well. I opted for disabling IPv6.