There are some existing solutions to this: early-ssh worked on previous Debian versions. There are other blog posts showing some methods but I found them unwieldy (ie a 35 second delay before you can type your passphrase.)

Here’s how I’m doing it. Unless otherwise mentioned, all of the following commands need to be run as root. The server name in the examples is “debian_headless”

At time of writing I was using versions Debian 7.0 Wheezy with packages cryptsetup 2:1.4.3-4, initramfs-tools 0.109.1 and dropbear 2012.55-1.3.

These steps may work on recent Ubuntu versions as well, please let me know if you try them out.

Overview

The very first thing a Debian system runs at startup is a minimal initramfs ramdisk which performs basic initialisation, mounts the main filesystem and starts the rest of the boot process.

This is the stage when an encrypted rootfs gets mounted. We need to customise it so we can ssh in and unlock the rootfs.

Install prerequisites

apt-get install dropbear initramfs-tools busybox

Dropbear is a slimline SSH server. This package installs into the initramfs and runs during early boot only, allowing you to log in remotely.

Authorise the user(s) you want to be able to login as root and unlock

Hopefully you’re already using SSH Key Authentication, otherwise have a read about it and why it’s a good idea. Set it up for the user(s) on the client systems by having them run ssh-keygen.

You have to add the public keys for these users to Dropbear’s list of authorized_keys for login.

For each client user you want to allow to unlock, take their public key file ~/.ssh/id_rsa.pub and append it to /etc/initramfs-tools/root/.ssh/authorized_keys

One way of doing it:

scp ~/.ssh/id_rsa.pub myuser@debian_headless:id_rsa.pub
ssh myuser@debian_headless
sudo sh -c "cat id_rsa.pub >> /etc/initramfs-tools/root/.ssh/authorized_keys"
rm id_rsa.pub

Alternative: Have a unique key for root/unlock

If you don’t want to use your usual user-level ssh key and passphrase to provide potential root-level access to your headless machine, generate a separate key (with a different passphrase) and then copy it across to authorized_keys instead:

ssh-keygen -f ~/.ssh/id_rsa_headlessunlock
scp ~/.ssh/id_rsa_headlessunlock.pub myuser@debian_headless:id_rsa.pub
(etc, as above.)

Add network hardware module

The initramfs doesn’t usually load network hardware modules, so you should add the name of the network hardware module directly to /etc/initramfs-tools/modules

You can find the module name via sysfs. For example:

$ grep DRIVER /sys/class/net/eth0/device/uevent
DRIVER=e1000

So to enable eth0 in this case, you’d add e1000 to /etc/initramfs-tools/modules.

(This may not be necessary on “normal” x86 machines with built-in ethernet, it’s definitely necessary using USB Ethernet on my ARM-based GK802 stick and it’s probably necessary on Raspberry Pi.)

Configure early boot IP Address

Initramfs needs to know the IP address to come up on. You could do this by passing the ‘ip=’ kernel command line parameter, but you can also just set it in an initramfs-tools config file:

Create /etc/initramfs-tools/conf.d/network_config with contents like this:

export IP=<client-ip>:<server-ip>:<gw-ip>:<netmask>:<hostname>:<device>:<autoconf>

<server-ip> can be left blank. Some possible examples:

export IP=dhcp

or

export IP=10.0.0.2::10.0.0.1:255.0.0.0:myserver:eth0

See the nfsroot kernel docs for full details. Remember if using DHCP that you’ll need a static DHCP entry configured so you can predict what IP your headless machine is going to have on startup.

Annoyance: Interface doesn’t reconfigure

It seems that after the main system boots, the network interface won’t automatically change to whatever is set in /etc/network/interfaces (because the interface is seen to be already “up” it gets left alone.) To work around this you could put a line ifdown eth0; ifup eth0 in /etc/rc.local, or just keep the initramfs network settings the same as the ones in /etc/network/interfaces.

Unlock Script

I’ve written a little initramfs-tools hook that generates scripts for easy unlocking of the rootfs.

Download this script file, then read it to make sure you’re not suspicious about what it’s going to do. Then copy it to /etc/initramfs-tools/hooks/mount_cryptroot and mark it executable:

chmod +x /etc/initramfs-tools/hooks/mount_cryptroot

There’s a section at the top of mount_cryptroot that allows you to set an ALLOW_SHELL variable if you want to be able to drop to a remote root shell for diagnostics. Note that this explicitly allows a remote root shell to anyone with an authorised SSH private key. I can’t guarantee it’s impossible to get to a root shell otherwise, but it’s at least not explicitly welcomed.

Update the initramfs

Update the initramfs so it includes all your changes:

update-initramfs -u

Add client SSH config entry

Because Dropbear has a different SSH host key to the normal system, OpenSSH will (rightly) complain if you try connecting to it on the same IP. The solution is to create a special host definition for the initramfs environment. A custom section in ~/.ssh/config:

Host headless_unlock
HostName <IP of headless>
User root
HostKeyAlias headless_unlock

Now you can log in to the unlock stage by simply typing ssh headless_unlock. (Replace headless with whatever name you want to use.)

If you authorized a unique private key to use for unlocking, you can add another line IdentityFile ~/.ssh/id_rsa_headless or similar.

Putting it all together

Here’s what it looks like to connect to my headless host and unlock the disk:

$ ssh headless_unlock

BusyBox v1.20.2 (Debian 1:1.20.0-7) built-in shell (ash)
Enter 'help' for a list of built-in commands.

Unlocking rootfs...
Enter passphrase: 
Success
Connection to 192.168.56.101 closed.

Easy!

Final word about security

When dealing with remote access and with encryption, it’s always worth thinking about security tradeoffs.

There’s the obvious risk that enabling Dropbear in the initramfs means allowing remote root logins to your system. Even with ALLOW_SHELL turned off there’s no guarantee that someone with access to an authorized SSH key (or Dropbear exploit) can’t log in, pop a root shell somehow, and cause havoc.

In addition, when I was looking for existing techniques to do this I came across a post exclaiming ssh unlocking was a terrible idea, because someone with physical access to your machine can then impersonate you remotely and you’ll never know.

I think this is worth considering, but it’s not as bad as it might seem. All encrypted disk access, with local or remote unlocking, relies on physical security of your device. Yes, if someone gets physical or root access they can steal your Dropbear host key and set up another server that pretends to be you and steals your keys. However, they already have physical or root access. They can just as easily insert malware in the initramfs that quietly captures your passphrase as you type it in in locally.

In either case, you have to be sufficiently sure noone is messing with you. With remote access that’s maybe more likely, but I’m not too worried. I’m using this on a USB stick PC that fits in my pocket, and I don’t want a burglar walking out my door with all my personal data. If some very dedicated person comes after my data, there would be easier ways to get it.

(Also, an obvious disclaimer – all the advice/scripts on this page is worth what you paid for it. I’m not a security professional. Please tell me if it seems broken/unsafe, but don’t blame me if you use it for something important and it isn’t safe enough.)

I bought this LG branded USB 2.0 Hub for $10 from PCDIY.

Seems to be an LG promotional item that PCDIY decided to resell. (I don’t mind, it works properly in Linux unlike most cheap hubs.)

However, the form factor is kinda big and the LG logo is pretty bright.

A popular alternative to security bits: screws so soft they attempt to disintegrate when you unscrew them.

Eck.

Screwed back together…

Bit bright…

Much nicer. Sticker was from This Is Not Art Festival a few years ago.

Next week: Turning an Macbook Pro into a Macbook Air via clever application of sew-on patches.

Here are two USB to Ethernet adapters:

One of them is sold on ebay for $3.85 AU ($3.99 US), including postage to Australia. The other is sold at Apple Stores for $29.¹

In Linux they both use the driver for an “ASIX AX88772A” USB to Ethernet converter, even though the Apple one reports as “Apple” and is sold only for the MacBook Air.²

The cheap adapter comes with drivers for OS X & Windows, as well.

When I ran a TCP throughput test with iperf, they both performed well. The Apple adapter measured throughput of 94.3Mbps. The cheap adapter measured 87.4Mbps. By comparison, the builtin ethernet on my laptop measured 94.8Mbps (after being set from gigabit to 100Mbps.)

Oddities

There are a few unusual things about the cheap adapter, though. The Linux kernel log says:

usb 3-2.1.2: New USB device found, idVendor=0b95, idProduct=772a
usb 3-2.1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
usb 3-2.1.2: Product: AX88x72A
usb 3-2.1.2: Manufacturer: ASIX Elec. Corp.
usb 3-2.1.2: SerialNumber: 000002
eth1: register 'asix' at usb-0000:0e:00.0-2.1.2, ASIX AX88772 USB 2.0 Ethernet, 00:8a:8d:8a:39:2b

Serial number “000002″? Hmm…

Also, the hardware MAC address prefix (00:8a:8d) isn’t any known vendor OUI (organisationally unique identifier.) Seems odd, although chipset vendors (like ASIX) often require the device manufacturer to register their own OUI (for instance the Apple adapter uses an Apple prefix.) For a no-name vendor, it makes sense to just make one up.

Spot the difference

Here’s a view inside the cheap adapter:

Inside the Apple adapter:

It seems $29 buys you shielding from interference. Underneath the cover:

Some similarities are visible:

• Both adapters have a 512 byte EEPROM onboard (Atmel AT93C66B.)³
• Both adapters have an Ethernet transformer to isolate the ethernet signals from the rest of the board (MEC TM1701M & LFE8423, respectively.)

The biggest difference is this: the Apple adapter contains a clearly labelled ASIX AX88772ALF USB to Ethernet bridge, the other adapter has an unmarked chip that is not made by ASIX.

ASIX don’t make any USB/Ethernet devices in a 32 pin package, their smallest package has 64 pins, same as the genuine one above.

It’s a clone!

A clever clone

Cloning ASICs isn’t new to the computer world. They’ve been around since the early 80s, maybe earlier.

This is not a simple copy, though. It’s been modified to make it much cheaper to produce. It might even be a brand new ASIC, created from scratch to be compatible.

To know if it is a modified knockoff or made from scratch we could dissolve the chip packages in acid (“decapping” them) and then look at the exposed silicon die under a microscope. Not quite that keen yet, though.

Clock Sync

The genuine AX88772A requires two crystal oscillators for accurate timing – a 25MHz oscillator for the ethernet interface clock, and a 12MHz oscillator for the USB interface clock. The cheap adapter only has a single 25MHz crystal.

How much does this save? A lot. The Apple adapter has two “TXC” brand oscillators (the shiny silver packages.) These cost around $1.33 each if you buy 15,000⁴. This wholesale cost alone is nearly the retail price of the cheap adapter on taobao.com.

By comparison, a single large crystal resonator like the one on the cheap adapter costs $0.16 from the same source (6% 20%, see below, of the cost.)

EDIT: Someone on reddit pointed out that I was using ‘oscillator’ incorrectly to refer to the large crystal, which is only the resonator part of an oscillator circuit. Fixed!

EDIT 2: Gerard points out in the comments that the two TXC chips are probably just crystal resonators (not full oscillators) as well. I checked the reference schematic for ASIX’s Demoboard and he’s right. Thanks Gerard! So the cheapest ones they could be are around $0.40 each (12MHz/25MHz), the cheaper board’s single crystal is then closer to 20% of the cost.

In the genuine chip, the 12MHz frequency would be used to derive 480MHz for USB 2.0 High Speed by using a PLL frequency multipler circuit. 12 x 40 = 480.

In the cheap one, I’m guessing the host’s packet sync must be used to synchronise the 480MHz clock. Without a high speed oscilloscope or a USB 2.0 logic analyser it’d be hard to tell how well this goes at meeting the 480MHz +/-500ppm requirement of the USB High Speed spec.⁵ ASIX themselves warn that if their precision 12MHz clock source has an incorrect capacitor it “can cause some problem during USB High Speed mode enumeration. For example, during the 100 times of repeatedly plug-and-unplug test, there may be 1 time that AX88772 may not be initialized properly“. You’d imagine this kind of problem is potentially much worse in the cheap adapter, although I haven’t yet noticed anything.

Parts Count

The Apple adapter also has many more small components – two inductors (the cheap adapter has none), over twenty five capacitors (the cheap adapter has only nineteen), more resistors. For the cheap adapter design, every fraction of a cent saved is important!

One thing that surprised me is that the cheap adapter has a functioning blue activity LED, that glows through the enclosure. The Apple adapter actually has a space on the PCB for this, but no LED in place (Apple’s designers presumably nixed it for aesthetic reasons.) I’m surprised the manufacturer paid the few cents to add this feature.

Software

For the manufacturer of cloned/compatible ASICs, an interesting bonus⁶ is driver support. The CD that came with the cheap adapter contains ASIX’s own drivers for Windows and OS X.

The Windows drivers are the exact same digitally signed ones that Microsoft distributes through Windows Update, meaning the adapters appear to have “passed” Windows Hardware Quality Labs testing. Something the actual device manufacturer surely couldn’t have afforded.

Reliability

In my simple tests both adapters seemed perfectly reliable, moving data back and forth quickly without any measurable errors. I ran an eight hour two-way ping flood (assuring plenty of collisions) with zero lost packets.

However, the cheap adapter is probably susceptible to (and a producer of) electrical noise. The Apple adapter is protected from an electrically noisy environment by its metal shielding, and extra decoupling capacitors on the board.

On the other hand, both USB and Ethernet contain mechanisms for dealing with errors introduced by interference. It’s possible the protocols are well engineered enough that you’ll never notice the difference.

It is also possible that the cloned ASIC will display hardware bugs that aren’t in the legitimate adapter. So far I haven’t found any (I tested some of the uncommon features like forcing 10Mbps, forcing half duplex, adjusting MTU.) I’d be interested to hear of any, though.

Conclusion

I’m probably unusual in that I find this world of cheap clone “shanzhai” hardware amazing. I’m fascinated that someone is out there redesigning existing silicon to make a knockoff that is smaller, cheaper but otherwise near-equivalent – to be used in devices that retail for less than $3.

I’d love to learn more about these secretive industries and the engineers who work in them.

How about you? Do you have any experience with dirt cheap hardware devices? How do you feel about shanzhai competing with regular firms’ R&D by cloning their hardware? Please leave a comment and let me know what you think.

1. Model MC704ZM / A1277
2. Internet lore seems to agree that any Mac running OS X 10.5.2 or later can use it.
3. Actually the part markings on the cheap adapter’s EEPROM don’t match Atmel’s datasheet, so it’s possible that one is a clone, rebadged, or old stock.
4. Not the same oscillators but the same brand, meeting the spec that ASIX requires: 25MHz & 12MHz. Of course both Apple and our noname manufacturer would pay less than Digikey’s prices, but it gives you some idea.
5. I think the Linux kernel could know the raw USB bit error rate, giving some idea of how precise the timing was, but the kernel doesn’t record it anywhere
6. Assuming you can ignore copyright law, and have no qualms about reproducing other companies’ intellectual property for profit

Unfortunately we’ve also struggled with a couple of long-standing problems. One of them is the PTFE “bowden tube” that encloses the filament between the extruder drive and the hot end. In the stock design, the tube is screwed into two threaded brass fittings at each end. Forcing the tube into the fitting makes a thread, it’s kind of “self-tapping.”

On some Huxleys this seems to hold firm, but ours popped out at the drive end repeatedly. Rethreading, glue, epoxy all failed. Ryan’s clever stop-gap worked for a while but put a lot of load on the extruder drive motor:

This is a modification of the Rep Rap Pro Huxley design to use push-to-fit pneumatic connectors instead of the brass fittings:

The design is inspired by DeuxVis’ Bowden cable push-fit connectors, but DeuxVis’ part doesn’t fit the Rep Rap Pro variant of the Huxley.

Better Alternative?

While writing this post just now (isn’t this always the way) I’ve found that eMaker is furnishing their Huxleys with pneumatic fittings that you can buy individually from them. They look like they’re small enough to not require a redesigned hot end. I hope these weren’t available when I started this project back in August, I searched a lot! If you’re looking at options you may do well with those smaller fittings and less other changes, depending on what availability & your requirements are.

Nevertheless, here are the instructions if you want to go the Make, Hack, Void way:

New Parts Required

• 2 printed parts, M6-Block & nozzle-mounting. You can get them from Thingiverse.

• 2 40mm M3 screws, which replace the the original 25mm vertical screws holding the nozzle-mounting to the aluminium heat block.

• 2 20mm M3 screws, these replace the 2 16mm screws that are used to mount the blue heatsink to the aluminium heat block. I think 25mm screws should also fit.

• Heat sink thermal compound (may be unnecessary, see below.)

• 1 small piece (no smaller than 38x8mm) of 2mm thick copper plate, or 2mm thick aluminium if copper isn’t available. You can go a bit thinner than 2mm if you like, down to about 1.1mm, but you’ll need to be more careful when assembling (see below.)

• 2 push-to-fit connectors. I used Parker Prestolok F8PB4M5, RS Components part number 287-3490 (link goes to the Australian RS site.) These are specified to hold 4mm PTFE tubing against up to 300psi of pneumatic pressure, at temperatures up to 100°C.

• 1 M5 nut, standard height.

• 30cm or so of PTFE tubing, 2mm ID 4mm OD. Our Huxley came with 3mm OD tubing, so we needed to swap. This is hard to get in Australia, easy to get in the US from places like McMaster-Carr. I was lucky that Andrew from the Campbell (Sydney) hackerspace was ordering a big lot from Aliexpress, so I paid for a 3m share in his order.

Cold end – extruder drive

Preparing the extruder drive is exactly the same as the stock RepRap Pro extruder drive assembly, only swapping out the “M6 Block” (ie the main body of the extruder drive) for the new printed one.

When you’re done, drop the M5 nut into the nut recess and then screw one of the two push-to-fit connectors into it:

The hot end preparation varies quite a bit from the standard instructions, however. Here it is step by step:

Making the copper spacer

When you screw the push-to-fit fitting into the aluminium heat block, it will foul on the blue heatsink. The 2mm copper (or aluminium) plate is a spacer to sit behind the heatsink.

You’re looking to make a plate that matches this face from the heat block, 38x8mm with 2 3mm holes placed, centered 3mm from each end:

Detail from “heatsink block.pdf” in original Huxley plans

I used a piece of 2mm copper scrap with the drill press and hacksaw in the hackerspace:

As you can see it doesn’t need to be too exact, there’s quite a bit of room at each side. The most important thing is getting the hole spacing correct (centres should be 32mm apart.)

Of course, if you were machining the Huxley parts from scratch then you could just modify the whole “heatsink block” part to use 10mm bar stock instead of 8mm bar stock, and adjust the drill hole offsets accordingly.

Preparation

Start by filing any irregularities on the printed nozzle mounting. In particular, the pathways for the two tall vertical screws need to be perfectly parallel so they can screw into the aluminium heater block as shown:

This is what it looks like fully assembled, note the parallel long vertical bolts.

Filing the printed material in-situ is a real pain, so if possible line all these parts up on the bench and make sure they’ll fit.

Take care not to file too much material, though, because those bolt pathways have to hold the hot end tight against the mounting.

Assembling the nozzle mounting

After removing the old nozzle mounting, start by screwing the new printed piece into the X carriage, using the same 4 self-tapper screws that held on the old piece:

Connector

Next, remove the machined brass fitting from the aluminium heat block and screw in the push-to-fit connector:

To tighten the connector against the nozzle, use the same technique as the original hot end: First, screw in the two parts until they touch. Then unscrew the push to fit connector by 1/4 turn, screw in the nozzle to match it again, and finally tighten the push to fit connector down against the nozzle. You can use a 10mm spanner for this.

If your spacer is less than 2mm thick then it’s a good idea to check now that the push-to-fit connector doesn’t stick out any further than it. If it does, you’ll need to tweak the angle of the nozzle so the connector ends up flatter when you tighten it down (when fully flat it only sits 1mm proud of the aluminium block.)

With a 2mm spacer there should be heaps of room regardless of angle, as you can see:

PTFE Spacer

At this point, you’ll need to take a sharp knife to the white PTFE spacer that sits on top of the heater block. Cut it into 3 pieces, cutting as close as you can to each of the two small holes while leaving them intact:

The middle piece is no longer needed, but the two end pieces are.

Attach hot end

Now you can use the two long vertical bolts to screw in the push-to-fit connector against the printed nozzle mounting, sandwiching in the two ends of the white PTFE spacer:

At this point you’ll want to hope that your pre-installation filing went well, or (like me) you’ll be awkwardly filing the bolt passageways in-situ so they come out parallel.

Once the mounting bolts are tightened, you can push in the filament tube:

Nearly done!

Heatsink Install

Finally, install the blue heatsink and spacer, using new M3 mounting bolts because the old ones aren’t quite long enough:

The whole thing should fit up snugly as shown:

A note about cooling

When I installed the heatsink, I made a point of cleaning the old mounting tape off the mating surface and then applying a very thin layer of good quality thermal heatsink compound between each mating face. I was quite worried that the push-to-fit connector might get too warm if the heat block wasn’t cooled efficiently enough.

Turns out I over-compensated and the newly assembly hot end was too effectively cooled! It couldn’t reach 240° for printing ABS, it would reach equilibrium at about 230°. I had to take the heatsink back off and remove some of the thermal compound! An alternative might have been to turn the fan speed down so the heatsink stayed warmer.

Whatever you do, make sure the aluminium heat block isn’t getting too hot. The push-to-fit connector is only rated to 100°C, and the surrounding ABS can also melt if the heat isn’t bled off effectively.

Running the hot end at 240° for an extended time, I measured the ends of aluminium block at no more than 40°. Which I think is a very safe temperature, I can’t see the middle of the block being significantly hotter than the ends.

It works!

The printer has done more than 5 hours of printing now, and I think this design is a success:

On display printing a business card holder at the Art, Not Apart festival

Source Code

The OpenSCAD files are available on Thingiverse, but there is also a full fork of the Huxley design on github with the changes in it.

Prior Designs

This wasn’t my first attempt at upgrading the extruder:

Tracing

The first thing I needed to do was redraw the two printed parts. The originals were made with Solidworks, which I’m sure is an excellent CAD program but it’s not Free Software and costs a bit more than my hobby budget can bear!

Instead, I retraced the STLs from the Huxley repository into OpenSCAD, the results are available in a different branch on github.

First draft

The first redesign mounted the push-to-fit connector on top of the nozzle bracket:

You can’t see it in these photos, but the original redesign needed a lot of filing & drilling before it actually matched the Huxley parts. All mistakes in my understanding of the design. Thanks to Adam to MHV for helping me with this part of the install, and getting the new extruder on the printer.

However, once installed this version worked fine for over a week, with several prints.

Second draft

The second design draft was a cleanup of the first, fixing dimensions to minimise drilling & filing and also beefing up the push-to-fit mount at the top of the nozzle mounting:

This redesign worked fine until I tried to print something that took up a large area of the 140x140mm bed. Turns out that the push-to-fit connectors, though sturdy when held statically, will eventually bend their way out if flexed back and forth. Having the X axis travel all the way back and forth, over and over, eventually worked it loose:

Moment of destruction photo, courtesy Chris Lee

Hence the final redesign with the push-to-fit connector recessed low behind the heatsink, with a collar at the top to hold the tube straight and fast.

Conclusion

Hopefully this redesign is the last one for the extruder. The MHV Huxley is broken again(!) right now, although this time it’s one of the Y Axis mounts and nothing to do with the bowden tube.

Hopefully someone will be able to repair that failure successfully without requiring a whole part redesign, though!

Demo page with data from the last week in September

A few months ago, in the dead of Canberra winter, we discovered our flat had a mould problem. Insidious disgusting mould had crept in around our walls. Much scrubbing ensued.

Black mould found behind a cupboard – eww!

We bought a dehumidifier in order to keep our flat dry and hopefully mould-free. The more advanced model had a humidistat and could automatically maintain a given humidity level. That model wasn’t in stock, though. Besides, surely I could build something better (famous last words!)

The Flatdrier project is my overblown personal project to monitor & control our dehumidifier.

Data Acquisition

The first step was monitoring temperature & humidity. We’d seen signs of mould in a couple of rooms, so I wanted to be able to keep track of several rooms plus outdoors.

The cheapest and easiest solution was this ebay weather station. $43 and it came with 3 remote units:

Photo courtesy of ebay seller marcmart.ukau – fine purveyors of weather stations, DC solenoids, off-brand Rubik’s cubes, and men’s wool jackets!

The weather station is pretty average quality. With the remote sensors all next to each other the readings vary by 1-2 degrees & 10-20% humidity. However each sensor seems consistent with itself, I think this is accurate enough for my purposes.

Hacking the weather station

The first step was pulling the weather station apart. The remote modules use a 433Mhz transmitter (the blue module in the photo marked “SL-TX583 WWW.HH.COM”) to send digital data back to a matching receiver in the base station:

With the data pin patched from a transmitter module into my Arduino, I wrote a quick Arduino sketch (later an MHVLib program) to sniff the signals so I could interpret them.

Turns out the protocol is very simple, with short & long pauses for 0s and 1s. To verify data it doesn’t use a checksum, it just repeats the same message multiple times.

Base Station

The base station had its own thermistor & humidity sensor onboard. I figured this was too good to waste, so I tried to reverse engineer it as well:

Both the thermistor and the humidity sensor are resistive sensors. The resistance varies with temperature & relative humidity, respectively. After “buzzing out” the circuit with my multimeter and sketching it, it seemed that instead of using ADCs the base station was using a comparator and timer to measure resistance.

This method measures the time taken to charge or discharge a capacitor through the sensor it is measuring. The time is proportional to the resistance (more resistance means less current flows to/from the capacitor, so it takes longer to fully charge or discharge.)

I found a good description of this technique on page 10 of TI’s publication Economic Measurement Techniques with the Comparator A Module (PDF).

I had a tiny microcontroller board lying around, the TI MSP430 F2013 evaluation board, so I loosely mounted it in the base station to measure the time between charge/discharge of the capacitor.

Unfortunately I didn’t notice that the F2013 has no onboard comparator (the next model down does, this one has a high-precision ADC!), so I was stuck using the high impedance digital inputs to detect “high” & “low” instead of specific threshold voltages. I think either the digital high/low thresholds are not consistent enough, or I misunderstood the circuit, because I never managed to take consistent readings.

After a few nights of poking the circuit and being confused, I abandoned the base station. I pulled the 433Mhz receiver module out of the base station, to connect directly to my Arduino.

Not wanting to lose a temperature input completely, I went to ebay and bought a $2 DHT-11 temperature/humidity sensor. This sensor is pretty crummy, the humidity values seem totally wrong (though I think maybe I cooked it during soldering), but the temperature readout provides useful data points.

Once I had all the hardware pieces together, I wrote a C program (plain AVR C) to run on the Arduino and monitor the temperatures.

Control Logic

I wanted to be able to track the temperature & humidity, so I could recall recent changes. I also wanted control – to switch the dehumidifier on and off by comparing the dew point to the temperature.

I also needed to stop the dehumidifier turning on in the middle of the night!

All these tasks could probably be accomplished by a single high-end microcontroller, but at the time I had an Arduino and the tiny MSP430 board. So I decided to make a client/server architecture, using our existing home wireless router (which runs the OpenWRT Linux distribution.)

A Python script on the router runs regularly to poll the latest values from the Arduino. It also makes decisions about when to turn the dehumidifier on and off.

Rather than managing a long-running process, the Python script just gets executed by cron every minute. It runs, takes a new sample from the Arduino, saves it to a CSV file, then sends either an “On” or an “Off” signal to the dehumidifier based on the recent sensor history (filtered to avoid hysterisis and midnight surprises.)

Dehumidifier Control

The dehumidifier is switched on and off by a wireless remote control powerpoint (Powertran A0342) that was left over from an older project:

This also communicates on 433Mhz. I already had a Jaycar ZW-3100 transmitter module, so I used this to transmit the on/off signals.

Conveniently I didn’t have to reverse engineer the switch protocol, there was an existing Arduino remote switch library and I just translated the “Blokkerswitch” signals into plain AVR C.

Putting it together

The PCB engraver at Make Hack Void made it easy to throw together a custom Arduino shield:

Realistically, a protoboard could have done this pretty easily. But it’s fun to use a new toy like the PCB Engraver, and it took less than an hour to make the board!

The PCB routing itself is a bit quick and nasty. For starters, I got the receiver module pinout backwards so it sticks out the side of the PCB! That’s since been fixed, but I think the existing board looks a bit like a sailboat!

Scrappy breakout shield – left-to-right is 433Mhz receiver, DHT-11 sensor, 433Mhz transmitter.

Web Interface

There’s a web page to view the temperature, humidity & the history of the dehumidifier state:

The date range picker control (thanks Dan Grossman!) lets you view any date range:

Here’s a demo page, using data from the last week in September. Looking over the history you’ll notice some inconsistencies with the dehumidifier coming on and off, this was caused by me tuning the algorithm sensitivity during the week.

Rather than make a server-side program to run on the router, I made the web page run entirely in client-side JavaScript. The script running in the web page downloads CSV files containing each day’s raw sample data (created by the Python script), parses them, and then displays the data.

I tried out CoffeeScript for the web programming. The program doesn’t need to be “production ready”, so it uses the in-browser compiler to compile inline CoffeeScript to JavaScript as the page loads.

To draw the graphs I used the very nice Flot charting framework.

An early version used the d3.js charting functions from NVD3.js, instead of Flot, but the browser rendering times were much slower and there were glitches with the rendering output. I didn’t spend much time trying to tune or fix D3, though. D3 is a very powerful framework and I’m sure I could make it work well, but this time I really only needed a simple “cookie cutter solution”, something NVD3 explicitly says it avoids.

Code

The project code is all up on github, with a README describing the structure.

It’s unlikely anyone else will have this same problem or exact same hardware, but the code is pretty modular so some parts will hopefully come in handy.

Irony

The ironic thing about this project is that we got the dehumidifier in early July but I only got the controller up and running in the second week of September, just in time for (hopefully) warmer weather and less condensation problems.

Maybe waiting a few days for the “smart” dehumidifier model to come back into stock wouldn’t have been such a bad idea, after all…

Someone posted a comment last week:

How is this phone holding up since you bought it last August? Are you still happy with it?

Ten months on and I’m still using the phone, so here’s my follow-up.

Build Quality

So far the Sonic has held up pretty well. I don’t give it any special care, apart from trying not to leave it in the same pocket as my keys.

The screen has developed some light scratches (key-pocket regime not strict enough), but I hadn’t really noticed them until I came to write this post.

The cover for the earpiece & notification LED fell off a couple of months ago. Didn’t seem to change anything.

The phone locks up and stops responding maybe once every 8 weeks, sometimes it comes back but sometimes I have to take the battery out. It’s not really enough to be a problem.

Performance

Back in August I said:

The Sonic runs Android 2.3 Gingerbread extremely well – performance is snappy, apps open quickly.

Today I’m a little less upbeat. Things have changed in ten months. Many app updates have come. It’s clear that developers are now focusing on newer models. Consequently, the Sonic feels like it has been getting fractionally slower over time.

One of the most obvious culprits was the Facebook app, although I actually uninstalled that one a few months ago. Updates brought many nifty features that didn’t work particularly well on a lower-RAM lower-processor smaller-screen device.

Similarly, the Swype keyboard (which I still use and love) seems to have become mildly more ponderous with each increment (although still lightning-fast if you’re using it on last year’s flagship phone.)

As a consumer, this kind of upgrade cycle seems to be the real downside to the “Android Fragmentation” problem.

In some cases you can get around this by seeking out sleeker app alternatives. For example, the Twitter client “Twicca” is much faster than any alternative I’ve tried.

Overall, the Sonic is still usable, but the difference will only get more pronounced. At least until all new development shifts to Android 4 or newer, which the Sonic will never run.

As a Phone

As I said in the original review, the Sonic has decent ringer and call volume. However, I’ve since had people tell me the call quality on their end is poor.

Music Player

I haven’t used “Huawei Player” since I wrote the review. PowerAMP is a very good music player, well worth paying for.

I also haven’t really noticed the sound quality issues that I wrote about in the first review. There is a contributing factor to that – I nearly always listen to music on my little Sennheiser in-ear headphones while walking somewhere outdoors, or on speakers plugged into the stereo at the Hackerspace where music competes with the sounds of angle grinders and mitre saws.

Battery Life

Battery is still good for a smartphone. Recently I turned off email and Twitter notifications¹, and in combination with dropping Facebook a few months ago I now easily go more than two full days without charging (but still with some online use.)

Camera

This is the one feature that makes me regularly dislike the Sonic. As I said in the original review, image quality is poor especially in bad light. Worse, the camera app is quite buggy – on my phone it sometimes takes a long time to start, and one in every few launches it crashes outright. Grabbing a quick snap is not really an option.

“Rooting” & “Custom ROMs”

Even though the factory Android 2.3 install is pretty good, I was really hoping for a port of Cyanogenmod. If only to get past some of the niggly bugs in Android 2.3.3 (Huawei haven’t released any official updates, to my knowledge.)

As it happens some unofficial Cyanogenmod 7.1 ports have surfaced, done by some Spanish developers.

Unfortunately they’re the kind of no-source-code no-guarantees dodgy-upload-site affair that seems to be pretty common in the Android modding community. It really jars me, coming from the broader FOSS community where there’s so much emphasis on trust, identity, and source code transparency. There’s just no way I trust these kind of ROMs enough to install them on my own phone, I want to be able to see what it’s built from (and by whom, and whether anyone is supporting the releases) and repeat the process myself if I feel like it.

Not that I think this is something to blame the developers for, it just keeps me from embracing it as an option. It’s up to you to decide whether that makes a paranoid free software zealot or merely fussy!

In a world where I had infinite time I’d love to try and work on more open, inclusive, Android development for the Sonic. Along with the trillion other things on my backburner.

Other Alternatives

There are a lot more alternatives now, especially given the Sonic price hasn’t gone down in 10 months (still AU$188.) I haven’t researched any of the cheaper options this time around. I did notice that now the Samsung Galaxy S3 is out, the Galaxy S2 (which eclipses the Sonic in every measurable way) is available cheaply on carrier-subsidised plans.

For that reason it’s really hard to recommend the Sonic or any other budget smartphone, unless you absolutely don’t want a 24-month contract.

Bottom Line Revisited

I don’t regret buying the Sonic. I’m not planning to replace it in the near future, despite the annoying camera.

However, if I was looking to buy today then I’d seriously consider signing a cheap 24-month contract for a Galaxy S2, or I’d look for something like the S2 on the used market.

That said, I wouldn’t be surprised to hear soon that Huawei have dropped the price of the Sonic. If it comes down closer to $100 then it might be too cheap to pass up!

1. I did this after reading The Information Diet and trying to build more conscious information consumption habits. With mixed success.

By following these steps, you should be able to use the FV-1 a bit like a normal desktop Linux computer. However, it’s still a long way from my dream goal of running XBMC for Linux.

(Edit 11 March – added a couple more notes about exactly how poor the UI graphics performance is.)

What This Is

A lightly customised installation of Debian Wheezy, set up initially with a minimal “LXDE core.” Kernel is 3.3rc from linux-samsung-soc, plus some minor patches for FV-1 hardware support.

Debian Installation

This configuration has the kernel loaded into the internal flash of the FV-1. However, the root filesystem with Debian is external – on either a USB device or an SD card.

If you want to reverse back to Android after testing Debian, you should be able to perform a normal FV-1 firmware upgrade using their “SDtool” to prepare an SD card.

In theory, it should also be possible to install Debian into the internal flash. However, the m-boot bootloader is quite peculiar so I haven’t tried.

How to install

1) Download an SD card image to flash the kernel – use this one if you to boot Debian from an SD card, and this one to boot Debian from a USB device. The only difference between the two is the kernel commandline in the config, you can see the configs here.

2) Work out what device name your SD card reader in your computer has. It will probably look like /dev/sdX (for USB card readers) or /dev/mmcblkX (for built-in card readers). I’ll be using /dev/sdX for the sample commands here. You can probably work this out by inserting an SD card, waiting for it to mount on the desktop, and then typing “mount” to see what is mounted.

3) Flash your SD card with the kernel upgrade. This will erase any content already on the card.
umount /dev/sdX1
dd if=kernel-boot-usb.img of=/dev/sdX
sync


4) Remove the SD card and insert it into the FV-1. Power on the FV-1. The red light will come on, followed by the green light. After 1 minute or so the green light will start blinking. This means the new kernel has finished flashing. Power off the FV-1 and remove the SD card.

5) Download the rootfs tarball. This is the filesystem that the kernel will boot into.

6) You then unpack the rootfs, to your SD card or USB device. If you only have one SD card, you can reuse the one which you just used for the kernel (although you’ll need to repartition and reformat it.) I suggest formatting as ext3.

7) Once your SD card is ready, unpack the rootfs:

cd /media/[path-to-my-SD-card-mountpoint]
sudo tar zxvf ~/path/to/rootfs.tgz
sync
cd ..
eject [path-to-my-SD-card-mountpoint]


8) Insert the card back to the FV-1, and power on.

9) The FV-1 will boot to a login screen, using the lightdm login manager. Alternatively, if you don’t have a screen then you can connect an ethernet cable – the FV-1 will establish a connection using DHCP and you will be able to ssh into it.

10) The default username is ‘user’ and the password is ‘android’

11) The stock image only has a very basic LXDE GUI desktop (lxde-core.) When you first log in, go to the LXDE menu (bottom-left) and choose Accessories -> LXTerminal.

12) Use “passwd” to change the default password.

13) Optionally, use “sudo adduser (username)” to add a new user then “sudo usermod -aG sudo (username)” to give them sudo. Then reboot and login as your new user, and “sudo deluser user” to remove the initial user.

14) If you’re not in the US, edit /etc/apt/sources.list and change to a local mirror for packages.

15) To install a full GUI LXDE environment, run “sudo apt-get update” then “sudo apt-get install lxde”. It may also be possible to remove lxde-core and install other desktop environments like Xfce4, GNOME, KDE – as described in the Debian documentation.

16) When installing LXDE (or any other environment), the gdm3 package may prompt you to choose a default login manager. Keep the current “lightdm”. Any new manager will need to be preconfigured to use Xwrapper.sh to launch X (see below.)

17) Happy computing! From now on, everything should behave like a standard Debian Wheezy system.

Keep reading for the status of the hardware support.

Things That Work

X11 graphics

Up to 1920×1080 over HDMI, 16-bit colour.

USB, ethernet, SD card slot.

Things That Might Work

HDMI audio output

I’m using an HDMI->DVI cable and a computer monitor, so I haven’t been able to test audio output.

Things That Don’t Work

Wifi & Bluetooth

This is another case where having the actual GPL source release would be very helpful. The wl12xx_sdio driver detects the chip correctly, sometimes successfully uploads firmware, and very occasionally I’ve managed to get it to associate and send/receive packets. More commonly it just crashes with MMC read/write errors.

My vague theories for why it is not working include:

• An incorrectly set clock frequency (if I force lower the SDIO frequency it becomes more reliable but not 100% reliable.)
• Samsung HSMMC driver bug of some kind.
• A floating GPIO line that I don’t know about.
• Unpublished differences between Samsung’s SWB-T37 chip and the original TI WL1271.

Changing screen resolution normally

The X server is launched via a custom wrapper script, /usr/local/bin/Xwrapper.sh, that sets the screen resolution for the HDMI output before X starts. Out of the box it is set for 720p (1280×720), but you can edit that file to get 1080p (1920×1080) or 480p (720×480.)

Detecting available screen resolutions

The mainline kernel has no support for DDC/EDID (receiving monitor identification data) over HDMI on the Samsung S5PV210 SoC.

Blanking/unblanking the screen

This includes restarting the X server – any time HDMI output is disabled and then reenabled it hard locks the system. I’ve committed a note on the fv1_hacks branch showing where the system locks the second time around, but I haven’t worked anything else out yet.

To work around the issue there’s also a custom xorg.conf installed, disabling all screen blanking functions.

I think the lockup may be related to the fact that (unlike fancier hardware) the FV-1 has no system-controlled voltage regulators. This means some HDMI subsystems that would otherwise be disabled when HDMI output stops stay running, and cause problems. This is only a theory, though.

Video acceleration

This is the big one. Currently single buffered, software acceleration only – means any graphics-intensive or interactive operations tend to use a lot of CPU time and can be very jerky.

The videobuf2-fb driver that the XServer runs on (ie a framebuffer driver that maps to a V4L2 output device) is only a proof of concept, made by Samsung. It would be possible to add some more features, at least in principle. Or alternatively to write an Xorg driver that outputs directly to v4l2 (this is how Android does it.)

3D acceleration

I’ve been messing with the PowerVR drivers for the SGX540 core, but it’s all a closed-source restricted mess and very unlikely it can be made to work. Hopefully the FSF will get some traction behind their goal of open source PowerVR SGX drivers.

SD card plugging & unplugging

For some reason the Card Detect interrupt doesn’t work on the SD card slot. An SD card will work fine if its inserted at boot time, but not if it’s plugged/unplugged later on. I’ve committed what I think should work to a branch (can even see the CD line changing state and attach the interrupt to it), but I can’t make it actually work.

RF remote control

I spotted data from this coming in over the i2c bus but I haven’t tried to decode it, or work out functions like pairing, or discover if it has its own GPIO interrupt line. There’s a kernel driver for the remote in the vendor’s kernel, but due to GPL violation that’s not of any help.

Restarting

The system won’t reboot, you have to shutdown and then cycle the power by plugging/unplugging the cord!

That’s All

I hope this is of some use. Patches are very welcome if anyone works any of the issues out. :)

I recently purchased two “Android TV” boxes from aliexpress, for investigation and general hackery.

You can buy these from various online vendors (dealextreme, aliexpress, etc.) Also under other names – for instance it’s sold here in Australia as the Kogan Agora TV.

The “original” product is the Flexiview FV-1, made by Mesada Technology, Shenzhen.

High-Level Specifications

• Cortex-A8 1Ghz CPU (Samsung S5PV210 SoC)
• 512Mb RAM
• Hardware supports 1080p video decoding, 2d/3d acceleration
• HDMI output
• Wifi, bluetooth, ethernet connectivity
• 3 USB 2.0 ports
• RF remote with wiimote-like infared pointing feature

Price varies between $130 & $150 online.

Bear in mind that like most of these “shanzhai-esque” products, what’s advertised on the web page may be different to what you get. There are a few similar-looking products out there – I’ve seen ones with SoCs from Rockchip and Amlogic. Also, specs and build quality can vary widely depending on vendor.

First Impressions

The hardware is cheap & cheerful. The marks and scratches visible in the photos weren’t me, it was like that out of the box. EDIT June 2012: Tony points out in the comments that there’s a thin film covering the piano finish, and he’s absolutely right. Here’s a quick photo, the top-left corner is a reflection.

Nevertheless, things mostly work and the build quality seems OK. One thing I did notice is that the video output can be a bit picky on the stock firmware – only some of my monitors worked, and I saw reports of some TVs not working. I think the requirement may possibly be supporting a 720p YPbPr video mode.

The other cheap and nasty bit is the little plastic “door” on the front covering 2 USB ports. It’s hard to open and close, and I doubt any serious user would leave it there for long.

Android TV?

Three word review of the “Android TV” experience: “really, don’t bother.”

If you’ve never heard of “Android TV”, that’s because it doesn’t really exist. There is a thing called “Google TV”, but those products are relatively rare and closed-source and not available for $130.

“Android TV” really means “the plain vanilla phone operating system, with a remote control that emulates a mouse & keyboard which you then use to control an interface designed for touch-screens.”

Sound nearly unusable? It is. A touchscreen interface is not a PC, which is in turn not a remote control interface. It’s horrid.

The device works pretty well with a wireless keyboard and/or mouse. Which is fine, but you’d be just as well off buying a tablet with an HDMI port (for less money) or even just hooking up an old PC.

I didn’t use Android much – I can confirm the browser works quite nicely. However, looking around the internet I saw a lot of reports of patchy video playback. Which is disappointing.

Firmware Updates

As usual for these devices, the vendor offers no after-sales support. Firmware updates are scattered around the net in strange places (others are hidden in various giant forum threads!)

So why did you buy two?

I bought these because I have a weakness for cheap low-power ARM hardware. I wanted to see how easy it was to port GNU/Linux, and maybe have a stab at porting XBMC. I’ll put details about running GNU/Linux in a follow-up blog post.

Of course, to get hacking, you need to open the cover:

Here’s the board. The sloppily attached cable is the serial console. Thanks to eDIY NZ for posting the serial cable pinout online – it’s a 3.3v lvTTL port, 115200bps. Looking top-to-bottom in this photo the pinout is 3.3v/TX/RX/GND. The hot glue was originally for strain relief!

For a serial link, I used my favourite source – a mobile phone USB<->Serial cable that I got for $2 at an op shop¹.

Also note the unpopulated USB port, some unpopulated front & rear connectors, and the button cell battery. The battery surprised me, I wonder why they didn’t drop it to save on BoM (bill-of-materials) cost.

The large clear LEDs at the front are the IR LED beacon for the “wiimote-style” remote. The red LED is for power (amusingly, HDMI standby power can come from the monitor and light it even when the unit is off!) The green LED is connected to a GPIO.

Under the covers

Pry off the RF shielding and things are a bit more interesting. “Open in New Tab” on the images if you want to be able to read chip names.

Left hand area contains:
* Power regulation.

Middle area contains:
* Asix AX88772A USB to 10/100 Ethernet adapter
* RAM
* System-On-A-Chip (Samsung S5PV210)
* Sandisk flash memory. I believe this is an “embedded MMC” interface – it shows up as an SD card to the kernel.

Right area contains:
* TI WL1271 combined Wifi & Bluetooth adapter, packaged by Samsung as the SWB-T37 and connected via SDIO.

On the underside, the most interesting thing is the Nordic Semiconductor nRF24LE1 2.4Ghz digital transceiver under the left-hand shield. This is connected via SPI and talks to a similar model IC in the remote control. I haven’t looked at the protocol yet.

Under the main cover, it looks like there is an unpopulated LCD interface. Plus something else totally unpopulated on the bottom left. Given that Mesada also make many GPS devices, maybe there is a car GPS that uses this board?

PCB identifier for this batch is 9000J-MSD-SACC-PCB-110107. Which I guess makes it just over one year old!

GPL? What’s a GPL?

As I’ve written before, GPL compliance for these kinds of products is generally poor. The FV-1 is no exception, as far as I know it is impossible to get source for the exact Linux kernel running on the device.

It is possible that distributors outside China, like Kogan here in Australia or “Flexiview NZ”, are able to supply GPL source code – they have more reasons to act ethically and not break the license, and potentially severe consequences if they are distributing a violating product. I don’t have any more information either way.

In the meantime, I bought this device anyhow because Samsung are generally proactive with open source development, and S5PV210 SoC support is part of the mainline Linux kernel. In practice, it was much more work than I anticipated to get a kernel building from source that runs on the device.

This is partially because of differences between the FV-1 and any available reference board. Partially because not all S5PV210 support has made it to the official linux-samsung-soc kernel. Partially because embedded development is always a bit of a pain…

Vendor Kernel on Device

The device comes with a 2.6.32 Android kernel loaded. Here’s a dump of the boot messages and dmesg, plus some interactions from the Android side – /proc entries, mount points, etc.

Among the custom additions in this kernel are:
* The TI vendor WL1271 driver (open source, but different to the Linux kernel one)
* The PowerVR SGX540 3D accelerator drivers
* In-kernel driver (it seems) for the RF remote, via SPI.

Also of note is that Android isn’t using a Framebuffer device for the HDMI output (the framebuffer in that kernel is for the LCD only.) Android must be using an earlier revision of the v4l2-based s5p-tv interface.

3.2 Kernel

As I said, it seems like no source exists for the 2.6.32 “vendorkernel” that ships on the device.

I now have a 3.2rc kernel that boots and can run a normal GNU/Linux, with xorg over framebuffer, available here on github. Not everything works yet, though. Some of the fixes are a bit hacky and need to be cleaned up. Hopefully, soon some of the support can go upstream to the mainline kernel. More on this kernel in a follow-up post.

Other Reference Kernels

After much searching, I found two other helpful S5PV210/S5PC110 kernel sources (not that I ever successfully booted them on the FV-1.)

The Samsung Galaxy GT-I9000 source code, available from opensource.samsung.com, is for very similar hardware.

A Chinese development board called the RealARM Realv210. This looks to be very similar to Samsung’s SMDKV210 development board. I think the FV-1′s reference board was either the Realv210 or the SMDKV210.

RealARM are distributing a number of useful resources, including a Linux 2.6.32 kernel for the Realv210 board. This reference has been very helpful in getting 3.2 up and running.

Mesada mboot

The last vendor surprise is the bootloader. “Mesada mboot” is the strangest embedded bootloader I’ve ever seen – among its features it appears to be able to both read and format EXT3 filesystems! Unfortunately, mboot is also closed source and no documentation is available.

Here’s a dump of mboot performing a full system update for an Android firmware

The good news is I’ve managed to reverse engineer enough of the format to “unpack” and “repack” mesada’s .osk firmware update files. This allows the flashing of a new kernel, or new system partitions, or even a new bootloader. The project is called mboot-tools and is up on github. There are some usage instructions in the README there, and lots of comments in the header file.

It should also be possible to use mboot_unpack & mboot_pack to make “custom Android ROMs” or possibly even as a first step to porting a newer version of Android (after all the hardware is very similar to Galaxy/Nexus S, which can run ICS.)

What’s Next

I’ll put a post up soon explaining how to boot Debian on the FV-1, explaining what currently works and what doesn’t.

1. “thrift store”

The best introductions to scratchbox2 that I’ve found are this wiki page from the plugcomputing people, and this nice succinct set of presentation slides from Riku Voipi.

One of the big annoyances has been seeking information, Google likes to replace sb2 (the main scratchbox2 command) with db2 and scratchbox2 with scratchbox.

Multiarch

I have found that scratchbox2 (as of 2.3.3) is a bit fiddly when targeting Debian wheezy or sid (testing or unstable.) This is due to the new Debian Multiarch support.

In a nutshell, multiarch means libraries and tools are moving from locations like /usr/lib to arch-specific locations like /usr/lib/[triplet] where triplet identifies the architecture, ie arm-linux-gnueabi for ARM systems.

The symptoms I saw were linking errors (or configuration failures) due to missing libraries, when you know they are installed on the target. The crucial test is whether “sb2 [some-simple-command]” fails while “sb2 -e [same-simple-command]” passes – this implies that the problem is in the host translation, not in the target (otherwise it would fail in the -e qemu’d environment, as well.)

Here’s what worked for me. In the file ~/.scratchbox2/[TARGETNAME]/gcc.config.lua, I sought out and changed the following two config items as follows, to add linker search paths:

extra_cross_compiler_args="   -L@SBOX_TARGET_ROOT@/usr/lib -L@SBOX_TARGET_ROOT@/usr/lib/arm-linux-gnueabi -L@SBOX_TARGET_ROOT@/lib  -L@SBOX_TARGET_ROOT@/lib/arm-linux-gnueabi",

extra_cross_ld_args="-rpath-link @SBOX_TARGET_ROOT@/usr/lib/arm-linux-gnueabi:@SBOX_TARGET_ROOT@/usr/lib:@SBOX_TARGET_ROOT@/lib:@SBOX_TARGET_ROOT@/lib/arm-linux-gnueabi",

I also ran the command

sb2-config setenv PKG_CONFIG_PATH /usr/lib/pkgconfig/:/usr/lib/arm-linux-gnueabi/pkgconfig/

(In all the above, replace arm-linux-gnueabi with your target architecture’s “triplet” of choice.)

As always, the solutions are simple and easy to follow once you know what they are. :) Please bear in mind also that this may not be the best solution, I’m still a scratchbox2 newbie.

Distro Versions

The other major gotcha, hinted at in the presentation I linked to, is that the host and the target distros must match exactly. If you have different library versions on each then you are headed for problems.

This can be tricky even when the distros match. At time of writing debian wheezy uses libc6-dev 2.13 on amd64 but emdebian wheezy uses libc6-dev-armel-cross 2.11 for the cross-compiler. The header files are different enough to cause builds to fail. To get matching headers I had to change to emdebian sid, which currently has libc6-dev-armel-cross 2.13.

An alternative, also hinted at in the presentation, is to set up a “tools” distro. This is a directory with a complete host-arch distribution, matching perfectly the target’s distro except for the architecture. It seems the idea there is to create a controlled single-purpose environment, instead of just sharing the host’s tools. Maybe that’s a cleaner path, especially if it lets you sandbox a stable distro release with predictable versions.

(Extended Rant, Feel Free to Ignore)

The header-mismatch problem seems to be caused because programs compiled under sb2 can search for headers in both the host directory /usr/include/, and also in the cross-compiler directory (ie /usr/arm-linux-gnueabi/include/), -and- in the target rootfs directory.

Exactly which header file gets chosen for which purpose seems to be somewhat haphazard – in the example I saw, a #include <fcntl.h> was pulling in /usr/include/fcntl.h which was then pulling /usr/arm-linux-gnueabi/include/bits/stat.h.

I haven’t dug deep enough into sb2 to see what’s really going on here with gcc include paths, but it seems like it should be possible to workaround this and ignore the host’s headers entirely – in favour of the cross-compiler headers & those installed in the target rootfs.

However, passing these paths explicitly is exactly the kind of drudgery that sb2 tries to avoid with its “magic”, so at this stage I’m leaving it – my program compiles & links, so that’s good enough for now. If I end up having to explicitly set everything then I may as well create a conventional cross-compiler environment.

Well, actually I learned this a few weeks ago while developing the Hairless MIDI<->Serial Bridge. But the blog post had to wait until today.

The Problems

• By default, serial latency with FTDI chips (including Arduino Duemilanove/Mega) on Windows & Linux can be quite high (>16ms) and unpredictable.
• In audio applications (like sending MIDI data), this can add enough latency to create audible artifacts.
• In Java-based applications that use it, librxtx introduces an additional 20ms of latency. I wasn’t using Java or librxtx, but you’ll want to read that if you are.

The good news is that you can reduce FTDI latency substantially with a simple tweak.

What’s Latency?

In this case, latency is the amount of time between when some data gets sent from one side (the Arduino), and received on the other side (the computer.)

In lots of cases latency doesn’t matter, or you accept higher latency in exchange for higher throughput. However, for real-time applications like MIDI controllers, you don’t want a noticeable delay between pressing a button and hearing the sound that it makes.

The consensus seems to be that for acceptable MIDI audio responses, you need to keep MIDI message latency under about 20ms.

FTDI Latency Timer

The problem stems from the Arduino’s “Serial to USB converter” chip, the FTDI FT232R. The FTDI can’t send a USB packet to the computer for every byte that comes from the Arduino’s microcontroller. Instead, it stores the serial data in an internal buffer and only sends a USB packet when the buffer is full, or after a period of time has elapsed. This period of time is determined by the FTDI Latency Timer, which is the reason why FTDI chips can give bad latency characteristics.

On Linux & Windows, the default latency timer setting is 16ms. For example, say you send a 3 byte MIDI message from your Arduino at 115200bps. As serial data, it takes 0.3ms for the MIDI message to go from the Arduino’s microcontroller to the FTDI chip. However, the FTDI holds the message in its buffer for a further 15.8ms (16ms after the first byte arrived), before the latency timer expires and it sends a USB packet to the computer.

Thankfully, the latency timer can be tweaked. The tweaking method varies between operating systems.

Linux

In proper Linux style, the kernel’s FTDI driver exposes a nice sysfs interface that lets you get and set the latency timer. For example, if your serial port is ttyUSB0:

# cat /sys/bus/usb-serial/devices/ttyUSB0/latency_timer
16
# echo 1 > /sys/bus/usb-serial/devices/ttyUSB0/latency_timer
# cat /sys/bus/usb-serial/devices/ttyUSB0/latency_timer
1

… that will lower the timer from 16ms to 1ms (the minimum), to reduce latency.

In my experience, the timer value won’t change immediately on an open serial port. If an application is using it then you’ll need to close and reopen it before the new value takes effect.

If you’re writing code, there is also a Linux-specific serial flag ASYNC_LOW_LATENCY that programmatically sets the latency timer down to 1ms. This is how Hairless MIDI<->Serial Bridge does it. You can see a succinct C code example in this patch I submitted to the ttyMIDI project.

In testing, I found that ASYNC_LOW_LATENCY also only works if you subsequently close the serial port and then reopen it (annoying, because setting the flag requires you have open()ed it already.)

Windows

FTDI’s own driver for Windows has a combo box in the Port Settings dialog that lets you choose the latency timer value. This Instructable has some screen shots showing how to find the setting in the Windows Device Manager control panel.

Programmatically, setting the timer is a bit hackier on Windows but not impossible. The FTDI driver saves the current latency setting for each device in the registry, so you can use Microsoft’s Registry API to write a new value, then reopen the serial port. The registry key is

SYSTEM\CurrentControlSet\Enum\FTDIBUS\-device id-\0000\Device Parameters

There is a code example for this hack in the Hairless MIDI<->Serial source code.

OS X

OS X does things differently. The driver bundle contains a file, /System/Library/Extensions/FTDIUSBSerialDriver.kext/Contents/Info.plist. This XML plist file describes different profiles for the serial port, including different LatencyTimer values, depending on how the FTDI identifies itself on the USB bus. FTDI’s own Technical Note on the subject [PDF] explains how to edit that value to change the latency.

The good news is that on OS X the latency timer defaults to 2ms for any FTDI FT232 that uses the default vendor & device USB IDs (0403:6001). This includes Arduino & clone FTDIs, so there is no real need to change anything.

How Fast?

I ran a test of the tweaked latency timers. The test sketch sends a MIDI note (3 bytes), then waits to see that same note echoed back. It does this many times and calculates the average delay.

With the latency timer set to 1-2ms, the entire round trip averages 18-19ms. This includes at least 1ms spent in the MIDI framework on the computer. So, based on those results, I estimate the one-way latency to be under 10ms. Excellent!

With a 16ms latency timer, the one-way latency would have been 25ms or more.

I don’t know how much of the 10ms latency is now coming from FTDI/USB layer, or from higher layers in the host operating system. It’s good enough for MIDI use, so I stopped investigating!

Which Arduinos?

Arduinos with FTDI chips include the Arduino Duemilanove & Mega, and some clones like the Seeeduino.

The newer Arduino Uno & Mega 2560 have a different AtMegaU8 chip, programmed to behave as a USB/Serial converter. According to tests I’ve seen, these have good latency characteristics.