Air Quality Egg: Shield Firmware Update

Based on my research findings (described in my last post), I spent some time digging into the Egg Shield hardware and firmware and I made some significant updates in hopes of improving results. I’m currently running an overnight experiment of the latest firmware, which I pushed out on GitHub (also for the Ozone and VOC add-ons). I also put load scripts out there to help ease the update process. I wanted to give an update on what I found and what I did.

Constant Heater Voltage

There was an observable random “spiking” phenomenon associated with the dynamic heater voltage control algorithm. I still don’t know the root cause of that, but I know that if I take that algorithm out, the associated phenomenon goes away, so the current version of the firmware sets the voltage at start-up and leaves it alone. I determined the settings for each heater experimentally by trying a setting, measuring H+ and H- and calculating the heater power, until I hit the datasheet targets.

Range Selection Algorithm / Fixed Point Math Rewrite

I was concerned about the frequent reports of apparent overflow conditions occurring. I looked into how to make better use of our range selection capability and rewrote it to select the range that yields results that are closest to the mid-range voltage. I believe this algorithm should provide more stable / continuous results than the original algorithm. In particular, I expect that it will be more immune to overflow conditions.

NO2 Hardware Bug – Workaround

In re-writing the Range Selection Algorithm, I was astonished to find an undiscovered hardware bug in the Egg Shield. I kept trying different settings on the range selection divider for NO2, but no matter what I did, two of the range settings always yielded the same analog-to-digital conversion result. I dug in and traced it back to the schematic (after hours of assuming I must be doing something wrong in software).

It turns out there is a node named “/NO2_R2_SEL” and a separate node named “/R2_SEL.” Those were meant to be the same node, but the tools don’t know that, so they never got connected in the layout. The effect is that it’s impossible to select a low-side resistance of R1 + R2 = 24.2 kΩ for NO2. <face-palm!> The firmware didn’t know about that bug either, so its model of the world didn’t agree with reality when it chose that range. This inconsistency was almost certainly also contributing to the instability in the NO2 readings we’ve been seeing. Anyway, the firmware agrees with the hardware now. I think it will be less accurate than it could have been were it not for the hardware bug, but I think this will be much better than what we had going on.

What To Do Next

I’ll follow up with a post tomorrow night with my overnight results. If you’re ambitious and know what you’re doing, the software and binaries for the firmware updates above is already out on GitHub and you can go ahead and re-flash your Egg Shield to hopefully start getting better results right away. As a reminder, Joe Saavedra also made a nice tutorial on how you can use an Arduino (or a Nanode) to reprogram a Sensor Shield. I plan to do a video to demonstrate that process as well in the next couple weeks. As always, if you’re uncomfortable reprogramming the hardware for yourself and  you are willing to pay for the round-trip shipping, I am willing to personally reprogram your hardware at no additional cost.

Air Quality Egg: Research Findings

I’ve been looking at a lot of the data being published and trends in questions on the forum, and there is a theme developing. The results are not consistent with expectations. We’re also now getting some reports from laboratory studies that are confirming there’s a problem. The collective information strongly suggests that the data the Air Quality Eggs are reporting is incongruent with the natural response of the sensors.

In light of this growing body of evidence, I built a much simplified reference circuit based on the sensor datasheets and recorded sensor readings along side the Egg for several hours. Sure enough, the reference circuit exhibited a fairly smooth / continuous response over time. The Egg did not. 

I decided the rational thing to do was to conduct a series of controlled experiments in an effort to establish the root cause of the variation between the Sensor Shield and the Reference Circuit behavior. This was a daunting proposition with so many potential variables in play, but I did my best to come up with some viable hypotheses. The fundamental differences between the reference circuit and the Sensor Shield / Air Quality Egg are:

• The Air Quality Egg had a fan (which affects airflow and electrical load), the reference circuit did not
• The Air Quality Egg was in an enclosure, the reference circuit was not
• The Sensor Shield dynamically controlled the heater voltage, the reference circuit did not
• The Sensor Shield dynamically selected the voltage divider, the reference circuit did not

For the first two bullets I did a series of tests that ultimately revealed the enclosure didn’t have much of an impact, but that disconnecting the fan from the Air quality Egg circuit has a very noticeable impact. As a quick aside, all of the following graphs have elapsed time in milliseconds on the x-axis and ADC counts on the y-axis. The following graph compares the response of the reference circuit, to an Air Quality Egg with a fan and enclosure, and to an Air Quality Egg with neither a fan nor an enclosure.

The difference is readily apparent. The data with the fan (red) has a bout 50 ADC counts of noise on it. The data with no fan (blue) in the circuit, however, still had a peculiarity of these randomly appearing spike discontinuities in the data. I made modified builds of the Sensor Shield firmware that took out the Heater Control algorithm, took out the Range Selection algorithm, and took out both.  The result of taking out both looked like this (with and without an enclosure) in comparison to the reference circuit.

Lo and behold, with the fan removed and the software sophistication removed, the Air Quality Egg response has neither noise nor spikes, and it seems to track the reference circuit quite well. I then added just the Heater Control algorithm back in and I got this result:

This confirms, in my view, that the Heater Control algorithm is the source of spiky-ness in the Air Quality Egg data. I’m currently studying the software intensely to come up with an improved version of the Sensor Shield firmware, and will post another update when I know more.

“Air Quality Egg” is now a trademark

We have followed in the footsteps of Arduino™ and received a trademark on the “Air Quality Egg”. This post explains why we did this, and it’s effects.

A quick note in case you are unfamiliar with Trademarks: This has no effect on the open source-ness of anything of the hardware and software. It is still 100% open source.

This blog post has a pretty good explanation why open source projects should trademark. A trademark prevents other people from calling something an “Air Quality Egg”. We did this for two reasons:
- It would be confusing to us if other people created a product with the same name.
- If might also confuse customers, and they might expect us to support all products called “Air Quality Egg”

Here is another good discussion on trademarks and open source.

Air Quality Egg status update

All,

It has been awhile. Here is an update on the Egg.

If you ordered:

• an egg with no add-ons (Just the basic egg),
• a shield only or
• a Kit,

Then we have shipped your order. If you don’t think we have, then please contact support AT wickeddevice.com

If you ordered an Egg with any add-ons (Ozone, VoC, etc), we have not shipped your egg yet, but will be doing so soon (starting next week).

If you ordered a radiation module, as noted in January, Joe Saveedra has been kind enough to design and help with delivering it.

Wicked Device announces strategic partnership with ITT Exelis

ITT Exelis (NYSE: XLS) and Wicked Device LLC have entered into a strategic partnership to explore innovative technology to provide real-time weather information at a hyper-local scale.

Wicked Device is developing an embedded system to integrate with Exelis’s Helios™ platform. The Helios platform enables traffic and other cameras to function as weather sensors, providing more accurate, real-time environmental intelligence for better decision-making and response.

The Helios platform integrates networks of surveillance cameras already in use to watch traffic, facility security and railroad assets for weather monitoring. It aggregates disparate images into a single source and uses algorithms and image processing to reveal actionable real-time environmental intelligence. Providing hyper-local weather data, versus regional data, to national and local agencies, utility companies, transportation services and first responders can help in their decision-making process, saving lives and resources.

AQE: Enhanced RGB User Feedback

Over the weekend I worked on making the color expressions of the Air Quality Egg more… well, expressive. The update is on Git-Hub so feel free to update your software (howto videos) for the Remote and Base units. This is the 1.03 version of both baselines (version number gets printed to the serial console).

As the Eggs have been reaching their destinations, feedback from users indicated that the setup process had some significant holes in diagnostic feedback. The most important of these was that the outcome of the pairing interval was invisible to the end user; the only way you knew that pairing succeeded was that the system started working a couple minutes later. If the system didn’t start working, you were left scratching your head and trying again. So I added an acknowledgement to the pairing behavior, and now there’s clear feedback about the outcome of the pairing interval: after the yellow pulses that signify pairing in progress, three magenta blinks indicate “pairing was not acknowledged” and three cyan (light blue) blinks indicate “pairing was acknowledged.”

The second feedback notable feedback confusion was related to Cosm provisioning. The previous behavior was to indicate solid green when provisioning succeeded, but *only the first time* after that you would never see green again. So now, if provisioning has previously succeeded, you get three green blinks.

In summary, the colors the Egg expresses now come in three “flavors”:

• Pulsing colors are used to indicate normal activity and progress (I know some people find this annoying, but I’m erring on the side of more feedback)
• Blinking colors are used to signify status
• Solid colors are used to indicate conditions that result in a restart
• Blinking/Solid green, blue, and cyan are used as “positive” indicators
• Blinking/Solid red and magenta are used as “negative” indicators

The following flowchart describes the feedback behavior of the 1.03 baseline.

AQE Bugs

The Short Story

There’s a bug with Cosm activation. If your Egg is not publishing data, here’s what you have to do, without getting into the details:

Option 1: Send us back your Air Quality Egg, along with prepaid return postage. We will re-program it for you.

Option 2: Update the software yourself. Steps:

1. Get an “FTDI cable.” You can get one from us ($22 or $13), or your local reseller (or if you have an Arduino laying around…)
2. Download the latest software from Git-Hub.
3. Reprogram your Eggs (we’ll post a video and some instructions to help in the next few days).

How to choose an option: We may discover more bugs in the future. Option 2 empowers you to reprogram your Eggs whenever additional bugs are found, or enhancements are published. If you are not comfortable doing that, choose Option 1.

The Longer Story

I have some unfortunate news. Provisioning with Cosm seems to have stopped working somewhere along the way. The good news is I’ve already got a software fix. The not-so-good news is that it’s pretty likely we’ve already shipped your Egg.

As I’ve described in other posts, when your Egg is first plugged into a network it reaches out to Cosm to declare its existence. In exchange for this declaration, (and by virtue of the fact that we’ve pre-registered your serial number) Cosm issues your Egg an API Key and Feed ID. The Nanode should extract these bits of information, remember them, and thereafter use that information to post information to the Cosm website under the prescribed Feed ID. This was all working, but something has changed. The software I wrote for the Base Egg wasn’t prepared for the change. I think the Cosm responses used to be fragmented into two or more packets and that is no longer the case. In a nutshell, Cosm is getting the message and issuing the activation data to the Egg, but the Egg is ignoring that response, and just keeps re-attempting activation.

The technical detail is in Provision.ino in the AQEBase sketch:

static void provisioningCallback (byte status, word off, word len) {

…

if (status == 1){ //we’re getting the body —

… all the parsing of the response happens here …

and  status is being returned as 0 from the EtherCard library so the response is not being parsed. The fix is stupidly easy: remove the guard and everything works as expected. Honestly, I can’t remember why I put the guard in there in the first place. It was probably cut-and-paste from an EtherCard example.

GREAT, SO NOW WHAT!?

The simple truth is that this will probably not be the last bug we find that needs to be patched. I wish that weren’t true and that I were a perfect programmer, but I’m not. We (Wicked Device) also can’t afford the shipping costs of offering to personally reprogram everyone’s Egg every time a bug is found and needs to be patched. Fortunately, the whole project is open source and I can push patches up to Git-Hub and announce when bugs are found to the Google Group and blog. I can also post instructions / videos showing you how to update your own units. The problem that remains is that you need a “special” cable to program the Egg units, commonly called an FTDI cable. In retrospect, I wish we had just included one with the Eggs, but we were already up against a wall with cost.

Here is a solution I think is reasonable: we will send you a suitable programming cable from our store (linked above), or  buy an FTDI cable (or equivalent) from the supplier of your choice. This will allow you to reprogram the Egg units anytime you need to. This cable will also be a necessary ingredient for performing sensor calibration in the future.

Please accept my apologies. We appreciate your understanding as we continue to do our best to support the Air Quality Egg community! This has certainly been a learning experience so far.

Mark Gibbs from Network World reviews the Air Quality Egg

On January 17th, Mark Gibbs from Network World has published a review of the Air Quality Egg. He made some great comments which we will be taking to heart (Better documentation, move the sticker off the box, both spot on). The review also had some comments we are quite proud of:

“And that’s it, the AQE system is a great example of plug n’ play done right.”

“… a huge thumbs up for delivering an ambitious solution to citizen-driven environmental monitoring.”

Egg voted top 20 Kickstarters of 2012

The Air Quality Egg was voted one of the top 20 Kickstarters of 2012!

http://www.kickstarter.com/year/2012?utm_campaign=Jan082013&utm_medium=email&utm_source=newsletter#air_quality_egg

Kickstarter Eggs Shipping

It has been a long road, and we are very happy to say that we are shipping our first batch of Kickstarter Air Quality Eggs. We are shipping plain eggs first (that is, with no add on boards).

This is what 100 Eggs looks like. The address labels are on the side so that they don’t cover up the packaging. (If the one upside down package is driving you nuts, don’t worry, we turned it around

Once you get your egg, remember to go to airqualityegg.com and enter the serial number, which is printed on the inside of the box. Finally, we will be shipping more soon – if your egg is not in this batch, you will be getting it in a week or three.