Air Quality Egg: Firmware/Software Update

Here’s the quickie summary. I reorganized the software on the Air Quality Egg to do the conversion math on the Nanode. Good news is this seems to be giving better results. The downside is it was a fairly broad impact, and I believe every board needs to be reprogrammed to get the system running at the latest baseline. I haven’t yet pushed the updated software and binaries up to Git Hub, but I’m running it and observing this feed, so far so good.

I’ve previously published instructions on how to reprogram the Nanodes using an FTDI-compatible programming cable in videos that are linked on the FAQ. I also plan to make a video that shows how to re-program the Sensor Shield and add-on modules, though that is admittedly not as easy as I’d like it to be for the average user.

I really want the fielded Eggs to get to latest version of the software though, so I’m willing to  reprogram units personally, at no additional cost, if you send them back to us and just pay for the round trip shipping. I know this is a real hassle for our international community members, with customs and all. If there are community members that are willing to be “local experts” for reproramming Eggs that would ease the process. If you are willing to be a volunteer in this capacity please let me know at support AT wickeddevice DOT com. It would be really great to have some local people help support Europe, Asia, and Australia.

Technical Details

Math Code Refactoring

The sensor shield code now responds with a record containing the following information in response to the EGG_BUS_SENSOR_BLOCK_RAW_VALUE request:

   uint32_t adc_value     :: the analog-to-digital conversion (ADC) count
   uint32_t low_side_r    :: the low side resistance associated with the ADC count
   uint32_t sensor_vcc    :: the high side voltage applied to the sensor divider circuit
   uint32_t adc_vcc       :: the reference voltage of the ADC
   uint32_t max_adc_value :: 2^(ADC # of bits), e.g. 2^10 = 1024 for a 10-bit ADC

Given this information, the Nanode computes the measured resistance as follows (using floating point math):

   vx = (adc_value / max_adc_value) * adc_vcc; // this is the voltage divider output voltage
   sensor_resistance = (sensor_vcc - vx) * low_side_r / vx;

And the value of the independent variable used for computing the interpolated response is calculated as sensor_resistance / R0, where the (mutable) calibration value R0 and the piece-wise linear response function table is retrieved from the sensor shield and applied as previously.

A few side note here:

• This refactoring effectively deprecates the EGG_BUS_SENSOR_BLOCK_MEASURED_INDEPENDENT field, as that quantity is now computed on the Nanode as a derived value. This is all obviously “hidden” in the EggBus library, but I did want to make that clear.
• I tried to keep enough information on the endpoint to allow for future flexibility. I could have included less information in the EGG_BUS_SENSOR_BLOCK_RAW_VALUE response, but it would be less generic. With the interface defined as above, a sensor endpoint could for example use a 24-bit ADC, or a different voltage reference, and so forth, without loss of generality.
• The terms may mean different things to different sensor endpoints, but they can rely on the Nanode to use them in downstream calculations as described above. The dust sensor, for example, has “occupancy” in “hundredths of a percent” as it’s raw value. To respect the calculation stream, the dust sensor now reports adc_value = 1, max_adc_value=1, adc_vcc=1, low_side_r=1, sensor_vcc=(occupancy + 1), and r0=100.

Data Refactoring

In the updated software, for each sensor there are three values reported to Cosm on every sample interval:

• Raw Value := the result of the sensor_resistance calculation above. Tagged with “aqe:data_origin=raw”
• Computed Value := the result of the interpolation calculation derived from the Raw Value, R0, and the piece-wise linear transfer function. Tagged with “aqe:data_origin=computed”
• R0 := reporting this to Cosm once calibration enters the picture we’ll need to recognize that this value may vary over time, but this should be a mostly constant. Tagged with “aqe:data_origin=r0”

One thing to note, I’m changing the meaning of the raw value feed here. Where previously it reflected the “independent variable” (i.e. the quantity resistance / r0), it now reflects the sensor resistance.

Library Updates

There have been some updates to the Ethercard library in recent months. I’m not sure, but perhaps the latest library baseline addresses the issue causing some Air Quality Eggs to freeze up in the field, requiring a restart to resume publishing data to the web.

Air Quality Egg: Shield Firmware Update

Based on my research findings (described in my last post), I spent some time digging into the Egg Shield hardware and firmware and I made some significant updates in hopes of improving results. I’m currently running an overnight experiment of the latest firmware, which I pushed out on GitHub (also for the Ozone and VOC add-ons). I also put load scripts out there to help ease the update process. I wanted to give an update on what I found and what I did.

Constant Heater Voltage

There was an observable random “spiking” phenomenon associated with the dynamic heater voltage control algorithm. I still don’t know the root cause of that, but I know that if I take that algorithm out, the associated phenomenon goes away, so the current version of the firmware sets the voltage at start-up and leaves it alone. I determined the settings for each heater experimentally by trying a setting, measuring H+ and H- and calculating the heater power, until I hit the datasheet targets.

Range Selection Algorithm / Fixed Point Math Rewrite

I was concerned about the frequent reports of apparent overflow conditions occurring. I looked into how to make better use of our range selection capability and rewrote it to select the range that yields results that are closest to the mid-range voltage. I believe this algorithm should provide more stable / continuous results than the original algorithm. In particular, I expect that it will be more immune to overflow conditions.

NO2 Hardware Bug – Workaround

In re-writing the Range Selection Algorithm, I was astonished to find an undiscovered hardware bug in the Egg Shield. I kept trying different settings on the range selection divider for NO2, but no matter what I did, two of the range settings always yielded the same analog-to-digital conversion result. I dug in and traced it back to the schematic (after hours of assuming I must be doing something wrong in software).

It turns out there is a node named “/NO2_R2_SEL” and a separate node named “/R2_SEL.” Those were meant to be the same node, but the tools don’t know that, so they never got connected in the layout. The effect is that it’s impossible to select a low-side resistance of R1 + R2 = 24.2 kΩ for NO2. <face-palm!> The firmware didn’t know about that bug either, so its model of the world didn’t agree with reality when it chose that range. This inconsistency was almost certainly also contributing to the instability in the NO2 readings we’ve been seeing. Anyway, the firmware agrees with the hardware now. I think it will be less accurate than it could have been were it not for the hardware bug, but I think this will be much better than what we had going on.

What To Do Next

I’ll follow up with a post tomorrow night with my overnight results. If you’re ambitious and know what you’re doing, the software and binaries for the firmware updates above is already out on GitHub and you can go ahead and re-flash your Egg Shield to hopefully start getting better results right away. As a reminder, Joe Saavedra also made a nice tutorial on how you can use an Arduino (or a Nanode) to reprogram a Sensor Shield. I plan to do a video to demonstrate that process as well in the next couple weeks. As always, if you’re uncomfortable reprogramming the hardware for yourself and  you are willing to pay for the round-trip shipping, I am willing to personally reprogram your hardware at no additional cost.

Air Quality Egg: Research Findings

I’ve been looking at a lot of the data being published and trends in questions on the forum, and there is a theme developing. The results are not consistent with expectations. We’re also now getting some reports from laboratory studies that are confirming there’s a problem. The collective information strongly suggests that the data the Air Quality Eggs are reporting is incongruent with the natural response of the sensors.

In light of this growing body of evidence, I built a much simplified reference circuit based on the sensor datasheets and recorded sensor readings along side the Egg for several hours. Sure enough, the reference circuit exhibited a fairly smooth / continuous response over time. The Egg did not. 

I decided the rational thing to do was to conduct a series of controlled experiments in an effort to establish the root cause of the variation between the Sensor Shield and the Reference Circuit behavior. This was a daunting proposition with so many potential variables in play, but I did my best to come up with some viable hypotheses. The fundamental differences between the reference circuit and the Sensor Shield / Air Quality Egg are:

• The Air Quality Egg had a fan (which affects airflow and electrical load), the reference circuit did not
• The Air Quality Egg was in an enclosure, the reference circuit was not
• The Sensor Shield dynamically controlled the heater voltage, the reference circuit did not
• The Sensor Shield dynamically selected the voltage divider, the reference circuit did not

For the first two bullets I did a series of tests that ultimately revealed the enclosure didn’t have much of an impact, but that disconnecting the fan from the Air quality Egg circuit has a very noticeable impact. As a quick aside, all of the following graphs have elapsed time in milliseconds on the x-axis and ADC counts on the y-axis. The following graph compares the response of the reference circuit, to an Air Quality Egg with a fan and enclosure, and to an Air Quality Egg with neither a fan nor an enclosure.

The difference is readily apparent. The data with the fan (red) has a bout 50 ADC counts of noise on it. The data with no fan (blue) in the circuit, however, still had a peculiarity of these randomly appearing spike discontinuities in the data. I made modified builds of the Sensor Shield firmware that took out the Heater Control algorithm, took out the Range Selection algorithm, and took out both.  The result of taking out both looked like this (with and without an enclosure) in comparison to the reference circuit.

Lo and behold, with the fan removed and the software sophistication removed, the Air Quality Egg response has neither noise nor spikes, and it seems to track the reference circuit quite well. I then added just the Heater Control algorithm back in and I got this result:

This confirms, in my view, that the Heater Control algorithm is the source of spiky-ness in the Air Quality Egg data. I’m currently studying the software intensely to come up with an improved version of the Sensor Shield firmware, and will post another update when I know more.

“Air Quality Egg” is now a trademark

We have followed in the footsteps of Arduinoâ„¢ and received a trademark on the “Air Quality Egg”. This post explains why we did this, and it’s effects.

A quick note in case you are unfamiliar with Trademarks: This has no effect on the open source-ness of anything of the hardware and software. It is still 100% open source.

This blog post has a pretty good explanation why open source projects should trademark. A trademark prevents other people from calling something an “Air Quality Egg”. We did this for two reasons:
– It would be confusing to us if other people created a product with the same name.
– If might also confuse customers, and they might expect us to support all products called “Air Quality Egg”

Here is another good discussion on trademarks and open source.

Air Quality Egg status update

All,

It has been awhile. Here is an update on the Egg.

If you ordered:

• an egg with no add-ons (Just the basic egg),
• a shield only or
• a Kit,

Then we have shipped your order. If you don’t think we have, then please contact support AT wickeddevice.com

If you ordered an Egg with any add-ons (Ozone, VoC, etc), we have not shipped your egg yet, but will be doing so soon (starting next week).

If you ordered a radiation module, as noted in January, Joe Saveedra has been kind enough to design and help with delivering it.

Wicked Device announces strategic partnership with ITT Exelis

ITT Exelis (NYSE: XLS) and Wicked Device LLC have entered into a strategic partnership to explore innovative technology to provide real-time weather information at a hyper-local scale.

Wicked Device is developing an embedded system to integrate with Exelis’s Helios™ platform. The Helios platform enables traffic and other cameras to function as weather sensors, providing more accurate, real-time environmental intelligence for better decision-making and response.

The Helios platform integrates networks of surveillance cameras already in use to watch traffic, facility security and railroad assets for weather monitoring. It aggregates disparate images into a single source and uses algorithms and image processing to reveal actionable real-time environmental intelligence. Providing hyper-local weather data, versus regional data, to national and local agencies, utility companies, transportation services and first responders can help in their decision-making process, saving lives and resources.

AQE: Enhanced RGB User Feedback

Over the weekend I worked on making the color expressions of the Air Quality Egg more… well, expressive. The update is on Git-Hub so feel free to update your software (howto videos) for the Remote and Base units. This is the 1.03 version of both baselines (version number gets printed to the serial console).

As the Eggs have been reaching their destinations, feedback from users indicated that the setup process had some significant holes in diagnostic feedback. The most important of these was that the outcome of the pairing interval was invisible to the end user; the only way you knew that pairing succeeded was that the system started working a couple minutes later. If the system didn’t start working, you were left scratching your head and trying again. So I added an acknowledgement to the pairing behavior, and now there’s clear feedback about the outcome of the pairing interval: after the yellow pulses that signify pairing in progress, three magenta blinks indicate “pairing was not acknowledged” and three cyan (light blue) blinks indicate “pairing was acknowledged.”

The second feedback notable feedback confusion was related to Cosm provisioning. The previous behavior was to indicate solid green when provisioning succeeded, but *only the first time* after that you would never see green again. So now, if provisioning has previously succeeded, you get three green blinks.

In summary, the colors the Egg expresses now come in three “flavors”:

• Pulsing colors are used to indicate normal activity and progress (I know some people find this annoying, but I’m erring on the side of more feedback)
• Blinking colors are used to signify status
• Solid colors are used to indicate conditions that result in a restart
• Blinking/Solid green, blue, and cyan are used as “positive” indicators
• Blinking/Solid red and magenta are used as “negative” indicators

The following flowchart describes the feedback behavior of the 1.03 baseline.

AQE Bug: Ridiculously High Readings

Short Story

I’ve been reading reports of really high (some would say ‘ridiculous’) values being reported by Air Quality Eggs in recent weeks as they’ve started to reach their homes. I think I’ve figured it out, and I’ve pushed an update to the Remote Egg software (more specifically, the Egg Bus Arduino Library). Basically there was a bug in the ‘extrapolation’ case of the getSensorValue function. If you want to skip reading the rest and go patch your Egg with the fix, just go download the latest zip from Git-Hub and reload the software as described in the tutorial videos we recently produced <LINK HERE>.

Details

In EggBus.c in the getSensorValue function, the way it works is it searches a table of (x, y) samples from the sensor hosting device (i.e. the Egg Shield) that relates the “raw” value from the sensor to a computed value that is something we are more used to thinking about.  For example, the table might take us from measured resistance in ohms (the raw value of the NO2 sensor) to concentration in ppb (parts per billion). The way the Nanode uses this table is it look for the two rows in this table that the raw value falls between and performs a linear interpolation – this is the most common case we should expect to see. There are of course a few other ‘edge cases’ that need to be handled: namely (1) exact matches to a table row, (2) values that are smaller than the smallest value represented in the table, and (3) values hat are larger than the largest value in the table. Exact matches are trivial, as we just return the mapped value with no further maths applied. The other two cases are slightly less trivial, but the principle is to calculate the slope of the sampled curve at either extreme and then perform a linear extrapolation in the corresponding direction off the sampled curve.

The problem was in the handling of the third case above – an extrapolation above the highest value in the table. Notably:

float EggBus::getSensorValue(uint8_t sensorIndex){

  // some setup code omitted here
  while(getTableRow(sensorIndex, row++, &xval, &yval)){
    real_table_value_x = x_scaler * xval;
    real_table_value_y = y_scaler * yval;

    // bunch of code here for the other cases omitted here...

    // store into the previous values for use in interpolation/extrapolation
    previous_real_table_value_x = real_table_value_x;
    previous_real_table_value_y = real_table_value_y;
  }

  // case 4: the independent variable is must be greater than the largest value in the table, so we need to extrapolate forward
  // if you got through the entire table without an early return that means the independent_variable_value
  // the last values stored should be used for the slope calculation
  slope = (real_table_value_y - previous_real_table_value_y) / (real_table_value_x - previous_real_table_value_x);
  return real_table_value_y + slope * (independent_variable_value - real_table_value_x);
}

I removed all the irrelevant stuff. Do you see the problem? Basically at the slope calculation right before the return is *guaranteed* to result in the calculation 0 / 0 which turns out to be Not-a-Number (nan) in floating-point-speak. So in the case where the “measured value” turns out to be on the high side, things would sort of fly off the handle. There is still the ‘problem’ that the values are coming out high to begin with, but that’s because we don’t have a calibration procedure at this point, and is a topic for a different post. Thanks for reading!

AQE Bugs

The Short Story

There’s a bug with Cosm activation. If your Egg is not publishing data, here’s what you have to do, without getting into the details:

Option 1: Send us back your Air Quality Egg, along with prepaid return postage. We will re-program it for you.

Option 2: Update the software yourself. Steps:

1. Get an “FTDI cable.” You can get one from us ($22 or $13), or your local reseller (or if you have an Arduino laying around…)
2. Download the latest software from Git-Hub.
3. Reprogram your Eggs (we’ll post a video and some instructions to help in the next few days).

How to choose an option: We may discover more bugs in the future. Option 2 empowers you to reprogram your Eggs whenever additional bugs are found, or enhancements are published. If you are not comfortable doing that, choose Option 1.

The Longer Story

I have some unfortunate news. Provisioning with Cosm seems to have stopped working somewhere along the way. The good news is I’ve already got a software fix. The not-so-good news is that it’s pretty likely we’ve already shipped your Egg.

As I’ve described in other posts, when your Egg is first plugged into a network it reaches out to Cosm to declare its existence. In exchange for this declaration, (and by virtue of the fact that we’ve pre-registered your serial number) Cosm issues your Egg an API Key and Feed ID. The Nanode should extract these bits of information, remember them, and thereafter use that information to post information to the Cosm website under the prescribed Feed ID. This was all working, but something has changed. The software I wrote for the Base Egg wasn’t prepared for the change. I think the Cosm responses used to be fragmented into two or more packets and that is no longer the case. In a nutshell, Cosm is getting the message and issuing the activation data to the Egg, but the Egg is ignoring that response, and just keeps re-attempting activation.

The technical detail is in Provision.ino in the AQEBase sketch:

static void provisioningCallback (byte status, word off, word len) {

…

if (status == 1){ //we’re getting the body —

… all the parsing of the response happens here …

and  status is being returned as 0 from the EtherCard library so the response is not being parsed. The fix is stupidly easy: remove the guard and everything works as expected. Honestly, I can’t remember why I put the guard in there in the first place. It was probably cut-and-paste from an EtherCard example.

GREAT, SO NOW WHAT!?

The simple truth is that this will probably not be the last bug we find that needs to be patched. I wish that weren’t true and that I were a perfect programmer, but I’m not. We (Wicked Device) also can’t afford the shipping costs of offering to personally reprogram everyone’s Egg every time a bug is found and needs to be patched. Fortunately, the whole project is open source and I can push patches up to Git-Hub and announce when bugs are found to the Google Group and blog. I can also post instructions / videos showing you how to update your own units. The problem that remains is that you need a “special” cable to program the Egg units, commonly called an FTDI cable. In retrospect, I wish we had just included one with the Eggs, but we were already up against a wall with cost.

Here is a solution I think is reasonable: we will send you a suitable programming cable from our store (linked above), or  buy an FTDI cable (or equivalent) from the supplier of your choice. This will allow you to reprogram the Egg units anytime you need to. This cable will also be a necessary ingredient for performing sensor calibration in the future.

Please accept my apologies. We appreciate your understanding as we continue to do our best to support the Air Quality Egg community! This has certainly been a learning experience so far.

Mark Gibbs from Network World reviews the Air Quality Egg

On January 17th, Mark Gibbs from Network World has published a review of the Air Quality Egg. He made some great comments which we will be taking to heart (Better documentation, move the sticker off the box, both spot on). The review also had some comments we are quite proud of:

“And that’s it, the AQE system is a great example of plug n’ play done right.”

“… a huge thumbs up for delivering an ambitious solution to citizen-driven environmental monitoring.”