Our products, Nanode, wireless, MCUs, electronics and more

Air Quality Egg: Shield Firmware Update

Posted: April 22nd, 2013 | Author: | Filed under: AQE | Tags: , , | Comments Off

Based on my research findings (described in my last post), I spent some time digging into the Egg Shield hardware and firmware and I made some significant updates in hopes of improving results. I’m currently running an overnight experiment of the latest firmware, which I pushed out on GitHub (also for the Ozone and VOC add-ons). I also put load scripts out there to help ease the update process. I wanted to give an update on what I found and what I did.

Constant Heater Voltage

There was an observable random “spiking” phenomenon associated with the dynamic heater voltage control algorithm. I still don’t know the root cause of that, but I know that if I take that algorithm out, the associated phenomenon goes away, so the current version of the firmware sets the voltage at start-up and leaves it alone. I determined the settings for each heater experimentally by trying a setting, measuring H+ and H- and calculating the heater power, until I hit the datasheet targets.

Range Selection Algorithm / Fixed Point Math Rewrite

I was concerned about the frequent reports of apparent overflow conditions occurring. I looked into how to make better use of our range selection capability and rewrote it to select the range that yields results that are closest to the mid-range voltage. I believe this algorithm should provide more stable / continuous results than the original algorithm. In particular, I expect that it will be more immune to overflow conditions.

NO2 Hardware Bug – Workaround

In re-writing the Range Selection Algorithm, I was astonished to find an undiscovered hardware bug in the Egg Shield. I kept trying different settings on the range selection divider for NO2, but no matter what I did, two of the range settings always yielded the same analog-to-digital conversion result. I dug in and traced it back to the schematic (after hours of assuming I must be doing something wrong in software).

It turns out there is a node named “/NO2_R2_SEL” and a separate node named “/R2_SEL.” Those were meant to be the same node, but the tools don’t know that, so they never got connected in the layout. The effect is that it’s impossible to select a low-side resistance of R1 + R2 = 24.2 kΩ for NO2. <face-palm!> The firmware didn’t know about that bug either, so its model of the world didn’t agree with reality when it chose that range. This inconsistency was almost certainly also contributing to the instability in the NO2 readings we’ve been seeing. Anyway, the firmware agrees with the hardware now. I think it will be less accurate than it could have been were it not for the hardware bug, but I think this will be much better than what we had going on.

What To Do Next

I’ll follow up with a post tomorrow night with my overnight results. If you’re ambitious and know what you’re doing, the software and binaries for the firmware updates above is already out on GitHub and you can go ahead and re-flash your Egg Shield to hopefully start getting better results right away. As a reminder, Joe Saavedra also made a nice tutorial on how you can use an Arduino (or a Nanode) to reprogram a Sensor Shield. I plan to do a video to demonstrate that process as well in the next couple weeks. As always, if you’re uncomfortable reprogramming the hardware for yourself and  you are willing to pay for the round-trip shipping, I am willing to personally reprogram your hardware at no additional cost.


Air Quality Egg: Research Findings

Posted: April 16th, 2013 | Author: | Filed under: Uncategorized | 1 Comment »

I’ve been looking at a lot of the data being published and trends in questions on the forum, and there is a theme developing. The results are not consistent with expectations. We’re also now getting some reports from laboratory studies that are confirming there’s a problem. The collective information strongly suggests that the data the Air Quality Eggs are reporting is incongruent with the natural response of the sensors.

In light of this growing body of evidence, I built a much simplified reference circuit based on the sensor datasheets and recorded sensor readings along side the Egg for several hours. Sure enough, the reference circuit exhibited a fairly smooth / continuous response over time. The Egg did not. 

I decided the rational thing to do was to conduct a series of controlled experiments in an effort to establish the root cause of the variation between the Sensor Shield and the Reference Circuit behavior. This was a daunting proposition with so many potential variables in play, but I did my best to come up with some viable hypotheses. The fundamental differences between the reference circuit and the Sensor Shield / Air Quality Egg are:

  • The Air Quality Egg had a fan (which affects airflow and electrical load), the reference circuit did not
  • The Air Quality Egg was in an enclosure, the reference circuit was not
  • The Sensor Shield dynamically controlled the heater voltage, the reference circuit did not
  • The Sensor Shield dynamically selected the voltage divider, the reference circuit did not

For the first two bullets I did a series of tests that ultimately revealed the enclosure didn’t have much of an impact, but that disconnecting the fan from the Air quality Egg circuit has a very noticeable impact. As a quick aside, all of the following graphs have elapsed time in milliseconds on the x-axis and ADC counts on the y-axis. The following graph compares the response of the reference circuit, to an Air Quality Egg with a fan and enclosure, and to an Air Quality Egg with neither a fan nor an enclosure.

Exp00002_no2_compare

The difference is readily apparent. The data with the fan (red) has a bout 50 ADC counts of noise on it. The data with no fan (blue) in the circuit, however, still had a peculiarity of these randomly appearing spike discontinuities in the data. I made modified builds of the Sensor Shield firmware that took out the Heater Control algorithm, took out the Range Selection algorithm, and took out both.  The result of taking out both looked like this (with and without an enclosure) in comparison to the reference circuit.

Exp00002e_no2_comparison

Lo and behold, with the fan removed and the software sophistication removed, the Air Quality Egg response has neither noise nor spikes, and it seems to track the reference circuit quite well. I then added just the Heater Control algorithm back in and I got this result:

Exp00002f_no2_comparison

This confirms, in my view, that the Heater Control algorithm is the source of spiky-ness in the Air Quality Egg data. I’m currently studying the software intensely to come up with an improved version of the Sensor Shield firmware, and will post another update when I know more.


“Air Quality Egg” is now a trademark

Posted: March 15th, 2013 | Author: | Filed under: Uncategorized | Comments Off

We have followed in the footsteps of Arduino™ and received a trademark on the “Air Quality Egg”. This post explains why we did this, and it’s effects.

A quick note in case you are unfamiliar with Trademarks: This has no effect on the open source-ness of anything of the hardware and software. It is still 100% open source.

This blog post has a pretty good explanation why open source projects should trademark. A trademark prevents other people from calling something an “Air Quality Egg”. We did this for two reasons:
- It would be confusing to us if other people created a product with the same name.
- If might also confuse customers, and they might expect us to support all products called “Air Quality Egg”

Here is another good discussion on trademarks and open source.


Air Quality Egg status update

Posted: February 14th, 2013 | Author: | Filed under: Uncategorized | Comments Off

All,

It has been awhile. Here is an update on the Egg.

If you ordered:

  • an egg with no add-ons (Just the basic egg),
  • a shield only or
  • a Kit,

Then we have shipped your order. If you don’t think we have, then please contact support AT wickeddevice.com

If you ordered an Egg with any add-ons (Ozone, VoC, etc), we have not shipped your egg yet, but will be doing so soon (starting next week).

If you ordered a radiation module, as noted in January, Joe Saveedra has been kind enough to design and help with delivering it.


Wicked Device announces strategic partnership with ITT Exelis

Posted: February 7th, 2013 | Author: | Filed under: Uncategorized | Comments Off

ITT Exelis (NYSE: XLS) and Wicked Device LLC have entered into a strategic partnership to explore innovative technology to provide real-time weather information at a hyper-local scale.

Wicked Device is developing an embedded system to integrate with Exelis’s Helios™ platform. The Helios platform enables traffic and other cameras to function as weather sensors, providing more accurate, real-time environmental intelligence for better decision-making and response.

The Helios platform integrates networks of surveillance cameras already in use to watch traffic, facility security and railroad assets for weather monitoring. It aggregates disparate images into a single source and uses algorithms and image processing to reveal actionable real-time environmental intelligence. Providing hyper-local weather data, versus regional data, to national and local agencies, utility companies, transportation services and first responders can help in their decision-making process, saving lives and resources.


AQE: Enhanced RGB User Feedback

Posted: February 4th, 2013 | Author: | Filed under: AQE, Software | Tags: , , | Comments Off

Over the weekend I worked on making the color expressions of the Air Quality Egg more… well, expressive. The update is on Git-Hub so feel free to update your software (howto videos) for the Remote and Base units. This is the 1.03 version of both baselines (version number gets printed to the serial console).

As the Eggs have been reaching their destinations, feedback from users indicated that the setup process had some significant holes in diagnostic feedback. The most important of these was that the outcome of the pairing interval was invisible to the end user; the only way you knew that pairing succeeded was that the system started working a couple minutes later. If the system didn’t start working, you were left scratching your head and trying again. So I added an acknowledgement to the pairing behavior, and now there’s clear feedback about the outcome of the pairing interval: after the yellow pulses that signify pairing in progress, three magenta blinks indicate “pairing was not acknowledged” and three cyan (light blue) blinks indicate “pairing was acknowledged.”

The second feedback notable feedback confusion was related to Cosm provisioning. The previous behavior was to indicate solid green when provisioning succeeded, but *only the first time* after that you would never see green again. So now, if provisioning has previously succeeded, you get three green blinks.

In summary, the colors the Egg expresses now come in three “flavors”:

  • Pulsing colors are used to indicate normal activity and progress (I know some people find this annoying, but I’m erring on the side of more feedback)
  • Blinking colors are used to signify status
  • Solid colors are used to indicate conditions that result in a restart
  • Blinking/Solid green, blue, and cyan are used as “positive” indicators
  • Blinking/Solid red and magenta are used as “negative” indicators

The following flowchart describes the feedback behavior of the 1.03 baseline.

Egg Color Patterns


AQE Bug: Ridiculously High Readings

Posted: February 4th, 2013 | Author: | Filed under: Uncategorized | Comments Off

Short Story

I’ve been reading reports of really high (some would say ‘ridiculous’) values being reported by Air Quality Eggs in recent weeks as they’ve started to reach their homes. I think I’ve figured it out, and I’ve pushed an update to the Remote Egg software (more specifically, the Egg Bus Arduino Library). Basically there was a bug in the ‘extrapolation’ case of the getSensorValue function. If you want to skip reading the rest and go patch your Egg with the fix, just go download the latest zip from Git-Hub and reload the software as described in the tutorial videos we recently produced <LINK HERE>.

Details

In EggBus.c in the getSensorValue function, the way it works is it searches a table of (x, y) samples from the sensor hosting device (i.e. the Egg Shield) that relates the “raw” value from the sensor to a computed value that is something we are more used to thinking about.  For example, the table might take us from measured resistance in ohms (the raw value of the NO2 sensor) to concentration in ppb (parts per billion). The way the Nanode uses this table is it look for the two rows in this table that the raw value falls between and performs a linear interpolation – this is the most common case we should expect to see. There are of course a few other ‘edge cases’ that need to be handled: namely (1) exact matches to a table row, (2) values that are smaller than the smallest value represented in the table, and (3) values hat are larger than the largest value in the table. Exact matches are trivial, as we just return the mapped value with no further maths applied. The other two cases are slightly less trivial, but the principle is to calculate the slope of the sampled curve at either extreme and then perform a linear extrapolation in the corresponding direction off the sampled curve.

The problem was in the handling of the third case above – an extrapolation above the highest value in the table. Notably:

float EggBus::getSensorValue(uint8_t sensorIndex){

  // some setup code omitted here
  while(getTableRow(sensorIndex, row++, &xval, &yval)){
    real_table_value_x = x_scaler * xval;
    real_table_value_y = y_scaler * yval;

    // bunch of code here for the other cases omitted here...

    // store into the previous values for use in interpolation/extrapolation
    previous_real_table_value_x = real_table_value_x;
    previous_real_table_value_y = real_table_value_y;
  }

  // case 4: the independent variable is must be greater than the largest value in the table, so we need to extrapolate forward
  // if you got through the entire table without an early return that means the independent_variable_value
  // the last values stored should be used for the slope calculation
  slope = (real_table_value_y - previous_real_table_value_y) / (real_table_value_x - previous_real_table_value_x);
  return real_table_value_y + slope * (independent_variable_value - real_table_value_x);
}

I removed all the irrelevant stuff. Do you see the problem? Basically at the slope calculation right before the return is *guaranteed* to result in the calculation 0 / 0 which turns out to be Not-a-Number (nan) in floating-point-speak. So in the case where the “measured value” turns out to be on the high side, things would sort of fly off the handle. There is still the ‘problem’ that the values are coming out high to begin with, but that’s because we don’t have a calibration procedure at this point, and is a topic for a different post. Thanks for reading!


AQE Bugs

Posted: January 26th, 2013 | Author: | Filed under: AQE, Repair, Software | Comments Off

The Short Story

There’s a bug with Cosm activation. If your Egg is not publishing data, here’s what you have to do, without getting into the details:

Option 1: Send us back your Air Quality Egg, along with prepaid return postage. We will re-program it for you.

Option 2: Update the software yourself. Steps:

  1. Get an “FTDI cable.” You can get one from us ($22 or $13), or your local reseller (or if you have an Arduino laying around…)
  2. Download the latest software from Git-Hub.
  3. Reprogram your Eggs (we’ll post a video and some instructions to help in the next few days).

How to choose an option: We may discover more bugs in the future. Option 2 empowers you to reprogram your Eggs whenever additional bugs are found, or enhancements are published. If you are not comfortable doing that, choose Option 1.

The Longer Story

I have some unfortunate news. Provisioning with Cosm seems to have stopped working somewhere along the way. The good news is I’ve already got a software fix. The not-so-good news is that it’s pretty likely we’ve already shipped your Egg.

As I’ve described in other posts, when your Egg is first plugged into a network it reaches out to Cosm to declare its existence. In exchange for this declaration, (and by virtue of the fact that we’ve pre-registered your serial number) Cosm issues your Egg an API Key and Feed ID. The Nanode should extract these bits of information, remember them, and thereafter use that information to post information to the Cosm website under the prescribed Feed ID. This was all working, but something has changed. The software I wrote for the Base Egg wasn’t prepared for the change. I think the Cosm responses used to be fragmented into two or more packets and that is no longer the case. In a nutshell, Cosm is getting the message and issuing the activation data to the Egg, but the Egg is ignoring that response, and just keeps re-attempting activation.

The technical detail is in Provision.ino in the AQEBase sketch:

static void provisioningCallback (byte status, word off, word len) {

if (status == 1){ //we’re getting the body —

… all the parsing of the response happens here …

and  status is being returned as 0 from the EtherCard library so the response is not being parsed. The fix is stupidly easy: remove the guard and everything works as expected. Honestly, I can’t remember why I put the guard in there in the first place. It was probably cut-and-paste from an EtherCard example.

GREAT, SO NOW WHAT!?

The simple truth is that this will probably not be the last bug we find that needs to be patched. I wish that weren’t true and that I were a perfect programmer, but I’m not. We (Wicked Device) also can’t afford the shipping costs of offering to personally reprogram everyone’s Egg every time a bug is found and needs to be patched. Fortunately, the whole project is open source and I can push patches up to Git-Hub and announce when bugs are found to the Google Group and blog. I can also post instructions / videos showing you how to update your own units. The problem that remains is that you need a “special” cable to program the Egg units, commonly called an FTDI cable. In retrospect, I wish we had just included one with the Eggs, but we were already up against a wall with cost.

Here is a solution I think is reasonable: we will send you a suitable programming cable from our store (linked above), or  buy an FTDI cable (or equivalent) from the supplier of your choice. This will allow you to reprogram the Egg units anytime you need to. This cable will also be a necessary ingredient for performing sensor calibration in the future.

Please accept my apologies. We appreciate your understanding as we continue to do our best to support the Air Quality Egg community! This has certainly been a learning experience so far.


Mark Gibbs from Network World reviews the Air Quality Egg

Posted: January 21st, 2013 | Author: | Filed under: AQE | Comments Off

NetworkWorldOn January 17th, Mark Gibbs from Network World has published a review of the Air Quality Egg. He made some great comments which we will be taking to heart (Better documentation, move the sticker off the box, both spot on). The review also had some comments we are quite proud of:

“And that’s it, the AQE system is a great example of plug n’ play done right.”

“… a huge thumbs up for delivering an ambitious solution to citizen-driven environmental monitoring.”


Air Quality Egg: The Making of a Sensor Data Point

Posted: January 15th, 2013 | Author: | Filed under: Uncategorized | Comments Off

I have been wanting to write this blog post for a while. I hope this article will give some insight to the Air Quality Egg community about how the system works end-to-end. I want to provide details about how the Air Quality Egg software and hardware work hand in hand to provide an extensible platform for getting data from the a transducer (sensor) to the internet. I’ll try and structure this in into sections are data-centric, framing it in the context of the “lifespan” of a sensor reading. I chose to write it with a biologically inspired vantage point, hopefully it’s not too lame.

Inception

When you turn on your Base Egg, it broadcasts its MAC address (unique ID) for a period of about 30 seconds (the pairing interval) while illuminating the shell with a yellow pulse. During this time, the Base Egg is offering itself for pairing in order to establish a data link with a Remote Egg. A Remote Egg, in turn, listens patiently for Base Egg advertisement broadcasts for about a 15 second period after power up. A Remote Egg hearing a Base Egg stores away that Base Egg address in persistent memory (EEPROM). Base Eggs are not “monogamous” and willingly accept data from any Remote Egg that is within range and has learned the Base Egg’s advertised address during the pairing interval. Neither do Remote Eggs “mate for life.” They can always be paired to a new Base through the same procedure. They are nevertheless “loyal” in that if they reset, hear no new Base Egg advertisements, and have previously paired with a Base Egg, they will resume communicating with that previously paired Base Egg.

Nurturing

Every sensor hosted by the system has a small microcontroller attached to it. The job of this microcontroller is three-fold. It:

  1. Provides the interface to the Egg Bus.
  2. Performs any tasks necessary to keep the sensor happy and safe, for example regulating the heater power for a CMOS ozone sensor.
  3. Is responsible for actually reading the sensor at the lowest level, be it through analog-to-digital conversion like the CMOS gas sensors, or a time-based digital occupancy measurement like the dust sensor.

This federated sensor management architecture separates the concern of how each sensor is dealt with into bite-sized, compartmentalized fragments, and allows the system to scale well locally. A major upside to the standardized interactions is that it allows you to plug in new sensors into the system without needing to change the software running on your Air Quality Egg. Another big upside is that new sensors can be developed by anyone because of the openness of the interface, software, and hardware.

Discovery

In the Remote Egg, Sensors are attached to the Egg Shield through the Egg Bus. The Remote Egg, at one minute intervals, searches the Egg Bus and finds attached sensors. Upon encountering a sensor, it interrogates it for a variety of information. Using the information gleaned by the Egg Bus, the Remote Egg calculates both a “raw” and “computed” sensor reading.

In a similar theme of discovery, the first time you plug your Base Egg into your network it reaches out to Cosm over the internet and attempt to negotiate the creation a feed where it will ultimately store your Egg’s data. This mechanism is enabled by the fact that each and every Air Quality Egg is labeled with a Unique ID that is also bound to the physical hardware, sort of like your Egg’s fingerprint! When you enter your Egg Serial Number into airqualityegg.com and fill out the form, this process is enabled.

Migration

Having discovered a sensor, and having calculated the raw and computed data points, the Remote Egg ships it wirelessly to its paired Base Egg in a Sensor Packet. The Base Egg, upon receiving, a Sensor Packet addressed to it, illuminates its shell a color (chosen from a predefined sequence) to give you, the proud parent of your Air Quality Egg, feedback that your Remote and Base Eggs are getting along.

Metamorphosis

The Base Egg then restructures the information it receives wirelessly into a format suitable for Cosm’s API. The Base Egg also decorates the datapoint with various tags to help applications search, filter, and group the data more efficiently. The Base Egg transmits the datapoint over the internet where it can live out the rest of its days as part of the world’s first open public Air Quality data set, and doing its part to give us a better understanding of air quality across time and space.

 Evolution

As the data set becomes larger and more dense, and as our understanding of the data grows, the hope is that our community will develop algorithms and procedures to improve the quality of the data through further post-processing. Techniques will also be hopefully be developed to help ‘calibrate’ your Egg’s sensors in a low cost, DIY way. As new sensors become available modules will be developed that will seamlessly plug into your existing Egg through the aforementioned Egg Bus connection, giving us an ever richer set of correlated data.