Adding the Nokia 5110 LCD to your Arduino Data Logger

Here I’ve added a the 5110 LCD to a logger recording data from a BME280 & Tipping Bucket Rain gauge. If the BME  survives in our field environment, this will become a standard configuration for our climate stations. I’m not holding my breath though, as we’ve tested half a dozen RH sensors so far and none of them have gone the distance in high humidity environments that occasionally go condensing.

This year I want to tackle some projects that need live data out, so I’ve been sifting through the many display options available for Arduino. Unlike flashier projects, my goal was to find one that I could add to existing logger builds without sacrificing too much of the multi-year lifespan I had worked so hard to achieve. The low power winner by a fair margin was the Nokia 5110 which you can pick up for around $2 from the usual sources. With the back-light off these displays pull between 100-400 μA, depending on the number of pixels turned on.

This screen uses a PCD8544 controller and the SPI protocol.  It will tolerate 5V, but it works best at 3.3V, which is perfect when you are driving it from an 8mhz ProMini. Each pixel on the display is represented by a single bit in the PCD8544’s RAM. Each byte in RAM correlates to a vertical column of 8 pixels. The X coordinate works on a per-pixel basis, and accepts values between 0 and 83. The Y coordinate accepts values of 0 – 5 which on this 48 pixel high screen, corresponds to 6 “rows of bytes” in the controller’s RAM. So bitmaps can only be displayed on a per row (& column) basis. The display is quite sluggish compared to competitors like the 0.96 I2C monochrome OLED and you have to handle any processing overhead on the Arduino.

Most hookup guides assume that you can spare six control lines to run the display, which is not the case when your logger already has three indicator LEDs, I2c devices, one wire sensors and a couple of voltage dividers on the go.  However if you are willing to add a few resistors and occasionally toggle the power, you can bring that down to three wires and a power pin.

So many libraries, so little optimization…

This screen’s been around for a very long time, so there’s are a huge number of easy to use, highly functional libraries for Arduino. But they tend to focus on things like speed or endless font options which are not important for most data logging applications. And these libs assume your project can afford to lose up to ⅓ of the available program & variable memory just driving the display. Most also require the hardware SPI lines, but our project needs those for SD cards, which are finicky enough without some pokey LCD gumming up the works: the 5110 maxes out at 4mbps, and this slows the bus significantly .

Those fat libs were non-starters for our project, and I had almost given up on this display when I found Ilett’s Ardutorial offering a bare-bones method more suitable for our resource limited data loggers. If you haven’t discovered Julians YouTube channel yet then you are in for a treat because if Andreas Spiess is the maker worlds answer to Werner Herzog, then Julian is surely their equivalent to Bob Ross.  I don’t know if he’s growing “Happy little trees” with his DIY hydroponics, but I can say that the gentle timbre of his “Gooood morning all” reduces stress faster than a warm cup of Tea.  And his “Arduino sandwiches” are brilliant examples of minimalist build technique.

Driving the Nokia 5110 with shiftout

Everything I’m presenting here builds on his tutorials, so grab a mug and give ’em a watch:

Tutorial #1 – Connecting and Initial Programming
Tutorial #2 – Getting Text on the Display
Tutorial #3 – Live Numerical Data

This software SPI method (originally from arduino.cc?) requires no library at all, and shiftout commands work with any combination of digital pins; saving those hardware SPI lines for more important jobs.

Initial setup is explained in video #1 using two functions

void LcdInit(void)
{
digitalWrite(RST, LOW);            // not needed with pin powering!
digitalWrite(RST, HIGH);           // see below for details
LcdWriteCmd(0x21);                 // extended commands 
LcdWriteCmd(0xB8);                 // set Vop(contrast) // you may need to tweak
LcdWriteCmd(0x04);                 // set temp coefficient 
LcdWriteCmd(0x14);                 // bias mode 1:40 // you may need to tweak this
LcdWriteCmd(0x20);                 // basic commands 
LcdWriteCmd(0x0C);                 // normal video
for(int i=0; i<504; i++) LcdWriteData(0x00);  // clear the sceen
} 
void LcdWriteCmd(byte cmd)
{
digitalWrite(DCmodeSelect, LOW);    // low for commands, high for data 
digitalWrite(ChipEnable, LOW);      // not need with pin-power
shiftOut(DataIN, SerialCLK, MSBFIRST, cmd);  // transmit serial data 
digitalWrite(ChipEnable, HIGH);     // not need with pin-power
} 

After that you need is a function to position the cursor and a font stored in a byte array (in this example called ASCII[][5])

void LcdXY(int x, int y)
{
LcdWriteCmd(0x80 | x);              // Column
LcdWriteCmd(0x40 | y);              // Row  
} 

Then three short cascading functions let you send a string of ascii characters to the display:

void LcdWriteString(char *characters)
{
while(*characters) LcdWriteCharacter(*characters++);
} 
void LcdWriteCharacter(char character)
{
for(int i=0; i<5; i++){
LcdWriteData(pgm_read_byte(&ASCII[character - 0x20][i])); 
}
LcdWriteData(0x00);            //one row of spacer pixels between characters
} 
void LcdWriteData(byte dat)
{
digitalWrite(DCmodeSelect, HIGH);    // High for data 
digitalWrite(ChipEnable, LOW);  
shiftOut(DataIN, SerialCLK, MSBFIRST, dat);  // transmit serial data 
digitalWrite(ChipEnable, HIGH);
} 

Julians original implementation includes a font.h file posted on hastebin, but I’ve rolled that 500 byte font array into some code based on his work and posted it to the Cave Pearl Project’s Github .  You will find lots of other examples based on the shiftout method on Github, but for some reason people insisted in retooling that tiny bit of code into, you guessed it, even more libraries

You’ll also find plenty of other drop-in font definitions with Google, but for small 5×7’s, it doesn’t take that long to roll your own by clicking the boxes in an online font creator. and then copying the byte pattern into a bin-hex converter. This also gives you the option of creating custom icons by using a non-standard bitmap for some of the less frequently used ascii characters. Keep in mind that you don’t need to store the entire alphabet if you are only sending a few letters to the screen (like ‘T+P’ or ‘RH%’, etc …) extracting only the letters you need to a reduced font array could save a lot of memory.

So your reduced font array could look something like this:

const byte ASCII[][5] =
{
{0x7f, 0x09, 0x19, 0x29, 0x46}  // 52 R
,{0x7f, 0x08, 0x08, 0x08, 0x7f} // 48 H
,{0x23, 0x13, 0x08, 0x64, 0x62} // 25 %
,{0x01, 0x01, 0x7f, 0x01, 0x01} // 54 T
,{0x08, 0x08, 0x3e, 0x08, 0x08} // 2b +
,{0x7f, 0x09, 0x09, 0x09, 0x06} // 50 P 
};  

If you do that you’ll need to send text to the screen character-by-character because the ASCII based [character – 0x20] calculation won’t work any more.

I tweaked Julians code in a couple of important ways. First, I added PROGMEM to move the font(s) into the program memory space. Second, I added a method to print large numbers to the screen by repeating the same WriteString->WriteCharacter->WriteData pattern two times: once for the “upper half” of the numbers, and then again for the “lower half” of the numbers after re-positioning the cursor to the next line.

To make this limited large-number font I first composed a black & white bitmap for each number with a graphic editor, and then loaded that .bmp file into the LCD assistant program as described in this instructables tutorial.  I started with a bitmap that was 11 pixels wide, by 16 pixels high (though you can use any arbitrary size you want – just remember to leave the blank spacer row at the bottom) and for this two-pass ‘sliced-letters’ method I set vertical & little endian encoding in LCD assistant. I then put the top 11 bytes in the Big11x16numberTops[] array & the lower 11 bytes for each number in the Big11x16numberBottoms[] array.

It takes two passes to print each large number to the screen:

LcdXY (0,2);              //top half of the double-size numbers
LcdWriteBigStringTops(dtostrf(voltage,5,2,string));  
LcdXY (0,3);              //bottom half of the numbers 
LcdWriteBigStringBottoms(dtostrf(voltage,5,2,string));

And some slightly modified functions that refer to the corresponding number-font array:

void LcdWriteBigStringTops(char *characters)
{
while(*characters) LcdWriteBigCharacterTops(*characters++); 
} 
void LcdLcdWriteBigCharacterTops(char character)
{
for(int i=0; i<11; i++){
if((character - 0x2d)>=0){
LcdWriteData(pgm_read_byte(&Big11x16numberTops[character - 0x2d][i]));
}
}
LcdWriteData(0x00);  //one row of spacer pixels inserted between characters  
} 

Those number printing functions could be eliminated with better use of pointers, but I liked having the readability afforded by a few extra lines of code.

Reducing the number of control lines

The stuff I posted on Github assumes you are using a standard 6-pin arrangement shown in most Nokia 5110 hookup guides you will find on the web. But once I had that wrangled, I realized that it would be possible to reduce the number pins needed to drive the display.  You will have to tweak that default example by commenting out the RS & CS commands if you implement the pin-power changes I’m suggesting here… 

I use Deans micro-plugs  for multi-wire applications like this. The unused pin here is the normal Vcc, with the A0 pin-power supply indicated with a dash on the red line.

The shiftout method can be used with any pins you want, and most of my builds have A0-A3 available. With those dedicated wires the PCD8544’s chip select (CS) line can be connected to GND telling the screen that it’s always the selected device. This would be bad if it was connected to the hardware SPI lines shared with the SD card, but since we are using re-purposed anlaog lines, there is no conflict. One minor drawback is that since we are now using all the analog lines (I use A6 & A7 too) we can’t read a floating pin for RandomSeed().

Getting rid of the RESET line is a little trickier. The data sheet says that the RS line must be low while power stabilizes and should then be pulled high within 100ms of power on. Several people create an auto-reset situation by connecting the screens reset to the Arduino’s reset line. Others make the low-high transition with an RC network across the supply for a delayed rising signal. This can even be driven by the DC line (which is low in command mode and high in data mode) 

But I had something else in mind, since I wanted to power the entire display from a digital pin because the power draw with the display off is still about 70uA. This accumulates into a significant amount of wasted power over a multi-year deployment.

Reducing Back-light Current

OR … a photocell divider embedded in that clear epoxy would let you enable the backlight dynamically – with the appropriate mosfet for the connection type.

If you use the back-light in the default configuration, the screen can potentially draw up to 80mA (4 white LEDs at 20mA each). The back-light pin is usually connected to a transistor, so you can PWM all 4 LEDs at once for variable lighting control, but the peak currents are still too high for direct pin-powering unless you add some kind of series resistor.  A 10k pot gives you a simpler method to adjust the screen brightness, but I found that a 3k3 series resistor brought the total display current down to ~1mA with decent readability ( & blue LEDs are brighter than white).  Adding an in-line slide switch provides a way to completely disable the back-light for long deployments.  With the entire display safely below Arduino’s pin-current limit, you can then power it by writing a driver pin high or low in output mode.

#define n5110PowerPin  A0          // power the 5110 screen from pin A0 (RED)
#define n5110modeSelect A1         // 6.1.9 D/C: mode select (BLUE)
#define n5110SData A2              // 6.1.7 SDIN: serial data line (WHITE) 
#define n5110SCLK A3 ;             // 6.1.8 SCLK: serial clock line (YELLOW)
// lines not needed any more: 
// #define n5110RST  Now -> 4.7k to power A0
// #define n5110ChipEnable  Now -> GND

This gives you a way to perform a hard reset any time you want provided you tie the screens RS line to that switched power with a 4k7 pullup resistor, and re-run the initialization sequence after restoring power.

Boards with pin vias on both sides make it easier to add the RST & CE connections. The orange wires shown here thread through the housing to a slide switch which disables the backlight connection for surface deployments. This example connects power to the backlight, but with other screens you might have to connect BACKLIGHT to GND.

Enabling the screen now looks like this in setup:

pinMode(n5110PowerPin, OUTPUT);
digitalWrite(n5110PowerPin, HIGH);
pinMode(n5110modeSelect, OUTPUT);
pinMode(n5110SData, OUTPUT);
pinMode(n5110SCLK, OUTPUT); 
LcdInit(); // shiftout takes control of Mode, Data & SCLK lines at this point

To turn off the screen you pull all the control lines low:

digitalWrite(n5110PowerPin, LOW);        
digitalWrite(n5110modeSelect, LOW);                
digitalWrite(n5110SData, LOW);                
digitalWrite(n5110SCLK, LOW);                         

All four control lines must be brought low when you de-power the display or you will get a 13mA leak current through the controller after vcc goes low. Only the power pin needs to be driven high to start the screen later in the main loop, but don’t forget to run the init each time you power up.

My tests so far have shown reliable operation of pin-powered 5110’s through more than 8000 ‘long-sleep’ power cycles. In applications where I want to display data on the screen on for long periods of time,  I still depower the screen during the new sensor readings. This lets me know when the logger is capturing data and forces a periodic re-synch with the bus. I don’t know how long these displays would run continuously without that step, but I’m sure the coms would eventually go AWOL without some kind of regular reset.

Potting the Nokia 5110 display

Contraction of the epoxy created pressure burns on the LCD when I did a single large pour to pot the screen.

No screen is much use on our project unless it can withstand some bumping around in the real world, and ideally we want one that is dive-able. For several years my go-to solution has been to pot surface mounted LED’s and sensors in Loctite E30CL. I like this epoxy because the slow cure usually sets clear because bubbles have time to rise to the surface without a vacuum treatment. My first attempts looked great the night of the pour, but I got a nasty surprise the following morning. You see I usually mount sensors in small ½-1 inch wells, but the 5110 required a ring more than 2” in diameter. The contraction of the epoxy in this 10mm deep well caused pressure marks on the edges of the screen, and a significant brown spot in the center of the display where the text became inverted.

Successive small pours worked better. Here the back-light reflects off of the edges of the epoxy that seeped under the screen before it finished setting. The display in this photo has a 3k3 series resistor in the backlight circuit.

The next attempt was much more successful, as I built up the epoxy a few mm at a time like the layers of an onion. As each layer hardened, it protected the screen from the contraction of the subsequent layers above.  The trick was to bring the first pour to the base of the pcb, and the second pour to “just barely” cover the surface of the screen. The epoxy penetrates about 1/3 of the way into the display housing but this does not interfere with readability as those edges are invisible under natural lighting conditions. That epoxy is actually under the LCD, in the air gap between the transparent glass LCD sandwich and the white reflector plastic which holds the thin LCD in place between the metal rim and the PCB.  I’ll try future pours at different angles to see if that lets the space under the LCD fill completely. Looking at the epoxy penetration, it’s clear that the black edges in pour #1 were places where the LCD was compressed on both sides, and the brown discoloration was from pressure on top with no support below. 

Seawater caused severe fogging of the potting epoxy after only three days in service. Originally a concession to my aging eyes, the large fonts really saved our bacon when this reaction occurred.

The results for the second batch looked good and the screens worked beautifully with full marine submersion for about two days. Then some kind of chemical reaction with the sea-water started fogging the epoxy, and by day three I was glad I’d created the large number fonts because the 5×7’s were completely unreadable.   Once we were back home, a bit of elbow grease & 800 grit removed the foggy surface rind, and a layer of conformal coating restored clarity. I think my next builds will add the coating to the epoxy surface at the start.

I also noted some screen discoloration from pressure at about 3m depth, indicating that even a thick layer of epoxy bows too much for a deeper deployment. I’ve ordered some 1/4“ plexiglass disks to provide a surface with a bit more chemical resistance, and will post an update on how that works after the next fieldwork trip. I’m hoping that provides a bit more pressure protection too, but the shore hardness of the epoxy is 85, and PMMA (plexiglass) is only a few steps above that at 90. I might try polycarbonate as well.

Other Fun stuff:

There is so much more to explore with this screen, including live graphing libraries, and display controls so I expect it will keep me amused for a while since I can add it to any of the current logger builds. Several are out in the wild now for long term tests, and I’m currently working on a script to move those fonts (and a few other things) into the 328p’s internal eeprom. If all goes well I’ll release that ultra low memory footprint version of the code shortly. 

Cheers for now.

Measuring Temperature with Two Clocks

I was pretty pleased about coming up with a new dithering method that lets you over-sample thermistors to better than 0.002°C with an Arduino.  I was bragging about this to a programmer friend recently, and suggested that achieving this with nothing more than a resistor on a pin might annoy engineers who’ve used more complicated circuits for essentially the same job.  His response to this surprised me:

“Actually, the ones you’re really going to piss off are the coders. It seems to me that your pin-toggle technique only works because the rail stabilization on those chips is bad. In my business, that’s known as ‘Coding to a Fault’.  You see, I spend my days trying to keep large systems on their feet, and these are a patchwork of woolly legacy code and new functions tied together by utilities passing data back and forth. Errors creep in, and that’s expected, because those translators necessarily have to change the format of the data. Often the coder who did the tweening knows about these issues, and documents them, but their manager won’t authorize time for a proper fix once the function is stable because it’s judged to be a harmless, non-lethal error, and the budget’s too tight.  Ten years later, it lands on my plate because some diligent person finally decided to take that issue off the bug-list, usually because it was getting on their nerves that the person before them had done such shoddy work. Of course by then the error’s been present in the system so long that other working groups have unwittingly written code that relied on the artifact.  So fixing the original problem causes a heap of trouble five steps down the line.  Those situations are the bane of my life…”

This 8 MHz 3.3V clone works well with the two-clock method. I think of these as simply a cheap breakout board for the 328P.

He had more to say on the matter, but I missed the rest because I was backing away from his increasingly animated gesticulations, which were punctuated by some weird facial ticks.  I paused for a moment at the front door, thinking I should at least thank him for the coffee, when “…in fact, I should probably take you out of the technological gene pool right now, before you spread this madness…” convinced me otherwise.  I finally understood why the engineers were so critical about this project, since the way I build our loggers ignores several of rules that they’ve dedicated their career to.  If I had any professional decency, I’d seek absolution by dumping all that eBay crap in the bin, and abstain from any future component purchases that lead me away from the righteous ±1% path.

A divine revelation from Rantwijk’s site.

But I have to confess – the devil is still finding work for these idle hands.  Especially after reading Joris van Rantwijk’s analysis of Arduino clock frequency, when I realized the resonators on cheap Pro Mini clones probably had so much thermal variation, that you could base a temperature sensing method on that alone. The key insight here is that the $1 DS3231SN boards I’m getting from eBay are the genuine article with ±2ppm stability. By using both of those components in the same device, I had a high accuracy angel on one shoulder and a high resolution daemon on the other.

DS3231’s can be set to output a 1Hz pulse on SQW simply by zeroing the  control register:

Wire.beginTransmission(104); //DS3231 RTC address
Wire.write(0x0e);            //control register
Wire.write(0);               //zeroing all bits outputs 1 Hz on SQW
Wire.endTransmission(); 

To build the temperature sensor, all I had to do was compare that thermally stable 1-Hz pulse to the corresponding number of system clock ticks from the rubbish oscillator on the Pro Mini clone. A dive into the programming forum at Arduino.cc led me once again to Nick Gammon’s site, where his page on Timers and Counters  provided several ways to measure the frequency of pulsed inputs.  This isn’t the first time I’ve spent a few days learning from Nick’s examples, and I own him a huge debt of gratitude for his insightful work.

I was already familiar with the hardware interrupts on D2 & D3, but the 328P’s input capture unit on D8 does the job with a special circuit that saves the timer values in another register. You then read back that frozen value, and it doesn’t matter how long after the interrupt executes, or what you do in that interrupt, the captured time is still the same. This lets you measure duration to a temporal resolution approaching two ticks of your system clock. (see reply #12 on Nick’s page)

Will the regulators temp-coefficient affect the oscillator?

Oscillators are sensitive to input voltage as well as temperature, but Rantwijk also looked into that, determining that the clock frequency on his 16 MHz Pro Mini varied by about +2.5 Hz / mV input.  The MIC5205 LDO on the Mini & most of the 3.3V clones quotes an output Temperature Coefficient [∆VO/∆T] of 40 ppm/°C. If I restrict my sensing range from 0-40°C, then my regulator’s voltage output would be expected to vary by about  0.00004*(3.3 output Voltage/°C) = 5.28 mV over that 40° range.  Multiplying Rantwijk’s observed 2.5 Hz/mV *5.28 mV suggests that thermal effects on the regulator would cause less than 14 Hz of variation. Given that he was seeing a variation of several thousand Hz in the ceramic oscillator over a similar temp. range, regulator based thermal errors can be considered negligible.

What about freq. variation due to your decreasing battery voltage?

I had to go digging to figure out what kind of effect a decreasing battery voltage would have. The 5205’s data sheet lists a line regulation [ ∆VO/VO ] of 0.05%.  Since my rail is at 3.3v, that suggests a rail variation of about 165 mV over inputs between 3.65-12V.  With Rantwijk’s 2.5 Hz / mV benchmark, that would cause about 400 Hz of frequency variation over the entire input range the 5205 will support. That’s about 10% of the expected clock frequency delta. I should see less than 1/3 of that error since the batteries I’m using will only swing from 3.65-6v,  but three percent is significant so I’m going to need some way to compensate for main battery discharge.  My freezer testing also sees a 0.5V drop on alkaline supply batteries when they get cold, but provided I use the same chemistry during my calibration & deployment runs, voltage “droop” from ambient temperature should be included in the calibration.

Will it Blend?

This graph was created by counting the number of system clock tick during a 1Hz pulse from a DS3231 RTC. The count was captured with Nick Gammon’s Input Capture Method (see reply #12) on pin D8. The red line is the temperature in °C from an si7051 reference [±0.13 °C] (left axis, Celcius), and the orange line is the corresponding change in the count (right axis, freq.(-798000)).

This test used a Pro Mini clone, and the clock count varied by 4056 ticks over 44.54°C. That’s similar to Rantwijk’s result, and provides an average resolution of 0.011°C.  I was expecting a nasty response curve, with the least amount of change near 20°C since most components are optimized for room temperature operation. But the fit was surprisingly linear:

The key to getting a clean run is to change the temperature as slowly as possible so your reference sensor doesn’t get out of sync with the oscillator on the main board. To achieve this I built a dry calibration box out of two heavy ceramic pots, with about 3kg of rice between the two pots.  Each pot then has its own lid. I drove this box through a 40° range by putting it in the refrigerator, and then setting it on top of the house radiators. I excluded the “rapidly changing” portions of the data from the final calibration set. I also noticed that some boards seem to have a 6 hour “settling time” before the startup hysteresis goes away and temp/frequency relationship stabilized. Even then you still see some non-linearity creeping into the Temp. vs. Freq. graph above 40°C, which forces you to a 3rd-order polynomial to include those higher temperatures.

Dealing with oscillator frequency drift

Decreasing rail voltage isn’t the only thing that could give me grief because virtually all oscillators suffer from age related problems.  According to Frequency Stability and Accuracy in the Real World:

“Drift often proceeds in one direction and may be predictable based on past performance, at least for a few days. In some oscillators drift may be more random and can change direction. Long-term drift will affect the accuracy of the oscillator’s frequency unless it is corrected for.”

This is a real issue for our loggers which are now being deployed beyond the two-year mark.  And since we are under water, or in a cave, I can’t discipline to some NTP server, or GPS signal.  Whatever method I use has to rely on the existing parts.  By deduction, the temperature sensor inside the DS3231SN must have a reasonable amount of long term stability, since that record is the basis for the corrections that let Maxim claim ±2ppm.  And though the temp. record from the DS3231 has only quarter-degree resolution and an datasheet spec of only ±3°C, I’ve been surprised many times by how well it compares to sensors with respectable accuracy:

Most of the time you will see a small offset between the RTC and the reference sensor, and you can correct that to three nines with a linear from the calibration dataset.  (a typical example: y = 1.0022x + 0.0646, R² = 0.9996) Another thing I’ve noticed is that the RTC’s temp. register starts to diverge from my reference sensors above 50°C. So the DS3231 should probably only be used to discipline temp. accuracy across the same  0°C to 40°C range specified for the ±2ppm rating.

After the initial calibrations are done, you can correct drift in the frequency-delta temperatures by comparing them to the RTC’s temperature register. The only fly in the ointment is that the RTC record is so crude that direct adjustments based on it have the potential to put ugly stair-step transitions all over the place.

Drift correction with an Leaky integrator

Drift correction is necessarily a long game, so I’m not really worried about individual readings; just the overall trend. Tracking this requires more readings than you would use in a typical moving-average.  In addition, the 328 doesn’t have much computational horsepower, and it’s easy for numbers extending past the third decimal place to over-run the memory.  One solution to those problems is the leaky integrator:

//I pre-load filter variables in setup for 25°C, but that is not necessary
uint32_t rtcfiltSum = 800000;    // the previous reading "memory" variable (32*25000)
uint32_t rtcFiltered= 25000;     // equivalent to a temp of 25C
int filterShift = 5;             // equivalent of dividing new readings by 32
float tempFloat; uint16_t newValue;

//bit shifting only works with positive integer values (so no negative temps!)
//so you have to convert the decimal-float temperature readings
tempFloat=TEMP_degC*1000;  //preserves three decimal places from the RTC temp. reading
newValue=(uint32_t)tempFloat;
// this is the filter itself - save rtcFiltered in the log file for drift correction
rtcfiltSum += (newValue - (rtcfiltSum >> filterShift)); 
rtcFiltered =(rtcfiltSum >> filterShift);

That is an hefty amount of filtering, so the value in rtcFiltered will lag behind the current temp reading by several hours. However using a shift of 5 means that each new data from the RTC’s temp register only adds 1/32 of that to the final filtered value.  Even if the RTC takes one of it’s gigantic 0.25°C steps, the filtered value changes by only 0.25/32 =  0.0078°C. This transmogrifies the crunchy RTC record into one with a resolution comparable the data we are deriving from the frequency delta.  Then the de-trending can be done without harming the small details in the high resolution record. I could probably change that to shift=4 for faster response as only 1/2 of the resulting 0.0156 steps would typically show up in the record on average.

So the method is basically:

1) Create an IIR low-pass version of RTC’s temperature register.

2) Use the same leaky integrator on the frequency based temperature, which is calculated from the calibration data’s fit equation.

3) Compare those two heavily filtered numbers to create a rolling adjustment factor that you apply to the frequency based temperature data. Because you are only trying to compare long term behavior, you want these to be ridiculously heavy filters (ie shift 5 or 6) which you would never use in normal sensor filtering applications.

A typical 15 minute sampling interval only generates 96 readings per day, so this leaky integrator approach would take 8 hours to adapt to large changes in the offset. So it won’t do a thing to help you with short term hysteresis in environments that see large daily temperature fluctuations.  However, it would gracefully handle the line regulation problem, and it will respond to aging offsets that change direction half way through a long deployment.  Many sensors suffer from drift issues like this, and its easy to see how this correction method could be applied to those situations if you could define a temperature relationship.

Of course this begs the question of why Maxim didn’t make that temperature record available at a higher resolution in the first place; especially when you see all those unused bits in the register:

I suspect the design engineers had higher resolution output in the original plans, but it was pulled when some bean counter pointed out that doing so would cannibalize existing sensor sales – especially the highly profitable DS18b20s. If that is the case, it certainly wouldn’t be the first time a company disabled capabilities in a design to protect other product lines.

When things go squirrelly

I’ve been talking about the resonator/clock as though it was an isolated system, but this method is really a test of all the components on the board, and sometimes those combinations  produce some pretty weird behavior.  For example: The Rocket Scream Mini Ultra has long been one of my favorite Arduino compatible boards, and I’ve use it in many different logger builds because of it’s efficient MCP1700 regulator. But it doesn’t seem to work with this two-clock method because of a spectacular discontinuity at around 23°C:

Any response curve that produces non-unique solutions like that will make it impossible to do a simple regression curve-fit. The 800ST ceramic resonator on the Mini Ultra I used for this test looks quite different from the metal-can style one on the cheaper clone board, and it’s larger size has me wondering if there was some kind of compensation circuitry that was suffering from really bad hysteresis. Of course, there is always the chance I made calculation error somewhere…but if so I still haven’t found it, and the same code was running on both boards. The fact that the temp-frequency graph showed an inverse relationship also has me scratching my head.

Where to go from here…

Every board/RTC combination will have to be calibrated, which is a bit of a pain if you’re not doing week long burn tests like we do.  But the thing that really amuses me about this approach is that the potential resolution is inversely proportional to the quality of the components you are using. Rantwijk found a temp-co of less than +0.97 Hz / °C on the Duemilanove so the method has access to 100x more resolution on a Pro Mini. If you’ve got a Genuino with a quartz  oscillator, it’s probably not worth your time to even try.

One problem with driving the dry-box down to sub-zero temps is that the freezer’s compressor cycle will muck up the data if you don’t have enough thermal mass. If you live in a suitable climate, outdoor temps give you better results, with the added benefit that your spouse won’t gripe that they can’t find the chicken to make dinner behind “all that crap” you’ve stuffed into the ACSF.  (Arctic Conditions Simulation Facility)

Unless . . . uhhhh . . . doesn’t the 328P’s internal clock use an RC circuit with even more variation than the external ceramic resonators?  If memory serves,  Atmel suggests calibration to wrangle that puppy down from the ±10% factory defaults to ±1% over the -40 to 85 °C operating range. (see AVR4013: picoPower Basics)  For an 8 MHz system, that works out to a delta of more than 600 ticks per degree.  If you used Nick’s code to count with the 328’s internal oscillator, you could reach ±0.001°C with this two-clock method.  That’s approaching what you’d get from an RTD, and all you need is a $1 DS3231SN module from eBay.  Since you’d be dealing with parts inside the 328, you likely won’t have the kind of board to board variation I ran into with the Mini Ultra.

To read ambient temps, you would have to take the count right after waking from about 10 minutes of sleep, so the chip was not reacting to it’s own internal heating.  And I suspect reconciling the accuracy of those readings could involve more calibration-penance than any self-respecting maker should ever have to pay.  But it is just sitting therewaiting to be done by someone willing to set some fusesand that voice whispering in your ear right now is just good honest intellectual curiosity. No sin in that. Right?

After these shenanigans, I propose we update that old saying:
“A man with one watch knows what time it is.  A man with two watches is never sure.”
with the postscript: “But at least he can measure the temperature to ±0.01°C, provided one of those clocks is a 99¢ piece of junk from eBay.” 🙂

 

Addendum: 2018-03-19:
Just found an interesting document from the fluke archive describing the drift on an RTD sensor described as 90-day accuracy: 8520A/PRT precision temperature system (see graph on page 1)  Whats interesting is that those curves seem to have a period of accelerated drift, and then flatten out again.

Exposure to high temperatures affects all resistors (which RTDs & thermistors are). Prolonged exposure to temperatures over ~100°C  will cause a higher drift rate compared to lower temperatures.  Our loggers are usually out on deployment for at least a year… I’d be kidding myself to think I could constrain drift to anything near this RTD’s performance with the RTC oscillator.

The 2017 Cave Pearl Project ‘Year in Review’

Run times for Cave Pearl loggers without RTC pin power (0.25 mA) and with RTC pin powering (0.1mA). Dashed lines are projections to the point where the unit would have performed an automatic low-voltage shut down.

It’s been an intense year for the Cave Pearl Project, with several major advances. Almost all of the the units in the field sleep at 0.1mA or less with the RTC pin-power modification, so we can finally push the deployment cycle out to a two year rotation. Somehow I doubt that Trish will be able to wait that long for her data, but at least now we have the option of just leaving things in place if we need to when the fieldwork doesn’t go according to plan.

I also started experimenting with SD card shut-down this year, and early records from those prototypes are in, suggesting that they will run for at least four years. However I’m not going to call this technique ready for prime time until we see at least twenty units deliver more than a year of saved data.  Accelerated bench tests can only tell you so much about how a unit performs in the real world.

Extended lifespans mean that we can leave loggers with other people so they can do exploratory deployments in locations we can’t get to. Natalie Gibb from Under the Jungle sent us a brilliant video from one of those deployments:

We couldn’t have produced anything that polished with our hokey little point & shoot cameras!

I didn’t have much time to update the blog in 2017, but even with only six posts the traffic continues to grow, and we are approaching 100,000 unique IP’s. I still consider this blog an experiment, since I had no idea how much traffic a nerdy rant about data loggers would get when I fired it up in 2014. I think we’ve passed some kind of threshold this year, because traffic analyzers are now hitting the site regularly, and the comments are being spammed with a heap of “monetize your blog” garbage. Are there really people out there willing to annoy that many people for a $2.50 average CPM? (Ok…rhetorical question….)

I mentioned in the 2016 review that European countries were occasionally passing the US in the daily stats. At the time I thought that this was just an artifact of some mention in a forum thread, but through 2017 it started happening more frequently and now the US traffic is really starting to fall off. This struck me as odd, given all chin wagging you hear about STEM education initiatives in the burbs.

So I did a some checking with Google Trends and it’s not my imagination; something happened in 2017 that reduced the number of searches for Arduino in the U.S. while those same searches are rising in the rest of the world. Sadly, I don’t think you have to look very far to understand what’s happening when the president states that dismantling “the DEP” is one of his goals, and then appoints a director that sued the agency several times to carry out that plan. The relationship between the executive branch and the research community is so deeply dysfunctional right now that the Center for Disease Control has been ordered not to use terms like “evidence-based” or “science-based” in official documents.  Is it possible that folks on the street are getting the message that science has somehow become un-American?

I’ve no doubt more will be written about this epic estrangement, and in keeping with the freaky zeitgeist we’re introducing the Cave Pearl Projects first annual “Ugliest Deployment of the Year” contest.  A worst-in-show collection of from coastal outflows, mangrove swamps and hydrogen sulfide pits:

Sponges growing on a flow logger after 27 month deployment in a hydrogen sulfide cave. This unit smelled so bad that it attracted all the feral cats from the neighborhood when we brought it home for cleaning.

Bio fouling on an estuary unit after only 6 months. It was deployed as a floater, and I was quite surprised that it was still buoyant enough to function. A swarm of stinging sea lice made this retrieval particularly memorable.

Pressure unit after 6 month stint in a cave feeding a coastal lagoon. While this units’s not that ugly per se, that outflow fed directly into a swimming area that sees hundreds of tourists per day expecting the joys of “crystal clear spring water”… Nuf said?

Honorable Mention: This was actually reef logger from 2015, but I’ve included it here because this was the deployment that really defined the “fugly enough to be cute” benchmark that future deployments will have to measure up to.

To qualify, these dogs had to run through the entire deployment because without primary data to support your hypothesis – you’ve got nothing. (no matter what they look like on the outside…)  You can’t just wave your hand and create alternative data because in research that’s called lying.  Somehow, this not-so-subtle distinction has become blurred in America, where science itself now seems to be under attack.  It’s remarkable that someone with such an obvious facility with numbers could fail to appreciate how economically valuable the insights from research can be.

I’m genuinely concerned that if the the U.S. continues to turn inward, choosing a path towards declining health and social outcomes, it will become less appealing to the kind of entrepreneurs that built the country in the first place. The winners-take-all legislation rolling out of Washington these days threatens to eliminate the things that make risk-taking innovation possible.  An important nugget that policy wonks always seem to forget was eloquently summarized in Jennifer Romolini’s Career Advice article:

“You’ll Suck at Everything the First Time You Do It.
You will probably suck the second and third time too.
Don’t get defensive about this; don’t decide that you should never do the thing again…”

This was my first attempt to build a data logger, and yeah, it was pretty crude. I was lucky to get 24 hours on 6xAA batteries and it was held together with hot glue. My current builds are probably going to pass four years at 100-200 ft depth.   If I gave up when it still looked like this piece of junk on my kitchen table, I’d have no idea that was even possible.  Even today I get criticism from engineers who say the project is pointless because it’s not ‘state of the art’. But they don’t know where we started, so they also have no idea how far it will go. The same holds true for your project, whatever that might be.

Without a functioning a social support system, only people with rich parents ever get that second chance. It’s no surprise that bankers could care less about this, since they don’t have to produce anything new to make their money. But these days it seems that silicon valley is also willing to focus on corporate profits rather than innovation, even if that means looking the other way while legislators dismantle systems those propeller-heads needed when they were just starting out.  If people thought WannaCry was bad in 2017, just wait till they see what AT&T and Comcast does with the net neutrality rollback. (And they called the bill the “Internet Freedom Act”…nope…that’s not Orwellian at all… )

America will not succeed in the 21st century simply by making stuff cheaper with learning systems that remove human beings from the process.  If that does happen, who’s going to have enough money to buy the consumer goods that keep the economy rolling? But they could reclaim that edge by re-imagining how technologies interact with people and the environment. Of course, no one from old money will support policies that might upset the profitable status quo, so the last thing they want is an educated population changing things with a bunch of new ideas.

Unfortunately, the ‘new’ money created by the internet revolution is now acting like the old.  With few exceptions, the major tech players are willfully dropping the ball, spinning up proprietary bank crypto-currencies when they could be using tools like blockchain to build efficient resource allocation systems for new energy sectors that use considerably less water and create far more jobs. It’s time for the de facto stewards of the internet to accept responsibility for their creations, which are becoming the real arbiters of trust, fairness and justice in our society.  A good start along that path would be to make the voting system more secure from external threats, and from internal ones.

If the tech sector fossilizes into a corporate platform oligopoly, scientists / inventors get lured away by countries 1/10th the size, and congress bends the knee to carbon energy’s hired guns and PhRMA’s lobbyists, then the US will essentially be handing the future to China.  But none of that needs to happen. Especially if people take a moment to think about the role science, innovation and invention played in ‘Making America Great’, and then vote for leaders who help those things to flourish in the future:

Arduino Tutorial: Adding Sensors to Your Data Logger

This in-cave micro-climate recorder had pressure & temperature sensors mounted in little wells of Loctite E30-CL epoxy. This sensor potting method is described in our Pro Mini build tutorials.  Weather sensing stations are the most popular type of Arduino-based Sensor project on the instructables.com website.

This post isn’t another How-To tutorial for a specific sensor because the Arduino community has already produced a considerable number of resources like that You’d be hard pressed to find any sensor in the DIY market that doesn’t give you a dozen cookbook recipes to follow after a simple Google search. In fact, you get so many results from “How to use SensorX with Arduino” that beginners are overwhelmed because few of those tutorials help people decide which type of sensor suits their skill level. This post attempts to put the range of different options you can use with a Cave Pearl data logger into a conceptual framework, with links to examples that illustrate the ideas in text.

One thing to note before you start is that many modern sensors will only accept 3.3v inputs, so UNO based projects need to check if the sensor they want to use is 5v tolerant. Most sensors from vendors like Adafruit put regulators on their breakout boards to handle this 3.3v-5v translation, but you may have to place level shifters between some of the more advanced digital sensors and an UNO based logger. Occasionally you run into the opposite situation where the sensor requires 5v (or more) forcing Pro Mini based systems to do the same thing.


Analog Sensors:

Some substances react to energy input by changing their physical or electrical properties. Arduinos can only read voltages, so to record these changes in the physical world some kind of circuit is needed to convert those properties into a voltage. Sensors that output continuously varying voltages in response to natural phenomenon are called analog sensors. Arduino pins A0 to A7 are analog input pins, and the ADC inside the microprocessor converts those voltages into a numerical value between 0 and 1023. It’s worth keeping in mind that those numbers are somewhat arbitrary depending on the reference voltage, and the behavior of your sensing circuit – so it’s up to you to figure out how to convert the raw ADC readings back into understandable units of the phenomenon you were trying to measure (like degrees Celsius for temperature, or m/second of wind speed, etc.)

A typical photoresistor divider from Sparkfun’s Voltage Divider Tutorial It’s worth noting that many LDRs go from 100 Ohm to over 1 MOhm. So you would have to change the series resistor to capture  a range that large.

The most common analog sensors are those that change their resistance in response to temperature (thermistors), light (photo resistors) or pressure (variants: force / stretch / bend )  If a sensor varies in resistance, you can turn that into a voltage by adding a fixed resistor to create a voltage divider circuit.  The non-sensing resistor in the divider is usually chosen with a value near the midpoint of the sensing devices range. For example, a photoresistor might vary between 1kΩ in the light and about 10kΩ in the dark, so a suitable resistor to pair it with would be ~5kΩ. For analog sensors that change by really small amounts, more sensitive Wheatstone bridge arrangements combine 2 or 4 sensors in the same circuit to expand the delta, but it’s the same basic idea: you are converting a change in resistance into a change in voltage.

Divider methods are referred to as ratiometric because the output voltage from the circuit is some fraction of the supply voltage determined by the resistances of the components. If the input voltage is doubled, the output voltage is doubled, so these circuits work fine on 5v UNO and on a 3.3v Pro Mini. By default the Arduino ADC takes a reading by comparing it to the same rail voltage supplying the resistive divider, and sensor nerds like me get all excited about this because it means that noise from your power supply will have no effect on the readings. You can squeeze more sensitivity per bit out of the Arduino’s humble 10-bit ADC by changing to a lower internal reference voltage. However once the Aref is different from your supply, that rail noise shows up on the divider output unless you squelch it out a smoothing capacitors.

Some light sensors get used in conjunction with a emitter for reflectance and ranging applications. You can create a reasonably good color sensor by combining an RGB LED emitter with a simple photoresistor. However, the humble LED  not only emits light, but can also be used to sense it because from a physics point of view, a diode is simply a PN junction, so a rectifier diode, a light emitting diode and a photo-diode are basically the same device.  Forrest Mims built some cool filterless photometers with LED sensors long before the mainstream media started waxing philosophical about ‘citizen science’.  (Also see: jpiat’s Li-Fi )

Most analog sensors are simple devices, but there are more complicated versions providing modified analog output, where some extra circuitry has been added to convert the highly non-linear response you get from typical resistance based sensors into the kind of straight y=m(x) relationship you get from a TMP36. This greatly simplifies the math required to convert your analog voltage readings into the real world property you were actually trying to measure. Some analog sensors (like thermocouples) generate tiny voltages, but those signals may so small that they need to be amplified before the Arduinos ADC can read them, so these analog sensors may also be sold with supporting electronic boards to boost the output.

Sensors can be mounted  inside a housing with a couple of layers of 3M Scotch Outdoor Mounting Tape. Sensors mounted this way have stayed in place for many years of deployment. The adhesive will sag somewhat under gravity if you expose your loggers to temps >55C.

At the top of the analog sensor food chain, there are complex Micro Electro Mechanical (MEMS) devices like accelerometers. In these sensors, silicon has been machined into very tiny physical devices made from springs, coils and flat sheets. These micro-cantilevers form capacitors that react to movement by changing a voltage and they are usually arranged in sets of three on x,y,&z axes. This means you need to read three separate input pins to capture a complete reading from the device. Since the Cave Pearl data loggers use pins A4 & A5 for communications with the RTC module, and A0 to track the main battery voltage, a complex analog sensor like the ADXL335 can use up all of the remaining analog inputs on the logger unless you build it with an Arduino that makes inputs A6 & A7 available. (the Pro Mini does, the UNO does not)  The limited number of analog input pins can motivate people to switch over to digital-bus sensors, though multiplexers provide another possible solution to the problem.

If you start with the project’s basic UNO logger script , adding a new analog sensor requires only three lines of code. Add

int AnalogSensorReading = analogRead(A0);  
// change A0 to match the input pin you connect the divider to

at the top of the main loop. Then add that new sensor data to the concatenated dataString which is saved to the SD card at the end of the main loop:

dataString += ", "; //comma separates new data from that already in the string
dataString = dataString + String(AnalogSensorReading);

That’s it. This simplicity is why analog sensors are usually the first ones people encounter when they are learning the ropes. Of course there are some advanced tricks you can play to supercharge Arduino’s humble 10-bit ADC, and you’ll find more useful tips over at Nick Gammon’s ADC tutorial.

Digital Sensors:

A bullet-proof de-bouncer from www.ganssle.com.    Compare this to the 5-key de-bouncing circuit from the IBM 705

Unlike analog sensors, digital sensors only output two voltages: High & Low. Usually the high voltage is the same as your power rail, and the low voltage is your system ground. In some ways that makes digital sensors easier to use, but there are some devils hiding in the details, and digital sensors cover the entire range from crude noob level devices to Gordian knots with more computational horsepower than the Arduino itself. Even the most complicated digital sensor usually has an analog sensor hiding somewhere at its core.

I group digital sensors into three conceptual categories:

Flippers,   Thumpers,  &   Thinkers

This is based on what kind of output they produce, rather than the complexity of their electronic circuits. And it’s not unusual for more advanced IC-based digital sensors to be as easy to use as the flippers & thumpers, because some kind soul has released a library that takes care of the gnarly low-level details.

1) Flippers

The humble push-button can be thought of as a crude pressure sensor that can be in only two states: open or closed. Add a couple of passive components for debouncing, and reed switches become the digital sensor of choice for event counting sensors like the tipping-bucket rain gauges you find in weather stations.  IR break-beam switches are another common implementation with on/off output.

When you first look into digital sensors there seems to be a bewildering array of different ‘breakout boards’ and ‘sensor modules’ for the Arduino. These are often sold in bundles of twenty, thirty or even sixty different pieces. Once you get a closer look at them, you notice that many these cheap sensor modules look similar to each other:

That’s because most of those boards are simply a voltage divider connected to one leg of a five cent comparator circuit, with a twenty cent trimming pot setting the voltage on the other side:

These boards switch their high/low output when the sensor voltage crosses the threshold set by the trimpot; changing the original analog voltage divider into an environmentally responsive threshold alarm. It’s such a generic circuit, and you could connect other resistive sensors and the board wouldn’t even notice. If you use these modules with the Cave Pearl loggers, look for boards that also break out that 4th analog pin so you can also read the sensors output with the ADC.

Integrating simple on/off digital sensors to your logger code would use almost the same pattern as the analog sensor reading:

pinMode(PinNumber, INPUT);  // Declaring the pin as an digital Input
int DigitalSensorReading = digitalRead (PinNumber);

dataString += ", ";         //comma that separates new data
dataString = dataString + String(DigitalSensorReading);

Those eBay boards are all somewhat redundant, since the Arduino has a built-in analog comparator on pins D6 & D7 already.  However there are many high/low output sensors with more complicated circuits that are not as easily replicated.  Proximity sensors can have complex internal circuitry and perhaps the most common of these more-advanced-but-still-simple sensors would be the passive infrared (PIR) motion sensors that seem to occupy every corner of the modern world.  Adafruit has a fantastic tutorial on how to use them with an Arduino, which also demonstrates how the boolean HIGH or LOW value you get back from digitalRead() can be used with if statements to select different courses of action:

Reading = digitalRead (PinNumber);

if(Reading == HIGH)
  {   Serial.println("input is HIGH");   }

if(Reading == LOW)
  {   Serial.println("input is LOW");   }

All the I/O pins on an Arduino can be used as digital inputs  (including the analog lines) and the cool thing about that is the circuitry hidden behind those pins inside the microprocessor. The  Schmitt trigger on each pin has read-high vs read-low threshold voltages. This lets you replicate what those cheap eBay modules do by replacing the fixed resistor in your analog voltage divider with variable one, and then connecting the output of that divider to a digital input pin. Inferring resistance (or capacitance) by timing threshold crossovers on a digital I/O pin can produce respectably high resolution analog readings because micro-controllers count time far more precisely than ADC’s measure voltage.

2) Thumpers:

Flippers change state slowly by microcontroller standards, and since they can be read with a single digitalRead() command, they won’t get you much cred at the local hacker-space. To get into the digital world’s caffeine-driven middle class you have to start working with Thumpers. These are sensors which convey information by varying the amount of time the sensor outputs a high voltage at a given frequency (called pulse-width modulation or PWM) OR by changing their output frequency with a fixed 50/50 split between on&off time (this is called frequency modulation or FM) .

This kind of output was common long before the Arduino existed because putting an analog sensor into the oscillator circuit feeding a 555 timer chip changes the pulses coming out the other end in proportion to the sensors resistance / capacitance / etc. You’d be hard pressed to find any environmental sensor that can’t be constructed with a couple of op-amps and a 555 (See: the conductivity sensing post for examples).

Three common methods for reading pulsed signals with an Arduino are:

  1. The pulseIn() Function
  2. External Interrupts
  3. Pin Change Interrupts

The output of the pulseIn() function is the time in microseconds that it took for the pin to go (or be) LOW, then go HIGH, then go LOW. This is the method of choice for PWM thumpers, and it is extremely easy to use provided the incoming signal is a clean square wave.
Unfortunately, it does not handle frequency modulation very well at the high end, because it’s susceptible to errors in timing when detecting the start and end of really short pulses.

Range finding sensors often output PWM signals, and the most popular of those is the HC-SR04 which is used for collision avoidance by just about every Arduino-based robot on the planet. Self-balancing robots are one of the maker movements “killer apps”, and it doesn’t hurt that the SR04 transceivers just happen to look like a pair of eyes. There’s currently a bit of a turf war between the SR04 and slightly more expensive IR rangefinders. Ultrasonic energy is absorbed by soft materials, and SR04’s are susceptible to interference & multi-path issues in environments where there are lots of flat rigid surfaces. Infrared sensors have a much more focused beam so you get better results finding small objects…like the other robot you’re currently doing battle with.  (For more precise work you can go upscale to the VL53L0X Time of Flight distance sensors, and if money is no object, you can take that all the way to LIDAR.  To get the highest level range-finding merit badge of them all: MaxBotix Sonar sensors let you play the game under water…)

A code-side implementation for the HC-SR04 could be as simple as this:

digitalWrite(triggerPin, HIGH);       // send out (transmit) the pings
delayMicroseconds(10);                // give the sensor 10 ms to settle
digitalWrite(triggerPin, LOW);        // stop the outgoing pings
duration = pulseIn(echoPin, HIGH);    // listen for the echo and return time. 
Distance2reflectingSurface = (duration/2) / 29.1;
// Divide by 2 since the sound ping travels out & back = twice the distance
// Speed of sound in air = 340 meters per second or ~29 microseconds per centimeter
// Then divide the duration by 29 cm = distance in centimeters.

External interrupts handle both PWM and FM efficiently with the limitation that there are only two hardware interrupt lines on a typical Arduino. The Cave Pearl loggers are already using D2 for the RTC wakeup alarms, and that leaves only D3 available for hardware methods calling attachInterrupt().

It’s worth noting that there is also a near IR (940nm) sibling in the TSL family: the TSL245

The TSL235R light-to-frequency sensor outputs a square wave (50% duty cycle) with a frequency proportional to light intensity. The TSL235 is self-contained, well calibrated, and very linear over the ultraviolet-to-visible light range of 320 nm to 700 nm. Calibration in manufacturing is something that most companies will try to avoid, and when you include the fact that this sensor works from 2.7-5.5v, you have a $3 sensor that’s nearly perfect for use with Arduino-based data loggers. Rob Tillaart has posted a simple bit of code that counts the interrupt pulses per second from this FM sensor over at the Arduino playground. It should be easy to integrate these functions into the Cave Pearl base code, and to modify it to work with any other FM output sensor. Data from light sensors usually requires post processing with somewhat complicated luminous efficiency calculations, but if you Google around you’ll find plenty of Arduino tutorials on those steps (also see: Insolation Models).

Only D2 & D3 support external interrupt signals by default, but with a little bit of extra code interrupt signals can be received on any of the Arduinos I/O pins.  Interrupts triggered from pins other than D2 & D3 are referred to as Pin Change Interrupts. Pin change interrupts are grouped into 3 ‘ports’ on the MCU. This means there are only 3 interrupt subroutines to handle input from all 20 pins. This makes the code somewhat more complicated than Rob Tillarts example, as it now needs to determine which pin triggered the ISR. That extra complication usually motivates people to use something like the PinChangeInt library for situations with a limited number of input pins.  Anemometers often use interrupt-based approaches because they work with output that’s so variable that it can’t really be classified as PWM or FM.

There are many great frequency counting libraries but it’s important to note the difference between ones which count the number of pulses during a fixed “gate interval” time, and those measuring the period of a single high frequency pulse.  Rob Tillart’s code uses the counting method, and this works well for relatively high frequencies, because many cycles are counted during the gate interval and this reduces error. At lower frequencies, very few cycles are counted, and the precision suffers, so measuring the elapsed time during a single cycle is a better option at low frequencies.

This image is from PJRC’s FreqCount Library page, which goes into more detail on the FM sensing process. It’s worth noting some of the other useful sensor libraries that Paul Stoffregen has released including: Onewire library for the ever popular DS18B20 temperature sensor, SerialFlash to simplify SPI memory builds, and their project blog is a font of other highly interesting things.

According to PJRC: Frequency Counting: works best for 1 kHz to 8 MHz and Period Measuring: works best for 0.1 Hz to 1 kHz.  There is some wiggle room there, and you should check your sensors data sheet to make sure your method matches the output range.

If you’d rather skip the libraries, you can get closer to the bare metal with the advanced timer over-flow methods described at www.fiz-ix.com and at Nick’s Timers & Counters page. Nick’s page describes a method to measure frequency with the input capture unit on pin D8. While that’s some pretty advanced code, it allows you to measure pulsed inputs to a resolution approaching the frequency of your system clock.

3) Thinkers

This Grove I2C hub connects several the sensors to the bus, so only one jumper set needs to be patched down to the logger platform. The mounting shown here was done with plumbers putty, which hardens quickly, and adheres well to most plastic enclosures.

Modern chip-based sensors offer high resolutions and complex signal processing capabilities that can be hard to replicate on the Arduino.  Most of these digital sensors send data using serial communication protocols over a common set of “bus” wires that are physically connected to all of the sensors. Serial protocols can be intimidatingly complex for beginners, but you rarely need to worry about the details because most of the vendors in the Arduino landscape release libraries to simplify the use of the sensors they sell. These libraries make it quite easy to work with complex sensors, and they are one of the reasons that companies like Adafruit and Sparkfun have such a dedicated following in the maker movement.

Newer versions of the Arduino IDE have a Library Manager which provides access to a large list of libraries with a one-click install. Sensors that have been used for a few years by the community often have a library available through the manager. However for new sensors, you usually have to download a library and manually place it in your

Username>Documents>Arduino>Libraries folder.

You can Learn more about installing Arduino libraries at:
Sparkfun :Installing an Arduino Library,    All About Arduino Libraries at Adafruit 
and  Installing Additional Arduino Libraries at Arduino.cc.

Note that these little bits of library code can be located in several different places on your hard drive but it’s best to keep the ones you add in your sketchbook folder because the Arduino Software (IDE) upgrades itself by first erasing everything in the program root directory: including any libraries that were stored there. Libraries in your personal document folders are not deleted during the Arduino Software (IDE) update process.

Here I’ve added a resistor which pulls the CS pin high to tell this ADXL345 accelerometer to communicate with the I2C protocol rather than SPI. The value of that pull-up resistor is not critical, so they can range from a 200Ω to 10kΩ.

The digital sensor protocols you are most likely to see used with an Arduino are SPI and I2C. It’s fairly common for chip-based sensors to support BOTH protocols and for those you usually add a pull-up or pull-down resistor to tell the chip which one to use.  SPI is preferred when fast communication is needed to move large amounts of data, but this is rarely the case for environmental monitoring. More importantly, the Arduino SD card libraries expect the SPI bus to be operating in Mode0. Adding a sensor to the Cave Pearl Logger which changes the SPI bus to one of the other three operating modes would prevent data from being saved until the bus was reset to Mode0. I have yet to find an SPI-only sensor that doesn’t have an I2C equivalent on the market.

 

I2C sensors are often the best choice for Cave Pearl Data Loggers.

The DS3231 RTC breakout module used on the Cave Pearl logger has a cascade port at one end, making a perfect attachment point for other I2C devices

The I2C bus or TWI (Two Wire Interface) allows a single master (the Arduino) to share communication lines with more than 100 slave devices (the sensors). Cave Pearls use an DS3231 RTC for timekeeping and the I2C breakout board carrying it provides 4.7k pullups resistors on the SDA and SCL communication lines. Each new I2C sensor gets connected to the same wires as the RTC board. If you have a good library to go with your sensor, about the only thing that might prevent it from working is a bus address conflict. Because I2C devices are all connected to the same wires, the Arduino needs a way to talk to only one device at a time. It does this using the I2C address of each sensor. (kind of like a phone number)

The first thing to do with a new sensor after connecting it to your Arduino, is run a bus scanner which queries every possible address to see if any devices are responding. If two devices are trying to use the same address, only one of them will show up in the scan, and sometimes neither of them will. Code for this basic utility for this can be found at the Arduino playground.

Running that bus scanner on a Cave Pearl data logger before any sensors are attached should produce:

This output screen tells us that the RTC breakout board is functioning and the I2C communications are working. It also tells us that the I2C “device addresses” are 0x57 ( this is the EEprom on that module ) and 0x68 (the DS3231 RTC).  Adafruit has compiled a list of typical I2C addresses for different sensors and scanning through that list for the two we are already using on the logger finds some potential conflicts:

0x57
MB85RC I2C FRAM (0x50 – 0x57)
MAX3010x Pulse & Oximetry sensor (0x57)  (uh-oh… this sensor will not work with our logger!)

0x68
This address is popular with real time clocks – almost all of them use 0x68!
AMG8833 IR Thermal Camera Breakout (0x68 or 0x69)
DS1307 RTC (0x68 only)
PCF8523 RTC (0x68 only)
DS3231 RTC (0x68 only)
MPU-9250 9-DoF IMU (0x68 or 0x69)
MPU-60X0 Accel+Gyro (0x68 or 0x69)
ITG3200 Gyro (0x68 or 0x69)

Some I2C devices have only one fixed address and but most offer a small range of different addresses that you can set by connecting different pins on the module to power or to ground. This will let you resolve an address conflict, but be make sure to make corresponding changes in your code if you change a sensor address away from it’s default. Most sensor libraries will have a modifiable parameter for the device address that is used to initialize the sensor. If you have a sensor with a fixed address, you will only be able to hook up one of those sensors to the logger at a time unless you add an I2C multiplexer to resolve the address conflict.

Once you’ve confirmed the sensors show up on a scan of the I2C bus, the next steps depend on the complexity of your sensor. I2C sensors that only do one thing can often be read with a minimal amounts of code after the #include <Wire.h> statement embeds the TWI library that’s built-in to the IDE . You often see this with a basic temperature sensors like the TC74

//All I2C coms start with a handshake transaction with the device @ address
Wire.beginTransmission(address);
Wire.write(0);  //Sends a bit asking for register 0, the data register of the TC74
Wire.endTransmission(); // nothing is sent over the wires until wire.end is executed

//then you request the temperature data from the TC74 sensor
Wire.requestFrom(address, 1);  //this requests 1 byte from the specified address
int celsius= Wire.read();

For sensors that do more complicated things there can be many more steps, especially during sensor initialization when you might have to configure the bit-depth of the readings, the sampling speed of the sensor, and a host of other options. I’ve posted an extensive tutorial about this for tech-savvy users (see: How to configure I2C sensors ) but for beginners the best approach to adding a new I2C sensor is

1) Find a suitable tutorial by typing “How to use SensorX with Arduino” in Google or by reviewing the tutorials available at: Hookup Guides at Sparkfun, the Sensor tutorials at Adafruit, or search for code examples and links at the sensors forum.

2) Download the associated sensor library and install it into your Documents>Arduino>Libraries folder

3) Add #include <SensorLibrary.h> to the start of your code

4) Initialize the sensor in startup following the code examples for your tutorial located on GitHub

5) Read the sensor in the main loop

Most libraries are written to provide InitializeTheSensor() and ReadTheSensor() functions that so steps 4) & 5) often end up adding only couple of lines to your code.

As a simple example look at the MCP9808 temperature sensor from Adafruit:

The Tutorial   &   The example code on Github

That script is quite small because the library condensed a lot of I2C handshaking down to

tempsensor.begin();  // initializes the sensor
tempsensor.readTempC();   // reads the sensor

For an example of a library driving a more complex sensor, look at the BMP180 pressure sensor from Sparkfun

Hookup guide   &  The example code on Github

There’s an important step at the start of the code:

#include <SFE_BMP180.h>
#include <Wire.h>
SFE_BMP180 pressure;    // creates an SFE_BMP180 object, called pressure
#define ALTITUDE 1655.0 // Altitude of SparkFun's HQ in Boulder, CO. in meters

In setup, theobjectname. appears before each call to library functions:

pressure.begin()       //initializes the sensor in setup

And then in the main loop, the sensor uses a four step process to complete one reading.

status = pressure.startTemperature();
delay(time);
status = pressure.getTemperature(T); 
// the temperature must be read before the pressure!

status = pressure.startPressure(3);  // with oversampling set to 3
delay(time);
status = pressure.getPressure(P,T);

Multi-step read procedures like that are quite common, because it takes time to capture high resolution readings, and in this case the temperature has to be sent as a correction factor for the pressure reading.

Then there are two more functions in the example program worth noting:

p0 = pressure.sealevel(P,ALTITUDE);
a = pressure.altitude(P,p0);

The pressure sensor returns absolute pressure, and Sparkfun have provided extra code in their library to do calculations which convert that number into sea level equivalent & altitude equivalent numbers.  That Sparkfun code example is pretty typical of what you get with libraries for more complex sensors, and it should not be too hard to just open two IDE windows to copy and paste the required pieces of code from the Adafruit & Sparkfun examples into the basic Cave Pearl Logger script on Github.  There is nothing magical about libraries: they are just pieces of code that you can read through yourself by opening the .cpp file listed in the same Github repository. I recommend that you always review the library code, as figuring out how someone else’s stuff works is an important part of learning how to program the Arduino.

Think about your housing-logger interconnections before you start your build.  My current favorite connectors are Deans Micro Plugs, which are available in 2,3,4,5,6,& 8 pin versions. Use a consistent color convention for different bus wires.

Most libraries will include a simple example sketch with the downloadable file. These show up in the IDE in the FILE>EXAMPLES> pull-down menu after the library is installed, so you don’t usually have to go all the way to Github like I did here. The included examples usually only initialize the sensor and print out some raw readings, but that’s exactly what you need to verify the sensor is working before you merge those bits into your own code.

The real benefit of a good library is not just the code, but the significant amount of time someone spent slogging through a sensors data sheet figuring out the correct sequence of operations.  Just because a library exists for your sensor does not mean that it is necessarily a good one – especially when you find them out in the wild. So you should test different libraries when you have options. I generally chose libraries that require the least amount of memory at compile time, and/or ones that give me access to the ‘raw’ sensor readings in addition to the processed output. Raw sensor readings let you do calculations later in Excel to make sure the library didn’t introduce an error somewhere.  Another thing to keep in mind is that sensor libraries don’t have to be continually upgraded like the software you run into on a more complex system. Once a sensor library is working, it will hang around for years with no updates because none are needed.

Well… this post swelled into another voluminous tome, but hopefully no one lost sight of the forest for the trees.  Generally speaking you can buy each type of sensor in all of the data output ‘flavors” described in this post.  As an example, there are both analog (voltage) & digital (pulsed) anemometers, and the digitals range from simple reed-switch thumpers to ones with onboard IC’s doing most of the raw signal processing to provide calibrated wind-speed numbers over an I2C bus.  Don’t mistake the Analog vs Digital divide as any indication that one kind of sensor is necessarily better for the job you are doing.  Same goes for my tongue-in-cheek categories for Flippers, Thumpers & Thinkers. They’re just conceptual tools to use when you are hunting through tutorials on instructables, or when you run into an intimidating wall of information like the Interfacing with Hardware page at Arduino.cc.

Although this post has been focused on capturing sensor data with a logger, you should also keep in mind that there are many different physical methods to measure the same phenomenon. Using the anemometer example, most people think of the traditional egg-cup spinners because that’s what they are used to seeing on rooftops, but heat-loss methods, and ultra sonic methods are also quite common. A Google search on ‘how to measure water level’ shows the incredible range of different sensors can be put to that simple task.  When you are faced with a range of methods like that, the ‘best sensor’ for the job is the one you can actually get working, and that usually boils down to the amount of programming complexity you are comfortable with. Good libraries can level the playing field quite a bit, making complicated sensors almost as easy to add to your data logger as basic analog voltage dividers.

Addendum 20171218

A few people have commented about my use of string variables in the basic logger code, and the general consensus is that the String class should be avoided because it can lead to memory fragmentation. It is better to use character arrays, but there is a significant learning curve there and strings will let you build a working data logger when you are just starting out. Majenko has one of the most concise summaries of steps to address this issue, and there is a reasonably good introduction to character arrays, and many other helpful concepts at the Starting Electronics: Arduino Programming Course  (see: Section 18).  Personally, I find that having to re-jig sprintf() statements when I want to add another sensor to my logger is a pain in the backside. A more memory friendly approach could be to simply open the file and then save the variables directly to the SD card without the string concatenation steps.

dataFile.open(FileName, O_WRITE | O_APPEND);
dataFile.print(AnalogSensorReading); dataFile.print(",");
dataFile.print(DigitalSensorReading); dataFile.print(",");
[...]
dataFile.close();

An alternative way to address String memory issues is to use the Pstring library by Mikal Hart.   “Print-to-String” is a lightweight Print-derivative string class that renders text into a character buffer that you define at the start of your program.

char DATABuffer[30];  //This character array receives the ascii characters
// it's worth noting that you can't move more than 30 bytes at a time 
// over the I2C bus due to limitations of the wire library buffer, 
// so my receiving arrays are usually [30] bytes long.

The data concatenation steps I described previously for the basic UNO logger:

dataString += ", ";      //comma that separates new data
dataString = dataString + String(DigitalSensorReading);

are slightly different for the more advanced Cave Pearl logger code which uses the Pstring library:

PString str(DATABuffer, sizeof(DATABuffer));// set the array as the receiving buffer
str = "";                                   // this empties the receiving buffer
str.print(CycleTimeStamp); str.print(",");  // this data is already in ASCII format
str.print(DigitalSensorReading); str.print(",");     // this data is an integer
//add more variables as needed  up to the [30] char limit
// separating each additional sensor reading with a comma

It does not matter what what the source variable format is – float, integer, etc – it all gets rendered into ascii by the str.print statements. And Pstring will never cause a buffer overflow because any excess data that you try to add to the DATABuffer is simply discarded. That receiving buffer will always contain valid (i.e. NULL-terminated) C string data. This makes the method much friendlier for people who are new to programming.

To save the sensor data stored in the char array buffer to the SD card use file.write:

file.open(FileName, O_WRITE | O_APPEND);
file.write(DATABuffer, sizeof(DATABuffer));
file.close();

 

Bil A. Phillips: an Explorer, a Teacher, and a Friend

For me this expression, while on a Nat. Geo. sponsored expedition in Belize, captures what made Bil one of the preeminent cave-diving instructors in the world. Every time I see it, I hear his voice saying ” You can do this better Ed.  Just remember what I taught you.”  And he had the uncanny ability to deliver that message with a smile.

Today, I am deeply saddened to hear of the passing of Bil A. Phillips, a great friend and a mentor to so many of us in the the Cave Diving community.

The research I write about on this blog would be almost impossible to do without help from lots of wonderful people, and I try to give them a shout of thanks wherever possible. But when it comes to Bil, words simply can’t express the incredible amount of encouragement, support, and guidance he has given us over the years.  He taught me to cave dive many years ago, when Trish was doing the work that became the foundation for her PHD. At the time I had no means of paying for the training, but this was, to him, irrelevant.  What he cared about was commitment, and passion for one of the most spectacular environments in the natural world.  Cave exploration was his life, so even the most exacting safety lesson (and there were many of those) was seamlessly woven into one of Bil’s real-life stories (of which there were many, many, more… )

There was a time when trips to ‘the Pit’ were a lot more challenging than they are today.  And if memory serves, I think Bil liked it better that way…

Our experience of his generosity is hardly unique, and the well oiled machine he created at Speleotech has been the pillar supporting an untold number of research expeditions over the years. And no element of the underwater work ever escaped his good-natured scrutiny because he was as much an explorer of ideas as he was of caves.  One trip would barely finish before he was tossing around mischievous plans for the next, and intense conversations about what needed investigation to protect his beloved caves, were as much a part of the diving experience with Bil as anything else.

The world has lost a true explorer, who mapped the Yucatan systems with a rare combination of technical mastery and a spirit for adventure.  I hope the Cave Pearl Project stands as a fitting tribute to the effect that man had on our lives.

Bil – we will miss you dearly.

<— Click here to continue reading the story—>

Arduino Tutorial: How to Configure I2C Sensors

I’ve spent the last year in the ‘uncanny valley’ of the Arduino. That’s the point where you understand the tutorials at Arduino.cc, but still don’t get much from the material on gitHub because trained programmers would never stoop to using the wire.h library when they could just roll their own in native C++ using the avr-g compiler.  The problem with establishing sensor communication at the level of the TWI peripheral inside the AVR is that there are so many fiddling details to keep track of that it quickly overruns the 7±2 things this average human can hold in his head at one time: Computers aren’t the only things that crash after a buffer overflow!  So this post is meant to be a chunking exercise for beginner-intermediate level people who want to get a new sensor working using the standard IDE.  I’ve tried to distill it all down to things that I run into frequently, but there’s still a lot of material here:  So pour yourself a cuppa before diving in...

The great strength of I2C is that you can put so many sensors on the same four wires. But for units with several pre-made modules connected you might have to remove a few smd resistors from the breakouts, or the pull-up on the bus might become too aggressive. Most of the time I just leave them on, so I can extend the wire length, or crank up the bus clock…

REGISTERS are simply memory locations inside an I2C device. The summary of how many registers there are in a given sensor, and what they control or contain is called a register map. Most of the information on the sensor’s datasheet is about explaining how each register functions, and they can be quite a slog to read through because the information is rarely presented in an intuitive way.

To give you a sense of what I mean by that: take a look at page 14 of the manufacturers datasheet for the ADXL345 accelerometer:

A document only a hardware engineer could love…

Then take a look at the interactive register map for that sensor over at the i2cdevlib site:

Even if you’ve never worked with registers before, jrowberg’s visual grid layout makes it easy to see how the sensor’s memory is divided into sections, which are doing different things.

There are many kinds of registers but for this introduction I am going to group them into three general types: Control, Data and Status registers, and provide brief examples of code that you can use to work with each of them. The functions named with the i2c_ prefix should be generic enough to work with most I2C sensors, but I’ll also be referring to a few specific cases to show how you might need to modify those basic functions.

1) Control Registers

Most sensors change how they operate based on the values stored in control registers. Think of control registers as banks of On/Off switches, which you turn on by setting a bit to 1 and turn off by setting that bit to 0.  I2C chip-based sensors often have a dozen or more operational settings for things like bit-depth, sampling speed, noise reduction, etc., so you usually need to set bits in several different control registers before you can actually take a reading. And sometimes there are “special chip functions” that perform some kind of post processing on those sensor readings that would be hard to replicate on the Arduino. These can add an extra layer of control settings to take care of when you initialize the sensor.

Arduino’s wire library can only transfer 8-bit bytes over the I2C bus, so that’s the smallest amount of information you can write into a register memory location at one time. This can potentially change eight of those control switches simultaneously and, for parameters that are controlled by more than one bit, sometimes it’s actually required that you set them in one register-writing operation.  Most people use byte variables for the sensor’s bus and register memory addresses, but once you’ve figured out the pattern you need to set up in control register switch-bits, it helps to write that information as a long form binary number (eg. 0b00001111) so you can see the on/off states when you read through your code. 

Writing a byte to a sensor’s control register can be done with four basic steps:

Wire.beginTransmission(deviceAddress);  // Attention sensor @ deviceAddress!
Wire.write(registerAddress);   // command byte to target the register location
Wire.write(dataByte);                           // new data to put into that memory register
Wire.endTransmission();

The I2C deviceAddress is set by the manufacturer but some can be modified from their defaults by connecting solder pads on the breakout board.  Since the bus address of a given sensor IC can vary from one module to the next I keep Rob Tillaart’s bus scanner handy to find them, and more importantly to discover when two sensors are fighting with each other by trying to use the same address on the bus.  The registerAddress moves a pointer inside the chip to the memory location you specified. You can think of this pointer as a read/write head and once that pointer is aiming at a specific register, the next byte you send along the wires will over-write the data that was previously stored there.

The startup default values for a given control register are often a string of zeros because all the chip functions being controlled by that register are turned off. Unfortunately this means you’ll find lots of poorly commented code examples out there where people simply write zero into a control register without explaining which of the eight different functions they were aiming for because seven of those were still at their default zero-values anyway.

Reading data from a sensors memory register(s) requires two phases:

Wire.beginTransmission(deviceAddress);    // get the sensors attention 
Wire.write(registerAddress);    // move your memory pointer to registerAddress
Wire.endTransmission();           // completes the ‘move memory pointer’ transaction

Wire.requestFrom(deviceAddress, 2); // send me the data from 2 registers
firstRegisterByte = Wire.read();             // byte from registerAddress
secondRegisterByte = Wire.read();       // byte from registerAddress +1

The first phase tells the I2C slave device which memory register that we want to read but we have complete the read operation in two separate steps because the wire library buffers everything behind the scenes and does not actually send anything until it gets the Wire.endTransmission(); command.  The second phase is the data reading process and you can request as many bytes as you want with the second parameter in Wire.requestFrom .  The memory location pointer inside the sensor increments forward automatically from the initial memory register address for each new byte that it sends.

These simple patterns are at the heart of every I2C transaction, and since they are used so frequently, they often get bundled into their own functions:


byte i2c_readRegisterByte (uint8_t deviceAddress, uint8_t registerAddress{
byte registerData;
Wire.beginTransmission(deviceAddress);              // set sensor target
Wire.write(registerAddress);                                     // set memory pointer
Wire.endTransmission();
// delete this comment – it was only needed for blog layout.   
Wire.requestFrom( deviceAddress,  1);     // request one byte
resisterData = Wire.read(); 
// you could add more data reads here if you request more than one byte
return registerData;           // the returned byte from this function is the content from registerAddress
}
// delete this comment – it was only needed to maintain blog layout
byte i2c_writeRegisterByte (uint8_t deviceAddress, uint8_t registerAddress, uint8_t newRegisterByte
 {
byte result;
Wire.beginTransmission(deviceAddress);
Wire.write(registerAddress);  
Wire.write(newRegisterByte); 
result = Wire.endTransmission(); // Wire.endTransmission(); returns 0 if write operation was successful
// delete this comment – it was only needed for blog layout.
//delay(5);  // optional:  some sensors need time to write the new data, but most do not. Check Datasheet.
if(result > 0)  
{ Serial.print(F(“FAIL in I2C register write! Error code: “));Serial.println(result); }
// delete this comment – it was only needed for blog layout. 
return result;    // the returned value from this function could be tested as shown above
//it’s a good idea to check the return from Wire.endTransmission() the first time you write to a sensor 
//if the first test is okay (result is 0), then I2C sensor coms are working and you don’t have to do extra tests

//NOTE: copy/pasting code from blogs/web pages is almost guaranteed to give you stray/302 errors because
//of hidden shift-space characters that layout editors insert. Look at the line your compiler identifies as
//faulty, delete all the spaces and/or retype it slowly and carefully ensuring you enter only ASCII characters.


Those two functions will let you control the majority of the I2C sensors on the market, provided you can figure out the correct pattern of bits to send from the datasheet. A common strategy for keeping track of the multi-bit combinations that you want to load into your sensor control registers is to declare them with #define statements at the beginning of your program, which replace the human readable labels with the actual binary numbers at compile time.

For example the ADXL345 can range from 3 samples per second to 1600 samples per second, depending on four bits in the ADXL345_BW_RATE register. A set of define statements to represent those bit combinations might look like:

byte ADXL345_Address=0x53;     // the sensors i2c bus address (as a hex number)
byte ADXL345_BW_RATE=0x2c;    // the memory register address
#define ADXL345_BW_1600  0b00001111
#define ADXL345_BW_800    0b00001110
#define ADXL345_BW_400    0b00001101
#define ADXL345_BW_200    0b00001100
#define ADXL345_BW_100    0b00001011
#define ADXL345_BW_50      0b00001010
#define ADXL345_BW_25      0b00001001
#define ADXL345_BW_12      0b00001000
#define ADXL345_BW_6        0b00000111
#define ADXL345_BW_3        0b00000110
etc…. Note that all of these combinations assume normal power mode (bit4=0)

So a command to set the sampling rate to 50 Hz could be written as:

i2c_writeRegisterByte(ADXL345_Address, ADXL345_BW_RATE, ADXL345_BW_50);

 The cool thing about using defines is that they do not use any ram memory like byte variables would. And you can usually find code examples on gitHub where someone has transcribed the entire register address list into a set of defines, which you can simply copy and paste into your own code. This saves you a great deal of time, though there’s always the chance they made a transcription error somewhere. Also note that typical datasheets & ‘c’ language examples express those numbers as hex “0x0F” instead of “0b00001111” and you can leave them in that format if you wish.

Writing a whole byte to a register is pretty straightforward, but it gets more complicated when you need to change only one of the bit-switches inside a control register. Then the standard approach is to first read out the register’s current settings, do some bit-math on that byte to affect only bit(s) you want to change, and then write that altered byte back into register’s memory location.

But bit-math syntax is one of those “devils in the details” that makes relatively simple code unreadable by beginners. The bit operators you absolutely must be familiar with to understand sensor scripts you find on the web are: the bitwise OR operator [|] , the bitwise AND operator [&], the left shift [<<] and the right shift [>>] operators.  Fortunately there is an excellent explanation of how they work over at the Arduino playground, with a set of bit-math recipes in the quick reference section that let you reach into a byte of data and affect one bit at a time.  Be sure to parenthesize everything when using bitwise operators because the order of operations can be counter-intuitive, and don’t worry if you have to look up the combinations every time because most people forget those details once they have their code working. I know I do. 

Two particularly useful procedures:

x &= ~(1 << n);   // AND inverse (~) forces nth bit of x to be 0. All other bits left alone
x |= (1 << n);       // OR forces nth bit of x to be 1.  All other bits left alone

And these let us add a third function to the standard set which will turn on or turn off one single bit switch in a sensors control register:

byte i2c_setRegisterBit ( uint8_t deviceAddress,  uint8_t registerAddress,  uint8_t bitPosition, bool state )  { 
 byte registerByte, result;
registerByte = i2c_readRegisterByte ( deviceAddress,  registerAddress ); // load the current register byte
// delete this comment – it was only needed to maintain blog layout
if (state) {   // when state = 1
  registerByte |= (1 << bitPosition);   //bitPosition of registerByte now = 1
//or use bitSet(registerByte, bitPosition); 
  }  
else {           // when state = 0
   registerByte &= ~(1 << bitPosition);   // bitPosition now = 0
//or use bitClear(registerByte, bitPosition); 
  }
// now we load that altered byte back into the register we got it from:
result = i2c_writeRegisterByte ( deviceAddress,  registerAddress,  registerByte );
return result;   // result =0 if the byte was successfully written to the register


The ADXL345 accelerometer supports low power modes that use about 1/3 less power than the ‘standard’ operating modes.  The sensor is not turned off, but the bandwidth is reduced somewhat, so the sensor responds more slowly to things like tap inputs.
An example which sets the single bit enabling this low power mode might look like:

i2c_setRegisterBit( ADXL345_ADDRESS,  ADXL345_BW_RATE,  5,  );

Many I2C sensors have power saving features like that which rarely get utilized. Note that bit position numbering starts with 0 and counts from the left OR the right hand side depending on the sensor manufacturer. 

Some devices have control registers that are 16-bits wide. These get treated as a pair of 8-bit bytes that are read-from or written-to sequentially. You only have to specify the device & register address once at the beginning of the process because the sensors internal memory pointers get incremented automatically during the process.

This adds an extra wire.write step to the basic register writing operation:

Wire.beginTransmission(deviceAddress);
Wire.write(registerAddress);
Wire.write(MSB_registerData);    // Send the “upper” or most significant bits
Wire.write(LSB_registerData);     // Send the “lower” or least significant bits
Wire.endTransmission();

The MCP9808 is a common temperature sensor that uses 16-bit control registers.  Setting “bit 8” of the CONFIG register to 1 puts the sensor into shut down mode between readings and setting that bit to 0 starts the sensor up again. (yes, that’s opposite to the usual on/off pattern…)  The 8-bit limitation of the I2C bus forces us to retrieve the register in two separate bytes, so bit 8 of the 16 bits described in the datasheet ends up in the zero bit position of MSB. 

A custom function shutting down the MCP9808 might look like this:  

#define MCP9808_i2cAddress          0x18    // defines in setup are an alternative to using variables
#define MCP9808_REG_CONFIG   0x01    // the compiler swaps the text-name for the # at compile time
// delete this comment – it was only needed to maintain blog layout
void mcp9808shutdown()      //since we used defines, we did not pass any byte variables into the function

 byte MSB, LSB;
 Wire.beginTransmission(MCP9808_i2cAddress);
 Wire.write(MCP9808_REG_CONFIG);
 Wire.endTransmission();
// delete this comment – it was only needed to maintain blog layout
 Wire.requestFrom(MCP9808_i2cAddress, 2); //request the two bytes
 MSB = Wire.read();       // upper 8 bits described in data sheet as 15-8
 LSB = Wire.read();        // lower 8 bits described as 7-0 in the datasheet
// delete this comment – it was only needed to maintain blog layout
 MSB |= (1 << 0); // bitmath forces MSB bit0 (which is ‘bit8’ in the datasheet) to value one
 // using MSB &= ~(1 << 0); here would start the sensor up again by forcing the bit to zero
// delete this comment – it was only needed to maintain blog layout
 Wire.beginTransmission(MCP9808_I2cAddress);  // now write those bytes back into the register
 Wire.write(MCP9808_REG_CONFIG);
 Wire.write(MSB);                          // the one we modified
 Wire.write(LSB);                           // unchanged
 Wire.endTransmission();
}


This ‘position x becomes position y’ translation is common stumbling block for beginners working with multi-byte registers – especially when you add reverse order position numbering into the mix.  But there’s another gotcha with 
control registers that’s even more frustrating if you don’t catch it on your first pass through the datasheet:  Sometimes there are special “write protection” registers that have to be set before you can change any of the other control registers, and these have to be changed back to their “protecting” state before those new settings take effect. You might not get any error messages, but nothing will work the way it should until you get the protection bits disabled and re-enabled in the right sequence. Fortunately less than 20% of the sensors I’ve worked with have this  feature.

Another thing to watch out for are old code examples on the web that appear to be using integer variables to store device and memory register locations, with statements like Wire.send((int)(eepromaddress >> 8));  The I2C wire library only sends bytes/uint8_ts, but people got away with this (int) cast  because it was being corrected behind the scenes by the library, which re-cast the value into a byte at compile time.  The (byte) data type on Arduino is interchangeable with the (uint8_t) variables you see in most C++ coding tutorials. 

2) Data registers

Unlike a control registers bank-of-switches, I think of data output registers as containers holding numbers which just happen to be stored in binary form. Since eight bits can only hold decimal system values from 0 to 255 you usually have to “re-assemble” larger sensor output values from bytes stored in consecutive memory locations. For sensors like the ADXL345 you can concatenate the two 8-bit bytes into one 16-bit integer variable by shifting the MSB left by 8 positions and merging in the LSB with a bitwise OR :

Wire.beginTransmission(deviceAddressByte);  // the pointer setting transaction
Wire.write(registerAddressByte);
Wire.endTransmission();

Wire.requestFrom(deviceAddressByte,2);       // request two bytes
LSB = Wire.read();                                                // byte from registerAddressByte
MSB = Wire.read();                                              // byte from registerAddressByte +1
int combined = (int)MSB;             // MSB now in rightmost 8 bits of combined int
combined = combined<<8;          // shift those bits to the left by 8 positions
combined |= LSB;     // logical OR keeps upper bits intact and fills in rightmost 8 bits

Those steps are usually written in one single line as:

int combined = (((int)MSB) << 8) | LSB;

There are several other ways to combine bytes and some sensors send the MSB first – so you have to check the register map in the datasheet to know the order of the bytes that arrive from the output registers when you request multiple.

Now if you are thinking that looked too easy – you’re right! Most hobby market I2C sensors only have a 12-bit ADC, and since memory is a limited resource there are often status register bits mixed in with the data held in the MSB. Since these bits are not part of the sensor reading, you need to &-mask them away before you combine the MSB & LSB. It gets trickier when the sensor output can be a positive or a negative number because signed and unsigned integers are distinguished from each other by a special “sign” indicator bit, which can accidentally be turned into a “number” bit by bit shifting. (see: ‘sign extension’ in that bit math tutorial )

The temperature data output register in the MCP9808 is a good example of both of these issues:

Bits 15-13 (which become the top 3 bits of the upperByte in the code below) are status indicator flags identifying when high & low temp. alarm thresholds have been crossed. Bit 12 is a sign bit (0 for +ve temperature or 1 or -ve temps). The remaining bits 11-8 (=bits 3-0 of the upperByte) are the most significant 4-bits of the 12-bit integer representing the temperature.

So a sensor-specific approach to reading the temp. from an MCP9808 might look like this:

int TEMP_Raw;
float TEMP_degC; 

// spacer comment for blog layout
Wire.beginTransmission(0x18);    // with mcp9808 bus address written in hex
Wire.write(0x05);                             // and the temperature output register
Wire.endTransmission(); 
Wire.requestFrom(0x18, 2); 
byte UpperByte = Wire.read();          // and sometimes the MSB is called the “highByte” 
byte LowerByte = Wire.read();          // sometimes called the “lowByte” 
// spacer comment for blog layout
UpperByte = UpperByte & 0b00011111;  // Mask away the three flag bits
//easier to read when the mask is written in binary instead of hex
// spacer comment for blog layout
//now we use a mask in a slightly different way to check the value of the sign bit:
if ((UpperByte & 0b00010000) == 0b00010000)  {          // if sign bit =1 then temp < 0°C
UpperByte = UpperByte & 0b00001111;                             // mask away the SIGN bit
TEMP_Raw = (((int)UpperByte) << 8) | LowerByte;    // combine the MSB & LSB
TEMP_Raw-= 256;   // convert to negative value: note suggested datasheet calculation has an error!
 }
else  // temp > 0°C  then the sign bit = 0  – so no need to mask it away
 {
TEMP_Raw= (((int)UpperByte) << 8) | LowerByte;
 }
// spacer comment for blog layout
TEMP_degC =TEMP_Raw*0.0625;


Typically a data output register will continue to hold the last sensor reading until it is refilled with the next one. If your sensor takes a long time to generate this new reading (30-250 ms is typical, while some can take up to a second) and you read the registers before the new data is ready, you can end up loading the previous sensor reading by mistake. That’s where status registers come to the rescue.

3) Status registers

These tell you if if a specified type of event has occurred and I think of these registers as a set of YES/NO answers to eight different questions. The most commonly used status register is data ready [usually labeled DRDY] which sets a bit to 1=true when a new sensor reading is available to be read from the related output registers. Another common status register is one that becomes true if a sensor reading has passed some sort of threshold (like a low temperature alert, or a falling/tilt-angle warning).

A function to check the true=1/false=0 state of a single DRDY bit inside an 8-bit status register might look like this: 

bool i2c_getRegisterBit (uint8_t  deviceAddress, uint8_t  registerAddress, uint8_t  bitPosition) {     
byte registerByte;
registerByte = i2c_readRegisterByte(deviceAddress, registerAddress);
 return ((registerByte >> bitPosition) & 0b00000001);  // or use (bitRead(registerByte, bitPosition))
 }
// delete this comment – it was only needed to maintain blog layout
 //  You could use i2c_getRegisterBit to check the DRDY status bit with a do-while loop
//  and only move on to reading the sensor’s data output registers after the DRDY bit changes to 1
// delete this comment – it was only needed to maintain blog layout
bool dataReady=0;
do {
dataReady= i2c_getRegisterBit(deviceAddress, statusRegAddress, DRDYbitPosition);  
} while ( dataReady==0 );        // processor gets cycled back through this loop until DRDY=1


Holding the processor captive in a status-bit-reading loop is very easy to do, but it is usually avoided unless you are trying to capture a series of sensor readings quickly.  Most status register bits can be mapped to physical alarm output lines on the sensor module, and these can be used to trigger a hardware interrupt pin (D2 & D3) on the Arduino.  This lets you to setup an interrupt service routine (ISR) which grabs that new reading even faster than a bit reading loop. And since hardware interrupts can be used wake a sleeping processor, the interrupt method also lets you put your data logger to sleep until something actually happens. 

The only drawback to the ISR method is that the sequence of register settings you need to follow to create hardware alarms is another multi-step process to add to your sensor initialization code.  The conceptual pattern is usually something like:

  1. Disable the sensor’s global interrupt control bit (if there is one)
  2. Enable the sensors triggering function   (eg:  a max. temperature alert)
  3. Load register(s) with the parameter value for that trigger (eg:  52.5°C)
  4. Turn on the status register that listens to that triggering function
  5. Map that status register bit to a hardware output line
  6. Re-enable the global interrupt control bit

This LSM303 combined accelerometer / magnetometer sensor has two alarm outputs in addition to DRDY. So you could map the Accelerometers DRDY signal to int1, and the Magnetometers DRDY to DRDY.  Just to make life interesting with this sensor, the 3-axis output data  registers are arranged in a different order  on the magnetometer than  they are on the accleerometer. This is typical for multi-sensor chips, which you handle like separate sensors even if they come in the same package – you can even put one to sleep mode while the other one is taking a reading.

Sensors can have many different status monitoring functions, but they usually have only one or two hardware alarm lines.  So the status register -> hardware output mapping (step 5) listed above sometimes involves its own sequence of register settings.  As example, the ADXL345 reads acceleration on three axes, and it has double-tap detection functions for each x,y,z direction. But the Arduino only has two incoming hardware interrupt lines. So generally speaking, you would map all three of those tap-detect status registers to the same outgoing alarm line on the sensor module, and then have the program figure out which axis actually triggered the alarm by reading the status registers later on. High & Low temperature sensor alerts are often mapped in a similar fashion because many breakouts only have one outgoing line: especially if the DRDY status register has been permanently connected to the only other physical alarm line.

A conceptual twist here is that most of the time, the hardware output actually moves the line LOW when the alarm is triggered, even if the status bit it’s mapped from is true=1=high when the actual event occurs. No matter what the status bit->alarm pattern is, any of the four possible interrupt triggers: HIGH, LOW, RISING & FALLING can be used to wake a sleeping 328p processor (though the datasheet states differently).  

Another thing to watch out for on the Arduino side is setting your ISR to respond to HIGH/LOW levels rather than RISING/FALLING edges: Level based interrupts will keep triggering as long as that line is HIGH/LOW. This could cause a sketch to run extremely slowly until the interrupt handler is disabled in your program. Even old analog reed-switched based sensors can suffer from this type of issue, as its not uncommon for something like a wind sensor to stop spinning right where the magnet is holding the reed-switch closed.  The thing that makes this choice somewhat tricky is that the most common type of sensor failure I see is one where the alarm stays on permanently.  If you set your interrupt to respond to LOW,  and the sensors starts self-triggering your event counters get pushed up to ridiculously large numbers – so it’s very easy to spot that failure in the data, and by the fact that the logger is usually kept awake till the batteries run dry.  If your ISR responds to FALLING, your counts go to almost zero in the same situation, and depending on the phenomenon you are recording it could be very easy to miss that a sensor problem has developed.  

For more information, there’s an excellent guide to interrupt handling over at the Gammon Forum. Probably the most important thing to keep in mind about using interrupts is that by default all interrupts are disabled once you are inside an interrupt subroutine so that the ISR can’t interrupt itself and create an infinite-recursion situation that over-runs the memory.  But the I2C bus relies on interrupts to function, along with timers and other important things.  So don’t try to change a sensor register while inside the ISR,  just set a volatile flag variable and deal with resetting registers later in the main loop.  The general rule of thumb is: “get in & get out ” as fast as possible, and I rarely have a sensor triggered ISR longer than this:

void  INT1pinD3_triggered()  {   INT1_Flag = true;   }

though sometimes I’ll also detachInterrupt(interrupt#) inside the ISR, to make sure it only fires once for things like button de-bouncing. 

Status registers are usually latched, and have to be reset by the I2C master after they are triggered. DRDY registers are cleared by reading information from the data registers they are associated with.  Most other status registers are cleared by reading the register’s memory location, which also turns off the hardware alarm signals that were mapped from them.  This is different from control registers which always have to be explicitly over-written to with new information to change them. If you are waking up a sleeping data logger based on something like a high temperature alert, you usually read the status registers to clear those alarms before enabling interrupts and putting your logger into a power-down state. Threshold based alarms allow interesting things like burst logging.

In Summary:

A good register map, and the four generic functions I’ve described here

  1. i2c_readRegisterByte
  2. i2c_writeRegisterByte
  3. i2c_setRegisterBit
  4. i2c_getRegisterBit

Should be enough to get a typical I2C sensor running, and you can easily tweak those functions to make custom versions for reading 16-bit registers and/or to mask the cruft out of data pulled from mixed registers.

After testing an I2C sensor combination, I pot them in epoxy. Detailed instructions here.

Initializing an I2C sensor is a multi-step process and the correct order of operations is often poorly explained in the data sheet because they are usually written “in reverse”.  Instead of a straightforward list saying “To get a reading from this sensor, do (1),(2),(3),(4), etc.” you find descriptions of the control register bits saying “before you set bit x in this register you must set bit y in this other control register”. When you look up that other control register you find that it too contains a sentence at the end saying “before you set bit y in this control register you must set bit z in this other control register”. So you have to work your way through the document, tracing all those links back until you find the things you were supposed to do first.  Finding the “prime control bit” can be such a time consuming process that it’s not unusual for people who figure out the sequence to wrap it all up into a sensor library so they never have to look at that damn datasheet ever again.

But if you use those libraries, keep in mind that they are probably going to configure your sensor to run at the highest possible bit-depth & data rate, unnecessarily burning away power in applications like data logging which might only need one reading every fifteen minutes.  So the majority of off-the-shelf sensor libraries should be seen as partial solutions, and you don’t really know what else your sensor is capable of until you read through the datasheet yourself.  As an example there are IMU’s out there that will do Euler angle calculations if you simply turn on those functions with the right control register. But libraries for those chips sometimes enable the bare minimum data output functionality, and then do computational handstands to accomplish those gnarly (long) calculations on the Arduino’s modest µC.

In addition there can be useful sensor functions hidden in plain sight, because the datasheet tells you how to turn them on & off, but gives you no clue when to do so. An example here would be humidity sensors like the HTU21D which has an on-chip heating element to help the sensor recover from long periods of condensation, but no status alert that would let you do this automatically. You could just run the heater once a day, but there is also no indication how long the sensor would last if you did that – just some vague references to “functionality diagnosis”. But then some manufacturers (Freescale and Sensirion come to mind…) commit more than just sins-of-omission, breaking away with non-standard I2C bus implementations to lock in customers. The logic there is that if you have to buy the one great sensor that only they make, it’s easier to buy the other four sensors for your device from them as well, rather than juggling low-level protocol conflicts. 

Another challenge when you are working with a new sensor is that Arduino’s C++ environment is not the same as vanilla C in some important ways. So many of the tutorials you find will describe methods that won’t work on an Arduino. Even when the code does compile, there are a number of different “styles” that are functionally identical when they pop out the other side of the compiler, so I’m still trying to wrap my head around the syntax that turns arrays into pointers when they get passed into functionsThat’s why I didn’t mention I2C eeproms in a post about memory registers: almost every multi-byte read/write example out there for EEprom’s uses array/pointer/reference thingies. If you absolutely have to read a series of sensor output registers into an array with a loop, my advice is to just make it a global until you really know what you are doing. And don’t try to store numbers in a char array, because the “temporary promotion” of int8_t’s to 16-bit during some operations can bung up the calculations.

But now it’s time to bring this thing to a close. While I’m still thinking about stuff I wish I’d known earlier, it occurs that a good follow-on to this post would be one about techniques for post-processing sensor data.  There are plenty of useful methods like Paul Badger’s digital smooth, and other code tricks like wrapping those functions in #ifdef #endif statements so those routines only get compiled when a sensor that actually needs them is connected to your logger.

That will have to wait for another day so for now I’ll just sign off with some links. Except for that last ranty bit, I’ve tried to stay out of the I2C handshaking weeds, because when you are up to your neck in bit banging, it’s easy to forget you were trying to measure the water level in a swamp.  But if that’s your thing, there’s some more advanced I2C code examples over at the Gammon Forum, an in depth reference to the Wire library at the Arduino playground , and some troubleshooting tips over at Hackaday.  Its also worth noting that I’ve used bit-shifting to extract bits, and concatenate 16-bit values from 8-bit registers. But you sometimes run into examples where people have uses structs & unions to do those tasks in a much more elegant way.

Addendum 2017-11-04

I wonder how many other sensors I could use this with? And if my pin-toggled oversampling method works on the ATtiny, this might provide better resolution than some commercial sensors; though I guess that would depend on how much data I could squeeze into 512 bytes of SRAM…

Somehow I always seem to run into a bunch of related material the day after I post something to this blog: There’s a cool little project over at Quad Me Up using ATtiny85 to turn an analog light sensor into an I2C slave device.  AN4418 from Maxim explains how to use I/O extenders to connect a compact-flash (CF) cards to the I2C interface, which is something I never thought I’d see. And then theres AN10658 from NXP with a method for sending I2C-bus signals over 100m. My own tests with the I2C sensors just hanging off the Arduino only reached about 20m.

 

Addendum 2017-11-05

Koepel over at the Arduino forum pointed out that the IDE supports some handy macros like bitSet(), bitClear(), and bitRead() that could replace the bit math & masking functions I described here. There’s also word(h , l) to combine two bytes, or highByte() and lowByte() to divide 16-bit variables into 8-bit two pieces. Those were new to me, so I thought I should list them here in case people run into sensor scripts using them.

I’ve also just found out that there are a small number of sensors there that require a ‘false’ modifier to be used at the end of an I2C transaction:  Wire.endTransmission(false);   This is called a repeated start, and the I2C master does not release the bus between writing the register-address and reading data with Wire.requestFrom();   The sensor responds to the I2C address with an acknowledge at the begin of the I2C transaction, and to each databyte that is written to the sensor, so the error code returned by endTransmission can still be used because it is a test if the I2C address was acknowledged by the sensor.

And there was another I2C quirk mentioned at the Gammon Forum:

“You can’t rely on the slave necessarily sending the number of bytes you requested. If you request 10 bytes, Wire.requestFrom() will return 10, even if only 5 have been sent. For the slave to terminate the communication early (ie. after sending less than the requested number of bytes) it would have to be able to raise a “stop condition”. Only the master can do that, as the master controls the I2C clock. Thus, if the slave stops sending before the requested number of bytes have been sent, the pull-up resistors pull SDA high, and the master simply receives one or more 0xFF bytes as the response…It does not help to use Wire.available() because that too will return the number of bytes requested.”

Addendum 2017-11-08

On my page about the DS3231 rtc I describe how to power that I2C chip from a digital pin during bus communication. That trick only works because the chip was designed to gracefully fail over to a backup coin-cell power supply. With other I2C sensors a leakage current might flow into the sensor through the pullup resistors, so you would have to power the bus pullups with the same digital pin to avoid this. And since the internal pullup resistors are enabled by default in the Wire library, you have to disable I2C before you could pin power that I2C device.  Also don’t try to de-power a whole module with decoupling capacitors through a digital output pin as that creates big current spikes and really needs proper switching with a PNP transistor of p-channel FET.  99.99% of the time its better to simply find a sensor with a really low sleep current sleep state that you can enter by setting a control register. The best sensors are ones that automatically go into these low current standby-states whenever they detect no traffic on the I2C bus: then you don’t have to do anything.

Another thing I discovered while working with that RTC was that it had a Wire.begin() call hidden in the library, but I was already starting the I2C bus normally during setup. So without knowing it the I2C hardware was being initialized a second time. As the I2C peripheral registers are set to the same value as in the first Wire.begin() call nothing bad happened. However I can see where it might get’s problematic if you call Wire.begin() accidentally because it was buried inside some sensor library while you were running a data transfer,  and the hardware is re-set to an idle state. 

Addendum 2017-11-09

Most of us are familiar with trying out different libraries to drive a sensor, but I’d be remiss if I didn’t mention that there are also some alternatives to Wire library for I2C. The one that gets the best reviews is the I2c Master Library developed by Wayne Truchsess at DSSCircuits. This lib has a faster throughput and a significantly smaller code size: the original Wire library adds about 796 bytes to your sketch when included whereas Wayne’s I2C library only adds 140 bytes. And it has built-in commands that replicate all of the functions I described in this post. For 16 bit registers Wayne points out:

“Technically when sending bytes to a slave device there is no difference between data and an address. In other words let’s say you have a three byte address and three bytes of data. You could use the write(address, regaddress, *data) by making the first byte of your multibyte address equal to regaddress and then combine the rest of the address and data together into *data.”

and that’s equally true with the wire library. The memory savings alone would be worth exploring, but perhaps I2C Master library’s most compelling feature is a ‘TimeOut’ parameter for all bus coms, which could keep your logger from getting stuck in a while-loop if one of your sensors goes AWOL, though I wonder if it still has the 0xFF problem mentioned above, if the sensor sends less than you requested?

Addendum 2017-11-10

I thought using an ATtiny to convert an analog sensor into an I2C device was a neat trick. But it seems that Andreas Spiess has taken the idea to a new level with three HC-SR04’saccessible through on a single AT.  His youTube video #42 with three Ultrasonic Sensors for Arduino walks through the process, with a vocal track that leaves you thinking Werner Herzog has started doing maker videos…

Addendum 2017-11-13

The IDE compiler has an annoying quirk when it runs into Wire.requestFrom in those I2C register routines because the compiler throws up warning messages whenever it feels it has to resolve an ‘ambiguous’ call:  (click to enlarge)

 Turns out that requestFrom has two different implementations, one that can take int arguments, and one that takes uint8_t arguments. If you put in something which has no type like a number (or something you declared with a #define) the compiler has to decide which implementation to use. In the case shown above it chose to use the (int,int) flavor even though device address was specified as uint_8 at the start of the function. 

Anyway, to make those warnings disappear, simply cast the two parameters in Wire.requestFrom to either (uint_8) or (int):

And all those compiler warnings will disappear.

Addendum 2017-12-14

Single I2C/SMBus Address Translator for those times when you have an unavoidable sensor bus address conflict. Or you can use an I2C multiplexer like the TCA9548A over at Adafruit which will let you use one I2C address to talk to the multiplexer and tell it which lane you want to enable.

 

 

 

Measuring Electrical Conductivity with Arduino

This post is a summary of my background research into electrical conductivity (EC) to serve as a backdrop for my own humble attempts at this interesting measurement challenge. I’m sure there are many other approaches that I’ve yet to discover, and if you know of one please leave a comment so that we can pass that knowledge on to others – Ed.

Obligatory blog-post monkey shot.

Pete & Trish doing profiles with a YSI EXO. As you might imagine, these puppies are pretty expensive. Now that we have A Flexible Arduino-Based Logging Platform to build on,  adding conductivity is our #1 priority.

The conduction of current through a water solution is primarily dependent on the concentration of dissolved ionic substances such as salt. Since most fresh water derives from relatively clean rainfall, variations in EC provide a way to track the chemical  and hydrological processes the water has been subjected to over time. High amounts of dissolved substances (usually referred to as salinity) can prevent the use of waters for irrigation and drinking, so conductivity ranks as one of the most important inorganic water quality parameters.

A huge number of resources are dedicated to measuring EC and rather than re-hashing all that material, I thought I’d start with links to a few good background reads:

Conductivity, Salinity & Total Dissolved Solids
-discusses the older TDS measurements in part per million (ppm) which makes assumptions about the charge carriers that don’t reflect real world environments.  The conversion factor from EC (which is the thing you actually measure) to TDS changes for different dissolved solids, so instruments from different manufacturers often give you different TDS readings for the same solution, because the companies made different assumptions about what’s in your water.  Because of this confusion, straight EC measurements in siemens have been adopted as the standard by the international scientific community. One siemens is equal to the reciprocal of one ohm (S = 1/Ω)  and is also sometimes also referred to as the mho (℧) in older literature.

Conductivity Theory & Practice
-a white paper that covers basic probe designs, and mentions some non intuitive things like geometry/field effect errors.

Conductivity Sensing at PublicLab.org
-many groups at PublicLab.com have been working on different types of conductivity sensors and their overview page is another excellent introduction to DIY approaches. In fact it’s so good that I will be referring to several of those projects in this post.

Aqueous conductivity is commonly expressed in millisiemens/cm (mS/cm) and natural waters range from 0.05-1.5 mS/cm for freshwater lakes & streams up to about 55mS/cm for sea water. Water up to 3 mS/cm can be consumed, though most drinking/tap water is below 0.8 mS/cm.  Many of the Cave Pearl Project’s installations are in coastal areas where tidally driven haloclines require our instruments to cover that entire “natural waters” range.  Groundwater can vary even more, with measurements being complicated by organic acids and/or significant amounts of dissolved limestone.  Salt water is chemically aggressive and water hydrolyzes above 0.4v, so the probes for high-conductivity environments are usually made of resistant materials such as platinum, titanium, gold-plated nickel or graphite, making them somewhat expensive.

Ways to Measure Conductivity:

There are so many different approaches to measuring EC that it’s taken me a while to digest it all into some working categories.  I expect to build at least one prototype for each of these methods just to see if I can make it work.

Density Based Methods

Refractometers and density based hydrometers are used by aquarium hobbyists. Better quality acoustic doppler flow sensors can also calculate density based on the speed of sound through the water and infer salinity from that. Given then number of acoustic anemometer projects out there, I’m surprised someone has not already adapted the method for underwater applications, though this may be due to the timing limits of the affordable transducers.

Resistance Based methods:

a) Use submerged probes as part of a resistor divider / bridge :
This common approach measures the resistance between two probes using some type of voltage divider. Resistance =1/conductance, which allows you to derive conductivity with your cell K constant since conductivity=(conductance * length)/(area).  AC oscillators are tacked on to reduce electrode polarization, and this forces you to add even more electronics on the output side to convert the signal back to DC for reading. The resistance between the probes changes by several orders of magnitude in environmental waters so different probe surface areas & divider resistors are usually required to cover a significant conductivity range. Above 50% sea water, the resistance between the probes doesn’t change very much, so this method tends to get used more frequently for fresh water environments.

b) Change the pulse frequency of a 555 timer circuit:
You can use resistance between the electrodes as part of an RC relaxation oscillator and then measure the 555’s square wave output frequency to determine the resistance.

This circuit from Thomas Allen’s site provides galvanic isolation, uses AC measurement, and the output frequency varies from about 42 Hz with the probes in air to > 8000 Hz depending on conductivity. You can buy this circuit on pre-made a module for $24  from the EME site.

Circuits and instructions can also be found at at PublicLab.org and there are many good tutorial videos describing 555 based EC sensors on YouTube.  At this point I’ve run into so many projects using this chip that I’d  be willing to bet every environmental sensor I’ve ever heard of could be cobbled together from a few op-amps and a low voltage 555 timer. There are several frequency counting libraries available to help you get started, and if you are ready to sink your teeth into some code, Nick Gammon has produced the some elegant solutions for pulse/frequency timing. Note there are some duty cycle issues

c) Time the discharge of a capacitor through the solution:
Jim Conner’s describes this method in his youTube video at
EC Probes – How they work, and how to build one


A circuit like this might be easy to implement on an Arduino if you can put the internal 1.1v reference onto the comparator that’s also built-in to the 328.  Microcontrollers count time with far better resolution than you get from their ADC’s, but that doesn’t mean there aren’t other issues to deal with. Given that you can try this method with practically no extra circuitry, I will definitely be prototyping a few of these.  Like the 555 based circuits, it will be interesting to see if the method bumps into timing & interrupt handling limits (100 kHz?) when you use it with seawater.

Capacitance based approaches:

You often see capacitance used for liquid level sensors and soil moisture probes, and some of these could be adapted for EC.  To me, the raindrop detection pcb’s you see on eBay have always looked like prime candidates for re-purposing as capacitive sensors.

The circuit described for the Chirp Moisture Sensor uses a fixed resistor and a non-contact probe to create a low pass filter whose cutoff frequency changes with capacitance,  which is affected by the electrolytes in solution. This filters an 1-8MHz square wave and the output voltage is accumulated other side of simple diode peak detector circuit for reading. 

You can also vary 555 timer output frequency by changing the capacitor in the tank circuit, or create more complicated oscillator circuits. No matter which cap-based method you use, the supporting electronics have to be located near the sensor – because just about any length of wire will add enough stray capacitance to throw off your measurements. Another thing to keep in mind is that common ceramic capacitors have some of the worst thermal coefficients and aging effects imaginable. Plastic film capacitors using Polyphenylene Sulfide (PPS ±1.5%) or Polypropylene (CBB or PP ±2.5%) are much better for sensor circuits like this, and having a digital capacitance meter on hand is probably a good idea too.

DIY capacitor plate sensors are going to be small values (picofarads ?) and the resulting RC time constants make these methods more suitable for fresh waters, as this leverages the relatively large solution resistance to give you more accurate interval timing.

Potentiometric (4 electrode) Methods

Four-electrode cells uses two “driver” pins to place an electric field across two other “reading” pins that lie between them: 

This paper describes a DIY 4-probe sensor that was used for soil moisture sensing, and you will find quite a few articles using potentiometric methods over at IEEE and Sensors

Nokia/Apple audio jacks came to mind as soon as I saw this diagram, and they might be available with gold plating.  4-electrode methods often measure the voltage between the read pins, which is divided by the exciter pin current to determine the solution’s impedance = 1/conductance.  To obtain the conductivity, the conductance is multiplied by the cell constant of the inner poles. Tracking the pin current lets you compensate for fouling on the plates, and the method can cover a wide range of conductivity. Like inductive methods, this approach tends to work better as the concentration increases. 

Inductive Methods

The conductivity measurement is made by passing an AC current through a toroidal drive coil, which induces a current in the solution. This induced solution current, in turn, induces a current in a second coil, called the pick-up toroid. The amount of current induced in the pick-up toroid is proportional to the solution conductivity. You get industrial grade performance out of this non-contact method in many different types of solutions, but you also need industrial amounts of power the drive the sender coil so it’s hard to implement on the kind of power constraints you see on stand-alone data loggers.  Inductive sensors require a 3 inch radius from any other surface (bio-fouling?) and you see this pretty clearly in the ‘donut on a stick’ sensor heads. It occurs to me that you see very similar components in a wireless charging system, but there’s a lot of devils hiding in those details – like shielding, etc.  It might be possible to press one of the production line proximity sensor chips into service for a low power solution, or simply try measuring changes in inductance due to the presence of salt water.

Off-the-shelf Solutions for Arduino:   (using 2-Electrode Resistance Methods)

TransparentSinglePixl
Atlas Scientific Conductivity Kit
A complete solution including calibration solutions, a range of probes and code libraries. All parts also sold separately: interface boards are ~$35 & EC probes come in around $120 each but they are durable enough for continuous long term submersion.  I2C data transfer is supported, so resolution is not limited to the Arduino’s ADC.  Whitebox labs Tentacle Shields ($35-$110) provide up to four galvanically isolated channels for full hydroponic rigs. Stand alone BNC carriers for $10.
$200
Gravity: EC Sensor Kit for Arduino (K=1)
Another complete K=1 kit solution, but the probes are not robust enough for long term submersion so several people replace the stock probe with the 208DH which is available on eBay for $35. Arduino ADC reads voltage.  The KnowFlow project uses the full set of DFrobot boards. DFR also has an inexpensive TDS kit, which cfastie has been testing over at publiclab.org
$70
Vernier CON-BTA EC probe
This 5v K=1 probe covers 0-20,000 μS/cm  in the high range, and the analog voltage output is read by the Arduino ADC. You need an inexpensive adapter board for the BTC connector, and they provide a basic library. One key feature is built in hardware temperature compensation with a 10k thermistor in the probe head. My tests show this reduces the usual 2%/° C reading variation down to about 0.5%/° C, so you still need to do your own calibration to get high accuracy. Like Atlas Scientific, Vernier has many other interesting sensors that are Arduino compatible.
$115
EC/pH Transmitters
This company offers a range of physically bulky turn-key solutions, with the $70 entry level unit claiming 0-5000 μS/cm (fresh waters) and continuous monitoring. Arduino ADC reads voltage. ~$200 units support PH with isolation.
$70-250
Sparky’s widgets MiniEC
An indie who makes several other useful sensor breakout boards, including PH. You have to build or locate your own probes, though they use a standard BNC connector like most EC probes.  Arduino ADC reads voltage output.
$24
EC-Salinity Probe Interface by Ufire
Designed around an ATtiny configured as an I2C slave, probably using the cap-discharge method.
$14.50
Hanna HI 73311 (K=1) Replacement probes
In the past we’ve used used these epoxy&graphite probes from Hanna DIST5 (HI 98311) and DIST6 (HI 98312) testers, which connect to a standard male audio jack.  You can also re-purpose one of the Vernier ABS/graphite probes if you get a used one cheap on eBay, and the Vernier probes have a 10k NTC thermistor built in, which you can read with a divider.
$55
Comercial Standard Solutions
For fieldwork, it’s often easier to transport the dry packets, and mix them on location.  Atlas sells calibration sets, but at the twice the cost of standards when you buy them in larger volumes.  You can find recipes for homemade calibration solutions at Reefnet Central and PublicLab. For a classroom situation, it’s much cheaper to mix secondary “lab standards” in larger quantities, and then test the resulting solutions with a commercial probe that’s been calibrated against commercial solutions. 5.566g of dry NaCl in 1 litre of distilled water will create an ~10,000 μS/cm solution, which you can dilute down for lower concentration standards.
$14/500ml

This photo from Bitnitting gives you a sense of the physical space needed for the Atlas breakouts and a ‘mini’ form factor Arduino.

Hydroponics hobbyists have putting these kits to good use over the years with notable examples like the long running forum thread on Billies Hydroponic Controller, and the well documented adventures over at the Bitnitting Blog.  The people at OpenCTD and other academic projects have put the Atlas boards into real world deployments.

But to me these commercial solutions still leave you stuck with those expensive electrodes which sometimes cost more money than you would pay for a used 4-pole device. More annoying is the fact that those cell constants do not line up with my goal of measuring the entire “fresh” to “marine” range with one sensor, thought if I could extend it a bit the K=10 probe comes close.  This is illustrated by the following graph from Andy Connelly’s Blog which is worth digging through as he has posted lots of other interesting material on calibration, reproducibility, signal detection, etc. 

 

Of course the cell constant changes as your probes get older and dirtier, so you have to re-calibrate them with standard solutions just about every time you want to take a new reading. I’m pretty sure I will end up making my own probes, probably out of Nichrome 80 wire as the vaping fad has made it common on eBay. Some have had good EC results with gold plated PCB traces. Feedback on the Arduino.cc forum suggests that Platinum-Rhodium Thermocouple Wire is another good option.  I’ve also been wondering about Ag/AgCl which is highly resistant to seawater and is commonly used for non-polarizing electrodes in medical/bio applications. (EKG electrodes?) It might also be a good idea to cobble together a DIY magnetic stirrer, based a PC fan and an old hard drive magnet

DIY 2-Probe EC Circuits

The easiest circuits to build yourself are the 555 timer oscillators, but there are plenty of quad-opamp solutions out there for people comfortable with a breadboard. The oldest example I’ve seen is this one by M. Ahmon from the Sept 1977 issue of Electronics magazine which uses the resistance of the solution to modify opamp output:

This circuit uses the first stage of the quad opamp in a Wien-bridge oscillator, reducing errors caused by electrolysis with a 1-kHz signal that gets attenuated by the solution’s resistance before it reaches the driving amplifier A2.  Pot P1 controls oscillator amplitude, and P2 adjusts gain of A2.  A3-A4 form a precision rectifier giving output voltage equal to absolute value of input voltage. This one chip solution seems to have been the basis for many of the current EC projects on the web, including these two exceptionally well documented examples:

Octavia’s EC/TDS/PPM Meter On Limited Budget

Daniel Kramnik’s Digital Salinometer Project

Similar circuits can be found on the breakout modules from Sparky’s Widgets and DFrobot . Using the solution’s resistance in the feedback divider controlling an op-amp is a neat idea, but having only one opamp there imposes hard limits the range you can measure with a given K value probe. There is a more advanced multi-opamp approach over at pulsar.be that can step over several decades.

On more recent EC projects I’m seeing single supply RRIO opamps for the oscillator & gain stages, which are easier to integrate with battery operated Arduino’s. (though any dual supply opamp can be used as a single supply in a pinch; since voltage is relative the opamp doesn’t know whether V- is a negative voltage or ground) To keep using an AC signal, this requires a virtual GND at 1/2 VCC, but the integration also gives you the option of getting rid of the oscillator entirely, since you can use PWM output as your source.

This is beautifully illustrated by the circuit from bhickman’s Conductivity & Temperature Meter over at PublicLab:

Ranging is accomplished with the (red) bank of R1 resistors, and (yellow)R2’s 5/6 can be substituted in for the probe (R8) with those known resistances can be used to track drift. The AC–DC converter stage is built with precision peak detectors. I think this is the best voltage divider approach I’ve seen to date.  To simplify things a bit, you might replace that output stage with an RMS-DC converter; though I’ve not seen any breakouts for those, and I hate working with raw SMD parts.

Sources of Error: 

Even with a clever circuit like the one above you still need to address things like temperature compensation before you get an accurate, repeatable, and stable device. Electrical conductivity measurements are typically referenced to 25 °C using standard temperature compensation factors (α). The conductivity of natural waters exhibit strongly nonlinear temperature behavior, though in practice linear correction factors are most frequently used.  NaCl-based solutions typically have a temperature coefficient (α) of 0.02-0.0214 (~ 2% change/degree C). So to convert your “ambient” conductivity measurement into 25°C “specific” conductivity, the simple linear conversion is:

EC25=ECambient /[ 1 + α (tambient – 25) ], α= 0.02

Field effect errors are significant, causing read errors if bare 2-pole electrode get within 2-5 cm of the solution container: which will completely mess up your calibration and cell constant determination. This is one reason that virtually every EC probe is encased inside a plastic shroud of some sort. That causes field effect errors too, but at least its the same error every the time, rather than one that varies depending on how far you are from the edge of the beaker. Four probe methods also require a fixed volume of solution between the driver electrodes, so the shroud provides that.

Grazing through the hydroponics forums shows plenty of people struggling with cross-sensor interference.  Most notably when conductivity probes affects the accuracy of a PH probe in the same tank.  Any time two devices are immersed in the same environment differences between them can generate ground loop voltages and induce currents which degrade the readings and exacerbate corrosion.  Sometimes you can address these issues with optical or I2C isolators. One helpful contributor at Arduino.cc suggests:

pH electrodes are very high impedance devices and the cabling and connectors are all important – even flexing a decent cable will distort the readings…. Ground loops are the enemy of pH and any other specific ion electrode. I used them a lot in difficult situations and the most trouble-free solution is always to put a buffer op amp (FET type) as close to the electrode as possible – some commercial electrodes come already equipped. Find a decent op amp like the old MAX406, high impedance techniques like PTFE insulators or simply keep the input pin off the board. Modern FET’s take single-sided supplies and run at better than 2-microamps – a 3.6-V lithium cell will give you in excess of 5-year’s trouble and ground loop -free operation. Once you have buffered the signal, you can use any cable you like. As a bonus, you can convert a pH electrode into an ammonia electrode by separating the water from the electrode with PTFE tape as used by plumbers.”

Well, I think that covers most of the stuff I had in my notes, and hopefully gathering it all here saves someone else from burning away that time. I have been experimenting with conductivity quite a bit lately, and I think I might have  come up with an analog approach that will allow people to play with conductivity on shoestring budgets. I just have a little more calibration testing to do before I let that one out of the bag  🙂