Wait…do I hear something?

The more I thought about it, the more the high power drain of the flow sensors got under my skin. How had I misjudged the performance that badly, after so many successful bench top tests? I was determined to get to the bottom of this mystery.  The simplicity of the “one sensor” drip loggers, which spend all of their time in sleep mode, meant I had a tool I could use for some process of elimination trials. So I made a few loggers that were identical except for the mcu and put them on my newly arrived µCurrent. But my initial results were all over the place. Some units slept at a nice comfortable 0.2 -0.3 mA, while others drew about 2mA, and a few pulled 5mA or more.

That last one is my spare flow sensor, and it came in around 5 mA , apparently confirming the high power use seen in the latest field deployment data.

But five milliamp! Was that thing even sleeping at all? I started wondering if adoption of the Rocket Scream sleep library meant that some of the boards were not sleeping properly, so I dug up the older non-RS code and ran those…nearly the same results, with about 0.03 mA more current because because I was not shutting down the BOD in the older sleep routines. But that small consistent difference told me that the units actually were going into sleep mode between sensor readings, so I went back to my original thought that voltage regulators were my energy vampires.

Then, as I was juggling things around, I switched an “apparently identical” 128mb Sandisk microSD card from the Rocket Scream unit into the TinyDuino flow sensor. And suddenly it dropped from an abysmal 5 mA sleep current down to less than one milli-amp.  A furious shell game followed and after locating the “best” 128mb microSD card, and using it (and the same battery module) the test units delivered:

So counterfeit SD cards were causing the high sleeping currents!

Looks like I have been sold several batches of bad cards from eBay, which do not go into the low current sleep mode I was expecting from the Sandisk spec sheets! (typically around 0.15 mA (see sect2)) When I tested each of the cards I have on hand, the Rocket loggers gave me sleep currents ranging from 0.24 to >5mA, with a cluster around 2mA, and another around 4-5mA.  This does not seem to be related to whether the card is 64, 128 or 256 mb.  I don’t see really bad screen printing but there are differences between cards with the same branding. To be honest, I figured that these small cards are so old, and worth so little money that nobody would bother cloning them. I was wrong.

At least I have identified the issue, so the shield regulators in the Tinyduino stack are off the hook for the bad power performance of my field units. The only problem with this new information is that I am fairly certain I put a bunch of crummy cards into the newly built loggers that we just deployed.  With six AA’s, the flow sensors should survive all right, but with excess current drain like that, the drip sensors will expire after a couple of months. (and likely go into some kind of weird brown-out power cycle loop until the sd cards are toast…)

Addendum 2014-09-22

If I can’t locate a reliable source for small low sleep current SD cards, I will look into Solarduino’s solution of putting an N-MOSFET on the ground line. A few folks have pointed out that might not be quite as easy as it sounds to turn the power to an SD card on and off like that: “because SD.h wrapper for SdFat has a bug that prevents multiple calls to begin()”. What I really want is a system that protects the data by cutting the power to the SD cards whenever the power supply falls low. I would like this control to latch off (in case there is battery rebound), and use an “independent” circuit that relies on passive components. I have some homework to do there because you obviously can’t pull the plug when you are in the process of actually writing to the SD card. I know from my Vcc logs that the new file creation event is biggest sustained load on the system – making it the thing most likely to trigger the cut-off if the mcu is not in control.

 Addendum 2014-09-24

It looks like some people have managed to test Sandisk microSD cards that sleep down around 70 uA.  If that’s true, then 300 uA sleep currents mean I have other power issues still to sort out on these loggers.  According to fat16lib, SanDisk claims some of their cards draw higher idle current in SPI mode if the unused pins are not pulled high: 

“The ‘RSV’ pins are floating inputs. It is the responsibility of the host designer to connect external pullup resistors to those lines. Otherwise non-expected high current consumption may occur due to the floating inputs.”

Evidently this is a perennial issue for beginners (like me), concisely expressed by MarkT over at the Arduino forum:

“This is the classic floating-inputs problem with CMOS logic.  Never leave an input pin unconnected.  If you do it can either oscillate or end up at 50% of the supply voltage (at which point both input transistors are conducting).  Either situation increases power consumption drastically (3 to 5 orders of magnitude for a single CMOS gate) compared to the normal static situation.  Now this might be a small fraction of current consumption of a running microcontroller, its going to dominate the sleep-mode consumption when nothing is being clocked.”

So I need to do a few tests to see how this helps lower the sleeping currents on my old small SD cards.

Addendum 2014-09-27

 After testing the 64, 128 & 256 mb cards I bought from eBay, I have found that if you have the “good” SD cards, the Rocket based loggers generally gravitates towards 0.2 mA or less, even if connections 8 & 9 are floating.

However many of my cards do need pullups (0r pulldown) resistors on the two data connections that are not used for SPI to keep them from floating, or the sleep currents are much higher. In the forums (and the datasheets) people seem to be recommending 50-100K pullups. I tested about 50 cards, and general result is: If this logger system+Sd card sleeps around 0.22mA with the pins floating, I already have a “good” card and pullups won’t change the sleep current by much. However about 1/2 the time a pulldown increases the sleep current of a low power sleeper, sometimes adding as much as 0.6mA (so total sleep current goes to about 1 mA). If there is no rise, then sleep current is unaffected by the pulldown.

If my system draws between 0.5 mA to 2mA with the two pins floating…then a pull down resistor on those two lines will usually brings the whole system down to about 0.25-0.35 mA sleeping current. A pullup does not change the sleeping current of these bad cards quite as much, usually reducing sleep currents by about 1/3-1/2 as much as a pull down.  So preventing the pins from floating is always good to reduce sleep current, but the worse a card is when pins 8&9 are floating, the more likely it is that pulldown will help it more than a pullup on those lines. This is very odd because no where in the data sheets does it specify to use a pulldown resistor. 

I am setting my pass/fail point for the entire logger at 0.33 mA, and if I don’t get to or below that with either pullup or pulldown, then I’m calling it a bad card and I won’t use it.  If the “sleeping system” current for this logger design is above 2 mA with pullup/down, then I have a REALLY bad counterfeit microSD card, and I just throw it in the rubbish bin (which happened to about 6 out of 50).

The WORST cards of all bounce down to a reasonably low sleep current when the system first goes to sleep, and then slowly increase over the course of 2-5 minutes, as I am watching the meter, even though the logger is completely asleep.  Those cards seem to “creep up”, whether I put a pullup, or a pulldown, on the unused lines. What’s interesting is that they don’t “jump up” like I would expect if they were waking …they just slowly increase the draw bit by bit. Of course I am watching this with a plain old multimeter, so I am only seeing “the average”.  Perhaps its a whole bunch of wake/sleep cycles in some kind of self triggering loop? I found 4-5 of these (of 50) and I am assuming that the card controller itself is NFG. A couple of these eventually went over 5 mA before I lost patience and just pulled the plug, but I might go back and let them run later to see how far they go. It might even be handy to have a duff card like this around when I want to bring the power supplies down to test power fault handling.

Generally, if I have a good card, it goes into sleep state as soon as the MCU sleeps, and you can see that on the meter because the numbers are completely stable right away. The crummy cards seem to wander around for a while, like they have to think about whether they actually want to go to sleep or not. Some keep on jitterbugging, even after they have gone into low sleep current territory. So far, I’d say about 50% of the cards from eBay have been ok, with the ones being sold as “used” being much better than the “new” ones.

Addendum 2015-01-10

Looks like I am not the only one who has had to grapple with SD card power issues.  Luke Miller, over at the Open Wave Height Logger project, spotted another more subtle power problem with some SD cards using his oscilloscope.  The new SD cards that he tested were drawing about 200 uA, which sounds much worse than the 60-70 uA Muve Music SD cards I am been using.  But his loggers operate at an incredibly fast sampling rate, and that means time to sleep is just as important as the actual sleep current. Until I get my hands on an old scope and see for myself, I need to consider the possibility that the cards I am using might not be as good as I think they are….

Addendum 2016-03-29

After experimenting some more with mixtures of hard vs soft pullups (& despite the fact that only CS actually requires it) I found that three lines had to be pulled up (w MISO as INPUT not output!)  before misbehaving cards would sleep properly. So add these lines to the beginning of your setup before you run sd.begin

// pulling up the SPI lines at the start of Setup with 328p’s internal resistors
pinMode([ChipSelect], OUTPUT); digitalWrite([ChipSelect], HIGH); //pullup SD CS pin
pinMode([MOSIpin], OUTPUT); digitalWrite([MOSIpin], HIGH);//pullup the MOSI pin
pinMode([MISOpin], INPUT); digitalWrite([MISOpin], HIGH); //pullup the MISO pin
delay(1);

I found that enabling these three internal 20K pullup resistors raises sleep current by between 1-5 μA, so it should not hurt your power budget too much, and could potentially save far more power by helping the SD cards go to sleep faster.

Addendum 2017-05-21

Well, I finally took the plunge and started cutting power to the SD cards: Switching off SD cards for Low Power Data Logging.   I left that step for last because as I was being cautious about anything that might put my precious data at risk, but so far (and I am still testing it…) is seems to be working OK. While Sandisk 250 512Mb cards are generally the best sleepers when the cards remain powered, testing so far shows that Nokia 256 MB cards have superior handling of the the de-powering & restarts events. (ie: the fewest housekeeping artifacts)

I finally have a moment to look at the data from the recently retrieved flow meters, and the results are not what I was expecting. This was a five month deployment, with the units operating from March 22, 2014 to August 25, 2014. The three pendulum units had identical data logging hardware consisting of a the Tinyduino, Sd shield, HMC5883L compass & BMA250 accelerometer, with a DS18B20 temp sensor and a 3-color 5050 indicator LED.  These units recorded Vcc using the 328’s internal 1.1 vref trick, and even with the usual caveat about 10% variability there, the flow sensors did not deliver anything like the low power consumption suggested by my dry test runs*:

(*One of the pendulums had a NCP1402 voltage regulator on the power module, so I will be ignoring that unit for now although the remaining AA cell voltages from that unit were surprisingly similar to the batteries from other the power modules?)

One key observation is that power consumption was similar in both units although Unit4 was creating three times as much sensor, EEprom & Sd card traffic as Unit3. This becomes more obvious if I project these curves down to 2800 mV, which is my cutoff to prevent unsafe Sd card operations:

The vertical lines here represent one month of operation, and Unit 4 (above in orange), which gathered almost 45000 samples in that time,  projects out to another two months of operation.

Despite a boost from slightly newer batteries and a longer 15 minute sample interval, the Unit 3 projection (in yellow) has almost exactly the same amount of time left on it’s 6x AA power pack. This would seem to imply to me that the quiescent current draw of my data loggers is far more important than the power used to drive the sensors. If I just do a quick ‘back of the envelope’ here, and I assume that the six AA’s deliver about 2000mA each,  we are burning through 12000 mAh in about 5200 hours (est.at 7 months)  =  2.3 mA average current.  Ouch!

So I went back and looked at the results from my longest bench top test. This was done with a configuration matching units 3&4 above, but racing along with a sensor read every ten seconds for most of the test. I plotted that voltage curve again with one month time indicators added:

Even with ~400 000 sensor read/record cycles it operated for almost four months – more than half of my lifespan projections for unit 3&4 and it was running on only 3 AA batteries. I had assumed that much bus traffic was the biggest load on the system by far, but perhaps it is time for me to re-consider things? The Tinyduino sensors each have a regulator, as does their Sd shield…have I been ignoring the forest for the trees?

We also had a stationary sensor unit on that last deployment, with a single MS5803-02 sensor, recording both temperature and pressure (barometric & water level). I hacked into a TinyCircuits light sensor board to provide the regulation and I2C level shifting needed for the 5803, so that system had one regulator there, and the one for it’s Sd card.  With only two voltage regs, and no power being used by the DS18B20 temp sensor (which draws  for almost 800 ms to load its 12 bit registers), the stationary unit projects out like this:

At 11 months this just squeaks back in my  design target of one complete year of operation.

These curves leave me with a couple of impressions:

Any regulator, even one with relatively low quiescent current, will draw at least as much juice over time as any of the bits you are actually powering with it in a long term application like a data logger. I might also need to take another look at the losses on the Shottky diodes isolating the battery banks, because even with all those regulators in play we are no where near spec sheet predictions here.

I needed better acclerometer sensitivity,so the new Cave Pearl builds have the sensors moved away from the main stack, and all of the power for the I2C bus is now runs through the single hacked light sensor board, similar to the stationery unit above.  While power was not my primary reason for doing this, I suspect this was the right way to go for power managment in the overall design.

And finally, the drip sensors have only one single regulator in their build, with the Sd card hanging right off of the pins.  And now I am playing with BOD fuses, probably exposing my precious data to even more hazard.  But it just might be that the humble three component logging platform actually surpasses the TinyDuino logger units, in terms of power use over time.  After some months of waiting, my eevblog µCurrent is finally on it’s way, so I will have more power test results to report soon. In future, I will do my bench tests with two identical units,  one of which will do sensor readings, etc., as normal, while the other one simply sleeps the whole time, so that I can isolate where the power is being used.

Addendum 2015-04-12

There have been quite a few field trials since this test was done, and the short version of the results is: TinyDuinio based loggers draw about 0.065 mA sleep current, which on a 6x AA battery pack will get you between 6-9 months of run time. Loggers built with generic pro-mini style boards draw 0.33 mA which will deliver the same 6-9 months of run time on 3x AA batteries. Having good low sleep current SD card is critical to the success of your data logger and you really need to test to make sure your sensors are going into low current sleep modes as well. Cheap eBay sensors from China often fail this test.