Category Archives: Developing a DRIP ⚶ sensor

An event logger counting drips from stalactite formations in dry caves.

For a case study showing the data we collect from these sensors, see: A Flexible Arduino-Based Logging Platform at http://www.mdpi.com/1424-8220/18/2/530

(Sensors is an open access journal, so the PDF is free to download)

Field Report 2015-12-14: Re-install the Drip Monitoring Network

New tufa based installation.

These large soft formations often form around the tree roots that drill down into the caves from the surface.

I brought six new units to cover attrition of the older drip sensors, but with only one failure in the last round, the scheduled replacements left us with several ‘old but still working’ loggers in hand.  With so many good stalagmite-forming records in the bag, we decided to look at some of the amorphous flow-stone draperies on the periphery of the cave. It will be interesting to see if we can develop a sense of how much these sources contribute to the water draining into the cave.

Although I had not yet determined if the Masons RH experiment was working, we decided to re-installed those multi-probe logger since they capture both temperature and barometric data, which is still very useful information for the project.

A roof-top deployment is nearby, for comparison.

With a roof-top deployment nearby, we hope that comparisons with ground level units might provide some idea how much rain is evaporating before it even hits the ground…

With the success of the rain gauges, we decided to disperse those loggers to topside locations ranging as far as Tulum. I will build more for the next deployment, as we would eventually like to have surface rain gauges at all of our cave monitoring sites (precipitation is as relevant to the flow loggers as it is to the drip sensors!).  That’s a tall order, so I am happy that nearly all the units are delivering one-year or more operation on a set of batteries, so we no longer have to change them on every trip. As readers of this blog know, it’s been a long road bringing the project to the point where we can just leave the loggers in place if we run out of time, without loosing any data.

 

 

 

In all, there are now more than twenty five loggers in operation at Rio Secreto, which is the largest concentration of instruments we’ve ever had at one location. My hope is that the information we are gathering today helps to preserve their cave when the growing metropolis of Playa del Carmen reaches their doorstep.

The road out to Rio Secreto skirts the boarder of the largest limestone quarry in the area. A dramatic example of the different perspectives you encounter about how the natural resources of Mexico should be utilized. Of course, as a Canadian, I see exactly the same issues playing out back home...

The road out to Rio Secreto skirts the boarder of the largest limestone quarry in the area. A dramatic example of the different perspectives you encounter about how the natural resources of Mexico should be utilized. Of course, as a Canadian, I see exactly the same issues playing out back home…

<— Click here to continue reading the story—>

Field Report 2015-12-08: The DIY Logging Rain Gauges Work!

Trish & Fernanda inspecting units before retrieval

Trish & Fernanda inspecting units before retrieval

We managed to squeeze in a short  fieldwork trip before the end of the year, and the growing number of loggers at Rio Secreto put that cave at the top of our list to give me enough time to service all the units.  It was also important that we get everything back into place before they were swamped by tourists wanting to spend their holidays in the sun, rather than shoveling snow.

I was very happy to see that only one single machine suffered a sensor failure, and this was one the surface drip units that we had cooked under the tropical sun during a previous deployment.  Some of our early monitoring stations are finally passing that critical one-year mark, so we can start to think about seasonal patterns in records that display this kind of short term variability:

Typical Drip Sensor record 2015, Rio Secreto cave

Drip count /15 min,  Station 10, Far Pool Cluster

RainGauge

I had no idea spiders were so fond of  living in climate stations…

We also had a several sensors on the surface, and I was really curious to see the the data from this first field deployment of the new rain gauges, given that so many of our cave records showed strong discontinuity events like the one above.  Not only did I want to see the quantitative data, I also wanted to know if the bottom shroud prevented the internal temperatures from going into to the 60°C range (which damaged several earlier loggers…)

And…. success! And both rain gauges were within 5% of each other, despite accumulations of bird poop & leaf litter, and one unit suffering from a slow tilt of nearly 10 degrees as the palapa roof shifted underneath it. With conversion ratios from my back-yard calibration, we were able to translate the drip counts directly into rain fall:

Rainfall (cm/day) data from one of our first rain gauge prototypes at rio Secreto

Rainfall (cm/day) from one of our first rain gauge prototypes at Rio Secreto

Trish had her doubts about this record initially, with so much rainfall occurring in what was supposed to be the local ‘dry’ season.  But after searching through data from nearby government weather stations, and comparing our surface record to the break-through events I was seeing in the drip data, we slowly became convinced that it had, in fact, been one of the rainiest dry seasons in quite a while.  We also had a beautiful temp record that showed the new cowlings pulled peak temperatures (inside the loggers) down by almost 20°C:

Rain gauge internal Temp from RTC registers.

Rain Gauge, Internal Temp (°C) from the DS3231 RTC register.

Hopefully this means that the SD cards are back in the safe operating zone, which I know from past failures is nowhere near the 85°C that Sandisk claims.

So the new rain gauges are working properly, adding another piece of hydrology instrumentation to the Cave Pearl lineup.  I would love to say that the Masons Hygrometers delivered another great success, but the analysis is turning out to be somewhat more complicated as the 96-98% RH variations pulled my wet bulb depressions right into the bit depth limit of the DS18b20’s , so I will have to keep you in suspense for a while as I chew on those numbers…

Addendum 2016-03-16

Well serves me right for counting my chickens: Turns out that the drip sensor based rain gauges suffer from spurious counts due to wind noise. But I’ve been running these guys at their highest sensitivity settings, so hopefully I can dial that back to reduce the problem. We also had the gauges on a soft palapa roof, which no doubt contributed to the problem.

<— Click here to continue reading the story—>

Field Report 2015-08-14: Install New Sensors at RioSecreto

Ready for Next Deployment at Rio

The next batch is ready to go!

Sensor failures & battery leaks meant that I had to rebuild some of the drip loggers retrieved last week, in addition to loading all the units with the latest code tweaks for sensitivity, buffering, etc.  Because the latest loggers have improved so much, Trish keeps telling me to retire the mixed bag of first-gens. But the maker geek in me just can’t resist the urge to give those aging units another run just to see how long they will last. (perhaps there’s a bit of self identification there..?)  I’ve heard some of my coder friends refer to this kind of thing as a one tweak loop; often in association with tales of corporate disaster. Fortunately, the hard time constraints of fieldwork are more than enough to curtail my mild O.C.D because when deployment day arrives, you either have it in the bag or you don’t.

The First Masons hygrometer sitting in a cluster of drip sensors

The first hygrometer, located in a cluster of drip sensors, with the “dry” bulb in the foreground, suspended far away from drip sources.

And this time round we had more than twenty units going in. This was the reason that Trish & Fernanda had done so much surveying on our previous day at Rio Secreto. While they were taking measurements, I joked with Trish about playing Goldilocks in a chamber that was literally heaving with beautiful stalagmites, but the truth is picking a good one that doesn’t introduce a sampling bias, is a heck of allot more challenging than you might think.  I left her to sort that out and do the manual counts,  because I was so keen to get the two Masons hygrometers installed.  I’m hoping that the DS18’s can deliver another success, like we saw with the underwater temperature strings.

WetBulb ConfigurationThe key to this approach is that the  wet bulb is hydrated by the run-off from a drip sensor station, hopefully allowing us to operate for long periods of time at these unsupervised locations. So those DS18’s have a very long wick wrapped around the rim of a drip logger with a cable tie. Encouraging evaporation like this is probably going to cause some mineral deposition, and there are a dozen other problems  listed in the textbooks.  But I am going to give it a shot anyway because there are few things as satisfying as discovering something you’ve built actually works, when authoritative sources say it won’t.

This SHT-11 RH sensor is deployed beside the Masons Hygrometers

And we also have a new SHT-11 humidity logger installed nearby.  The sensor itself is the soil moisture rig from Seeed, with a copper sintered mesh over the sensor that could ‘theoretically’ withstand full immersion. I tried to suspend the sensor head so that random drips from above would be deflected away, but even with liberal amounts of silicone on that breakout, I’ll be happy if we get a month of clean data from it before it suffers the same fate at the HTU21D’s we tried earlier. That ought to be enough to calibrate the Masons.

This unit detects drips down to 12cm drip fall

Once we had the new toys in place, we could finish placement of the drip stations.  A couple of the older stations suffered repeated knock-overs, so we decided to relocate those. One of the new sites was near a beautiful pancake-stack formation, but it only had a 12cm drip fall. That’s a very small amount of kinetic energy for my sensors to detect, and I watched this unit for 30 minutes to make absolutely sure the logger was picking it up.  Though the drips made no sound at impact, the indicator LED was piping OK.

The new rain gauge loggers on their first real world installation

The new rain gauge loggers on their first real world installation.

We wrapped up the day by putting the new logging rain gauges on the roof of a building, beside our solar shielded pressure/temp/r.h. sensors. My hope is that those funnels give us a more quantitative record from the little drip sensors in the base, in addition to protecting them from that merciless tropical sun.

Several people have pointed out to me that weather stations are getting pretty cheap these days, and then asked me why I don’t just use something off the shelf.  Generally, I just get a weird look when I reply: “Sure, but where’s the fun in that?” Fortunately for Trish & I, the people at Rio Secreto understand, and they have allowed us to use a corner of their amazing cave once again for the project.  Thanks again you guys!

<— Click here to continue reading the story—>

Field Report 2015-08-06: Retrieve Drip Sensors from RioSecreto

Fernanda taking readings from a known survey station out to a drip logger

Fernanda taking readings from a cave survey station out to a new logger installation.

Fieldwork typically starts with retrieving the loggers deployed on the last visit. Over the last year Rio Secreto has grown to become our biggest single installation, so it is usually the first place on our visit list. It helps that they have plenty of dedicated staff to help out, and Fernanda Lases was able to join us for the day of collecting loggers and surveying their locations. As we gathered the drip sensors, we did manual counts of drips/15min, and took notes on the overall appearance of each installation. A couple had been knocked over by high water events, but most were still perking away happily where we had left them.

As this was near the peak of the local rainy season I was expecting to see some nice environmental response in the data with increasing drip rates.  But the majority of the records had slowly tapered off since March, and the temperature in the cave had risen slightly.  This is the record from drip logger #021, which was typical of the trend:

021 Drip Record

Note: The black line on the drip record is a daily (96 point) moving average. The temperature record is from the DS3231 RTC inside the logger housing, which only has 0.5°C resolution. My other tests have shown that they are much more accurate than the ±3°C that Maxim specifies.

I was not expecting to see reduced counts.  Trish mentioned that she has seen up to six month offsets between surface precipitation and cave drips, but that was in some caves on Vancouver Island.  I’m still scratching my head on this one as there is precious little soil in the area, and I would have thought the limestone was just too porous to provide much storage.

We put a pressure & relative humidity recorder in the cave on the last trip, which I was hoping would run for more than a couple of days this time around. I used a slower epoxy for the potting, with a couple of weeks to cure before going into the field. And the logger did provide a complete record, but as I feared the R.H sensor flat-lined shortly after being placed in the cave:

036 Pressure & R.Humidiy Probe

There was no direct moisture contact with the RH sensor.

There was no direct moisture contact with the RH sensor.

That’s a decent barometric record from the MS5805-02, and at least a hint at the results I might be able to squeeze out of the Mason’s hygrometers if that design can resolve the small wet bulb depressions we will see with humidity bouncing between 95-98% all the time.  The R.H. breakout circuit board was still clean & shiny under the epoxy, with no evidence of moisture intrusion, so I think this high humidity environment is just too much for those humble HTU21D’s.

We also had an underwater logger built with an MS5803, and with the barometric record from #036 we could derive the changes in the cave’s fresh water level:

Derived Water level from two pressure sensors

The spike in that record coincides with thunderstorms that hit the area on June 13-14th, and the local weather records indicate that 5 to 6 cm of rain fell per day during that period. It is interesting that both water level and water temperature return to their previous trends so quickly, and I am keen to see if that precipitation shows up in our other records from further down the coast. If I go all squinty, I can convince myself that some of the drip records were affected by the event, but most of them showed no effect at all.

Even baking under the full tropical sun, some fungus has managed to colonize the ABS plastic. That is one tough organism!

Even baking under the full tropical sun, fungus still managed to colonize the ABS plastic on the cap. That is one tough little organism!

Unfortunately, I can’t confirm the rainfall directly because both of my surface drip sensors croaked. The older 024 unit suffered an SD card failure (as it did last time) and the newer 034 unit drained it’s batteries rapidly when the ADXL345 started self triggering, which would have kept the mcu awake and drawing full power the entire time. The prime suspect in both cases is thermal cycling, with our hardy RTC’s showing some 60°C peaks. It’s worth noting that after replacing the sensor & SD cards I managed to get both loggers working so the 3.3v Rocket Ultras I am using survived the high temps. The DS3231’s had less than 10 seconds of drift after the ordeal.

High surface temps cook the drip sensors

Hopefully the new rain gauge housings I’ve brought along this trip will shield my little drip sensors well enough to prevent this from happening again. I’m sure it doesn’t help that I have the accelerometers set to fairly high sensitivity for this application.

Trish handles the big-picture analysis when we have so many logs to go through, but there are always plenty of  ‘mini’ experiments buried in the data for me to chew on: including confirmation that the loggers pin-powering their RTC’s during µC up-time saw coin-cell voltage drops between 0.03 – 0.1 volts.  And these units did not see clock drift significantly different than the non pin-powered units (~5-10sec / 4 months), giving me confidence that this method to reduce sleep current is worth adopting on more of my builds. (though I will be tracking things with a 4.7meg divider) The small drifts that I could confirm all seemed to have the clocks advancing, rather than loosing time.

Addendum 2015-08-22

Given all the μSD cards I’ve killed off in the surface loggers, it seems pretty incredible that some people have been re-flowing SD cards directly onto breakout boards. That requires bringing them up to about 200°C for a short interval. Clearly long term medium temperature exposures are not the same as short high temperature ones. I’ve also had a fair number of cards “shake” from their holders during a deployment, so this soldering idea got my attention.

<— Click here to continue reading the story—>

Field Report: 2015-03-17 Drip Logger Service Visit

Everett was one of the grad students who also came down to help with the N.U. trip.

I pressed Everett (a Northwestern grad student) into helping me with the manual counts when I collected the drip loggers.

In March my wife led a Northwestern University earth sciences trip to the Yucatan Peninsula. While she was busy with all the necessary preparations for the students who would be arriving shortly, I slipped away for a couple of hours to retrieve the loggers we left at Rio Secreto last year. With so many new units in that deployment, I was really chomping at the bit to see how they faired.

As usual we had some good news, and some bad news from the deployed loggers. Actually, we probably set a new record on the bad news side of things as the two relative humidity loggers that I cobbled together before the December trip went bananas as soon as we brought them into the caves. The HTU21D sensor on unit 030 died on Dec. 20th, one single day after it was deployed, while the sensor on 028 lasted for four days before it pooped out.  Both delivered crazy readings the whole time even though they seemed to be working fine on the surface.

capt

Even potted under epoxy, the solder contacts were severely oxidized.  I suspect that moisture was able to “creep” along the surface of the breakout board, because of the area exposed around the humidity sensor.

The epoxy in the sensor wells had turned yellow & rubbery, even though it was clear and rock hard when the units were deployed . But these sensor caps were assembled just days before the flight, and I used 5 minute epoxy to speed the build, rather than the usual 30 minute stuff. So I am thinking that the moisture resistance of the faster curing epoxies is much lower. Perhaps it’s time for me to investigate some new urethane options with lower permeability? It is also possible that PVC solvent residue interfered with the epoxy’s chemistry because I built them so quickly.

 

Dispite its "splash proof" rating, this MS5805-02 quit after one month in the cave

Despite its “splash proof” rating, this MS5805-02 died after one month  in the cave. It had no “direct” water contact.

The loggers kept running after the R.H sensors stopped working but they eventually both quit long before draining the AA battery packs, which leads me to conclude that rusty contacts eventually shorted the I2C bus, preventing the RTC alarms from being set. We also lost one of the pressure sensors, and a TMP102 board. In fact the only sensor still fully operational when I pulled the loggers was the MS5803-02 pressure sensor, once again showing just how robust those pressure sensors are under their white rubber caps.

 

PressureSensorOnPalaparoof

The white ball is an older first gen housing for an underwater pressure unit, and the black cylinder above is a drip sensor, acting as a crude rain gauge. I don’t know who collected the rain water.

I left a new RH&Pressure unit in the cave, which was made with E30CL and had more than a month of test runs under its belt before going into the field. Even with fully cured epoxy, there is still the possibility that moisture will penetrate through the exposed RH sensor, so I will look into moving that sensor off the main housing for my next builds.

We also had some sensors on the surface during this last deployment, and they faced dramatically different challenges under the full tropical sun.  The pressure logger had been re-purposed from a four month cave deployment. It sported a DS18b20 temp sensor, and an MS5803-05 pressure sensor, which both performed beautifully in the underwater environment.

But as you can see from the pressure record (in mBar) things did not go so well this time around:

DIY Cave Pearl data loggers based on Arduino Microcontrollers

I was expecting daily fluctuations of a few millibars so there is no way I can believe that local highs reached 1200 mBar…but what happened?  This pressure sensor had been used first for an under water deployment so it had a layer of Qsil silicone over top if it. This caused a -50 mbar offset, but did not seem to give us any other problems in the thermally stable cave environment.  But with full sun exposure this logger saw huge daily temperature variations (detailed below) I believe this caused the silicone above the sensor to expand and contract; exerting enough physical pressure to overwhelm the more subtle barometric readings. Unfortunately I did not have time to look at this data while we were in the field, so the unit was redeployed, although this time in a more sheltered spot under the palapa roof.

Now for the good news:

The drip sensor which we left beside that pressure logger on the surface delivered a respectable record despite the fact that it had no collecting funnel:

DIY Cave Pearl data loggers based on Arduino Microcontrollers

That peak of 8000 counts (/15 min.) is about 9 drips per second on the surface of the unit which, with all the delays I put in the code to suppress double count artifacts, might be approaching the max response of the sensor itself. With no way to capture the water, gentle foggy rain events would not have triggered the impact sensor, so there is a good chance that a significant amount of precipitation did not get recorded. But what makes this record so impressive to me is the RTC temperature log from inside the housing:  (in °C)

DIY Cave Pearl data loggers based on Arduino Microcontrollers

The black end cap actually started melting grooves into the white PVC of the drip logger housing.

The black end cap actually started melting grooves into the white PVC of the drip logger housing.

The spec sheet maximum for the DS3231 is 70°C,  and the Arduino mcu‘s limit is 85°C.  Even so, with daily peaks reaching nearly 60° I am quite surprised that the batteries did not pop.  The little logger did not escape this trial by fire completely unharmed, as the micro SD card went from a nice low current sleeper to pulling around 1 mA all the time. The data was intact, but I can only surmise that the high temps cooked some of its control circuitry. The upper ABS surface also changed from a neutral frosted white to a slightly fluorescent green/yellow color, presumably because of intense UV exposure. After replacing the batteries & SD card, the unit was put back on roof for another run.  Just to be on the safe side I added a second unit in case that first one gives out.

While I leave heavy weight analysis of the hydrographs to the expert on the team, I couldn’t help peaking to see if these surface storms affected the in-cave drip flow rates. I was quite surprised to see that the precipitation events had small effects on some of the counts, while barely registering as a blip on others that were quite nearby. This is the record from DS20 (15 min bins, with a purple overlay of surface record that is not on the same scale):

DIY Cave Pearl data loggers based on Arduino Microcontrollers

And this is the record from DS02, located less than 5m away in the same chamber:

DIY Cave Pearl data loggers based on Arduino Microcontrollers

Given the thin soils of the area,I suspect that much of that brief rain evaporated shortly after surface contact, or the dry season vegetation was just sitting there like a sponge, able to absorb most of it quickly.

The whole group of loggers represents a mixed bag of first and second generation builds with many different “mini” form factor Arduino boards in them. I left the batteries in a couple of units back in December so I could see some longer term battery discharge curves:

DS01_02_Longruntest

These two units were using three lithium AA’s, which I knew from the 1st generation test results, are about 2/3 depleted when they hit that 5000 mV shoulder. This tells me that DS01 would probably have delivered nine months of operation on these cells. This is very good news because even the loggers I built with no-name eBay clones (MIC5205 v.regs) sleep around 0.33 mA if they have good SD cards. So it should be safe to put them on a six month rotation schedule.

In addition to their drip counts, several of the loggers were running with different eeprom buffering levels to help me understand how this affected the power budget. I won’t wade into all of that data here but two of the most directly comparable records are from units 26 &  27:

Logger # starting voltage sleep current
# records buffered  V.drop/8500 records
26 5243 mV 0.28 mA 512 30 mV
27 5198 mV 0.26 mA 96 33 mV

Unit 26 was handicapped by a slightly higher sleep current and a starting voltage above the lithium plateau (I often see a small quick drop on 3xAA lithiums above 5200 mV) The fact that it still delivered a smaller voltage drop on the batteries over the three month run implies that increasing the size of the eeprom buffer does improve performance. Logger 26 had a 32K eeprom so it only experienced 16 SD card writing events, while the smaller 4K buffer on unit 27 required 87 SD writes.  Both loggers created six separate log files during the run and the cumulative drip counts were comparable.  It’s still a close call, and the increased buffering does not providing a huge savings, perhaps on the order of 5-10%.  Since the extra I2C eeproms only cost $1.50, and the coding to support them is trivial, I consider that to be an easy way to get another month of run time. As with the buffering tests I did back in 2014, it’s clear that all those eeprom page-writes (3mA x 5msec  + mcu uptime) take a significant amount of power. But at least they are not subject to the random latency delays you see with SD cards.

I added larger LED limit resistors to each logger on this service visit, so even if the drip rates pick up dramatically during the wet season, power used by the interrupt will be reduced compared to the runs since August. All units that were capable are now buffering about five days worth of data to the eeproms. The current crop of “best builds” with Rocket scream boards and pin-powered RTC’s, are getting down to 0.15 mA, implying that they should be good for a year long deployment provided the SD cards & sensors hold out. Of course I don’t count chickens anymore, no matter how good the numbers look. Though all units delivered a full data set on this deployment, two of them suffered dramatic increases in sleep current when their ADXL’s suddenly started consuming more power. You can easily spot when these kind of sensor failures occur by looking at the power supply voltage log:

020_SleepCurrentChange2

I am sure there are more gems buried in the data, which I will post here as they are uncovered.

<— Click here to continue reading the story—>

Field Report 2014-12-20: Our 2nd Drip Sensor Deployment at Rio Secreto

Now that we had all the flow sensors under water, the last big item on the to-do list was the installation at Rio Secreto. New memory cards brought all the 1st generation units down to a more reasonable 0.33 mA sleep current.  The new builds were even better, coming in around 0.22 mA, and they have 32k eeproms allowing them to buffer up to five days worth of data. Not wanting to risk too much on an untried idea, only three of the new builds are pin-powering the RTC, allowing them to get below 0.15 mA between drip counts. I have high hopes for this whole generation, and each has slightly different settings, to help me zero-in on an optimal approach to the eeprom buffering.

Ben Schwartz, Edward Mallon, Fernanda Lasas, Aubrey X

Ben Schwartz, Ed Mallon, Fernanda Lasas, Aubri Jensen

We were very happy to have other cave researchers join us for the day. It never hurts to have extra people on hand for the manual counts, because recording those initial drip rates the “old fashioned way” can take a substantial amount of time. It’s also good to have a fresh set of eyes on the installation, as I often miss things that are embarrassingly obvious while I juggle the low level details about each sensor in my head.  Being able to point out errors like that, with a sense of humor like Ben’s, is one of those vital fieldwork skills that never seems to make it onto the resume, but probably should. Of course, when you bring a bunch of karst researchers together in a beautiful system like Rio Secreto, the conversations start to sound a bit like this. (just substitute in the word “stal”)  I noticed a distinctly inverse relationship between our progress through the cave, and the number of syllables per second  😉

Here Aubrey and Ben are timing drips at a typical cluster.

Aubri and Ben record drip intervals at a 3-unit cluster.

The first chamber (which we started calling Fernanda’s Lab) had a beautiful stalagmite garden on offer, and we tried to cluster drip sensors on actively growing stalagmites of different heights and shapes.  Understanding how drip dynamics and vadose zone chemistry contribute to stalagmite growth is something of a holy grail for my wife, as it underpins all the paleoclimate records that people derive from cave formations. Of course if figuring that out was easy, someone would have done it already.  So we set to work locating drips with the “right stuff”.  Five units ended up in the stall garden, with another cluster off at a more distant pool.  Units 04, 05 and 07 were replaced in their former locations with more robust anchoring this time, as these locations had been washed out by a high water event during the last deployment.

Hopefully the ledge will give some measure of protection

I put the new babies under a dry ledge to keep liquid water away from the humidity sensors.

The last data set gave us a gorgeous pressure & temperature record from the MS5803-02 and I continue to be impressed by sensors from Measurement Specialties.  This time round we had two pressure & RH sensors to deploy in the cave, but I have no idea if those HTU21d’s will survive such long exposure to the high humidity cave environment.  The  5bar pressure logger that we patched up with glue was going to be deployed on the surface, with a drip sensor beside it acting as a crude rain gauge so we can get a better sense of what the cave drips are responding to. By the end of the day we had fifteen instruments on site, making this our largest installation ever!

<— Click here to continue reading the story—>

Field Report: 2014-12-11 Our First Real-World Drip Sensor Data!

Our deployment sites were within site of survey stations, so we could triangulate surface locations later.

Our sites were all close to survey stations, so we can work out the locations for top side sensor co-deployments later.

Trish retrieved the first generation of drip sensors from Rio Secreto yesterday, but we did not have time to open them up until this morning. So breakfast consisted of me concatenating the separate csv files, while she brought some substantial spreadsheet whammy to bear analyzing the data.  We were both pretty excited to find that the first generation of drip sensors had actually worked, and I can only imagine how our little science/nerd fest looked to the other folks at the Turtle Bay Cafe, as they nursed their Mexican holiday hangovers with caffeine.

Trish described how one of our installation sites had experienced a flooding event, which knocked over some of the the sensors. While she was re-assembling the scattered stations, she recorded another set of manual drip counts, so we could verify our numbers. Shortly after the deployment in August, I had made the unwelcome discovery that many of those early drip loggers had been deployed with fake SD cards, which use excessive amounts of power. But I had also installed lithium batteries to provide a bit more juice (~2900 mAh/cell), and I was happy to see that four of the original six units were still running. Whew!

I won’t bore you with the low level details, because I think the graphs speak for themselves. This drip count from is from DS01, binned to a 15 minute interval:Cave Pearl Drip Sensor DATA

Fernanda Lasas,  who kept an eye on our little loggers during this deployment, confirmed that there was indeed an event, with rainfall and a jump in the local water table, right around that first point of inflection, and that it was “pretty rainy” for more than a month after that. This resolved that question I had about the validity of data from DS03, which had failed early.  While there was considerable variability in drip counts from one stalagmite to the next (as predicted by my wife back in August…), we saw congruent trends in the different records from this chamber. With so few stations in this first deployment,  I am not sure what else one could say at this point.

The second chamber we instrumented had considerably lower drip rates, which was somewhat ironic considering that it was the one that flooded during the deployment. This record was from DS07 (shown in the photo above) with the count recorded every 15 min:DripSensor07_Data_RioSecreto_Aug-Dec_2014

While it doesn’t take a rocket scientist to see when the unit was knocked over, I have added a once-per-day x,y,z axis reading to the code, so we can be more certain that the sensors hold a fixed position on future deployments. You do see that spike reflected in the readings from the other chamber, although it was muted against the higher base rate. And it’s even good to see a long string of zeros in the data, as this tells me that my sensors are not generating false positives from internal noise. Accelerometers can be twitchy when you run them at high sensitivity.

The SD card debacle meant that most of these loggers burned through their batteries much faster than I would have liked. But after looking at the plots, I realized that we had a reasonably good natural experiment testing the capacity of the Energiser advanced lithium (EA91) batteries we used.  So in the spirit of “making lemonade”, I thought I would post the combined Battery Voltage (mV)  vs  Sleep Current (mA) results:

Battery Voltage vs Sleep CurrentWhile that’s not a Danish performance test, it shows me that the Cave Pearls really do need to get sleep currents below 0.3 mA if I’m going to get a year out of three AA’s. You do not see that distinct “shoulder & elbow” pattern (see DS05) in the discharge curve of alkaline batteries, which give you a nice gradual decline in voltage.  If a set of lithium cells is going to die while they are still at 4.4 volts, I need to raise the cut-off in the code to protect my data. Unit 3 did have a corrupt file on the SD card, so I probably lost a couple of weeks there by not intercepting the SD writes soon enough. I have reduced the number of records per file, which will put less information at risk from a lost file like this, though it means more work stitching the data together later.

It took us most of the day to sift through it all,  but in the evening I still had a time to fire up a few new units, and review the overnights from the ones I had started the day before. In the process I discovered a problem with the new humidity sensors I had cobbled together just before the trip, which were claiming that our room hit 200% RH sometime around noon. The raw data looked fine, so I suspected a problem with the floating point calculations on the Arduino; something I ran into with the temperature sensors. I locked them into a bag with some desiccant packages to see if forcing them into a low range over night would help me isolate the problem.  They were still running the code that had performed beautifully back home, but during those tests they not been exposed to such high humidity for significant amounts of time.

Still, after a banner day like this, I can’t get too torn up about it.  🙂

Addendum 2014-12-16

I am happy to report that all of these first generation drip sensors went down to ~0.33mA after I replaced the fake SD cards. And only one of those first gen units was built with Rocket Ultra, the rest were a variety of cheap Pro-Mini style clones from eBay.  I will leave the existing batteries in DS01, so that the next deployment gives me more of that discharge curve.

Addendum 2015-01-07

Another little gem lies in the record for DS04, which had a sleep current just under one milliamp.  That unit looks like it would have gone another couple of weeks before hitting the rapid fall-off at 4.4v , giving about 4 months of operation. Now 4(months) x 30(days) x 24 (hours) = 2880, which, more or less, is the number of milliamp hours you get from a single AA battery. But the drip loggers have three batteries, so my ‘live’ duty cycle power drain is roughly 2x that being consumed when the unit is sleeping. DS04 wrote about 10000 records to the SD card and with 64 records buffered to the 4k eeprom on the RTC, that translates to about 160 SD card write events. The cumulative drip count approached 11000. The drip-interrupt routine does not do very much so I think writing data to the eeprom & SD card are my biggest power users. The new generation units have 32k eeproms, and I have them set to buffer a range from 96 to 512 records to see if I can quantify that power use a bit better.

Addendum 2015-03-02

A friend put me onto a way of graphing battery capacity in a way I had never seen before: Ragone Plots. I am still wrapping my head around the idea, but the convergence of all those battery power curves seems to indicate that there is very little advantage to using lithium batteries in low power applications like data loggers unless you need to operate at very cold temperatures.

Addendum 2016-08-31

Just breezing over some of these older posts, and realized the mistake I made on 2015-01-07.  When you put batteries in series you add voltage, but not power, so 2880 mAh would have been the budget for the entire battery pack. Fortunately for me, the blog traffic was so low back then that no one pointed out the gaff.  I’m leaving it in, as it shows what kind of a learning curve I was on at that point, and the project continues to progress despite my many misunderstandings.

<— Click here to continue reading the story—>

Project Update: the Next Generation of Drip Sensors

It’s been a while since I made a post, as the basement workshop is in now in full swing assembling units for the next round of fieldwork. But I recently found out (thanks to Fernanda at Rio Secreto) that one of the drip sensors we installed in August croaked early, with a suspicious order of magnitude increase in the number of drips it recorded just before the battery took a nose dive:

Unit3_RioS_PowerCurveGraph_201400726-1010

Local climate stations don’t show that kind of increase in rainfall, and the batteries did not leak,  so my first thought is that the ADXL345 went through some kind of failure, self-triggering as it brought the whole unit down. But before that event, the battery voltage was good (I used lithium AA’s, which is why they started so high), so even with fake SD cards pulling alot of sleep current I think the other units will survive until we can retrieve them. To be honest, those first drip sensors were a dogs breakfast of different components, and most were cobbled together the week before the trip, so I am surprised that we only have one dead soldier so far. The fact that the data on the SD card survived a complete loss of power is also very good news.  I am actually looking forward to examining that unit, as it is the first hardware failure we have had in the field(I wish I could say the same thing for my code…)

Cave Pearl Drip Sensor Data Logger

With the usual quid pro quos of a highly variable duty cycle, this kind of sleep current might get me in to multi-year territory on 3 AA cells.

For the new set of builds, I have made a few changes to reduce the power consumption of the logging platform.   With a design goal of one year, continuous operation, I have decided to leave out the Pololu  switch, as it wastes too much power when you only have 3 AA’s. The Muve music SD cards from eBay have all been great low current sleepers, bringing my typical build with a Pro Mini clone board into the 0.25mA range. Powering the Vcc line on the RTC from a pin on the Arduino saves another 0.089 mA, and taking advantage of the MCP1700 on the Rocket Ultra brings my best builds down to 0.13mA  with a sleeping SD card (~ 60 µA) and an ADXL345 running (~ 50 µA) to provide interrupts.  Of course, there is always the chance that the RTC will not survive repeated switching on it’s power supply, so I will only test this technique on a couple of units in the next deployment.  If they survive without running the CR2032 coin cells dry, I will adopt this modification for the whole Cave Pearl line. It doesn’t sound like much, but that little RTC breakout will eat 600 mAh (1/4 of a AA) if you power it from Vcc for a year.

When you disconnect Vcc, the DS3231 goes into a 3uA timekeeping mode.

Here I have lifted the Vcc line from the RTC breakout board.  When this line is pulled low by the Arduino, the DS3231 goes into a 3uA timekeeping mode, saving 89uA

As a point of comparison, I tested an “unregulated” TinyDuino stack with the same code, SDcard, RTC, ADXL and battery pack, and it only went down to 0.55 mA while sleeping. Lifting the tiny light sensor board I2C lines, and having the SD card in place, means that the Tinyduino stack actually had two live voltage regulators, while my Rocket based loggers only have one. (Also, I don’t think they are pulling up the unused data lines on their SD shield, so floating pins might be increasing the SD sleep current a bit.)  And while having a regulator on every shield affects my particular application, I can see lots of good reasons for that aspect of their design. I still love the overall form factor of the TinyDuino’s, and the significant amounts time I spend soldering these loggers together has me patiently waiting for TinyCircuits to produce a generic voltage regulated, level shifted, I2C communications shield, so I could spend more of my time fixing code bugs instead of soldering (hint..hint!) It takes me about four hours to build one of these drip loggers from scratch.

I simply let the wires I am already using to tap the cascade port I2C lines poke up enough to give me solder points for the EEprom. Don't forget to remove the pullups on the EEprom board.

I simply let the wires I am already using to tap the I2C cascade port poke through as solder points for the EEprom.

The final upgrade in this iteration the addition of a 32K AT24C256 eeprom. With the testing I did back at the beginning of the project, I know there is a chance that this much I2C bus traffic (buffering up to 512 records if I fill the whole thing)  might not save much power, but it lets me standardize the data handling to one SD writing event per day no matter how many sensors are attached, or how short the sample interval is.  I will have a couple of the drip sensors buffering five days worth of data on the next deployment, to see if using this eeprom to condense five smaller SD card events into one big one  (which takes about 2 seconds to complete)  reduces the long term power consumption of the loggers. And finally, if the unit has a catastrophic power failure that completely toasts the SD card, there is a good chance that I will be able to retrieve the records buried in the eeprom for forensic analysis.

Having a second eeprom in the system also lets me “reserve” the little 4K AT24C32 on the RTC breakout for calibration data. While this is not that important for the drip sensors, it will come in quite handy as my flow sensors mature, and I start to tackle things like hard & soft iron correction on the magnetometer.  The drip loggers are so quick to make & deploy, they have become my low-risk test platform for new ideas.

Drip sensor test rig

Life was easier when I only had one calibration stand. Testing multiple units at the same time turns you into the proverbial man with two watches…

And speaking of test platforms, I now have two testing riggs set up for the drip loggers as they come off the bench, and they have taught me a very valuable lesson: Whenever you invent a new device, you also have to develop a new calibration protocol; which can turn out to be more challenging than building the actual device. Initially, I thought that all I would need for the job was a graduated cylinder and a funnel with some kind of flow limiter.  But my initial test results were all over the place, with the counts varying by 30% or more from one run to the next. At first I blamed the accelerometer thinking “It must be registering multiple counts, or it is just not firing the interrupt when it should”. I tried different sensitivity settings, varied timing delays, everything I could think of. But nothing seemed to improve the consistency between runs, and after many tests with crummy numbers, I was beginning to think I must be missing something fundamental.  I was so frustrated by this I decided to just sit there, staring at the little LED and counting the drops myself for as long as it took to figure this out… And I am really glad I did, because an hour into this caffeine powered exercise in utter boredom I had sore eyes, but I knew beyond shadow of a doubt that those sensors were working fine. Every drop was being counted. Every last one. So what was throwing the numbers?

Surface splatter from a drip impact. The green light is the LED indicating the count.

I made several attempts to capture photos the surface splash after a drip hits, as I thought this was throwing the counts off. Here you see the green LED light being triggered with five or six splash-drips flying away from the point of impact.

Google didn’t bail me out on this one, and all Wikipedia gave me was an article on how drip volume depended on the relationship between surface tension and tip radii.  But both funnels had the same 8mm pex tubing, hand polished smooth, as the drip source point, and they were still giving dramatically different results. It was time to call in the big guns, so I asked Trish to let me root around the Web of Science the next time I was at her office. That lead me to this paper:

Controls on water drop volume at speleothem drip sites:  An experimental study
by Christopher Collister and David Mattey, published in the Journal of Hydrology.

I used heat-shrink to get nice sharp 5mm diameter tube tip.

I  also used heat-shrink to make a much smaller  5mm  drip tube, further reducing the drip size.

In that paper there is a graph (pg 264) that illustrates: “At fast drip rates with drip intervals less than about 10s, drop volumes are seen to increase with a greater degree of scatter in the mass of individual drops.”  Their plot had almost 50% drip volume variability when the drip interval fell to 1 second. I had been testing my units at 3-6 drips per second or more, because I was worried about clipping the high end with all the delays I put in the code to prevent double counts from splash back. I also thought that with more runs, and larger the volumes of water (which increased pressure above the drip), I would be reducing the errors.  I went back with a magnifying glass, and sure enough, I could see the water surface tension shaking like a struck drum after each drip detached, and this wobbling increased dramatically with the rate of flow. Fortunately the paper also provided a solution to my problem:  “The results confirm that drop volumes are essentially constant at drip intervals greater 10 seconds.”  So I completely changed tactics, going for lower volumes and slower drip rates. On the very next try, I was getting data like this:

Unit 10: 500 mL
Drip Count: Time to complete: Drips/Min (max)
6289 135 min 62 dpm
6281 138 min 64 dpm
6314 141 min 63 dpm
Unit 10: 1000 mL
13097 246 min 70 dpm

At about one drip per second, I am still running these a bit fast. It will be a while before I get “ideal” interval tests done as they will take at least ten times longer to perform.  You can see how the larger 1000 mL volume pushed the flow rate up a bit, increasing the 500ml equivalent count up to 6548.  But even at 1 second intervals, I am pretty sure my handling of the graduated cylinder is on par with drip variation as a source of error. And the two different test rigs are within a couple of percent of each other now as well.

Whew!

Addendum 2014-12-02

Since the GSA, I have had people asking when I would have the drip counters for sale, and this post will likely trigger a few more enquiries. Right now every unit I can put together is earmarked for my wife’s research, or projects with other researchers who are helping me improve the overall design. So before that trickle becomes a flood, I should point out that plain old tipping bucket sensors are dirt cheap, and they are pretty easy to hook up to any arduino logger you can find (some people even hook them directly to bicycle speed loggers or calculators. And if you want it wireless, you can use a doorbell). There are some very interesting alternatives in the weather station station market (like the Hydreon RG-11, though at 15-50mA it’s a non starter for battery powered loggers) There are also plenty of lab-based drip counters out there that use approaches such as IR beam deflection or even photo interrupters / optoisolators.  A fair number of people are working on acoustic rain sensors. And finally there is already a drip counter on the market specifically for cave research: The Driptych Stalagmate. The Driptych uses a piezo sensor to detect the impact of the drips. While I have never seen one in person, I can guess that if they are able to use the voltage spike from the piezo to trigger the interrupt line directly, their loggers might be able to operate with an extremely low sustained power draw. In comparison, my accelerometer based approach will probably always pull at least 0.06 mA to keep the sensor ticking over, so I have a ways to go before I reach the four years of operating life they offer, not to mention all the other little bugs I still have to sort out. I am just having a heck of allot of fun doing it.

Addendum 2015-05-29

I recently tried replacing the accelerometers with some very sensitive 18020P shake switches. This removed the constant power drain of the accelerometer, but they also made it fiendishly difficult to adjust the sensitivity. Those simple vibration detectors are not standardized very well, so you have to rotate the switch in 3d space to find the best response angle for each individual one. (unlike the accelerometers, where its just a simple register setting) Tilt it a little too far and the switch will just self trigger, not far enough and you have to hit the thing very hard (relative to drip energy) to trigger the interrupt.  And when you have it in the right physical orientation for “light” drip impacts to register, the spring continues vibrating for quite a long time after receiving a “hard” drip impact.  Those “echo” triggers stretch well past 150-200ms, causing the problem that when the water is really flowing, and the drop volumes increase causing harder hits, the settling time gets longer and longer, so the sensor itself imposes a fairly low maximum detectable drip rate after which the spring mechanism is just triggering all the time.

<— Click here to continue reading the story—>

Field Report 2014-08-26: The First Drip Sensor Deployments

Selecting test locations. (Photo courtesy of Monica Wnuk)

We were spoiled for choice of locations in a beautifully decorated system like this.  (Photo courtesy: Monika Wnuk)

Yesterdays success with the flow meters meant we could now turn our attention to deploying the new drip sensors. Trish had ongoing work in the Rio Secreto Nature Reserve section of Pol Tunich cave, included an interesting student project monitoring calcite raft deposition. A maintenance visit to those experiments was already on the schedule, so it was an obvious choice to use that site for our first drip sensor deployment as well. Even better, Rio Secreto had several capable science liaison staff, some of whom were conducting experiments of their own nearby.

Units are tethered so flood conditions don't wash them away.

When you do something for the first time in the field, you tend to improvise allot, especially when you are trying to apply the Goldilocks principle to something as random as cave drips. But Trish seemed to know exactly what she was after, and with the help of our guide from Rio Secreto, we quickly started locating ‘gotas’ that were ‘just right’.  We knew from Kayleen’s video that the water level in the cave could be highly variable so we tethered the units located close to the water level to assure that flood conditions would not wash them away.

 

ComparisonTest

They are intentionally placed at an angle, so that water does not pool on the sensor surface and cause spurious readings.

We also wanted to set the sensors in areas with multiple drips in relatively close proximity.  Trish has always been fascinated by the question of why some drips readily form deposits while others quite nearby,  possibly with similar chemistry, are much less capable of creating decorations. With this in mind our guide led us to a wonderful “stalagmite garden”  which had many excellent drip sites for comparison.  We found that the soft, slightly concave rubber bottom on the drip sensor housing made it very easy to perch the sensors without harm to the stalagmites. Several units were set up in short order, and as we exited the cave we could see their little green indicator lights blinking away in the darkness behind us.  Even with the help of lithium AA batteries, there is a fair bit of uncertainty about how long these units will actually run, so I was quite happy to hear our guide Fernanda say that she would stop in to check on them when she had the chance. The sensors are designed to light a red led when the battery voltage falls too low, but I told her to remove and disconnect them before that if they showed any other unusual behavior. These units were assembled with an assortment of cheap clone boards, most of which use the MIC5205 voltage regulator, and I will be especially keen to see the performance of the one build with the Rocketsream Ultra, as it’s MCP1700 delivered some very low sleep currents during my bench top testing.  My hope is that we get at least four months of good data from all of them before the batteries expire.

<— Click here to continue reading the story—>

 Addendum 2014-12-12:

This first deployment was a great success, which you can read about here.

Drip Sensor Update: The Gamma Build

I suggested in the last post that a new build usually comes together on the third iteration, so I though I would post a few photos of the current drip sensor, to show how much they changed in that short series:

Alpha, Beta & Gamma builds

Alpha, Beta & Gamma builds of the Cave Pearl Drip Sensor

That Alpha would only detect drops in the very center of the housing, and often registered double hits because of the splash back from the complex surface topography, which also caused a buildup of water on the surface, further interfering with the signal. So I removed the pressure and temperature sensor wells from the Beta, and used a heat gun to bow the surface and shed water.  I hand sanded the pvc to make the strike surface thinner and more responsive, which increased the “reliable” sweet spot to the diameter of the circle you see on drawn there. This worked well, but as you might imagine removing that hour long fabrication step rose to the top of the priority list (if necessity is the mother of invention, laziness is surely the father…) I also did not want to penetrate the housing for those standoffs if I could avoid it because this device has to maintain integrity with constant water impact at exactly that spot.

JB Plasicweld bonds the accelerometer to the cap

Plastic-weld bonds the sensor, preserving the integrity of the housing

My solution to both problems was to solvent weld a four inch knock out cap to the top of the hard pvc shell.  The green color you see there is ABS to PVC transition cement, that I learned about back at the beginning of the project when I was mangling Leggo bricks for internal scaffolding. The ABS is translucent, so the LED no longer needs a portal of clear epoxy because the light passes right through.  The knockout is thin and stiff, which eliminates all that sanding and improves the response of the accelerometers so much that now the device even responds to loud noises, and almost the entire surface sensitive to the smallest drip.  As a result I now have to tweak the settings to reduce the sensitivity so we don’t get too many false positives.  But on the third build of a new sensor, that’s the kind of problem you want to have.  I though the flat surface might resurrect the water pooling problem we had on the Alpha, but in the end it just ended up being a lesson in how I still miss things that are really darned obvious, because simply setting the drip sensor up with a slight tilt will shed the water without affecting the operation at all.  (I am just glad I realized this before some three year old came along and pointed it out to me…)

These guys are only $1.5 each, and are much easier to work with than the plastic micro SD adapters

Much easier to solder those jumpers now.

Wirefold

The trickiest part of the build is routing the wires to reduce strain. Thin silicone 26awg wire helps quite a bit there.

There have been a few improvements to the logging platform as well.  I found these really inexpensive  raspberry pi SD card adapters that are mounted on their own pcb, which neatly solves the melting problem that the cheap plastic  adapters give you if you linger too long with your soldering iron.

Drip Sensor Gamma BuildSo this is the new baby, and I am now running burn tests on loggers using a few different clone boards, including a Rocket Scream Mini Ultra, as they have some very interesting power saving features that might significantly increase the operating time, while keeping the overall simplicity of the three component design.  I expect a few more issues will arise in testing, so I will hold off posting the code to Github, till I am sure that they are behaving properly.  I am still a bit stunned that these drip sensors came together so quickly, but perhaps the last six months of work on the flow meters had something to do with that. 🙂

<— Click here to continue reading the story—>  .

Addendum

Adafruit just posted a video from the National Science Foundation showing how water droplets move on various hydrophobic surfaces.  Way cool…

Addendum 2014-12-11:

We deployed our first batch of these drip sensors in August, and when we went back to get them in December, we were delighted to find that the first real world run was a resounding success.