Category Archives: Expedition Reports & Updates

Photos, videos & reports from the field.

Field Report 2015-12-10: Sensor Precision vs Accuracy vs Drift

Testing multiple flow meter deployment configurations.

We compared different fin position configurations & three build generations on this deployment.

With drip sensor service under way, we started our usual round of dives to exchange the flow sensors. We often use our favorite coastal outflow as an equipment shakedown dive because it’s fairly shallow and we know the system so well at this point that it feels more like the project workshop than an actual cave. We monitor tide levels here with a pressure sensor, and we also had the 25cm DS18b20 temperature string in the system this time round (but I will report on those results later)

So close to the ocean, this system delivers tidal signals like ‘old faithful’, and we have another gorgeous data set (in triplicate!) to keep my favorite karst hydrologist happy:

Uncorrected Raw Tilt angle as a proxy for water flow velocity in one of our coastal outflow monitoring sites.

Raw Tilt angle in degrees (as a proxy for water flow velocity)

Those high flows line up nicely with the large rain events recorded at Rio Secreto, and with peak displacements above eighty degrees, I suspect that the drag fins are bumping the ceiling of the cave; clipping some of our high end. But that’s still a beautiful time series, and it reminds me that now I really have to get my hands on a logging ADCP so I can calibrate the 2″ housings to point velocity. Unfortunately past experience has already shown us that acoustics often don’t like being in caves with low ceilings, so I would also have to do that testing in some other system or get my hands on a concentrating beam ADV logger to avoid any reflection issues.

The other gorilla in the room is that age old question of accuracy vs precision. This shows up most clearly in the two temperature records from our pressure sensor, which sports a 12-bit Adafruit MCP9808, and a 24-bit MS5803-05BA (which also records temperature for it’s internal corrections) located right beside each other:

MCP9808 vs ms5803 temperature data...?

Temperature (°C): MCP9808 (Top)  vs MS5803 (Bottom)

That pesky pressure sensor....

That pesky pressure logger….

The bit depth limitation of the 9808 shows up pretty clearly against the 5803’s beefy ADC, but my dilemma is that that the factory calibration on the Adafruit sensor is ±0.25°C, while all the lovely data from the M.S. sensor comes with a quid pro quo of ±2.5°C.  I’d be happy to cherry pick diurnals out of the high rez record, but even without a trends over top it’s obvious that the two sensors have diverging behavior. (though both sensors claim great drift stability?). This is the kind of thing that drives a builder like me nuts because it hints that we might have another creeping problem like the TMP102 pressure sensitivity that nearly took out a whole generation of loggers. I think I will have to start recording the RTC temp register (which is protected inside the housing) so I have another data set to compare to these two surface mounted sensors.

What I would really like to know is if there was some way I could squeeze higher precision out of the 9808.  I keep finding off-hand references in the forums suggesting that if you average ~16 readings, you get another decimal place of resolution out of your sensor. But with temperature sensing IC’s doing massive amounts this when they convert 12-bit readings, they might already have reached the point of diminishing returns. This question is also relevant to the ubiquitous DS18b20’s we’re using, because they are so stable that 16 readings in a row usually just gives me the same number 16 times. Does this mean that averaging has already taken any bit-depth enhancing noise right out of the signal?

<— Click here to continue reading the story—>

Field Report 2015-12-08: The DIY Logging Rain Gauges Work!

Trish & Fernanda inspecting units before retrieval

Trish & Fernanda inspecting units before retrieval

We managed to squeeze in a short  fieldwork trip before the end of the year, and the growing number of loggers at Rio Secreto put that cave at the top of our list to give me enough time to service all the units.  It was also important that we get everything back into place before they were swamped by tourists wanting to spend their holidays in the sun, rather than shoveling snow.

I was very happy to see that only one single machine suffered a sensor failure, and this was one the surface drip units that we had cooked under the tropical sun during a previous deployment.  Some of our early monitoring stations are finally passing that critical one-year mark, so we can start to think about seasonal patterns in records that display this kind of short term variability:

Typical Drip Sensor record 2015, Rio Secreto cave

Drip count /15 min,  Station 10, Far Pool Cluster


I had no idea spiders were so fond of  living in climate stations…

We also had a several sensors on the surface, and I was really curious to see the the data from this first field deployment of the new rain gauges, given that so many of our cave records showed strong discontinuity events like the one above.  Not only did I want to see the quantitative data, I also wanted to know if the bottom shroud prevented the internal temperatures from going into to the 60°C range (which damaged several earlier loggers…)

And…. success! And both rain gauges were within 5% of each other, despite accumulations of bird poop & leaf litter, and one unit suffering from a slow tilt of nearly 10 degrees as the palapa roof shifted underneath it. With conversion ratios from my back-yard calibration, we were able to translate the drip counts directly into rain fall:

Rainfall (cm/day) data from one of our first rain gauge prototypes at rio Secreto

Rainfall (cm/day) from one of our first rain gauge prototypes at Rio Secreto

Trish had her doubts about this record initially, with so much rainfall occurring in what was supposed to be the local ‘dry’ season.  But after searching through data from nearby government weather stations, and comparing our surface record to the break-through events I was seeing in the drip data, we slowly became convinced that it had, in fact, been one of the rainiest dry seasons in quite a while.  We also had a beautiful temp record that showed the new cowlings pulled peak temperatures (inside the loggers) down by almost 20°C:

Rain gauge internal Temp from RTC registers.

Rain Gauge, Internal Temp (°C) from the DS3231 RTC register.

Hopefully this means that the SD cards are back in the safe operating zone, which I know from past failures is nowhere near the 85°C that Sandisk claims.

So the new rain gauges are working properly, adding another piece of hydrology instrumentation to the Cave Pearl lineup.  I would love to say that the Masons Hygrometers delivered another great success, but the analysis is turning out to be somewhat more complicated as the 96-98% RH variations pulled my wet bulb depressions right into the bit depth limit of the DS18b20’s , so I will have to keep you in suspense for a while as I chew on those numbers…

Addendum 2016-03-16

Well serves me right for counting my chickens: Turns out that the drip sensor based rain gauges suffer from spurious counts due to wind noise. But I’ve been running these guys at their highest sensitivity settings, so hopefully I can dial that back to reduce the problem. We also had the gauges on a soft palapa roof, which no doubt contributed to the problem.

<— Click here to continue reading the story—>

Cave Pearl Data Loggers at GSA 2015

We had two presentations at the Geological Society of America conference last weekend in Baltimore.  Trish spoke in the Karst Processes and Speleogenesis session about the challenges of new instrument development, and presented a snapshot of our growing long-term record which includes ocean sensors,  surface precipitation & cave drip monitors, with flow sensors inland and at coastal discharges:

Cave Pearl Logger deployment Record to Sept 2015

You can see attrition of the early prototypes pretty well in a graph like this.  We lost units from the 3rd generation of flow sensors due to epoxy failures, and much of the fourth failed due to pressure effects on our temperature sensors.  Despite having to wake for several hundred thousand processor interrupts,  the drip sensors are faring much better overall.  But I have been replacing them anyway, as the recent builds have much better power consumption. Even with improvements to each logger, you still reach a point where field logistics put a limit on how many units you can deploy with a small two person team.

While we aren’t at the whole catchment/watershed scale yet, we hope that our little DIY monitoring network will grow into the kind of coverage you see in Critical Zone Observatories.  Trish ended her talk by walking through some data from a very large storm event to show how it impacted the different monitoring stations. You could see the effect on people as they watched the individual sensor records building into a system wide picture of the hydrology.

Presenting the Cave Pearl loggers at the GSA conference 2015

There was a a lot of interest from younger academics, as they understand the potential for open source kit to help them compete with senior colleagues who can afford $5-10k for each piece of equipment.

That interest spilled over to the poster session the next day where I had a selection of different builds on display. I was a little embarrassed as many of these were rough early prototypes & bookshelf calibration units. (because all the really good loggers are currently out on deployment 🙂  There was a steady stream of people with questions that kept me at the poster all day and both of us were on deck for the huge crush of people at the end of the day. All that positive feedback was great for the ego, and it was especially nice to talk to people from other research groups working with the Arduinos. Each of us is tackling different questions, leading to a good diversity of hardware builds and approaches.

I think things are reaching critical mass because at this point many of the smaller independent projects people have been working on have seen enough field trials to cross the tipping point from rough prototypes to capable research equipment. DIY instrumentation will have a much larger presence at future conferences, and some of us have even started scheming about a whole session dedicated to it. If we can wedge it in between the teaching and the fieldwork schedule, perhaps next year we will bring the Pearls over to the AGU.


The conference generated enough interest to show up on the blog’s traffic: not so much as a change in the number of visitors, but as an increase in the number of pages being read:

I think pages per visitor is probably a better metric of interest, though I am sure humble stats like these don’t even count as a drop in internet bucket. But hey, you’ve got to start somewhere…

<— Click here to continue reading the story—>

Interview: Cave Pearl Project Team: 2015-09-20

Now that we have recovered from the last round of fieldwork, I finally managed to blow the dust off an old Macintosh in the basement and produce a little promo for the project:

Still needs some polish, but not bad for a few quick clips in the living room. My next task is to update my “How to build a data logger” posts to include all the new improvements.

Addendum 2015-08-24:

The new build series tutorials are now live to help folks get started.

<— Click here to continue reading the story—>

Field Report 2015-08-17: Flow Sensors go back into Akumal Bay

The surfaces were covered with hard deposits

Even after scrubbing, those surfaces were still covered with hard deposits. I wonder if hydrophobic paint would prevent this?

With the the dive deployments done, and the Rio Secreto installation out of the way, it was time to start wrapping up the trip.  Sometimes we are forced to leave the open water flow units with Gabriel at C.E.A., but he had important meetings that morning and I had enough time on to install them on my own. As we talked about potential sites for other units, I laid out the loggers, cables etc. on the table. He was somewhat  surprised to see the condition of the older units.


Treating Ocean units with muriatic acid

Nothing like a good hydrochloric acid bath at the end of a long deployment.

You see when we retrieved B3 & B4 at the start of the trip, months in the sea had coated them with so much bio-growth that they looked like something from “Pirates of the Caribbean”.  On previous trips, hard scrubbing and bad language were enough to sort them out, but after that failed I knew we were going to need bigger guns. After googling my way through a few chemical resistance charts, we popped down to the hardware store for a bucket, and bottle of muriatic acid. And as we hoped it was highly effective, but I was biting my nails as we watched the loggers, and the data they still contained, fizzing away like seltzer tablets. Fortunately those EPDM O-rings held up, and after a few hours in the soup, I was finally able to scrub away the crud and get to our data.  I did my best to keep the sensor wells away from the acid, as the epoxy there was already getting pretty old.

So by the time I was ready to swim out into the bay, our flow sensors had gone through something of transformation:

B4 before & B4 after

B4 before                                      &                                 B4 after

I spent a fair bit of time locking down a new anchor plate for B3, with sea urchins and rolling surf conspiring to make that a challenge. And I don’t know if it was the fact that I was further out on the reef, or that I just did not move like the tourists, but I swear critters came of the woodwork just to see what I was doing.  The barracudas were probably drawn in by the shiny metal surfaces on the camera, and at one point, while I was busy looking down to capture some clips of B4 in motion, a sea turtle swam right into me. I know it sounds funny, but an impact from something that big when you are floating in the sea can really scare the willies out of you.  When I spun round to see what happened, there were three more beside me (…probably doing the turtle equivalent of laughing…inside…)  But by this time the loggers were installed, and I was too worn out from all the swimming to spend any time watching them.  Reluctantly, I headed back in.

Of course, things always happen when you are not looking for it, and as I made my way to shore I noticed a spotted eagle ray swimming nearby.  I was in the boating lane at the time, and decided that trying to capture a photo was not worth getting run over, so I just kept going. However, once I reached the shallower sandy area, he reappeared right under me, and this time I dug the camera out of the mesh bag I had been using to ferry the loggers around:

…and I think I will use that to sign off on a brilliant fieldwork trip.


By the end of all this, my field-notes went over 50 point-form pages of observations, readings, etc., and there is no way I could relate more than a small sampling of that here. Once the major diving is done, we try to grab a little social time with friends as we drop off various bins of gear to be stowed till next time.  Trips like this could never happen without the help of the dive community in Tulum, and we are grateful for all the help they have given us over the years. Of course a proper list of thank-yous would be even longer than my field notes, but I’d like to give a special shout out to Bil, Robbie, Kim, Natalie, Jeff and Gosha. I hope some of this blog-y catharsis makes you laugh, and some of it makes you proud, because you are all an important part of it.


<— Click here to continue reading the story—>

Field Report 2015-08-14: Install New Sensors at RioSecreto

Ready for Next Deployment at Rio

The next batch is ready to go!

Sensor failures & battery leaks meant that I had to rebuild some of the drip loggers retrieved last week, in addition to loading all the units with the latest code tweaks for sensitivity, buffering, etc.  Because the latest loggers have improved so much, Trish keeps telling me to retire the mixed bag of first-gens. But the maker geek in me just can’t resist the urge to give those aging units another run just to see how long they will last. (perhaps there’s a bit of self identification there..?)  I’ve heard some of my coder friends refer to this kind of thing as a one tweak loop; often in association with tales of corporate disaster. Fortunately, the hard time constraints of fieldwork are more than enough to curtail my mild O.C.D because when deployment day arrives, you either have it in the bag or you don’t.

The First Masons hygrometer sitting in a cluster of drip sensors

The first hygrometer, located in a cluster of drip sensors, with the “dry” bulb in the foreground, suspended far away from drip sources.

And this time round we had more than twenty units going in. This was the reason that Trish & Fernanda had done so much surveying on our previous day at Rio Secreto. While they were taking measurements, I joked with Trish about playing Goldilocks in a chamber that was literally heaving with beautiful stalagmites, but the truth is picking a good one that doesn’t introduce a sampling bias, is a heck of allot more challenging than you might think.  I left her to sort that out and do the manual counts,  because I was so keen to get the two Masons hygrometers installed.  I’m hoping that the DS18’s can deliver another success, like we saw with the underwater temperature strings.

WetBulb ConfigurationThe key to this approach is that the  wet bulb is hydrated by the run-off from a drip sensor station, hopefully allowing us to operate for long periods of time at these unsupervised locations. So those DS18’s have a very long wick wrapped around the rim of a drip logger with a cable tie. Encouraging evaporation like this is probably going to cause some mineral deposition, and there are a dozen other problems  listed in the textbooks.  But I am going to give it a shot anyway because there are few things as satisfying as discovering something you’ve built actually works, when authoritative sources say it won’t.

This SHT-11 RH sensor is deployed beside the Masons Hygrometers

And we also have a new SHT-11 humidity logger installed nearby.  The sensor itself is the soil moisture rig from Seeed, with a copper sintered mesh over the sensor that could ‘theoretically’ withstand full immersion. I tried to suspend the sensor head so that random drips from above would be deflected away, but even with liberal amounts of silicone on that breakout, I’ll be happy if we get a month of clean data from it before it suffers the same fate at the HTU21D’s we tried earlier. That ought to be enough to calibrate the Masons.

This unit detects drips down to 12cm drip fall

Once we had the new toys in place, we could finish placement of the drip stations.  A couple of the older stations suffered repeated knock-overs, so we decided to relocate those. One of the new sites was near a beautiful pancake-stack formation, but it only had a 12cm drip fall. That’s a very small amount of kinetic energy for my sensors to detect, and I watched this unit for 30 minutes to make absolutely sure the logger was picking it up.  Though the drips made no sound at impact, the indicator LED was piping OK.

The new rain gauge loggers on their first real world installation

The new rain gauge loggers on their first real world installation.

We wrapped up the day by putting the new logging rain gauges on the roof of a building, beside our solar shielded pressure/temp/r.h. sensors. My hope is that those funnels give us a more quantitative record from the little drip sensors in the base, in addition to protecting them from that merciless tropical sun.

Several people have pointed out to me that weather stations are getting pretty cheap these days, and then asked me why I don’t just use something off the shelf.  Generally, I just get a weird look when I reply: “Sure, but where’s the fun in that?” Fortunately for Trish & I, the people at Rio Secreto understand, and they have allowed us to use a corner of their amazing cave once again for the project.  Thanks again you guys!

<— Click here to continue reading the story—>

Field Report 2015-08-12: Success with DS18B20 1-Wire Temperature Chains!

OMG! It worked. Woot!

Yep, that’s me grinning like a fool. It might not be an X-prize, but it worked! IT WORKED!

We were already happy with the flow & drip sensor data from this trip because, despite the TMP102 problems, most of the units had performed brilliantly.  But for me, the real prize of the season was going to come from the underwater temperature strings that began their first ‘real-world’ trials on the last trip. Because the cave they were in presented some challenges, we didn’t pull those units until we had a few good dives under our belts. Back in March, I had just changed over to slimmer builds in 2″ PVC pipe,  and the flow meters at this site were the deepest deployment so far for those new housings. So every unit in this cave was testing something important, and I hoped to take plenty of photos despite the fact that we were right at our little cameras limit.

Trish had line duty (for this part of our dive), and she captured a reasonable shot of me inspecting first flow meter from there. Though I had only been at the sensor for a few moments, you can already see a ball of dust starting to form overhead:

After that we installed a new ceiling anchor, with a descender rod to put a Pearl in the deeper saline flow.  Unfortunately, all that faffing perc’d out the site, so I only managed a tiny clip of one temperature string in-situ before the cloud of pea soup drifted over:

That logger supported a fairly short 5m sensor chain, with 19 nodes spaced at 25cm. It was installed across the halocline, and I admit that I was concerned that (with a minimum increment of 0.06°C) those humble DS18B20’s would not have enough resolution to track the fresh/salt water interface. In addition, I had assembled the string in segments that were linked by my new diy underwater connectors, so these builds had more potential failure points than I even wanted to think about.  It’s probably a good thing that that our dive schedule was so full that I didn’t have time to look for LED heartbeat pips while we were still under water.

Following that long dive, we had a bumpy crawl back to the main road which put more than a few new scratches onto the rental car. After one bone-shaker, Trish observed a logger going through it’s startup sequence on the indicator LED.  A power blip like that during an SD write could toast the card, destroying all our data!  I asked her to cradle the new babies till we got back to the paved highway. When we reached Tulum, we returned our tanks, stowed the gear, and bolted down a couple of tacos in record time. I might even have exceeded the speed limit a bit on the way back to the room…

But after some tough dives, and months of waiting, this was the result from #045:

Cave Pearl data loggers - DS18b20 Temperature string
Note: I inverted the temperature axis (left side) to match the physical situation: the saline water was warmer at depth, with a cooler fresh cap layer. The black traces are 96 point (1 day) moving averages.

The deeper saline water was a full degree warmer than the shallow fresh water, giving plenty of spread for the DS18’s.  And with 25cm spacing we managed to plant one sensor right in the middle of the halocline, capturing its cycles of expansion and shearing away. And that’s just the raw data!  Even without the calibration corrections it was easy to see that we nailed it. Unit #046 gave an equally complete log, but with its larger 50cm spacing the sensors straddled that fresh/salt boundary, so we simply have an empty gap on the plot. Of course to a karst hydrologist, knowing the limits of mixing zone is also useful information

After the initial excitement over that temperature data died down, I proceeded through my usual set of post deployment checks. I was keen to compare the power curves from the two loggers we had deployed:TempLoggerPowerCurves

I knew those sensors were going to pull a substantial amount of juice during 12-bit conversions, but putting twelve AA batteries in the housing was still something of a shot in the dark for me back in March. While both curves looked smooth , there was something odd about #046 using less power than #045, because it had one more sensor (total of 20) and they were stretched out over 10 meters of cable so #046 also had a more aggressive pull-up resistor on the bus.  A bit more poking around and I noticed that I had accidentally reversed a cell in one of the banks  (those with sharp eyes probably  spotted that in the photo above) so #045 had actually run on only three sets of AA batteries. Shottky’s isolate each bank against battery failures, but it’s nice to know that they also protect the little loggers from my own dumb mistakes. With 46’s full complement of 4x3AA batteries only loosing 0.5 v over almost four months, I’m confident these loggers could approach my one-year operating target on a fresh set.

The marine heat shrink tubing adhesive after four months

After four months under water the adhesive on that marine-grade heat shrink looked a bit flaky, but there is ECL30 potting the wires inside the adapter so I’m not worried about the seal.

Of course that’s predicated on everything staying water-tight, so I examined each temperature sensor very carefully: looking for evidence of water damage. Most of the nodes were filled with hard epoxy (E-30CL), and for some I was trying out a more flexible urethane. (U-09LV )  They were all remarkably clean, with only one node showing yellowing, and that one was a botch where I had split the original sheathing under the heat gun, and had to re-wrap it. I was also pleased to see that my DIY underwater connectors proved to be robust. They were all were bone dry inside, with no hint of oxidation on the contacts. It was looking like we would be able to re-deploy these units right away!

While I was examining the hardware, Trish had been chewing on the data from both of the loggers. At one point she started making funny “Hmmm…” noises which I know she only makes when she disagrees with something, but is being too polite to say so.  (You hear that kind of thing a lot at academic parties…)  When I asked her what was wrong, she showed me the pressure log from one of the MS5803’s:


Damn! Even with surface barometric corrections it was obvious that the rising trend in this record was out of sync with our other water level recorders.  And with 1 millibar of pressure being approximately equivalent to 1 cm of water depth, the implied 4m delta is simply ridiculous. I immediately suspected that the Qsil 216 I had put over the pressure sensor was doing something weird. I’ll need to do some homework to sort out what actually happened, but I’m guessing the silicone started absorbing moisture at depth since the stable cave environment is unlikely to cause problems from thermal expansion.

#046 was ready to roll the next morning.

#046: ready to roll the next morning.

So we didn’t get a hat-trick on these first builds, but by 2 am I had new batteries in place, the clocks updated, and I had carefully peeled away the silicone over the MS5803s. (hopefully without damaging the factory gel caps). If the overnight run looked good, we hoped to install #046 in deeper cave (~24m) the next day.

<— Click here to continue reading the story—>

Field Report 2015-08-10: Diagnosing a TMP102 Temperature Sensor Failure

Over the next few days we did cave dives to exchange units at sites along the coast. Some of these systems were too deep for our camera, and in the shallower systems the visibility could get a bit grim because if you stay in one place too long, air bubbles rise up and cause “percolation” of little pieces of rock from the ceiling.  So to dismiss any illusions people might have that science diving looks like what they show you on TV, here’s a clip from one of those claustrophobic installations:

Even so, things went well.  Each night I had a new crop of loggers to download and check for my “standard set” of parameters (battery, RTC coincell, sleep current, clock drift, etc.) which I compared to the readings taken before the deployment. The recent retrievals were from the last generation of 3-inch pvc builds, and while they all looked great on the outside, some of them stopped running long before we pulled them out. It can be tricky to figure out the problem when a logger has several I2C sensors on it because the very things that make it so easy to use those breakout boards, also hide problems from you: essentially they don’t just run digitally, they also tend to die that way.  Issues tend to show up much earlier in the analog data, and one useful diagnostic is the voltage log from the main battery pack (read with a simple resistor divider).  Alkaline AA’s usually provide a nice smooth decline over time, so when I see something like this:


Unit w BMA180, TMP102, HMCL5883l. Deployed in fresh water at 5m.

…it’s an immediate red flag.

But unit #011 had a complete set of data logs, with no hint in the sensor records that something strange occurred.  That peak in temperature was pretty large, but it also corresponded to the June 13/14 event that showed up my Rio Secreto & Akumal Bay records, so I accepted it as valid data.  My next thought was that the unit had suffered a μSD failure, but this logger was still successfully going to sleep at a reasonable 0.26 mA. Rapidly entering a low current sleep state is the best indicator I’ve found so far that a card’s control circuitry is still working as it should. Hmmm….

Then next unit with problems was also from a relatively shallow fresh water deployment, and it ran for only two weeks before shutting down:

BMA180, TMP102, HMCL5883l, Deployed in fresh water at 14.5m Depth

Unit w BMA180, TMP102, HMCL5883l.  Deployed in fresh water at 14.5m.

Leak corosion after sensor failure drained the batteries

The power module from #015:  alkalines almost always leak if something pulls them below 0.5v.

This logger was positively pressurized when I broke the seal, with only 0.22 volts at the battery module. When I connected a new battery the unit would did restart properly, producing a strange pattern on the indicator LEDs. Fortunately even with only two weeks of data, what we did have in the log make it pretty clear who the culprit was. It was extremely unlikely the temperature would have spiked like that on the same day that the battery curve took a nose dive.

The next unit presented another mystery, because it had a nice smooth power curve though it had reached it’s low voltage shut down point too early. The temperature data looked reasonable:

Unit w BMA180, TMP102, HMCL5883l. Deployed in fresh water at 22.5m Depth

Unit w BMA180, TMP102, HMCL5883l. Deployed in salt water at 22m.

These loggers have dual 3xAA power packs, and they normally deliver at least eight months of operation. While #016 seemed to restart OK, with a normal (~o.24mA) sleep current, it then crept up to 1.65 mA.  I left the unit running for about 5 minutes and by that time the sleep current was jumping all over the place, varying from 0.5mA to 4 mA at random. This behavior continued after I swapped in a known good SD card, so I was stumped. Once again the data in the sensor logs looked fine.

I had one more unit from this build generation to open up, and it too had developed a problem over the deployment:


Unit w BMA180, TMP102, HMCL5883l. Deployed in salt water at 24m.

No rust at all...?

I have to trim the boards a bit to fit them in the sensor wells but I am careful not to cut any traces or vias. There was no visible rust on those breakouts at all.

That hockey stick bend on the power curve was another smoking gun, especially in the context of that temperature data. The deep saline water in these caves is remarkably stable, so when I saw a temperature rise alignment like we had with #015, I started pulling wires. Sure enough, the sleep current problem on #016 disappeared as soon as the TMP102 was disconnected and #015 fired up properly as well.

Since epoxy failures took out a fair number of units in 2014, I did a detailed inspection of the potting, but found nothing to indicate that water had reached the sensors. Since the deeper units were most affected, my current working hypothesis is that these sensors were actually suffering from the effects of pressure, rather than moisture. (…but good luck finding that in your ‘Recommended Operating Conditions’)

Anyway, I disconnected the TMP102’s from all the loggers of that generation, (including the ones that were still working…) and set the code to read the internal RTC temperature register instead. Test runs with that were good, and these units went back out for deployment a few days later even though all that housing mass will put a huge lag into the temperature record.  I don’t think I will be using TMP102 sensors again on my projects, but of course there is nothing to say the MCP9808’s from Adafruit won’t develop a similar problem over time.  So I might end up back at the humble DS18Bs which I used on my earliest builds:  they have proved to be remarkably resilient over time.

Addendum 2016-03-17

The MCP9808’s did indeed suffer a similar fate.


Field Report 2015-08-07: Retrieve Flow Sensor from Akumal Bay

Gabriel, Patricia Beddows, Marco A. Montes Sainz

Left to Right: Gabriel Sanchez Rivera / Patricia Beddows  / Marco A. Montes Sainz

I was up pretty late downloading the loggers from Rio Secreto the day before, so we had a late breakfast at Turtle Bay Bakery & Cafe the next morning. While the only decent coffee in Akumal has become something of a necessity for my aging brain, our corner table is also something of an office away from home for Trish, who knows so many people in the area that sometimes it’s hard to escape from all the hugs and hand-shakes.  With sufficient caffeine in my bloodstream I was ready to hit the reef with Marco, who had been keeping an eye on our little loggers through the Sargasso seaweed invasion that has affected coastlines throughout the Caribbean this year. He had already taken south bay unit out of the water due to a zip tie failure on the support rod.  I wondered if those dense floating mats had snagged the shallow unit, putting enough stress on the ties to break them?

Removing B4 from float line

Pulling B4 from it’s anchor rod.  (photo by Marco)

Since the B3 logger was already dry, that left only the Pearl at the north end of the bay. This B4 unit is the oldest continuously running logger on the project (it’s first underwater stint was back in March 2014) and is still running on it’s original Tinyduino core. Since the sensor is now well past my original one year design goal, I am tempted to retire it to the “bookshelf museum” as these old dogs feel like Russian tanks next to the new builds. But this project also embodies what the guys over at Boston Robotics distill down to “Build it, Break it, Fix it”, so I really want to see how long this DIY flow sensor will last.  As far as I know, this is the longest marine exposure test anyone has ever done with JB weld, or Loctite E30Cl epoxy on hardware store PVC.

And the little loggers did not disappoint, delivering a gorgeous four month record of water temperature, and tilt angle (my proxy for flow velocity)

Data from B4 Cave Pearl Data logger

This gave me another look at that June 13/14 event, and it must have been something! It almost doubled the relative flow velocities (probably more than that due to non-linearity, etc) and it pulled the mid-column temperature in the bay down by three degrees Celsius. To put sixty five degrees of deflection in perspective, here is a video clip of the relative motion of the floating logger, on the day we retrieved it:

This thing was brand new 4 months ago

That pivot joint was brand new four months ago…

I’m happy that the unit wasn’t ripped from it’s mooring by the storm, and that I installed the new super duper PVC pivot joints on that last trip. I am sure the old zip-tie swivels would have completely let go. In addition to the rough conditions, there is marine life colonizing all exposed surfaces.  When I took a closer look, the pivot joint was making some distinct “crunchy” noises – indicating something was trying to take up residence inside the tubing. The logger itself is now so hairy that I think the buoyancy is being affected. Hmmmm….

<— Click here to continue reading the story—>

Field Report 2015-08-06: Retrieve Drip Sensors from RioSecreto

Fernanda taking readings from a known survey station out to a drip logger

Fernanda taking readings from a cave survey station out to a new logger installation.

Fieldwork typically starts with retrieving the loggers deployed on the last visit. Over the last year Rio Secreto has grown to become our biggest single installation, so it is usually the first place on our visit list. It helps that they have plenty of dedicated staff to help out, and Fernanda Lases was able to join us for the day of collecting loggers and surveying their locations. As we gathered the drip sensors, we did manual counts of drips/15min, and took notes on the overall appearance of each installation. A couple had been knocked over by high water events, but most were still perking away happily where we had left them.

As this was near the peak of the local rainy season I was expecting to see some nice environmental response in the data with increasing drip rates.  But the majority of the records had slowly tapered off since March, and the temperature in the cave had risen slightly.  This is the record from drip logger #021, which was typical of the trend:

021 Drip Record

Note: The black line on the drip record is a daily (96 point) moving average. The temperature record is from the DS3231 RTC inside the logger housing, which only has 0.5°C resolution. My other tests have shown that they are much more accurate than the ±3°C that Maxim specifies.

I was not expecting to see reduced counts.  Trish mentioned that she has seen up to six month offsets between surface precipitation and cave drips, but that was in some caves on Vancouver Island.  I’m still scratching my head on this one as there is precious little soil in the area, and I would have thought the limestone was just too porous to provide much storage.

We put a pressure & relative humidity recorder in the cave on the last trip, which I was hoping would run for more than a couple of days this time around. I used a slower epoxy for the potting, with a couple of weeks to cure before going into the field. And the logger did provide a complete record, but as I feared the R.H sensor flat-lined shortly after being placed in the cave:

036 Pressure & R.Humidiy Probe

There was no direct moisture contact with the RH sensor.

There was no direct moisture contact with the RH sensor.

That’s a decent barometric record from the MS5805-02, and at least a hint at the results I might be able to squeeze out of the Mason’s hygrometers if that design can resolve the small wet bulb depressions we will see with humidity bouncing between 95-98% all the time.  The R.H. breakout circuit board was still clean & shiny under the epoxy, with no evidence of moisture intrusion, so I think this high humidity environment is just too much for those humble HTU21D’s.

We also had an underwater logger built with an MS5803, and with the barometric record from #036 we could derive the changes in the cave’s fresh water level:

Derived Water level from two pressure sensors

The spike in that record coincides with thunderstorms that hit the area on June 13-14th, and the local weather records indicate that 5 to 6 cm of rain fell per day during that period. It is interesting that both water level and water temperature return to their previous trends so quickly, and I am keen to see if that precipitation shows up in our other records from further down the coast. If I go all squinty, I can convince myself that some of the drip records were affected by the event, but most of them showed no effect at all.

Even baking under the full tropical sun, some fungus has managed to colonize the ABS plastic. That is one tough organism!

Even baking under the full tropical sun, fungus still managed to colonize the ABS plastic on the cap. That is one tough little organism!

Unfortunately, I can’t confirm the rainfall directly because both of my surface drip sensors croaked. The older 024 unit suffered an SD card failure (as it did last time) and the newer 034 unit drained it’s batteries rapidly when the ADXL345 started self triggering, which would have kept the mcu awake and drawing full power the entire time. The prime suspect in both cases is thermal cycling, with our hardy RTC’s showing some 60°C peaks. It’s worth noting that after replacing the sensor & SD cards I managed to get both loggers working so the 3.3v Rocket Ultras I am using survived the high temps. The DS3231’s had less than 10 seconds of drift after the ordeal.

High surface temps cook the drip sensors

Hopefully the new rain gauge housings I’ve brought along this trip will shield my little drip sensors well enough to prevent this from happening again. I’m sure it doesn’t help that I have the accelerometers set to fairly high sensitivity for this application.

Trish handles the big-picture analysis when we have so many logs to go through, but there are always plenty of  ‘mini’ experiments buried in the data for me to chew on: including confirmation that the loggers pin-powering their RTC’s during µC up-time saw coin-cell voltage drops between 0.03 – 0.1 volts.  And these units did not see clock drift significantly different than the non pin-powered units (~5-10sec / 4 months), giving me confidence that this method to reduce sleep current is worth adopting on more of my builds. (though I will be tracking things with a 4.7meg divider) The small drifts that I could confirm all seemed to have the clocks advancing, rather than loosing time.

Addendum 2015-08-22

Given all the μSD cards I’ve killed off in the surface loggers, it seems pretty incredible that some people have been re-flowing SD cards directly onto breakout boards. That requires bringing them up to about 200°C for a short interval. Clearly long term medium temperature exposures are not the same as short high temperature ones. I’ve also had a fair number of cards “shake” from their holders during a deployment, so this soldering idea got my attention.

<— Click here to continue reading the story—>