Field Report 2014-08-26: Old Flow Sensor Inspection

The drip sensor deployments left me with an couple of hours free time that evening, which gave me a chance to take a closer look at the flow sensors we pulled the day before.

From the same batch?

Different corrosion  although nuts & bolts were identical

The most obvious impact of the near marine exposure was the rust that had accumulated on the stainless steel bolts and ballast washers. (no spec on the bolts, but the lock nuts were 18-8) While they fasteners were all purchased at the same time, they showed dramatic variation in the amount of oxidization they sustained. I can only presume these are the result of the manufacturing process leaving scratches which acted as nucleation sites. Even the fasteners that suffered significant oxidization remained secure and they were relatively easy to remove once the surface rust had been brushed away.

Still clear, and nothing growing on the surface.

E-30CL still  clear, with nothing growing..

Some pitting on the JB weld surface.

Some pitting on the JB weld surface. I had some concern that the iron particles in the J-B weld might induce galvanic corrosion on the other metal parts.

Both epoxies proved to be far more robust than the manufacturers testing indicated, with the Loctite showing some surface fogging on two units, while remaining perfectly clear on the other one. The grey JB marine weld changed from a smooth surface to one with significant grit (~400 grit sandpaper?). I suspect the pitting is a result of the iron particles in their formulation rusting out of the the matrix, and I will try to get these puppies under a microscope later.  The rubber 0-rings were still in pretty good shape although they had a significant layer of bacterial slime on the exposed surfaces which I cleared off with a touch of isopropl alcohol. I suspect that any material with suflur in it is a banquet for critters the low energy cave environment, but the O-rings certainly look like they will survive for at least a year. (something for me to keep in mind with the bungee anchors though, as the older one’s are at 9 months submersion now)

Three of the four units pulled their 6 x AA power supplies into the 3.3 volt range (as read by the Atmel internal 1.1 vref trick) ; more power drain than my earlier tests had indicated for a 5 month run. But those bench-top tests were done too fast to include self discharge, without isolation diodes, and the real world batteries had been exposed to a relatively high humidity for the duration. (I have added 10 gram desiccant packs to the current crop.)

Perhaps the most interesting power consumption result was from the one unit that included a voltage regulator in the power supply module. I was unable to measure the cell voltage directly till a few days after the units were disconnected, but after the rebound period the AA’s supplying the NCP1402-3.3V Step-Up regulator were at 1.35v, while identical cells that had powered the unregulated Tinyduinos were at 1.4 v.  That’s a pretty small difference given that the nominal efficiency of the regulator is around 75%.

I will have to analyze the rest of the data later because the little net-book I have with me doesn’t have the gumption to chew on data sets of nearly 34000 records. So now we have the three older model Cave Pearls (and a pressure sensor!) cleaned up and in working condition… I think it’s time to put some thought into our next experiment!

Field Report 2014-08-26: The First Drip Sensor Deployments

Selecting test locations. (Photo courtesy of Monica Wnuk)

We were spoiled for choice of locations in a beautifully decorated system like this.  (Photo courtesy: Monika Wnuk)

Yesterdays success with the flow meters meant we could now turn our attention to deploying the new drip sensors. Trish had ongoing work in the Rio Secreto Nature Reserve section of Pol Tunich cave, included an interesting student project monitoring calcite raft deposition. A maintenance visit to those experiments was already on the schedule, so it was an obvious choice to use that site for our first drip sensor deployment as well. Even better, Rio Secreto had several capable science liaison staff, some of whom were conducting experiments of their own nearby.

Units are tethered so flood conditions don't wash them away.

When you do something for the first time in the field, you tend to improvise allot, especially when you are trying to apply the Goldilocks principle to something as random as cave drips. But Trish seemed to know exactly what she was after, and with the help of our guide from Rio Secreto, we quickly started locating ‘gotas’ that were ‘just right’.  We knew from Kayleen’s video that the water level in the cave could be highly variable so we tethered the units located close to the water level to assure that flood conditions would not wash them away.

 

ComparisonTest

They are intentionally placed at an angle, so that water does not pool on the sensor surface and cause spurious readings.

We also wanted to set the sensors in areas with multiple drips in relatively close proximity.  Trish has always been fascinated by the question of why some drips readily form deposits while others quite nearby,  possibly with similar chemistry, are much less capable of creating decorations. With this in mind our guide led us to a wonderful “stalagmite garden”  which had many excellent drip sites for comparison.  We found that the soft, slightly concave rubber bottom on the drip sensor housing made it very easy to perch the sensors without harm to the stalagmites. Several units were set up in short order, and as we exited the cave we could see their little green indicator lights blinking away in the darkness behind us.  Even with the help of lithium AA batteries, there is a fair bit of uncertainty about how long these units will actually run, so I was quite happy to hear our guide Fernanda say that she would stop in to check on them when she had the chance. The sensors are designed to light a red led when the battery voltage falls too low, but I told her to remove and disconnect them before that if they showed any other unusual behavior. These units were assembled with an assortment of cheap clone boards, most of which use the MIC5205 voltage regulator, and I will be especially keen to see the performance of the one build with the Rocketsream Ultra, as it’s MCP1700 delivered some very low sleep currents during my bench top testing.  My hope is that we get at least four months of good data from all of them before the batteries expire.

<— Click here to continue reading the story—>

 Addendum 2014-12-12:

This first deployment was a great success, which you can read about here.

Field Report 2014-08-25: Retrieve & Deploy New Flow Sensors

We started the day with breakfast at Turtle Bay Cafe, and once I had enough caffeine in my bloodstream to engage more than two brain cells at the same time, I reviewed the data on the SD cards from the overnight test runs. They all looked good.  Over breakfast we met up with Monika Wnuk, a multimedia journalist and documentary photographer from Northwestern University, who wanted to interview Trish for a water & development story she was working on. Yesterday, when she heard about our abbreviated schedule, she volunteered to help with the sensor preparation, and to provide shore support for our deployment dive. I was glad for the assistance, as two scheduled days were now being merged into one single operation.

Pre-dive planning with Bill, Trish, Monica & Jeff.

Pre-dive planning with Monica, Bil, Trish & Jeff.

Our diving field work almost always begins with a visit to Speleotech in Tulum, to see our long time friend Bil Phillips. Bil taught me to cave dive many years ago, and I still have much to learn from that remarkable man, who is without doubt one of the most dedicated cave explorers in the world. We also had the good fortune of meeting another good friend, Jeff Clark, who loaned me some equipment I needed for the days dive. The dive community in Tulum has always been generous to visiting researchers because they understand, more than most people, what is at risk with the rapid development that is happening in the region.  We all share a passion for protecting the caves as both a vital water resource, and as areas of natural beauty & wonder.

Checking for rotation, damage, etc.

Inspecting the old units for rotation, damage, etc.

With the kit sorted, we headed out to our main deployment site where I began to adjust the buoyancy of the new sensor units. With the new internal copper ring of ballast mass (45g), and heavier
aluminum battery holders, it only took 2-3 external washers to bring each unit to my target of 15 grams negative. This is slightly heavier than the last deployment but I am expecting any reduction in the tilt angle to be more than compensated by the 14bit 1g resolution of the new BMA180 accelerometers.  With
calibration out of the way,  Trish and I set off on the dive. High tide at the coast meant the system was experiencing very low flow, so we had a relaxed swim, with three new pendulums and a pressure sensor stowed neatly in the mesh bag by my side.

Old vs. New

New  vs. Old

Once at the site, the first task was to do a general inspection of the old units, noting anything unusual in my dive notebook.  After almost five months of submersion, there was plenty of rust on the stainless steel bolts and one of the units needed it’s anchor plate replaced.  Using the checklist I had prepared earlier, we swapped each unit in succession with it’s replacement.  In the calm conditions, percolation obscured our view a bit as our bubbles meandered around the ceiling of the cave, but it was still a very simple operation to exchange flow sensors.

Once the new units in place, we did a final inspection swim:

…checking that the new units were secure, with the X axis of the accelerometers oriented toward north.  While this is not strictly necessary with magnetometers inside the units,  I can use it as a rough confirmation of the compass bearings when I get the chance to do some proper data analysis later. I gathered the old sensors into the mesh bag and we made our way out of the cave.  I am not sure I can fully express the excitement that an inventor feels returning from a dive like this, but it’s very, very cool.

I think there is an ocean and a sunset in this picture. But at the time, we did not even notice it.

There is an ocean and a beautiful sunset in this picture. But at the time, I don’t think we even noticed it. (photo courtesy Monika Wnuk)

Back at the surface we had a chance to do a better visual inspection of the old units, which all appeared to be intact. I had some concern about the hull penetrations, as none of the epoxies were rated for long duration marine exposure. But the indicator LEDs were still piping on schedule, telling us that they were all still running.  Back at the dorms, we were equally thrilled to find complete data sets recorded on the SD cards.  (I will post more on the actual data after we have a chance to work on it.)

 <— Click here to continue reading the story—>

Field Report 2014-08-24: Assemble & Test all data logger units

Re-assembling the units

Putting the Pearls back together again.

Life intervened while we were still mid flight to Mexico, making it clear that this trip was going to be much shorter than expected. We would have only 5 days on the ground instead of the planned 12.
I started re-assembling the new drip sensors in the car as soon as we left the Cancun airport. With a 15 minute sampling schedule they would need a full 16 hours before data buffered in the eeprom would be written to the SD cards.  Later that day I put the flow sensors back together and loaded the final deployment scripts. With the flow sensors using a short five minute sample interval,  they only need 8 hours to complete a full cycle. While all the units had been tested at home, I always want to see at least one more successful run before a unit is installed into a cave.

One dead soldier before it even goes in the water

We had one dead soldier before it even went into  the water

The software for the unit with three accelerometers still needed some final tweaking, so I set to work on that (I had the epoxies clamped on that one till the last possible minute before packing) but it turned out that the ADXL345 would only read on the x & y axis.  As it was too late to fix the problem,  I just commented that sensor out of the code.  The resulting script will only give me data from the BMA180 and the BMA250 but that is still enough to bridge the data sets from our main installation site. Fortunately the three temperature pearl was running well, and I will be quite keen to see how those sensors differ in their behavior once the unit is in the actual cave environment.

Addendum 201409-10: Looks like I might have made a mistake with the ADXL345. Turns out that many of them are so badly calibrated from the factory that the Z axis is almost useless unless you change what’s in the offset registers. I live and learn.

Project Update: Gearing up for field work 2014-07-20

GluingSensorsThe last two weeks have been a complete blur of sanding, gluing & soldering, with simultaneous assembly of five next generation of flow meters, and six new dry cave drip sensors.  With the leftover beta unit I used for the long bookshelf power drain test, and the new 5 Bar pressure sensing unit, we will have 13 deploy-able Cave Pearls with us this trip.

Packed & Ready to go..

Packed & Ready to go..

Laying all this out, I realized that while our clothing for would easily fit into a single carry on, I could not remember the last time my wife and I traveled without two 49.99 pound suitcases of equipment. I make detailed component identification sheets for the TSA inspectors, who always return the favor by putting their own little pieces of paper in our luggage.  So far it has worked out OK.

Flow meter updates:

Since the last deployment, I have located better sensors for the flow meters and tweaked quite a number of things in the physical build. The new accelerometer is the BMA180, the only 14bit 1g accelerometer I could find on the market.  The venerable DS18B20 temperature sensor has been replaced by the Sparkfun TMP102, in my quest for completely interchangeable I2C sensors. Several new epoxies will also be tested in this build, including a very expensive Arctic Alumina thermally conductive epoxy to see if it will improve the temperature sensor response.

RosettaStone1

Does a logger with three temperature sensors know how warm it is?

I have built two special “Rosetta Stone” units for this deployment. One has the Ms5803-05, TMP102 & Ds18B20 temperature sensors, while the other has three accelerometers in it.  Data from these units will give me a head-to-head comparison of the sensor performance, and allow me to unite the data sets that we are generating with each successive build. The multi-accelerometer unit will sport the BMA180, BMA250 and the ADXL345. I think the 180 will come out on top, but there is always the chance that the increased sensitivity of that acclerometer will contribute more noise than accuracy to the overall performance.

The anchoring system now uses modular 50 cm long connecting rods, so we can hang the pendulums at different heights in the cave passage simply by adding or removing sections. (and they are much easier to transport in the luggage) Replacing the stainless steel rim bolts with nylon removed about 80 grams of ballast mass, so I have embedded a ring of copper inside the upper clam-shell. Hopefully this will improve the accuracy of  the data from the compass sensor.

Drip Sensor Progress:

Similar to one of the early flow meters, this would probably withstand submersion for quite a while.

Similar to one of the early flow meter designs, this would probably withstand complete submersion for quite some time…

Developed from a minimalist three component design, these are the first data loggers I have constructed where the mcu board possesses an always-on voltage regulator. Using only 3 AA batteries to power systems already sandbagged by the relatively high quiescent current of the MIC5205, really forced me to look for other ways to conserve power. Thanks to at tip from one 0f the advanced users at the Arduino Playground I discovered that the heartbeat LED pips are still quite visible when I use a whopping 10k limit resistor. This brings the 20mA LED currents well below 1mA, and if I sleep the processor while the LED is on, we save another 5mA for the pip duration. This little tweak will become part of my standard build from now on. Wherever possible I have replaced delay statements with brief MCU sleeps, and I am only reading Vcc once before the SD card writing process, since that is the only time the information becomes important. I originally conceived these units around the Sparkfun Pro Mini,  but early bench tests are indicating that the Rocket Scream Ultra could be the board of choice for future Cave Pearl data loggers.

<— Click here to continue reading the story—>

More Duracell battery failures this week

Houston we have a problem...

Houston we have a problem…

I have been doing run tests on the newest flow sensor builds, and this morning I had a little red error-state led waiting for me on a sensor that I had assembled the night before. I assumed I had botched a solder joint, or broken a wire somewhere while loading everything into the housing, but instead it turned out to be another Duracell battery failure.

The dead power module

The dead power module

Normally I would ignore this but this is the third Duracell battery failure I have had in a week. One is ‘meh’, two is just ‘bad luck’, but three…that’s a problem that requires attention.  My first though was a bad batch, but the batteries are not all from the same package, each being a slightly different variant of the marque.  Interestingly I have a log of the voltage curve during the start of this problem as I track the power supply via the internal 1.1v reading trick on the Atmel processor:     (mV on the vertical axis)

VoltageduringBatteryFail

The logger recorded about 500 samples before going into shut-down mode.  My overnight runs race along at one sample recorded every 15 seconds just to give them a workout, so this graph only represents about two hours of operation. But still, that is one weird looking voltage curve (…especially with those ‘gaps’, and the voltage was increasing??). Since I have not seen one like this before, I thought I would post it here. The power module has 2 banks of 3AA batteries, isolated from each other by shottky diodes, so this probably represents one set trying to keep the logger alive while the other bank went down hill (while venting internal pressure?) Then when the bad battery leaked enough, the alkali affected the others, and the whole power supply pooped out.

Another from this week

Another from this week

I don’t know why the logger stopped at this point either, as all of these voltages were within the units operating range, so I have to suspect that some kind of “burst” occurred around record 500, that pulled the voltage down below the 2800mV sleep trigger in the code. Certainly there was battery goo sprayed all over the inside of the housing.  The cells that leaked were still delivering 1.3 volts each when I pulled the unit apart in the the morning, so this certainly was not caused by running the cells dry.  My current hypothesis is that the new Keystone Electronics battery holders are so strong, that they are exerting pressure on the battery housing causing some kind of containment failure. Until I get a handle on this,  I will have to go hunting for a replacement brand before the next field deployment.

P.S. The Tinyduino logging platform seems to have survived the event, and is now ticking away in the basement for a proper shake-down run.

Project Update: Power Tests & Battery issues

3LPowerTestEnds

I’ve been waiting for this red LED to light for quite some time..

The power drain test of the spare TinyDuino based pendulum unit (identical to the ones we deployed back in March) has finally completed.  The unit captured sensor readings every minute from April 23rd to June 9th (about 62000 records), at which point I bumped it up to a 10 second sample frequency to accelerate the test, which continued until Aug 6th.  Three AA Energiser Max alkaline batteries (rated at 2850 mAh each) powered the unit, and given the extremely low current, I would guess that we used at least half of this power before we hit the 2800 mV cutoff I set in the code to prevent unsafe SD card writing. While the HMC5883L compass & BMA250 accelerometer had voltage regulators & level shifters on each shield,  the unit drove the DS18B20 temperature sensor, the AT24C32 eeprom, and the DS3231 RTC with the raw Vcc (including the A4/A5 I2C lines), showing that those components are robust enough to handle a fairly large voltage swing:

3LPowerDrainGraph
Y axis: Vcc (millivolts)         X Axis: # of sensor records recorded

In total almost 390000 sensor records were recorded to the SD card. With 42 eeprom page writes per cycle (and inefficient Ascii character encoding via pString) the 4k AT24C32 on the $2 RTC board was filled almost 9300 times in this test.  This is approximately the same amount of data we would process from one year of operation at a 1.5 minute sample interval…and the design supports six batteries: three more than were used in this test.

This opens up two possible directions for the physical build: I can reduce the housing size so that it only accommodates three batteries, or I can keep the current dimensions, with some reasonable faith that even if we get some pretty bad cells, the loggers will still last for a year.  For now, I think I will take what’s behind curtain number two because another interesting event happened this last weekend….

At least we know the solvent weld is water tight!

At least we know the ABS to PVC solvent weld is tight!

I set up one of the new drip sensors with Duracell batteries straight from the package, and I left it under the rain gutter down-spout to give it something to record from time to time.  A few days later I returned to find the housing bowed out from internal pressure, and found upon opening that one of the batteries had leaked quite badly.  I think the electrolyte in alkaline batteries is potassium hydroxide, and as you can see it corroded the aluminum battery holder significantly. The PVC of the housing was unaffected, and the excess fluid simply pooled in the bottom of the rubber cap.

test

The package said these batteries were good to 2017

Fortunately the divider that monitors battery voltage put the Arduino to sleep as soon as the failing cell brought the supply below the 3.4 volts required by the regulator on the Pro Mini clone used in this build. This occurred within one day of the start of the unit, and the only electronic component to expire was the ADXL345 accelerometer, which probably suffered from too much physical strain on its solder joints, as it was welded to the ABS cap which was strongly convex before I opened the clamp.

So what happened here? The unit was in a shaded area, so I don’t think it failed due to excessive heat.  The batteries were purchased from Amazon, so it seems unlikely that they were counterfeit.  I could not find any scientific surveys or lab tests to determine how often consumer batteries leak like this, but it does reinforce the benefit of testing the units for a couple of days on the surface, before we deploy them.  I wonder if we will see any more of this…

Drip Sensor Update: The Gamma Build

I suggested in the last post that a new build usually comes together on the third iteration, so I though I would post a few photos of the current drip sensor, to show how much they changed in that short series:

Alpha, Beta & Gamma builds

Alpha, Beta & Gamma builds of the Cave Pearl Drip Sensor

That Alpha would only detect drops in the very center of the housing, and often registered double hits because of the splash back from the complex surface topography, which also caused a buildup of water on the surface, further interfering with the signal. So I removed the pressure and temperature sensor wells from the Beta, and used a heat gun to bow the surface and shed water.  I hand sanded the pvc to make the strike surface thinner and more responsive, which increased the “reliable” sweet spot to the diameter of the circle you see on drawn there. This worked well, but as you might imagine removing that hour long fabrication step rose to the top of the priority list (if necessity is the mother of invention, laziness is surely the father…) I also did not want to penetrate the housing for those standoffs if I could avoid it because this device has to maintain integrity with constant water impact at exactly that spot.

JB Plasicweld bonds the accelerometer to the cap

Plastic-weld bonds the sensor, preserving the integrity of the housing

My solution to both problems was to solvent weld a four inch knock out cap to the top of the hard pvc shell.  The green color you see there is ABS to PVC transition cement, that I learned about back at the beginning of the project when I was mangling Leggo bricks for internal scaffolding. The ABS is translucent, so the LED no longer needs a portal of clear epoxy because the light passes right through.  The knockout is thin and stiff, which eliminates all that sanding and improves the response of the accelerometers so much that now the device even responds to loud noises, and almost the entire surface sensitive to the smallest drip.  As a result I now have to tweak the settings to reduce the sensitivity so we don’t get too many false positives.  But on the third build of a new sensor, that’s the kind of problem you want to have.  I though the flat surface might resurrect the water pooling problem we had on the Alpha, but in the end it just ended up being a lesson in how I still miss things that are really darned obvious, because simply setting the drip sensor up with a slight tilt will shed the water without affecting the operation at all.  (I am just glad I realized this before some three year old came along and pointed it out to me…)

These guys are only $1.5 each, and are much easier to work with than the plastic micro SD adapters

Much easier to solder those jumpers now.

Wirefold

The trickiest part of the build is routing the wires to reduce strain. Thin silicone 26awg wire helps quite a bit there.

There have been a few improvements to the logging platform as well.  I found these really inexpensive  raspberry pi SD card adapters that are mounted on their own pcb, which neatly solves the melting problem that the cheap plastic  adapters give you if you linger too long with your soldering iron.

Drip Sensor Gamma BuildSo this is the new baby, and I am now running burn tests on loggers using a few different clone boards, including a Rocket Scream Mini Ultra, as they have some very interesting power saving features that might significantly increase the operating time, while keeping the overall simplicity of the three component design.  I expect a few more issues will arise in testing, so I will hold off posting the code to Github, till I am sure that they are behaving properly.  I am still a bit stunned that these drip sensors came together so quickly, but perhaps the last six months of work on the flow meters had something to do with that. 🙂

<— Click here to continue reading the story—>  .

Addendum

Adafruit just posted a video from the National Science Foundation showing how water droplets move on various hydrophobic surfaces.  Way cool…

Addendum 2014-12-11:

We deployed our first batch of these drip sensors in August, and when we went back to get them in December, we were delighted to find that the first real world run was a resounding success.

Building your sensors & underwater housings from scratch

I reviewed my build journal recently, and found enough themes in there for another bit of bloggy catharsis.  No one should mistake this as the advice of an expert in anything, as I have a long way to go before I start collecting karma points at the playground. But at least I can claim that these ideas are well tested, because as the saying goes, I never make the same mistake twice – I make it 5 or 6 times… just to be sure.

Prototyping:

Only RTC & interrupt lines get soldered to the mcu.

Only RTC, SD, & interrupt lines soldered to the mcu.

Join the forums: Everyone says read the datasheets first….well from this beginners perspective that’s B.S.  Even when I started to understand all the terms I was reading, it was still a major leap of understanding to realize what the information in the data sheet implied… (and I still go through it with each new sensor)  If you want to get rolling on something start with the discussion forums, because it’s likely that someone there has already walked down your path, or one very close to it. Thirty minutes Goggling Sparkfun, Stack Exchange, or searching through places like the Arduino Playground, will get you farther than several hours reading the datasheet. Of course you will eventually end up doing that too...just don’t start there.  Often the forums lead you to some well commented GitHub code examples.  It is so much easier to understand the data sheet, when you have a piece of relevant code in front of you at the same time. Datasheets are written solely for corporate electronics engineers, because unless you represent the 10 to 100 thousand unit MOQ,  you simply don’t show up on company radar.

( Sometimes it even seems that a company deliberately tries to hide the functions built into their own ICs from the maker community.  For example: the mysterious Digital Motion Processor (DMP) functions of the invensense 9150. I have yet to find a single example of an Arduino script accessing these features though the 9150 is commonly used in quad copters. I’d have thought that having Euler angle & 9DOF quaternion calculations done for free, would have drawn some serious attention from those those guys.)

You need one completely “trusted” set of kit:  Which you will pay dearly for, but you need it to test out each new component you are thinking of adding to your project. (I still haul out my original Uno for this from time to time, because the darned thing is virtually indestructible…)

More than a few of the cheap eBay boards are D.O.A.

About 1 in 10 of the cheap eBay boards are D.O.A. This  ADXL345 board had a pretty typical alignment skew: X & Y axes were ok, Z did not read.

Connect new parts one at a time:  Do not add to your grief by connecting two new untested pieces of equipment to each other at the same time. (like, for example, a new FTDI board, and an Arduino clone of off eBay….grrrrr….) I am slowly learning what it is that you are paying for when a given sensor module sells for $20 from a trusted source like Sparkfun and $2 from China.  It’s probably not the components (although I hear there are fake IC’s out there) but something else that’s just as important.  If you watch the Tiny Circuits promo vid, around the 5 minute mark you see a person manually re-positioning the components that the pick and place machine laid down, and then around 7:15 you see someone testing each board before shipping. At this point I have seen enough of the eBay stuff that I am pretty sure that these two steps are never done on the clone boards. So If you go down that route, you are implicitly agreeing to do those jobs yourself (in addition to cleaning off all the excess flux left on the boards…)  This is not a problem when I am just hacking something together, and then testing it on the bench (or in the bathtub…) but not a risk I will take lightly for the fieldwork units.

And while we are on the topic of those cheap modules, they almost always arrive with connections broken out for both SPI and I2C, but you will likely use them as one or the other, meaning that you often need non adjacent pins soldered onto the breakout. I used to fiddle with crappy metal “helping hands” things (now replaced by a far more effective Panavise Jr)  to get those single pins soldered into place, but now I simply use a bread board to hold the pins in the correct locations before soldering:

Many thanks to my bother Mike for showing me this soldering technique!

Many thanks to my bother for showing me this soldering technique!

Modularize your designs:   Breadboards were never very practical for me because most of my projects get bashed around, and are supposed to run while the whole thing is in motion. But even after the prototype stage my sensors are not usually soldered directly to the data loggers. Yes, it’s a pain constantly making custom all those custom interconnect cables, especially since I color code everything – I2C bus, interrupt lines, …everything. And once you get an early prototype working, immediately build another one, and do all of your tests, etc., on the second unit, with the first working unit just sitting on the shelf.  Then if something stops working, you can use process of elimination with the known good modules to isolate the cause of the problem quickly. Often this technique helps me identify that the problem is actually in the software, and not with the hardware at all.  But I am still wading my way through the wonderful world of crimp connectors for ones that will prove robust enough for fieldwork. (note: I’ve adopted Deans Micro Plugs for my current builds…)

Coding:

Hit the verify button often, even when you make “simple” changes:  Don’t wait till you have been working for half an hour because humans can only remember 7±3 things (for this human, even less than that…)  You will be backtracking allot and error messages in the AVRc programming environment are usually about as helpful as a baby crying. You know something is wrong, but often you cant figure out what the problem is from the feedback it give you.  Something as simple as: “Hey man,  you left out a semicolon at the end of line 17” can be reported with anything from a one line “Expected X before Y” error to twenty (or more…) lines of cryptic compiler faults. Sometimes Google is your only friend when this happens.

The utility of a piece of code is directly proportional to how dangerous it is to use: For example: Global #define statements are tricky…especially when you are using libraries, because you never know when you have tried to “re-define” something hidden in one of those libraries. Interrupt handling is another example of something tremendously useful, and really easy to screw up when you have more than one sensor generating interrupts in the same piece of code, at the same time.

Software faults almost always show a repeating pattern of behavior: At least that’s been my experience so far. Bad wiring, or some other issue with the physical build, (like a v. regulator thermaling out…) hangs the system in a way that is much more unpredictable.  Of course, finding those patterns is a heck of a lot easier when your project is a data logger in the first place…

Physically putting “stuff” together:

Third time’s a charm:  If your alpha works at all; that’s a great success. The holes will be in the wrong place, you will use too much epoxy, and it will be a clunky octopus of dodgy wiring. But even if it only runs for a single day, it has done the job of showing if the idea will work. (and they make such stylish book-ends…right?) The second build lets you sort out the right physical locations for all the components, and eliminates that hour of laborious hand-sanding that you really did not need to do.  This is also the model that lets you work out most of the software bugs, because it is usually much easier to open & close without breaking fragile jumpers all the time.  But for me at least,  it’s the third build that usually comes together in a way that feels right.

This type of bench vise is worth finding. I picked this used one up for $10, and it has been fantastic for holding parts together while epoxies cure & solvents weld PVC parts.

This type of bench vise is handy. I picked this one up for $10, at a garage sale and it has been fantastic for holding parts together while epoxies cure & solvents weld.

Adhesives & epoxies can be one of the most expensive components in your physical build:  (I wasn’t expecting this one) I go through a fair bit of Loctite E30-CL  (or E00CL for better PVC bonding) for my hull pen-etrations, and I love the fine control of the applicator gun.  But the mixing nozzles alone are more than a buck each, and they are elegantly design to waste a large volume of that precious epoxy with every use, which then solidifies inside the nozzles turning them into expensive single use devices.  That irritated me enough to start experimenting, and it turns out that Goof-off  (mainly xylene, and 2-ethanol) cleans out those applicator nozzles relatively easily. (Acetone would also work) I usually buy my adhesives from Zoro Tools. (note: their product search function is terrible, I use Google Shopping to hunt for stuff on Zoro’s site)  There are plenty of cheap Loctite sellers on eBay, but many are hawking expired lots.  In a pinch, I fall back to good old JB weld, and I use PlasticWeld putty absolutely everywhere because it nicely adheres sensor boards to both PVC & ABS in a way that is strong, but still removable (sort of..) if you have to repair something. Be careful not to bridge contacts with your adhesives on really sensitive sensors.  Most of these epoxies do not conduct electricity, so I use JB weld to shore up weak jumpers on I2C lines all the time. I have had problems with some sensors not working after gluing, probably because the cured epoxy added capacitive effects to the circuits. I have also started experimenting with 3M’s VHB double sided tape (for low surface energy plastic & acrylic substrates) to see if that is easier to work with. (Note: after testing the VHB did not work as well on the pvc as the standard 5lb mounting tape)

(…and if paying an “arm and a leg” for your adhesives wasn’t bad enough, those cheerful, brightly colored labels hide some fairly dire warnings about skin contact literally causing cancer, or vapors so dangerous that use of the product anywhere other than a open field is guaranteed to give you brain damage & significantly harm your … uhhh … “reproductive capacity”. Don’t believe me? Get a 200x microscope and read that fine print for yourself… just make sure you do it some place far away from stoves, heaters, electric motors and all other sources of ignition…like soldering irons 🙂 )

The shop course that you only took in high school to boost your marks will finally start to come in handy:  Turns out that working with PVC is almost the same as working with wood, so I was surprised by how much time I spent scoping woodworking forums while figuring out how to cut weird angles and then re-assemble things.

BigIron

An angle grinder pulls the rust off of old school tools in short order.

You can build anything with three pieces of  “iron”: A good soldering iron, a drill press, and a 13″ bench top scroll saw (which cuts PVC and trims circuit boards beautifully).  I have not regretted for one second putting down $100 for my little Haiko from Adafruit. (Wish I had bought the thing at the very start of this adventure.) But my other two irons are vintage Sears Craftsman models which cost next to nothing because when I bought them, they looked like flea-market clunkers from the days before plastic was invented. Cleaning them up was worth the effort because they weigh a ton, and are rock solid stable. In fact, these days I actually go out of my way to get old tools for my workshop. So if you see me one morning, at yet another garage sale, don’t be surprised if my hands are full of weird drill bits,  strange clamps, and other rusty lumps of metal that are even older than I am.

And last but not least:

I know everyone has their own taste, but I find that Pandora’s “Chillout” channel (in the Dance/Electronic genre section) just has a really good vibe for working on the bench. Spacey electronica wallpaper-music smoothly delivers the hyper focus I need to sustain a long soldering session, especially when the clock ticks on into the wee hours… .

Addendum 2014-07-29 

I have another one to add to this list but it stands outside of any category because it seems to apply equally. I call it the ‘Principal of Equivalent Annoyance’, and it’s just the observation that the dumb little things on a project take up at least as much time as the big important things. For example: when you connect allot of parts together, the physical behavior of your wires matters more than you’d expect because the difference between good insulation and stiff hard plastic puts quite a bit of pressure on the wires when you fit everything into the housing. Once and a while I am left with some excess lead wire from a sensor and when I open up the left over multicore cable I sometimes find a meter or so of the perfect wire.  Soft silicone insulation, 12 or more strands, no tinning… the stuff is fantastic to work with and is so flexible that it doesn’t stress the connections. Of course it never has any identification marks on it that might lead me to the source.  So I am slowly working my way through every brand of hook-up wire on the market, searching for this holy grail, and wasting more time on it than I even want to think about.  I suspect this is kind of thing happens on every project.

Addendum 2014-09-13

There are a few contenders so far in my ongoing search for “the perfect wire”: 

Adafruit sells 26AWG in multiple colors. This is my current favorite wire because they sell it in easy to handle packages (~0.95/2m) and they have the only grey & orange colored stuff I have found so far. It probably sounds silly to mention the packaging, but from other vendors like Hobby king, or eBay, you can expect to spend at least an hour untangling the spaghetti when your wire arrives. Their 30AWG is 0.70/2m, and is very handy for running jumpers across the surface of a breakout board.

Sparkfun sells their red & black silicone hook up wire at a good price, I just wish they had more colors.

Cal Test Electronics CT2956 Test Lead Wire:  (24awg $9 for 10 m, green, white) It is soft multi-strand bare copper and there is some variability in the stiffness as they seem to use different stranding depending on the color you order, but all of it is much more flexible than pvc insulated wire.  The insulation is nice and thin, so it crimps well into the standard 0.1″ connectors, and it is a bit stiffer than the Turnigy, so it will hold a bend you put in the wire.

Turnigy Soft Silicone Wire: (24awg $.60-$1 per meter, red, black, yellow blue) which is popular in the RC plane/Quadcoper crowd. Multi-strand “tinned” copper, with lots of colors. The very soft insulation is about twice the thickness of the Cal Test, but it still crimps nicely. (note: Hobby King does not let you mix warehouses on orders…)

There are two drawbacks to to using really soft silicone wire: You can not push the female crimp connector ends into the housings as you normally would with jumper wire. You have to dig in and pull the connector through the enclosure with tweezers.  The other issue is that because the silicone conducts heat so much faster than pvc, you will burn your fingers more often while soldering if you are not careful. In return,  you can stuff as much spaghetti as you want into a very tight housing without putting pressure your the solder joints in the process.  The wire stripper from Pololu, works well on both types of wire.

Addendum 2016-03-11

More eBay vendors for silicone jacket wire are appearing over time, and they are a great place to find a wider selection of colors like brown, purple, & pink. With module & jumper builds like mine, you need as many different colors as you can get your hands on. 

Addendum 2015-01-09 

I received a 10″ craftsman 21400 band saw for Christmas this year. And just like the night & day transition that occurred when I went from a crummy soldering iron to the lovely Hakko, my cuts have now become more accurate and virtually effortless.  No more risking my fingers trying to cut pipe on the table saw! And free-handing with a band saw accomplishes 90% of what I could do with the the jig saw as well.  If you can only afford to own one cutting machine, then I am now convinced it should be a bench-top band saw. Wish I had it from the beginning…

Addendum 2015-01-16

Looks like I am not the only one who is irritated by the waste generated by the Loctite 50ml nozzles. The folks over at RCuniverse confirm that acetone makes a good solvent for cleaning out the mixing tubes, and that its easier to do if you remove the white plastic mixing baffles first. They also mention that putting the nozzles in bags in the freezer will prevent the epoxy from setting, so I might try this trick to see if I can get the tubes to “self clean” by gravity. Looks like All-Spec is the cheapest source for the nozzles, but I keep finding “generic” mixing nozzles on eBay that claim they work with both 3M and Loctite adhesives. Haven’t tried them yet though.

Addendum 2015-02-02

I bought some of the cheep MA5.4-17S 17 element mixing nozzles, and although the fit perfectly onto the 50ml dispenser, they were much shorter than the 15cm, 20 element, Loctite 98623 nozzles I usually use:

TopCheap_BottomLoctite

I thought this was a good thing, as I hate wasting all the epoxy that is left in the dispenser after every use and these smaller ones only retained 1.68ml in the barrel. I did some test pours under nearly identical conditions, and the smaller nozzles left a significant “swirly” pattern when the cured epoxy was examined close up a few days later:

Left side: Loctite Nozzle Right Side: Short Generic Nozzle

Left side: Loctite Nozzle                                                   Right Side: Short MA5.4-17s nozzle

I am interpreting this as a density gradient formed by inadequate mixing in the shorter nozzles. The epoxy has still set hard as a rock so I am not sure if this compromises the integrity too much. There are also listings for MA6.3-20S and MA6.3-21S nozzles, and I will try some of those next time.

(Note: the MA6.3’s worked great, as did the $10 applicator guns on eBay)

A New Drip Sensor for Cave Research

 … we interrupt this hydro-metric-flow-thingy blog to bring you an important announcement …

Trish wiring up our home made drip sensors in 2004

Trish installing her drip sensors in a B.C. cave

Back in 2004, my wife did a post-doc at McMaster university with Dr. Derek Ford & Dr. Henry Schwartz, that focused on drip hydrology in caves.   She set up a monitoring network of sensors to determine drip rates & water chemistry and we were very fortunate at that time to have the help of Kenrick Chin, who built custom data loggers for the project. While I was not capable of addressing the electronics back then, I did manage to create some DIY tipping-bucket sensors, which Trish combined with thermistors & conductivity sensors into a portable assembly. The resulting contraptions yielded an impressive data set, but we had minor problems with moisture/corrosion, and major problems with cave critters who absolutely love chewing on plastic cable insulation.

So with that as back story, I am very pleased to introduce a working beta of the new model “D” line of Cave Pearls, for research in “dry” cave environments.  This unit is a drip sensor, prototyped with the ProMini based data logging platform discussed in my previous post, bringing the total parts cost in around $25 (+ 4 hours building time):

.
The unit sleeps until the vibration of a drip impact triggers a hardware interrupt that wakes the Arduino, which then updates a counter variable, pips the LED, and goes right back to sleep.  When the RTC interrupt fires (which is adjustable) the counter data then gets written to the EEprom or SD card.  I actually had multiple sensors on the Alpha unit (temp, pressure, etc) but the back-splash from the protruding sensor wells on top caused spurious drip count readings. So, like the under-water models, the dry cave sensors will have different builds to record the various aspects of the cave environment.

DripSensorBetaWhile re-examination of the 2004 cave sensors has been on the back burner for a while, this drip counter was a direct result of my search for better tilt sensors for the flow meters.  Those investigations lead me to several projects building DIY seismometers, and it occurred to me that if accelerometers were that sensitive, it might be possible to detect drips falling onto the housings.  In the course of evaluating the ADXL345 (and rejecting it from the flow sensors due to relatively low bit depth at 2g) the well commented code posted by wyojustin & Kevin Stevenard introduced me to tap sensing registers.  The “Eureka” moment that followed spurred on a week of furious prototyping, and this new drip counter is the result.  I still have a bit of refining to do as I optimize timing & sensitivity settings, but as you can see from the video, the basic idea works down to around 10 cm fall distance. If I set the sensitivity high enough to catch drips closer than that the acclerometers internal noise starts to generate false positives.

While not directly related to the flow meters, I have a feeling that this kind of positive feedback/serendipity is going to happen with increasing frequency from here on, so I would like to officially retract my statements in the last post, where I implied that the Pro Mini based data loggers were not going to affect the project that much.  A week of soldering later, and I can say with confidence that having a $10 datalogger to prototype with is a huge step forward, even if the specific build never goes into the field.

 … we now return to the regularly scheduled  pend-aqueous-hydropod programming …

<— Click here to continue the story—>

Mods to the ADXL345 on a regulated bus.

Mods to use an ADXL345 on a 3.3volt bus. On newer builds I now only use boards that already have a 3.3v input (Geetech, CJMCU-105, etc.)

Addendum 2014-07-15  

I almost forgot… for those of you wanting to build one of these drip sensors, the ADXL345 was one of the cheep boards from eBay . When all you want is tap sensing, things that the clone boards are notorious for like offsets, drift, etc. don’t matter that much.  Most of these breakouts come with a voltage regulator already installed, but since I was driving the I2C lines from the pro mini, which already has a 3.3v regulator, I had to de-solder the vReg (& its isolation cap.) from the breakout board and then bridge the vcc line straight across. (see top)  And since I2C lines are running through the RTC breakout , they already have 4.7kΩ pullups on them, so I also had to lift the little smd resistors from the accelerometer breakout board. You can see the empty solder pads, just below the chip in the photo.

Addendum 2015-01-10

Just stumbled across someone trying to use capacitance and IR to detect drips without interacting with the drip itself. I think the capacitance idea has legs, but I would worry about all the condensation you get on objects in a cave giving you grief with IR approaches.