Category Archives: Lessons learned.

If it’s true that people learn from their mistakes, then I guess I qualify as ‘a life long learner’.

The 2017 Cave Pearl Project ‘Year in Review’

Run times for Cave Pearl loggers without RTC pin power (0.25 mA) and with RTC pin powering (0.1mA). Dashed lines are projections to the point where the unit would have performed an automatic low-voltage shut down.

It’s been an intense year for the Cave Pearl Project, with several major advances. Almost all of the the units in the field sleep at 0.1mA or less with the RTC pin-power modification, so we can finally push the deployment cycle out to a two year rotation. Somehow I doubt that Trish will be able to wait that long for her data, but at least now we have the option of just leaving things in place if we need to when the fieldwork doesn’t go according to plan.

I also started experimenting with SD card shut-down this year, and early records from those prototypes are in, suggesting that they will run for at least four years. However I’m not going to call this technique ready for prime time until we see at least twenty units deliver more than a year of saved data.  Accelerated bench tests can only tell you so much about how a unit performs in the real world.

Extended lifespans mean that we can leave loggers with other people so they can do exploratory deployments in locations we can’t get to. Natalie Gibb from Under the Jungle sent us a brilliant video from one of those deployments:

We couldn’t have produced anything that polished with our hokey little point & shoot cameras!

I didn’t have much time to update the blog in 2017, but even with only six posts the traffic continues to grow, and we are approaching 100,000 unique IP’s. I still consider this blog an experiment, since I had no idea how much traffic a nerdy rant about data loggers would get when I fired it up in 2014. I think we’ve passed some kind of threshold this year, because traffic analyzers are now hitting the site regularly, and the comments are being spammed with a heap of “monetize your blog” garbage. Are there really people out there willing to annoy that many people for a $2.50 average CPM? (Ok…rhetorical question….)

I mentioned in the 2016 review that European countries were occasionally passing the US in the daily stats. At the time I thought that this was just an artifact of some mention in a forum thread, but through 2017 it started happening more frequently and now the US traffic is really starting to fall off. This struck me as odd, given all chin wagging you hear about STEM education initiatives in the burbs.

So I did a some checking with Google Trends and it’s not my imagination; something happened in 2017 that reduced the number of searches for Arduino in the U.S. while those same searches are rising in the rest of the world. Sadly, I don’t think you have to look very far to understand what’s happening when the president states that dismantling “the DEP” is one of his goals, and then appoints a director that sued the agency several times to carry out that plan. The relationship between the executive branch and the research community is so deeply dysfunctional right now that the Center for Disease Control has been ordered not to use terms like “evidence-based” or “science-based” in official documents.  Is it possible that folks on the street are getting the message that science has somehow become un-American?

I’ve no doubt more will be written about this epic estrangement, and in keeping with the freaky zeitgeist we’re introducing the Cave Pearl Projects first annual “Ugliest Deployment of the Year” contest.  A worst-in-show collection of from coastal outflows, mangrove swamps and hydrogen sulfide pits:

Sponges growing on a flow logger after 27 month deployment in a hydrogen sulfide cave. This unit smelled so bad that it attracted all the feral cats from the neighborhood when we brought it home for cleaning.

Bio fouling on an estuary unit after only 6 months. It was deployed as a floater, and I was quite surprised that it was still buoyant enough to function. A swarm of stinging sea lice made this retrieval particularly memorable.

Pressure unit after 6 month stint in a cave feeding a coastal lagoon. While this units’s not that ugly per se, that outflow fed directly into a swimming area that sees hundreds of tourists per day expecting the joys of “crystal clear spring water”… Nuf said?

Honorable Mention: This was actually reef logger from 2015, but I’ve included it here because this was the deployment that really defined the “fugly enough to be cute” benchmark that future deployments will have to measure up to.

To qualify, these dogs had to run through the entire deployment because without primary data to support your hypothesis – you’ve got nothing. (no matter what they look like on the outside…)  You can’t just wave your hand and create alternative data because in research that’s called lying.  Somehow, this not-so-subtle distinction has become blurred in America, where science itself now seems to be under attack.  It’s remarkable that someone with such an obvious facility with numbers could fail to appreciate how economically valuable the insights from research can be.

I’m genuinely concerned that if the the U.S. continues to turn inward, choosing a path towards declining health and social outcomes, it will become less appealing to the kind of entrepreneurs that built the country in the first place. The winners-take-all legislation rolling out of Washington these days threatens to eliminate the things that make risk-taking innovation possible.  An important nugget that policy wonks always seem to forget was eloquently summarized in Jennifer Romolini’s Career Advice article:

“You’ll Suck at Everything the First Time You Do It.
You will probably suck the second and third time too.
Don’t get defensive about this; don’t decide that you should never do the thing again…”

This was my first attempt to build a data logger, and yeah, it was pretty crude. I was lucky to get 24 hours on 6xAA batteries and it was held together with hot glue. My current builds are probably going to pass four years at 100-200 ft depth.   If I gave up when it still looked like this piece of junk on my kitchen table, I’d have no idea that was even possible.  Even today I get criticism from engineers who say the project is pointless because it’s not ‘state of the art’. But they don’t know where we started, so they also have no idea how far it will go. The same holds true for your project, whatever that might be.

Without a functioning a social support system, only people with rich parents ever get that second chance. It’s no surprise that bankers could care less about this, since they don’t have to produce anything new to make their money. But these days it seems that silicon valley is also willing to focus on corporate profits rather than innovation, even if that means looking the other way while legislators dismantle systems those propeller-heads needed when they were just starting out.  If people thought WannaCry was bad in 2017, just wait till they see what AT&T and Comcast does with the net neutrality rollback. (And they called the bill the “Internet Freedom Act”…nope…that’s not Orwellian at all… )

America will not succeed in the 21st century simply by making stuff cheaper with learning systems that remove human beings from the process.  If that does happen, who’s going to have enough money to buy the consumer goods that keep the economy rolling? But they could reclaim that edge by re-imagining how technologies interact with people and the environment. Of course, no one from old money will support policies that might upset the profitable status quo, so the last thing they want is an educated population changing things with a bunch of new ideas.

Unfortunately, the ‘new’ money created by the internet revolution is now acting like the old.  With few exceptions, the major tech players are willfully dropping the ball, spinning up proprietary bank crypto-currencies when they could be using tools like blockchain to build efficient resource allocation systems for new energy sectors that use considerably less water and create far more jobs. It’s time for the de facto stewards of the internet to accept responsibility for their creations, which are becoming the real arbiters of trust, fairness and justice in our society.  A good start along that path would be to make the voting system more secure from external threats, and from internal ones.

If the tech sector fossilizes into a corporate platform oligopoly, scientists / inventors get lured away by countries 1/10th the size, and congress bends the knee to carbon energy’s hired guns and PhRMA’s lobbyists, then the US will essentially be handing the future to China.  But none of that needs to happen. Especially if people take a moment to think about the role science, innovation and invention played in ‘Making America Great’, and then vote for leaders who help those things to flourish in the future:

The 2016 Cave Pearl Project ‘Year in Review’

That's a chain with 24xDS18b20 sensors pulling only 0.15 mA sleep current. Woot!

That’s a chain with 24 DS18b20 sensors pulling only 0.15 mA sleep current. Woot!

We made great strides in 2016 with development of new calibration procedures and continued refinement of the housings & connectors for multi-sensor builds. At this point I’ve cobbled together more than 130 loggers for the Cave Pearl Project. While I still have a way to go before I reach Gladwell’s 10,000 hours, we haven’t spent $71 million, and I think I’m starting to get the hang of it. Of course, I felt that way last year too, so perhaps that’s just as good as it gets when you are figuring things out as you go along. It’s not unlike that moment when your boss calls you in for the annual performance review, and then starts the meeting by asking you to rewrite your job description, again, because less than half of the work you actually did last year was in the previous one.

Despite all that building, the number of loggers out on deployment leveled off around sixty five as newer units were often used to retire earlier generations out of the fleet. It’s worth noting that those old dogs are all still running, but field logistics have reached the point where we can’t expand the project unless everything on active duty has more than a year of operating life.  Fortunately the latest Pearls are consistently delivering sleep currents around 0.1mA , so builds with more than three AA cells are probably going to run for two years or more. 

No one ever seems to bother our equipment installations...I wonder why that is?

No one ever seems to bother our equipment installations.  I wonder why that is?

The temp. string loggers are currently the most complicated builds, but they have been delivering some very impressive data sets. Despite their flagship status, most are deployed in sites with a nasty mix of salt water and hydrogen sulfide that’s been destroying the marine grade stainless steel anchor weights. As they take a while to assemble, on-site service and redeployment will be the norm for  those logger installations until we get more of them into circulation. Rapid turn-arounds like that were one of the original goals of the project, but that’s only possible in practice because none of our current sensors are affected the bio-fouling that accumulates over a deployment.

There are plenty of new sensor combinations on the todo list as we round out the hydrology tool kit with more cheap & cheerful ‘tattle-tale’ builds. The recent discovery that you can stretch the I2C bus out to 10’s of meters has opened up many new sensor possibilities.  Long-cable MS5803 pressure sensors are now being put into service for borehole logging and if I can reduce the thermal equilibration time, that sensor also shows promise as part of a drop profiler.  From very the beginning, we’ve had our sights set on a cheap reliable CTD with extended logging capability.

Our oldest monitoring stations are reaching the three year mark, which means we are starting to see repeating annual cycles in the data. As an example, here is the temperature and tilt record from one of the reef stations in Akumal Bay:

annualcycles

The occasional hurricane means there are some lovely ( …to a hydrologist  🙂 ) event records peppered throughout, but the real prize is the baseline data that you only get from multi-year deployments. From the builders perspective, those long records also give me a sense of the inter-unit consistency, which is looking  good despite substantial  changes to the loggers over time. As the project matures, more of my time is being dedicated to sensor calibration and normalization.

Though we rarely have enough visibility to capture it with our little point&shoot camera, the underwater procedures are getting smoother too:


Cave Pearl data loggers

At  ~6000 unique IP’s a month, traffic continues to grow, though I will have to manage more than one post per month if the project is ever going to be a real contender for science geek fame.  The UNO logger has now passed the RTC post in the daily rankings, which shows that the number of interested beginners is also increasing.  The US dominates the overall traffic at (with about 50% of the total), and Germany leads the European traffic, often coming in second only to the US.  If other DIY sites are seeing a similar trend, then I think it’s pretty obvious why Germany continues to be the economic powerhouse of Europe

Many years ago I did a short stint as a high school science teacher and some of my good friends are still in the trenches.  Conversations with them often highlight two things:  How their job now depends on standardized test results (so they don’t have much room to change the curriculum), and that resource budgets are shrinking to zero. These are dedicated people who are more than willing to go the extra mile if they can get their hands on the right material.  So I’ve been working to expand the online tutorials into a set that would help a teacher add Arduino based lessons to their curriculum, even if they have do it out of their own pocket:

2016 was another great run of Trish's Instrumentation & field methods course. It's impressive with how quickly the students pick up the Arduino, and everyone got their final projects running though most of them had never held a soldering iron before.

2016 saw another great run of Trish’s Instrumentation & Field Methods course. It’s impressive how quickly the students pick up the Arduino system, and everyone got their final projects running though most of them had never used a soldering iron before.

  1. Build your own Arduino classroom
  2. Arduino UNO Logger for Beginners
  3. Using the UNO for Data Acquisition
  4. ProMini based Data Logger (update)

I’m sure that most people missed the significance of the DAQ post when it came out, but in terms of teaching it’s probably the most important one in the set. The IDE’s new plotting function makes it easy to do live demonstrations in front of a classroom simply by adding one print statement to the code:  nothing I’ve tried before even comes close to this level of simplicity.

I use my  ‘UNO-scope’ all the time now because I can do a quick screen capture  or I can send a series of runs to a serial text monitor. In both cases, calculations can be done on the Arduino before the output is sent, converting the raw ADC readings into something more meaningful like current or power.  Then I can go hunting for unexpected events by graphing it up in Excel.  This approach has let me track down things like SDcard weirdness that are very hard to capture during normal logger operation because they only happen when you pass some threshold inside the flash controller:

Cave Pearl data loggers

I’ve overlaid the bold numbers, but that’s a screen capture of the IDE output. Modern scopes can do this kind of thing too, but you’ll be hard pressed to find one for the price of an Arduino clone on eBay. With a low value shunt resistor, you can push the ADC clock pre-scalars into the 20-40kHz range, which is more than adequate for the kind of diagnostic work I’m doing, and most stuff you’d be trying to demonstrate in a high school physics class. 

DIY Cave Pearl data loggers based on Arduino Microcontrollers

My PVC housings sport smaller separate wells for each sensor as I’m trying to reduce the surface area exposed to pressure & salt water. So far we have only deployed them to ~30m, but deeper sites are on the calendar in 2017.

3D printers are finally reaching a price point where people can afford one to help them pursue other more interesting hobbies, and as key patents continue to expire,  new/old high-end tools are entering the consumer market: Forget 3D-Printed Knick-Knacks.  But it’s worth noting I haven’t used any of these tools on the project because I can still build more durable housings out of plumbing fittings from the local hardware store. Total part cost on those is ~$15, and I probably spend less time making them than I would repairing holes or fixing mesh errors on a constantly evolving 3D printed version. But I’m sure the strength and print quality tipping point will occur eventually… probably when DLP’s with stronger resins reach the kind of price point we are currently seeing for extrusion printers.

I will add more tutorials to that set over time, and hopefully we’ll manage to publish a paper or two this year.  The plan is to put them in open source journals so everyone in the world has access, and if we spin up all the collaborative projects we’ve been planning with other researchers, 2017 promises to be a very busy year.

Addendum 2017-01-10

2016 was also a banner year for the Maker movement in terms of media coverage. So I thought I would add selection of those articles here:

Why the Maker Movement Matters: Part 1, the Tools Revolution

Why the Maker Movement Matters: Part 2, Agility

 

Sometimes those geeky editorials make me laugh, but even then they are still thought provoking.  It’s also good to see more thoughtful criticism and self reflection going on as the movement matures it’s way through the Hype Cycle (beautifully illustrated by this 100+ year old debate about the Wright brother’s)

Making It

Makerspace: Towards a New Civic Infrastructure

Why I Am Not a Maker

Unfortunately that also means the market has grown to the point that the big boys want a piece of the action. While this probably won’t be another  “Embrace, Extend & Extinguish” situation, commercial players inevitably increase the pressure to commoditize the product into easier to use (& thus more sellable) packages. I can see good arguments to support this but I have some concern about developments (like the ESLOV) which eliminate the user’s exposure to actual code: turning great problem-based learning exercises into plug & play activities.  Unless you let students see ‘under the hood’, they’ll walk away thinking technology is about connecting little black boxes for participation marks.  By now it’s clear that we are heading toward a IoT powered world of self driving carsBaxter bots, and staffless stores.  So I can’t help thinking that unless our young people can handle Arduino level programming tasks, they won’t qualify for a job making toast.

Addendum 2017-04-30

On the topic of media: looks like the one-word journals are finally starting to notice the open source hardware movement. Better late than never, though I suppose with all the mergers going on, the whole makers movement isn’t much more than a rounding error on corporate scale balance sheets. 

<— Click here to continue reading the story—>

Field Report 2015-08-10: Diagnosing a TMP102 Sensor Failure

Over the next few days we did cave dives to exchange units at sites along the coast. Some of these systems were too deep for our camera, and in the shallower systems the visibility could get a bit grim because if you stay in one place too long, air bubbles rise up and cause “percolation” of little pieces of rock from the ceiling.  So to dismiss any illusions people might have that science diving looks like what they show you on TV, here’s a clip from one of those claustrophobic installations:

Even so, things went well.  Each night I had a new crop of loggers to download and check for my “standard set” of parameters (battery, RTC coincell, sleep current, clock drift, etc.) which I compared to the readings taken before the deployment. The recent retrievals were from the last generation of 3-inch pvc builds, and while they all looked great on the outside, some of them stopped running long before we pulled them out. It can be tricky to figure out the problem when a logger has several I2C sensors on it because the very things that make it so easy to use those breakout boards, also hide problems from you: essentially they don’t just run digitally, they also tend to die that way.  Issues tend to show up much earlier in the analog data, and one useful diagnostic is the voltage log from the main battery pack (read with a simple resistor divider).  Alkaline AA’s usually provide a nice smooth decline over time, so when I see something like this:

011_PS&Temp

Unit w BMA180, TMP102, HMCL5883l. Deployed in fresh water at 5m.

…it’s an immediate red flag.

But unit #011 had a complete set of data logs, with no hint in the sensor records that something strange occurred.  That peak in temperature was pretty large, but it also corresponded to the June 13/14 event that showed up my Rio Secreto & Akumal Bay records, so I accepted it as valid data.  My next thought was that the unit had suffered a μSD failure, but this logger was still successfully going to sleep at a reasonable 0.26 mA. Rapidly entering a low current sleep state is the best indicator I’ve found so far that a card’s control circuitry is still working as it should. Hmmm….

Then next unit with problems was also from a relatively shallow fresh water deployment, and it ran for only two weeks before shutting down:

BMA180, TMP102, HMCL5883l, Deployed in fresh water at 14.5m Depth

Unit w BMA180, TMP102, HMCL5883l.  Deployed in fresh water at 14.5m.

Leak corosion after sensor failure drained the batteries

The power module from #015:  alkalines almost always leak if something pulls them below 0.5v.

This logger was positively pressurized when I broke the seal, with only 0.22 volts at the battery module. When I connected a new battery the unit would did restart properly, producing a strange pattern on the indicator LEDs. Fortunately even with only two weeks of data, what we did have in the log make it pretty clear who the culprit was. It was extremely unlikely the temperature would have spiked like that on the same day that the battery curve took a nose dive.

The next unit presented another mystery, because it had a nice smooth power curve though it had reached it’s low voltage shut down point too early. The temperature data looked reasonable:

Unit w BMA180, TMP102, HMCL5883l. Deployed in fresh water at 22.5m Depth

Unit w BMA180, TMP102, HMCL5883l. Deployed in salt water at 22m.

These loggers have dual 3xAA power packs, and they normally deliver at least eight months of operation. While #016 seemed to restart OK, with a normal (~o.24mA) sleep current, it then crept up to 1.65 mA.  I left the unit running for about 5 minutes and by that time the sleep current was jumping all over the place, varying from 0.5mA to 4 mA at random. This behavior continued after I swapped in a known good SD card, so I was stumped. Once again the data in the sensor logs looked fine.

I had one more unit from this build generation to open up, and it too had developed a problem over the deployment:

017_PS&TEMP

Unit w BMA180, TMP102, HMCL5883l. Deployed in salt water at 24m.

No rust at all...?

I have to trim the boards a bit to fit them in the sensor wells but I am careful not to cut any traces or vias. There was no visible rust on those breakouts at all.

That hockey stick bend on the power curve was another smoking gun, especially in the context of that temperature data. The deep saline water in these caves is remarkably stable, so when I saw a temperature rise alignment like we had with #015, I started pulling wires. Sure enough, the sleep current problem on #016 disappeared as soon as the TMP102 was disconnected and #015 fired up properly as well.

Since epoxy failures took out a fair number of units in 2014, I did a detailed inspection of the potting, but found nothing to indicate that water had reached the sensors. Since the deeper units were most affected, my current working hypothesis is that these sensors were actually suffering from the effects of pressure, rather than moisture. (…but good luck finding that in your ‘Recommended Operating Conditions’)

Anyway, I disconnected the TMP102’s from all the loggers of that generation, (including the ones that were still working…) and set the code to read the internal RTC temperature register instead. Test runs with that were good, and these units went back out for deployment a few days later even though all that housing mass will put a huge lag into the temperature record.  I don’t think I will be using TMP102 sensors again on my projects, but of course there is nothing to say the MCP9808’s from Adafruit won’t develop a similar problem over time.  So I might end up back at the humble DS18Bs which I used on my earliest builds:  they have proved to be remarkably resilient over time.

Addendum 2016-03-17

The MCP9808’s did indeed suffer a similar fate.

 

Testing Power Use with Complex Datalogger Duty Cycles

There is an old saying that goes: “Yesterdays solutions are today’s problems” and I think that now describes this project. You see the first year of development was focused on pretty straightforward hardware issues, and solving each one produced significant gains in performance. Now that I am consistently seeing sleep currents in the 0.1-0.2 mA range (with an SD card (~80uA) & live Adxl345 (~50uA) along for the ride), I am hunting for more elegant ways to extend the operating time while maintaining the simple three component core of the original design. With a 3xAA power pack now providing almost a year of operation for some sensor configurations, I also have the task of developing a method for testing the loggers that can discriminate between subtle code changes with relatively short runs. But artificially reducing the sleep interval between samples distorts the result enough that it’s hard to make good projections. I am slowly coming to realize that testing & calibration are the real heavy lifting when you build any new device.

The new A544 cells arrived at > 7 volts which was too high for the regulator on the Ultras. So I took them down to 5.6 volts with a Zenner. The rare earth magnet soldered to the diode wire gets zapped by the heat from the iron, so you need a second little magnet to hold each battery connection securely. You can also stack a "set" of button cells with these magnets, giving you more options for low power tests.

The new A544 cells arrived at > 7 volts which was too high for the regulator on the Ultra. So I took them down to 5.6 volts with a Zenner that stops the discharge before it goes too far. The rare earth magnet soldered to the leads gets zapped by the heat from the iron, so you need a second little magnet to hold each battery connection securely.

Each new trick I try, like finding another place where I can put the cpu to sleep, adds complexity to code that has “once per day events” and “once per week” events, and soon there will be “only if the delta between two readings is greater than x” events. Most of these are so short that a multimeter can’t catch them but even when a friend donated an old Tektronics to help me try to get a handle on the duty cycle, I faced the challenge of displaying currents ranging from less than 0.1mA to 80mA SD writes with variable duration. To make things more interesting, some of the cheap sensor boards I have been noodling around with have components of unknown origin & dubious quality, which introduce yet another set of variables.

Even with my mediocre scope-skills the forums had convinced me that the SD card was the elephant in the room. So I tried to avoid SD use by adding an external 32k eeprom which let me buffer five or more days worth of data before having to fire up the external storage. Problem solved…or so I thought. I was quite surprised by data from the last deployment that showed using this approach to reduce SD writes by a factor of five only delivered a 5-10% improvement overall.  I had overlooked the fact that the AT24C256 eeprom pulls 3mA for five milliseconds per pagewrite. This was nearly as much current as the Rocket Ultra I was using, not to mention a significant extension of the cpu uptime for multi sensor units that were buffering up to four eeprom pages per record. All of that activity adds up.

So I took another look at buffering data in SRAM, which I flirted with at the beginning of the project. But my script was now much larger than those early versions, leaving barely 500 bytes free.  I know the real coders out there probably laugh at my use of Pstring & Ascii but that lets me add a new sensor by changing a couple of print statements, and adaptability has always been of my primary design goals. To maintain that simplicity I went searching for an Arduino with more headroom and the Moteino Mega over at LowPower Labs seemed to fit the bill with it’s 1284P offering an extravagant 16K of sram (compared to just 2K on the 328p). It also used a low drop out MCP1700 series regulator like the Ultras, and there was support for RFM transceivers.  With the Mega’s larger footprint, I decided to try them first on the larger dry cave platforms:

Rocket Ultra (left) VS Moteino Mega (right) loggers with pin powered RTCs. I break out LED, I2C and one-wire with Deans micro connectors, and you can see the 4.7K one-wire pullup above the main power supply divider on the Mega. The 32K eeprom is tucked under the RTC, which is inverted on the Moteino build to make changing the coin cell easier.

Rocket Ultra (left) VS Moteino Mega (right) based data loggers with pin powered RTCs and 2×4.7MΩ voltage dividers monitoring both the power supply voltage and the rtc backup battery.  I break out LED, I2C and one-wire with Deans micro connectors, and you can see a 4.7K one-wire pull-up above the main divider on the Mega. A 32K eeprom is tucked away under the rtc breakout, which I flipped over on the Moteino build to make it easier to change the CR2032.

For a standardized test, I set both loggers buffering 96 records (= one day @ 15min intervals) in drip sensor configuration. I added the I2C eeprom to the Moteino logger to make the builds as similar as possible, but it does not get used. Instead I store the raw sensor data in integer arrays. So there is no Pstring/ascii use on the Mega logger until I write the data to the SD cards.  With matched acclerometers & cards, both loggers sleep at 0.18 mA so the the only difference between them should be the data handling.  One thing I did not catch from the LowPowerLab specifications was that the 16mhz Mega draws ~12 mA (while awake) in this configuration as compared to the Ultra builds which perk along at just over 4mA. I figured that with SRAM storage the mcu up time would be so much shorter that it would not matter.

With super caps to buffer SD write events, you can drive the loggers with a very small battery. Rare earth magnets let you connect to the ends without a holder and you can make a multi-layer magnet/button cell sandwich to build low power options at just about any voltage. Those are 5v 1farad supercaps in series, so I don't bother to balance them as they should be able to handle leakage asymmetry when the battery input is only 5.6 volts

With a parallel bank of super caps to buffer SD events, you can drive the loggers with small batteries that have high series resistance. Rare earth magnets let you connect without a holder and you can build multi-layer magnet/button cell stacks to create low power options at different voltages. Those are 5v 1farad supercaps so I don’t bother to balance them as they should be able to handle any leakage asymmetry when the battery input is only 5.6 volts. The graphs below had no low volt blips at all, so this series/parallel arrangement of 4 of them was probably more capacity than I needed.

I still had not sorted out the oscilloscope issues but I realized that I could flip the problem around: instead of struggling to display the effect of every little tweak to the duty cycle why not provide a fixed amount of power and just see how long the unit runs. It’s a data logger, so I already have a time stamp and a battery voltage reading with every record.  A couple of people suggested capacitors, but even a 1F supercap only gives you about 0.27 mAh per volt, translating into a few hours of operation for my loggers. I needed longer runs than that because the Moteino was going to loose the data in it’s sram buffer when the unit browned out (I can always dig into eeproms later for the last few records on the Ultra).  A bank big enough for multi day runs was going to be expensive, and is probably a hazard for my little bots.

Fortunately there are a host of small form factor batteries out there for things like fire alarms, medical devices, etc. Energiser’s A544 seemed to fit the bill at 6 volts & 150 mAh: promising to power the Pearls in “sleep current” mode for about 40 days. Even better, they were alkaline cells, so their discharge curve would be more like the AA’s used in real world deployments. There was some risk that these little cells would drop to the low voltage cutoff when the SD write current spikes occurred, so I added a few super caps to buffer those loads. I then set the units up on a book shelf where they would not be triggered and waited for my “baseline” load result.

This is the voltage record from the two different logger platforms, when they were powered by a single 150mAh A544:

A544_FirstPowerTest
(I stopped the test after a month, because I couldn’t take these suspense any longer. There were few sensor interrupts during the test, so this was a baseline power use comparison)

I was sure the SRAM buffering Moteino logger would come out far ahead of the Rocket build that was sandbagged by all that I2C eeprom traffic. But if you correct for the slightly higher starting voltage those two curves are so close to each other they might well have come from the same machine. So there is no longevity boost from SRAM buffering if I use an mcu that draws 3x as much current, but at least I now have a good way to test the loggers without waiting too long for results. This result also agrees with some of my earliest drip sensor results which hinted that the sampling/buffering events were consuming 2/3 of the power budget.

For the next round of tests I will put them on the calibration rigs to see how the A544’s handle the interrupts being triggered all the time. Presumably the Moteinos will draw more power there so I will need to normalize the results to match drip counts. To go beyond the conservative one day buffering I will need some way to capture data from the SRAM buffer before the units power down, so perhaps I will end up using the eeprom on those Moteino Mega builds after all. We will use a few Mega based drip sensors set for very long buffering (8-10 days?) on the next real world deployment. I also have a feeling that the DS18B20 temperature strings would benefit more from SRAM buffering than these simple drip sensors, as they poll up to 40 sensors per record. That’s a lot more data to shuffle around.

Addendum 2015-07-05

Hackaday just posted about [Majek] putting “live” data into Arduino’s flash ram (which is normally not accessible after startup) via a Optiboot hack.  This opens up another possible data buffering strategy, though I am not sure if it could handle the duty cycle of a long deployment. Or it might let you do calculations with the 328p that would otherwise run out of space.  So this is an interesting development that involves no extra hardware, which is usually good news for the power budget. I had already been wondering if calibration data could be stored in flash with Progmem, but that solution only works for data that is not changing all the time.

Addendum 2016-01-06

We finally have some data from the first field deployment of Moteino based loggers which store sensor readings in ram (array variables), rather than buffering all the data as ascii characters in an external eeprom like my 328p based loggers do.

Here is the power curve from a Moteino:

057-M_Drip_BatteryCurve

Battery (mV): 3xAA supply, 0.18 mA sleep current, 5 days of data in ram

And here is a directly comparable build using a rocket scream ultra with a slightly higher drip count (ie: number of processor waking events) over the duration of the deployment.

Cave Pearl data loggers

Battery (mV): 3xAA supply, 0.18 mA sleep, 5 days of data buffered to AT24C256 eeprom

So once again, these performance curves are so close that it makes no odds. But on the bright side, this confirms that accelerated testing with 150mAh A544 batteries does give me results that translate into real world. So this is still pretty good news even if the 1284’s did not deliver the magic performance bullet I was hoping for.

Addendum 2016-02-15

If I wanted something a bit beefier than the 150 mAh in the A544’s, I could hack my way into a 9v battery, and use half of the set of 500 mAh AAAA batteries you find inside. That would give me about 1/4 the power of the AA batteries I typically use on deployment.

Addendum 2016-08-15

I finally figured out how to view  individual logger events using an Arduino UNO as a DAQ with the serial plotter tool built into the IDE:

Cave Pearl data loggers  I’m quite tickled about being able to replicate a task that you normally would need an oscilloscope to see.  Of course my chances of actually catching one of those big unpredictable SD card latencies (from something like  age related wear-leveling) is still pretty low, so I will continue to use this A544 method for solid longevity predictions.

 

The 2014 Cave Pearl Project ‘Year in Review’

With the last field reports of 2014 out the door, I feel I should post something about the project having completed it’s first year as a “serious endeavor”.  At the risk of sounding like A.J.Jacobs,  it has been “A Year of Living Inventorly”, and the fact that I had never done anything like this before did not stop me from trying to build one of the simplest underwater dataloggers ever made. Looking back, I am very pleased by how much the Cave Pearls developed, both in terms of the electronics:

Platform_tryptic_Jan2014

and the physical build:

Cave Pearl Physical Build Tryptic

It’s been one long string of learning by doing, and with no “formal” training in electronics, I have been using a basic process of elimination approach to tackle each problem as I went along.  Each time I built a one, it became a little easier to see where the next improvements could be made.  I became fairly obsessed about the project and my wife, with her usual good humor, coined the phrase “nerdling” to describe the days spent cutting & soldering in the basement, instead of pursuing my other passions.  But left brain/right brain tropes have always sounded odd to me because inside my head both tasks feel pretty much the same.

The highlights of the year were the field deployment trips:

Date #deployed Logger / Sensor combination
Dec. 2013 2 Alpha Flow Sensors go under water for the first time.
Mar. 2014 3 Beta Flow Sensors   Tinyduino based loggers with Bma250 accelerometers, housing have no latch clamps
1 1st gen. Underwater Pressure Sensor
Aug. 2014 3 3rd gen. Flow sensors Rocket based logger platforms, Rosetta Stone builds to compare sensors, BMA180 accelerometers adopted, epoxy failure in this generation
3 Beta Flow sensors re-deployed  in the open ocean
2 Pressure Sensors    MS5803 based,   2bar & 5bar
6 1st gen. Drip Sensors    deployed at Rio Secreto
Dec. 2014 10 Flow sensors a mix of 3rd & 4th generation units, most go into deeper low flow saline zones, 1 commercial co-deployment and a fresh/saline co-location at one site
3 Beta Flow units 2 rebuilt for open ocean deployment, one unit has been underwater continuously ~ 10 months
11 Drip Sensors 1st & 2nd generation, @ Rio Secreto
2 Pressure sensors 2bar in water & 5bar @ surface
2 1st gen. R.Humidity, Pressure & Temp. loggers

The first time I make a machine, it usually takes a week or two of experimentation, and plenty of epoxy, wire & PVC goes into the bin. But now that the logging platform is more developed,  it is getting easier for me to come up with new sensor/logger combinations. While the underwater housing remains the most time consuming part of each build, the overall improvements in design, and in my own skill level, show clearly in the number of units we have been able to deploy over time:

Cave Pearl DIY housings made from PVC

So I guess that’s a report card of sorts, and I owe thanks to all the Arduino makers out there who shared their code & experience, because they really made it possible for me to pick it up as I went along.  But even with that help, there were still a few bits of information that would have been really nice to know a year ago, that I that just did not find anywhere on the web.  So I thought I would address some of those questions.

How much does it cost to build this stuff?

As a long time freelancer, I keep pretty detailed records. By the end of 2014, the cumulative hardware cost came to about $10,000  including all electronics, tools, and pvc parts. Yep. Really. The waste generated in the process, as you hunt around for the “right parts” is significant, and in the early stages I build at least two units that never leave the bench for every one that makes it into the field.  Now I try to buy “just enough” material to build the next 3 or 4 units, no matter how ridiculous the shipping charges are. (often more than the parts themselves)  If you try to order the cheaper parts from eBay you end up with 2-3 week shipping delays, which is a real impediment to your prototyping workflow.  The problem is that every design change leaves you with a handful of leftover bits from the “old” version, and I now have a substantial collection these orphans in boxes all over my workshop.  Even with diligent effort about 1/2 of the electronic parts I purchased did not get used. (at least not yet…the prototyping is going faster now that I have a critical mass of components on hand) Also, I would advise anyone to buy better tools right at the start, rather than wasting your money working your way up through crummy soldering irons, and low budget saws. A good table top band-saw & drill press are essential.

Should I make a blog for my project?

At the beginning, I really had no idea if it was worth the time and effort to maintain a blog like this. But I know several people who are smarter than I am, working on equally cool projects that no one will ever hear about because they have not put the word out.  So for them I thought I would post a snapshot of the stats from this little opus, which has not been subjected to any “How to promote your blog” efforts because I don’t have any spare time left.  I post links in forums when I have something useful to contribute, and a handful of people have given a nod to my project, but that’s about it:

CavePearlBlogTraffic_2014

In total there were 6861 visitors in 2014, generating 19,456 page views.
At the start of the new year, the stats look like this:

CavePearlBlog_TypicalWeeklyTraffic

As you might expect, at least 1/2 of all traffic comes from the U.S. The UK and Germany lead the western European contingent, which adds another 1/3 of the total, while the remainder is sparsely distributed around the globe.  I used to get really excited when some new country generated a big spike in traffic to the blog. Then my brother explained how botnet attacks work. This went a long way to curing my daily compulsion to check the wordpress stats because the truth is they probably don’t care that much about underwater data loggers in the Republic of Elbonia.  Still, some stats are useful, like the fact that 99.9% of the traffic comes in from Google searches.  Ignoring the home pages, posts with content on “How to do X with an Arduino” are by far the most popular:

CavePearlBlog_TopPages

I was surprised to see the RTC page rise so high in the ranking and I was probably just the first person to gather that information together in one place.  When a page does rise above the one-per-day hit rate that just about everything seems to get (from search engine spiders?),  I maintain that page with regular updates, and that seems to be paying off. I also take the time to cross-link extensively between old and new posts, because several people have complained that they could not navigate around “my website”. There are still many people out there who do not know how blogs work, and almost no one uses the built in search feature.

Where does the project go from here?

On the physical side, I need to make the process of assembling the underwater housings less labor intensive, and I need to improve the overall response to flow rates less that 1 cm/second because the deep saline circulation in those caves is very slow. Wherever possible the design also needs to make in-water procedures for things like buoyancy adjustment faster and easier.

On the electronics side, I will hunt for ways to reduce sleep power consumption, while continuing to use cheap components in my basic “off the shelf” design criterion. I have to find or develop better sensor calibration methods, and there are still many code issues to sort out. With some basic improvements in buffering and data handling during processor up-time, I think I could still see significant improvements in the power budget. Using other data formats is one obvious approach, because it’s all basically integer data. But I will probably stick with ASCI for a while yet, as PString makes the code very easy to modify when I change sensor configurations. This is important because the “sensor caps” are interchangeable modular units, independent of the logging platform and the rest of the housing.

I am finally getting comfortable tackling new sensors at the datasheet level and I now have a general sense of how to get communication working between breakout boards and the Arduino.  I accept the criticism that so far I have been making DIY versions of sensor & logger combinations that are already available on the market; in essence trading build  time for money.  With a year of work under my belt, I would say that this trade is not worth it for equipment that you only need one of,  because getting that first prototype operational usually costs you more than you would have spent on the commercial sensor. It starts to be worth it when you reach about ten units, because by then you have all your parts sourced and you can build them pretty quickly.  From that perspective, this project is just starting to reach a break-even point, with the added benefit that we can make custom builds for a particular research question.  Of course, sorting out a new instrument is so immensely satisfying that I would be doing it no matter how many we actually needed.   My hope is that by the end of 2015 I have the chops to develop genuinely new devices that no one has ever thought of.

But looking at the page stats again reminds me that I probably shouldn’t spend too much time writing posts like this one.  I need to get back to the bench and figure out how to do more stuff with an Arduino…and then write about that.

<— Click here to continue reading the story—>

What to expect buying cheap electronic components from eBay

The first time I deliberately purchased something with DRM.

The first time I ever intentionally purchased something with DRM on it. But for me, 1 gig is plenty of space! With Sandisks wear leveling, and a 1 million hour MTBF, (yeah…right…)  I am hoping that a few years in someones cell phone before I get them won’t matter. About 10% of these used cards fail to pass my low current sleep tests, and since they only cost $2, I just throw those away.

After playing the SD card shell game for the last week, I think I have finally found a way to make sure that I have authentic Sandisk cards for my data loggers. Muve music microSD cards were custom built by Sandisk for Crickets music service, and they put some kind of blocking routine into them that prevents you from seeing 3/4 of the space on the card. As you might imagine, extra custom circuitry which makes a product less functional affects its market value, so they are unlikely to be seen as a profitable target by counterfeiters.  Of course I put these used cards through a full test routine, formatting them with SDformater, and then making them sweat through a good bout of H2testw’s verify. And even though what I am after is stable low sleep current, I keep an eye on the meter during the testing. Some of the fake cards I ran into earlier draw unusually large amounts of power during write operations, even if they sleep OK.

More adventures in eBay land:

Going through all that motivated me to make a post on the approximate nature of buying cheap electronic components on the internet. Of course there are warnings about this all over the place. But when I was just starting out, very few people took the time to provide details so I had to learn this from scratch.  The easiest way to convey this is with a table showing the photo from the listings  vs  what actually shows up in the mail somewhere between 2 & 8 weeks later. I would say the current hit rate for actually getting what I thought I was ordering from vendors outside the U.S. is about 60%. The rest of the time you get something that resembles the item in the listing…more or less:

Photo Shown Typical substitutions Comment
Square boards have 4 mounting holes, important in tight spaces. roundBoardRTC Don’t count on fine details like mounting holes or board dimensions matching the listing. Batteries can be missing -or- Cr2032 -or LIR2032,  no matter what is shown in the photos.
JY-MCULow profile 3 color & quite visible with 10k limiters. Brightness is highly variable in eBay 5050s. 2 Other leds …or one of 5 other types randomly intermixed on same order. Sometimes they have limiters, sometimes they don’t. Silk screen labels almost never match the actual pin colors indicating defective runs.
MolexBoard flipup If you get the Molex connectors they both work well enough, but if you don’t have the overhead in your build for the flip up style, it’s kind of annoying to get them.

I could go on, but hopefully that gives you a general idea. And this is in no way limited to eBay. But these kind of replacements are usually not hard to work around, and since the parts only cost a buck or two, I’ll make parallel orders from other vendors, just to be sure I get something I can work with. Order different quantities from each vendor when you do simultaneous orders, because the parts could arrive weeks later, and there will be nothing written on the package in English to help you figure out which vendor sent which shipment.  Always do a “test” order of a small quantity, before ordering multiple units from an eBay vendor, and don’t be surprised if the product changes like this from one batch to the next anyway.

More concerning are part substitutions directly on the boards.  The more components on a breakout, the higher the chance that at least one of them will not be what they claim it is. Complex combinations like you see on IMU’s are a red flag:

Photo Shown Comment
IMU GY-81-3205 10DOF IMU modules are usually described as having the a 1g BMA180 accelerometer (which I use with the HMC5883L in the flow meters) but from places like Miniinthebox, Gearbest, Amazon, etc. they actually ship with crappy 2g BMA020 on board. Of 5 vendors I tried, only NYplatform actually sent this board with a BMA180.

If I’m rooting around on eBay for a sensor board to try out, I get one with as few components as possible. For example, I use the ADXL345 in my drip sensors. These things are common as dirt, and almost as cheap, but the boards are often laden with dubious quality voltage regulators, power LEDs, etc.  In general, the less cruft there is, the more likely it is that it will work, and I always look for boards that have a 3.3v power input.

Avoid: Ahh, that’s better…
adxl345-1 adxl345-2 adxl345-3

Of course this is still no guarantee, especially when a conditioning procedure is needed for the sensor to meet spec.  But when the parts are only a couple of bucks, I don’t have worry that much about frying them when I am just noodling around.  (I killed several of these things before I realized it that JB 5-minute plastic epoxy dissolves IC housings before it cures…) And it’s just the magic of eBay economics that you usually pay more, sometimes much more, for the breakouts with fewer parts on them.

Inspect all solder joints. The lower image shows a pretty typical alignment skew on the cheap SD adapters.

Inspect all solder joints. The lower image shows a pretty typical alignment skew on the cheap SD card adapters. Of course I did not discover it until I had already soldered it into place inside one of my loggers . . .

If you paying peanuts for a sensor that normally sells for $20-$30, you should expect a 20-25% failure rate, with about half of those failures being subtle, like for example, one axis on an accelerometer not reading at the same bit depth as the other two, or only reading in the positive, but not negative direction, etc. Always test your cheap sensors against a  a “known good” board and one thing I have found to be particularly diagnostic is how much current an IC draws.  If the milliamps match the data sheet, there is a good chance that the sensor is Ok.  This is especially true of low current modes, as I have run into several cheap sensors that give decent readings, but never go into low current sleep modes. This makes them useless for data logging applications.

It’s usually a much safer bet to get simple passive components on the web, and the difference between low end parts, and brand name stuff is often an order of magnitude or more.  When you are just starting out, finding cheap component kits can get you rolling for without breaking the bank (although the thin wires & non standard labeling will eventually get on your nerves) Sadly, I did not find out about those until I had already paid extortionate amounts to radio$hack, etc.

I was most of the way through the bundle by this point, but by then I was angry enough to take a photo of it.

I was most of the way through the bundle by this point, but by then I was angry enough to take a photo of it.

And speaking of places where staff react like I a speaking another language if I use the words like “capacitor” in conversation,  I have to add that it really does make a difference if  you order stuff from a vendor who actually uses the products they are selling.  For example, I recently ordered a large amount of wire from HobbyKing. Not only did they take quite a while to ship it to me from their US warehouse (for items listed as “in stock” when ordered) but they looped the many different types of wires into a single large intertwined bundle. It took ages to untangle that mess, leaving me with the impression that they are just a warehouse operation, charging a markup simply to repackage stuff that they too are ordering from China.

Everything gets a bath in 90% isopropl alcohol to remove flux residue.

Everything gets a bath in 90% isopropl alcohol to remove flux residue.

I guess that if I was to sum it all up, I’d say I still get lots of stuff from dodgy online vendors if I am risking $5 or less. The more expensive an item is, the more likely I am to go upmarket to a vendor like Sparkfun or Pololu.  For example, when I need a tool like a soldering iron, or a multimeter, I check what they reccomended at EEVblog, then I see what they think about it on Adafruit.  Those guys really know their stuff, and I never have to worry about dropping a couple of hundred bucks on their recommendation. (something I would never do with an eBay vendor from Hong Kong)

Anyway, I hope that helps someone out there who is just getting started (like I was not so long ago). Even with the best research beforehand, I still end up wasting about 30% of the money I spend on components, because there are alot of elements to any item that one just has see & use to understand. And there is always seems to be some “physical” factor that the experts thought was too obvious to mention, that dramatically affects your particular project… like… for instance…never putting your coffee down on the work bench beside your newly built data logger 😉

Drip Sensor Update: The Gamma Build

I suggested in the last post that a new build usually comes together on the third iteration, so I though I would post a few photos of the current drip sensor, to show how much they changed in that short series:

Alpha, Beta & Gamma builds

Alpha, Beta & Gamma builds of the Cave Pearl Drip Sensor

That Alpha would only detect drops in the very center of the housing, and often registered double hits because of the splash back from the complex surface topography, which also caused a buildup of water on the surface, further interfering with the signal. So I removed the pressure and temperature sensor wells from the Beta, and used a heat gun to bow the surface and shed water.  I hand sanded the pvc to make the strike surface thinner and more responsive, which increased the “reliable” sweet spot to the diameter of the circle you see on drawn there. This worked well, but as you might imagine removing that hour long fabrication step rose to the top of the priority list (if necessity is the mother of invention, laziness is surely the father…) I also did not want to penetrate the housing for those standoffs if I could avoid it because this device has to maintain integrity with constant water impact at exactly that spot.

JB Plasicweld bonds the accelerometer to the cap

Plastic-weld bonds the sensor, preserving the integrity of the housing

My solution to both problems was to solvent weld a four inch knock out cap to the top of the hard pvc shell.  The green color you see there is ABS to PVC transition cement, that I learned about back at the beginning of the project when I was mangling Leggo bricks for internal scaffolding. The ABS is translucent, so the LED no longer needs a portal of clear epoxy because the light passes right through.  The knockout is thin and stiff, which eliminates all that sanding and improves the response of the accelerometers so much that now the device even responds to loud noises, and almost the entire surface sensitive to the smallest drip.  As a result I now have to tweak the settings to reduce the sensitivity so we don’t get too many false positives.  But on the third build of a new sensor, that’s the kind of problem you want to have.  I though the flat surface might resurrect the water pooling problem we had on the Alpha, but in the end it just ended up being a lesson in how I still miss things that are really darned obvious, because simply setting the drip sensor up with a slight tilt will shed the water without affecting the operation at all.  (I am just glad I realized this before some three year old came along and pointed it out to me…)

These guys are only $1.5 each, and are much easier to work with than the plastic micro SD adapters

Much easier to solder those jumpers now.

Wirefold

The trickiest part of the build is routing the wires to reduce strain. Thin silicone 26awg wire helps quite a bit there.

There have been a few improvements to the logging platform as well.  I found these really inexpensive  raspberry pi SD card adapters that are mounted on their own pcb, which neatly solves the melting problem that the cheap plastic  adapters give you if you linger too long with your soldering iron.

Drip Sensor Gamma BuildSo this is the new baby, and I am now running burn tests on loggers using a few different clone boards, including a Rocket Scream Mini Ultra, as they have some very interesting power saving features that might significantly increase the operating time, while keeping the overall simplicity of the three component design.  I expect a few more issues will arise in testing, so I will hold off posting the code to Github, till I am sure that they are behaving properly.  I am still a bit stunned that these drip sensors came together so quickly, but perhaps the last six months of work on the flow meters had something to do with that. 🙂

<— Click here to continue reading the story—>  .

Addendum

Adafruit just posted a video from the National Science Foundation showing how water droplets move on various hydrophobic surfaces.  Way cool…

Addendum 2014-12-11:

We deployed our first batch of these drip sensors in August, and when we went back to get them in December, we were delighted to find that the first real world run was a resounding success.