How to Monitor Leaky Radioactive Water Tanks
Written by Akiba   
Saturday, 24 August 2013

I just got back from my vacation in Shenzhen yesterday and was going to relax a bit until Monday when Iíd start working on some new designs. When I was reading the New York Times this morning, a particular article caught my attention. Hiroko Tabuchi wrote an interesting article about the worsening crisis going on at the Fukushima Dai-Ichi nuclear power plant.

The radioactive waste water is leaking into the ocean and the leaks seem to be getting larger. There are nearly 1000 tanks at the plant to store the radiated water which are prone to leaks. Also, there are no water level gauges in the tanks and only two men patrol the tanks every day to check for leaks. What caught my attention in the article was when it mentioned TEPCO had no reliable way to check the storage tanks for leaks. That was actually one of those ďhmmmÖ.reallyĒ moments for me.

The problem of detecting water leakage in the tanks can actually be restated as a need for continuous monitoring of water levels, a common problem. In fact, I ran a workshop in Dharamsala, India in the Himalayas earlier this year to teach locals to implement automated water level monitoring in storage tanks that provided drinking water to Tibetan Buddhist monks and students.

2013-04-12 DSC03305

There are a lot of different solutions for water level monitoring, but in this situation, there are a couple of constraints. The structures need to be retrofitted without disturbing the water, doing anything to come in contact with the radiated water is probably undesirable, and any access to the storage tanks are probably at the top of the tanks which are likely a few stories tall. This means that access is extremely inconvenient.

This is actually an ideal case study for a wireless sensor network. A remote device consisting of a battery powered, water level sensor and a wireless transmitter would be placed on top of each storage tank. The remote devices would communicate with an aggregator device connected to the internet and uploading data to a server periodically. Finally, the problem is then reduced to parsing web data to visualize the tank levels and possibly color code or send alarms when the water levels in any particular tank start to drop.

Sure itís all fine and dandy to talk about this theoretically, but letís go one step further and implement a mock system to continuously monitor water levels in a tank, transmit them wirelessly, and upload the data to the internet.

First, itís important to take care of the water level sensing part. Based on the limitations I mentioned previously, one of the best ways to handle non-contact water level sensing is to use a sonar based sensor. This type of sensor emits an ultrasonic audio chirp and then waits for the echo. The distance can be calculated based on the time it takes for the echo to arrive. In this case, Iíll be using an industrial grade, waterproof sonar sensor from MaxSonar: the MB7389 . Itís waterproof, pre-calibrated, and has a range of 5m. There are also 10m versions available but I just happen to have the MB7389 on-hand. It outputs analog, serial, and pulse width data, but in this case, Iíll be using the analog output. Itís easier to deal with and at this point, we mostly care about prototyping quickly to check feasibility.

20130324-DSC01465-2

The sensorís analog output will be fed into the analog input of a 900 MHz freakduino which is an Arduino-compatible board that I make with an integrated 900 MHz transmitter. Iím choosing to use 900 MHz in this case because it has longer range and lower attenuation through free space. By controlling the transmitter power and antenna type, itís possible to get ranges up to a few kilometers using a stock board.

  2013-08-24 DSC07320

The software side is hopefully pretty straightforward. In the setup routine, weíll be initializing the analog input pin, initializing the chibi wireless stack, and then setting the channel and address of the device. Since this would be used in Japan, Iím setting the radio to channel 8, the equivalent of 920 MHz which falls in the license free band in Japan. Each device also needs to have a different address with 64k addresses to choose from. In the loop section, we wait for a period of 30 seconds to elapse, read the sensor, convert it to a value, and then transmit it to the receiver. The water level sensor transmitter code can be downloaded here.

This takes us to the receiver side. For this, Iím using the Arashi 900 MHz Ethernet Gateway, another board I made specifically for wireless sensor networks. For the internet data aggregator service, Iím using Xively (formerly Cosm, formerly Pachube). They put out a handy little Arduino library which works great for using their service. You first need to sign up for a free developer account on Xively. Once thatís done, youíll need to create a data feed page. The data feed page will show the API Key and the feed ID and you can basically just plug both of those values into the source code to set up the Xively interface. FYI, I've never gone beyond the free developer account and I have no connection to Xively.

2013-08-24 DSC07338

Once thatís all taken care of, we need to implement the meat of the code. Most of the setup code is boilerplate code like setting up the serial port and chibi wireless stack. The Xively setup code can be a bit intimidating but you can pretty much just copy it verbatim for each use. In the loop section, we just wait until we receive data wirelessly. When we receive data, I'm doing some minor formatting of the data to convert it to floating point and then just take that data and upload it to Xively. Here's the water level sensor receiver code for the Arashi gateway board.

That takes us past the feasibility check so now we know that itís possible to implement an automated, continuous monitoring solution to check the water levels in the storage tanks. If you know what youíre doing, getting here takes the better part of a few hours. This actually makes me a little bit sad since two and a half years have passed since those radiated water storage tanks have existed and they still just have two poor guys climbing each tower to check the water levels every day.

2013-08-24 DSC07322 2013-08-24 screens1

Beyond the feasibility check, we now need to fill in the details and harden the design. As I first mentioned, the remote nodes on top of the water towers will be battery powered. This puts some very hard constraints on the design. For wireless nodes, power is everything. Weíre going to need to put the system to sleep most of the time so that it will consume as little power as possible. Then, weíll wake up, take a measurement, transmit it, and then go back to sleep as quickly as possible. To do this, weíll need to add a switch to power on and off the sensor so that it doesnít suck down power while the system is sleeping. The software will also need to be written to power the system down in the proper sequence and weíll have to figure out a method to periodically wake up for the transmission.

Power management is almost an art form and engineers go through crazy contortions to squeeze out every last microamp of current from a design. In this case, weíre going to make a lot of modifications to the software and some to the hardware to reduce the system power consumption. To start off, the main modification to the hardware is to add a power switch in the form of a PNP transistor tied to an I/O on the MCU. I decided to mill a board to accommodate this. Itís really overkill, but it makes things so much prettier and cleaner. On the software side, we initialize the power switch as an output and then enable power to the sensor.    

2013-08-24 screens6-schematic 2013-08-24 DSC07332 2013-08-24 DSC07330

For the wake mechanism, Iím using the watchdog timer interrupt inside the MCU. Normally, Iíd use the watchdog timer to reset the system in case it hangs somewhere, but it can also be used to wake the system from a deep sleep mode.

The loop function changes drastically too. Now, rather than loop continously, the MCU will only go through one loop iteration and then go to sleep. Basically, the MCU will wake up, do whatever is in the loop once, go back to sleep again, and wait for the watchdog interrupt to wake it up. The powerdown sequencing consists of sleeping the radio first, and then shutting down different parts of the MCU. The UART and ADC consume a lot of power so those get shut down. Also, the I/O are all forced into input mode so nothing can leech power from them. On wakeup, the reverse happens. After all this is implemented, the system now consumes approximately 330 uA at 2.4V in sleep mode and 82 mA in full active mode. If we sleep most of the time and just transmit data every 10 minutes, the systems can theoretically last over 6 months on a pair of AA NiMH batteries. Not too shabby. You can download the modified transmitter code here .

2013-08-24 DSC07327 2013-08-24 DSC07328

To finish things off, I went out and purchased a trash can to use as a mock water storage tank and then modified it to install the water level sensor.

2013-08-24 DSC07325 2013-08-24 DSC07323 2013-08-24 DSC07324

 I tested the sensor on the empty can. It read 460 mm to the sensor with nothing inside which corresponds to what I saw with my tape measure. I then added approximately 140 mm of water and then tested the sensor again. It now read approximately 320 mm to the surface of the water which also corresponds to both the math and what I measured with the tape measure.

2013-08-24 DSC07336 2013-08-24 screens2-in can

2013-08-24 DSC07335 2013-08-24 screen3-in can boxes

This mock setup is similar to the opening at the top of most water storage tanks. If the storage tanks at Fukushima Dai-Ichi don't have an opening at the top, then it still is feasible to cut or drill a small opening to fit a sensor into. Once there's access, it's possible to automate the monitoring of the tank. Speaking of which, you can view the Xively feed for the mock storage tank in real time here. Note: the feed may be up or down since I'm still doing development on it.  

2013-08-24 screens7-xively

I didnít want to get too complicated since this was just a spur of the moment prototype I threw together but there are various things Iíd do if I wanted to take things further. Iíd definitely add a solar input and battery charge circuitry to it so that it could have an indefinite lifetime. Itís already possible to monitor the battery voltage on the board so Iíd also transmit the battery status wirelessly. That way, itís possible to see what the battery charge level is on all devices. Finally, Iíd add an external watchdog reset to improve the reliability of the system. At this point, itís robust enough to deploy and would probably be a much better solution to the storage tank leakage problem than anything TEPCO has currently.

So there you have it. This is my proposal for a solution to the Fukushima Dai-Ichi storage tank leakage problem in tutorial form. Itís both open source software, hardware, uses low cost components, automated, and is much better than anything Iím seeing them use.

Updated 2013-08-24: There has been some discussion on rad-hard requirements for this design. If this is the case, one solution would be to put the electronics in a lead-lined metal enclosure with only the antenna protruding. High radiation fields shouldn't affect communication signals in much the same way as strong light doesn't affect communication signals. I'm also the designer of the radiation monitoring equipment used by Safecast which has been used numerous times in the exclusion zone. There have been no instances of bit-flipping due to radiation that I know of and all the data collected there has come out intact.

 

Updated 2013-09-12: Took the system offline after three weeks. It had a good run, and I'll be repurposing the equipment for some agricultural monitoring projects :)
Hits: 42937
Trackback(0)
Comments (20)Add Comment
I disagree
written by AlBundy, August 25, 2013
I can believe your solution is sonar sensor and wireless transmission. Why? The implemented solution should be design thinking in low maintenance and long live.

The simpler solutions, lesser problems. The application does not needs a great resolution.

Think in float sensors: http://www.ebay.com/bhp/water-tank-float-switch
report abuse
vote down
vote up
Votes: +1
Radioactiv
written by Wilfried, August 25, 2013
Unfortunately, the radioactivity on site is enough to render most electronics useless after a short while. Even so called radioactive-hardened electronics are proving to fail quite quickly, give false positives, etc.
report abuse
vote down
vote up
Votes: +12
Why Run Backwards, You'll Vomit!
written by Matt Saunier, August 25, 2013
Sorry, but I don't trust anyone who crimps network cables, but fails to understand that Bell Operators Give Better Service.
report abuse
vote down
vote up
Votes: +0
...
written by Akiba, August 25, 2013
@Wilfried: I'm pretty sure there are challenges in implementation. There always are, but the main point is to try something out, and iterate as needed. I wrote this to describe one possible solution using commonly available technology. It's far more efficient than the current solution of having two people check each tank individually for leaks. Even if only some of the tanks get automated, the work load on the two men decrease quite a bit for each tank they don't have to check.
report abuse
vote down
vote up
Votes: +3
...
written by Aaron, August 25, 2013
Don't forget to set up some crypto for the transmissions - you don't want a prankster spoofing low water level readings.
report abuse
vote down
vote up
Votes: +0
...
written by KTOWN, August 25, 2013
You would have hoped that a company capable of building and operating a nuclear power plant could figure out how to monitor water level in a container. But then ... look how things have worked out for TEPCO so far, so maybe we shouldn't (sadly) be all that surprised.
report abuse
vote down
vote up
Votes: +1
...
written by Akiba, August 25, 2013
KTOWN!!!
report abuse
vote down
vote up
Votes: +0
...
written by Zeke4ther, August 25, 2013
You might consider incorporating solar power panels in your design to keep the batteries charged.
report abuse
vote down
vote up
Votes: +0
...
written by Akiba, August 25, 2013
Yes, I mentioned towards the end where I'd have solar charging capability. Since this was just a quick proof of concept, I didn't want to go overboard on the design. If I were doing this seriously, that would be one of the things I'd include.
report abuse
vote down
vote up
Votes: +0
Over engineered in my opinon
written by WhizKid, August 25, 2013
Hi, I am sorry but I believe that your solution is over engineered and the point about adding the solar panel and battery charger is simply too much.

For the credibility, I am what they call an 'instrumentation and control engineer'. And I am design automation systems for offshore and onshore plants for a living.

Tank level gauging systems are very mature and there are probably a hundred ways to do it. For tanks the height of ones present in this japanese reactor plant, they usually use a float type level sensor. A motor is connected via a pulley to a float. And the motor rotates to always keep a pre-determined tension in the wire/cable. A tachometer can then be used to measure the liquid height based on the distance motor has traveled.

Another way used is to install strain gauges between the tanks' 'feet' and foundation, the weight will then give you the amount of liquid in the tanks. But this solution is probably out of question now.

The easiest method it the one using differential pressure transmitters. It works on the principle of the pressure difference between two points. One probe can be exposed to atmospheric pressure where as the other can be in contact with the liquid pressure. (and no, these the radioactive water will not have any issue here. These transmitters are designed to withstand the most corrosive fluids. Also the probe can be hundred of meters long. The pressure drop across that distance can be calculated easily). Now the tanks will definitely have quite a few water pipes leaving from it. Usually those kind of lines have tee-off points for various purposes, to drain, to sample, to connect a calibrated pressure / flow meter for maintenance purposes. We can connect a pressure differential transmitter to any suitable place. May be right onto the tank or a few hundred meters from the tank. Depending on the site situation.

These plants are designed to what we call 'SIL-4' the highest 'Safety Integrated Level' (you can read a wiki article on it if you google). SIL 4 calls for a probability of failure in a hour of 10^-8 or 10^-9. Even the most critical systems on an oil plant doesn't require more than SIL-3. SIL-4 is specifically designed to use with Nuclear power plants. Usually SIL-4 can only be satisfied using what we call 'triple redundancy technique'. Here three instruments are installed to measure the same process parameter and the result from all 3 is voted. If one of the instrument value is different its cast out as faulty. So these tank must also have atleast 3, not one tank level measuring system.

I would also like to emphasize that wireless transmission and power via solar panels is not a requirement here. Due to SIL-4 requirements the plant would have several pair of cables going to the tank for power and instrumentation. Most of the instruments are what we call 'loop powered' and use the measuring line as power too. These only need 4mA to power up and work. Thus if the engineers at the plant are not able to make this instrumentation work there must be a pretty good reason for it. Most probably due to unavailability of physical access to the tank it self.

Please also note that these tanks were 'designed' to hold radioactive water. So the sensor system used would have been chosen keeping this design criteria in view. So please don't talk to me about radiation hardened systems or bit flipping. Nuclear reactors are around for almost 60 years and all the instrumentation technology surrounding it is very very mature.

This whole saga reminds of a story I heard during one of the seminars I was attending. In a particular packaging plant they were facing a problem of empty boxes ending up on the last conveyor. The engineers used several methods. Using a strain guage to measure the weight and then use a pneumatic cylinder to push away the empty boxes. or using x-rays to analyse the contents etc. to the cut the story short nothing worked reliably. One of the technicians then solves the problem by directing an industrial grade fan towards the conveyor and it simply blows off the empty boxes. So may be we should also try using the fan in this case too smilies/smiley.gif

Cheers
report abuse
vote down
vote up
Votes: +10
...
written by Akiba, August 25, 2013
@WhizKid: Thanks for the insightful post. It's interesting to hear how things would be done from an instrumentation engineer's point of view. A big question for me is that if its not so difficult to do, why haven't they done anything yet. Any system of automated measurement is better than none at all. If you're correct and its possible to get power and comms cables to the tops of the towers, then things will be much simpler. My experience though is that cabling is difficult when instrumenting equipment which is why wireless is an interesting alternative for industrial sensing.
report abuse
vote down
vote up
Votes: +0
...
written by WhizKid, August 25, 2013
@Akiba, To be frank I am not sure whats going on in there. We, the out side public doesn't know all the facts. What I am trying to say is that, this whole issue is not a problem from an engineering point of view. The tanks will most certainly have some kind of instrumentation still working. But I don't really want to try an propose a solution when I don't have all the information.

However I would want to clarify the point about using wireless sensing in an industrial setting. Wireless sensing is almost never used to monitor a critical plant parameter. There are certain applications which are exceptions such as UGV used to transport materials in a warehouse or shop floor but you would certainly not want a distillation towers' pressure or temperature or a flowmeters' reading via a wireless transmission.

The primary reason here is what is defined as an Isochronous data transfer. In a process control setting, the data or information about a particular parameter is as important as the time at which it was generated. Almost all the industrial data transfer protocols (Modbus, Fieldbus, Profibus etc) cater to it. This usually means a guaranteed response time of 100ms or less. This the same reason why these protocol don't cater to resending the data if it is not received by the process controller (i.e there is no handshaking or acknowledgement of data received). Because 'late data' is as bad or even sometime worse as no data. On the other hand information loss is a big factor and catering to it is a big part of all wireless protocols. A wireless transmission will almost never be able to keep up with the stringent requirements needed by a closed loop controlled process. This is the reason why you will see oil refineries which span several square kilometers but every last sensor is connected to the main control station via a cable.
report abuse
vote down
vote up
Votes: +1
...
written by Akiba, August 25, 2013
I agree and I wouldn't use wireless for a critical or time-sensitive parameter. In this case, the problem is that the alternative is manual observation which is completely inefficient. If there was an automated solution they had plans for it's a different story. For the water level monitoring in the storage tank, its not a critical factor to the plant operation. Being off by a minute or ten is probably not serious. I'm for any solution, cabled or not, and this was just a proposal to hopefully start off a discussion. As an engineer, I'm interested in the best solution in the proper timeframe. Whether its cabled or wireless doesn't matter as long as it works.
report abuse
vote down
vote up
Votes: +2
...
written by chickeee, August 26, 2013
I think TEPCO ismonitoring the water levels but is unable to find/fix the sources of the leaks. I don't know why that is tho
report abuse
vote down
vote up
Votes: +1
...
written by Rob S, August 26, 2013
Why not IR thermography?
report abuse
vote down
vote up
Votes: +1
...
written by Rahul Sen, August 26, 2013
It is very interesting to have discussion about wireless communication in industrial domain. But as i understands, wireless systems were never meant to replace the existing system. As per ietf RFC5673 "Industrial Routing Requirements in Low-Power and Lossy Networks" section 3- http://tools.ietf.org/html/rfc5673
"Cable is perceived as a more proven, safer technology, and existing,
operational deployments are very stable in time. For these reasons,
it is not expected that wireless will replace wire in any foreseeable
future; the consensus in the industrial space is rather that wireless
will tremendously augment the scope and benefits of automation by
enabling the control of devices that were not connected in the past
for reasons of cost and/or deployment complexities. But for LLNs to
be adopted in the industrial environment, the wireless network needs
to have three qualities: low power, high reliability, and easy
installation and maintenance."
Apart from this also check Annexure I of IEEE 802.15.4e-2012
http://standards.ieee.org/find...-2012.html
report abuse
vote down
vote up
Votes: +0
...
written by Akiba, August 26, 2013
Yeah, I think that was what I was getting at. If the water towers are difficult to cable or would take a long time to cable up, then wireless is a viable alternative. If cables are already present, then it's always the best solution and should be used instead.
report abuse
vote down
vote up
Votes: +0
Good solution
written by MattT, August 28, 2013
I think some folks are losing sight of the scope of the project. This device isn't intended to help manage a high speed mission-critical operations process. It's just intended to reduce the need to send people out into a highly hazardous environment to check for something as simple as a leak. It's not ben necessary to know how full the tank is, just whether it's becoming accidentally less full. It's certainly not over-engineered: There's a part that determines how far the water surface is from the top of the tank, and a part that sends that information to monitoring staff--with an optional part to keep the power source charged up and reduce the need for sending someone out to change the batteries. Simple and straightforward. Great job!
report abuse
vote down
vote up
Votes: +2
...
written by Akiba, August 28, 2013
Thanks for the kind words smilies/smiley.gif
report abuse
vote down
vote up
Votes: +0
Radiation hazard
written by Mike Swift, August 29, 2013
If radiation levels are low enough to allow a worker to visit each tank each month or so the levels are not going to be high enough to cause problems with your measuring equipment, or radios. Besides the only part of your system that would have any visibility to the radioactive water would be the front of the sensor. All other parts are on the top of the tanks, and could be shielded from most gamma radiation by just placing a steel or lead plate under the equipment.
report abuse
vote down
vote up
Votes: +1

Write comment

busy