Sorta OT: ICS Hack at Florida WTP

There has been reporting in the past day of an intentional intrusion into a drinking water treatment plant in Florida, near Tampa Bay, in which an intruder intentionally increased the sodium hydroxide pump to 100x its normal setting, which would have made the water dangerously alkaline if it had not been caught quickly.

Because the general-interest press likes to hype up the (real) spectre of terrorist attacks against water infrastructure, it's being called a control system "hack", and there's been talk of powerful international cyber weapons.

The details I've been able to see tell a much simpler technical story: the intruder used a remote desktop tool to access the SCADA computer, and the supervising engineer literally watched them use the mouse and keyboard to enter a higher setpoint for the NaOH dose pump.

It still could have been a nefarious international terrorist intent on poisoning the Super Bowl. But the reporting, both from national and international news outlets and the Tampa Bay Times, is that the plant had an ordinary remote access system, which they used regularly.

Evidently the hacker accessed the system twice: the first time, an operator assumed the remote access was his supervisor working from home.

'Scuse me while I go change my remote access credentials...

Its scary how many systems i come across in the water industry that are poorly protected. VNC with no password, no VPN etc.

The ability to change a SP to such an abnormal value though is also a control system design failure. Our drinking water standards here have hard limits on certain parameters, and i always hard code these. So even if someone did manage to increase the SP for a chemical dosing system, the final water monitoring will pick it up and shut the plant down.

Of course if the intruder is into the PLC then all bets are off.

What I see often is that an organizations IT department don't facilitate any remote access options, so the resourceful operators find a way around. You can't manage what you don't know about, and for them it's all about being able to do their jobs without needing to drive 45 minutes at 2am to swap duty on a pump or adjust a dose rate.

I'm lucky that my main client here has an excellent IT security consultant who has built a robust and secure VPN network. Two factor authentication using smart phone to get into the VPN, then Citrix to bring up a SCADA terminal (followed by login to change anything on SCADA). PLC remote access restricted to small usergroup, everything logged. Traffic between VPN and plant networks restricted to handful of ports only.

Its possible to make this kind of thing secure if you know what you're doing, but it's a combination of defenses required, not just a single thing like a password or a firewall.

Surely the SCADA had users and password protection ?
Also, no limits on dangerous setpoint values. That a value is set to 100x its normal value should by itself have triggered an alarm.

There should be 3 layers of protection plus procedures must be in place.
- Access to the internal network should be by login.
- The devices on the network should all be password protected, wether HMIs or PLCs.
- HMI and PLC programs should have limits and checks of critical parameters.
- Also, there must be procedures in place in case anyone is granted access from outside. I.e. positive identification and an personal login - not using someone elses login.

Every day when I go home after working in the water industry I give thanks to God for my personal well. And I tell all my family and friends, what ever you pay for water, it ain't enough. The infrastructure is largely junk. I will drink pond water, but I won't drink some of the water in certain municipalities. Sweet tea at lunch in (unnamed town)? No thanks, I'll have a Diet Dr. Pepper in a can please.

I have seen some Teamviewer password that are stupidly simple. So simple two different towns hundreds of miles apart thought of the same stupid one. I have seen one unsecured VNC connection too.

If we get involved in the controls, we do hard limits on dangerous settings and add security controls inside the SCADA as well as more controls on who and how to get there to begin with.

TeamViewer can be dangerous

https://www-vice-com.cdn.ampproject...3/hacker-poison-florida-water-pinellas-county

The problem is not with Teamviewer, it is with only having a single barrier in the shape of a username and a password, which may become compromised.

JesperMP said:

there must be procedures in place in case anyone is granted access from outside. I.e. positive identification and an personal login - not using someone elses login.

Click to expand...

It may not be obvious from what I wrote, but the "positive identification" means 2-factor login, or that a timelimited password is provided by the site being accessed, or some other similar robust method.

"...The hacker changed the sodium hydroxide from about 100 parts per million to 11,100 parts per million," the Sheriff explained.

So the hacker changed the setpoint to over 100x the normal setpoint. That would mean that the chemical feed pump would run at 100 times its normal speed. No. Chemical feed pumps are usually oversized by only a factor of 2 or so from their normal operating rate. Metering pumps tend to be non-linear at the very low range of their speed and wouldn't dose properly if they were that oversized. I think the programmer forgot to clamp the setpoint to a reasonable range or even one the pump could obey. That's a bug apart from the suspected hacking.

I already have a nervous customer who read this news report asking me to remove all the chemical feed setpoints from their HMI. Uggh. There's presently no other way to change some of them. Thanks for the work, but this is not useful work.

KirkC said:

I already have a nervous customer who read this news report asking me to remove all the chemical feed setpoints from their HMI. Uggh. There's presently no other way to change some of them. Thanks for the work, but this is not useful work.

Click to expand...

Discuss with your customer how to implement security. Instead of patching the various parameters with limits, the real issue is that an intruder could gain access in the first place. There are simple and robust methods to make gaining access for an intruder much harder, if not impossible.

Bet Teamviewer will come out with the 2step verification and protection, in case they don't have it already.

JesperMP said:

... if not impossible.

Click to expand...

It's never impossible: if you can get in, they can get in.


Detection, damage control, and recovery are the key areas to focus on, because ultimately it is going to happen.


I'm not saying don't harden defenses as much as reasonable, just don't think hardening is the end of due diligence.

JesperMP said:

Discuss with your customer how to implement security. Instead of patching the various parameters with limits, the real issue is that an intruder could gain access in the first place. There are simple and robust methods to make gaining access for an intruder much harder, if not impossible.

Click to expand...

Nothing is impossible in the software world.


Even with strong IT service at multinational companies. If you are from dennmark, have you heard of Maersk, one of the biggest boat shipping company? Their whole network was wipped out in a few hours, worldwide, in one day. A single piece of software was compromised on a single computer and then the virus used a zero day exploit in windows to spread and it destroyed all their network.


The (long and fascinating) story here https://www.wired.com/story/notpetya-cyberattack-ukraine-russia-code-crashed-the-world/


We have also seen recently hackers compromising an IT network software manager called Solarwinds and distributing a backdoor embedded with the offical updates of the software. Most of the biggest US companies including the likes of Microsoft and Cisco and US federal administrations were running the trojan horse infected software. It went unnoticed more than one year.


The safest will always be to not connect a PC to the network and limit the possibility of remote access to the smallest.

JesperMP said:

Discuss with your customer how to implement security. Instead of patching the various parameters with limits, the real issue is that an intruder could gain access in the first place. There are simple and robust methods to make gaining access for an intruder much harder, if not impossible.

Click to expand...

setpoint limits also protect against in house stupidity or mistakes....

Iner said:

Nothing is impossible in the software world.

Click to expand...

That is what I said.

Iner said:

Even with strong IT service at multinational companies. If you are from dennmark, have you heard of Maersk, one of the biggest boat shipping company? Their whole network was wipped out in a few hours, worldwide, in one day. A single piece of software was compromised on a single computer and then the virus used a zero day exploit in windows to spread and it destroyed all their network.

Click to expand...

Yes I am well aware of that case. Tt was professional hackers, and they used a zero day exploit, something that there arent that many of.

What the water works mentioned in this thread did is analog to putting the key to the front door under the mat.

There are 2 simple methods that will make it much much harder for an intruder to gain access:
1. 2-factor login.
2. Instead of perpetual access to the internet, the internet connection and the logon is only granted from someone on the inside, and then only for a timelimited period.