EC News DeskOn 5 February 2021, unidentified cyber actors obtained unauthorised access, on two separate occasions, approximately five hours apart, to the supervisory control and data acquisition, SCADA, system used at a local municipality’s water treatment plant in Florida. The unidentified actors accessed the SCADA system’s software and altered the amount of sodium hydroxide, a caustic chemical, used as part of the water treatment process.
Water treatment plant personnel immediately noticed the change in dosing amounts and corrected the issue before the SCADA system’s software detected the manipulation and alarmed due to the unauthorised change. As a result, the water treatment process remained unaffected and continued to operate as normal.
The unidentified actors accessed the water treatment plant’s SCADA controls via remote access software, TeamViewer, which was installed on one of several computers the water treatment plant personnel used to conduct system status checks and to respond to alarms or any other issues that arose during the water treatment process.
All computers used by water plant personnel were connected to the SCADA system and used the 32-bit version of the Windows 7 operating system. Further, all computers shared the same password for remote access and appeared to be connected directly to the Internet without any type of firewall protection installed.
Here are some of the top industry comments.
Marty Edwards, Vice President of Operational Technology Security, Tenable.
The attack against the City of Oldsmar’s water treatment system is what OT nightmares are made of. If successful, the damages of the attack would have been catastrophic. This story highlights just how quickly and covertly a subtle, and potentially deadly, change can be made. This is precisely why the security community has been warning about the rising threats to OT for the last decade-plus.
The days of isolated OT networks are long gone. In its place is a highly dynamic and complex environment of smart OT technology, modern IT and everything in between. Attackers have capitalised on these converged networks to move laterally from one system to another, making the compromise of just one device even more dangerous.
Luckily, the plant operators were able to detect the unauthorised changes to the sodium hydroxide levels immediately. Had it not been for their quick actions, this story could have ended very differently.
All organisations that operate critical infrastructure, such as water supplies, must invest in the people, processes and technology required to keep these systems safe. This wasn’t the first attack of its kind and it certainly won’t be the last.
Morgan Wright, Chief Security Advisor at SentinelOne.
It was fortunate that one of the plant operators happened to see the unauthorised remote access and was able to take preventive action. However, luck should never be the cornerstone of effective network and endpoint protection. Remote access software is a necessary tool in managing vital systems like water and power. Yet, time and time again, the steps taken to protect unauthorised access pale in comparison to the critical nature of the infrastructure the software is deployed in.
These older systems were never designed to be connected directly to the internet. As a result, they lack the robust security needed to defend critical resources. Beyond that, it has also been discovered that the remote access software used a weak, shared password. Fortunately, the attacker appears to have lacked the sophistication of a well-trained nation-state adversary. Otherwise, even being lucky would not have been enough to stop a terrible tragedy.
Daniel Kapellmann Zafra, Manager of Analysis, Mandiant Threat Intelligence.
Since last year, Mandiant Threat Intelligence has observed an increase in cyber incidents by novice hackers seeking to access and learn about remotely accessible industrial systems. Many of the victims appear to have been selected arbitrarily, such as small critical infrastructure asset owners and operators who serve small populations. Through remote interaction with these systems, actors have engaged in limited-impact operations but none of these cases has resulted in damage to people or infrastructure.
Fortunately, industrial processes are often designed and monitored by professional engineers who incorporate safety mechanisms to prevent unexpected modifications. We believe that the increasing interest in industrial control systems by actors of this nature is the result of the increased availability of tools and resources that reduce the barrier to learn about and interact with these systems.
While the incident does not appear to be particularly complex, it highlights the need to strengthen the cybersecurity capabilities across the water and wastewater industry similarly to other critical infrastructure sectors.
Sam Curry, Chief Security Officer, Cybereason.
With the US Secret Service and FBI involved in trying to determine the cyber culprits poisoning the Pinellas County, Florida water supply, this is another reminder that cyber threats against critical infrastructure networks are real.
For nearly one year since the beginning of the Covid-19 pandemic, threat actors have carried numerous acts of war against research companies, hospitals, and other first responders. These attacks are brazen, shocking, and downright maniacal. While this attack was not against Florida’s two largest counties, Miami-Dade or Broward County, any attempt to poison a water supply should raise the eyebrows of local and state officials.
What is surprising about the manipulation of chemical levels in Florida’s water supply is the bad actors tipped their hand without first doing proofs of concept or stockpiling attacks for later use. What we do not know is if any successful attacks have taken place over the past few months and possibly not reported.
It is premature to infer what the motive of the attackers was and who they are. The actor at this point could be script kiddies, terrorists, criminal ransom, nation state of any other actor. The correct response should be due process, investigate, understand, learn, improve, follow the investigation and data, and constantly get better.
Acts of War are determined by the State and among states. If the US can point to a culprit and say it is, then that is what matters. The details thus far are scant, but we will all be listening to the post-mortem and hope the current administration provides a deeper response and holds the adversaries responsible for this act. To be clear, the investigation is what matters. Where it leads, who it involves and how we interpret that are all to be determined.
Oliver Tavakoli, CTO at Vectra AI.
There is a spectrum of critical infrastructure, from nuclear power plants on one end to relatively small water treatment facilities at the other end, and IT spend, and the maturity of security practices employed reflect that spectrum.
The attack on the Oldsmar water treatment plant leveraged a remote control application that did not appear to have been deployed in the most secure manner and had been superseded by a newer tool several months before the attack.
Leaving mothballed access methods in place after they have reached end-of-life certainly speaks to a lack of mature security practices.
But the bigger question comes down to how much we are willing to budget, in this case, in the form of higher water rates, for smaller organisations to keep themselves reasonably secure. Or should smaller organisations look to form regional groups to handle IT and security as the scale required to achieve reasonable operational agility and security becomes too costly at such a small scale?
Ram Narayanan, Country Manager, Check Point Software Technologies Middle East.
Supervisory Control and Data Acquisition or SCADA as it is popularly known, is an automated software control system that monitors industrial control systems, ICS, and provides data insights to industrial supervisors about the condition of the entire operation.
While the system is automated, the data output is monitored by human eyes in a separate control room through a graphical user interface. Data is collected via various sensors, controllers and IoT connected devices and monitored by a SCADA software. Operators can use the interface to make changes to the operating processes in real-time. These changes might include turning valves on or off and or changing the temperature of a thermostat and so on.
In the past 2 years, ICS environments are targeted by hackers who try to disable or hijack controllers and sensors of key processes. Also, every IoT device connected to the network feeding data into the system is a potential security vulnerability. Many of today’s attacks on OT and ICS networks are found to be based on IT attack vectors, such as spear phishing, endpoint and ransomware.
Generally, Operational Technology, OT, systems lack basic security controls and have vulnerabilities like legacy software, lack of encryption, and default configurations that make it easy for attackers to enumerate and compromise OT systems. Systems connected to unaudited dial-up lines or remote-access servers also give attackers convenient backdoor access. Security gaps are also created when IT and OT personnel differ in their approach to securing industrial controls.
Industrial control system environments need to be properly secured to prevent cyber criminals from attacking. The major threats that such OT systems face are DDoS attacks, Web application attacks, malware, and command injection and parameters manipulation that allows invalidated data not verified as legitimate system traffic which allows attackers to execute arbitrary system commands on OT systems. The increasing connectivity of industrial control systems, ICS, and the convergence of OT and IT networks expands the attack surface of industrial manufacturing and critical infrastructure facilities.
Here are our top three guidelines for protecting an ICS environment from cyberthreats:
Monitor systems
If you don’t monitor your control systems, you are vulnerable to attack. The best way to monitor your ICS is with SCADA.
Proper segmentation
Make sure your cybersecurity solution enables clear segmentation between OT and IT or Internet, to stop Internet threats from crossing to OT environments and disrupting processes or causing damage. With access control, restrict access to resources in OT environment and all networks.
Utilise threat intelligence
Stay ahead of new threats with threat intelligence solutions that aggregate intelligence from 100 million endpoints, gateways, and IoT devices worldwide.
Cameron Camp, Security Researcher at ESET.
In the Florida case, criminals used remote access tools to gain a foothold and change chemical levels in the water supply, ramping them up to potentially hazardous levels.
That is worrisome, including because hackers would normally have to gain specific knowledge of water treatment management systems, a very specific target demographic. That is not a spray and pray attack; it is targeted and takes some time to craft and deploy. And while this incident wasn’t a super stealthy zero-day attack, chances are that somebody was interested in the target for quite some time.
First, the attackers identify the target, they gather information and form a plan. Once access has been gained, they then need to scour the network for the control systems that interact directly with the water treatment process. Again, this can take significant time and planning.
Once potential targets have been identified, attackers need to understand what role those targets have in the chemical process and what access those systems have to the physical equipment involved in production, whether valves, relays, level sensors, thermocouples or other controls.
Then they have to craft a specific attack within the context they are able to assess along the way, and then launch at a precise time that would have the best odds of success, all while maintaining undetected access to all the systems in the chain.
In the case of Oldsmar, once the attack was launched, there were other systems in place that provided feedback that could alert staff in time to scuttle the attack. That is the good news. The bad news would be that they might have been under silent attack for weeks or months prior to the actual poisoning attempt and did not know it.
My colleague Tony Anscombe wonders why the Oldsmar facility did not have a thoroughly vetted and implemented plan in accordance with CISA sector-specific guidance for water and wastewater systems, including measures like two-factor authentication, 2FA, and similar controls.
It is very helpful that those guidelines are made available for small municipalities to ramp up quickly, even if they don’t have access to cybersecurity ninjas on staff which can be very expensive with typical small-town budgets.
Meanwhile, expect to see future exploit attempts against other municipalities. Ransomware attempts would be an obvious follow-on trend.
What can small towns do? They should take the time to understand and implement the guidance available, which may be as simple as adding or enforcing 2FA, patching systems, implementing good change control processes and training staff on cyberhygiene.
Also, do a practice drill assuming a breach and think like a hacker to stop them from getting in. It is a good idea as well to have a plan in place in case a ransomware attack happens; that way, small towns won’t be faced with the untenable prospect of explaining to the citizens why they just spent public money to stop an attack that shouldn’t have happened in the first place.
Stefan Schachinger, Product Manager, Network Security – IoT, OT, ICS at Barracuda.
A key takeaway for many organisations is to ensure that their remote access solution is configured correctly as this is a key tactic used by hackers to gain access. Given that this method of remote access is very popular, many other companies in the utilities area and in other verticals are at risk of becoming victims of the same kind of attack.
It also demonstrates just how significant the impact of such an attack could be, as a serious threat to people and the environment is possible. In this case, fortunately the attacker was not very cautious, and a diligent employee was able to stop him.
The pandemic has been a big driver of digitalisation, and many organisations have had to introduce new means of remote access to overcome limitations in mobility. However, the majority of insecure remote access solutions have been around for years. It is just a matter of time until somebody with bad intensions finds out. The reality is, air-gapped systems have become rare and even the last ones remaining will surely disappear.
That said, it is important to highlight that remote access is not insecure by definition. Instead of using screen sharing tools, my suggestion is to work with either a VPN or Zero Trust Network Access, ZTNA, solution with multifactor authentication.
Traffic from remote devices should be inspected to ensure there is no malicious software entering the OT infrastructure. Have a granular firewall rule set in place to allow what is necessary only, and use antivirus, IPS systems and Advanced Threat Protection for analysis. Especially if maintenance personnel from other companies, such as machine vendors, require remote access to certain assets, an approval mechanism should be implemented. Furthermore, that kind of remote access should be disabled automatically after a specified period to avoid systems being unintentionally exposed for extended periods.
The perimeter is just one aspect, but of course it is important to protect the OT network against threats from outside in the best possible way. Secure remote access is key. The other question is how we can protect the OT network against threats from inside. Having to deal with outdated and vulnerable systems is not unusual.
In that particular case at the water utility in Florida as far as I know Windows 7 was in use. But that is just an example, and many companies have to deal with similar challenges as a result of long machine lifecycles in OT. Because it is not possible to protect the system itself, for example by updating or installing endpoint security, the system needs to be protected and isolated as far as possible.
Micro-segmentation can divide the OT network into smaller segments where only legitimate traffic is allowed. In conjunction with full next generation security on firewalls between the segments, that is a powerful tool. That concept is also known as virtual patching. If an attacker or malicious software finds a way into the network, for instance on a USB stick, it cannot spread.
An insecure remote access tool is rather easy to resolve. That could be the low hanging fruit to begin with. The next step should be to prepare against threats which find another way inside the network. There are so many attack vectors, and each one needs to be considered. At the end of today, defense in depth is the only way to achieve a high security level. Different layers of defense will give the attacker a hard time.
Julissa Caraballo, Product Marketing Manager at Beyondtrust.
With OT systems, such as SCADA systems and ICS, increasingly exposed to the Internet and often easily discoverable by tools like Shodan, it is imperative that the pathways into the environment be properly locked down. Time and time again, exploits such as this one demonstrate that basic or unsecured remote access tools absolutely do not cut it in any environment where security is an important consideration.
Government agencies and enterprises throughout the world have an immense responsibility to keep citizens and customers safe, and their data secure at all times. Unfortunately, the use of consumer-grade remote access tools is rampant, and frequently inadequate, inappropriate, and out of compliance, for the use cases to which it is applied.
As federal and state agencies continue to be the target for threat actors, it’s incumbent upon IT and security teams to harden remote access to protect critical infrastructure and sensitive data to the level it demands and the public expects.
