CISA released an urgent message warning water utilities to disconnect their logic controllers from the internet in the face of rising cyberattacks.
The hacks target internet-facing programmable logic controllers (PLCs) that control equipment and allow machinery to communicate. They monitor and control the water pressure, chemical dosing, and other factors to ensure the water is safe.
Many of the PLCs are apparently open to the internet and use default credentials, allowing a remote attacker to easily take them over.
I assume it is this CISA: https://en.wikipedia.org/wiki/Cybersecurity_and_Infrastructure_Security_Agency
But if they do that then how will one guy monitor and manage 20 plants at the same time?
OT networks are utter shit and everyone has known that for at least a decade.
I will take things that never should be connected to Internet for $200 Alex.
Remote monitoring? Sure. Remote managing? Fuck that. These facilities shouldn’t be unstaffed for days at a time.
It’s called paying people to watch the systems during off-hours. It’s often way cheaper than setting up firewalls, setting up alerts, maintaining patches, paying a VPN company to manage connections, paying another company to handle authentication, and so on. Security constraints are getting so bad at my office I’m seriously thinking of how we can do more things with pen and paper.
The things you mentioned probably aren’t the problem. It’s not hard or expensive to setup secure remote access for systems.
The real problem is that a lot of this very expensive and very specific infrastructure equipment is also very old, and frequently does not support newer and more secure protocols.
Source : I’ve been trying to get gas generators to fire off email alerts using modern authentication with conditional access for about a week now.
Changing tech is usually a lot harder than adoption. Better methods and protocols will keep emerging to fight new digital threats but with physical threats usually don’t change. Only exception is social engineering (e.g. attacking a locked door by tricking someone into opening it). Even then a physical presence is required and someone can’t enter a locked room from 2000 miles away.
Security constraints are so bad at home I want to just buy paper books or go to library.
Oh for sure, air gapped sensors that relay information are going to be necessary. We should be building these systems with absolute security because they are critical to infrastructure.
Some people will say it’s too hard to retrofit onto legacy equipment, but all it really means is adding a secondary system on top of the legacy system. Just a sensor array that has no interaction with existing systems.
Right. Why would you do this? Monitoring being available on the Intranet, of course. But controllers on the open internet?
Might as well really focus on ease of use and just make a live website with one big button that says “cut off all electricity and turn all the water into poison, lol”
And guarded by default passwords
It’s so fucking dumb. They are getting attacked, and they still need to be told to unplug? What the actual fuck?
yeah, but like…that won’t happen to me.
security through obscurity is nothing but a fantasy.
It is much more like yeah but like… That isn’t my job. I just scoop out the human poop.
The hacks target internet-facing programmable logic controllers (PLCs)
Why the fuck
is your PLC
facing the fucking internet!Jesus Zombie Christ an a pogo stick. Has no one been paying attention for the last two decades? Seriously, we learned this sort of lesson in Two Thousand and fucking Three. Your critical assets do not get public IP addresses.
You’d be surprised. Especially on city utilities.
It usually happens in 2 steps.
First step is that everything gets connected to the LAN and you can only access the PLC network from within the building’s network. Then some time later, management finds out that keeping someone on-call to go out costs a fortune, so they request that access be made so they can make the change from anywhere.
The IT team argues security, but no new hardware can be provisioned and a developing a new process is too hard. Then the magical phrase is uttered: “Just make it work”. So IT punches a hole in the firewall, adds a NAT rule, and job done.
Except you skipped the part where IT is ridiculously underfunded and treated like crap (“We’re a utility, not a tech company!”)
With constant turnover, the person receiving the “Just make it work” call is about 6 layers removed from the person who originally set up the infrastructure, and probably completely unaware of all the previous security holes put in “because it’s air-gapped, it doesn’t matter.”
Sadly yes, this is exactly what happens. And it ends up being IT holding the bag at the end, with the managers having long since cashed out their stock options and left.
“Well, we just bought this new monitoring platform from my cousin’s company and they need access to it. No, their platform doesn’t support any real secure protocols and won’t route over a VPN, it needs to be on the same subnet as the devices. Ok, then just open all ports to the network so they can connect. No, they don’t know what port or protocol, it also runs out of my cousin’s house so it’s on a dynamic IP so you can’t create some type of access rule for it. Just put an any/any rule in and it’s good. What do you mean we already have a monitoring platform? Well, this one is better!”
“No, their platform doesn’t support any real secure protocols and won’t route over a VPN, it needs to be on the same subnet as the devices. […]”
They won’t know that much. They don’t understand that their job isn’t running a water department, it’s running the computers that maintain the water department and that includes security. For someone familiar with computer security it’s easy to recognize negligence, but until the law sees it as criminal negligence nothing’s going to change.
Small town municipal water departments aren’t going to be able to afford the guy holding the shield against international state actors.
Literally this. It starts with “everything needs to be on an air gapped LAN so we can control it.” And then suddenly that air gapped LAN isn’t air gapped, because someone got tired of paying the call-out fee every time the on-call person had to drive to the site to push a few buttons on a keyboard. But figuring out a proper VPN (or at least a reverse proxy with access control) was too difficult or expensive, (management thinks the point is to save money on call out fees, not spend money on proper hardware) so they just poked a bunch of port-forwarded holes in the firewall and called it a day.
What’s extra ridiculous is VPNs are not that hard. I am fantastically awful with networking and even I can wrap my head around a WireGuard tunnel.
The issue is doing one at scale, with proper access controls. Also, let’s not skip past the whole “network isn’t air gapped anymore” side of things. Now you’re having to maintain firewalls, patch firmwares, manage users, enforce good password hygiene, etc… All without any additional IT support or hardware, because management is doing this as a cost cutting measure.
I can pretty much guarantee that the systems that get hacked aren’t the ones that do everything right. They’re the ones who have a single shared VPN password, haven’t updated their firewall since it was installed 8 years ago, and they haven’t even changed the default passwords for their control systems. Because they got tired of answering phone calls about what the password was, so they just made every username and password something stupid like admin/admin.
“network isn’t air gapped anymore” side of things. Now you’re having to maintain firewalls, patch firmwares, manage users, enforce good password hygiene,
This is all shit they should be doing now. They have to have a firewall to get to the net, so they should be maintaining and patching it. Managing users is standard AD shit, and password hygiene is t as easy, but setting reqs in AD for complexity goes a long way (as does not forcing passwords to rotate on an arbitrary basis).
And yes, I know it doesn’t happen in a lot of places, which is why I said ‘should’. Having been on the cleanup side of multiple security incidents, doing the basics is harder than you’d expect for a lot of places …
Not only that but it said many of them were using the default login credentials
I did some work for a company that owned a /16 public IP range and used it on their internal network. I had to put a safety rated control system on it that ran mining machines that weighed 60 tons and cost a couple of million bucks each.
They gave me a subnet range for the system that was clearly non-private and I was like, “ummmmm, is that on the Internet?”
Apparently it wasn’t routed to the rest of the Internet but it always felt like it was one misconfigured router away from disaster.
I am repeating what I said elsewhere on a different comm:
I call FULL bullshit on this being Iran.
I personally worked on the cybersecurity incident that happened to St Paul municipal and water systems last year as the Incident Response Technical lead.
The threat actors were not Iranian then. They were Russian.
I personally rebuilt most of their networks, VMware hosts, storage, and servers. The majority of the VMware environments did not have config backups and were running 5 and 6 branches of VMware, only two had 7+.
Their network was insecure. They had IPSEC VPNs between the various location, but there was no further security in that respect. Their firewalls were not being kept up. AAAAND… I warned them that we thought they were still compromised.
In fact, the forensics team called them out as being clean on a group call one week into the incident, but while we were on the call with leadership, they got RE-infected.
I had my team compile a massive set of recommendations for them and I wrote the report. Water Systems specifically. They did not listen to us, apparently. The forensics team also did this.
They had FBI, national guard (fucking incompetent), and law enforcement (useless) in all meetings.
If they got into Water, they’ll get back into city and municipal again. I guarantee it.
This is not Iran. It’s the fucking incompetence of city government and Russian threat actors.
Minor nit: Russia and Iran are already working together on shaheds and targeting
No surprise at all if they also coordinate cyber
Whatever. I worked the fucking case. The infiltrators were Russian. After that, it’s a political or financial negotiation.
Yeah and Russia shares intel with Iran.
Not saying I ever accessed it, but there’s a generator in Germany open to the web without security. It makes a few MW. My guess is hospital or any other critical infrastructure.
Went on the public tour of our local waste-treatment facility. Really fascinating stuff. Highly recommend it if you’re even slightly curious.
There are definitely industrial-scale automation systems. Mostly PLCs, but also SCADA for monitoring. Lots of 20-30 year-old tech. Those networks have no reason to be connected to the internet.
But there were also office equipment and personal devices everywhere. All it takes is for someone wanting to monitor the state of a digester on their laptop from home.
You just crossed the beams.
definitely industrial-scale automation systems. Mostly PLCs, but also SCADA for monitoring. Lots of 20-30 year-old tech. Those networks have no reason to be connected to the internet.
You’re talking a massive nation with more systems like this than you could ever imagine. Probably more than all of Europe combined.
They are in every state of repair and disrepair that you could imagine, and many have been receiving government funding for at least at decade now (def more, but I can only speak to the last ten years or so) to replace these systems, including SCADA and PLCs.
So you have a patchwork of countless systems, in every level of tech modernity that you can imagine.
The newer systems usually have an internal network of sorts, and they might have a portal for someone outside the network to log in to remotely monitor.
The problem seems to be that these idiots kept the default credentials, etc.
But to answer you’re question, they’re not all 20-30 years old, and network connectivity (without being open to the outside internet, obviously), is a very useful feature when replacing those old systems with newer ones.
That’s why.
Automation engineer here who does a lot of SCADA
There are soooo many good ways to securely monitor stuff from home. But especially in critical infrastructure it’s just usually air gapped which is unbeatable in cyber security.
Tours of any local facility are super cool! I find it all so interesting, seeing how the things we use work!
not so funny when it isn’t happening in the middle east but right at your doorstep
A lot of this is often down to shitty IT practices by the vendor who has little IT knowledge let alone decent security insight.
I helped support a new PCL installation (technician control PCs and software upgrade) in this exact scenario many years ago. The vendor came in expecting to treat our system as their own. Direct remote access to several over-specced, physical servers from the outside world, available on our core network, shared account names (and passwords!) across all devices, every account a local admin and their own domain admin account so they can make changes as required… Needless to say our security guy almost broke a rib from laughing so hard!
What they got was very different. I ended up doing fairly basic troubleshooting to get the shitty software working without it needing admin rights to run - XP era software running on Win7 clients so it was meeting basic folder and reg key write permissions - everything else was locked down as tight as we could. They didn’t even get admin on the servers.
A few years later I heard they’d reported that our site was their most reliable. Probably due to the fact that they (or the techs) couldn’t fuck with anything without our permission. Apparently their original configuration is normal in the industry… Shudder
This has been talked about for years and about how exposed SCADA is. I just assumed that it was fixed/isolated years ago. The fix appears to have been to stop talking about it.
That’s why NIS2 in Europe demands that OT is on a separate network from IT. My old workplace (well, the head of FO Technology and his gang anyway) was up in arms about it because it “was a waste of time” (and completely ruin the ideal architecture he had in his mind).
why is any critical control equipment internet accessible.
I understand some machines might need internet to send reports and make document filings and shit. But it should not be the critical control equipment that makes the system function. And there should be a massive physical gap between the internet machines and the critical machines.
Jesus christ, even by the abysmally low bar I have for humanity, it still finds ways to dig itself a path under it.
The entire tech sector screamed about the vulnerability of IoT devices, plenty of which are now participants in zombie botnets.
Does anyone remember in 2015 when YESCO billboards by the thousands were hacked to show Goatse instead of an AT&T ad? Pepperidge Farm remembers. IoTs are still vulnerable, including big municipal ones and, yes, Flock Safety cameras.
And CISA
was severely brokenhad its staff severely reduced by DOGE.In unrelated news the CDC, also a victim of DOGE, is entirely unprepared for the next epidemic.
