Over the years, I’ve worked on investigating and cleaning up many breaches, and in nearly every single instance, the IT team that designed and managed the environment had no concept that their system could be exploited in the manner it was. Another commonality is that nearly all of those breaches resulted from a chain of weaknesses, some of which were consciously “accepted”. I argue that it is difficult to design a system resilient to attack if one does not know the tactics adversaries use, and it is equally difficult to assess risks without understanding how controls help block adversarial techniques.
For National Cyber Security Awareness Month, my hope is that people responsible for designing and assessing IT environment take time to learn about adversarial tools and techniques to design more robust environments and processes. This is, unfortunately, not a one-time event, though: techniques change over time, and we need to keep up with the latest trends.
The downside, I suppose, to this advice is that red team can be quite addictive and we’ll lose many competent IT people to the pen test puppy mill.
Zero trust networks are quickly becoming all the rage in the IT world. Building proper defenses into each endpoint and relying on strong authentication schemes seems intuitively right. I’ve had several recent discussions with smart people about how dull the old world of network-based firewalls which grant implicit permissions based on the network location of a particular device (i.e., inside vs. outside the firewall), which is just another way of saying that we should move toward zero trust.
But all this presumes that the “endpoints” can be secured. Many pieces of IT infrastructure, including switches, servers, firewalls, and many other devices contain administrative interfaces that have security that is on par with, or slightly worse than, the average home router. In the past few years, we’ve seen many, many problems with the lights-out management interfaces for various servers; we’ve seen a (so far) non-stop parade of authentication bypass/hardcoded passwords in Cisco devices; and we’ve seen many other devices using various badly configured or exploitable services running on these interfaces, like dropbear, libssh, and others.
These interfaces should NOT be exposed to untrusted networks. That, sadly, means that we need to continue on with architecting, at least to some extent, well thought out networks.
The concept of a “management network” is not new. I was first introduced to the concept over 20 years ago, and I suspect the idea was already old by that point. Remember that a management network, by definition, is a concentration of sensitive interfaces and user sessions that have administrative privileges. A lot has been written about the design of management networks by people much smarter than I am, but I’ll give some ideas/observations here:
- Ensure that only authorized people and devices are able to connect to the management network
- Monitor activities on the management network for indications of unauthorized devices or users
- Keep the number of devices on the management network as small as possible – A one to one relationship would be optimal, but often impractical
An often-overlooked aspect of vulnerability management are software components that exist on a system, such as PHP, Apache Struts, and Ghostscript. These components are often dependencies to other applications. If the packages are installed through a normal package manager, like yum or apt, updates should be applied during periodic updates. There are three things to be aware of, though:
- If a package goes end of life, like what is about to happen PHP5, updates may simply and silently stop being applied, leaving a potentially vulnerable piece of software running on a system.
- If a component is custom compiled, a package manager will not apply updates. Note: this is an argument in favor of using binaries provided by main stream repositories
- Vulnerability scans may not be able to detect vulnerabilities in such components, particularly if using unauthenticated scans.
As we move toward infrastructure-as-code, maintaining these inventories should be less taxing, since the configuration definition for systems should explicitly contain the packages installed. If not, then you’re doing IAC wrong.
Create a list of all these components that exist in your environment, and determine what process is used to identify a vulnerability in them and ensure each is updated when necessary. Many may be updated in the normal course of running operating system updates, while others may require manual tracking to identify when to download, compile, and install updated source code.
It’s hard to manage what you don’t know you have.
Sometimes, despite our best efforts, ransomware will successfully invade our systems. The need for good back ups should be well known by now, but here are a few recommendations:
- Several organizations impacted by the likes of SAMSAM have opted to pay the ransom to recover their data, despite having good backups. This is apparently happening because the time and cost to restore all the impacted systems and data from backup is substantially higher than the cost of the ransom. I previously wrote about why it’s a bad idea to simply clean an infected or compromised system and paying the ransom to get back to operations faster is basically doing just that. I argue that an organization that is tempted to pay the ransom in such a case probably did not properly assess the RTO and RPO that were needed when designing and implementing its recovery program.
- Be aware that some “backup” schemes use real time or near real time replication between systems in remote sites. Ransomware-encrypted files will be replicated to the backup location nearly instantaneously. Remember to make backups or take snapshots if you’re using such a recovery scheme.
- This shouldn’t have to be said, but you (probably) don’t have a backup if you haven’t tested your backup. Test them.
- Over time, we will likely see a shift to injecting techniques that render backups useless to help force ransom payment.
- One of the insidious aspects of some ransomware like SAMSAM is that is can effectively take out all systems on a network. Consider your ability to initiate recovery if all of your administrators have locked up workstations and your (ugh) Sharepoint repository of recovery plans are all encrypted.
- I previously wrote a longer post on how to prevent ransomware infections.
Remote Desktop Protocol, RDP, is becoming a common entry point for bad actors, including many POS terminal breaches, and the main delivery method for enterprise-grade ransomware, like SAMSAM. An underground economy has developed around finding and then selling credentials to access various organizations through RDP.
Though it is a legitimate tool for administering Windows systems remotely, RDP should never be exposed directly to the internet. Firewalls should be configured to disallow RDP access from Internet sources, and systems that run RDP must not permit accounts with default or weak passwords.
Use a service like Shodan to scan your organization’s address ranges to identify RDP services exposed to the Internet. Workstations should be configured via GPO to disable RDP, enabling it only by exception.
We alternately hear “people are the first line of defense” or “people are the last line of defense” in cyber security. I haven’t figured out which one is true. Regardless, we need to understand that there are limits to the effectiveness of awareness training and that our first line of defense or our last line of defense (whichever is correct) is quite fallible.
It comes as no surprise to anyone that training humans is not like defining a rule base in a firewall. We tell a firewall what network traffic to permit, and what to block based on attributes of the traffic. Similarly, we train our employees on how to identify and resist various types of attacks. Firewalls will dutifully and predictably follow the rules it was programmed with. Humans, however, are a different story.
Let’s imagine for a moment that we have developed a perfect security awareness program. It clearly communicates dos and don’ts, how to spot attacks, how to report problems, and so on, in a way that is memorable and engaging. I propose that the outcome will be significantly less than perfect, because of the following factors:
- People act irrationally under stress from things such health problems, family problems, medication, and lack of sleep
- Any given person will act upon the same set of conditions differently based on the time of day, proximity to lunch, day of the week, and many other factors that affect his or her frame of mind at the time
- People in a business setting generally have incentives that may, at least some of the time, run contrary to the recommendations of awareness training, such as project deadlines, management expectations, and so on.
This should tell us that awareness training is, at best, a coarse screen that will catch some problems, but allow many others to pass unimpeded. As such, we should focus on providing awareness education that provides the biggest value, in terms of outcomes, and then focus our remaining effort on enhancing process and technical controls that are designed to provide more predictable, and repeatable security outcomes, similar to the operation of a firewall.
On a related note, I personally think it’s irresponsible to pin the safety of an organization’s systems and data on an employee recognizing that a potentially sophisticated attack. For this reason, I think it is incumbent on us to develop and implement systems that are resilient to such attacks, and allows employees to focus on their job duties.
Web applications are among the most common entry points in data breaches and network intrusions. In the best of conditions, defending web applications can be challenging, but I’ve observed that the are often orphaned as priorities change, staff turns over, and in the wake of organizational changes.
This is not a problem in all organizations, particularly in smaller firms that have a monolithic IT department that manages all technology, though I have seen the problem in small companies, too.
Similar to tracking servers and workstations, organizations should have a system in place to track the ownership of applications, and periodically revalidate ownership and force reassignment when the designated owner leaves or transfers.
The application owner should be responsible for understanding all of the components that compose the application and ensure that each component is properly vulnerability scanned and patched, or shut down and deleted if no longer needed.
Organizations should be on the lookout for applications not being properly maintained through vulnerability scans and other tools that may be available. Unmaintained applications should be treated as an incident to investigate.
While I previously wrote that the cloud is not a magical place, I think it’s important to point out that there is a sickness in the IT world. It’s insidious and seems to hang around Kanban boards like West Nile laden mosquitos hang around a pond. Of course, I’m talking about exposed S3 buckets and NoSQL/MongoDB databases.
The fundamental issue appears to be that the those who configure these environments do not know what they don’t know. We need to take down this sickness. Unfortunately, there is no blinky box that can fix this problem*. Rather, employee awareness and support are needed. For example, include a segment in your organization’s mandatory security training to engage the IT or IT security team for guidance on the proper use of such services. Yes, this may encourage some people who may not otherwise have thought to copy the contact database into an S3 bucket, and may drive up work on the IT team, but it’s better than the alternative. If you offer help rather that harsh criticism, you may just get people to ask for that help.
I suppose it should go without saying that your organization’s IT and security teams should themselves know how to properly use these services as a start.
*depends on your willingness to believe CASB vendor marketing pitches. YMMV.
Recently disclosed details about the Equifax data breach indicate that, in addition to the Apache Struts vulnerability that initially led to the breach, some security tools had stopped working for an extended period of time, and only after those tools were brought back online was the breach detected.
There are many potential reasons for security technology to fail, but quite often we don’t recognize that they have failed, because we are only monitoring for alerts, or simply assuming that the security “thing” is quietly doing its job. When those technologies do fail and we’re not aware that they’ve failed, a key element of our security program is not working, and if we’re not actively monitoring for those failures, we don’t know that we’re blind or unprotected.
For this reason, I recommend developing a set of ongoing test cases that are implemented along with new security technology to help ensure that the technology is operating as expected and raise an alert when it fails in some way. For example, a SIEM should be configured to trigger an alert if a log source does not provide a log within a certain timeframe, which may indicate that the logging service died on the host, or some network issue is preventing logs from being sent to the SIEM. Another example might be a periodic injection of a particular type of network “attack” (in a relatively safe manner, of course) designed to trigger an IPS block and alert, in a manner that tests both the blocking (did the “attack” make it to the destination?) and the alerting (did the “attack” result in a generated alert?).
These test cases should be developed to measure the ongoing effectiveness of all the key functionality that the security technology provides.
Here we are, after decades of security enhancements, blinky boxes, and hundreds of hours of security awareness training and companies still get compromised through email. My movement to drive everyone back to using pine, mutt, and elm for email has failed miserably, so here are my next best recommendations:
- Strongly consider not doing email, or at least email filtering, on your own. I don’t advocate for particular technology vendors, but most of the big names, like Proof Point and others, have pretty good mail filtering capabilities that you’re just not going to match. Save your efforts for security programs that are unique to your organization. Email is a commodity service these days.
- Prepend the tag “[external]” to the subject line of incoming email from the Internet to serve as a visual cue for employees. It’s not foolproof, particularly in the context of business email compromises where malicious emails can originate locally, but it can help and gives some fodder for awareness training.
- If you do use a service, such as Proof Point, that rewrites URLs in emails and/or add the “[external]” tag, be wary of the way in which you run phishing simulation exercises. If the simulation emails appear to come from outside the organization, but do not have the “[external]” tag, or do not have URLs rewritten in the way that all other external emails do, employees will quickly learn to identify the simulation emails based on those characteristics, rather than the characteristics you want them to observe.
- Tailor awareness training by role. If someone has a job that requires them to open attachments from strangers, such as is the case with recruiters, don’t give them training that tells them not to open such attachments. At best, it’s confusing. Rather, provide guidance on the proper means for various roles in the organization to do their jobs in a safe manner.
- Be aware that every hacker and her dog are trying to get into your organization’s email and act accordingly. Require two factor authentication for mail access, particularly for any cloud-based mail that is accessible straight from the Internet.