I sat down witih Jeanne Friedman last week and recorded a short podcast preview for my upcoming presentation at RSA 2012 “Remediation Statistics: What Does Fixing Application Vulnerabilities Cost?” You can see her post online here and you can download the 8 minute podcast here. The actual session is ASEC-302 and will be Thursday, March 1st, 2012 at 9:30am in Room 302. Hope to see folks there!
The good news is that if you got your stakeholders aligned in an Inception phase and if you know what you are going to do and how you are going to do it from a Planning phase, actually fixing the vulnerabilities is usually pretty easy: you just do what you said you were going to do. That said, there are some potential pittfalls.
For starters - you can’t always just start fixing vulnerabilities. You first have to have a working development environment where you can make changes and test them. For applications being actively developed with good configuration management this should be pretty easy. For the 10 year old end-of-lifed ASP Classic application that no one has touched in five years it can be more challenging. This can be a huge time waster and wading through a couple of messy projects has renewed my belief in the value of configuration management and environment documentation.
Also once you fix vulnerabilities you have to both make sure your fixes worked and make sure you didn’t break the application. The way of the world is that the last person to touch the code is assumed to be responsible for anything that goes wrong, so you want to be sure to be able to demonstrate that security fixes aren’t responsible for any downtime. If your application has a suite of automated unit and functional tests they will really be paying dividends at this point. If you agreed that a “fix” would be considered successful if it no longer turned up in a scan then you’ll need to re-run a scan. If your fix stanards were more involved then you will need to runt through and confirm that the vulnerabilities have been addressed. Again – automation can be a big help for certain types of vulnerabilities.
Finally you can’t declare victory until your updated – and tested – code is pushed into production. Using an existing update window is preferable or you may have to do an out-of-cycle release. The important thing is that the code goes live. If vulnerabilities are supposed to be fixed and the next round of testing from your external auditors indicates that they haven’t been you should expect to get a nasty phone call. So far every time I have seen this situation in a well-run remediation project it has been because code updates or mandated configuration changes had not actually been pushed into the production environment.
Here is a short video where we talk a bit about the Execution phase for software security remediation projects:
Here is a presentation outlining some of our research on the actual costs of remediating different classes of vulnerabilities as well as where time gets spent during the execution of remediation projects:
You can read the full HOWTO Guide for Software Security Vulnerability Remediation here:
Normally information security creative writing is reserved for marketing brochures talking about Advanced Persistent Threats, but I thought I would share a little something I’ve been working on. This is an excerpt from an infosec romance novel I have in progress titled “The Remediation Diaries”
After we sent out our assessment report we hadn’t heard a peep from the application team until Alice the development manager came to me and complained one of her developers, Bob, was behind schedul and had been “screwing around” all week doing some sort of “security crap.” Wandering the halls until I found him, a premonition hit me that the conversation was going to take a frightening turn. I felt a hollowness in the pit of my stomach as I looked into the puppy dog eyes of a developer who wanted nothing more than to prove his competence and win the aproval of the security team.
“The report was pretty long and a little confusing, but in there it said the XSS vulns were because we weren’t escaping text before we sent it to the browser.” I breathed a small sigh of relief. At least Bob read the report and seemed to understand the important parts. Things were looking up. “So I wrote an ‘escape’ function that turns all the left and right brackets into the escape sequences.” My heart started racing – did Bob really think that was a sufficient answer? “Then I went around and pasted it in the code to fix a bunch of the pages – I think I got them all.” My throat closed up. This was not going well. “I didn’t want to bother you so I went and did that for all the injection vulnerabilities, too. Can we run another test?” My heart stopped beating and I was convinced I was in full-on cardiac arrest. I smelled toast burning – was I having a stroke?
I struggled to keep my cheerful poker face. What was the kindest way to handle this? How could I encourage Bob’s enthusiasm without crushing his soul? My brain was shouting: That’s a crappy way to make an HTML escape function! And it doesn’t work for HTML attributes! And why would anyone think that HTML escaping was the same as SQL escaping? And why wouldn’t you just use parameterized queries to fix SQL injection properly? Why didn’t they know how to fix this stuff right the first time? If only I had helped the developers plan a little bit before they ran off on their own…
[Side note: What do you think? Do I have a future as a novelist? Or should I stick with my day job?]
On a more serious note, this story typifies the types of unproductive efforts we see with poorly-planned remediation efforts. Just as the Inception phase is needed to make sure that the stakeholders are on the same page, efforts needs to be planned so that there is unambiguous agreement on issues such as:
Which specific vulnerabilities are going to be fixed and when?
Specifically how should the vulnerabilities be fixed?
What sort of follow-on testing is going to be performed and what is the standard by which vulnerabilities will be considered to be successfully addressed?
Developers are smart people. The vast majority of them also have the best of intentions – they want to write good code and they want to write secure code and, if given the choice and the time, they would prefer to do things the “right” way. But if you rile them up and don’t provide guidance – especially when it comes to unfamiliar territory like security fixes – you are likely to be disappointed with the results. And if you are disappointed then you can be sure that development managers and other stakeholders are going to be both disappointed and angry. In addition, they aren’t going to be interested in funding your next remediation effort. Up-front planning and explicit guidance means the difference between successful vulnerability remediation and ineffective time-wasting so spend the time with developers to get this right the first time.
Imagine this scenario: Your development team builds an application and puts it into production. Down the road, a customer asks you to do a security assessment. You run a scanner against the application and perhaps even do some manual penetration testing. The result is you end up with a long list of vulnerabilities and the customer wants them fixed. So the security team meets with the development team and the exchange goes something like this:
Thanks, security team! Very helpful! Come back any time…
If you’re going to spend the time and resources diverting development teams from building new features to fix security vulnerabilities all parties involved owe it to themselves to make sure the effort is successful. Based on our experience doing software security remediation projects the ones that are approached in a thoughtful and structured manner tend to do far better than ones based on a mandate of “FIX IT!” We’ve developed a HOW-TO guide for software security remediation projects outlining just such a structure, and these projects start with an Inception phase.
The Inception phase is used to get all the stakeholders together and on the same page. Software security remediation projects are typically software development projects, not security testing projects and they need to be estimated and project managed as such. They also force people from different parts of an organization with different goals to work together. Before moving forward, teams need to agree on things like:
Approximate budget and where the budget is coming from
Desired (but realistic) timeline
Specific compliance or audit issues that must be addressed
Initial project success criteria (“fix all the CRITICAL and HIGH vulnerabilities” or “fix all public-facing cross-site scripting”)
Given this shared understanding the involved parties can start to work on planning the actual remediation effort, but in the absence of a consensus the remediation project likely does not have a clear mandate and this is a recipe for project failure.
Here is a short video talking a bit about the Inception phase for software security remediation projects. Hopefully you find it to be a bit more constructive than the previous one.
Also, you can read the full HOWTO Guide for Software Security Vulnerability Remediation here:
We are starting to see some data from the industry about how long it takes most organizations to fix vulnerabilities they’ve identified in their software. Two useful sources are:
These provide information about the prevalence of different types of vulnerabilities as well as how how long vulnerabilities tend to stay in software and is a reflection of the calendar time that these vulnerabilities exist. At Denim Group we have also released some of our data on how long it takes to fix different types of vulnerabilities and our data reflects the level of effort required to make fixes. The data we released can be found online here:
The combination of these types of data sources should be helpful for organizations trying to craft a strategy for addressing vulnerabilities in their software. The data about vulnerability lifespans can help you to benchmark yourself against industry peers and set goals for what sort of exposure window you are willing to accept in your organization (although I would argue that software security vulnerability lifespans are far too long right now). The data about the level of effort required for fixes can help you to plan the resources required for remediation projects. Availability of data sets like this allows security analysts to have “grown up” conversations with management. Think along the lines of “to keep pace with peers in our industry we should be doing these things…” versus “cross-site scripting is scary…” More “grown up” conversations should lead to better-allocated budgets and ultimately to better-managed risk.
The slides from my SOURCE Boston 2011 presentation “The Real Cost of Software Remediation” are now online:
We’ve been doing remediation work for a number of years so I’m happy to start talking about more of the things we’ve seen. I’ve maintained for a while that finding vulnerabilities is usually pretty easy and that fixing vulnerabilities is where organizations need to focus more effort. Hopefully some of the lessons we have learned will help other organizations start to plan and execute remediation projects of their own. I think the remediation project framework we put together, when combined with some of the remediation statistics we are releasing should help. We have these and a number of other resources in our online Remediation Resource Center.
The security industry is beginning to release data that focuses on the prevalence of different types of vulnerabilities and incidents. However interesting, such data falls short of providing crucial information to aid organizations with their software remediation efforts. This presentation provides statistical data from 15 different web application remediation projects in order to provide real insight into the costs of remediating application-level vulnerabilities. The data addresses pressing questions, including how much time is spent on different phases of remediation projects (inception, planning and execution), and how much time is required to remediate different classes of vulnerabilities. Based on this data, analysis is also provided so organizations can make decisions about which vulnerabilities should be fixed and which should be left, how to schedule vulnerability remediation into software project schedules, and activities organizations should undertake in order to prevent the most costly vulnerabilities from occurring in the first place.
