By William T.
Update: Ory Segal (@orysegal) from IBM Rational reached out with a simpler method to handle this natively in AppScan.
During a recent security assessment, I encountered several issues while performing Denim Group’s standard baseline test using the automated scanning tool IBM Rational AppScan. I have compiled the problems I encountered (as well as the solutions) in the hopes that some or all of it will be helpful in future application security assessments.
For those unfamiliar with the tool, the basic functionality begins with configuring AppScan to log into a given web application. Once AppScan knows how to log in, it performs an exploratory phase in order to discover the pages, forms, and features that comprise the web application[1]. Alongside this, AppScan looks for certain URL or form patterns and generates a set of tests to run on each page. Then the attack phase begins, in which the generated attacks are executed against the application.
The problems I encountered began when I was configuring the login process and opened AppScan’s internal browser.
Handling Browser Support
The first problem was that the target web application was rejecting login requests coming from AppScan’s internal browser[2]. The message stated: “Your browser is not supported! For security reasons, your browser version is not supported and will not be allowed access into the Online Banking system…”
Clearly, the application requires an up-to-date version of the browser being used. But how was I to update the AppScan internal browser? The answer is that I didn’t have to update any software, but I did need to trick the web application into thinking that the browser being used was up-to-date. This particular web application uses the “User-Agent” HTTP request header in order to determine browser version, and thus I needed to modify the “User-Agent” header value that AppScan sends to the web application. This is done by changing the “Communication: User-Agent header value” AppScan configuration setting (Found in Scan > Scan Configuration > Advanced Configuration):
Figure 1: Changing the AppScan internal browser’s User-Agent header value.
The value above was modified from the default value of “Mozilla/4.0 (compatible; MSIE 6.0; Win32)” to “Mozilla/5.0 (Windows NT 6.1; WOW64; rv:10.0.2)”.To get a valid header value, I used an updated browser connected to a proxy tool, such as OWASP ZAP or Burp, to navigate to the application. I then copied the “User-Agent” header value from the request (stored or intercepted by the proxy tool) into the above AppScan configuration setting. This satisfied the web application’s browser version requirements, and I proceeded with the login configuration.
Handling Dynamic Authentication
AppScan allows for several types of login management: Recorded login, Prompted login, and Automatic login. Prompted (manual) login is typically discouraged due to the large number of times AppScan will need to perform a login during testing. Automatic login works for most instances, but develops problems when an application requires second-factor authentication in addition to a standard username and password login. The second-factor authentication implemented by the web application I was testing randomly picked one of four security questions to ask during the login process. Thus, Recorded login was my best option and I proceeded to record the login sequence:
Figure 2: The recorded login sequence in the Login Management scan configuration settings.
In the above screenshot are the requests recorded for the login sequence. The first GET request retrieves the login page and sets the public session identifier. The next request, a POST, submits the login credentials. The POST request following submits the answer to the security question, whose response contains a redirect to the web root, where the now present Logout link can be used for in-session detection.
My next issue arose when AppScan was attempting to automatically log in during the Explore or Test phases with this sequence but failing. I realized the problem was that my recorded sequence contained an answer for just one of the four possible security questions. When a different question was returned in the response to the first POST request, an invalid answer would be submitted with the second POST request. My solution to this problem was to go into the web application and set the answers to all the security questions associated with the account to the same value. For this scan, I set all answers to the value “asdf”.
Upon running the attack phase, however, I realized that the recorded login still had issues which prevented AppScan from being able to maintain or create a session with the web application. Looking at the scan log, I saw AppScan attempting to perform the login sequence but not being able to find the in-session identifier. Thus, I concluded that something else must be wrong in the recorded login sequence. To confirm this I decided to take a closer look at the HTTP requests being sent during login. Using a browser whose traffic was fed through a proxy tool, I inspected several instances of the second-factor authentication page:
|
POST /login/authenticate HTTP/1.1
Cookie: SESSIONID=bFjbluc7gDgxz4Nfxvne4GAB373nn2nEdu5MdVa;
Content-Length: 53
Accept: image/jpeg, image/gif, image/pjpeg, application/x-ms-application, application/xaml+xml, application/x-ms-xbap, */*
Referer: https://xxx.xxx.xxx.xxx/login/authenticate
Accept-Language: en-US
Content-Type: application/x-www-form-urlencoded
Host: xxx.xxx.xxx.xxx
Connection: Keep-Alive
Cache-Control: no-cache
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:10.0.2)
X-ForwardTo=&questions=62636&answer_62636=asdf
|
Figure 3: The first second-factor authentication request sent.
|
POST /login/authenticate HTTP/1.1
Cookie: SESSIONID=ILHZgOwgbEJM53P5DQZj6R6PVAMbSX8P7VypRH6;
Content-Length: 53
Accept: image/jpeg, image/gif, image/pjpeg, application/x-ms-application, application/xaml+xml, application/x-ms-xbap, */*
Referer: https://xxx.xxx.xxx.xxx/login/authenticate
Accept-Language: en-US
Content-Type: application/x-www-form-urlencoded
Host: xxx.xxx.xxx.xxx
Connection: Keep-Alive
Cache-Control: no-cache
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:10.0.2)
X-ForwardTo=&questions=3468&answer_3468=asdf
|
Figure 4: The second second-factor authentication request sent.
I discovered that the name of the challenge question answer parameter (‘answer_xxxxx’) is dynamic – it was different every time the security challenge was presented. This meant that I needed a way for the login sequence to handle a dynamically named parameter.
After many hours, I decided that AppScan could not, by itself, deal with this issue. I needed to use the recorded login sequence, but also needed some way to recognize the specific login request which contained the security challenge answer and modify it so the answer parameter was named correctly. I noticed that in the request which submitted the challenge question answer, another parameter was also submitted (the ‘questions’ parameter in the above request). This parameter contained the dynamic reference id value of the challenge question presented during login. The dynamic part of the answer parameter’s name was this reference id value. Thus, my solution required that I find a way to detect the second-factor authorization request, extract the value of the ‘questions’ parameter to get the challenge question reference id, and modify the name of the challenge question’s answer parameter to reflect the reference id value.
To do this, I looked to the proxy tools mentioned earlier in this post. I decided to use Burp Proxy for performance reasons (although OWASP ZAP provides analogous functionality). To configure Burp Proxy, I first needed AppScan to send its traffic to the proxy tool, so I modified AppScan’s connection settings (Scan > Scan Configuration > Connection).
Figure 5: AppScan Communication and Proxy scan configuration settings.
Under the “Communication and Proxy” section, I selected the bullet for “Use custom proxy settings” and entered the address and port that the proxy tool is set up to use. In Burp, I found this information by clicking the Proxy tab, then the Options sub-tab and looking at the “proxy listeners” section:
Figure 6: Burp proxy options.
Once this was set up, I needed to configure Burp to perform the parameter modification. Under the Options sub-tab of the Proxy tab, I scrolled down to the “match and replace” section. This section allows Burp to look at a specific part of all requests and responses, be it the request headers, request body, response headers, or response body. Since I needed to modify a request parameter, the closest option was for Burp to look at the request body (selected via dropdown list). The remaining two fields in this section are for the regular expression to perform matching on, and the regular expression for how the selected part of the request should be modified if a valid match is encountered. After some trial and error, I found that Burp uses a rather Perl-esque regular expression syntax, which was helpful for matching and extracting the value out of the request body. I say Perl-esque because I was able to use parentheses to group specific elements in the match pattern, and then in the replace pattern extract the matched elements via local special variables (of the form $_, or $1, $2, $3, etc):
Figure 7: Burp match and replace.
The regular expression I used for matching the security challenge login request was:
([0-9]+)&answer_[0-9]+=[A-Za-z0-9]*
This regular expression should only match on the second-factor security challenge request because it is the only request using these specific parameters (see Figure 4 or 5 above). For help understanding regular expressions, please refer to: http://www.regular-expressions.info/reference.html.
The regular expression used to modify the request body was:
$1&answer_$1=asdf
When a successful match is made on an incoming request, ‘$1’ refers to the value inside the first set of parentheses in the match expression. This is how I extracted the dynamic challenge question reference id and used it to modify the request. The answer parameter value is set to ‘asdf’ because that is the string I set as every security question’s answer earlier.
That’s it! Whenever AppScan needs to perform a login during testing, it will execute the recorded login sequence. This login sequence is fed through the proxy tool Burp, which modifies the second-factor authentication request to account for the dynamic nature of the challenge question answer field. It is worth noting that I attempted to use OWASP ZAP to perform the needed match and replace function, but it could not process the traffic with the same efficiency that Burp could manage.
[1] It is worth noting that a manual exploration of the application is usually needed to ensure complete coverage of pages in the application. To do this, click the ‘Manual Explore’ button in AppScan and a browser will pop up in which it is possible to navigate to undiscovered pages. I usually perform a submit on any forms or features I can find to ensure AppScan picks up the form parameters to generate tests for.
[2] When I was viewing the scan log and the web application was blocking the internal browser, I saw a message similar to “Cannot connect to host: xxx.xxx.xxx.xxx”.
Contact us for help with your security assessments.
Thanks to Denim Group team member William for writing this post! -Dan
