Is OWASP Benchmark Any Good?

A short analysis of the OWASP Benchmark project towards comparing the SAST tools

OWASP Benchmark

Straight from the source code repository, here’s the goal of the project.

A single test case represented with an Java source file and a complementing XML file.
  • Path Traversal
  • Insecure Hash Algorithm
  • Trust Boundary Violation, CWE 501
  • Insecure Encryption Algorithm
  • Command Injection
  • SQL Injection
  • Insecure Random Number Generation
  • LDAP Injection
  • Cross Site Scripting
  • Missing Cookie Secure Attribute, CWE 614
  • XPath Injection

A Simple Analysis

If we leave the basic weakness analysis engine processes to locate a vulnerability aside, the main goal behind the project is measuring the accuracy.

Dead code

In the code block below, assume the param comes from a dangerous source. It turns out that the else statement contains a dead code. With that line never runs, there shouldn’t be any vulnerability as the tainted data never arrives at DangerousMethod.

// Simple if statement that assigns constant to bar on true condition
int num = 86;
if ((7 * 42) - num > 200) bar = "This_should_always_happen";
else bar = param;
DangerousMethod(bar); // a synthesized sink

Complex Data Structures

A similar pattern that we can find to mess with the SAST tools is using the data structures such as ArrayLists. In the code block below, assume that the param comes from a dangerous source.

String bar = "alsosafe";
if (param != null) {
java.util.List<String> valuesList = new java.util.ArrayList<String>();

valuesList.remove(0); // remove the 1st safe value

bar = valuesList.get(1); // get the last 'safe' value
DangerousMethod(bar); // a synthesized sink

Simple Inter-Procedural Calls

Here’s another example. ProcessBuilder is a dangerous sink that the data arrives to it should be traced. On the code block below, the args contains a param which comes from an external class method call that returns a hardcoded value.

SeparateClassRequest scr = new SeparateClassRequest(request);
String param = scr.getTheValue("BenchmarkTest00051");
String[] args = {a1, a2, "echo " + param};

ProcessBuilder pb = new ProcessBuilder(args);
public String getTheValue(String p) {
return "bar";

Configurational Values

The source code reads a property value with a strong cryptographic algorithm as a default value when the key is missing in the properties file.

String algorithm = benchmarkprops.getProperty("cryptoAlg1", "AES/ECB/PKCS5Padding");
# This file contains various property values used by various test cases in the OWASP Benchmark

A Short Critique

Although it contains a web based interface, the application itself is not designed to include code flows or design complexity that even a normal web application usually has, such as services, repositories, etc.

A classified benchmark test cases of FlowBlot.NET against SAST tools.
Cookie cookie = new javax.servlet.http.Cookie("SomeCookie", str);



Comparing SAST tools is a hard task. There are good deliberately vulnerable source code projects out there, such as OWASP Benchmark, OWASP Web.Goat, FlowBlot.NET, however, they are not enough to decide every criteria under the analysis.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store


CodeThreat is a static application security testing (SAST) solution. Visit