We are happy to announce that version 1.08 of Securibench Micro has been released. Unlike Securibench, which contains large, real-life applications, Securibench Micro is a series of small test cases designed to excercise different parts of a static security analyzer. Each test case in Securibench Micro comes with an answer, which simplifies the comparison process.
All test cases included in this release can be installed on a standard application server such as Tomcat. So, in addition to using test cases contained in Securibench Micro to put a static analyser thorough its paces, Securibench Micro may be used to compare the effectiveness of runtime techniques such as penetration testing tools.
These test cases suffer from a variety of vulnerabilities including
and potentially many others.
Below are some common questions:
The goal of Securibench Micro is to test the capabilities of security testing tools. Often the information about a particular analysis is hard to discern behind the marketing jargon and non-standard terminology. Unlike Securibench, which collects "natural" open-source benchmark applications, Securibench Micro addresses this problem by creating a large standard set of artificial test cases. Securibench Micro and Securibench were created as part of the Griffin Security Project at Stanford University.
Securibench Micro is a suite of micro benchmarks written in Java and using J2EE libraries. Each test cases represents a small self-contained servlet that can be tested in isolation. Most benchmark programs in Securibench Micro are designed to have security vulnerabilities embedded in them.
The overaching goal was to design an in-depth suite of benchmarks that would take the capabilities of a particular static analyzer to the limit. At the same time, we wanted to have a suite of benchmarks, all of which are executable so that they are amendable to manual penetration testing or some form of dynamic analysis.
| Type | # of entries |
| arrays | 10 |
| basic | 42 |
| collections | 15 |
| factories | 3 |
| inter | 14 |
| pred | 9 |
| reflection | 4 |
| sanitizers | 6 |
| session | 3 |
| strong updates | 5 |
| aliasing | 6 |
| data structures | 6 |
| Total | 123 |
Securibench Micro is designed as a number of packages designated to test a particular feature set of source-level security vulnerability scanners. Currently there are packages that test analysis interprocedural features, handling of collections, handling of predicates, handling of reflection, and more.
Securibench Micro comes with an Ant installation script build.xml. Before running ant, please update file build.properties to refer to your server installation directory. Before running ant, make sure you have xdoclet available on your system. (If you are missing xdoclet, unzip it to a directory of your choice and change xdoclet variable in build.properties to refer to it.)
It is not strictly necessary to run ant install if you just intend to manually study or run static analysis tools on the test cases located in src/securibench/micro. Installation is only necessary if you indent to have running versions of the benchmarks running on the server. Since most of these micro-benchmarks have fairly obvious security holes, it's not recommended that you install Securibench Micro on a machine that is externally accessible.
Version 1.08 of Securibench Micro ships with 96 test cases in 10 categories. Show in the table are more detailed statistics about the number of tests in each category. This information can be generated by running script stat.pl in src/securibench.
| Name | Description | Bug count |
|---|---|---|
| arrays | ||
| Arrays1.java | a simple array test | 1 |
| Arrays2.java | a more complex array test | 1 |
| Arrays3.java | a more complex array test | 1 |
| Arrays4.java | a test of array indeces | 1 |
| Arrays5.java | strong updates to array elements | 0 |
| Arrays6.java | strong updates to array elements | 1 |
| Arrays7.java | test of array initializers | 1 |
| Arrays8.java | test of array initializers | 1 |
| Arrays9.java | multidimentional arrays | 1 |
| Arrays10.java | comple multidimentional array test | 1 |
| basic | ||
| Basic1.java | very simple XSS | 1 |
| Basic2.java | XSS combined with a simple conditional | 1 |
| Basic3.java | simple derived string test | 1 |
| Basic4.java | test path sensitivity just a bit | 1 |
| Basic5.java | moderately complex test of derived strings | 3 |
| Basic6.java | complex test of derived strings | 1 |
| Basic7.java | complex test of derived strings involving a string buffer... | 1 |
| Basic8.java | test of complex conditionals | 1 |
| Basic9.java | chains of value assignments | 1 |
| Basic10.java | chains of value assignments | 1 |
| Basic11.java | a simple false positive because of two calls to String.to... | 2 |
| Basic12.java | a simple conditional; both branches should be taken | 2 |
| Basic13.java | use getInitParameter instead | 1 |
| Basic14.java | use the servlet context and casts | 1 |
| Basic15.java | test casts more exhaustively | 1 |
| Basic16.java | simple heap-allocated data strucure | 1 |
| Basic17.java | simple heap-allocated data strucure | 1 |
| Basic18.java | protect agains simple loop unrolling | 1 |
| Basic19.java | simple SQL injection with prepared statements | 1 |
| Basic20.java | simple SQL injection | 1 |
| Basic21.java | SQL injection with less commonly used methods | 4 |
| Basic22.java | basic path traversal | 1 |
| Basic23.java | path traversal | 3 |
| Basic24.java | unsafe redirect | 1 |
| Basic25.java | test getParameterValues | 1 |
| Basic26.java | test getParameterMap | 1 |
| Basic27.java | test getParameterMap | 1 |
| Basic28.java | complicated control flow | 2 |
| Basic29.java | recursive data structures | 2 |
| Basic30.java | field sensitivity | 1 |
| Basic31.java | values obtained from cookies | 2 |
| Basic32.java | values obtained from headers | 1 |
| Basic33.java | values obtained from headers | 1 |
| Basic34.java | values obtained from headers | 2 |
| Basic35.java | values obtained from HttpServletRequest | 6 |
| Basic36.java | values obtained from HttpServletRequest input stream | 1 |
| Basic37.java | StringTokenizer test | 1 |
| Basic38.java | StringTokenizer test with a false positive | 1 |
| Basic39.java | StringTokenizer test | 1 |
| Basic40.java | MultipartRequest test | 1 |
| Basic41.java | use getInitParameter instead | 1 |
| Basic42.java | use getInitParameterNames | 1 |
| collections | ||
| Collections1.java | simple collection deposit/retrieve | 1 |
| Collections2.java | collection deposit/retrieve, check for false positives | 1 |
| Collections3.java | collection of collections | 2 |
| Collections4.java | test of iterators | 1 |
| Collections5.java | test of iterators | 1 |
| Collections6.java | test of maps | 1 |
| Collections7.java | test of map iterators | 1 |
| Collections8.java | collection copying | 1 |
| Collections9.java | more complex collection copying | 0 |
| Collections10.java | more complex collection copying | 0 |
| Collections11.java | interprocedural collection passing | 1 |
| Collections11b.java | simple collection deposit/retrieve | 1 |
| Collections12.java | collection copying through an array | 1 |
| Collections13.java | more complex collection copying through an array | 1 |
| Collections14.java | more complex collection copying through an array | 1 |
| factories | ||
| Factories1.java | simple factory problem with toLowerCase | 1 |
| Factories2.java | simple factory problem with String.toString | 1 |
| Factories3.java | factory problem with a string wrapper | 1 |
| inter | ||
| Inter1.java | simple id method call | 1 |
| Inter2.java | simple id method call | 2 |
| Inter3.java | chains of method calls | 1 |
| Inter4.java | store stuff in a field | 1 |
| Inter5.java | store stuff in a field | 1 |
| Inter6.java | bug in class initializer | 1 |
| Inter7.java | bug in class initializer | 1 |
| Inter8.java | multi-level context sensitivity test | 1 |
| Inter9.java | simple object sensitivity | 2 |
| Inter10.java | more complex object sensitivity | 2 |
| Inter11.java | object sensitivity and context sensitivity combined | 1 |
| Inter12.java | collection as a static field | 1 |
| Inter13.java | recursive case | 1 |
| Inter14.java | interprocedural loop | 1 |
| pred | ||
| Pred1.java | simple if(false) test | 0 |
| Pred2.java | simple correlated tests | 0 |
| Pred3.java | simple correlated tests | 0 |
| Pred4.java | correlated test with an int variable | 1 |
| Pred5.java | correlated test with a complex conditional | 1 |
| Pred6.java | correlated test with addition | 0 |
| Pred7.java | correlated test with multiple variables | 0 |
| Pred8.java | using an array element as in a predicate | 1 |
| Pred9.java | using an array element as in a predicate | 1 |
| reflection | ||
| Refl1.java | reflective call of a method | 1 |
| Refl2.java | reflectively access a field | 1 |
| Refl3.java | reflectively create a class and access its field | 1 |
| Refl4.java | bug in class initializer | 1 |
| sanitizers | ||
| Sanitizers1.java | simple sanitization check | 1 |
| Sanitizers2.java | simple sanitization check | 0 |
| Sanitizers3.java | safe redirect | 0 |
| Sanitizers4.java | buggy sanitizer | 2 |
| Sanitizers5.java | encode and then decode | 1 |
| Sanitizers6.java | sanitizers for directory traversal | 0 |
| session | ||
| Session1.java | simple session test | 1 |
| Session2.java | test of session false positives | 1 |
| Session3.java | test of session enumeration | 1 |
| strong_updates | ||
| StrongUpdates1.java | simple test of strong updates | 0 |
| StrongUpdates2.java | simple test of strong updates | 0 |
| StrongUpdates3.java | strong updates in data structures | 0 |
| StrongUpdates4.java | a more tricky test of when we can't assume a strong updat... | 1 |
| StrongUpdates5.java | making a shared servlet field thread-local | 0 |
| aliasing | ||
| Aliasing1.java | simple aliasing because of assignment | 1 |
| Aliasing2.java | simple aliasing false positive | 1 |
| Aliasing3.java | simple aliasing of an array element | 1 |
| Aliasing4.java | simple aliasing with casts | 1 |
| Aliasing5.java | interprocedural argument aliasing | 1 |
| Aliasing6.java | aliasing with copy propagation | 7 |
| datastructures | ||
| Datastructures1.java | simple test of field assignment | 1 |
| Datastructures2.java | multiple objects of the same type | 1 |
| Datastructures3.java | simple nexted data | 1 |
| Datastructures4.java | simple nexted data (false positive) | 1 |
| Datastructures5.java | nested data in a loop | 1 |
| Datastructures6.java | nested data in a loop | 1 |
SecuriBench has been developed at Stanford as part of the Griffin application security project.
DroidBench -- a collection of benchmarks for evaluating taint analysis tools for Android apps
A collection of large benchmarks from the same effort: Stanford SecuriBench.
Foundstone's Hackme Books.
OWASP's WebGoat, which is included as part of SecuriBench.
The following publications of the Stanford SUIF group at Stanford are related to Securibench and Securibench Micro. Note that these publications are from a while back (2005 or so) and much research has been done since then.