I was participating in a code review today and was reminded by a senior architect, who started working as an intern for me years ago, of a testing technique I had used with one of his first programs. He had been assigned to create a basic web application that collected some data from a user and wrote it to a database. He came into my office, announced it was done and proudly showed it to me. I walked over to the keyboard, entered a bunch of junk and got a segmentation fault in response.
Although I didn’t have a name for it, that was a standard technique I used when evaluating applications. After all, the tried and true paths, expected inputs and easy errors will be tested early and often as the developer exercises the application using the basic use cases. As Boris Beizer said, “The high-probability paths are always tested if only to demonstrate that the system works properly.” (Beizer, Boris. Software Testing Techniques. Boston, MA: Thomson Computer Press, 1990: 76.)
It is unexpected input that is useful when looking to find untested paths through the code. If someone shows me an application for evaluation the last thing I need to worry about is using it in an expected fashion, everyone else will do that. In fact, I default to entering data outside the specification when looking at a new application. I don’t know that my team always appreciates the approach. They’d probably like to see the application work at least once while I’m in the room.
These days there is a formal name for testing of this type, fuzzing. A few years ago I preferred calling it “gorilla testing” since I liked the mental picture of beating on the application. (Remember the American Tourister luggage ad in the 1970s?) But alas, it appears that fuzzing has become the accepted term.
Fuzzing involves passing input that breaks the expected input “rules”. Those rules could come from some formal requirements, such as a RFC, or informal requirements, such as the set of parameters accepted by an application. Fuzzing tools can use formal standards, extracted patterns and even randomly generated inputs to test an application’s resilience against unexpected or illegal input.