What are unit tests?

here's a great, but kinda technical definition StandardDefinitionOfUnitTest

but, here's a more general example and an attempt to illustrate "why code is so crappy" and some of the difficulties in translating requirements and designs into "high quality" software.

Let's say we have a screen to enter credit card information. Let's further suppose that one of the requirements is to "Validate the credit card information". Let's then say that "Validate the credit card information" means specifically:

card types

The software written to perform this validation will have a number of "code" units that likely would translate into functions. One of these functions would likely be "validate length of card number".

Note: right now we're saying that "function"=="code unit" to simplify things.

Let's walk through what we might do to validate JUST Visa and Mastercard length. We'll write a function that will take two inputs (card type, and card number) and verify the card number has the proper number of digits. This function should return an error message if the card is not the right length or an empty string if it IS valid.
Let's say that the developer writes the following code (this is pseudo code):


validateCardLength (cardType,cardNumber)
if cardType equals "MC" and cardNumber.length != 16 return "Master Card must be 16 digits"
if cardType equals "VISA" and (cardNumber.length != 16 and cardNumber.length != 13) return "Visa must be 13 or 16 digits"
return ""


How many test cases should we write? What should they be?

Lets explore!

The trivial solution is to write 1 test case... something like this


test
validateCardLength("MC","1234123412341234") equals ""


This is a "happy path" test and it "kinda" verifies that the code does what we want in the optimal case. The problem is that it doesn't cover the second and third lines of the function. This test will never verify if you can even enter a valid visa or not. So to get the complete happy path for this VERY simple function we'd need:


test
validateCardLength("MC","1234123412341234") equals ""
validateCardLength("VISA","1234123412341234") equals ""
validateCardLength("VISA","1234123412341") equals ""


This is good, we know you can at least enter cards with the RIGHT number of digits, but kinda the whole point of this function is to verify you can't enter cards with the WRONG number of digits. Let's add to our tests then:


test
validateCardLength("MC","1234123412341234") equals ""
validateCardLength("VISA","1234123412341234") equals ""
validateCardLength("VISA","1234123412341") equals ""

validateCardLength("MC","123412341234123") equals "Master Card must be 16 digits"
validateCardLength("VISA","123412341234123") equals "Visa must be 13 or 16 digits"



So, for this simple thing, the minimum number of tests most people would agree are necessary is about 5. This is actually mathematically implied by Cyclomatic Complexity, but that is beyond the scope if this. When developers talk about "coverage", they are referring to how many of these branches have they covered with tests. Our first example would typically imply 20% coverage and the last example would imply 100% coverage.

Now my question is, are we done with our unit testing for this function?

I would argue that yes, we're done testing the "card length" function, but there are a bunch of "unanswered" questions that a typical developer (or architect) still has to answer in the larger scope. What immediately comes to mind is "what happens if you pass an invalid card type?"
Right now, the validateCardLength function happily ALWAYS returns that the card length is A-OK.

Just to further illustrate how unit testing doesn't catch everything, here are some other sources of errors that these tests will likely miss:

• What happens if I pass a 100 character string as a card number? (whoops, the "length" function can only handle strings up to 50 chars)


• What happens if the system that passes card type uses the code "M2" or "MC" to BOTH mean mastercard? (requirements didn't specify the exact codes)


• What happens if an intermediate component encrypts the card number before sending it to the function? (does the encrypted string have the same length as the source string?)


• What happens if some intermediate code automatically trims the card length to 16 characters? (this is a fun one which will lead to lotsa finger pointing)
  for non-developers or those who just don't get it: This is a problem because if the input string is trimmed to 16 characters, this means that when a user enters 17 characters the "validateCardLength" will return "" because it's only getting the string ALREADY trimmed to 16 characters (so the length == 16). This is particularly entertaining because it is a common theme in the software development world to do "helpful" little things like this and totally screw things up in unexpected ways.
  

So for this mindlessly trivial piece of software, here are a number of things that need to be tested as well as a number of ways that 100% testing will totally miss problems. For those of you that fly on airplanes, think of how complicated avionics software must be and let me know when you stop shivering...