xUnit

Principles

1. Keep Test Logic out of Production Code
2. Tests should be easy to run.
3. Tests should be easy to write and maintain.
4. Tests should be repeatable / consistent.

Terminologies

Patterns

Benefits of writing tests

Tests as Specification. The very act of thinking through various scenarios in enough detail to turn them in tests helps us to identify those areas where the requirements are ambiguous or self-contradictory. This helps us improve the quality of the specification which in turn helps improve the quality of the software it specifies.

Test First enable us to clarify specification and design decisions. Example Driven Development (EDD) is a variation of TDD. Frameworks for EDD: RSpec (Ruby), JBehave (Java).

Tests as Bug Repellent. Prevent bugs from happening.

Defect Localization. If we have made our unit tests fairly small by testing only a single behavior in each, we should be able to pinpoint the bug pretty quickly based on which test is failing. This is one of the big advantages of unit tests over customer tests. The customer tests will tell us that some behavior expected by the customer isn't working. The unit test will tell us why. We call this phenomena Defect Localization. If we have a failing customer test with no unit tests failing, that is an indication of a Missing Unit Test. All these benefits are wonderful but we cannot achieve them if we don't write tests to cover off all possible scenarios each unit of software needs to cover. Nor will we get the benefit if the tests themselves have bugs in them. Therefore it is crucial to keep the tests as simple as possible so that they can be easily seen to be correct. Writing unit tests for our unit tests is not a practical solution but we can and should write unit tests for any Test Utility Method (page X) to which we delegate complex algorithms needed by the test methods.

Tests help us understand the SUT (System Under Test).

Tests as Documentation.

Tests as Safety Net. Making changes to this code is risky because we never know what we might break and we have no way of knowing whether we have broken something! This forces us to work very slowly and carefully by doing a lot of manual analysis before we make changes. When working with code that has a regression tests suite we can work much more quickly. We can adopt a more experimentaly style of changing the software. "I wonder what would happen if I changed this? The effectiveness of the safety net is determined by how completely our tests verify the behavior of the system. Missing tests are like holes in the safety net. Incomplete assertions are like broken strands. Each can let bugs of various sizes through.

Things to keep in mind when writing tests

Tests should reduce (and not introduce) risk.

Tests Must Do No Harm. Keep Test Logic out of Production Code principle directs us to avoid putting test-specific hooks into the SUT (System Under Test). It is certainly desirable to design the system for testability but any test-specific code should be plugged in by the test and only in the test environment; it should not exist in the SUT when it is in production. Don't Modify the SUT. That is, we must be clear about what SUT we are testing and make sure that we don't replace the parts we are testing with test-specific logic.

Tests should be easy to run. Most of us want to write code. Testing is just a necessary evil. Automated tests give us a nice "safety net" so that we can work more quickly but we will only run the automated tests frequently if they are real easy to run. What makes tests easy to run? There are four specific goals: They must be Fully Automated Tests so they can be run without any effort. They must be Self-Checking Tests so they detect and report any errors without manual inspection. They must be Repeatable Tests so they can be run multiple times with the same result. Ideally, each test should be an Independent Test that can be run all by itself. With these four goals satisfied, one push of a button (or keyboard shortcut) is all it should take to get the valuable feedback the tests provide.

Unrepeatable Tests usually come about because we are using a Shared Fixture (page X) of some sort (and I include any persistence of data implemented within the SUT in this definition.) When this is the case, we must ensure that our tests are "self-cleaning" as well. When cleaning is necessary, the most consistent and foolproof way is to use a generic Automated Teardown (page X) mechanism; it is possible to write tear down code for each test but this can result in Erratic Tests when not implemented correctly in every test.

Tests should be easy to write and maintain. Coding is a fundamentally hard activity. We need to keep a lot of information in our head as we work. When we are writing tests, we should be focused on testing and not on the coding of the tests. This means that tests need to be simple; simple to read and simple to write. They need to be simple to read and understand because it is hard to test the automated tests themselves. We want to Minimize Test Overlap so that only a few tests are affected by any one change.

We should strive to Verify One Condition per Test by creating a separate Test Method (page X) for each unique combination of pre-test state and input. Each Test Method should drive the SUT through a single code path. The one main exception to Test Methods being short is those customer tests which express real usage scenarios of the application. They are a useful way to document how a potential user of the software would go about using it; if these involve long sequences of steps then the Test Methods should reflect this.

Test one thing at a time. Trying to verify too much functionality in a single test is bad. It makes it harder to pinpoint the problem, and often slow down the tests. It is particularly important when we do test-driven development because we write our code to pass one test at a time and we want each test to introduce only one new bit of behavior into the SUT. We should strive to Verify One Condition per Test by creating a separate Test Method (page X) for each unique combination of pre-test state and input. Each Test Method should drive the SUT through a single code path.

Test should be expressive. We should build a library of Test Utility Methods, Domain Specific Language (DSL), that constitute a domain-specific testing language. This allows the test automater to express the concepts what they wish to test without having to translate the thoughts into much more detailed code. Creation Methods (page X) and Custom Assertion (page X) are good examples of the building blocks that make up this Higher Level Language. The DRY principle should be applied to test code in the same way it is applied to production code.

Minimize overlap between tests. We want to write our tests in such a way that the number of tests impacted by any one change is quite small. That means we need minimize overlap between tests. We should strive to Verify One Condition per Test. Ideally, there should only one kind of change that would cause a test to require maintenance. System changes that impact fixture set up or tear down code can be encapsulated behind Test Utility Methods to further reduce the number of tests directly impacted by the change.

Philosophies

Test First vs Test Last. When tests are written first:

1. The design of the system is inherently testable. When we design program, there are design decisions that make it easy for testing. This does not mean that we mix test code with production code.
2. We write only enough code to make the tests pass, the production code tends to be a small (clean).
3. Functionality that is optional tends not to be written (Just In Time design)
4. The tests tend to be more robust because the right methods are provided on each object based on the tests' needs.

Test-by-Test or All-At-Once. The test-driven development process encourages us to "test a bit, code a bit, test a bit more." Some developers prefer to identify all the tests needed by the current feature before starting any coding. This has the advantage of letting them "think like a client" or "think like a tester" and avoids being sucked into "solution mode" too early. Test-driven purists argue that we can design more incrementally if we build the software one test at a time. "It's easier to stay focused if we only have a single test failing." Many test drivers report not using the debugger very much because the fine-grained testing and incremental development leave little doubt about why tests are failing; the tests provide Defect Localization and the last change we made (which caused the problem) is still fresh in our minds.

This is especially relevant when talking about unit tests because we can choose when to enumerate the detailed requirements (tests) of each object or method. A reasonable compromise is to identify all the unit tests at the beginning of a task (possibly roughing in empty Test Method (page X) bodies) but only coding a single test body at a time. We could also code all the Test Method bodies and then disable all but one of the tests so we can focus on building the production code one test at a time.

With customer tests, we probably don't want to feed the tests to the developer one by one within a user story but it does make sense to prepare all the tests for a single story before development of the story is started. Some teams prefer to have the customer tests for the story identified before they will estimate the effort to build the story as the tests help frame the story.

Outside-In or Inside-Out. Designing the software from the outside inwards implies thinking first about black-box customer tests (a.k.a. "story tests") for the entire system and then thinking about unit tests for each piece of software we design. Along the way we may also implement component tests for the large-grained components we decide to build.

Each of these sets of tests causes us to "think like the client" well before we start thinking like a software developer. We focus first on the interface we provide to the user of the software whether it be a person or another piece of software.

Some people prefer to design outside-in but then code inside-out to avoid dealing with the "dependency problem". This requires anticipating the needs of the outer software when writing the tests for the inner software. It also means that we don't actually test the outer software in isolation of the inner software. This avoids the need for Test Stubs (page X) or Mock Objects in many of the tests.

Other test drivers prefer to design and code outside-in. Writing the code outside-in forces us to deal with the "dependency problem". We can use Test Stubs to stand in for the software we haven't yet written so that the outer layer of software can be executed and tested. We can also use the Test Stubs to inject "impossible" indirect inputs (return values, out parameters or exceptions) into our system under test (SUT) to verify that it handles them correctly.

Once the subordinate classes have been built, we could remove the Test Doubles from many of the tests. Keeping them gives us better Defect Localization at the cost of potentially higher test maintenance cost.

State or Behavior Verification. From writing code outside-in it is a small step to verifying behavior rather than just state. The "statist" view is that it is sufficient to put the SUT into a specific state, exercise it, and verify that it is in the expected state at the end of the test. The "behaviorist" view is that we should specify not only the start and end state of an object but the calls it makes to its dependencies. That is, we should specify the details of the calls to the "outgoing interfaces" of the SUT.
The behaviorist school of thought is sometimes called behavior-driven development. It is evidenced by the copious use of Mock Objects or Test Spys throughout the tests. It does a better job of testing each unit of software in isolation.