Today’s first session is Making Test Automation with Web Applications Work with Jeremy Miller. He’s the guy behind StoryTeller, which is the automated testing tool we use at work.

What makes a good test? Any test that isn’t repeatable is useless.

Test with the finest grained mechanism that tells you something important. Don’t test business logic through the screen. It does make sense to test your UI “magic” (conventions, routing, presentation logic, etc.).

White box testing. The goal of testing is not to make sure your application is perfect. It is about reducing risk to an acceptable level. So, white box testing is very valid, since it tends to be cheaper.

Collapse your application into a single process/appdomain. Even if you have multiple services – bring them all together into one process. You solve your functionality problems and ignore the communication issues. You then can test the communication separately.

Test data sets suck. Define your test data in your test in a declarative fashion.

Diving straight into code and Storyteller. I kind of feel sorry for anyone who hasn’t seen Storyteller before, because there is no explanations about what they are doing with it. They are using RavenDB, a document database that can run in process. This helps them easily create new databases for each test without a lot of the relational overhead. They just put in the documents they need for the test.

There are lots of issues with test data. If you aren’t careful it will lead to brittle tests. This is why you should strive to tie it to the tests themselves instead of defining the traditional large sets of data. If you set up your data as part of your test, it becomes clearer to those trying to understand what the test is testing.

There exists an embedded SMTP server you can use to test email. They also use a tool called PortFinder that finds an open port you can use to listen on, which could be very useful for testing things that use TCP/IP.

If your tests are running all of the time (in continuous integration) then they are worse than worthless because they hold you back.

They use webdriver to run their UI layer and have defined what they call “Gestures” in Storyteller to perform standard UI operations.

Provide element handlers for each type of web element to facilitate making less brittle tests.

Separate your test expressions from you screen drivers. Break out the screen driving logic so that your actual tests are testing logic, It reduces brittleness.

Thread.Sleep is the easiest way to make your test suck, take too long, and unreliable. Build reasonable waiting tools that are more graceful.

It can be valuable to have a stub date/time class so that you can have better control over time. You may even want to consider taking over the system clock to give you friendly timestamp-sensitive tests.

At this point I had to bail. Too much magic, not enough meat.