How to Create Maintainable Acceptance Tests

This post summarizes what I’ve learned from various sources about making acceptance or black-box tests maintainable. This topic is of great interest to me because I believe in the benefits that acceptance tests can bring (such as living documentation) but I’m also very much aware that it is all too easy to create an unmaintainable monster whose weight eventually crushes you. So the question is how to navigate the minefield to get to the golden apple?

The key elements that contribute to the maintainability of acceptance tests are:

1. Aligned business, software, and test models => small change in business requires only a similarly small change in the software and a small change in tests (Gojko Adzic explains that very well in his JavaZone 2012 talk Long-term value of acceptance tests)
   • The key to gaining the alignment is to use business language in all the three models from the very start, building them around business concepts and relationships
2. Testing under the surface level, if possible
   • Prefer to test your application via the service layer or at worst the servlet layer; only test on the UI level if you really have to and only as little as possible for UI is much more brittle (and also difficult to test)
   • The more you want to test the more you have to pay for it in the terms of maintenance effort. Usually you decide so that you cover the part(s) of the application where the most risk is – the best thing is to do cost-benefit evaluation.
3. Isolating tests from implementation by layers of test abstraction
   • Top layer: Acceptance tests should only describe “what” is tested and never “how” to test it. You must avoid writing scripts instead of specifications.
   • Layer 2: Instrumentation – right below the acceptance test is an instrumentation layer, which extracts input/output data from the test and defines how to perform the test via a high-level API, provided by the next level (we could say a test DSL) such as “logInUser(X); openAccountPage();”
   • Layer 3: High-level test DSL: This layer contains all the implementation details and exposes to the higher layer high-level primitives that it can use to compose the tests without depending on implementation details (ex.: logInUser may use HtmlUnit to load a page, fill a form, post it)

(And of course many, if not all, of the rules for creating maintainable unit tests apply as well.)

Note To Self: How to Solve Vagrant Destroy Failing with “Error in API call” in ffi.rb

Sometimes “vagrant destroy” fails with an exception from the depths of the virtualbox Ruby gem. A solution might be to use the command-line VirtualBox tool VBoxManage to forcibly stop the machine.

Visualize Maven Project Dependencies with dependency:tree and Dot Diagram Output

The dependency:tree goal of the Maven plugin dependency supports various graphical outputs from the version 2.4 up. This is how you would create a diagram showing all dependencies in the com.example group in the dot format:

mvn dependency:tree -Dincludes=com.example -DappendOutput=true -DoutputType=dot -DoutputFile=/path/to/output.gv

(The output is just a text file with the extension Graphviz gv.)

To actually produce an image from the dot file you can use one of dot renderers, f.ex. this online dot renderer (paste into the right text box, press enter).

You could also generate the output f.ex. in the graphml format & visualize it in Eclipse.

Note: Thanks to the reader Not Relevant for pointing out the right extension and a typo.

Key Lessons from the Specification by Example Course, Day 1

I’m taking part in a course of Specification by Example, lead by Gojko Adzic. Here I want to summarize the key things I’ve learned in the first day of this entertaining and fruitful course thanks to both Gojko and my co-participants.

If you haven’t heard about Specification by Example (SbE) before (really?!), then you need know that its main concern is ensuring that you build the right thing (complimentary to building the thing right), which is achieved by specifying functionality collaboratively with business users, testers, and developers, clarifying and nailing them with key examples, and finally, where it is worth the effort, automating checks of those examples to get not only automated acceptance tests but, more importantly, a “living documentation” of what the system does that never gets out of date. Best to read the key ideas described by Gojko himself or the SbE Wikipedia page.

uCertify

Some guy from uCertify, which offers preparation kits for various programming exams in Java and other areas, asked me to review their PrepKit service. I hadn’t really the time to do so but I’m sure it’s awsome,  and it even work on Macs. So if you want to prepare for an exam, you might consider them.

I’m very thankful for them for teaching me to never promise anything I’m not really commited to do.

Annual Blogging Report 2011

The WordPress.com stats helper monkeys prepared a 2011 annual report for this blog.

Here’s an excerpt:

Madison Square Garden can seat 20,000 people for a concert. This blog was viewed about 62 000 times in 2011. If it were a concert at Madison Square Garden, it would take about 3 sold-out performances for that many people to see it.

Click here to see the complete report.

Most interesting links of December

Recommended Readings

• The Netflix Chaos Monkey – how to test your preparedness for dealing with a system failure so that you won’t experience nasty wakeup when something really fails in Sunday 3 am? Release a wild, armed monkey into your datacenter. Watch carefuly what happens as it randoly kills your instances. This is exactly what Netflix does with their with their cloud infrastructure – also a great inspiration for my recent project. Do you need to be always available? Than consider employing the chaos monkey – or a whole army of monkeys!
  (PS: There is also a post with a picture of the scary monky.)

Links to Keep

• BDD: Write specifications, not scripts (from the Concordion site) – relatively brief yet very enriching practical description of how to do behavior-driven development a.k.a. Specification by Example right, the key point here being “Write specifications, not scripts.” It says why not (for scripts overspecify -> are brittle, specs should be stable) and how to do it (decouple the stable spec and the volatile system via fixture code, expose minimal stuff to the spec, perhaps evolve a DSL between the fixture and the system). It also lists common “smells” of BDD done wrong. If it still isn’t clear to you, read the Script to Specification Makeover example (or perhaps read it anyway). BTW, Concordion is a new tool for doing BDD based on JUnit and HTML, which was created as a response to the weaknesses of Fit[Nesse], i.e. exactly the tendency to do scripting instead of specifications. It looks very promissing to me!

SW Utilities

• (Linux/Mac) Autojump – superfast navigation between favorite directories in the command line (via Jake McCrary) – it keeps track of how much time you spend in each directory and when you execute j <substring of directory path/name>, it jumps into the most frequently used one matching the substring. It awesome! (You can also run jumpstat to see the statistics.)
• (Linux) Tcpkill – service/network outage testing (via Jake McCrary) – kill connections to or from a particular host, network, port, or combination of all – useful e.g. when you want to test that your software is resilient to the outage of a particular service or server – less brutal than actually killing the database etc. instances. We need to test that our application recoveres properly when one of our MongoDB nodes dies so this may be quite useful.
• Manik Hot Deploy Plugin for Maven Projects (v1.0.2 in 5/2011; older version in the Marketplace) – plugin that can do hot and incremental deployment to any app server (simply by copying to its hotdeploy directory or the directory of an installed webapp) whenever you run mvn install or automatically whenever sources change, multi-module support

Clojure Corner

• Jake McCrary: Continuous Testing With Clojure and Expectations – continuous test runner lein-autoexpect for Clojure tests written using the library expectations by Jay Field.
• Jake McCrary: Quickly Starting a Powerful Clojure REPL – Clojure REPL only two steps away: 1) Run Emacs, 2) Execute M-x clojure-swank (no more need to open an existing Leinigen project) – the trick is to install the Leinigen plugin swank-clojure and use Jake’s elisp function clojure-swank that automatically starts the swank-clojure server. (I had to hack the function for the clojure-swank output contained “null” instead of “localhost”, likely due to incorrect DNS setup.)

AWK: Extract Logs for the Given Date(s) from a Log File

If your log file has entries like these:

2011-12-10T22:00:27.996+0000 [http-8080-1] INFO  my.package.MyClass Hello, I'm alive!
2011-12-11T17:05:46.811+0000 [http-8080-15] ERROR my.package.MyClass  - Error caught in DispatcherServlet
        at my.package.MyServiceClass(MyServiceClass.java:36)
...
2011-12-11T17:06:10.120+0000 [http-8080-14] DEBUG my.package.MyClass Whoo, that has been a long day!

Then you can use the following bash script snippet to extract logs only for a particular day or consecutive days, including everything – even lines not starting with the date such as stacktraces – between the first log of the date up to the first log of a subsequent date (default: yesterday):

LOGFILE_ORIG="$0"; LOGFILE="${LOGFILE_ORIG}.subset"
if [ -z "$LOGDAY" ]; then LOGDAY=$(date +%F -d "-1 days"); fi
if [ -z "$AFTERLOGDAY" ]; then AFTERLOGDAY=$(date +%F -d "$LOGDAY +1 days"); fi
echo "Extracting logs in the range (>= $LOGDAY && < $AFTERLOGDAY) into $LOGFILE ..." awk "/^$LOGDAY/,/^$AFTERLOGDAY/ {if(!/^$AFTERLOGDAY/) print}" $LOGFILE_ORIG > $LOGFILE

This date format works on Linux. Date is very flexible and can provide dates in any format, not only yyyy-mm-dd. You may also want to read more about Awk ranges and other tips.

You would run it in one of the following ways:

$ ./analysis.sh /path/to/logfile.log
$ LOGDAY=2011-12-12 AFTERLOGDAY=2011-12-17 ./analysis.sh /path/to/logfile.log

Quiz: What’s the Best Test Method Name?

Which of the following names of test methods do you think to be the best?

(Notice that we could leave out “payment_” from the last name if it is clear from the context, i.e. from the fixture [a fancy name for test class] name.)

According to the holy book of Clean Code, the code should make visible the intent as much as possible. According to the testing guru Kent B., a test should be telling a story to its reader – a story about how the code should be used and function. According to these two and my own experiences from reading a lot of (test) core written by other people, the last one is absolutely the best. However you have the right to disagree and discuss

PS: I firmly believe that calling a test method “test()” should be punishable.

Getting Started with Amazon Web Services and Fully Automated Resource Provisioning in 15 Minutes

While waiting for a new project, I wanted to learn something useful. And because on many projects we need to assess and test the performance of the application being developed while only rarely there is enough hardware for generating a realistic load, I decided to learn more about provisioning virtual machines on demand in the Cloud, namely Amazon Web Services (AWS). I’ve learned a lot about the tools available to work with AWS and the automation of the setup of resources (machine instances, security groups, databases etc.) and automatic customization of virtual machine instances in the AWS cloud. I’d like to present a brief introduction into AWS and a succinct overview of the tools and automation options. If you are familiar with AWS/EC2 then you might want to jump over the introduction directly to the automation section.