My e2e Top 3

Having an automated end to end test suite can be a super comforting assurance that your application is in a ready state.  Nothing says 'ship it' like having just run a user-like test against every independent feature in the application with no mocks.  Maintaining that test suite can become like battling a hydra if the tests are not properly built though.  I was on a team a few years back that spent about 40-60% of the day maintaining existing cucumber tests.  That is counterproductive and just plain painful.  People get cranky about that sort of thing.

Before I dive in let's lay out a working definition of End to End tests for this post.  I define End to End (e2e) tests to be those that interact with the live application through the user interface and very rarely, if ever, mock functionality regardless of who maintains that functionality.  Tools such as Cucumber, Protractor, Geb, and RSpec are commonly in used to write these tests.  Techniques such as ATDD or BDD generally yield e2e tests.  E2E tests are at the top of the testing pyramid and there are far fewer of them than unit tests.

OK, academia complete...on to the good stuff...

Thing One - specify behavior, not implementation

You would think that a practice rooted in the name Behavior Driven Development would make this point obvious, but we continue to get this wrong.  Any time you find yourself using words like "Page", "Button", "Link", "Text Box", "Drop Down", "Radio Button", etc. in an e2e test specification it is a smell and you should consider whether or not there is a better way.  The reason may not be obvious at first, but what occurs with these tests is that they become increasingly brittle over time.  As the application matures and the implementation morphs these tests become a maintenance headache.  Although there is more to it than just names, they honestly best illustrate the difference.

Tests that should serve you well over time

• The XYZ Application should allow the user to enter and store address information.
• The User Information feature should allow the user to navigate to the Item Ordering feature.
• The user should be able to view all previously ordered items in History.

Tests that will eventually make you wish you never wrote them

• The XYZ Application should have an address form with Street Address, City, State, Postal Code fields.
• The user should be taken to the Order Page upon clicking the Place Order Button on the User Information page.
• The user should see a table including Order Date, Item Name, and Order Cost on the History Screen.

Thing Two - use abstractions to isolate implementation awareness

In most cases these days this really boils down to using the page object pattern.  Regardless, the important point is that there should be exactly one place in the entirety of your e2e test suite where location of a given UI element takes place.  There should be an object (used very loosely here) whose single responsibility is knowing how to get a hold of the UI and interact with the elements.  Any e2e specification should use these objects to locate and retrieve UI elements and and never do so directly in the spec.

For example, there is a fantastic explanation of how to accomplish this in AngularJS/Protractor over on the ThoughtWorks blog.  I implemented this pattern over the last few months with a web development team at a client site and we have all been very pleased with the outcome.

Thing Three - don't test against specific data, but if you must, control it

I cracked open a failing e2e spec the other day that was expecting 4 rows of data in an HTML table  and only found 3.  I didn't see anything in the 'arrange' portion of the spec that was loading the data so I scrolled up to the 'beforeEach'.  Nothing there either.  Hmmm...OK...Pardon me while I flip a table!

In all seriousness, if for some reason you are compelled to look for specific data in an e2e spec, be sure the spec is managing the test data.  A test that is written to be dependent on data from a completely separate process is going to fail and probably sooner than you think.  Better still is to try and keep most of the e2e specs data independent.  If the behavioral assertion can be vague enough to simply check that there is data, for instance, then let it be less precise for the sake of always being accurate.  There should be a unit test somewhere that handles making sure inputs equal outputs and the name doesn't end up in the address attribute of an object.

A data independent approach can occasionally leave holes in assurance.  For instance, if the wrong data field is bound to the view and the e2e test doesn't inspect the value of the data in the view the mistake can go unnoticed.  In these specific cases decide what is least painful for your team, but try to keep control of things so that the test doesn't fail next week.  Maybe a mock is in order, or an inserted row in the back end database, or perhaps a regex to ensure the data at least meets certain criteria.

Thanks for reading this far.  Happy testing!