Testing is full of organizational challenges that take a lot of our time.  Let’s face it, changing requirements, backend updates that influence UI behavior, busted cases, and fixing simple grammatical/syntactical errors all take their toll — both on us and our test documentation.  

If the above describes more of your life as a QA tester than you’d care to admit, let me shine a ray of sunshine into your day.  The difficulties of day-to-day testing life won’t go away, but (as I discovered) there are better ways to engage with and manage these challenges.  After a recent, successful trial with TestRail, we’ve chosen to evaluate it for the next few months to determine just how good a fit it really is for us.

The following are three standout attributes that really caught our attention.  

1. Test Case Organization and Orchestration

TestRail enables testers to create test cases very quickly.  Like super-fast.  Prior to TestRail, our method for creating test cases was a rather tedious and inefficient Google spreadsheet.  Apparently, this isn’t all-too-uncommon among the crowd offering TestRail testimonials.  Depending on your preference (either for a long, descriptive or step-based test case), a simple 5-8 step test case can be created in a matter of seconds to a minute.  

Also, the level of organization TestRail enables when it comes to writing, grouping, and executing test cases is, well, a revelation.  Once a decent amount of cases have been written, they can be executed in a test run.  Cases completed within a test run contribute real-time feedback as to the success of the run as a whole and the individual cases being tested.  Previously, this kind of data came at the end of testing via JIRA reports.  Now, we can see it as testing is underway.  

2. JIRA Integration

Speaking of Atlassian’s JIRA Software, it’s a great tool.  TestRail makes using JIRA that much better, though.  After the aforementioned spreadsheet was filled out, that information had to then be translated to a bug ticket in JIRA — manually.  

Now, with TestRail’s JIRA integration capability, the results logged against a test case can be pushed directly into a defect report in JIRA.  I won’t lie, the implementation is a little bit of a faff because the ticket TestRail creates is titled with the case’s title in TestRail.  This forces me to perform updates to the ticket in JIRA because of our project’s naming conventions.  Obviously, this won’t affect everyone.  

Bottom line, the ability to push 80-90% of the necessary defect information to JIRA while operating in TestRail is a huge productivity boost. 

3. Reporting 

Ok, this one is probably the biggest for us.  If you’re a QA tester and a follower of Donald Firesmith, then Common System and Software Testing Pitfalls is either on your reading list or soon to be.  Apparently, a major pitfall of software testing isn’t using metrics to guide decision-making.  Who would’ve thunk it?  

Maybe we’re just not that into JIRA reports.  JIRA has some great features for showing sprint velocity, issues solved versus issues created, etc.  What it doesn’t show, however, is how often, when, and under what conditions a test case fails or has failed.  Needless to say, we’ve come up with all sorts of pulse checks, stability assessments and such.  That’s not something we have to endure any longer, though!

TestRail’s reporting feature is a huge plus for us because we have better visibility into every part of the QA process.  Whether it’s in-the-moment test case passes or failures, a review of the latest test run or major milestone, or a super-granular performance report on a single test case over the last quarter, it’s possible.  

Additionally, manually or automatically generated reports can be shared with all — or a very few, specific — parties.  Options include email links to reports in test rail (fine if the recipient is a registered user) or attachments (great for non-user stakeholders).  Email reports can also be scheduled at regular intervals.  These can be viewed in TestRail at one’s leisure or can be emailed automatically at runtime.  

Wrapping Up

Is TestRail a QA dream app?  Well, for me at least, it’s too early to tell.  However, it checks all of my boxes at this time.  Perhaps familiarity will breed a certain level of contempt.  Then again, it may just be the perfect tool for our QA team to grow with for years to come.

So, my questions for you are . . . 

1. Have you demoed TestRail?
2. What are some standout features to you and your team?
3. What’s something you hate about TestRail?