Triaged Tester

February 11, 2009

Code coverage – The strategy

Filed under: Code Coverage,Stratergies — Triaged Tester @ 1:05 pm
Tags: ,

Here’s how I approach my job, and how coverage helps.

 

1. I divide the code under test into three categories.

·         High risk code could cause severe damage (wipe out data, give non-obviously wrong answers that might cost a user a lot of money), or has many users (so the cost of even minor bugs is multiplied), or seems likely to have many mistakes whose costs will add up (it was a tricky algorithm, or it talks to an ill defined and poorly-understood interface, or I’ve already found an unusual number of problems).

·         Low risk code is unlikely to have bugs important enough to stop or delay a shipment, even when all the bugs are summed together. They would be annoyance bugs in inessential features, ones with simple and obvious workarounds.

·         Medium risk code is somewhere in between. Bugs here would not be individually critical, but having too many of them would cause a schedule slip. There’s good reason to find and fix them as soon – and as cheaply – as possible. But there are diminishing returns here – time spent doing a more thorough job might better be spent on other tasks.

 

Clearly, these are not hard-and-fast categories. I have no algorithm that takes in code and spits out “high”, “medium”, or “low”. The categories blend together, of course, and where debatable code lands probably don’t matter much. I also oversimplify by treating risk as being monolithic. In reality, some medium risk code might be high risk with respect to certain types of failures, and I would tailor the type of testing to the blend of risks.

 

2. I test the high risk code thoroughly. I use up most of the remaining time testing the medium risk code. I don’t intentionally test the low risk code. I might hope that it gets exercised incidentally by tests that target higher-risk code, but I will not make more than a trivial, offhand effort to cause that to happen.

 

3. When nearing the end of a testing effort (or some milestone within it), I’ll check coverage.

·         Since high risk code is tested thoroughly, I expect good coverage and I handle missed coverage as described earlier.

·         I expect lower coverage for medium risk code. I will scan the detailed coverage log relatively quickly, checking it to see whether I overlooked something – whether the missed coverage suggests cases that I’d really rather a customer weren’t the first person to try. I won’t spend any more time handling coverage results than I would for thorough testing (even though there’s more missed coverage to handle).

·         The coverage for low risk code is pretty uninteresting. My curiosity might be piqued if a particular routine, say, was never entered. I might consider whether there’s an easy way to quickly try it out, but I won’t do more. So, again, coverage serves its purpose: I spend a little time using it to find omissions in my test design.

Advertisements

Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at WordPress.com.

%d bloggers like this: