Michael Eriksson
A Swede in Germany
Home » Software development | About me Impressum Contact Sitemap

How to handle bug tracking

Introduction

There are many guides out there on how to write good bug-reports (or even, illogically, on how to write good bugs), including http://www.chiark.greenend.org.uk/~sgtatham/bugs.htmle and http://www.softwaretestinghelp.com/how-to-write-good-bug-report/e. Something that is regrettably lacking, is a good guide on how to read and handle bug reports—which many developers and almost all managers are very poor at. This page is an attempt to provide a partial remedy to this problem.


Side-note:

The exact terminology to use is somewhat disputed, with other alternatives being e.g. “error” and “issue”. The most commonly used terms in this context, however, is “bug”, and I will stick to that for now. I note that the runner-up, “issue”, has a wider meaning and includes e.g. requests for assistance that are not caused by software errors. Parts of the below will apply to this wider term; other parts will not.


I have made extensive experiences with bug tracking over the years, and I have both used it and seen it used from a number of perspectives, including the developers, the users, the testers, and the release managers.

Attitude towards users

  1. Be thankful for all input. Do not assume that you are doing the user a favour by fixing “his” bug—he may very well take the view that he is doing you a favour by reporting “your” bug, having spent fifteen minutes writing a bug report on something he likely would never encounter again (could easily work-around, live with, whatnot). Things will go smother, and your product will benefit, if you adapt the same attitude—he is helping you, not the other way around.

    Making unreasonable demands on the user, using rude language, or otherwise being unappreciative, will lead to an alienated user and a poorer product.

    Note that a paying consumer may rightfully and righteously take the view that your employer is wasting his time and money, and may be in a less than good mode to begin with. Playing the “be grateful” card (even by implication) will not go over well.

    Also note that many companies make odd impositions for a user to be able to register a bug: I have, e.g., been made to register online with a dozen mandatory and irrelevant fields (that I duly filled with dummy information), had to wait half an hour for a confirmation email with my login data, and only then been able to actually enter my bug report. If a user is somewhat lacking in gratitude at that point, this is entirely your employers fault.

  2. Be kind to newcomers. Consider that the bug reporter may know considerably less about computers than you do, have less experience with bug-tracking systems (in particular, your local one), and will almost certainly know less about the application the bug occured in than you do.

    Handle e.g. non-native speakers similarly. You are not perfect either.

  3. Conversely, the user can be a highly competent developer with twice your experience and an IQ two standard deviations higher, so make very sure not to be a condescending medium fish.

  4. When requesting information, tell the user why. If you do, he is much more likely to give it to you. Example: The first link above states

    Somebody reported a bug to me once, and I asked him to try a command that I knew wouldn’t work. The reason I asked him to try it was that I wanted to know which of two different error messages it would give. Knowing which error message came back would give a vital clue. But he didn’t actually try it - he just mailed me back and said "No, that won’t work". It took me some time to persuade him to try it for real.

    The author sees this as a user error. I see it as a developer error: Had he stated upfront “This command will give one of several error messages. Knowing which one will tell me [...].” then he would almost certainly have received the cooperation he wanted.


    Side-note:

    This principle applies much more generally: As good as always, it is appropriate to tell a counter-part why some particular piece of information is needed. (Off the top of my head, the only exceptions are when the reasons are obvious, respectively the requestee is just an aidee of some sort to the requester.) Doing so increases the probability for cooperation, is more polite, and can help the counter-part to help the requester, should his request be incomplete, irrelevant, whatnot. This applies even when he, theoretically, has the right to unconditionally order the information to be given.


  5. Do not demand that users provide extensive extra tests of e.g. other versions or plattforms, that they try to re-install something before reporting a bug, or similar. A user who has given a correct report on how the bug occured on his system, as it was at the time, has done his job—the rest is your job. (Asking for his further help, without making demands and without making your work contingent on this help, is perfectly in order.)

    Note, in particular, that many bugs can be found in thirty seconds by actually looking at the code; sometimes a developer can even know what is wrong upon hearing the problem description. Forcing a user to spend twenty minutes (sometimes even considerably more) doing research for such a bug is extremely user unfriendly.

  6. Similarly, do not demand that the user provides a patch himself, even if you do work on an OSS/FSS software. He already has helped by reporting the bug, and it is unreasonable of you to demand further help from him in a rude manner. A “Sorry, we appreciate your report, but we do not have the resource to handle that right now. If you write a patch yourselves, however, we will be happy to integrate it.” is perfectly in order; a “We are not here to do your work for you. If you want it fixed, fix it yourself.”, OTOH, is not in anyway acceptable (and, yes, I have seen such formulations on repeated occasions—fortunately, always directed at others, else there would have been a word or two said in response).

  7. Note that detailed information on plattforms and similar may take considerable effort on behalf of the user—and are not always necessary. It is often the case that an experienced developer, who happens to be a user, can rule out the value of some additional information aforehand. It is often better to give basic information about the system, with the indication that more information will be provided upon request—in particular, as the user cannot reasonably forsee all information that can be relevant.


    Side-note:

    If you always want particular informations, regardless of the circumstances, write code to check for it yourself. The user can then be given instructions along the line “Please always append the output of [application name] –environment.” (or similar).


Attitude towards bugs

  1. Be thankful for all input. (Yes, I said that already.) All products have bugs, and it is in your best interest to know of as many as possible (in your product). You may be happier not knowing, not being bothered by complaints, or with your head in the sand; but you will be better of knowing. Foster a mentality of “Every bug report is a gift.”, and your software will be better off for it.

  2. Do not reclassify bugs for better appearances: I have repeatedly seen major revisions of e.g. bug severities when a dead-line approaches. Typically, the organisation has some rules about the number of critical, severe, major bugs that are allowed before a release can be made—for a good reason. When these numbers are not reached in time, then the release should be delayed, in order to ensure a sufficient quality of product, to avoid excessive bug fixing post-release, not to ruin the customers opinion, etc. However, in reality, a game of “How can we re-classify this bug?” starts, where the bugs are deliberately moved into the wrong categories to enable the release to go ahead as planned.

    This should not be confused with a limited correct re-classification that can validly occur at any time. Warning signs for the wrong kind include negotiations about severities with stake-holders (including giving promises in return for being allowed to re-classify), the opinions of stake-holders or developers being over-ruled without logical and factual arguments, decisions made by managers and project managers who cannot truly judge the situation and stand to benefit from the short-term keeping of the deadline, and obviously the sheer quantity of bugs involved.


    Side-note:

    But should there not be some amount of lee-way and discretion here? Getting the product out on time is important too!

    True. That discretion, however, should be built in into the rules, so that cheating the system is not needed, and the bugs can retain their correct status. If the rules are too inflexible, then lobby to have the rules changed in an orderly manner. Also beware that the rules, as they are, already may have considerable flexibility, and that they are now being broken after this flexibility has already been used up. Consider e.g. a rule made according to a reasoning of “Ideally, we should have no severe bugs. This, however, may be too restrictive, so we give [someone] the discretionary power to let through one severe (known) bug per 50.000 lines of code, with an upper limit of three severe bugs.”; and assume a situation where a program has four severe bugs: As likely as not, someone will start to complain, “It is just one bug extra, the rules should have some leeway!”—completely defeating purpose of both the rule and the original leeway.

    (In addition, my personal opinion is that deadlines are changeable, scope of product and features is changeable, but quality concerns are not. The opposite is short-sightened and, often, unethical.)


  3. Do not throw out old bugs reports in a blanket manner when moving on to a new version—even if a substantial re-write! These bugs may still be present for several reasons, including that the offending code may still be present (alleged re-writes often contain considerable copying...). Even if the code is changed, similar algorithms, patterns, and other solutions may be present and have re-introduced the bug. The bug may be a result of another bug in a third-party library. Etc.

    Correspondingly, each and every (open) bug report present for the old version that could conceivably, even in a thousand-to-one shot, still be present, should be explicitly tested. Ideally, serious closed bugs should be checked too (unfortunately, this will almost always be unrealistic).

  4. Do not scorn minor or “trivial” bugs: Not only should every bug be at least considered for fixing (although some may be eventually be left out), but a great occurence of many minor bugs can be warning sign of an underlying problem. It is, in particular, possible that an apparent minor bug is just one minor manifestation of a major one—even a show-stopper.

  5. Do not refuse treatement with the view that the bug occured in an older version, without having explicitly checked whether it applies in newer versions.

  6. Note that restrictions to e.g. plattforms and versions need not indicate the true scope. If the user reports that V1.3 on Windows NT shows a certain behaviour, then it is not allowed to automatically conclude that the problem only exists there. In particular, adding to this error by ruling that the plattform or version is so rare that the bug can be neglected is inexcusable. Until otherwise proven, assume that a bug is present everywhere.

  7. Beware of the impact of different plattforms, libraries, etc., and do not throw a bug report out with the claim that you cannot reproduce it. Unless the bug is trivial or the user gives signs of not having done his homework, take the bug sufficiently seriously to at least think possible causes through, have a look at the source code, etc.

    If this turns up nothing, then ask the user for more details to find out what you are doing differently. If, and only if, he cannot reproduce it anymore, then mark the report as invalid.

    Beware, in particular, of misconfigurations on your side: At E4, I originally had a bug report for a webpage rejected, because the internal QA team could not reproduce the JavaScript error that I reported. As it later turned out, the nit-wits actually had the display of JavaScript errors turned off in their Internet Explorers... (I note that these people tested more or less exclusively through access over the web and using IE. Not activating these error messages was about as professional as a near-sighted health inspector going about his work without glasses/contacts.)

  8. Similarly, beware of sporadic bugs and heisenbugs. That a bug cannot be reliably reproduced, does not automatically mean it is not there. Notably, if a bug occurs only occassionally, e.g., due to a threading issue, then chances are good that other bugs will be caused by a similar mechanism—one, possibly, the once in a life-time freak incident that costs your employer two million dollars.

  9. Not every bug must be fixed (although this is certainly the ideal), and sometimes circumstances (lack of time, money, developers, control over third party products, backwards compatibility, ...) can make a bug fix impossible even in the long term; however, no bug must ever be swept under the carpet.

Working with bugs

A few general semi-related notes that will likely be moved elsewhere later (possibly to a page dealing with debugging):

  1. If a bug (pertaining to logic) is verified, always write a test case for it. This allows to automatically verify that the solution is correct, that the bug does not re-occur later and is not present in other versions/on other plattforms, and may help in detecting similar bugs. In fact, in many cases in can make sense to write a test case immediately, even based on just a report. Notably, more test cases is almost always better.

  2. When many bugs come from one single method/class/module, consider a complete re-write: Where there is more, there will be many.

  3. Read up on debugging techniques and experiences elsewhere, e.g. http://c2.com/cgi/wiki?HeroicDebugginge.