How to create a Minimal, Complete, Verifiable Example

Anything related to Keen Modding.
Post Reply
User avatar
Fleexy
Site Admin
Posts: 490
Joined: Fri Dec 12, 2008 1:33 am
Location: Bloogton Tower

How to create a Minimal, Complete, Verifiable Example

Post by Fleexy »

Every so often, a bug report comes in for a program I wrote. The people sending these reports want to help me improve the program - that's why they took the time to inform me of the problem - but sometimes the reports simply don't contain enough information to identify or reproduce the issue. Now, as a StackOverflow contributor, I review a few downvoted questions from new users every day and what I've found is that what makes downvoted questions subpar is that they're missing a Minimal, Complete, Verifiable Example, or MCVE for short. An MCVE is a necessity for a "please debug my code" question on Internet sites, but I believe it's also an excellent tool for bug reports, problem solving in general, and even education.

To help me and other mod/tool creators quickly find and fix problems, I suggest bug reporters create an MCVE. It will take a little longer than saying "it doesn't work!", but it will save lots of debugging time, make me want to help you, and might even show you how to work around the problem before a fix is produced. But before you can create an MCVE, it helps to know what it actually is:

Minimal means that you've boiled the problem down to something that can be reproduced with a minimum of fiddling around with setup. In programming, this means stripping out working/irrelevant code. In bug reports, it means identifying the simplest sequence of actions or a specific set of relevant files that causes the problem to appear. Take a bit of time to eliminate unrelated steps; additional actions in unrelated sections of the program start wild goose chases. "Minimal" does not mean, however, than you elide important information from the report or send a single terse sentence. That leads me to...

Complete means that you describe exactly what happened. Did the program crash? If so, please include the error message or the log file. Did it corrupt your data? Send the original files and the mangled version. (I might even help you recover your changes.) Did it do something unexpected? Tell me what you hoped would happen and what actually happened. If after completing the "minimal" part of the investigation you discover that the problem only occurs in a specific mode or configuration, by all means send along the config files. Make sure to provide the program name and version! It never hurts to include your OS details as well (e.g. Windows 7 x64).

Verifiable means that your steps to reproduce the problem actually result in the issue appearing. If your report contains the word "sometimes", that's a good indication you need to revisit the "minimal" part of MCVE. If your report states that a feature simply doesn't work, please send the files you're using it on - it's likely that the tool creator tested the feature at least once with some set of files. If the tool you're using is very extensible, you might try disabling third-party components to see if the problem persists. Of course, in some environments (e.g. real-mode DOS), "sometimes" is unavoidable thanks to the style of memory management that's in play. In that case, try to find a sequence of actions that most often reproduces the problem, or take a memory dump (if you can) when the problem appears.

Example means that you demonstrate the problem. Whether that's through textual descriptions, a series of screenshots, or a video, it's great. Thanks for taking the time to report an issue!

I understand that not everybody has the time, skills, or inclination to create an MCVE for problems they encounter. A vague bug report is better than no report as long as some version of the problem can be reproduced after some tinkering. If the above process is a bit too bureaucratic for you, hop on IRC or Discord or another live communication medium. That way, I or the tool/mod creator can walk you through the collection of relevant information.

Here's to more stable tools and mods,

Fleexy

This article is accessible at the shortlink http://tiny.cc/km-mcve
levellass
Posts: 3001
Joined: Wed Oct 11, 2006 12:03 pm
Location: Ngaruawahia New Zealand

Post by levellass »

I hear you. So many times I get things like 'In level 3 the game crashed.' Nice, when, where, what was the error message? Every playtester should read this.


Also, I hate bugs that aren't verifiable, the kind that seem to pop up randomly.
allennicol
Posts: 1
Joined: Wed Apr 05, 2017 7:51 am

Re: How to create a Minimal, Complete, Verifiable Example

Post by allennicol »

thanks
Post Reply