Can I set up a unit test to run many times and return percentage success?
I'm new to unit testing and have been scouring the web trying to figure out how to automate my unit tests even more. I am creating a registration scheme and what I want to do is validate it with many hard disk serial numbers, application numbers, etc. I want to generate a registration key and then test it to make sure it decodes correctly. I am looking to automate the run of a test (with built-in Visual Studio test environment) with different inputs, and after thousands of runs, figure out what% of the tests were successful and unsuccessful. Is it possible? Below is my test method:
[TestMethod]
public void GeneratingValidKeyTest()
{
int numReadDevices;
string appNum = "123";
string hddSerial = "1234567890";
string numDevices = "12";
string regNumber = Registration.GenerateKey(appNum, numDevices, hddSerial);
Assert.IsTrue(Registration.CheckKey(regNumber, appNum, out numReadDevices, hddSerial), "Generated key does not pass check.");
Assert.AreEqual(int.Parse(numDevices), numReadDevices,"Number of registered devices does not match requested number");
}
source to share
If you are using NUnit you can customize the ValueSources series to feed to your method.
If you have a separate source of values for appNum, hddSerial, and numDevices, you get the number of checks for appNum * hddSerial * numDevices.
However, you shouldn't try to find the percentage of tests. The goal of unit testing is to ensure that all test cases pass.
To take Max's example and make it ValueSources:
[Test]
public void DivideTest([ValueSource("AppNums")]string appNum, [ValueSource("Serials")]string hddSerial, [ValueSource("NumDevices")]string numDevices)
{
string regNumber = Registration.GenerateKey(appNum, numDevices, hddSerial);
Assert.IsTrue(Registration.CheckKey(regNumber, appNum, out numReadDevices, hddSerial), "Generated key does not pass check.");
Assert.AreEqual(int.Parse(numDevices), numReadDevices,"Number of registered devices does not match requested number");
}
static object[] AppNums =
{
"1", "2", "3", "4", "5", "6", "7"
};
static object[] Serials =
{
"EXAMPLE", "ANOTHER", "AND AGAIN"
};
static object[] NumDevices =
{
"1", "2", "3", "4", "5", "6", "7"
};
source to share
Personally, I find very little value in the "throw a lot of random data on the method and make sure everything works" method.
If you give 500 different serial numbers to a method and they all work, that's great. But what specific scenarios are they testing in your code? If you cannot answer this question, you are probably duplicating test scripts and, more importantly, scripts are missing .
Instead of throwing test cases at the wall and seeing which sticks, analyze your code and identify critical criteria for success and failure, as well as tests for ships that meet those criteria. This has the side benefit of making your tests more verbose and giving your team members a better idea of what the code should be doing just by reading the test names. Instead, GeneratingValidKeyTest
your tests should be named so that they describe what they are testing.
As an example, let's say you are building a calculator. With your approach, you would throw in a ton of extra cases - 1 + 1, 1 + 3, 5 + 30, etc. But most likely you will miss 1+ Int32.MaxValue
. Or maybe you won't try adding negative numbers. Or check what happens if the input is not a valid number. Etc.
Good tests make you think of all these scenarios as you write them.
source to share
In NUnit it will look like
[Test, TestCaseSource("DivideCases")]
public void DivideTest(string appNum, string hddSerial, string hddSerial)
{
string regNumber = Registration.GenerateKey(appNum, numDevices, hddSerial);
Assert.IsTrue(Registration.CheckKey(regNumber, appNum, out numReadDevices, hddSerial), "Generated key does not pass check.");
Assert.AreEqual(int.Parse(numDevices), numReadDevices,"Number of registered devices does not match requested number");
}
static object[] DivideCases =
{
new object[] { "123", "3", "4" },
new object[] { "12223", "35", "54" },
new object[] { "12123123", "23", "14" }
};
source to share
You can use Microsoft Unit Test Framework and force it to read test data from data source. The advantage of using MSTest is that it will work in Express Editions of Visual Studio.
However, you will not get the error rate and I agree with @DanielMann, instead you should make sure that your tests cover all possibilities and that they all pass.
So, given that you did this and now you have a list of cases to test, you can use the below code. He uses DataSourceAttribute
:
[TestClass]
public class RegistrationTests
{
public TestContext TestContext { get; set; }
[TestMethod]
[DataSource(
"System.Data.OleDb",
@"Provider=Microsoft.Jet.OLEDB.4.0;Data Source=""C:\MySolution\Serial numbers.mdb""",
"Serials",
DataAccessMethod.Sequential
)]
public void GeneratingValidKeyTest()
{
// Arrange
int numReadDevices;
var appNum = TestContext.DataRow["appNum"].ToString();
var hddSerial = TestContext.DataRow["hddSerial"].ToString();
var numDevices = TestContext.DataRow["numDevices"].ToString();
// Act
var regNumber = Registration.GenerateKey(appNum, numDevices, hddSerial);
// Assert
Assert.IsTrue(
Registration.CheckKey(regNumber, appNum, out numReadDevices, hddSerial),
"Generated key does not pass check."
);
Assert.AreEqual(
int.Parse(numDevices), numReadDevices,
"Number of registered devices does not match requested number"
);
}
}
In the test explorer window, you will get the same result (failed tests are shown first):
Test Name: GeneratingValidKeyTest Test FullName: MySolution.UnitTest.RegistrationTests.GeneratingValidKeyTest Test Source: C:\MySolution\MySolution.UnitTest\RegistrationTests.cs : line 15 Test Outcome: Failed Test Duration: 0:00:00,017832 Result1 Name: GeneratingValidKeyTest (Data Row 1) Result1 Outcome: Failed Result1 Duration: 0:00:00,0082728 Result1 StackTrace: at MySolution.UnitTest.RegistrationTests.GeneratingValidKeyTest() in C:\MySolution\MySolution.UnitTest\RegistrationTests.cs:line 27 Result1 Message: Assert.AreEqual failed. Expected:<11>. Actual:<12>. Number of registered devices does not match requested number Result3 Name: GeneratingValidKeyTest (Data Row 0) Result3 Outcome: Passed Result3 Duration: 0:00:00,0089332 Result3 StackTrace: Result3 Message:
source to share