Code verification and unit testing
As soon as the list of codes and functions that you write grows, you will need to have a unified way of ensuring all your functions work appropriately, regardless of the potential future internal changes that are made to the functions. This is called unit testing: a software development process in which the smallest testable parts of an application, called units, are individually and independently scrutinized for proper operation. Unit testing can be done manually, but if you have a long list of functions (which you most often have), you’d want to automate the testing process.
The grand goal in unit testing is to reduce the risk of encountering potential problems when running the code in the smallest possible units of the code. This means,
- ensuring the code has the correct behavior when given the proper input data.
- ensuring the code robustness to exceptions and invalid input data, meaning that it does not crash when it reaches unexpected situations during the code execution and gracefully handles the error, without interruption.
Because of the goals for which the unit tests are designed, they are mostly written and used by the developers of the code.
Unit testing approaches in MATLAB
There are many ways to write tests for codes. Now, if you asked each software developer to write a unit-test for specific software, each would likely come up with their own set of rules and tests of the software. You will end up with many tests, that will generally only be usable by the developer that wrote the tests.
A more reasonable approach would be to have a unified method of unit-testing, i.e., a unit-test framework, available to all programmers for the purpose of verifying their codes. A unit-test framework provides consistency for how the unit tests for your project are written. There are many test frameworks to choose from for just about any language you want to program with, including MATLAB.
Just like the case for the choice of programming language, almost every programmer has a strong opinion their own as to which test-framework is the best. Research what’s out there and use the one that meets the needs of your organization.
The framework will provide a consistent testing structure to create maintainable tests with reproducible results. From a product-quality and business point of view, those are the most valuable reasons to use a unit-test framework. When you write software, you should also think of a quick and simple way to develop and verify your logic in isolation. Once you make sure you have it working solidly by itself, then you can proceed to integrate it into the larger project solutions with great confidence.
Since MATLAB is a organizational effort (as opposed to Python, for example), it has its own unified set of rules and conventions for unit testing. For more information, see this page.
Script-based unit-tests
The simplest method of unit-testing in MATLAB is to write tests in scripts that verify the accuracy of the output or the expected behavior of MATLAB scripts, functions, or classes that you write. For example, you can use the assert function to test whether actual output values match the expected values. Or you can test whether the output variables have the correct size and type. To run your test scripts you will have to use MATLAB’s built-in function runtests.
Example
To see how it works, let’s look at an example from MATLAB manual. Suppose you want to write a MATLAB function, rightTri.m, in your current MATLAB® folde that takes lengths of two sides of a triangle as input and returns the three angles of the corresponding right triangle. The input sides are the two shorter edges of the triangle, not the hypotenuse.
function angles = rightTri(sides)
A = atand(sides(1)/sides(2));
B = atand(sides(2)/sides(1));
hypotenuse = sides(1)/sind(A);
C = asind(hypotenuse*sind(A)/sides(1));
angles = [A B C];
end
Create test-script
Now, in your working folder, create a new script, rightTriTest.m. Each unit test checks a different output of the rightTri function. A test-script must adhere to the following conventions,
- The name of the script file must start or end with the word ‘test’, which is case-insensitive.
- Place each unit test into a separate section of the script file. Each section begins with two percent signs (%%), and the text that follows on the same line becomes the name of the test element. If no text follows the %%, MATLAB assigns a name to the test. If MATLAB encounters a test failure, it still runs remaining tests.
- In a test script, the shared variable section consists of any code that appears before the first explicit code section (the first line beginning with %%). Tests share the variables that you define in this section. Within a test, you can modify the values of these variables. However, in subsequent tests, the value is reset to the value defined in the shared variables section.
- In the shared variables section (first code section), define any preconditions necessary for your tests. If the inputs or outputs do not meet this precondition, MATLAB does not run any of the tests. MATLAB marks the tests as failed and incomplete.
- When a script is run as a test, variables defined in one test are not accessible within other tests unless they are defined in the shared variables section (first code section). Similarly, variables defined in other workspaces are not accessible to the tests.
- If the script file does not include any code sections, MATLAB generates a single test element from the full contents of the script file. The name of the test element is the same as the script file name. In this case, if MATLAB encounters a failed test, it halts execution of the entire script.
Now, in rightTriTest.m, write four tests to test the output of rightTri.m. Use the assert function to test the different conditions. In the shared variables section, define four triangle geometries and define a precondition that the rightTri function returns a right triangle.
% test triangles
tri = [7 9];
triIso = [4 4];
tri306090 = [2 2*sqrt(3)];
triSkewed = [1 1500];
% preconditions
angles = rightTri(tri);
assert(angles(3) == 90,'Fundamental problem: rightTri not producing right triangle')
%% Test 1: sum of angles
angles = rightTri(tri);
assert(sum(angles) == 180)
angles = rightTri(triIso);
assert(sum(angles) == 180)
angles = rightTri(tri306090);
assert(sum(angles) == 180)
angles = rightTri(triSkewed);
assert(sum(angles) == 180)
%% Test 2: isosceles triangles
angles = rightTri(triIso);
assert(angles(1) == 45)
assert(angles(1) == angles(2))
%% Test 3: 30-60-90 triangle
angles = rightTri(tri306090);
assert(angles(1) == 30)
assert(angles(2) == 60)
assert(angles(3) == 90)
%% Test 4: Small angle approximation
angles = rightTri(triSkewed);
smallAngle = (pi/180)*angles(1); % radians
approx = sin(smallAngle);
assert(approx == smallAngle, 'Problem with small angle approximation')
In the above,
- Test 1 tests the summation of the triangle angles. If the summation is not equal to 180 degrees, assert throws an error.
- Test 2 tests that if two sides are equal, the corresponding angles are equal. If the non-right angles are not both equal to 45 degrees, the assert function throws an error.
- Test 3 tests that if the triangle sides are 1 and sqrt(3), the angles are 30, 60, and 90 degrees. If this condition is not true, assert throws an error.
- Test 4 tests the small-angle approximation. The small-angle approximation states that for small angles the sine of the angle in radians is approximately equal to the angle. If it is not true, assert throws an error.
Run tests
Execute the runtests function to run the four tests in rightTriTest.m. The runtests function executes each test in each code section individually. If Test 1 fails, MATLAB still runs the remaining tests. If you execute rightTriTest as a script instead of by using runtests, MATLAB halts execution of the entire script if it encounters a failed assertion. Additionally, when you run tests using the runtests function, MATLAB provides informative test diagnostics.
result = runtests('rightTriTest');
Running rightTriTest
..
================================================================================
Error occurred in rightTriTest/Test3_30_60_90Triangle and it did not run to completion.
---------
Error ID:
---------
'MATLAB:assertion:failed'
--------------
Error Details:
--------------
Error using rightTriTest (line 31)
Assertion failed.
================================================================================
.
================================================================================
Error occurred in rightTriTest/Test4_SmallAngleApproximation and it did not run to completion.
---------
Error ID:
---------
''
--------------
Error Details:
--------------
Error using rightTriTest (line 39)
Problem with small angle approximation
================================================================================
.
Done rightTriTest
__________
Failure Summary:
Name Failed Incomplete Reason(s)
===========================================================================
rightTriTest/Test3_30_60_90Triangle X X Errored.
---------------------------------------------------------------------------
rightTriTest/Test4_SmallAngleApproximation X X Errored.
The test for the 30-60-90 triangle and the test for the small-angle approximation fail in the comparison of floating-point numbers. Typically, when you compare floating-point values, you specify a tolerance for the comparison. In Test 3 and Test 4, MATLAB throws an error at the failed assertion and does not complete the test. Therefore, the test is marked as both Failed and Incomplete.
To provide diagnostic information (Error Details) that is more informative than ‘Assertion failed’ (Test 3), you can pass a message to the assert function (as in Test 4), or you can also consider using the more-advanced function-based unit tests.
Revise test to use tolerance
Save rightTriTest.m as rightTriTolTest.m, and revise Test 3 and Test 4 to use a tolerance. In Test 3 and Test 4, instead of asserting that the angles are equal to an expected value, assert that the difference between the actual and expected values is less than or equal to a specified tolerance. Define the tolerance in the shared variables section of the test script so it is accessible to both tests.
For script-based unit tests, manually verify that the difference between two values is less than a specified tolerance. If instead you write a function-based unit test, you can access built-in constraints to specify a tolerance when comparing floating-point values.
% test triangles
tri = [7 9];
triIso = [4 4];
tri306090 = [2 2*sqrt(3)];
triSkewed = [1 1500];
% Define an absolute tolerance
tol = 1e-10;
% preconditions
angles = rightTri(tri);
assert(angles(3) == 90,'Fundamental problem: rightTri not producing right triangle')
%% Test 1: sum of angles
angles = rightTri(tri);
assert(sum(angles) == 180)
angles = rightTri(triIso);
assert(sum(angles) == 180)
angles = rightTri(tri306090);
assert(sum(angles) == 180)
angles = rightTri(triSkewed);
assert(sum(angles) == 180)
%% Test 2: isosceles triangles
angles = rightTri(triIso);
assert(angles(1) == 45)
assert(angles(1) == angles(2))
%% Test 3: 30-60-90 triangle
angles = rightTri(tri306090);
assert(abs(angles(1)-30) <= tol)
assert(abs(angles(2)-60) <= tol)
assert(abs(angles(3)-90) <= tol)
%% Test 4: Small angle approximation
angles = rightTri(triSkewed);
smallAngle = (pi/180)*angles(1); % radians
approx = sin(smallAngle);
assert(abs(approx-smallAngle) <= tol, 'Problem with small angle approximation')
Now, rerun the tests,
result = runtests('rightTriTolTest');
Running rightTriTolTest
....
Done rightTriTolTest
__________
All the tests pass.
Create a table of test results.
rt = table(result)
rt =
4x6 table
Name Passed Failed Incomplete Duration Details
_______________________________________________ ______ ______ __________ ________ ____________
'rightTriTolTest/Test1_SumOfAngles' true false false 0.051194 [1x1 struct]
'rightTriTolTest/Test2_IsoscelesTriangles' true false false 0.010316 [1x1 struct]
'rightTriTolTest/Test3_30_60_90Triangle' true false false 0.011041 [1x1 struct]
'rightTriTolTest/Test4_SmallAngleApproximation' true false false 0.010778 [1x1 struct]
Function-based unit-tests
There is another method of unit-testing in MATLAB based on functions (rather than scripts). In this case, your test function is a single MATLAB file that contains a main function and your individual local test-functions. The discussion of function-based unit-tests goes beyond the scope of our goal here, but you can learn more about it here
Class-based unit-tests
There is yet another method of unit-testing in MATLAB based on the concept of classes and objects (rather than function or scripts). As with the case for function-based unit-tests, the discussion of class-based unit-tests goes beyond the scope of our goal here, but you can learn more about it here
Summary: unit testing
Unit testing is a component of test-driven development (TDD), a pragmatic methodology that takes a meticulous approach to build a product by means of continual testing and revision.
Unit testing has a steep learning curve. The programmer or the development-team needs to learn what unit testing is, how to unit-test, what to unit-test and how to use automated software tools to facilitate the process on an on-going basis. The great benefit to unit testing is that the earlier a problem is identified, the fewer compound errors occur. A compound error is one that doesn’t seem to break anything at first but eventually conflicts with something down the line and results in a problem.
There is a lot more to unit testing and the existing MATLAB frameworks for it than what we discussed here. However, covering all those topics would require a dedicated course for unit testing, which is certainly beyond the capacity of this course.