MATLAB: VerifyError in a script based unit test

MATLABtestcaseunit testunittestverifyerror

Hi, my question is regarding the script-based unit testing framework. I would like to use the verifyError functionality and am struggling to generate a suitable testCase object that I can use in my script-based unit test.

For example, the run suite is called with:

import matlab.unittest.TestSuite;
import matlab.unittest.TestCase;
import matlab.unittest.TestRunner;
import matlab.unittest.plugins.TAPPlugin;
import matlab.unittest.plugins.ToFile;
suite = testsuite('unitTests','IncludingSubpackages',true);
runner = TestRunner.withTextOutput();
tapFile = fullfile(getenv('WORKSPACE'), sprintf('%s_results.tap', 'test'));
runner.addPlugin(TAPPlugin.producingOriginalFormat(ToFile(tapFile), 'LoggingLevel', 3, 'OutputDetail', 3));
results = runner.run(suite);

In the folder unitTests I would have the following script based unit test:

% Test to evaluate foo


% Initial Set-Up


a = 1;
%% Test 1. Verify Error Condition


% How can I get a test case here to call either:


% verifyError(testCase, @()foo('badArguement'), 'SomeError');


% or 


% testCase.verifyError(@()foo('badArguement'), 'SomeError');

It seems that the following is not valid, as this instantiation is only for interactive use.

% Test to evaluate foo
% Initial Set-Up
a = 1;
testCase = matlab.unittest.TestCase.forInteractiveUse;
%% Test 1. Verify Error Condition
% How can I get a test case here to call either:
% verifyError(testCase, @()foo('badArguement'), 'SomeError');
% or 
% testCase.verifyError(@()foo('badArguement'), 'SomeError');

% Test to evaluate foo
% Initial Set-Up
a = 1;
%% Test 1. Verify Error Condition
testCase = matlab.unittest.TestCase.forInteractiveUse;
% How can I get a test case here to call either:
% verifyError(testCase, @()foo('badArguement'), 'SomeError');
% or 
% testCase.verifyError(@()foo('badArguement'), 'SomeError');

Best Answer

Hello,

Since there is no testCase provided for script based tests, they don't support the Qualification API, and that definitely is one reason someone might want to use function based tests instead. Can you describe some of your reasoning for using script based tests? Do you prefer them over functions or classes? This may be helpful to determine whether we should look into providing the qualification api to script based tests or if you are happy to use functions.

Thanks,

Andy

Related Solutions

MATLAB: How to pass values from inside a TestRunner run

You can use a TestRunnerPlugin to store data from the test run. I've included some example code below to show how this might work. The basic outline is as follows:

The test logs information using a Diagnostic subclass and TestCase.log
A plugin listens to the DiagnosticLogged event and stores the logged data on itself
After running the test, you can then access the data on the plugin

Example test code:

classdef MyTest < matlab.unittest.TestCase
    properties (ClassSetupParameter)
        idx = num2cell(1:3);
    end
    properties
        MyData;
    end
    methods(TestClassSetup)
        function createData(testCase,idx)
            testCase.MyData = idx;
            testCase.log(4, DataDiagnostic(testCase.MyData));
        end
    end
    methods(Test)
        function a(testCase)
        end
    end
end

Plugin code:

classdef MyPlugin < matlab.unittest.plugins.TestRunnerPlugin
    properties(SetAccess=private)
        Data = {};
    end
    methods(Access=protected)
        function testCase = createTestClassInstance(plugin, pluginData)
            testCase = createTestClassInstance@matlab.unittest.plugins.TestRunnerPlugin(plugin, pluginData);
            testCase.addlistener('DiagnosticLogged', @plugin.logDiagnostic);
        end
    end
    methods(Access=private)
        function logDiagnostic(plugin, src, evd)
            if isa(evd.Diagnostic, 'DataDiagnostic')
                plugin.Data{end+1} = evd.Diagnostic.Data;
            end
        end
    end
end

Diagnostic class code:

classdef DataDiagnostic < matlab.unittest.diagnostics.Diagnostic
    properties(SetAccess=immutable)
        Data;
    end
    methods
        function diag = DataDiagnostic(data)
            diag.Data = data;
        end
        function diagnose(~)
        end
    end
end

Usage:

>> suite = testsuite('MyTest');
>> runner = matlab.unittest.TestRunner.withTextOutput;
>> plugin = MyPlugin;
>> runner.addPlugin(plugin);
>> runner.run(suite);
Running MyTest
...
Done MyTest
__________
>> plugin.Data
ans =
    1×3 cell array
      {[1]}    {[2]}    {[3]}

References:

MATLAB: How to assign the Test Diagnostic provided by the Unit Testing framework in the Command Window to a variable

Hi Malek,

You can get programmatic access to this information by creating and installing a TestRunnerPlugin for the test run. This plugin can then listen to qualification failed events (like verification failures, assertion failure, etc) and this diagnostic information is contained in the TestDiagnosticResult property of the event data as a cell array of strings.

Here is a quick example that save all of the test diagnostics for failing verifications in test methods. Note this could expand to cover other cases like assertions, assumptions, etc and other scopes like TestClassSetup, but this might help get you started.

classdef TestDiagnosticSaverPlugin < matlab.unittest.plugins.TestRunnerPlugin
      properties
          TestDiagnosticData
      end
      methods (Access = protected)
          function runTestSuite(plugin, pluginData) 
              plugin.TestDiagnosticData = []; % Reinitialize for each test run
              runTestSuite@...
                  matlab.unittest.plugins.TestRunnerPlugin(plugin, pluginData);
          end
          function testCase = createTestMethodInstance(plugin, pluginData)
              testCase = createTestMethodInstance@...
                  matlab.unittest.plugins.TestRunnerPlugin(plugin, pluginData);
              testName = pluginData.Name;
              testCase.addlistener('VerificationFailed', ...
                @(~,event)plugin.storeTestDiagnosticsData(event,pluginData.Name));
          end
      end
      methods (Access = private)
          function storeTestDiagnosticsData(plugin,eventData,name,failureType)
              s.Name = {name};
              s.TestDiagnostics = eventData.TestDiagnosticResult;
              plugin.TestDiagnosticData = [plugin.TestDiagnosticData; struct2table(s)];
          end
      end
  end

With the above plugin you could then install it on a runner:

>> import matlab.unittest.TestRunner;
>> import matlab.unittest.TestSuite;
>> suite = TestSuite.fromFile('mbd_ZaehlerUnitTest.m')
>> runner = TestRunner.withTextOutput;
>> plugin = TestDiagnosticSaverPlugin;
>> runner.addPlugin(plugin);
>> runner.run(suite)
>> plugin.TestDiagnosticData

Hope that helps! Here are a few links you might find interesting to help you do this:

Writing plugins overview

Example of creating a custom plugin

QualificationEventData

Best Answer

Related Solutions

MATLAB: How to pass values from inside a TestRunner run

MATLAB: How to assign the Test Diagnostic provided by the Unit Testing framework in the Command Window to a variable

Related Question