MATLAB: Error stack trace visible with verifyError

MATLABunittest

Hello all,

How can I make my test running silently when using a VerifyError which checks an error is occurring? Now it displays the entire exception trace.

My testcase is defined as:

          function testcaseFHwrong(testCase)
              cd(testCase.foldername);
              try
                  testCase.verifyError(@() LoadFaultTable('Faulttablewrong.xls','SilentMode',1),'readfaulttable:NoFaultTable');
              catch ME
                  disp(ME);
                  testCase.verifyTrue(False);
              end
          end

The function LoadFaultTable will issue an error when being run, and this error is correctly verified by the unit test framework.

The unit test is triggered using:

import matlab.unittest.TestSuite;
import matlab.unittest.TestRunner;
import matlab.unittest.plugins.TAPPlugin;
import matlab.unittest.plugins.ToFile;
suite = TestSuite.fromPackage('test_LIB_SDS_LoadFaultTable');
runner= TestRunner.withTextOutput;
tapFile = 'FHtest.tap';
plugin = TAPPlugin.producingOriginalFormat(ToFile(tapFile));
runner.addPlugin(plugin);
delete(tapFile);
result = runner.run(suite);

The command line shows the error stack trace when running this function. I did not expect this behavior

regards, Han

Best Answer

Hi Hans,

I think that there is still some key ingredient missing. I wrote the following stub for LoadFaultTable:

    function LoadFaultTable(varargin)
    error('readfaulttable:NoFaultTable', 'Something went wrong');

With the following surrounding test:

 classdef ErrorTest < matlab.unittest.TestCase
     properties
         foldername = '.';
     end
     methods(Test)
         function testcaseFHwrong(testCase)
               cd(testCase.foldername);
               try
                   testCase.verifyError(@() LoadFaultTable('Faulttablewrong.xls', ...
                       'SilentMode',1),'readfaulttable:NoFaultTable');
               catch ME
                   disp(ME);
                   testCase.verifyTrue(False);
               end
         end
     end
 end

...and the output looks fine:

 >> runtests
 Running ErrorTest
 .
 Done ErrorTest
 __________
 ans = 
   TestResult with properties:
           Name: 'ErrorTest/testcaseFHwrong'
         Passed: 1
         Failed: 0
     Incomplete: 0
       Duration: 0.011415858
 Totals:
    1 Passed, 0 Failed, 0 Incomplete.
    0.011416 seconds testing time.

I think there is still something else at play here. Is the source code (LoadFaultTable) the thing which is printing the stack trace? Can you show the stack trace? Perhaps the call to CD removes the source and/or test from the path (here the cd is basically a no-op)?

Also, did you put the try-catch in there as a debugging step? Typically you don't need to use a try-catch when using the Throws constraint or verifyError. Taking out the try-catch produces the same expected, empty output when running the test.

Related Solutions

MATLAB: How to assign the Test Diagnostic provided by the Unit Testing framework in the Command Window to a variable

Hi Malek,

You can get programmatic access to this information by creating and installing a TestRunnerPlugin for the test run. This plugin can then listen to qualification failed events (like verification failures, assertion failure, etc) and this diagnostic information is contained in the TestDiagnosticResult property of the event data as a cell array of strings.

Here is a quick example that save all of the test diagnostics for failing verifications in test methods. Note this could expand to cover other cases like assertions, assumptions, etc and other scopes like TestClassSetup, but this might help get you started.

classdef TestDiagnosticSaverPlugin < matlab.unittest.plugins.TestRunnerPlugin
      properties
          TestDiagnosticData
      end
      methods (Access = protected)
          function runTestSuite(plugin, pluginData) 
              plugin.TestDiagnosticData = []; % Reinitialize for each test run
              runTestSuite@...
                  matlab.unittest.plugins.TestRunnerPlugin(plugin, pluginData);
          end
          function testCase = createTestMethodInstance(plugin, pluginData)
              testCase = createTestMethodInstance@...
                  matlab.unittest.plugins.TestRunnerPlugin(plugin, pluginData);
              testName = pluginData.Name;
              testCase.addlistener('VerificationFailed', ...
                @(~,event)plugin.storeTestDiagnosticsData(event,pluginData.Name));
          end
      end
      methods (Access = private)
          function storeTestDiagnosticsData(plugin,eventData,name,failureType)
              s.Name = {name};
              s.TestDiagnostics = eventData.TestDiagnosticResult;
              plugin.TestDiagnosticData = [plugin.TestDiagnosticData; struct2table(s)];
          end
      end
  end

With the above plugin you could then install it on a runner:

>> import matlab.unittest.TestRunner;
>> import matlab.unittest.TestSuite;
>> suite = TestSuite.fromFile('mbd_ZaehlerUnitTest.m')
>> runner = TestRunner.withTextOutput;
>> plugin = TestDiagnosticSaverPlugin;
>> runner.addPlugin(plugin);
>> runner.run(suite)
>> plugin.TestDiagnosticData

Hope that helps! Here are a few links you might find interesting to help you do this:

Writing plugins overview

Example of creating a custom plugin

QualificationEventData

MATLAB: How to pass values from inside a TestRunner run

You can use a TestRunnerPlugin to store data from the test run. I've included some example code below to show how this might work. The basic outline is as follows:

The test logs information using a Diagnostic subclass and TestCase.log
A plugin listens to the DiagnosticLogged event and stores the logged data on itself
After running the test, you can then access the data on the plugin

Example test code:

classdef MyTest < matlab.unittest.TestCase
    properties (ClassSetupParameter)
        idx = num2cell(1:3);
    end
    properties
        MyData;
    end
    methods(TestClassSetup)
        function createData(testCase,idx)
            testCase.MyData = idx;
            testCase.log(4, DataDiagnostic(testCase.MyData));
        end
    end
    methods(Test)
        function a(testCase)
        end
    end
end

Plugin code:

classdef MyPlugin < matlab.unittest.plugins.TestRunnerPlugin
    properties(SetAccess=private)
        Data = {};
    end
    methods(Access=protected)
        function testCase = createTestClassInstance(plugin, pluginData)
            testCase = createTestClassInstance@matlab.unittest.plugins.TestRunnerPlugin(plugin, pluginData);
            testCase.addlistener('DiagnosticLogged', @plugin.logDiagnostic);
        end
    end
    methods(Access=private)
        function logDiagnostic(plugin, src, evd)
            if isa(evd.Diagnostic, 'DataDiagnostic')
                plugin.Data{end+1} = evd.Diagnostic.Data;
            end
        end
    end
end

Diagnostic class code:

classdef DataDiagnostic < matlab.unittest.diagnostics.Diagnostic
    properties(SetAccess=immutable)
        Data;
    end
    methods
        function diag = DataDiagnostic(data)
            diag.Data = data;
        end
        function diagnose(~)
        end
    end
end

Usage:

>> suite = testsuite('MyTest');
>> runner = matlab.unittest.TestRunner.withTextOutput;
>> plugin = MyPlugin;
>> runner.addPlugin(plugin);
>> runner.run(suite);
Running MyTest
...
Done MyTest
__________
>> plugin.Data
ans =
    1×3 cell array
      {[1]}    {[2]}    {[3]}

References:

Best Answer

Related Solutions

MATLAB: How to assign the Test Diagnostic provided by the Unit Testing framework in the Command Window to a variable

MATLAB: How to pass values from inside a TestRunner run

Related Question