Hi there
I have a problem. We use the Matlab testing framework to analyze our codebase. To track the results in our CI system TeamCity we use the TAP-format. Here we have the following problem:
If a test includes a TestClassSetup section, the TAP results show up only at the end, and not during the exection. This results in a few issues for us: – Timestamps might not be correct – If informative output is given per test-case, it is not shown together with the assertion statment.
We use the following (simplified) snippet to identify out TestSuite and execute it:
testSuite = matlab.unittest.TestSuite.fromFolder('.'); runner = matlab.unittest.TestRunner.withNoPlugins(); runner.addPlugin(matlab.unittest.plugins.TAPPlugin.producingOriginalFormat()); results = runner.run(testSuite);
With the following two classes the issue is reproducible (the content is of course made up & meaningless…):
classdef SomeTest < matlab.unittest.TestCase properties (TestParameter) param = {1, 2}; param2 = {1, 2}; end methods (TestClassSetup) function someSetup(testCase) pause(0.1); end end methods (Test) function testMethod(self, param, param2) fprintf('I''m here, with the params: %f/%f\n', param, param2); pause(0.1); self.assertGreaterThan(param, param2); end end end classdef SomeOtherTest < matlab.unittest.TestCase properties (TestParameter) param = {1, 2}; param2 = {1, 2}; end methods (Test) function testMethod(self, param, param2) fprintf('I''m here, with the params: %f/%f\n', param, param2); pause(0.1); self.assertGreaterThan(param, param2); end end end
If you copy all three files into one folder, and execute the runner, you'll see the output (assertions are simplified):
1..8 I'm here, with the params: 1.000000/1.000000 not ok 1 - SomeOtherTest/testMethod(param=1,param2=1) # ================================================================================ # Assertion failed in SomeOtherTest/testMethod(param=1,param2=1) and it did not run to completion. # ================================================================================ # I'm here, with the params: 1.000000/2.000000 not ok 2 - SomeOtherTest/testMethod(param=1,param2=2) # ================================================================================ # Assertion failed in SomeOtherTest/testMethod(param=1,param2=2) and it did not run to completion. # ================================================================================ # I'm here, with the params: 2.000000/1.000000 ok 3 - SomeOtherTest/testMethod(param=2,param2=1) I'm here, with the params: 2.000000/2.000000 not ok 4 - SomeOtherTest/testMethod(param=2,param2=2) # ================================================================================ # Assertion failed in SomeOtherTest/testMethod(param=2,param2=2) and it did not run to completion. # ================================================================================ # I'm here, with the params: 1.000000/1.000000 I'm here, with the params: 1.000000/2.000000 I'm here, with the params: 2.000000/1.000000 I'm here, with the params: 2.000000/2.000000 not ok 5 - SomeTest/testMethod(param=1,param2=1) # ================================================================================ # Assertion failed in SomeTest/testMethod(param=1,param2=1) and it did not run to completion. # ================================================================================ # not ok 6 - SomeTest/testMethod(param=1,param2=2) # ================================================================================ # Assertion failed in SomeTest/testMethod(param=1,param2=2) and it did not run to completion. # ================================================================================ # ok 7 - SomeTest/testMethod(param=2,param2=1) not ok 8 - SomeTest/testMethod(param=2,param2=2) # ================================================================================ # Assertion failed in SomeTest/testMethod(param=2,param2=2) and it did not run to completion. # ================================================================================
What I would expect is that also in the second case the Assertion statements (and the ok / not ok TAP flags) are aligned with the fprintf-statements.
Has anyone an idea?
Best Answer