Forum SquashTM

hello,

With SquashTM 1.14.2 and SquashTA 1.9, I have a strange comportment: SquashTA does not execute the tests in the order requested by SquashTM.

The testsuite.json file SquahTM generates and provides to Jenkins job uses the same order than the one shown in SquashTM user interface.
But SquashTA does not uses this order.

Here infos picked from Jenkins logfile job:

[ul]
[li]testsuite.json file: [code]
{"test": [
{"param":{"TC_REFERENCE":"1-PREQ-ENV-SYST-003"}, "script":"10-Systeme/1-PREQ-ENV-SYST-003.ta", "id":"1374"},
{"param":{"TC_REFERENCE":"1-PREQ-ENV-PGSQL-007"}, "script":"11-SGDB/postgreSQL/1-PREQ-ENV-PGSQL-007.ta", "id":"1375"},
{"param":{"TC_REFERENCE":"3-CFMT-PGSQL-005"}, "script":"11-SGDB/postgreSQL/3-CFMT-PGSQL-005.ta", "id":"1376"},
{"param":{"TC_REFERENCE":"1-PREQ-ENV-SYST-001"}, "script":"10-Systeme/1-PREQ-ENV-SYST-001.ta", "id":"1377"},
{"param":{"TC_REFERENCE":"3-CFMT-PGSQL-006"}, "script":"11-SGDB/postgreSQL/3-CFMT-PGSQL-006.ta", "id":"1378"},
{"param":{"TC_REFERENCE":"1-PREQ-ENV-SYST-002"}, "script":"10-Systeme/1-PREQ-ENV-SYST-002.ta", "id":"1379"},
{"param":{"TC_REFERENCE":"1-PREQ-ENV-ORA-001"}, "script":"11-SGDB/Oracle/1-PREQ-ENV-PGSQL-001.ta", "id":"1380"},
{"param":{"TC_REFERENCE":"3-CFMT-SYST-001"}, "script":"10-Systeme/3-CFMT-SYST-001.ta", "id":"1381"},
{"param":{"TC_REFERENCE":"1-PREQ-ENV-SYST-005"}, "script":"10-Systeme/1-PREQ-ENV-SYST-005.ta", "id":"1382"},
{"param":{"TC_REFERENCE":"1-PREQ-ENV-SYST-008"}, "script":"10-Systeme/1-PREQ-ENV-SYST-008.ta", "id":"1383"},
{"param":{"TC_REFERENCE":"1-PREQ-ENV-SYST-006"}, "script":"10-Systeme/1-PREQ-ENV-SYST-006.ta", "id":"1384"},
{"param":{"TC_REFERENCE":"3-CFMT-SYST-009"}, "script":"10-Systeme/3-CFMT-SYST-009.ta", "id":"1385"},
{"param":{"TC_REFERENCE":"3-CFMT-APA-005"}, "script":"12-Serveur_HTTP/Apache/3-CFMT-APA-005.ta", "id":"1386"}
]}
[/code][/li]

[li] SquashTA uses this order: [code]
[INFO] Beginning execution of test 1-PREQ-ENV-SYST-003.ta
[INFO] Beginning execution of test 1-PREQ-ENV-SYST-001.ta
[INFO] Beginning execution of test 1-PREQ-ENV-SYST-002.ta
[INFO] Beginning execution of test 3-CFMT-SYST-001.ta
[INFO] Beginning execution of test 1-PREQ-ENV-SYST-005.ta
[INFO] Beginning execution of test 1-PREQ-ENV-SYST-008.ta
[INFO] Beginning execution of test 1-PREQ-ENV-SYST-006.ta
[INFO] Beginning execution of test 3-CFMT-SYST-009.ta
[INFO] Beginning execution of test 1-PREQ-ENV-PGSQL-007.ta
[INFO] Beginning execution of test 3-CFMT-PGSQL-005.ta
[INFO] Beginning execution of test 3-CFMT-PGSQL-006.ta
[INFO] Beginning execution of test 1-PREQ-ENV-PGSQL-001.ta
[INFO] Beginning execution of test 3-CFMT-APA-005.ta
[/code] [/li]

[/ul]

Hi,
tests are modeled as standalone test cases, without dependencies between them. As such, there is no guarantee of order between individual test scripts.
The ordering rules are as follows:
[ol]
[li]if you don't use ecosystems (ie: only one level of tests, all under src/squashTA/tests - see documentation):
[ol]
[li]default ecosystem setup script (named setup.ta)[/li]
[li]all remaining tests scripts except for the default ecosystem teardown script[/li]
[li]default ecosystem teardown script (teardown.ta)[/li]
[/ol]
[/li]
[li]if you use ecosystems:[/li]
[ul]
[li]all tests of a given ecosystem are executed when this ecosystem is executed[/li]
[li]there is no guarantee on ecosystem execution order[/li]
[li]each ecosystem is executed in the following order:
[ol]
[li]ecosystem setup script (setup.ta)[/li]
[li]all other scripts except teardown script, in any order[/li]
[li]ecosystem teardown script (teardown.ta)[/li]
[/ol]
[/li]
[/ul]
[/ol]

Hello,

Thanks for your detailed reply.

I understand the "standalone test cases" model that explain this out-of-order execution by SquashTA.
However, since test cases can be fully ordered in SquashTM, it sounds somewhat incoherent with SquashTA behaviour, no ?

Do you think that SquashTA comportment might change in future releases, eg use the order of TC provided by SquashTM ?

Best regards

Hello !
Ordering tests is not planned for now as far as I know.
Please note that, while tests execute without particular order TA side, results are posted back in the relevant TM execution results, which means that seen from Squash TM the final result will keep the on-screen order.
If you are refering to the manual test system in Squash TM, which allows ordering, it is a completely independant system, and therefore does not follow the same rules.

Sorry that I can't tell you more.

Best regards