Analyzing the Effectiveness of a System Testing Tool for Software Product Line Engineering

Supplemental Material

Descriptive Statistics
The experiment analysis was performed based on descriptive statistics. According to Mann [1], Descriptive statistics is the discipline of quantitatively describing the main features of a data collection.

BoxPlots

The Figures below present the experiment and replications Bloxplots. In descriptive statistics, a boxplot is a convenient way of graphically depicting groups of numerical data through their five-number summaries: the smallest observation (Min.), lower quartile (1st. Quartile), median, upper quartile (3rd. Quartile), and largest observation (Max.).

M1. Designed Test Cases (DTC)

Click here to view a bigger image
M1. DTC - Experiment
Without Tool With Tool
Min. 10.00 18.00
1st. Quartile 12.25 21.25
Median 17.50 25.50
Mean 15.70 26.10
3rd. Quartile 18.00 31.50
Max. 21.00 34.00
Sd. 3.83 6.06

M2. Efficiency in Test Case Design (ETCD)

Click here to view a bigger image
M2. ETCD - Experiment
Without Tool With Tool
Min. 0.11 0.42
1st. Quartile 0.16 0.73
Median 0.21 1.15
Mean 0.22 1.18
3rd. Quartile 0.29 1.70
Max. 0.32 1.76
Sd. 0.08 0.52

M3. Efficiency in Test Cases Execution (ETCE)

Click here to view a bigger image
M3. ETCE - Experiment
Without Tool With Tool
Min. 0.24 0.45
1st. Quartile 0.37 0.61
Median 0.43 0.69
Mean 0.50 0.68
3rd. Quartile 0.68 0.74
Max. 0.78 0.89
Sd. 0.18 0.13
M3. ETCE - Replications
Without Tool With Tool
Min. 0.50 0.97
1st. Quartile 0.67 1.13
Median 0.73 1.87
Mean 1.07 2.00
3rd. Quartile 1.51 2.36
Max. 1.90 3.63
Sd. 0.60 1.00


M4. Number of Errors Found (NEF)

Click here to view a bigger image
M4. NEF - Experiment
Without Tool With Tool
Min. 0.00 2.00
1st. Quartile 1.25 2.25
Median 3.00 5.00
Mean 2.90 5.20
3rd. Quartile 4.00 7.75
Max. 7.00 10.00
Sd. 2.23 2.97
M4. NEF - Replications
Without Tool With Tool
Min. 2.00 5.00
1st. Quartile 4.50 6.00
Median 5.00 9.00
Mean 5.71 8.20
3rd. Quartile 6.50 10.00
Max. 11.00 11.00
Sd. 2.81 2.20


M5. Test Case Effectiveness (TCE)

Click here to view a bigger image
M5. TCE - Experiment
Without Tool With Tool
Min. 0.17 0.09
1st. Quartile 0.175 0.10
Median 0.20 0.17
Mean 0.20 0.17
3rd. Quartile 0.22 0.24
Max. 0.26 0.24
Sd. 0.34 0.07
M5. TCE - Replications
Without Tool With Tool
Min. 0.11 0.17
1st. Quartile 0.22 0.21
Median 0.26 0.31
Mean 0.26 0.28
3rd. Quartile 0.31 0.34
Max. 0.37 0.38
Sd. 0.09 0.07


M6. Efficiency in Finding Errors (EFE)

Click here to view a bigger image
M6. EFE - Experiment
Without Tool With Tool
Min. 0.03 0.06
1st. Quartile 0.08 0.07
Median 0.13 0.10
Mean 0.14 0.13
3rd. Quartile 0.20 0.18
Max. 0.28 0.23
Sd. 0.09 0.07
M6. EFE - Replications
Without Tool With Tool
Min. 0.08 0.17
1st. Quartile 0.22 0.21
Median 0.13 0.37
Mean 0.19 0.50
3rd. Quartile 0.28 0.75
Max. 0.60 0.83
Sd. 0.19 0.22

References
[1] Mann, Prem S. (1995) Introductory Statistics, 2nd Edition, Wiley.