Analyzing the Effectiveness of a System Testing Tool for Software Product Line Engineering

Supplemental Material

Hypotheses Testing

In order to reduce the dataset errors, we eliminated the outliers. The Figure below describes the guideline that we follow to test the hypotheses, Null Hypothesis (H0n) and Alternative Hypothesis (H1n). The tests were primarily presented for a significance level of 5%.


M1. Designed Test Cases (DTC) - Experiment

Shapiro-Wilk Normality Test
Without Tool With Tool
W 0.897 0.881
p-value 0.201 0.135
Obs. Normal Normal
Levene Test
df 1
F 5.16
Pr(>F) 0.036
Variance Different
Wilcoxon Test
- -
W 95
p-value 0.0007
Obs. H02 Rejected

M2. Efficiency in Test Case Design (ETCD) - Experiment

Shapiro-Wilk Normality Test
Without Tool With Tool
W 0.912 0.874
p-value 0.294 0.112
Obs. Normal Normal
Levene Test
df 1
F 21.54
Pr(>F) 0.0002
Variance Different
Wilcoxon Test
- -
W 100
p-value 0.00018
Obs. H02 Rejected

M3. Efficiency in Test Cases Execution (ETCE) - Experiment

Shapiro-Wilk Normality Test
Without Tool With Tool
W 0.925 0.988
p-value 0.443 0.993
Obs. Normal Normal
Levene Test
df 1
F 0.96
Pr(>F) 0.341
Variance Equal Var.
t-test
t 2.356
df 16
p-value 0.03
Obs. H03 Rejected

M3. Efficiency in Test Cases Execution (ETCE) - Replications

Shapiro-Wilk Normality Test
Without Tool With Tool
W 0.809 0.871
p-value 0.051 0.104
Obs. Normal Normal
Levene Test
df 1
F 1.27
Pr(>F) 0.276
Variance Equal Var.
t-test
t 2.18
df 15
p-value 0.004
Obs. H03 Rejected

M4. Number of Errors Found (NEF) - Experiment

Shapiro-Wilk Normality Test
Without Tool With Tool
W 0.969 0.890
p-value 0.173 0.893
Obs. Normal Normal
Levene Test
df 1
F 5.01
Pr(>F) 0.03
Variance Different
Wilcoxon Test
- -
W 51
p-value 0.34
Obs. H14 Rejected

M4. Number of Errors Found (NEF) - Replications

Shapiro-Wilk Normality Test
Without Tool With Tool
W 0.934 0.855
p-value 0.067 0.591
Obs. Normal Normal
Levene Test
df 1
F 0.0045
Pr(>F) 0.94
Variance Equal Var.
t-test
t 2.04
df 15
p-value 0.058
Obs. H14 Rejected

M5. Test Case Effectiveness (TCE) - Experiment

Shapiro-Wilk Normality Test
Without Tool With Tool
W 0.90 0.84
p-value 0.376 0.07
Obs. Normal Normal
Levene Test
df 1
F 4.84
Pr(>F) 0.04
Variance Different
Wilcoxon Test
- -
W 18.5
p-value 0.34
Obs. H15 Rejected

M5. Test Case Effectiveness (TCE) - Replications

Shapiro-Wilk Normality Test
Without Tool With Tool
W 0.96 0.86
p-value 0.87 0.09
Obs. Normal Normal
Levene Test
df 1
F 0.0037
Pr(>F) 0.95
Variance Equal Var.
t-test
t 0.65
df 14
p-value 0.52
Obs. H15 Rejected

M6. Efficiency in Finding Errors (EFE) - Experiment

Shapiro-Wilk Normality Test
Without Tool With Tool
W 0.952 0.89
p-value 0.737 0.202
Obs. Normal Normal
Levene Test
df 1
F 0.99
Pr(>F) 0.335
Variance Equal Var.
t-test
t -0.4
df 15
p-value 0.69
Obs. H16 Rejected

M6. Efficiency in Finding Errors (EFE) - Replications

Shapiro-Wilk Normality Test
Without Tool With Tool
W 0.85 0.86
p-value 0.18 0.06
Obs. Normal Normal
Levene Test
df 1
F 0.305
Pr(>F) 0.58
Variance Equal Var.
t-test
t 2.35
df 15
p-value 0.03
Obs. H06 Rejected