12.2% regression in scimark_server_tuned_large_FFT in build osol_1002-143
scimark_server_tuned_large_FFT :

--------------------------------------------------------------------------------
Historical scimark_server_tuned_large_FFT results on oaf223_64
--------------------------------------------------------------------------------
Build Result Status Description Related Bugs
--------------------------------------------------------------------------------
osol_1002-143 10.32 RED 12.2% regression
osol_1002-142 11.75 RED 16% regression
osol_1002-141 ~ YELLOW Comparison not statistically significant6965584
osol_1002-140 14.34 GREEN unchanged 6957827
osol_1002-139 14.06 RED 2.4% regression 6953095
osol_1002-138 ~ YELLOW Comparison not statistically significant6950903
osol_1002-137
osol_1002-136 ~ YELLOW Comparison not statistically significant6942207
osol_1002-135 ~ YELLOW Baseline data results vary too much (stddev=4.1%)6938683
osol_1002-134a ~ YELLOW Test data results vary too much (stddev=4.1%)
osol_1002-134
osol_1002-133
osol_1002-132 ~ YELLOW Comparison not statistically significant
osol_1002-131 14.49 GREEN unchanged
osol_1002-130 ~ YELLOW Comparison not statistically significant
osol_1002-129 ~ YELLOW Comparison not statistically significant
osol_1002-128 GREEN
osol_1002-127 GREEN
osol_1002-126 ~ YELLOW Test data results vary too much (stddev=4.1%)
osol_1002-125 ~ YELLOW Test data results vary too much (stddev=4.2%)
osol_1002-124 ~ YELLOW Baseline data results vary too much (stddev=4.7%)
osol_1002-123 ~ YELLOW Test data results vary too much (stddev=4.7%)
osol_1002-122 ~ YELLOW Comparison not statistically significant
osol_1002-121 15.16 GREEN 2.8% improvement
osol_1002-118 14.75 GREEN Not applicable / multimodal
osol_1002-117 14.37 RED Not applicable / multimodal
osol_0906-111b2
osol_0906-111b
osol_0906-111a
osol_0906-111 14.52 GREEN Not applicable / multimodal
osol_0906-110
osol_0906-109
osol_0906-108
osol_0906-107 12.36 RED Not applicable / multimodal
osol_0906-106a 14.77 First Result

Using osol_1002-142 as baseline
24 baselines. Average is 11.75 PctStDevis 13.35%
Plot_mean: 10.32 , overall change: 12.17 %
Iteration Result Change Colour
1 10.117 -13.89 % RED
2 10.980 -6.54 % RED
3 10.545 -10.25 % RED
4 10.069 -14.30 % RED
5 10.192 -13.25 % RED
6 10.037 -14.57 % RED
Number of runs: 6
With all numbers shown, overall change will be -12.14%
Previous build:osol_1002-142 11.752 0.02 % YELLOW
The found results are: 1 YELLOW, 6 RED including the old result
Std. Deviation as % from Mean is 3.59
TTest against baseline: 0.0003835350

Outliers
=========
3 high outliers in base
Can't find the machine in /etc/bootparams.
Benchmark res_range: 45:-0.5 (not ship)

Detailed statistical comparison of results is available at:
http://perfwww.ireland/cgi-bin/detail.cgi?rig=oaf223_64&benchmark=scimark_server_tuned_large_FFT&build=osol_1002-143&base=osol_1002-142&clean=FALSE

Using s10_74L2a as baseline
No Baselines
Plot_mean: 10.32 , overall change: 100.00 %
Warning: plot_per is 0.00, but overall change is 100.00. Will update later.
Iteration Result Change Colour
1 10.117
2 10.980
3 10.545
4 10.069
5 10.192
6 10.037
Number of runs: 6
With all numbers shown, overall change will be 0.00%
Previous build:osol_1002-142 11.752
Std. Deviation as % from Mean is 3.59
TTest against baseline: 1.0000000000

Outliers
=========
Can't find the machine in /etc/bootparams.
Benchmark res_range: 45:-0.5 (not ship)

Detailed statistical comparison of results is available at:
http://perfwww.ireland/cgi-bin/detail.cgi?rig=oaf223_64&benchmark=scimark_server_tuned_large_FFT&build=osol_1002-143&base=s10_74L2a&clean=FALSE

http://perfwww.ireland/cgi-bin/publictools/shipII.cgi?&BUILD=snv&BNAMES=scimark_server_tuned_large_FFT&tSysteM=oaf223_64

The machine is a :Ultra 45 2 x 1600MHz 2048 MB ram
The same rig is seeing a larger regression in another java metric so I am
updating the title. I will test the change in java version first.

Using osol_1002-142 as baseline
3 baselines. Average is 11465 PctStDevis 0.56%
Plot_mean: 7326.40 , overall change: 36.10 %
Iteration Result Change Colour
1 7321 -36.15 % RED
10 7366 -35.76 % RED
2 7315 -36.20 % RED
3 7261 -36.67 % RED
4 7315 -36.20 % RED
5 7325 -36.11 % RED
6 7361 -35.80 % RED
7 7346 -35.93 % RED
8 7290 -36.42 % RED
9 7364 -35.77 % RED
Number of runs: 10
With all numbers shown, overall change will be -36.10%
Previous build:osol_1002-142 11465 -0.00 % YELLOW
The found results are: 1 YELLOW, 10 RED including the old result
Std. Deviation as % from Mean is 0.47
TTest against baseline: 0.0000225160

Outliers
=========
Can't find the machine in /etc/bootparams.
Benchmark res_range: 45:-0.5 (not ship)

Detailed statistical comparison of results is available at:
http://perfwww.ireland/cgi-bin/detail.cgi?rig=oaf223_64&benchmark=jbb2005_64_vm&build=osol_1002-143&base=osol_1002-142&clean=FALSE
It looks like the regression is down to the Java version change.

scimark_server_tuned_large_FFT

mean change% num stddev% build
11.75 0.02% 24 osol_1002-142
10.28 -12.47% 20 3.37 osol_1002-143
10.79 -8.19% 3 4.15 osol_1002-142-6968622-java-143

Baseline used: osol_1002-142

I am rerunning

osol_1002-142-6968622-java-143 here is osol_1002-143 with 'runtime/java'
from osol_1002-143.

osol_1002-142 version:
java version "1.6.0_18"
Java(TM) SE Runtime Environment (build 1.6.0_18-b07)
Java HotSpot(TM) Server VM (build 16.0-b13, mixed mode)

osol_1002-143 version:
java version "1.6.0_21"
Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
Java HotSpot(TM) Server VM (build 17.0-b16, mixed mode)
The jb2005 regression also has java as it's root cause:

mean change% num stddev% build (results)
15227.00 0.00% 3 osol_1002-142 (15242 15145 15294)
9910.90 -34.91% 10 0.61 osol_1002-143 (9811 9923 9951 9886 9935 9851 9914 9854 10004 9980)
9995.00 -34.36% 1 osol_1002-142-6968622-java-143-6968622-rerun (9995)

Baseline used: osol_1002-142

Here are the rig details:

The rig seeing this regression is:
Ultra 45:oaf223_64:2 x 1600MHz:2048 MB:XVR-100

The regession was in jbb2005_vm_svr which can be found:
http://perfwww.ireland/Benchmarks/Java/jbb05.tar.gz

Explorer:
http://perfwww.ireland/cgi-bin/internaltools/machineStats.cgi?osol_1002-143:oaf223_64

Statis before regression:
http://perfwww.ireland/Results9/osol_1002-142.oaf223_64/jbb2005_vm_svr/3/statit.bin.txt
http://perfwww.ireland/Results9/osol_1002-142.oaf223_64/jbb2005_vm_svr/2/statit.bin.txt
http://perfwww.ireland/Results9/osol_1002-142.oaf223_64/jbb2005_vm_svr/1/statit.bin.txt

Statits after regression:
http://perfwww.ireland/Results9/osol_1002-142-6968622-java-143-6968622-rerun.oaf223_64/jbb2005_vm_svr/1/statit.bin.txt