18.15% regression in sci_64bit_FFT in build snv_108
sci_64bit_FFT :


Historical sci_64bit_FFT Results on w991280_64 for snv_108
Baseline (build snv_85) 11.31
________________________________________________________________________________
 Build %Change %B2B Mean %Range %Stdev Runs TTest

snv_104 0 -0.02 11.19 0.51 0.00 13 0.751
snv_105 0 0.08 11.20 0.47 0.00 18 0.132
snv_106 0 -0.01 11.20 0.43 0.00 15 0.806
snv_107 0 0.03 11.20 0.15 0.00 6 0.522
snv_108 0 -18.15 9.17 0.41 0.00 3 0.000

Using snv_107 as baseline
6 baselines. Average is 11.199 PctStDevis 0.00%
Plot_mean: 9.17 , overall change: 18.12 %
Iteration Result Change Colour
1 9.1917 -17.93 % RED
2 9.1562 -18.25 % RED
3 9.1539 -18.27 % RED
Number of runs: 3
With all numbers shown, overall change will be -18.15%
Previous build:snv_107 11.200 0.00 % YELLOW
The found results are: 1 YELLOW, 3 RED including the old result
Std. Deviation as % from Mean is 0.23
TTest against baseline: 0.0000150196

Outliers
=========
The latest touch of results was 2 days ago.
Benchmark res_range: 10:-1 (not ship)

Detailed statistical comparison of results is available at:
http://perfwww.ireland/cgi-bin/detail.cgi?rig=w991280_64&benchmark=sci_64bit_FFT&build=snv_108&base=snv_107&clean=TRUE

Using s10_74L2a as baseline
No Baselines
Plot_mean: 9.17 , overall change: 100.00 %
Warning: plot_per is 0.00, but overall change is 100.00. Will update later.
Iteration Result Change Colour
1 9.1917
2 9.1562
3 9.1539
Number of runs: 3
With all numbers shown, overall change will be 0.00%
Previous build:snv_107 11.200
Std. Deviation as % from Mean is 0.23
TTest against baseline: 1.0000000000

Outliers
=========
The latest touch of results was 2 days ago.
Benchmark res_range: 10:-1 (not ship)

Detailed statistical comparison of results is available at:
http://perfwww.ireland/cgi-bin/detail.cgi?rig=w991280_64&benchmark=sci_64bit_FFT&build=snv_108&base=s10_74L2a&clean=TRUE

http://perfwww.ireland/cgi-bin/publictools/shipII.cgi?&BUILD=snv&BNAMES=sci_64bit_FFT&tSysteM=w991280_64

The machine is a :Netra-T12 12 x 1200MHz 24576 MB ram
Looking at the results, the regression looks real. The first experiment will be to binary chop the putbacks.
The binary chop is NOT tracking this regression. The next step will be to swap out any / all changed drivers (that changed between snv_108 to snv_107).

 mean change% num stddev% build
11.20 0.04% 6 6809183_snv_107-nd
11.20 -0.02% 6 0.15 6809183_snv_108-nd

Baseline used: 6809183_snv_107-nd
The all driver swap is not showing any improvement either, so unlikely to be at fault.

mean change% num stddev% build
9.17 -0.03% 3 snv_108
9.17 0.05% 6 0.38 snv_108-6809183-all-drv-snv_107

Baseline used: snv_108

Revisiting the build to build comparison we not that the results remain down in snv_108 -> snv_111.

 Build %Change %B2B Mean %Range %Stdev Runs TTest
snv_100a -- -- 9.22 0.67 0.00 3 --
snv_101 0 21.09 11.17 0.64 0.00 3 0.000
snv_101a 0 0.17 11.19 0.17 0.00 3 0.462
snv_102 0 0.07 11.19 0.20 0.00 3 0.462
snv_103 0 -0.03 11.19 0.31 0.00 9 0.707
snv_104 0 -0.02 11.19 0.51 0.00 13 0.751
snv_105 0 0.08 11.20 0.47 0.00 18 0.132
snv_106 0 -0.01 11.20 0.43 0.00 15 0.806
snv_107 0 0.03 11.20 0.15 0.00 6 0.522
snv_108 0 -18.15 9.17 0.41 0.00 3 0.000
snv_109 0 0.15 9.18 0.50 0.00 3 0.528
snv_110 0 -0.03 9.18 0.63 0.00 5 0.876
snv_111 0 -0.16 9.16 0.81 0.00 3 0.612

I've check the version of java in snv_107 and snv_108 and it has changed between the two builds.
I think it would be worth swapping out the version of java and see if this makes a difference.
I've included two summaries (and a more detailed comparison) below. The summaries show that swapping out the version of java with the version form snv_107 is showing an improvement in line with regression.


irperf# pwd
/results9/snv_108-6809183-jdk1-6-0_11.w991280_64/sci_64bit_FFT
irperf# d_builds snv_107 snv_108 snv_108-6809183-jdk1-6-0_11
mean change% num stddev% build
11.20 0.00% 6 snv_107
9.17 -18.15% 3 0.23 snv_108
11.21 0.10% 10 0.17 snv_108-6809183-jdk1-6-0_11

Baseline used: snv_107
mean change% num stddev% build
- - - - no results
irperf# d_builds snv_108 snv_108-6809183-jdk1-6-0_11
mean change% num stddev% build
9.17 -0.03% 3 snv_108
11.21 22.26% 10 0.17 snv_108-6809183-jdk1-6-0_11

Baseline used: snv_108

A more detailed summary is given here:
/results9/snv_108-6809183-jdk1-6-0_11.w991280_64/sci_64bit_FFT
irperf# d snv_108
Using snv_108 as baseline
3 baselines. Average is 9.1699 PctStDevis 0.00%
Plot_mean: 11.21 , overall change: -22.25 %
Iteration Result Change Colour
1 11.204 22.18 % GREEN
1_1 11.221 22.37 % GREEN
2 11.220 22.36 % GREEN
2_1 11.185 21.98 % GREEN
3 11.207 22.22 % GREEN
3_1 11.231 22.48 % GREEN
4 11.202 22.17 % GREEN
4_1 11.230 22.46 % GREEN
5 11.179 21.91 % GREEN
5_1 11.230 22.46 % GREEN
Number of runs: 10
With all numbers shown, overall change will be 22.26%
No usable previous results for this benchmark
The found results are: 10 GREEN
Std. Deviation as % from Mean is 0.17
TTest against baseline: 0.0000006511

Outliers
=========
snv_108-6809183-jdk1-6-0_11 isn't running on the machine any more
The latest touch of results was 0 days ago.
Benchmark res_range: 10:-1 (not ship)
No usable previous results available for driver comparison

Detailed statistical comparison of results is available at:
http://perfwww.ireland/cgi-bin/detail.cgi?rig=w991280_64&benchmark=sci_64bit_FFT&build=snv_108-6809183-jdk1-6-0_11&base=snv_108&clean=TRUE
I also swapped in reverse the version of java in snv_107 (with that from snv_108) we are seeing an improvement compared to snv_108 and no change compared to snv_107

mean change% num stddev% build
9.17 -0.03% 3 snv_108
11.21 22.26% 10 0.17 snv_108-6809183-jdk1-6-0_11
11.19 22.06% 5 0.15 snv_107-6809183-jdk1-6-0_12

Baseline used: snv_108


mean change% num stddev% build
11.20 0.00% 6 snv_107
9.17 -18.15% 3 0.23 snv_108
11.21 0.10% 10 0.17 snv_108-6809183-jdk1-6-0_11
11.19 -0.06% 5 0.15 snv_107-6809183-jdk1-6-0_12

Baseline used: snv_107

From these results it appears to me that the version of java running on snv_108 (java.version: 1.6.0_12)
is causing the regression.

I'm not sure who the correct product / cat / subcat is for this one at the moment.
I'm not sure who to re-assign this onto - maybe you guys can help.
I've tracked down an 18% regression in snv_108 to lie with the version
of java 1.6.0_12. If I run 1.6.0_11 on it we do not see the regression.
Rerunning the java swap to clarify.
The rerun of the java does suggest that the version of java is the problem:

==== snv_107 ====
Java(TM) SE Runtime Environment (build 1.6.0_11-b03)

==== snv_108 ====
Java(TM) SE Runtime Environment (build 1.6.0_12-b04)

==== snv_107-jdk1-6-0_12-b04-6809183-rerun ====
Java(TM) SE Runtime Environment (build 1.6.0_12-b04)

==== snv_108-jdk1-6-0_11-b03-6809183-rerun ====
Java(TM) SE Runtime Environment (build 1.6.0_11-b03)


Comparison with snv_107 as a baseline:

mean change% num stddev% build
11.20 0.00% 6 snv_107
9.17 -18.15% 3 0.23 snv_108
11.20 -0.00% 3 0.17 snv_107-jdk1-6-0_12-b04-6809183-rerun
11.18 -0.18% 3 0.05 snv_108-jdk1-6-0_11-b03-6809183-rerun
Baseline used: snv_107

I am slightly confused however, as I would have expected snv_107-jdk1-6-0_12-b04-6809183-rerun to show a regression compared with snv_107. However, what we are concerned about is snv_108 (where the original regression lies). When we swap out the java in favour of the java from snv_107 we find the regression goes away. I am going to assign to java / java / other as a starting point.