3.61% regression in spec_14_jvm98Servergm in build snv_121
spec_14_jvm98Servergm :



Using snv_120 as baseline
6 baselines. Average is 293.06 PctStDevis 0.53%
Plot_mean: 282.49 , overall change: 3.61 %
Iteration Result Change Colour
1 282.74 -3.52 % RED
2 281.91 -3.80 % RED
3 282.81 -3.50 % RED
Number of runs: 3
With all numbers shown, overall change will be -3.61%
Previous build:snv_120 293.06 0.00 % YELLOW
The found results are: 1 YELLOW, 3 RED including the old result
Std. Deviation as % from Mean is 0.18
TTest against baseline: 0.0000023866

Outliers
=========
Can't find the machine in /etc/bootparams.
The latest touch of results was 0 days ago.
Benchmark res_range: 10:-1 (not ship)
Driver Version changes...
  -qlc SunFC Qlogic FCA v20090519-2.36 (2.36 -> 2.37)


Detailed statistical comparison of results is available at:
http://perfwww.ireland/cgi-bin/detail.cgi?rig=blade002_64&benchmark=spec_14_jvm98Servergm&build=snv_121&base=snv_120&clean=TRUE



http://perfwww.ireland/cgi-bin/publictools/shipII.cgi?&BUILD=snv&BNAMES=spec_14_jvm98Servergm&tSysteM=blade002_64

The machine is a :Netra-CP3220(Alonso) 32 x 1000MHz 16376 Megabytes ram
The binary chop is not tracking this regression.

mean change% num stddev% build
291.61 0.00% 3 6871771_onnv_121_10144_1164
291.48 -0.04% 3 0.87 6871771_onnv_122_10254_1169


The rerun confirms the regression.

mean change% num stddev% build
293.94 0.00% 3 snv_120-6871771-rerun
281.81 -4.13% 3 0.60 snv_121-6871771-rerun


There has been a change in the version of java, this is the next step.

==== /results9/snv_120-6871771-rerun.blade002_64 ===
java version "1.6.0_13"
Java(TM) SE Runtime Environment (build 1.6.0_13-b03)
Java HotSpot(TM) Server VM (build 11.3-b02, mixed mode)
==== /results9/snv_121-6871771-rerun.blade002_64 ===
java version "1.6.0_15"
Java(TM) SE Runtime Environment (build 1.6.0_15-b03)
Java HotSpot(TM) Server VM (build 14.1-b02, mixed mode)
Swapping the version of java from snv_120 to snv_121 is yielding a 3.59% improvement, in line with the regression we are seeing.


/results9/snv_121-6871771-javaswap-jdk1-6-0_13-6871771-rerun.blade002_64/spec_14_jvm98Servergm
irperf# d snv_121-6871771-rerun
Using snv_121-6871771-rerun as baseline
6 baselines. Average is 281.43 PctStDevis 1.03%
Plot_mean: 291.52 , overall change: -3.59 %
Iteration Result Change Colour
1 293.35 4.24 % YELLOW
2 288.96 2.68 % YELLOW
3 293.10 4.15 % YELLOW
4 292.46 3.92 % YELLOW
5 291.62 3.62 % YELLOW
6 289.61 2.91 % YELLOW
Number of runs: 6
With all numbers shown, overall change will be 3.59%
No usable previous results for this benchmark
The found results are: 6 YELLOW
Std. Deviation as % from Mean is 0.63
TTest against baseline: 0.0000698448

Outliers
=========
Can't find the machine in /etc/bootparams.
The latest touch of results was 0 days ago.
Benchmark res_range: 10:-1 (not ship)
No usable previous results available for driver comparison

Detailed statistical comparison of results is available at:
http://perfwww.ireland/cgi-bin/detail.cgi?rig=blade002_64&benchmark=spec_14_jvm98Servergm&build=snv_121-6871771-javaswap-jdk1-6-0_13-6871771-rerun&base=snv_121-6871771-rerun&clean=TRUE


/results9/snv_121-6871771-javaswap-jdk1-6-0_13-6871771-rerun.blade002_64/spec_14_jvm98Servergm
irperf# cat 1/summary.txt | grep Java
Java(TM) SE Runtime Environment (build 1.6.0_13-b03)
Java HotSpot(TM) Server VM (build 11.3-b02, mixed mode)

/export/results9/snv_121-6871771-rerun.blade002_64/spec_14_jvm98Servergm
irperf# cat 1/summary.txt | grep Java
Java(TM) SE Runtime Environment (build 1.6.0_15-b03)
Java HotSpot(TM) Server VM (build 14.1-b02, mixed mode)

I'm swapping in the reverse direction to confirm.
We ran this experiment in reverse, installed snv_120 and swapped the version of java with that from snv_121. Here we are seeing the regression, which we expect.

mean change% num stddev% build
293.31 0.00% 6 snv_120-6871771-rerun
284.57 -2.90% 3 0.31 snv_120-6871771-javaswap-jdk1-6-0_15-6871771-rerun


So in summary, we are seeing a 3.61% regression between snv_121 and snv_120. When we swap out the version of java from snv_120 into snv_121 we see an improvement in snv_121 compared with original snv_121 and numbers are in line with snv_120. When we swap the version of java from snv_121 into snv_120 we see a regression in snv_120 compared with the original snv_120 and numbers are inline with the original snv_121 numbesrs.

mean change% num stddev% build
293.06 0.00% 6 snv_120
282.22 -3.70% 12 0.66 snv_121
284.57 -2.90% 3 0.31 snv_120-6871771-javaswap-jdk1-6-0_15-6871771-rerun
291.52 -0.52% 6 0.63 snv_121-6871771-javaswap-jdk1-6-0_13-6871771-rerun


We are seeing this in a number of different machines / metrics - I am confirming this at present and will subsequently close the bugs as duplicates of this one.