14.9% regression in specjvm08_64bit in build s10u9_01
specjvm08_64bit :

--------------------------------------------------------------------------------
Historical specjvm08_64bit results on oaf315_64
--------------------------------------------------------------------------------
Build Result Status Description Related Bugs
--------------------------------------------------------------------------------
s10u9_01 14.08 RED 14.9% regression
s10u8_08a 16.54 GREEN unchanged
s10u8_08
s10u8_07 16.5 GREEN 1.8% improvement
s10u8_06
s10u8_05
s10u8_04
s10u8_03
s10u8_02
s10u8_01 16.21 RED 1.1% regression
s10u7_08 16.39 GREEN unchanged
s10u7_07 ~ YELLOW Comparison not statistically significant
s10u7_06 ~ YELLOW Comparison not statistically significant
s10u7_05 16.39 GREEN unchanged
s10u7_04 16.38 GREEN unchanged
s10u7_03 16.36 GREEN unchanged
s10u7_02 16.38 GREEN unchanged
s10u7_01 16.3 GREEN unchanged
s10u6_07b 16.3 GREEN unchanged
s10u6_07a 16.34 GREEN unchanged
s10u6_07
s10u6_06
s10u6_05 16.27 GREEN unchanged
s10u6_04 ~ YELLOW Comparison not statistically significant
s10u6_03 16.3 First Result

Using s10u8_08a as baseline
9 baselines. Average is 16.539 PctStDevis 0.60%
Plot_mean: 14.08 , overall change: 14.87 %
Iteration Result Change Colour
1 14.08 -14.87 % RED
2 14.089 -14.81 % RED
3 14.08 -14.87 % RED
4 14.109 -14.69 % RED
5 14.039 -15.11 % RED
6 14.06 -14.99 % RED
Number of runs: 6
With all numbers shown, overall change will be -14.89%
Previous build:s10u8_08a 16.544 0.03 % YELLOW
The found results are: 1 YELLOW, 6 RED including the old result
Std. Deviation as % from Mean is 0.17
TTest against baseline: 0.0000000000

Outliers
=========
Can't find the machine in /etc/bootparams.
Benchmark res_range: 10:-1 (not ship)
Driver Version changes...
  -scsi_vhci SCSI VHCI Driver 1.57 (1.57 -> 1.60)
  -sata SATA Module v1.25 (1.25 -> 1.26)
  -sd SCSI Disk Driver 1.547 (1.547 -> 1.550)
  -consconfig console configuration 1.35 (1.35 -> 1.36)
  -consconfig_dacf Consconfig DACF 1.36 (1.36 -> 1.37)
  -mpt MPT HBA Driver v1.92 (1.92 -> 1.96)
  -st SCSI tape Driver 1.255 (1.255 -> 1.256)


Detailed statistical comparison of results is available at:
http://perfwww.ireland/cgi-bin/detail.cgi?rig=oaf315_64&benchmark=specjvm08_64bit&build=s10u9_01&base=s10u8_08a&clean=FALSE

Using s10_74L2a as baseline
No Baselines
Plot_mean: 14.08 , overall change: 100.00 %
Warning: plot_per is 0.00, but overall change is 100.00. Will update later.
Iteration Result Change Colour
1 14.08
2 14.089
3 14.08
4 14.109
5 14.039
6 14.06
Number of runs: 6
With all numbers shown, overall change will be 0.00%
Previous build:s10u8_08a 16.544
Std. Deviation as % from Mean is 0.17
TTest against baseline: 1.0000000000

Outliers
=========
Can't find the machine in /etc/bootparams.
Benchmark res_range: 10:-1 (not ship)

Detailed statistical comparison of results is available at:
http://perfwww.ireland/cgi-bin/detail.cgi?rig=oaf315_64&benchmark=specjvm08_64bit&build=s10u9_01&base=s10_74L2a&clean=FALSE

http://perfwww.ireland/cgi-bin/publictools/shipII.cgi?&BUILD=s10u9&BNAMES=specjvm08_64bit&tSysteM=oaf315_64

The machine is a :Ultra 40 (64bit) 1 x 2010MHz 2048 MB ram
The rerun is back and confirms the regression:

/results9/s10u9_01-6916527-rerun.oaf315_64/specjvm08_64bitUsing s10u8_08a-6916527-rerun as baseline
16 baselines. Average is 16.53 PctStDevis 0.00%
Plot_mean: 14.06 , overall change: 14.94 %
Iteration Result Change Colour
1 14.13 -14.52 % RED
2 14.06 -14.94 % RED
3 14.009 -15.25 % RED
4 14.089 -14.76 % RED
5 14 -15.31 % RED
6 14.07 -14.88 % RED
Number of runs: 6
With all numbers shown, overall change will be -14.94%
No usable previous results for this benchmark
The found results are: 6 RED
Std. Deviation as % from Mean is 0.35
TTest against baseline: 0.0000000000

Outliers
=========
1 high outliers in base
Can't find the machine in /etc/bootparams.
Benchmark res_range: 10:-1 (not ship)
No usable previous results available for driver comparison

Detailed statistical comparison of results is available at:
http://perfwww.ireland/cgi-bin/detail.cgi?rig=oaf315_64&benchmark=specjvm08_64bit&build=s10u9_01-6916527-rerun&base=s10u8_08a-6916527-rerun&clean=FALSE
The generic gate did not track the regression. There are no feature gate putback, scheduling a driver swap.
The regression has been introduced via the Java changes in s10u9_01.

mean change% num stddev% build JVM
16.54 0.03% 9 s10u8_08a 1.5.0_20-b02 mixed mode
14.08 -14.89% 6 0.17 s10u9_01 1.5.0_23-b01 mixed mode
14.11 -14.69% 3 0.14 s10u8_08a-6916527-java-s10u9_01 1.5.0_23-b01 mixed mode

Baseline used: s10u8_08a

The s10u8_08a-6916527-java-s10u9_01 run here is a fresh install of s10u8_08a with these packages
from s10u9_01: SUNWj6rt SUNWj6rtx SUNWj6cfg SUNWj5rt SUNWj5rtx SUNWj5cfg
As the results above show switching from 1.5.0_20-b02 to 1.5.0_23-b01 on s10u8_08a introduces the
regression we are seeing in s10u9_01.

s10u8_08a -> s10u8_08a with 1.5.0_23-b01

Using s10u8_08a as baseline
9 baselines. Average is 16.539 PctStDevis 0.60%
Plot_mean: 14.11 , overall change: 14.69 %
Iteration Result Change Colour
1 14.109 -14.69 % RED
2 14.089 -14.81 % RED
3 14.13 -14.57 % RED
Number of runs: 3
With all numbers shown, overall change will be -14.69%
No usable previous results for this benchmark
The found results are: 3 RED
Std. Deviation as % from Mean is 0.14
TTest against baseline: 0.0000000000

Outliers
=========
Can't find the machine in /etc/bootparams.
Benchmark res_range: 10:-1 (not ship)
No usable previous results available for driver comparison

Detailed statistical comparison of results is available at:
http://perfwww.ireland/cgi-bin/detail.cgi?rig=oaf315_64&benchmark=specjvm08_64bit&build=s10u8_08a-6916527-java-s10u9_01&base=s10u8_08a&clean=FALSE


The rig seeing this regression is:
Ultra 40 (64bit):oaf315_64:1 x 2010MHz:2048 MB:Nvidia Quadro NVS 285

The regession was in specjvm08_64bit which can be found:
http://perfwww.ireland/Benchmarks/Java/specjvm08.tar.gz

Explorer:
http://perfwww.ireland/cgi-bin/internaltools/machineStats.cgi?s10u9_01:oaf315_64

Statis before regression:
http://perfwww.ireland/Results9/s10u8_08a.oaf315_64/specjvm08_64bit/8/statit.bin.txt
http://perfwww.ireland/Results9/s10u8_08a.oaf315_64/specjvm08_64bit/2/statit.bin.txt
http://perfwww.ireland/Results9/s10u8_08a.oaf315_64/specjvm08_64bit/5/statit.bin.txt

Statits after regression:
http://perfwww.ireland/Results9/s10u8_08a-6916527-java-s10u9_01.oaf315_64/specjvm08_64bit/1/statit.bin.txt
http://perfwww.ireland/Results9/s10u8_08a-6916527-java-s10u9_01.oaf315_64/specjvm08_64bit/2/statit.bin.txt
http://perfwww.ireland/Results9/s10u8_08a-6916527-java-s10u9_01.oaf315_64/specjvm08_64bit/3/statit.bin.txt