3.8% regression in scimark_server_tuned_large_LU in build s10u9_05
scimark_server_tuned_large_LU :

--------------------------------------------------------------------------------
Historical scimark_server_tuned_large_LU results on oaf341_64
--------------------------------------------------------------------------------
Build Result Status Description Related Bugs
--------------------------------------------------------------------------------
s10u9_05 647.12 RED 3.8% regression
s10u9_04 672.72 GREEN unchanged 6934770
s10u9_02 673.09 GREEN unchanged 6926231
s10u9_01 673.7 GREEN unchanged 6916537
s10u8_08a
s10u8_08
s10u8_07 671.71 GREEN unchanged 6881975
s10u8_06
s10u8_05
s10u8_04
s10u8_03
s10u8_02
s10u8_01 674.54 GREEN unchanged 6851852
s10u7_08 674.25 GREEN unchanged 6830126
s10u7_07 ~ YELLOW Comparison not statistically significant6823785
s10u7_06 ~ YELLOW Comparison not statistically significant6826800
s10u7_05 673.85 GREEN unchanged
s10u7_04 673.57 GREEN unchanged
s10u7_03
s10u7_02
s10u7_01 674.59 GREEN unchanged
s10u6_07b 673.18 GREEN unchanged
s10u6_07a 673.97 GREEN unchanged
s10u6_07 673.06 GREEN unchanged
s10u6_06 675.34 GREEN unchanged
s10u6_05 671.42 GREEN unchanged
s10u6_04 672.95 GREEN unchanged
s10u6_03
s10u6_02
s10u6_01
s10u5_10 674.02 GREEN unchanged
s10u5_09 ~ YELLOW Comparison not statistically significant
s10u5_08
s10u5_07 ~ YELLOW Comparison not statistically significant
s10u5_06 674.14 GREEN unchanged
s10u5_05 672.98 GREEN unchanged
s10u5_05 672.98 GREEN unchanged
s10u5_04 671.76 GREEN unchanged
s10u5_03 675.05 GREEN unchanged
s10u5_02 673.4 GREEN unchanged
s10u5_01 672.72 GREEN unchanged
s10u4_12b 672.82 GREEN unchanged
s10u4_12 674.65 GREEN unchanged
s10u4_11 673.91 GREEN unchanged
s10u4_10 673.6 GREEN unchanged
s10u4_09 673.72 GREEN unchanged
s10u4_08 668.49 GREEN unchanged
s10u4_07 668.79 GREEN unchanged
s10u4_06 670.02 GREEN unchanged
s10u4_05a 668.23 GREEN unchanged
s10u4_05 669.29 GREEN unchanged
s10u4_04 670.43 GREEN 24.2% improvement
s10u4_03
s10u4_02 539.89 GREEN unchanged
s10u4_01 538.32 GREEN unchanged
s10u3_10
s10u3_09 539.16 GREEN unchanged
s10u3_08 539.23 GREEN unchanged
s10u3_07a
s10u3_07 536.85 First Result

Using s10u9_04 as baseline
6 baselines. Average is 672.72 PctStDevis 0.13%
Plot_mean: 647.12 , overall change: 3.81 %
Iteration Result Change Colour
1 647.87 -3.69 % RED
2 647.56 -3.74 % RED
3 648.50 -3.60 % RED
4 645.68 -4.02 % RED
5 645.68 -4.02 % RED
6 648.50 -3.60 % RED
7 645.99 -3.97 % RED
Number of runs: 7
With all numbers shown, overall change will be -3.81%
Previous build:s10u9_04 672.72 0.00 % YELLOW
The found results are: 1 YELLOW, 7 RED including the old result
Std. Deviation as % from Mean is 0.20
TTest against baseline: 0.0000000000

Outliers
=========
Can't find the machine in /etc/bootparams.
Benchmark res_range: 10:-1 (not ship)
Driver Version changes...
  -fp SunFC Port v20100212-1.82 (1.82 -> 1.83)


Detailed statistical comparison of results is available at:
http://perfwww.ireland/cgi-bin/detail.cgi?rig=oaf341_64&benchmark=scimark_server_tuned_large_LU&build=s10u9_05&base=s10u9_04&clean=FALSE

Using s10_74L2a as baseline
No Baselines
Plot_mean: 647.12 , overall change: 100.00 %
Warning: plot_per is 0.00, but overall change is 100.00. Will update later.
Iteration Result Change Colour
1 647.87
2 647.56
3 648.50
4 645.68
5 645.68
6 648.50
7 645.99
Number of runs: 7
With all numbers shown, overall change will be 0.00%
Previous build:s10u9_04 672.72
Std. Deviation as % from Mean is 0.20
TTest against baseline: 1.0000000000

Outliers
=========
Can't find the machine in /etc/bootparams.
Benchmark res_range: 10:-1 (not ship)

Detailed statistical comparison of results is available at:
http://perfwww.ireland/cgi-bin/detail.cgi?rig=oaf341_64&benchmark=scimark_server_tuned_large_LU&build=s10u9_05&base=s10_74L2a&clean=FALSE

http://perfwww.ireland/cgi-bin/publictools/shipII.cgi?&BUILD=s10u9&BNAMES=scimark_server_tuned_large_LU&tSysteM=oaf341_64

The machine is a :Xeon 2soc 4 x 1600MHz 2043 MB ram
We are also seeing a ~275% improvement in Monte Carlo, which are holding out
accross the board in s10u9_05 for x64 and s10u9_04 for sparc.

s10u9_05 to s10u9_04 comparisons on all machines for scimark_server_tuned_large_Monte
================================================================================
         | s10u9_05 | s10u9_04 |
%Change | Mean %STD Runs | Mean %STD Runs | Machine
  -1.31 | 51.21 1.30 2 | 51.89 0.27 2 | blade003_64 (Netra-CP3010)
  78.59 | 159.26 0.00 1 | 89.18 1.25 8 | blade004_64 (Netra-CP3220(Alonso))
  76.58 | 129.24 0.00 1 | 73.19 1.27 8 | blade016_64 (Netra-CP3020(Andretti))
  74.96 | 158.67 0.00 2 | 90.69 1.87 4 | oaf321_64 (Ultra 20 M2(Munich) (64bit))
 278.34 | 181.27 0.58 7 | 47.91 0.78 6 | oaf341_64 (Xeon 2soc)
   0.07 | 27.37 0.00 1 | 27.35 0.00 11 | oaf415_64 (T5120(Huron))
  -0.00 | 27.37 0.00 1 | 27.37 0.00 3 | oaf432_64 (T6320)
   0.33 | 33.22 0.00 1 | 33.11 0.68 3 | oaf438_64 (T5220(Huron))
   0.04 | 27.36 0.00 2 | 27.35 0.00 7 | oaf465_64 (T5220(Huron))
 247.75 | 273.50 0.00 1 | 78.65 0.00 6 | oaf483_64 (X8450(Scorpio))
  73.38 | 187.49 0.00 2 | 108.14 0.00 3 | oaf513_64 (X4100 M2(Galaxy 1F))
 230.90 | 258.24 0.00 2 | 78.04 0.00 6 | oaf541_64 (X6250(Wolf))
  -0.01 | 11.04 0.00 1 | 11.05 0.00 8 | oaf555_64 (Netra-T2000(Ontario))
   0.23 | 27.37 0.00 2 | 27.30 0.52 5 | oaf582_64 (Netra-T5220)
  -0.78 | 32.90 0.00 1 | 33.15 0.52 5 | oaf591_64 (T6340)
   0.02 | 27.37 0.00 2 | 27.36 0.00 14 | oaf650_64 (T5120(Huron))

s10u9_04 to s10u9_03 comparisons on all machines for scimark_server_tuned_large_Monte
================================================================================
         | s10u9_04 | s10u9_03 |
%Change | Mean %STD Runs | Mean %STD Runs | Machine
  33.65 | 9.20 0.00 1 | 6.89 0.00 4 | blade001_64 (Netra-CP3060(Montoya))
  -3.74 | 80.09 3.44 11 | 83.20 5.43 5 | blade002_64 (Netra-CP3220(Alonso))
 127.79 | 51.89 0.27 2 | 22.78 2.98 5 | blade003_64 (Netra-CP3010)
 115.33 | 27.36 0.00 4 | 12.71 2.49 8 | blade015_64 (Netra-CP3260(Monza))
   1.70 | 78.01 2.33 6 | 76.70 2.32 7 | oaf024_64 (Opteron Desktop (64bit))
 121.19 | 48.94 0.82 6 | 22.12 0.00 6 | oaf044_64 (V210)
 123.41 | 36.93 0.38 21 | 16.53 1.05 12 | oaf166_64 (SunBlade1000 (2 X UltraIII))
 148.95 | 52.32 0.43 9 | 21.01 1.78 6 | oaf170_64 (V490(Sebring))
 127.54 | 79.89 0.00 12 | 35.11 3.59 12 | oaf223_64 (Ultra 45)
   0.21 | 116.32 0.40 18 | 116.08 0.09 2 | oaf225_64 (Ultra 40)
  33.04 | 11.04 0.00 6 | 8.30 0.00 1 | oaf252_64 (T2000(Ontario))
 126.36 | 73.15 0.68 12 | 32.32 1.88 4 | oaf266_64 (SunBlade1500)
   2.15 | 109.32 0.38 2 | 107.02 2.59 2 | oaf295_64 (X4600(Galaxy4))
   2.47 | 99.73 0.00 4 | 97.33 2.12 3 | oaf302_64 (V40z(Stinger4))
   0.08 | 128.01 0.16 4 | 127.90 0.22 8 | oaf315_64 (Ultra 40 (64bit))
   2.45 | 91.61 0.00 3 | 89.42 2.08 6 | oaf320_64 (Ultra 20 M2(Munich) (64bit))
  -0.87 | 90.69 1.87 4 | 91.48 0.19 6 | oaf321_64 (Ultra 20 M2(Munich) (64bit))
   0.43 | 47.91 0.78 6 | 47.71 0.63 7 | oaf341_64 (Xeon 2soc)
   5.29 | 87.44 0.86 8 | 83.05 11.34 12 | oaf344_64_2048M (Xeon 4core)
   0.12 | 74.81 0.00 14 | 74.72 0.30 6 | oaf351_64 (Ultra 40 M2 (64bit))
 140.36 | 103.06 0.64 3 | 42.88 0.66 4 | oaf399_64 (V890)
  -0.12 | 70.49 0.00 1 | 70.58 0.00 8 | oaf403_64 (X3220)
 112.74 | 27.35 0.00 11 | 12.86 2.33 4 | oaf415_64 (T5120(Huron))
 110.49 | 27.37 0.00 3 | 13.00 0.00 3 | oaf432_64 (T6320)
  -2.51 | 89.06 1.98 3 | 91.35 0.46 5 | oaf447_64 (Ultra 20 M2(Munich) (64bit))
  -0.00 | 101.72 0.10 7 | 101.73 0.10 9 | oaf456_64 (Opteron 8356 4core 4soc)
   0.01 | 124.68 0.00 3 | 124.67 0.00 6 | oaf459_64 (X8440(Mira))
 110.36 | 27.35 0.00 7 | 13.00 0.00 13 | oaf465_64 (T5220(Huron))
  -0.04 | 78.63 0.00 2 | 78.66 0.00 5 | oaf482_64 (X8450(Scorpio))
  -0.01 | 78.65 0.00 6 | 78.66 0.00 4 | oaf483_64 (X8450(Scorpio))
  -0.97 | 88.35 0.80 8 | 89.21 0.00 1 | oaf488_64 (Xeon 5430 2soc)
  -0.01 | 108.14 0.00 3 | 108.15 0.00 2 | oaf513_64 (X4100 M2(Galaxy 1F))
  -0.40 | 96.12 0.00 10 | 96.51 0.84 11 | oaf537_64 (X6450(Hercules2))
  -0.61 | 107.50 1.50 6 | 108.15 0.00 6 | oaf539_64 (X6220(Gemini))
   0.97 | 78.04 0.00 6 | 77.29 0.62 5 | oaf541_64 (X6250(Wolf))
 112.25 | 27.37 0.00 5 | 12.89 1.90 6 | oaf559_64 (Netra-T5220(Huron))
  -1.38 | 115.67 1.77 6 | 117.29 0.78 6 | oaf562_64 (X2100 M2(Leo))
   0.98 | 134.28 0.18 7 | 132.98 0.18 6 | oaf575_64 (X2270(Wasp))
 116.46 | 48.56 0.54 4 | 22.43 0.89 16 | oaf579_64 (V210)
  33.08 | 11.05 0.00 3 | 8.30 0.00 3 | oaf581_64 (Netra-T2000(Ontario))
  98.51 | 27.30 0.52 5 | 13.76 0.00 5 | oaf582_64 (Netra-T5220)
 -12.00 | 118.09 0.00 6 | 134.20 0.00 7 | oaf584_64 (Ultra 27(Volans))
   1.53 | 90.73 0.62 7 | 89.36 1.05 8 | oaf586_64 (Opteron Desktop)
 110.24 | 27.33 0.00 8 | 13.00 0.00 6 | oaf587_64 (T5120(Huron))
 111.58 | 33.15 0.52 5 | 15.67 1.91 6 | oaf591_64 (T6340)
  -0.52 | 107.57 0.00 4 | 108.13 0.42 6 | oaf601_64 (X8400(Barcelona))
   0.02 | 108.43 0.00 8 | 108.41 0.00 6 | oaf603_64 (X8420(Capella))
   0.53 | 124.91 1.48 6 | 124.25 1.39 6 | oaf604_64 (X8440(Mira))
  -0.88 | 123.57 1.68 6 | 124.67 1.47 11 | oaf605_64 (X8440(Mira))
   0.31 | 104.46 1.99 6 | 104.13 1.99 7 | oaf606_64 (X8440(Mira))
   4.28 | 46.28 0.37 5 | 44.38 4.91 2 | oaf641 (Asus)
 113.70 | 27.36 0.00 14 | 12.81 2.34 6 | oaf650_64 (T5120(Huron))
  40.58 | 85.61 0.72 3 | 60.90 0.00 4 | opel09_64 (M4000)
  -1.78 | 27.66 2.87 8 | 28.16 2.04 20 | perx-18 (V65)
It looks like the root cause of this regression is a change from default java version.


For this rig the default in s10u9_04 is

java version "1.5.0_23"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_23-b01)
Java HotSpot(TM) Server VM (build 1.5.0_23-b01, mixed mode)

and in s10u9_05 is

java version "1.6.0_18"
Java(TM) SE Runtime Environment (build 1.6.0_18-b07)
Java HotSpot(TM) Server VM (build 16.0-b13, mixed mode)



It looks like there are issues with exactly when the default changes from
1.5 to 1.6 (cr's 6931507, 6933335), so for this comparison I installed
s10u9_04 and then manually changed the default version to 1.6.0_18 (as it was on the s10u9_05 run)
and retested.

Here are the results:


mean change% num stddev% build (version)
672.72 0.00% 6 s10u9_04 (1.5.0_23)
647.12 -3.81% 7 0.20 s10u9_05 (1.6.0_18)
646.78 -3.86% 2 0.10 s10u9_04-java-s10u9_05 (1.6.0_18)


Baseline used: s10u9_04

Comparing s10u9_04 -> s10u9_05 here for all the scimark metrics:

Mean Change #-res StdDev% Metric
364.99 7.96% 7 0.10 scimark_server_tuned_large_Composite
30.64 3.92% 7 0.03 scimark_server_tuned_large_FFT
647.12 -3.81% 7 0.20 scimark_server_tuned_large_LU
181.27 278.35% 7 0.58 scimark_server_tuned_large_Monte
603.14 0.32% 7 0.08 scimark_server_tuned_large_SOR
362.81 6.99% 7 0.12 scimark_server_tuned_large_Sparse
What is the JDK version under test?

The composite score is most important, rather than the individual subtests.
How does the composite score compare?
Hi James,

Both questions are already awnsered in comment 2! There is lots there,
so here is the summary.

For this rig the default in s10u9_04 is

java version "1.5.0_23"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_23-b01)
Java HotSpot(TM) Server VM (build 1.5.0_23-b01, mixed mode)

and in s10u9_05 is

java version "1.6.0_18"
Java(TM) SE Runtime Environment (build 1.6.0_18-b07)
Java HotSpot(TM) Server VM (build 16.0-b13, mixed mode)


Comparing s10u9_04 -> s10u9_05 here for all the scimark metrics:

Mean Change #-res StdDev% Metric
364.99 7.96% 7 0.10 scimark_server_tuned_large_Composite
30.64 3.92% 7 0.03 scimark_server_tuned_large_FFT
647.12 -3.81% 7 0.20 scimark_server_tuned_large_LU
181.27 278.35% 7 0.58 scimark_server_tuned_large_Monte
603.14 0.32% 7 0.08 scimark_server_tuned_large_SOR
362.81 6.99% 7 0.12 scimark_server_tuned_large_Sparse