Affected tests:
gc/gctests/StringInternGC
gc/gctests/StringInternSyncWithGC
vm/gc/compact/Compact_InternedStrings
vm/gc/compact/Compact_InternedStrings_NonbranchyTree
vm/gc/compact/Compact_InternedStrings_Strings
vm/gc/compact/Humongous_InternedStrings

Tests fails with:

;; Using jvm: "/bpool/local/aurora/sandbox/sca/vmsqe/jdk/pit/8/product/solaris-amd64/jre/lib/amd64/server/libjvm.so"
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 2560752 bytes for Chunk::new
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (allocation.cpp:374), pid=19929, tid=73
#
# JRE version: Java(TM) SE Runtime Environment (8.0-b84) (build 1.8.0-ea-b84)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.0-b26-internal-201304050410.amurillo.hs25-b26-snapshot mixed mode solaris-amd64 compressed oops)
# Core dump written. Default location: /bpool/local/aurora/sandbox/results/ResultDir/StringInternGC/core or core.19929
#

--------------- T H R E A D ---------------

Current thread (0x0000000005291800): JavaThread "gc.gctests.StringInternGC.StringInternGC$StringGenerator@3cd1a2f1" [_thread_in_vm, id=73, stack(0xfffffd7fa03da000,0xfffffd7fa04da000)]

Stack: [0xfffffd7fa03da000,0xfffffd7fa04da000], sp=0xfffffd7fa04d8360, free space=1016k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x1231cd3] void VMError::report(outputStream*)+0x8e7;; __1cHVMErrorGreport6MpnMoutputStream__v_+0x8e7
V [libjvm.so+0x1232ea2] void VMError::report_and_die()+0x506;; __1cHVMErrorOreport_and_die6M_v_+0x506
V [libjvm.so+0x8c8bb8] void report_vm_out_of_memory(const char*,int,unsigned long,const char*)+0x74;; __1cXreport_vm_out_of_memory6FpkciL1_v_+0x74
V [libjvm.so+0x558ccf] void*Arena::grow(unsigned long,AllocFailStrategy::AllocFailEnum)+0xb3;; __1cFArenaEgrow6MLnRAllocFailStrategyNAllocFailEnum__pv_+0xb3
V [libjvm.so+0x104d438] char*resource_allocate_bytes(unsigned long,AllocFailStrategy::AllocFailEnum)+0xb8;; __1cXresource_allocate_bytes6FLnRAllocFailStrategyNAllocFailEnum__pc_+0xb8
V [libjvm.so+0xb335a0] unsigned short*java_lang_String::as_unicode_string(oopDesc*,int&)+0x100;; __1cQjava_lang_StringRas_unicode_string6FpnHoopDesc_ri_pH_+0x100
V [libjvm.so+0x1111030] oopDesc*StringTable::intern(oopDesc*,Thread*)+0xa0;; __1cLStringTableGintern6FpnHoopDesc_pnGThread__2_+0xa0
V [libjvm.so+0xc52211] JVM_InternString+0xf5;; JVM_InternString+0xf5
C [libjava.so+0x18f25] Java_java_lang_String_intern+0x9;; Java_java_lang_String_intern+0x9
J java.lang.String.intern()Ljava/lang/String;
J gc.gctests.StringInternGC.StringInternGC$StringGenerator.run()V
v ~StubRoutines::call_stub
V [libjvm.so+0xb2dbbe] void JavaCalls::call_helper(JavaValue*,methodHandle*,JavaCallArguments*,Thread*)+0xa26;; __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_+0xa26
V [libjvm.so+0xb2a41f] void JavaCalls::call_virtual(JavaValue*,KlassHandle,Symbol*,Symbol*,JavaCallArguments*,Thread*)+0x45b;; __1cJJavaCallsMcall_virtual6FpnJJavaValue_nLKlassHandle_pnGSymbol_5pnRJavaCallArguments_pnGThread__v_+0x45b
V [libjvm.so+0xb2a808] void JavaCalls::call_virtual(JavaValue*,Handle,KlassHandle,Symbol*,Symbol*,Thread*)+0x60;; __1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_pnGSymbol_6pnGThread__v_+0x60
V [libjvm.so+0xc424b8] void thread_entry(JavaThread*,Thread*)+0xb8;; __1cMthread_entry6FpnKJavaThread_pnGThread__v_+0xb8
V [libjvm.so+0x1175acf] void JavaThread::run()+0x4f3;; __1cKJavaThreadDrun6M_v_+0x4f3
V [libjvm.so+0xf74b46] java_start+0x9d2;; java_start+0x9d2
C [libc.so.1+0x12257d] _thrp_setup+0xa5;; _thrp_setup+0xa5
C [libc.so.1+0x122820] _lwp_start+0x0;; _lwp_start+0x0

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J java.lang.String.intern()Ljava/lang/String;
J gc.gctests.StringInternGC.StringInternGC$StringGenerator.run()V
j nsk.share.runner.ThreadsRunner$ManagedThread.run()V+57
v ~StubRoutines::call_stub

All tests have similar stack:

V [libjvm.so+0x104d438] char*resource_allocate_bytes(unsigned long,AllocFailStrategy::AllocFailEnum)+0xb8;; __1cXresource_allocate_bytes6FLnRAllocFailStrategyNAllocFailEnum__pc_+0xb8
V [libjvm.so+0xb335a0] unsigned short*java_lang_String::as_unicode_string(oopDesc*,int&)+0x100;; __1cQjava_lang_StringRas_unicode_string6FpnHoopDesc_ri_pH_+0x100
V [libjvm.so+0x1111030] oopDesc*StringTable::intern(oopDesc*,Thread*)+0xa0;; __1cLStringTableGintern6FpnHoopDesc_pnGThread__2_+0xa0
V [libjvm.so+0xc52211] JVM_InternString+0xf5;; JVM_InternString+0xf5
C [libjava.so+0x18f25] Java_java_lang_String_intern+0x9;; Java_java_lang_String_intern+0x9
j java.lang.String.intern()Ljava/lang/String;+0

Should notice that some tests on interned strings passed:
vm/gc/compact/Compact_InternedStrings1
vm/gc/compact/Compact_Strings_InternedStrings
vm/gc/compact/Compact_TwoFields_InternedStrings
vm/gc/compact/Humongous_InternedStrings1