When we define the same class several times in the same class loader, we end up with multiple instanceKlass objects which will be never deleted (see attached test). The offending code is in SystemDictionary::resolve_from_stream() where we don't take into account that find_or_define_instance_class()/define_instance_class() can throw an exception.

The fix is to call loader_data->add_to_deallocate_list(k()) for the instance class.

Notice that this is a regression from jdk7 which was introduced with the new metaspace implementation.
I also think there's a second leak when we are running with -XX:+AllowParallelDefineClass or -XX:+UnsyncloadClass an a parallel capable class loader (but I still have to extend the test to also cover that). In that case find_or_define_instance_class() may return a different instance of instanceKlass and again the old one will never be deleted. The fix is similarly trivial to the first one: check the result of find_or_define_instance_class() and call loader_data->add_to_deallocate_list(k()) in the cases where it returns a new instance.