$ jruby -J-showversion -X+C -J-d32 --server -J-Xbatch bench/bench_define_method_methods.rb
java version "1.7.0_02"
Java(TM) SE Runtime Environment (build 1.7.0_02-b12)
Java HotSpot(TM) Server VM (build 23.0-b06-internal, mixed mode)

                                                                  user system total real
control, simple method, 10k * 100 invocations 0.418000 0.000000 0.418000 ( 0.345000)
define_method(:foo) {1}, 10k * 100 invocations 0.954000 0.000000 0.954000 ( 0.955000)
eval'ed define_method(:baz) {1}, 10k * 100 invocations Formatter.java:2472:in `format': java.lang.NullPointerException
        from Formatter.java:2423:in `format'
        from String.java:2845:in `format'
        from RubyModule.java:439:in `calculateAnonymousName'
        from RubyModule.java:410:in `calculateName'
        from RubyModule.java:401:in `getName'
...

Disabling OptimizePtrCompare makes the test pass:

$ jruby -J-showversion -X+C -J-d32 --server -J-Xbatch -J-XX:-OptimizePtrCompare bench/bench_define_method_methods.rb
java version "1.7.0_02"
Java(TM) SE Runtime Environment (build 1.7.0_02-b12)
Java HotSpot(TM) Server VM (build 23.0-b06-internal, mixed mode)

                                                                  user system total real
control, simple method, 10k * 100 invocations 0.401000 0.000000 0.401000 ( 0.341000)
define_method(:foo) {1}, 10k * 100 invocations 0.976000 0.000000 0.976000 ( 0.976000)
eval'ed define_method(:baz) {1}, 10k * 100 invocations 2.715000 0.000000 2.715000 ( 2.715000)
define_method(:bar) {a = 1}, 10k * 100 invocations 0.779000 0.000000 0.779000 ( 0.779000)
b = 1; define_method(:baz) {b = 2}, 10k * 100 invocations 0.759000 0.000000 0.759000 ( 0.759000)
...