FULL PRODUCT VERSION :
java version "1.6.0_10"
Java(TM) SE Runtime Environment (build 1.6.0_10-b33)
Java HotSpot(TM) Server VM (build 11.0-b15, mixed mode)


FULL OS VERSION :
Linux 2.6.27-9-generic #1 SMP Thu Nov 20 21:57:00 UTC 2008 i686 GNU/Linux

A DESCRIPTION OF THE PROBLEM :
The javadoc of java.lang.instrument.Instrumentation.retransformClasses says:
"The retransformation may change method bodies, the constant pool and attributes. The retransformation must not add, remove or rename fields or methods, change the signatures of methods, or change inheritance. These restrictions maybe be lifted in future versions."

The JVMSpec says in $2.10.2:
"The signature of a method consists of the name of the method and the number and type of formal parameters (§2.10.1) of the method. A class may not declare two methods with the same signature."

  To my understanding this means that adding or removing the 'synchronized' of a method does not change the signature of this method. Though doing so results in an exception.

THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: Did not try

THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Did not try

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
(1) compile all files in package 'agent' and put them into a jar file
 (1.1) use the following manifest entries
Premain-Class="agent.Agent"
Agent-Class="agent.Agent"
Can-Redefine-Classes="true"
Can-Retransform-Classes="true"
Can-Set-Native-Method-Prefix="true"

(2) compile Main class
(3) start java with -javaagent option


EXPECTED VERSUS ACTUAL BEHAVIOR :
== EXPECTED ==
as removing/adding the 'synchronized' modifier is not part of the signature (according to the VM Spec), we expect the retransformation to succeed.
alternatively, putting a hint in the javadoc of retransformClasses what is meant by 'method signature' could help.

== ACTUAL ==
java.lang.UnsupportedOperationException: class redefinition failed: attempted to change method modifiers.

ERROR MESSAGES/STACK TRACES THAT OCCUR :

Caught UnsupportedOperationException
     [java] java.lang.UnsupportedOperationException: class redefinition failed: attempted to change method modifiers
     [java] at sun.instrument.InstrumentationImpl.retransformClasses0(Native Method)
     [java] at sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:124)
     [java] at agent.Rewriter.init(Rewriter.java:60)
     [java] at agent.Agent.init(Agent.java:129)
     [java] at agent.Agent.premain(Agent.java:138)
     [java] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     [java] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
     [java] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     [java] at java.lang.reflect.Method.invoke(Method.java:597)
     [java] at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:323)
     [java] at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:338)


REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
Note: this example uses ASM

== AGENT Code ==
public class Agent {

    private static Rewriter rw;

    private static void init( Instrumentation instr ) {
        rw = new Rewriter(instr);
        instr.addTransformer((ClassFileTransformer) rw, true);
        rw.init();
    }

    /**
     * @param args
     */
    public static void premain( String args, Instrumentation inst ) {
        try {
            init(inst);
        } catch (Throwable t) {
            System.err.println("Rewriter Agent caught exception: "
                    + t.getMessage());
            t.printStackTrace();
        }
    }

    public static void agentmain( String agentArgs, Instrumentation inst ) {
        try {
            init(inst);
        } catch (Throwable t) {
            System.err.println("Rewriter Agent catched exception: "
                    + t.getMessage());
            t.printStackTrace();
        }
    }
}

package agent;

import java.lang.instrument.ClassFileTransformer;
import java.lang.instrument.IllegalClassFormatException;
import java.lang.instrument.Instrumentation;
import java.lang.instrument.UnmodifiableClassException;
import java.security.ProtectionDomain;

import org.objectweb.asm.ClassAdapter;
import org.objectweb.asm.ClassReader;
import org.objectweb.asm.ClassWriter;

public class Rewriter implements ClassFileTransformer {

    private final Instrumentation instrument;

    /*
     * this method cannot be called from within the constructor, as the newly
     * created class might not be registered at the Instrumentation class yet
     */
    public void init() {

        try {
            instrument
                    .retransformClasses(new Class<?>[] { java.util.Hashtable.class });
        } catch (UnmodifiableClassException umc) {
            System.err.println("Caught UnmodifiableClassException!");
            umc.printStackTrace();
        } catch (UnsupportedOperationException uso) {
            System.err.println("Caught UnsupportedOperationException");
            uso.printStackTrace();
        }
    }

    public Rewriter( Instrumentation instr ) {
        instrument = instr;
    }

    public byte[] transform( ClassLoader loader, String className,
            Class<?> classBeingRedefined, ProtectionDomain protectionDomain,
            byte[] classfileBuffer ) throws IllegalClassFormatException {

        if (classBeingRedefined != null && className != null
                && className.startsWith("java/lang/Hashtable")) {

            ClassReader cr = new ClassReader(classfileBuffer);
            ClassWriter cw = new ClassWriter(cr, ClassWriter.COMPUTE_MAXS);
            ClassAdapter ca = new ObjectSynchronizationAdapter(cw, className);
            byte[] b = cw.toByteArray();

            try {
                cr.accept(ca, ClassReader.EXPAND_FRAMES);
            } catch (Throwable t) {
                System.out.println("Exception: " + t);
                t.printStackTrace();
            }
            return b;
        }
        return null;

    }
}

package agent;

import org.objectweb.asm.ClassAdapter;
import org.objectweb.asm.ClassVisitor;
import org.objectweb.asm.MethodVisitor;

public class ObjectSynchronizationAdapter extends ClassAdapter {
    private String className;

    public ObjectSynchronizationAdapter( ClassVisitor cv, String _className ) {
        super(cv);
        className = _className;
    }

    @Override
    public MethodVisitor visitMethod( int access, String method_name,
            String desc, String signature, String[] exceptions ) {
        MethodVisitor mv = super.visitMethod(access, method_name, desc,
                signature, exceptions);
        mv = new MethodSynchronizationAdapter(mv, access, method_name, desc,
                className);
        return mv;
    }
}

package agent;

import org.objectweb.asm.MethodVisitor;
import org.objectweb.asm.commons.AdviceAdapter;

public class MethodSynchronizationAdapter extends AdviceAdapter {

    public MethodSynchronizationAdapter( MethodVisitor mv, int acc,
            String name, String desc, String className ) {
        super(mv, acc & ~ACC_SYNCHRONIZED, name, desc);
    }
}

== Main Program ==
package test;

public class Main {

    public static void main( String[] args ) {

    }

}

---------- END SOURCE ----------