The MXBean spec does not explicitly say that recursive data structures such as the following cannot be mapped into Open Types:

public class Node {
    @PropertyNames({"adjacent"})
    public Node(Set<Node> adjacent) {
        this.adjacent = adjacent;
    }
    public Set<Node> getAdjacent() {
        return adjacent;
    }
    private final Set<Node> adjacent;
}

But they can't, basically because recursive data structures imply cycles, and immutable types such as CompositeType and CompositeData can't be used to build cycles.

The spec should explicitly say that recursive data structures are forbidden, and suggest workarounds. Typically, a workaround is to add a level of indirection, like this:

public class Node {
    @PropertyNames({"name", "adjacent"})
    public Node(String name, Set<String> adjacent) {
        this.name = name;
        this.adjacent = adjacent;
    }
    ...
}
public class Graph {
    @PropertyNames({"nodeNames"})
    public Graph(Map<String, Node> nodeNames) {
        ...
    }
    ...
}