com/sun/jdi/JdwpAllowTest.java shows failures on AIX if prefixLen of mask is larger than 32 in IPv6 case

In parseAllowedMask in socketTransport.c, prefixLen of mask is compared with a maxValue (32 for IPv4, 128 otherwise). This fails if it is larger than 32, because getaddrinfo seems to detect IPv4 family, if IPv6 address has set only some of the last 32 Bits. So we take the wrong maxValue.


Example:
images/jdk/bin/java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:0,allow=0:0:0:0:0:0:10:0/106
Error in allow option: '106'
ERROR: transport error 103: invalid netmask in allow option
ERROR: JDWP Transport dt_socket failed to initialize, TRANSPORT_INIT(510)

Here is the final official statement from IBM regarding this issue:
================================================

The way that our name resolution code determines if an IP address is IPv4 or IPv6 is based on two macros.
#define IS_IPV4ADDR6(a) \
        (((a).s6_addr32[0] == 0) && \
         ((a).s6_addr32[1] == 0) && \
         ((a).s6_addr32[2] == 0xffff0000))

#define IS_COMPATADDR6(a) \
        (((a).s6_addr32[0] == 0) && \
         ((a).s6_addr32[1] == 0) && \
         ((a).s6_addr32[2] == 0) && \
         ((a).s6_addr32[3] > 1))
If the first eight bytes are zero and the next four bytes are 0xFFFF0000, it's considered a mapped IPv4 address. If the first twelve bytes of the address are zero, and the last four bytes are greater than 1, the code considers the IP address to be an IPv4-compatible IPv6 address. In both cases, getaddrinfo returns an h_addrtype of AF_INET.

I did some further research and found that this is the same behavior of the open source BIND code from the Internet Systems Consortium. Their macros are slightly different, but the functionality is the same.

Per https://users.isc.org/~each/doxygen/bind9/ipv6_8h-source.html:
/*%
 * IPv4 compatible
 */
#define IN6_IS_ADDR_V4COMPAT(a) \
        (((a)->s6_addr32[0] == 0) && \
         ((a)->s6_addr32[1] == 0) && \
         ((a)->s6_addr32[2] == 0) && \
         ((a)->s6_addr32[3] != 0) && \
         ((a)->s6_addr32[3] != htonl(1)))

/*%
 * Mapped
 */
#define IN6_IS_ADDR_V4MAPPED(a) \
        (((a)->s6_addr32[0] == 0) && \
         ((a)->s6_addr32[1] == 0) && \
         ((a)->s6_addr32[2] == htonl(0x0000ffff)))

As bizarre as this seems, AIX's behavior regarding IPv4/IPv6 determination via getaddrinfo matches up with ISC's code. (In fact, we get AIX's name resolution code from ISC.) This means the code is working as designed.

===============================
End of the official statement from IBM.