FULL PRODUCT VERSION :
java -version
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
Linux rmp-0075919300a78ce99-c-ea 3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 20 20:32:50 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

EXTRA RELEVANT SYSTEM CONFIGURATION :
point your primary nameserver in /etc/resolv.conf to nameserver that you have access to

A DESCRIPTION OF THE PROBLEM :
Do reverse DNS lookups in multiple threads. Bring down 'named' service in primary nameserver. DNS lookups will hang for 5s intermittently and often until nameserver is backup.



STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Run the following program to do reverse DNS lookup like

java Test 10.16.33.7

cat Test.java

import java.net.*;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class Test {
    static {
        java.security.Security.setProperty ("networkaddress.cache.ttl" , "1");
    }
    public static void main (String args[]) throws Exception {
        System.out.println("DEFAULT DNS TTL: "+java.security.Security.getProperty("networkaddress.cache.ttl"));
        ExecutorService executorService = Executors.newFixedThreadPool(4);
        while (true) {
            for (int i = 0; i < 4; i++) {
                executorService.submit(() -> {
                    try {
                        long start = System.currentTimeMillis();
                        System.out.println(InetAddress.getByName(args[0]).getHostName() + " " + (System.currentTimeMillis() - start));
                        //System.out.println(InetAddress.getAllByName(args[0])[0]+ " "+(System.currentTimeMillis() - start));
                        //System.out.println(InetAddress.getAllByName("10.16.36.12") + " "+(System.currentTimeMillis() - start));
                    } catch (Exception e) {
                        e.printStackTrace();
                    }
                });
            }
            Thread.sleep(2000);
System.out.println("");
        }
    }
}

This will print output like

DEFAULT DNS TTL: 1
ejp001ea.test.com 16
ejp001ea.test.com 9
ejp001ea.test.com 8
ejp001ea.test.com 8

ejp001ea.test.com 4
ejp001ea.test.com 4
ejp001ea.test.com 3
ejp001ea.test.com 4

ejp001ea.test.com 4
ejp001ea.test.com 3
ejp001ea.test.com 3
ejp001ea.test.com 3

ejp001ea.test.com 3
ejp001ea.test.com 3
ejp001ea.test.com 3
ejp001ea.test.com 4

ejp001ea.test.com 4
ejp001ea.test.com 3
ejp001ea.test.com 3
ejp001ea.test.com 3


Now login to primary nameserver and bring down named service.

sudo service named stop

OR run the following rule in nameserver

sudo iptables -A INPUT --src src_java_hostip -p udp --dport 53 -j REJECT


Now output will look like

ejp001ea.test.com 4
ejp001ea.test.com 3
ejp001ea.test.com 4

ejp001ea.test.com 5
ejp001ea.test.com 6
ejp001ea.test.com 5
ejp001ea.test.com 5



ejp001ea.test.com 5007
ejp001ea.test.com 5007
ejp001ea.test.com 5007
ejp001ea.test.com 5007
ejp001ea.test.com 2
ejp001ea.test.com 3
ejp001ea.test.com 3
ejp001ea.test.com 3
ejp001ea.test.com 2

ejp001ea.test.com 2


ejp001ea.test.com 5009
ejp001ea.test.com 5009
ejp001ea.test.com 5009
ejp001ea.test.com 2
ejp001ea.test.com 2
ejp001ea.test.com 5007
ejp001ea.test.com 3


ejp001ea.test.com 5008
ejp001ea.test.com 5006
ejp001ea.test.com 5006
ejp001ea.test.com 2
ejp001ea.test.com 2

ejp001ea.test.com 5006
ejp001ea.test.com 3

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Fails over to secondary nameserver without hanging for 5s.
ACTUAL -
Thread hangs for 5s

ERROR MESSAGES/STACK TRACES THAT OCCUR :
stacktrace looks like this

java.lang.Thread.State: RUNNABLE
        at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
        at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
        at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
        at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
        at java.net.InetAddress.getAllByName(InetAddress.java:1192)
        at java.net.InetAddress.getAllByName(InetAddress.java:1126)

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
import java.net.*;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class Test {
    static {
        java.security.Security.setProperty ("networkaddress.cache.ttl" , "1");
    }
    public static void main (String args[]) throws Exception {
        System.out.println("DEFAULT DNS TTL: "+java.security.Security.getProperty("networkaddress.cache.ttl"));
        ExecutorService executorService = Executors.newFixedThreadPool(4);
        while (true) {
            for (int i = 0; i < 4; i++) {
                executorService.submit(() -> {
                    try {
                        long start = System.currentTimeMillis();
                        System.out.println(InetAddress.getByName(args[0]).getHostName() + " " + (System.currentTimeMillis() - start));
                        //System.out.println(InetAddress.getAllByName(args[0])[0]+ " "+(System.currentTimeMillis() - start));
                        //System.out.println(InetAddress.getAllByName("10.16.36.12") + " "+(System.currentTimeMillis() - start));
                    } catch (Exception e) {
                        e.printStackTrace();
                    }
                });
            }
            Thread.sleep(2000);
System.out.println("");
        }
    }
}
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Do reverse DNS lookup in one thread.