Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8164900

Add support for O_DIRECT

XMLWordPrintable

    • Icon: Enhancement Enhancement
    • Resolution: Fixed
    • Icon: P3 P3
    • 10
    • None
    • core-libs
    • None
    • b29
    • generic
    • linux

      This is a proposal for adding O_DIRECT support into JDK 9 for reading/writing from/to a file on Linux platform. O_DIRECT is a file-open flag to pass to OPEN(2). It tries to minimize cache effects of the I/O to and from this file. File I/O is done directly to/from user-space buffers. Please refer to http://man7.org/linux/man-pages/man2/open.2.html for more details.

      The advantage of this feature is to provide consistent and predictable IO throughput, bypassing the influence of Linux filesystem cache. It is useful for applications which already have their own caching layers, such as databases Apache HBase and Apache Cassandra. In addition, for applications like Apache Spark shuffle phase that have random file accesses and no access locality (read the file only once from a random location), this feature also helps performance by minimizing trashing the filesystem cache.

      The proposal adds 4 API methods to java/io/FileInputStream.java, java/io/OutputStream.java to enable the feature. In addition, it add O_DIRECT with two more modes “ro” (read-only and direct) and “rwo” (read-write and direct) to java/io/RandomAccessFile.java.

      public FileInputStream(String name, boolean direct) throws FileNotFoundException

      public FileInputStream(File file, boolean direct) throws FileNotFoundException

      public FileOutputStream(String name, boolean append, boolean direct) throws FileNotFoundException

      public FileOutputStream(File file, boolean append, boolean direct) throws FileNotFoundException

      Many database applications written in C/C++ such as MySQL and MongoDB have already used O_DIRECT for certain DB operations. Many Java based BigData applications such as Cassandra are already using it from third party library named JNA. (https://github.com/java-native-access/jna) Our internal version of Hadoop Distributed File System (HDFS) that has O_DIRECT enabled via JNI functions also show significant throughput improvements.

      The initial version of the JDK9 patch is available at http://cr.openjdk.java.net/~vdeshpande/8164900/webrev.00/ . By using this patch, we saw good results up to 1.8x improvement on random read throughput on a micro workload named IOMeter with both SATA SSDs and Intel PCIeSSDs. The workload is available at https://github.com/persado/iometer/blob/master/src/main/java/com/persado/oss/iometer/IOMeter.java

      Please review the patch and let us know your feedback.

      Code contributed by Lucy Lu (yingqi.lu@intel.com)

            bpb Brian Burkhalter
            vdeshpande Vivek Deshpande (Inactive)
            Votes:
            1 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - 4 weeks
                4w
                Remaining:
                Remaining Estimate - 2 days
                2d
                Logged:
                Time Spent - Not Specified Time Not Required
                Not Specified