Provide a work-around to broken Linux 32 bit "Exec Shield" using CS for NX emulation (crashing with SI_KERNEL)

XMLWordPrintable

    • b53
    • x86
    • linux

        JDK-8015837 contains several crashes on 32 bit Linux due to a bug a patch from RedHat, part of "Exec Shield".

        RHEL 5 & 6 are affected, and earlier Ubuntu releases 10.04 (LTS), 11.04, and 12.04 have also taken in the patch. Whilst earlier Exec Shield patches had a switch, current incantations have no work-around, no way to disable.

        Details of the problem see RH bug filed: https://bugzilla.redhat.com/show_bug.cgi?id=996149

        To summarize: to emulate NX feature on X86_32 code segment is used to limit execution to the highest executable VA. There is a tiny race on SMP MM invalidation code which can cause the lazy CS update code in trap handling to think a general protection fault wasn't cause by itself. This results in sending the JVM a useless SIGSEGV with si_code:SI_KERNEL, results in JVM signal handling forcing a dump.

        The suggested work around (limited to 32 bit Linux): is to enable execution (PROT_EXEC) on a high address and execute some code.

        The memory layout on these distros is similar: TASK_LIMIT set to 3G (0xC0000000), and the main stack typically grows down from around here, so:
        1) using a page in main stack at JVM start up, test for said layout (don't bother if its not what we expect, i.e. exotic Linux config/distro)
        2) enable execution (mprotect)
        3) Place a single "ret" instruction and call it.

        This will cause CS to be updated to a high enough address to basically disable any further updates (due to code generation), effectively disabling the feature (which is already of questionable value, NX is the way to go).

              Assignee:
              David Simms
              Reporter:
              David Simms
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: