background radius don't work in windows 8 on 3D3Pipeline, but it is work on SWPipeline.
Code to reproduce :

 @Override
    public void start(Stage primaryStage) {
        FlowPane node = new FlowPane();
        node.setMinSize(100, 100);
        node.setStyle("-fx-background-color: green;"
                + "-fx-background-radius: 60px;");
        VBox root = new VBox(50);
        root.getChildren().add(node);
        
        Scene scene = new Scene(root, 300, 250);

        primaryStage.setTitle("Hello World!" + VersionInfo.getRuntimeVersion());
        primaryStage.setScene(scene);
        primaryStage.show();
    }


Prism verbose ouput :

Prism pipeline init order: d3d sw
Using t2k for text rasterization
Using native-based Pisces rasterizer
Using dirty region optimizations
Not using texture mask for primitives
Not forcing power of 2 sizes for textures
Using hardware CLAMP_TO_ZERO mode
Opting in for HiDPI pixel scaling
Prism pipeline name = com.sun.prism.d3d.D3DPipeline
Loading D3D native library ...
succeeded.
Direct3D initialization succeeded
(X) Got class = class com.sun.prism.d3d.D3DPipeline
Initialized prism pipeline: com.sun.prism.d3d.D3DPipeline
Maximum supported texture size: 4096
OS Information:
Windows version 6.2 build 9200
D3D Driver Information:
NVIDIA Quadro FX 1400
\\.\DISPLAY2
Driver nvd3dumx.dll, version 9.18.13.745
Pixel Shader version 3.0
Device : ven_10DE, dev_00CE, subsys_024310DE
 vsync: true vpipe: true