First, this is the exact same behavior and cause as this bug in SWT that I reported:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=387980

I'll repeat it here, adapted for JavaFX ...

The zoom factors as reported by JavaFX in ZoomEvent.getZoomFactor() and ZoomEvent.getTotalZoomFactor() on OS X can become negative if you pinch (zoom out) sufficiently within a single gesture. These should obviously never be negative.

(This is an extreme example that the JavaFX reported total zoom factor doesn't correspond to the actual movement of the fingers. If you move the fingers from 10 cm apart to 5 cm apart, the total zoom factor should be around 0.5.)

The bug can easily be observed in this example code:
http://docs.oracle.com/javafx/2/events/GestureEventsExample.zip
The shapes flips inside out when pinched sufficiently.

The fix is also trivial in principle and, apart from confusingly named parameters and methods, can be fully implemented in GestureSupport.handleDeltaZooming(). I personally would however construct a correct relative scale factor (a.k.a. "multiplicative delta" in your implementation parlance) as close as possible to the native code by adding 1.0 to NSEvent.magnification.

The thing is, the documentation for NSEvent.magnification is badly written:
http://developer.apple.com/library/mac/documentation/cocoa/Reference/ApplicationKit/Classes/NSEvent_Class/Reference/Reference.html#//apple_ref/occ/instm/NSEvent/magnification

A better description, including example code, can be found at:
http://developer.apple.com/library/mac/documentation/cocoa/conceptual/EventOverview/HandlingTouchEvents/HandlingTouchEvents.html#//apple_ref/doc/uid/10000060i-CH13-SW17