Summary

Create a new public javafx.scene.robot.Robot API based on the existing internal com.sun.glass.ui.Robot class.

Problem

The JavaFX Robot class is currently in an internal (not exported) package as com.sun.glass.ui.Robot. AWT has a public robot API java.awt.Robot as it is very useful for creating GUI tests. As an example, JavaFX uses the robot for its own internal tests. The TestFX third-party library currently uses the robot by adding the appropriate --add-exports arguments but this is not ideal. It is possible to try and use the existing public API of JavaFX to create a robot by utilizing Event.fireEvent but this is not ideal because it requires having a scene instance. The purpose of a robot is to do "global" actions at the OS level (key presses, mouse movements, screen captures, etc) independent of the JavaFX scene graph.

Solution

Move com.sun.glass.ui.Robot to javafx.scene.robot.Robot as a new addition to the JavaFX public API. Segregate the internal private robot API from the public one as much as possible by using the peer class com.sun.glass.ui.GlassRobot. Fix some inconsistencies between the native robot implementations so that the public API acts the same on all supported platforms. Convert JavaFX's internal tests to use the new public robot API.

Specification

javafx/scene/robot/Robot.java

package javafx.scene.robot;

/**
 * A {@code Robot} is used for simulating user interaction such as
 * typing keys on the keyboard and using the mouse as well as capturing
 * graphical information without requiring a {@link javafx.scene.Scene}
 * instance. Robot objects must be constructed and used on the JavaFX
 * Application Thread.
 *
 * @since 11
 */
public final class Robot {

    /**
     * Constructs a new {@code Robot} that can be used for simulating user
     * interactions.
     * If a security manager is present, the application must have the
     * {@link javafx.util.FXPermission} {@code "createRobot"} permission in order to construct
     * a {@code Robot} object.
     *
     * @throws IllegalStateException if this object is constructed on a thread
     * other than the JavaFX Application Thread.
     * @throws SecurityException if a security manager exists and the application does not have the
     * @link javafx.util.FXPermission} {@code "createRobot"} permission
     */
    public Robot() {}

    /**
     * Presses the specified {@link KeyCode} key.
     *
     * @param keyCode the key to press
     * @throws IllegalStateException if this method is called on a thread
     * other than the JavaFX Application Thread.
     */
    public void keyPress(KeyCode keyCode) {}

    /**
     * Releases the specified {@link KeyCode} key.
     *
     * @param keyCode the key to release
     * @throws IllegalStateException if this method is called on a thread
     * other than the JavaFX Application Thread.
     */
    public void keyRelease(KeyCode keyCode) {}

    /**
     * Types the specified {@link KeyCode} key.
     *
     * @implSpec This is a convenience method that is equivalent to calling
     * {@link #keyPress(KeyCode)} followed by {@link #keyRelease(KeyCode)}.
     * @param keyCode the key to type
     * @throws IllegalStateException if this method is called on a thread
     * other than the JavaFX Application Thread.
     */
    public void keyType(KeyCode keyCode) {}

    /**
     * Returns the current mouse {@code x} position in screen coordinates.
     *
     * @return the current mouse {@code x} position
     * @throws IllegalStateException if this method is called on a thread
     * other than the JavaFX Application Thread.
     */
    public double getMouseX() {}

    /**
     * Returns the current mouse {@code y} position in screen coordinates.
     *
     * @return the current mouse {@code y} position
     * @throws IllegalStateException if this method is called on a thread
     * other than the JavaFX Application Thread.
     */
    public double getMouseY() {}

    /**
     * Returns the current mouse (x,y) screen coordinates as a {@link Point2D}.
     *
     * @return the current mouse (x,y) screen coordinates
     * @throws IllegalStateException if this method is called on a thread
     * other than the JavaFX Application Thread.
     */
    public Point2D getMousePosition() {}

    /**
     * Moves the mouse to the specified (x,y) screen coordinates.
     *
     * @param x screen coordinate x to move the mouse to
     * @param y screen coordinate y to move the mouse to
     * @throws IllegalStateException if this method is called on a thread
     * other than the JavaFX Application Thread.
     */
    public void mouseMove(double x, double y) {}

    /**
     * Moves the mouse to the (x,y) screen coordinates specified by the
     * given {@code location}.
     *
     * @param location the (x,y) coordinates to move the mouse to
     * @throws IllegalStateException if this method is called on a thread
     * other than the JavaFX Application Thread.
     */
    public final void mouseMove(Point2D location) {}

    /**
     * Presses the specified {@link MouseButton}s.
     *
     * @param buttons the mouse buttons to press
     * @throws IllegalStateException if this method is called on a thread
     * other than the JavaFX Application Thread.
     */
    public void mousePress(MouseButton... buttons) {}

    /**
     * Releases the specified {@link MouseButton}s.
     *
     * @param buttons the mouse buttons to release
     * @throws IllegalStateException if this method is called on a thread
     * other than the JavaFX Application Thread.
     */
    public void mouseRelease(MouseButton... buttons) {}

    /**
     * Clicks the specified {@link MouseButton}s.
     *
     * @implSpec This is a convenience method that is equivalent to calling
     * {@link #mousePress(MouseButton...)} followed by {@link #mouseRelease(MouseButton...)}.
     * @param buttons the mouse buttons to click
     * @throws IllegalStateException if this method is called on a thread
     * other than the JavaFX Application Thread.
     */
    public void mouseClick(MouseButton... buttons) {}

    /**
     * Scrolls the mouse wheel by the specified amount. Positive {@code wheelAmt}
     * scrolls up whereas negative scrolls down.
     *
     * @param wheelAmt the (signed) amount to scroll the wheel
     * @throws IllegalStateException if this method is called on a thread
     * other than the JavaFX Application Thread.
     */
    public void mouseWheel(int wheelAmt) {}

    /**
     * Returns the {@link Color} of the pixel at the specified screen coordinates of the
     * primary screen.
     *
     * @param x the x coordinate to get the pixel color from
     * @param y the y coordinate to get the pixel color from
     * @return the pixel color at the specified screen coordinates
     * @throws IllegalStateException if this method is called on a thread
     * other than the JavaFX Application Thread.
     */
    public Color getPixelColor(double x, double y) {}

    /**
     * Returns the {@link Color} of the pixel at the screen coordinates relative to the
     * primary screen specified by {@code location}.
     *
     * @param location the (x,y) coordinates to get the pixel color from
     * @return the pixel color at the specified screen coordinates
     * @throws NullPointerException if the given {@code location} is {@literal null}
     * @throws IllegalStateException if this method is called on a thread
     * other than the JavaFX Application Thread.
     */
    public Color getPixelColor(Point2D location) {}

    /**
     * Returns a {@link WritableImage} containing the specified rectangular area of the
     * primary screen. If the given {@code image} is {@literal null}, or if the given
     * {@code image} is too small, a new {@code WritableImage} will be created and returned.
     * Otherwise, the given {@code image} is re-used.
     * <p>
     * If the {@code scaleToFit} argument is {@literal false}, the returned
     * {@code Image} object dimensions may differ from the requested {@code width}
     * and {@code height} depending on how many physical pixels the area occupies
     * on the screen. For example, in HiDPI mode on the Mac (aka Retina display) the
     * pixels are doubled, and thus a screen capture of an area of size (10x10) pixels
     * will result in an {@code Image} with dimensions (20x20). Calling code should
     * use the returned images's {@link Image#getWidth()} and {@link Image#getHeight()}
     * methods to determine the actual image size.
     * <p>
     * If {@code scaleToFit} is {@literal true}, the returned {@code Image} is of
     * the requested size. Note that in this case the image will be scaled in
     * order to fit to the requested dimensions if necessary, such as when running
     * on a HiDPI display.
     *
     * @param image either {@literal null} or a {@code WritableImage} that will
     * be used to place the screen capture in
     * @param x the starting x-position of the rectangular area to capture
     * @param y the starting y-position of the rectangular area to capture
     * @param width the width of the rectangular area to capture
     * @param height the height of the rectangular area to capture
     * @param scaleToFit If {@literal true}, the returned {@code Image} will be
     * scaled to fit the request dimensions (if necessary). Otherwise, the size
     * of the returned image will depend on the output scale (DPI) of the primary
     * screen.
     * @return the screen capture of the specified {@code region} as a {@link WritableImage}
     * @throws IllegalStateException if this method is called on a thread
     * other than the JavaFX Application Thread.
     */
    public WritableImage getScreenCapture(WritableImage image, double x, double y,
                                          double width, double height, boolean scaleToFit) {}

    /**
     * Returns a {@link WritableImage} containing the specified rectangular area of the
     * primary screen. If the given {@code image} is {@literal null}, or if the given
     * {@code image} is too small, a new {@code WritableImage} will be created and returned.
     * Otherwise, the given {@code image} is re-used.
     *
     * @implSpec This method is equivalent to calling {@code getScreenCapture(x, y, width, height, true)},
     * that is, this method scales the image to fit the requested size.
     * @param image either {@literal null} or a {@code WritableImage} that will
     * be used to place the screen capture in
     * @param x the starting x-position of the rectangular area to capture
     * @param y the starting y-position of the rectangular area to capture
     * @param width the width of the rectangular area to capture
     * @param height the height of the rectangular area to capture
     * @return the screen capture of the specified {@code region} as a {@link WritableImage}
     * @throws IllegalStateException if this method is called on a thread
     * other than the JavaFX Application Thread.
     */
    public WritableImage getScreenCapture(WritableImage image, double x, double y,
                                          double width, double height) {}

    /**
     * Returns a {@link WritableImage} containing the specified rectangular area of the
     * primary screen. If the given {@code image} is {@literal null}, or if the given
     * {@code image} is too small, a new {@code WritableImage} will be created and returned.
     * Otherwise, the given {@code image} is re-used.
     *
     * @implSpec This method is equivalent to calling {@code getScreenCapture(image, region, true)},
     * that is, this method scales the image to fit the requested size.
     * @param image either {@literal null} or a {@code WritableImage} that will
     * be used to place the screen capture in
     * @param region the rectangular area of the screen to capture
     * @return the screen capture of the specified {@code region} as a {@link WritableImage}
     * @throws IllegalStateException if this method is called on a thread
     * other than the JavaFX Application Thread.
     */
    public WritableImage getScreenCapture(WritableImage image, Rectangle2D region) {}

    /**
     * Returns a {@link WritableImage} containing the specified rectangular area of the
     * primary screen. If the given {@code image} is {@literal null}, or if the given
     * {@code image} is too small, a new {@code WritableImage} will be created and returned.
     * Otherwise, the given {@code image} is re-used.
     * <p>
     * If the {@code scaleToFit} argument is {@literal false}, the returned
     * {@code Image} object dimensions may differ from the requested {@code width}
     * and {@code height} depending on how many physical pixels the area occupies
     * on the screen. For example, in HiDPI mode on the Mac (aka Retina display) the
     * pixels are doubled, and thus a screen capture of an area of size (10x10) pixels
     * will result in an {@code Image} with dimensions (20x20). Calling code should
     * use the returned images's {@link Image#getWidth()} and {@link Image#getHeight()}
     * methods to determine the actual image size.
     * <p>
     * If {@code scaleToFit} is {@literal true}, the returned {@code Image} is of
     * the requested size. Note that in this case the image will be scaled in
     * order to fit to the requested dimensions if necessary, such as when running
     * on a HiDPI display.
     *
     * @param image either {@literal null} or a {@code WritableImage} that will
     * be used to place the screen capture in
     * @param region the rectangular area of the screen to capture
     * @param scaleToFit if {@literal true}, the returned {@code Image} will be
     * scaled to fit the request dimensions (if necessary). Otherwise, the size
     * of the returned image will depend on the output scale (DPI) of the primary
     * screen.
     * @return the screen capture of the specified {@code region} as a {@link WritableImage}
     * @throws IllegalStateException if this method is called on a thread
     * other than the JavaFX Application Thread.
     */
    public WritableImage getScreenCapture(WritableImage image, Rectangle2D region, boolean scaleToFit) {}
}