I was looking at extending Abbot/Costello to work with at least WebView components embedded in Java; but noticed that there isn't a way to make mouse co-ordinates into some sort of location in the displayed page.

Ideally you would be able to go from mouse->dom->offset and back. This makes it possible to record a click and abstract what the user has done. Currently there isn't enough information in the API to do this.