There is a huge native memory leak in awt code on Mac OS X. It was first identified by running NetBeans on JDK 8. The leak can be also reproduced using SwingSet2 with the following steps:
1) Run SwingSet2 demo on JDK 8u5 or JDK 9
2) move mouse over the SwingSet2 window - for example make circles over the window

the leak can be seen Activity Monitor and in details in vmmap output. The following output is captured, while moving mouse over SwingSet2 window.
bobik:~ thurka$ jps
33784 SwingSet2.jar
33786 Jps
bobik:~ thurka$ date; vmmap 33784 | grep ^DefaultMallocZone
Wed Apr 23 09:53:52 CEST 2014
DefaultMallocZone_0x109d33000 48.2M 78842 32.8M 68%
bobik:~ thurka$ date; vmmap 33784 | grep ^DefaultMallocZone
Wed Apr 23 09:54:12 CEST 2014
DefaultMallocZone_0x109d33000 239.2M 3646723 217.8M 91%
bobik:~ thurka$ date; vmmap 33784 | grep ^DefaultMallocZone
Wed Apr 23 09:54:32 CEST 2014
DefaultMallocZone_0x109d33000 466.2M 7873686 437.3M 93%
bobik:~ thurka$ date; vmmap 33784 | grep ^DefaultMallocZone
Wed Apr 23 09:55:11 CEST 2014
DefaultMallocZone_0x109d33000 928.2M 16488624 883.2M 95%
bobik:~ thurka$ date; vmmap 33784 | grep ^DefaultMallocZone
Wed Apr 23 09:55:44 CEST 2014
DefaultMallocZone_0x109d33000 1.3G 23483264 1.2G 95%

As you can see DefaultMallocZone of SwingSet2 application raised to 1G in two minutes.