Now that we have NMT malloc limits (JDK-8291878), we can remove `MallocMaxTestWords`, save some atomic counts per malloc and some code.

However, since the behavior of MallocMaxTestWord is somewhat different, its not a 1:1 translation:

- MallocMaxTestWords caused malloc to return NULL, faking an OOM. NMT malloc limits cause a fatal error. So we need the ability to optionally return NULL instead of fatal error.
- MallocMaxTestWords did not count frees, so the arguments given need to be adjusted to the more correct way of NMT malloc limit accounting.