From a discussion on https://github.com/openjdk/jdk/pull/20202

Nobody talks about removing the decoder; I meant removing malloc from the decoder.
I was under the impression that would be very difficult to do (not that I've looked into it) - hence the acceptance of its use. A malloc-less decoder doesn't sound trivial.
 @tstuefe
Member
tstuefe commented 2 days ago
I still dislike the misuse of the enum. Among other things, this is not threadsafe,
I was under the impression the rest of the flag bits were immutable, hence no actual thread-safety issue for hs_err reporting. But yes in general this does make printFlags not-thread-safe. Your alternative bitmap design sounds reasonable.

Nobody talks about removing the decoder; I meant removing malloc from the decoder.
I was under the impression that would be very difficult to do (not that I've looked into it) - hence the acceptance of its use. A malloc-less decoder doesn't sound trivial.
A very simple idea could be to pre-allocate a "memory balloon" at the start of the JVM and use that one up at hs-err time. Possibly optionally, depending on how large the balloon is. But some dozen KB probably would not hurt in any case, and may solve the majority of malloc deadlocks we see in hs-err such that the remaining cases are so super rare that we don't care.

Done smartly, one would integrate this somehow into os::malloc such that, at the beginning of error reporting, we switch over to "balloon use" and use that one up until either error handling is done or the balloon is gone. The nice thing about this would be that it would work for every component.
 @dholmes-ora
Member
dholmes-ora commented yesterday
A very simple idea could be to pre-allocate a "memory balloon" ... Done smartly, one would integrate this somehow into os::malloc
In other words os::malloc becomes a custom allocator that has a small memory pool available for use during error reporting. That would be a general purpose solution ...
 @tstuefe
Member
tstuefe commented yesterday
A very simple idea could be to pre-allocate a "memory balloon" ... Done smartly, one would integrate this somehow into os::malloc
In other words os::malloc becomes a custom allocator that has a small memory pool available for use during error reporting. That would be a general purpose solution ...
In hs-err case - we expect a small amount of small mallocs - we could just allocate from the top of the balloon, and maybe allow re-adding the topmost allocation (to handle free of the very last malloc). That would not require a full-blown allocator that would allow arbitrary frees, which I agree would be over the top.
 @dholmes-ora
Member
dholmes-ora commented yesterday
Not sure how you could avoid a full blown allocator though if you expect to fall-back to malloc if the pool/balloon is empty, as you would need to track what free's relate to the pool/balloon.
 @tstuefe
Member
tstuefe commented yesterday •
Not sure how you could avoid a full blown allocator though if you expect to fall-back to malloc if the pool/balloon is empty, as you would need to track what free's relate to the pool/balloon.
Two solutions:

very very simple: just don't free during error reporting. It may lead to a bit of memory loss, but we should not malloc that much anyway. And arguably its the healthier option since if the libc is corrupted, it is actually more likely to crash or deadlock on free()
somewhat simple: Assuming the balloon is just a simple memory region, on os::free, if in error reporting, check if pointer points into the balloon. If yes, don't call free(). Costs two pointer comparisons.