-
Bug
-
Resolution: Unresolved
-
P3
-
17.0.1
-
generic
-
generic
ADDITIONAL SYSTEM INFORMATION :
x86_64 Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
Linux 5.15.59-gentoo
OpenJDK Runtime Environment Temurin-17.0.1+12 (build 17.0.1+12)
A DESCRIPTION OF THE PROBLEM :
After about a month of running application with kafka producer library we start to constantly get the following exception:
java.lang.IllegalMonitorStateException: current thread is not owner
at java.base/java.lang.Object.wait(Native Method)
at org.apache.kafka.common.utils.SystemTime.waitObject(SystemTime.java:55)
at org.apache.kafka.clients.producer.internals.ProducerMetadata.awaitUpdate(ProducerMetadata.java:119)
at org.apache.kafka.clients.producer.KafkaProducer.waitOnMetadata(KafkaProducer.java:1048)
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:907)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:886)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:774)
despite the fact that the method Object.wait is in a double synchronization block.
here is a part of application log with timestamps:
app start = 2022-11-16T10:28:16.080Z[UTC] (1668594496080 millis)
kafka err = 2022-12-03T03:54:49.080Z[UTC] (1445193 stamp)
monitor = 2022-12-03T04:02:16.080Z[UTC] (1445640 stamp)
[2022/12/03 03:54:47.100] (ERROR): <org.apache.kafka.clients.producer.internals.Sender> [Producer clientId=altair.1, transactionalId=altair.1] Aborting producer batches due to fatal error
org.apache.kafka.common.errors.ProducerFencedException: There is a newer producer with the same transactionalId which fences the current one.
[2022/12/03 03:54:47.100] (WARNING): <KafkaDispatcher> unable to abort kafka transaction
org.apache.kafka.common.errors.ProducerFencedException: The producer has been rejected from the broker because it tried to use an old epoch with the transactionalId
[2022/12/03 03:54:47.102] (WARNING): <KafkaDispatcher> error occurred while processing request
org.apache.kafka.common.errors.ProducerFencedException: There is a newer producer with the same transactionalId which fences the current one.
[2022/12/03 04:02:17.338] (WARNING): <KafkaDispatcher> error occurred while processing request
java.lang.IllegalMonitorStateException: current thread is not owner
at java.base/java.lang.Object.wait(Native Method)
at org.apache.kafka.common.utils.SystemTime.waitObject(SystemTime.java:55)
at org.apache.kafka.clients.producer.internals.ProducerMetadata.awaitUpdate(ProducerMetadata.java:119)
at org.apache.kafka.clients.producer.KafkaProducer.waitOnMetadata(KafkaProducer.java:1048)
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:907)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:886)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:774)
REGRESSION : Last worked in version 8
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
The application works find about a month but when we receive some error from kafka library (1445193 timestamp in jit log) JIT compiles method KafkaProducer::waitOnMetadata and shortly after that (1445640 timestamp in jit log) when the problem in kafka layer disappears JIT re-compiles the code again and we start to constantly get IllegalMonitorStateException.
EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
process successfully waits on monitor object
ACTUAL -
IllegalMonitorStateException
---------- BEGIN SOURCE ----------
======== ProducerMetadata ===========
public synchronized void awaitUpdate(final int lastVersion, final long timeoutMs) throws InterruptedException {
long currentTimeMs = time.milliseconds();
long deadlineMs = currentTimeMs + timeoutMs < 0 ? Long.MAX_VALUE : currentTimeMs + timeoutMs;
time.waitObject(this, () -> {
// Throw fatal exceptions, if there are any. Recoverable topic errors will be handled by the caller.
maybeThrowFatalException();
return updateVersion() > lastVersion || isClosed();
}, deadlineMs);
if (isClosed())
throw new KafkaException("Requested metadata update after close");
}
======== SystemTime ==================
public void waitObject(Object obj, Supplier<Boolean> condition, long deadlineMs) throws InterruptedException {
synchronized (obj) {
while (true) {
if (condition.get())
return;
long currentTimeMs = milliseconds();
if (currentTimeMs >= deadlineMs)
throw new TimeoutException("Condition not satisfied before deadline");
obj.wait(deadlineMs - currentTimeMs);
}
}
}
---------- END SOURCE ----------
CUSTOMER SUBMITTED WORKAROUND :
To disable JIT.
FREQUENCY : rarely
x86_64 Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
Linux 5.15.59-gentoo
OpenJDK Runtime Environment Temurin-17.0.1+12 (build 17.0.1+12)
A DESCRIPTION OF THE PROBLEM :
After about a month of running application with kafka producer library we start to constantly get the following exception:
java.lang.IllegalMonitorStateException: current thread is not owner
at java.base/java.lang.Object.wait(Native Method)
at org.apache.kafka.common.utils.SystemTime.waitObject(SystemTime.java:55)
at org.apache.kafka.clients.producer.internals.ProducerMetadata.awaitUpdate(ProducerMetadata.java:119)
at org.apache.kafka.clients.producer.KafkaProducer.waitOnMetadata(KafkaProducer.java:1048)
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:907)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:886)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:774)
despite the fact that the method Object.wait is in a double synchronization block.
here is a part of application log with timestamps:
app start = 2022-11-16T10:28:16.080Z[UTC] (1668594496080 millis)
kafka err = 2022-12-03T03:54:49.080Z[UTC] (1445193 stamp)
monitor = 2022-12-03T04:02:16.080Z[UTC] (1445640 stamp)
[2022/12/03 03:54:47.100] (ERROR): <org.apache.kafka.clients.producer.internals.Sender> [Producer clientId=altair.1, transactionalId=altair.1] Aborting producer batches due to fatal error
org.apache.kafka.common.errors.ProducerFencedException: There is a newer producer with the same transactionalId which fences the current one.
[2022/12/03 03:54:47.100] (WARNING): <KafkaDispatcher> unable to abort kafka transaction
org.apache.kafka.common.errors.ProducerFencedException: The producer has been rejected from the broker because it tried to use an old epoch with the transactionalId
[2022/12/03 03:54:47.102] (WARNING): <KafkaDispatcher> error occurred while processing request
org.apache.kafka.common.errors.ProducerFencedException: There is a newer producer with the same transactionalId which fences the current one.
[2022/12/03 04:02:17.338] (WARNING): <KafkaDispatcher> error occurred while processing request
java.lang.IllegalMonitorStateException: current thread is not owner
at java.base/java.lang.Object.wait(Native Method)
at org.apache.kafka.common.utils.SystemTime.waitObject(SystemTime.java:55)
at org.apache.kafka.clients.producer.internals.ProducerMetadata.awaitUpdate(ProducerMetadata.java:119)
at org.apache.kafka.clients.producer.KafkaProducer.waitOnMetadata(KafkaProducer.java:1048)
at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:907)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:886)
at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:774)
REGRESSION : Last worked in version 8
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
The application works find about a month but when we receive some error from kafka library (1445193 timestamp in jit log) JIT compiles method KafkaProducer::waitOnMetadata and shortly after that (1445640 timestamp in jit log) when the problem in kafka layer disappears JIT re-compiles the code again and we start to constantly get IllegalMonitorStateException.
EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
process successfully waits on monitor object
ACTUAL -
IllegalMonitorStateException
---------- BEGIN SOURCE ----------
======== ProducerMetadata ===========
public synchronized void awaitUpdate(final int lastVersion, final long timeoutMs) throws InterruptedException {
long currentTimeMs = time.milliseconds();
long deadlineMs = currentTimeMs + timeoutMs < 0 ? Long.MAX_VALUE : currentTimeMs + timeoutMs;
time.waitObject(this, () -> {
// Throw fatal exceptions, if there are any. Recoverable topic errors will be handled by the caller.
maybeThrowFatalException();
return updateVersion() > lastVersion || isClosed();
}, deadlineMs);
if (isClosed())
throw new KafkaException("Requested metadata update after close");
}
======== SystemTime ==================
public void waitObject(Object obj, Supplier<Boolean> condition, long deadlineMs) throws InterruptedException {
synchronized (obj) {
while (true) {
if (condition.get())
return;
long currentTimeMs = milliseconds();
if (currentTimeMs >= deadlineMs)
throw new TimeoutException("Condition not satisfied before deadline");
obj.wait(deadlineMs - currentTimeMs);
}
}
}
---------- END SOURCE ----------
CUSTOMER SUBMITTED WORKAROUND :
To disable JIT.
FREQUENCY : rarely
- relates to
-
JDK-8276560 nsk/jvmti/scenarios/capability/CM01/cm01t007 failed with IllegalMonitorStateException
- Closed