-
Bug
-
Resolution: Fixed
-
P3
-
8, 11, 17, 21, 22
-
b22
-
generic
-
generic
ADDITIONAL SYSTEM INFORMATION :
All systems and versions, as far as I know
A DESCRIPTION OF THE PROBLEM :
Some MIDI files contain a sequence of events that I call "Interrupted Running Status":
1. MIDI Channel event with status byte
2. (optional) more MIDI channel events without status byte (using running status)
3. Meta or SysEx event
4. Event without status byte
The intention of the file creators is that the non-status event (4) uses the running status from the channel event (1) before the interruption.
I have looked at hundreds of real-world MIDI files with this behaviour, and I'm sure that this assumption is correct.
But I'm open to discuss this further, if requested.
The SMFParser interprets these files incorrectly, as soon as it reads the non-status byte (4).
- It reads the non-status byte
- It assumes that the status byte of the preceding Meta/SysEx must also be used as the status byte for the next event.
- It silently ignores the non-status byte (4)
- It interprets the next bytes as a part of the assumed Meta or SysEx message.
This is clearly a bug, for the following reasons:
- Running status is only allowed for channel commands, according to the SMF specification.
- In the specification it is clearly stated that running status must not be used for Meta or SysEx events.
- A byte must never be silently ignored.
- It is not what the file creators expect.
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Run the test case below. It contains, parses and prints the content of 3 small MIDI files.
Each file contains the following sequence:
- 90 3c 7f: Note-ON (C)
- ff 01 04 54 65 73 74: Meta Text (Test)
- 90 3c 00: Note-OFF (C)
- 90 3e 7f: Note-ON (D)
- 90 3e 00: Note-OFF (D)
- ff 01 04 54 65 73 74: Meta Text (Test)
- ff 2f 00: Meta (End of Track)
File 1 starts every message with a status byte.
File 2 uses (interrupted) running status for all channel events apart from the first one.
File 3 is nearly identical to file 2 - only a single delta time has a different value.
EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Two possible results can be regarded as valid: A strict solution and a tolerant one.
Strict solution:
The SMF specification contains this sentence: "Sysex events and meta-events cancel any running status that was in effect."
That means, all files with an interrupted running status are illegal.
A Meta or SysEx event should reset the running status to 0.
For all interrupted running statuses this will always result in a reliable InvalidMidiDataException.
In the provided test case that means, only the first file can be parsed. File 2 and 3 fail with InvalidMidiDataException.
Tolerant solution:
Only channel events should update the running status, but not Meta or SysEx events.
If an interrupted running status is found, use the status byte of the last channel event before the interrupting event.
For the provided test case that means, all 3 files can be parsed and produce the same output on the command line.
I would prefer to use the tolerant solution.
I have already implemented an according bugfix and inspected the resulting data from many real-world MIDI files.
The resulting data makes sense in the vast majority of files, apart from just a few that are completely broken.
I will create a pull request and discuss my arguments there or on the mailing list client-libs-dev, if needed.
ACTUAL -
The non-status byte is ignored. The next bytes are interpreted as if they belong to a Meta or SysEx message. So it depends on the following bytes what happens:
- InvalidMidiDataException
- EOFException
- No exception but misinterpreted messages
In the provided test case, only file 1 is parsed correctly.
File 2 fails with an InvalidMidiDataException
File 3 does not fail but is interpreted incorrectly. Three channel messages are missing but instead we have an additional Meta message without any sense.
---------- BEGIN SOURCE ----------
import java.io.ByteArrayInputStream;
import javax.sound.midi.MidiSystem;
import javax.sound.midi.Sequence;
import javax.sound.midi.Track;
public class BugreportIRS {
public static void main(String[] args) {
int[][] files = new int[][] {SMF_1, SMF_2, SMF_3};
for (int i = 0; i < files.length; i++) {
byte[] content = toByteArray(files[i]);
System.out.println("processing file " + (i+1));
Sequence seq;
try {
seq = MidiSystem.getSequence(new ByteArrayInputStream(content));
} catch (Exception e) {
System.out.println("...failed with exception " + e);
System.out.println();
continue;
}
System.out.println("messages:");
Track track = seq.getTracks()[0];
for (int j = 0; j < track.size(); j++) {
byte[] msg = track.get(j).getMessage().getMessage();
for (byte b : msg) {
System.out.print(Integer.toHexString(b & 0xFF) + " ");
}
System.out.println();
}
System.out.println();
}
}
private static byte[] toByteArray(int[] integers) {
byte[] bytes = new byte[integers.length];
for (int i = 0; i < integers.length; i++) {
bytes[i] = (byte) integers[i];;
}
return bytes;
}
// MIDI file without running status - should work equally before and after the bugfix
private static int[] SMF_1 = {
0x4D, 0x54, 0x68, 0x64, 0x00, 0x00, 0x00, 0x06, 0x00, 0x01, 0x00, 0x01, 0x00, 0x80, // file header
0x4D, 0x54, 0x72, 0x6B, 0x00, 0x00, 0x00, 0x24, // track header
0x00, 0x90, 0x3C, 0x7F, // delta time + Note-ON (C)
0x40, 0xFF, 0x01, 0x04, 0x54, 0x65, 0x73, 0x74, // delta time + META message (text)
0x20, 0x90, 0x3C, 0x00, // delta time + Note-OFF (C)
0x20, 0x90, 0x3E, 0x7F, // delta time + Note-ON (D)
0x60, 0x90, 0x3E, 0x00, // delta time + Note-OFF (D)
0x20, 0xFF, 0x01, 0x04, 0x54, 0x65, 0x73, 0x74, // delta time + META message (text)
0x00, 0xFF, 0x2F, 0x00 // delta time + META message (end of track)
};
// MIDI file with running status, interrupted by a META message - failed before the bugfix
private static int[] SMF_2 = {
0x4D, 0x54, 0x68, 0x64, 0x00, 0x00, 0x00, 0x06, 0x00, 0x01, 0x00, 0x01, 0x00, 0x80, // file header
0x4D, 0x54, 0x72, 0x6B, 0x00, 0x00, 0x00, 0x21, // track header
0x00, 0x90, 0x3C, 0x7F, // delta time + Note-ON (C)
0x40, 0xFF, 0x01, 0x04, 0x54, 0x65, 0x73, 0x74, // delta time + META message (interruptor)
0x20, 0x3C, 0x00, // delta time + Note-OFF (C) - running status
0x20, 0x3E, 0x7F, // delta time + Note-ON (D) - running status
0x60, 0x3E, 0x00, // delta time + Note-OFF (D) - running status
0x20, 0xFF, 0x01, 0x04, 0x54, 0x65, 0x73, 0x74, // delta time + META message (text)
0x00, 0xFF, 0x2F, 0x00 // delta time + META message (end of track)
};
// MIDI file with running status, interrupted by a META message - succeeded before the bugfix
// but with wrong interpretation of the data
private static int[] SMF_3 = {
0x4D, 0x54, 0x68, 0x64, 0x00, 0x00, 0x00, 0x06, 0x00, 0x01, 0x00, 0x01, 0x00, 0x80, // file header
0x4D, 0x54, 0x72, 0x6B, 0x00, 0x00, 0x00, 0x21, // track header
0x00, 0x90, 0x3C, 0x7F, // delta time + Note-ON (C)
0x40, 0xFF, 0x01, 0x04, 0x54, 0x65, 0x73, 0x74, // delta time + META message (interruptor)
0x20, 0x3C, 0x00, // delta time + Note-OFF (C) - running status
0x0D, 0x3E, 0x7F, // delta time + Note-ON (D) - running status
0x60, 0x3E, 0x00, // delta time + Note-OFF (D) - running status
0x20, 0xFF, 0x01, 0x04, 0x54, 0x65, 0x73, 0x74, // delta time + META message (text)
0x00, 0xFF, 0x2F, 0x00 // delta time + META message (end of track)
};
}
---------- END SOURCE ----------
CUSTOMER SUBMITTED WORKAROUND :
None that I'm aware of.
FREQUENCY : always
All systems and versions, as far as I know
A DESCRIPTION OF THE PROBLEM :
Some MIDI files contain a sequence of events that I call "Interrupted Running Status":
1. MIDI Channel event with status byte
2. (optional) more MIDI channel events without status byte (using running status)
3. Meta or SysEx event
4. Event without status byte
The intention of the file creators is that the non-status event (4) uses the running status from the channel event (1) before the interruption.
I have looked at hundreds of real-world MIDI files with this behaviour, and I'm sure that this assumption is correct.
But I'm open to discuss this further, if requested.
The SMFParser interprets these files incorrectly, as soon as it reads the non-status byte (4).
- It reads the non-status byte
- It assumes that the status byte of the preceding Meta/SysEx must also be used as the status byte for the next event.
- It silently ignores the non-status byte (4)
- It interprets the next bytes as a part of the assumed Meta or SysEx message.
This is clearly a bug, for the following reasons:
- Running status is only allowed for channel commands, according to the SMF specification.
- In the specification it is clearly stated that running status must not be used for Meta or SysEx events.
- A byte must never be silently ignored.
- It is not what the file creators expect.
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Run the test case below. It contains, parses and prints the content of 3 small MIDI files.
Each file contains the following sequence:
- 90 3c 7f: Note-ON (C)
- ff 01 04 54 65 73 74: Meta Text (Test)
- 90 3c 00: Note-OFF (C)
- 90 3e 7f: Note-ON (D)
- 90 3e 00: Note-OFF (D)
- ff 01 04 54 65 73 74: Meta Text (Test)
- ff 2f 00: Meta (End of Track)
File 1 starts every message with a status byte.
File 2 uses (interrupted) running status for all channel events apart from the first one.
File 3 is nearly identical to file 2 - only a single delta time has a different value.
EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Two possible results can be regarded as valid: A strict solution and a tolerant one.
Strict solution:
The SMF specification contains this sentence: "Sysex events and meta-events cancel any running status that was in effect."
That means, all files with an interrupted running status are illegal.
A Meta or SysEx event should reset the running status to 0.
For all interrupted running statuses this will always result in a reliable InvalidMidiDataException.
In the provided test case that means, only the first file can be parsed. File 2 and 3 fail with InvalidMidiDataException.
Tolerant solution:
Only channel events should update the running status, but not Meta or SysEx events.
If an interrupted running status is found, use the status byte of the last channel event before the interrupting event.
For the provided test case that means, all 3 files can be parsed and produce the same output on the command line.
I would prefer to use the tolerant solution.
I have already implemented an according bugfix and inspected the resulting data from many real-world MIDI files.
The resulting data makes sense in the vast majority of files, apart from just a few that are completely broken.
I will create a pull request and discuss my arguments there or on the mailing list client-libs-dev, if needed.
ACTUAL -
The non-status byte is ignored. The next bytes are interpreted as if they belong to a Meta or SysEx message. So it depends on the following bytes what happens:
- InvalidMidiDataException
- EOFException
- No exception but misinterpreted messages
In the provided test case, only file 1 is parsed correctly.
File 2 fails with an InvalidMidiDataException
File 3 does not fail but is interpreted incorrectly. Three channel messages are missing but instead we have an additional Meta message without any sense.
---------- BEGIN SOURCE ----------
import java.io.ByteArrayInputStream;
import javax.sound.midi.MidiSystem;
import javax.sound.midi.Sequence;
import javax.sound.midi.Track;
public class BugreportIRS {
public static void main(String[] args) {
int[][] files = new int[][] {SMF_1, SMF_2, SMF_3};
for (int i = 0; i < files.length; i++) {
byte[] content = toByteArray(files[i]);
System.out.println("processing file " + (i+1));
Sequence seq;
try {
seq = MidiSystem.getSequence(new ByteArrayInputStream(content));
} catch (Exception e) {
System.out.println("...failed with exception " + e);
System.out.println();
continue;
}
System.out.println("messages:");
Track track = seq.getTracks()[0];
for (int j = 0; j < track.size(); j++) {
byte[] msg = track.get(j).getMessage().getMessage();
for (byte b : msg) {
System.out.print(Integer.toHexString(b & 0xFF) + " ");
}
System.out.println();
}
System.out.println();
}
}
private static byte[] toByteArray(int[] integers) {
byte[] bytes = new byte[integers.length];
for (int i = 0; i < integers.length; i++) {
bytes[i] = (byte) integers[i];;
}
return bytes;
}
// MIDI file without running status - should work equally before and after the bugfix
private static int[] SMF_1 = {
0x4D, 0x54, 0x68, 0x64, 0x00, 0x00, 0x00, 0x06, 0x00, 0x01, 0x00, 0x01, 0x00, 0x80, // file header
0x4D, 0x54, 0x72, 0x6B, 0x00, 0x00, 0x00, 0x24, // track header
0x00, 0x90, 0x3C, 0x7F, // delta time + Note-ON (C)
0x40, 0xFF, 0x01, 0x04, 0x54, 0x65, 0x73, 0x74, // delta time + META message (text)
0x20, 0x90, 0x3C, 0x00, // delta time + Note-OFF (C)
0x20, 0x90, 0x3E, 0x7F, // delta time + Note-ON (D)
0x60, 0x90, 0x3E, 0x00, // delta time + Note-OFF (D)
0x20, 0xFF, 0x01, 0x04, 0x54, 0x65, 0x73, 0x74, // delta time + META message (text)
0x00, 0xFF, 0x2F, 0x00 // delta time + META message (end of track)
};
// MIDI file with running status, interrupted by a META message - failed before the bugfix
private static int[] SMF_2 = {
0x4D, 0x54, 0x68, 0x64, 0x00, 0x00, 0x00, 0x06, 0x00, 0x01, 0x00, 0x01, 0x00, 0x80, // file header
0x4D, 0x54, 0x72, 0x6B, 0x00, 0x00, 0x00, 0x21, // track header
0x00, 0x90, 0x3C, 0x7F, // delta time + Note-ON (C)
0x40, 0xFF, 0x01, 0x04, 0x54, 0x65, 0x73, 0x74, // delta time + META message (interruptor)
0x20, 0x3C, 0x00, // delta time + Note-OFF (C) - running status
0x20, 0x3E, 0x7F, // delta time + Note-ON (D) - running status
0x60, 0x3E, 0x00, // delta time + Note-OFF (D) - running status
0x20, 0xFF, 0x01, 0x04, 0x54, 0x65, 0x73, 0x74, // delta time + META message (text)
0x00, 0xFF, 0x2F, 0x00 // delta time + META message (end of track)
};
// MIDI file with running status, interrupted by a META message - succeeded before the bugfix
// but with wrong interpretation of the data
private static int[] SMF_3 = {
0x4D, 0x54, 0x68, 0x64, 0x00, 0x00, 0x00, 0x06, 0x00, 0x01, 0x00, 0x01, 0x00, 0x80, // file header
0x4D, 0x54, 0x72, 0x6B, 0x00, 0x00, 0x00, 0x21, // track header
0x00, 0x90, 0x3C, 0x7F, // delta time + Note-ON (C)
0x40, 0xFF, 0x01, 0x04, 0x54, 0x65, 0x73, 0x74, // delta time + META message (interruptor)
0x20, 0x3C, 0x00, // delta time + Note-OFF (C) - running status
0x0D, 0x3E, 0x7F, // delta time + Note-ON (D) - running status
0x60, 0x3E, 0x00, // delta time + Note-OFF (D) - running status
0x20, 0xFF, 0x01, 0x04, 0x54, 0x65, 0x73, 0x74, // delta time + META message (text)
0x00, 0xFF, 0x2F, 0x00 // delta time + META message (end of track)
};
}
---------- END SOURCE ----------
CUSTOMER SUBMITTED WORKAROUND :
None that I'm aware of.
FREQUENCY : always