MultiTouch MIDI Recommendation

MultiTouch MIDI

This is an extension of http://rrr00bb.blogspot.com/2011/08/midi-instruments-on-ios.html, of which this could be a more focused restatement.

In order to facilitate the correct interpretation of MIDI instruments that are based on iPad and iPhone instruments, we need to be specific about some of the areas that MIDI leaves up to the implementer (to get wrong in most cases!). This is being written for iOS developers who are writing MIDI instruments.

Wizdom Music is involved with, or actually makes these instruments at the current time:

• MorphWiz
• MorphWizMidi
  
• SampleWiz
• Geo Synth
• Haken Continuum (not owned by Wizdom, just very involved)

These instruments, and just about every iOS instrument that is not slavishly mimicking the limits of a piano controller, will encounter problems representing MIDI input and output in a way that matches common sense on non-piano controllers. They have more stringent pitch handling requirements than a piano controller. They all potentially can have a pitch bend per finger, not per controller. They are a special case of Omni mode; actually something that's half way between single channel and omni.

These new instruments need to be able to do smooth bends from the lowest renderable note to the highest at the fullest pitch resolution. A simple use case is to just hand your ipad to a child. Putting one finger on the glass and dragging it around for the duration of an entire song, or doing so with two fingers bending in different directions; this is obvious, and so needs to be made to work within what MIDI standardizes.

They are also a special case in that we are not doing bending with any real pitch wheels, and these virtual pitch wheels have no real top and bottom position. So we include a definition of a Non-Registered-Parameter-Number (NRPN) that functions as a note tie. It allows for bends to be as wide as you would like, independent of bend width. It's best to NOT set the bend width away from the small default, because the bend width only has 14 bits of resolution, which is quite small.

The NRPN is a standard feature of MIDI, but it's up to us to standardize any definitions that we create.

Frequency

MIDI specifies notes because it was originally tightly tied to discrete keys going up and down. The frequency of the note was implicit. But when pitch wheels become involved you can make the desired frequency explicit. Assume for a moment that we set the pitch wheel to mean plus or minus only one semitone:

frequency = baseFrequencyLowC * 2^((note + bend)/12)

Note is an integer value, and bend is a number between -1 and 1. This gives the exact intended frequency. The synth should not be taking any liberties and do exactly what is stated here.

Pitch Wheels

Pitch wheels are assumed to be centered at zero when the synth starts. Once messages come in, the last value that the pitch wheel was set to for a channel should stay where it is until a new bend message comes in. This is important because if the pitch wheel is not centered and a note comes on, the initial pitch on attack includes the bend in its frequency - with no portamento, and when the note is released, it also releases at the exact frequency; which includes that channel's bend value.

Assume that the pitch wheel is set to the 2 semitone default here. We behave as if we had gotten this when there is a reset of MIDI:

bend ch1 0%
bend ch2 0%
....
bend ch16 0%

So if we know this...

off ch1 33
...
bend ch1 100%
on ch1 33
off ch1 33

Because the bend width is 2 semitones, then this note strikes exactly the same pitch as B on attack and on release. There should be no audible difference from:

off ch1 33
...
on ch1 35
off ch1 35

If this isn't done right, then fretlessnes doesn't work right. Fretlessness is the foundation to getting correct *great* pitch handling. It's the only way to get microtonality working right as well, because real-world microtonality requires actual fretlessness (not MIDI tuning tables). Choral singers do NOT actually sing in 12ET and violinists don't play in it; they will tend to Just and Pythagorean intervals (which arise by the laws of physics, not man-made music theory) in the absence of other instruments providing the constant reference tones to keep that from happening. This is even more true with ethnic instruments like Sitar. It's also true that every instrument has characteristic intonation differences that helps to identify it.

So, these pitch issues are not something to be taken lightly as 'it never happens in the real world'. This is part of the reason why electronic music is still having trouble getting past the 'uncanny valley' into being convincing. The ability to start solving this problem is one of the more exciting things about using an iPad (or any continuous surface) as an instrument.

MIDI Channels

Omni is provided as a way to have a synth render MIDI that was originally being sent out across multiple channels. But what seems to typically get done is to just substitute the original channel number with its own before interpretation, as an easy hack that mostly works. Like this:

on ch1 33
bend ch1 +25%
on ch2 33
bend ch2 -25%
off ch1 33

The expected result is two instances of the same note where one is bent up a quartertone, and the other is bent down a quartertone. At the end of this the note bent down a quartertone is still playing. So, if we are doing omni, just treat it all like it's channel 1:

on ch1 33
bend ch1 +25%
on ch1 33
bend ch1 -25%
off ch1 33

This is wrong, given what the original channels were, and it's what most synths will do. First, we end up with silence at the end of the sequence, a dropped note (almost as bad as a stuck note), because we told the only note 33 on ch1 to be silent. The second thing is instead of chording two of the same note at detuned values (ie: chorusing), the first note 33 just got bent to the value of the second instance of 33.

So specifically, you must pass this test:

on ch1 33
bend ch1 +25%
on ch2 33
bend ch2 -25%
off ch1 33

The note should still be playing, detuned a quartertone here. This means that there are two separate instances of A, simultaneously played, chording, possibly out of phase, etc. This is no stranger than playing A and A# together. It should work, and this happens all the time on string instruments.

New Note - New Channel

You don't know if a note will be bent in the future, so it must have its own pitch wheel. If you run out of channels and have to share a note with another channel then you will get artifacts if one of the notes on that channel needs to be bent.

As a compromise, you don't have to monopolize all 16 channels for this. You just need a reasonable number, which if you are emulating a string instrument, is generally going to be the number of strings, or maybe just the size of the largest playable chord.

Adding more channels does help if you can afford it in performance, because a note being turned off is not the end of its life; releases are still happening when the note is turned off.

Single Patch "Omni"

Just because there is only one patch at a time, it should not mean that the MIDI data should be misinterpreted like the above situation. The channels specified in the original MIDI output exist to keep the various expression parameters separate. It is so important that the pitch wheel is kept separate; this intent should never be violated on pitch, even if it has to be violated for other parameters.

As a hack to make the separate pitch bends work right, a typical way to do this is to setup a multi-timbral synth to simply duplicate the same exact patch and settings across all of the channels (or just a subset of them). But just because you have a synth that can only load one patch should not change the meaning of the bend messages.

Overlaps

One of the issues created by MIDI channeling is overlaps. You must not get a silence when note on/off pairs for same note overlap. This happens when you try to play more notes than there are channels available to play them on:

on ch1 33
on ch1 33
off ch1 33

This should play two notes and then silence.

on ch1 33
on ch2 33
off ch1 33

This should play two notes, where you can hear two of the exact same note playing, then one of them cuts off. You should not hear silence yet. This happens because on a string instrument, you will trill on the exact same note to play very fast, like guitar picking:

on ch1 33
on ch2 33
off ch1 33
off ch2 33

Note: The MIDI specification states that it is up to the implementer what happens when note off and note on are not exactly balanced. But as with internet protocols "Be conservative in what you send and liberal in what you understand".

http://www.gweep.net/~prefect/eng/reference/protocol/midispec.html

"
If a device receives a Note On for a note (number) that is already playing (ie, hasn't been turned off yet), it is up the device whether to layer another "voice" playing the same pitch, or cut off the voice playing the preceding note of that same pitch in order to "retrigger" that note.
"
This is a disaster in the making. It is up to the implementer, but you will get a stuck note if the client and server choose different ways. Don't take that statement to mean that you should create a synth that will leave a stuck note when the notes don't balance.

Most synths take the (on, note, vol) tuple to be setting the state for a note, not pushing an instance of the note onto a stack, to be popped by a note off. If you are liberal in what you send, then you will only add to the number of synths that get stuck notes. If you need to play a voice twice, you should be using the channels. This is consistent with the practice of new note new channel that we are using here.

Think about the fact that (on, note, 0) is an alias for a note off. (on, note, vol) leaves vol as a variable. you can have balances get undone if the vol is equal to zero even though the intent wasn't to turn the note off.

Release Time

When a note is turned off, that's not the end of its life. That's the beginning of the release phase only. The frequency that it releases on will change if you mess with the pitch wheel for that channel before the sound is gone. It will be an annoying artifact for instruments like hammered dulcimers with special tunings (Indian music, etc).

Because every new note down picks a new channel of the 16 available, we need to carefully cycle through the available channels evenly. This will maximize the amount of time that a channel has been dead before we steal that channel to play a new note.

Note Tie Non Registered Parameter Number (restated from previous post)

It is not impossible to bend MIDI notes to any width you want at fullest possible resolution. the problem is that there is no defacto or dejure standard on how this is done. Imagine a piano player trying to simulate a bend, and it's on our channel cycling instrument....

bend ch1 0%

bend ch2 0%

...

bend ch16 0%

on ch1 33

...

off ch1 33

on ch2 34

...

off ch2 34

on ch3 35

...

off ch3 35

on ch4 36

...

So he's playing chromatics to simulate the bend because that's the best he can do. But if we are on a synth that inserts bend messages, the synth can at least bend from one chromatic to the next like this:

bend ch1 0%

bend ch2 0%

...

bend ch16 0%

on ch1 33

bend ch1 20%

bend ch1 40%

bend ch1 60%

bend ch1 80%

bend ch1 100%

off ch1 33

on ch2 34

bend ch2 20%

bend ch2 40%

bend ch2 60%

bend ch2 80%

bend ch2 100%

off ch2 34

on ch3 35

bend ch3 20%

bend ch3 40%

bend ch3 60%

bend ch3 80%

bend ch3 100%

off ch3 35

on ch4 36

bend ch4 20%

bend ch4 40%

bend ch4 60%

bend ch4 80%

bend ch4 100%

So, this would be a smooth bend, except we hear the note retrigger every time we reach the next chromatic. So let's say that we have a special message that notes that there is a note tie coming and that it's done when the next note on appears.

bend ch1 0%

bend ch2 0%

...

bend ch16 0%

on ch1 33

bend ch1 20%

bend ch1 40%

bend ch1 60%

bend ch1 80%

bend ch1 100%

tie ch1 33

off ch1 33

on ch2 34

bend ch2 20%

bend ch2 40%

bend ch2 60%

bend ch2 80%

bend ch2 100%

tie ch2 34

off ch2 34
bend ch3 0% #note that from tie to note on, we expect bends and a note off to happen

on ch3 35

bend ch3 20%

bend ch3 40%

bend ch3 60%

bend ch3 80%

bend ch3 100%

tie ch3 35

off ch3 35

on ch4 36

bend ch4 20%

bend ch4 40%

bend ch4 60%

bend ch4 80%

bend ch4 100%

We can continue this from the lowest note on the keyboard to the highest for a super-wide bend. It is at the full pitch resolution as well because we aren't playing tricks with the MIDI bend width. It is also the case that if we broadcast this both to a piano that can't bend, and the synth that understands, we get a similar result. It degrades gracefully on the piano, and sounds perfect on the synth that understands. We can use this to track up to 16 fingers at arbitrary pitches (in MIDI range of course!) bending in whatever wild directions they need.

The NRPN looks like this in our code:

#define TRANSITION 1223

static inline void sendNRPN(int ochannel,int msg,int val)
{
//B0 63 6D
//B0 62 30
//B0 06 100
int lsb = msg&0x7f;
int msb = (msg>>7)&0x7f;
//midiPlatform_sendMidiPacket7(0xB0+ochannel, 0x63, msb, 0x62, lsb, 6, val);

midiPlatform_sendMidiPacket3(0xB0+ochannel, 0x63, msb);
midiPlatform_sendMidiPacket3(0xB0+ochannel, 0x62, lsb);
midiPlatform_sendMidiPacket3(0xB0+ochannel, 6, val);
}

static inline void retriggerNewMidiNote(int finger,float midiFloat,int vol,int expr)
{
int channel = midiFingerUsesChannel[finger];
if(channel >= 0)
{
int ochannel = midiChannelOChannelSent[channel];
sendNRPN(ochannel,TRANSITION,midiChannelNote[channel]);
}
stopMidiNote(finger);
startNewMidiNote(finger,midiFloat,vol,expr);
}


Let us know if there is something unreasonable about that message. I haven't used NRPNs before, and since we write both ends of it, they could both be 'wrong' and work just fine between our synths.

Bend Width RPN

Just in case it isn't completely obvious what happens when we change the bend width slider, here's the code that tells the synth our assumed bend width when we move that slider:


static void bendRangeSliderMoved() {
// Check the current bend range with the previous. If it has changed, send the appropriate RPN.
static int previousBendRange = -1;
int bendRange = bendRangeSlider.scaledValue;
if (bendRange != previousBendRange)
{
NSLog(@"bend range has changed from %d to %d", previousBendRange, bendRange);

previousBendRange = bendRange;

// Activate the pitch-bend sensitivity (bend range) RPN.
SoundEngine_sendMIDIPacket3(0xB0, 101, 0);
SoundEngine_sendMIDIPacket3(0xB0, 100, 0);

// Send the new bend range value.
//BUG: earlier versions sent 2*bendRange, which was wrong
SoundEngine_sendMIDIPacket3(0xB0, 6, bendRange );
SoundEngine_sendMIDIPacket3(0xB0, 38, 0);

// Deactivate the pitch-bend sensitivity RPN.
SoundEngine_sendMIDIPacket3(0xB0, 101, 127);
SoundEngine_sendMIDIPacket3(0xB0, 100, 127);

// Flush the MIDI queue.
midiPlatform_flush();
}

// Update the bend range with the sound engine.
SoundEngine_midiBendsInSemitones(bendRange);

// Flag that the texture needs to be updated.
bendRangeSlider.needsNewTexture = YES;
}