MIDI and Over-Abstraction

Let me know specifically where this is wrong. I have actual code that works, so even if it's wrong, it's not theoretical - it works very well.

I am getting my thoughts together on the rationale for doing MIDI the way that I have. The feedback I get on what I am trying to do shows a divide between people that worry about keeping hardware simple versus those trying to keep software simple. It is instructive to first talk not about MIDI and sound rendering, but about OpenGL and drawing APIs.

Cooked Standards

The OpenGL that I am used to is the desktop 1.x versions of it. It provides a state machine that feels a lot like an assembly language for some real hardware device. But it is not painfully low level, because it provides the things that you are almost guaranteed to need in every app. The basic idea is that it will do buffering and matrix transforms for you, and you just feed it what you want. For example pseudo code:

loadIdentityMatrix();

setCameraPointLookingAt(x,y,z, atX,atY,atZ);

turnOnLightSource(lightX,lightY,lightZ, lightR,lightG,lightB);

...

push(TRIANGLES);

vertex(x,y,z, r,g,b);

vertex(x+dx,y,z, r,g,b);

vertex(x+dx,y+dy,z, r,g,b);

...

pop(TRIANGLES);

...

render();

What is really cool about doing things this way is that you don't have to supply all of the equations for moving points around in a 3D projection. You don't have to build any kind of API to buffer data. These things are provided for you, because you will need this functionality in every app. But newer versions of OpenGL threw all of this away. The main reason for it is that the huge number of APIs to cover every scenario for lighting and coordinate transforms keep growing. Their existence makes the spec unstable. These 'fundamental' things such as lighting, buffering, and coordinate transforms are really conveniences that should be out of the standard.

Raw Standards

The problem with this is that something like OpenGL needs to make the implementation as low-level as possible, while still allowing broad compatibility. So, the requirement for programmable shaders does not mean that it should just be shoehorned into the existing specification. What they did was to introduce an even lower-level API as the new standard, where old code is essentially emulated on top of it. The new APIs are kind of like this:

setVerticesArray(arrayOf3DVertices);
setColorsArray(arrayOfColorsForThe3DVertices);
setNormalsArray(arrayOfNormalsForThe3DVertices);
...
setVertexShaderProgram(compiledVertexShader1);
setFragmentShaderProgram(compiledFragShader1);
...
doRendering();

So if you want the old-code look, then you use an API that looks like the old code, but ultimately turns into this underneath. If you need the new features, then you write a framework that lives inside of your framework that approximates the old APIs where you can, and goes its own way where it must do so. This keeps the specification from being larded with everything that the designers didn't think of. The hardware has a smaller burden of what it must recognize, as does the OpenGL graphics driver. But the burden moved to the programmer to use (or write) a library over top of it to get it going. In short, they solved the stability problem by making the primitives more general.

So newer versions of OpenGL recognize that in the end, all of this code configures shaders in the graphics pipeline and builds buffers of different kinds of vertices. So new versions of OpenGL don't even bother providing 'fundamental' things like lighting, buffering, and coordinate transforms. The burden is on the developer to create these things. This is not a hassle if OpenGL developers use a standard software library above the raw library. All that has happened is that complexity moved out of the hardware (or at least the API that hides the hardware) into the client software.

VRML and other SceneGraph APIs went nowhere. They provided really easy abstractions, at the expense of creating objects that were too high level. But in the end, the ability to simply make shaded/textured triangles won. Making the API too high level simplifies things for somebody who insists on writing directly to the hardware, at the expense of imposing limitations. The limitations need to be worked around, so then garbage creeps into the spec because the spec is too high level. For OpenGL, staying very low level is the strategy for having the spec withstand change.

It makes the applications more complex to build, at the expense of taking out parts of the spec that are subject to much variation, by only including primitives.

MIDI Is Too High Level, Making The Spec Too (F-ing) Large

MIDI seriously suffers from this over-abstraction problem. Because hardware devices speaking raw MIDI with very little computational power on-board are the primary focus, it is understandable that the model is of turning notes on and off and bending them. In this view, it isn't the controller's job to either: 1) determine exact pitch 2) manipulate note on/off to create solo mode 3) manipulate the envelope of a note. This view is tightly tied to the assumption of discrete keyboard keys that send a MIDI message when the key goes down. The idea of completely continuous controllers could be shoe-horned in as an incompatible addition to the spec, but that's not a good thing because it takes an already complex spec and makes it more complicated without even being backwards compatible. What would be better is to make the primitive elements more general, and make controllers handle it themselves in a backwards compatible way.

MIDI isn't primitive enough. The abuses of the spec that are currently possible, but dicey, need to be standardized so that they are totally expected and legitimate. Note on/off are clearly fundamental. Bends are fundamental, but their semantics are underspecified. You should be able to compute a frequency given a note and a bend value, and work in actual frequencies. This is because frequencies are fundamental, even though MIDI doesn't quite get this concept. Envelopes on the sound are fundamental as well.

Bend And Legato

Because almost all MIDI devices only support the on/off/bend messages, and ONLY that reliably, it's foolish to implement huge parts of the MIDI spec and simply demand that synths and controllers understand all of it. This is especially true on iOS, where such demands will simply result in incompatibility on the part of developers who will simply implement what they need to ship. I am sticking to what is known to work everywhere; and then using the NRPN to putty in the cracks in such a way that it is not a disaster if the synth doesn't understand. This is analagous to rendering 3D graphics as simple triangles if a special shader is unusable. This principle is also why web pages are not usually completely broken when different browsers are used against non-portable pages. A wrong pitch due to an unrecognized message is far more debilitating than getting a right pitch with a note re-trigger at an inappropriate time, especially because many patches don't have a noticeable attack anyway.

Fundamental Elements

So, these are the fundamental elements to build up a correctly functioning MIDI synth that has control over pitch, polyphony, and legato:

0x90 - note on/off

0x80 - note off (i send 0x90, but should recognize 0x80 as equivalent

0xe0 - bend (pitch bend setting still applies)

0xbxxxxx - a special 'note tie' NRPN that states that the next note on and note/off pair are actually tied together.

The first rule is that there are no real 'notes', only frequencies that we make by a combination of midiNote number and bend. We don't try to re-tune the notes, but use pitch bend in combination with note to get the exact pitch. We also place all current notes into unique channels, and try to behave reasonably when this is not possible (ie: channel bend reflects last note down). This is the only really reasonable way to do this because the note on that we choose is what we *name* the note. This note name is what a piano that doesn't understand bends will play if it's asked to play our note.

Because we can always exceed the bend width, note tie says to 'continue note in the same state from the note turning off to the next one turning on'. This note can, and usually does change channels, because of the requirement for every note going down to go down into its own channel. You have to hold off on reusing a channel for as long as possible, because when a note is turned off, it will still respond to bends while it is releasing.

Keeping It Simple In Software, Hardware Isn't Our Problem

We are not worried about requiring a full-blown brain in the client, as hardware vendors might object to. Moving complexity out of the synth and into the controller makes an incredible amount of sense on iOS. This is because the controller will need some of the low level details in order to render itself on the screen. We have the pitch implied by where the finger touches, the pitch that we are drifting to due to fretting rules, and the actual pitch being played. We need to know all of this information in the controller. The standard MIDI equivalent would simply have the controller knowing about where the fingers are, and being more-or-less ignorant of what the synth is doing with this information. So in our case, the controller manipulates the pitch wheel to create the fretting system, and the synth has no idea what intonation we are using. It's not the synth's business to know this.

Similarly with polyphony rules, the synth can't just have a 'solo mode' setting. AlephOne and Geo both have a per-string polyphony that essentially adds the concept of 'polyphony groups'. The polyphony groups act similar to channels in that the controller will turn notes on and off to get the right polyphony behavior. This way we can chord and do legato at the same time. It's a controller-dependent thing, and it's not the synth's business to know any of this.

Similarly with legato. Legato *usually* tracks polyphony by playing attack on the first note down in the polyphony group. But in reality, on a string instrument, whether to pick or legato a note is decided on a per note-basis. It's not a mode that is enabled or disabled for the whole controller.

Because almost nothing recognizes more than note on/off/bend, anything else that the MIDI spec states is quite irrelevant in practice. The note tie addresses something that nothing in the spec does, and doubles as the legato, and it's not a major problem if it's not implemented. To somebody implementing a synth, a small number of primitives (only one thing beyond the standard elements) gives good overall behavior.

There is also the issue of the same note being played multiple times. AlephOne does chorusing. It doesn't do this with any post-processing effects. It works by playing the same note, microtonally displaced, twice everywhere. This is one of the reasons why simply picking note numbers and bending them around is a bad idea. On a guitar, the high E note is played simultaneously from 2 or 3 positions all the time. The assumption that you bend a key is rooted in the idea of a keyboard with one key per 'note'.

Hacks

So, yeah, this is a hack with current MIDI. OSC is too general (in the sense that OSC messages have no inherent semantics, just syntax - it's XML hell all over again). And what I have read of MIDI proprosals that aren't made of on/off/bend seem unworkable in practice. If we are all on iOS and MIDI becomes too complex and not compatible with existing hardware anyway, we will simply throw the protocol away entirely and use on-board synths.

Real World Implementation

This is actually impemented in Geo Synthesizer and AlephOne on the client end, and in SampleWiz, ThumbJam (where it is especially good, specifically on the JR Zendrix patch), and Arctic Synth on the server end (I believe - I don't have a new enough version of the OS to run it). But anyways, it's not a theoretical idea. It has been implemented multiple times, and is demoed here (switching intonations while playing and doing polyphonic bends - and single polyphonic playing of this works fine against a lot of synths that have weak MIDI implementations):

http://www.youtube.com/watch?v=OYvPeFPc7Zw&list=UUF0WJOds_PNANtUgE-XA4WQ&index=1&feature=plcp

A Complete Pitch Geometry

In previous posts, I explained how close 53ET is to being a 5-limit system, and how to create intuitive diagrams that do away with obscure lists of pitch ratios and vague explainations. Pitch geometry is a way of diagramming the actual harmonic series as viewed on a Just Fourths tuned string instrument so that things line up (or mis-align) based on the harmonic series. This gives us a practical way to view it without getting overly technical.

And best of all, it's baked directly into a real instrument that can be played. You can see simple ratios up to 13 with it. In particular, you can see everything from 2/1, 3/2, 4/3, 5/4, ... , 16/15. Which happens to be every "superparticular" ratio (meaning (N+1)/N) up to the commonly used Just Major Semitone. It's the Just Intonation version of the chromatic note. Click the picture to see the fine details!

I would like to mention this book by Cameron Powers:

http://www.amazon.com/Lost-Secrets-Perfect-Harmony-Indigenous/dp/1933983183

It is about Middle Eastern scales in particular, but it's also very much about the sort of future that touchscreen instruments are going to bring with them; Just Intonation becoming normal practice once again, assisted by the digital age.

Geo Synthesizer (aka Pythagoras, Geo Synth) is not just a fun toy for playing with virtuosity on iPhone and iPad. It was designed from the beginning to feed virtuosity in every way that it can. From allowing for very fast chromatic play, to learning the deep magic of the harmonic series (Just Intonation based microtonality), or just playing fretless; it is designed to teach you things you might not already know. This way you can learn the satisfaction of constantly improving your musicianship, rather than the short-lived satisfaction of an instrument that's merely easy to use.

Octaves, or 2-limit (2,1/2)
This is a trivial system of notes. It is everything you can create out of positive and negative powers of 2, meaning: 2^-1, 2^0, 2^1,... (ie: 1/2, 1, 2,...). It is obvious that octaves are one of the more basic equivalences that anybody can tune by ear alone.

Pythagorean Tuning, or 3-limit (3/2, 4/3 - blue lines)

Tuning a piano is not as straightforward as it seems, because the instrument is a real-world resonating body. In the real-world, the 12 tone system is a numerological fiction that roughly approximates the behavior of the harmonic series. Like a pair of drummers that play different time signatures but have the same number of beats per minute, a pair of waves that progress at a rate in simple ratios will regularly meet at the exact same point in the shortest time possible.

A simple way to tune a piano would be to start from one note that we will use as the reference note that is "in tune", and listen to the fifth overtone in the string. We then tune another string to that pitch, or an octave lower than it to stay in the same octave range. By the time we have done this to 5 strings, we have a Pythagorean Tuned pentatonic scale. If we do this for 7 strings, then we have a Pythagorean Tuned diatonic scale. It is obvious that these seven notes will not be equally spaced when we play them in order, because there will be whole tones and half tones. But the unevenness goes even deeper than this, because if we extend out to 12 strings, then we have a Pythagorean tuned chromatic scale. This scale will sound beautiful as long as you don't do anything that assumes that the 12 tones wrap around at the ends. This 12 tone scale overshoots the octave by about 1/9 of a Just whole tone.

The Blue lines that go up and down by fourths and fifths are the 3-limit. So this is every ratio that you can make with powers of primes up to the number 3. For example: 1/3, 2/3, 3/2, 9/8, etc. Because of how the 2-limit (powers of octaves!) and the 3-limit (powers of fifths, and therefore fourths as well) line up, it very nearly creates a 53 note per octave equal tempered scale. It isn't exact, but it is close enough that you can forget about the numbers not lining up at the extremities, as it's a perfect Pythagorean scale in practice. It is the first usefully close approximation to a Just Circle Of Fifths.

Just Intonation, or 5-Limit (5/4, 6/5 - green lines)
Usually, when Just Intonation is spoken of, 5-limit is what is meant. This means inclusion of perfect thirds, major and minor. It is a very fortunate coincidence that the 53 note per octave scale, happens to line up almost exactly. Almost every world-music system is using some variant of this scale. Flexible intonation instruments like Sitar, Violin, Voice, etc will invariably use this system or some subset of it.

Higher Limits, 7-limit (yellow), 11-limit (turquoise), 13-limit (purple)

These lines were recently added so that I could locate limits that get used in special contexts. Arabic Maqam is quoted as using a few different variations on what a "quartertone" is. It is typically notated as the dead center between a 12ET major third (or described as a 3/4 tone). But this is not what actually happens. Those intervals literally taken are awful for chording. The just intoned variations that can be found by ear and practice are what are used.

The scale families known as "Bayati" have quartertones in their bottom tetrachord. This would mean, D, E quarterflat, F, G. If what is meant is the note that is very close to the chromatic quartertone, then it means the ratio 11/10. This ratio, as well as 12/11 is unreachable in 53ET, which is otherwise excellent for this type of music. If what is meant is the note that fits almost exactly into 53ET, then it means 13/12.

Count The Lines

In the picture above, I can go over the various pitch ratios counter-clockwise.

• 3/2 - This is the Just fifth. It is a blue line
• 4/3 - This is a Just fourth. It is blue because it's the dual of 3/2.
  
• 5/4 - This is the Major Third. It is green. (to the left of F# on bottom row in the picture)
  
• 6/5 - This is the Minor Third. It is green.
• 7/6 - Yellow.
  
• 8/7 - Yellow.
  
• 9/8 - The Just Whole Tone. Grey. This is a scale building block
• 10/9 - The Minor Just Whole Tone. Gray. It arises often.
• 11/10 - Turquoise. A high quartertone. It falls almost exactly between two 53ET frets.
  
• 12/11 - Turquoise. The quartertone that matches a chromatic Quartertone closely. It falls almost exactly between two 53ET frets.
  
• 13/12 - Purple. A low quartertone that almost falls in the 53ET scale.
• 14/13 - Purple. A low quartertone. It falls almost exactly between two 53ET frets. Note: because of this pattern of quartertones, some people advocate splitting the way that 12ET is split to 24 to get quartertones. If you use 106ET, then you can specify these quartertones pretty exactly.
  
• 15/14 - Gray.
• 16/15 - The Just Major Semitone. A building block of scales.
• ...
• ... (they get smaller and smaller)
  
• ...
• 81/80 - It's not explicitly shown here, but it is very special. It almost coincides with 1 53ET fret, which is the distance between the vertical lines. It is the difference between two whole tones and a major third. Ie: (9/8 * 9/8) / (4/5) = 81/64 * 4/5 =81/80. The 53ET fret is also almost the amount by which 6 whole tones overshoot an octave. Many tuning systems want two whole tones to equal a major third, so tempering is done to make this so.
  

The good thing about the way it is drawn out onto the instrument, you don't have to think too hard about pitch ratios or anything like that. You just line up lines. The colors give an indication what family the pitch ratio is in. This style of playing doesn't use frets to keep you in tune, but uses instruments with a lot of resonance, so that the resonance is what keeps you in tune; which is getting help from the laws of physics rather than hard constraints.

Amazingly, it's not hard to play pretty accurately on the phone. You can certainly be more accurate than 12ET is. You can turn up the delay feedback to the top, and set the delay rate to 0.01seconds to make the audio resonate with a drone pitch (which I can't fine tune, but it's fretless, so you can adjust!). What is important is knowing exactly what you are doing with pitches, and by exactly how much you are falling short.

Read my older post on 5-limit for a deeper explanation of Pitch Geometry for 53ET and Just Intonation. It's almost identical to the user interface for the instrument, but is much more clear.

http://rrr00bb.blogspot.com/2011/08/53et-5-limit-in-geo-explained.html

MultiTouch MIDI Recommendation

MultiTouch MIDI

This is an extension of http://rrr00bb.blogspot.com/2011/08/midi-instruments-on-ios.html, of which this could be a more focused restatement.

In order to facilitate the correct interpretation of MIDI instruments that are based on iPad and iPhone instruments, we need to be specific about some of the areas that MIDI leaves up to the implementer (to get wrong in most cases!). This is being written for iOS developers who are writing MIDI instruments.

Wizdom Music is involved with, or actually makes these instruments at the current time:

• MorphWiz
• MorphWizMidi
  
• SampleWiz
• Geo Synth
• Haken Continuum (not owned by Wizdom, just very involved)

These instruments, and just about every iOS instrument that is not slavishly mimicking the limits of a piano controller, will encounter problems representing MIDI input and output in a way that matches common sense on non-piano controllers. They have more stringent pitch handling requirements than a piano controller. They all potentially can have a pitch bend per finger, not per controller. They are a special case of Omni mode; actually something that's half way between single channel and omni.

These new instruments need to be able to do smooth bends from the lowest renderable note to the highest at the fullest pitch resolution. A simple use case is to just hand your ipad to a child. Putting one finger on the glass and dragging it around for the duration of an entire song, or doing so with two fingers bending in different directions; this is obvious, and so needs to be made to work within what MIDI standardizes.

They are also a special case in that we are not doing bending with any real pitch wheels, and these virtual pitch wheels have no real top and bottom position. So we include a definition of a Non-Registered-Parameter-Number (NRPN) that functions as a note tie. It allows for bends to be as wide as you would like, independent of bend width. It's best to NOT set the bend width away from the small default, because the bend width only has 14 bits of resolution, which is quite small.

The NRPN is a standard feature of MIDI, but it's up to us to standardize any definitions that we create.

Frequency

MIDI specifies notes because it was originally tightly tied to discrete keys going up and down. The frequency of the note was implicit. But when pitch wheels become involved you can make the desired frequency explicit. Assume for a moment that we set the pitch wheel to mean plus or minus only one semitone:

frequency = baseFrequencyLowC * 2^((note + bend)/12)

Note is an integer value, and bend is a number between -1 and 1. This gives the exact intended frequency. The synth should not be taking any liberties and do exactly what is stated here.

Pitch Wheels

Pitch wheels are assumed to be centered at zero when the synth starts. Once messages come in, the last value that the pitch wheel was set to for a channel should stay where it is until a new bend message comes in. This is important because if the pitch wheel is not centered and a note comes on, the initial pitch on attack includes the bend in its frequency - with no portamento, and when the note is released, it also releases at the exact frequency; which includes that channel's bend value.

Assume that the pitch wheel is set to the 2 semitone default here. We behave as if we had gotten this when there is a reset of MIDI:

bend ch1 0%
bend ch2 0%
....
bend ch16 0%

So if we know this...

off ch1 33
...
bend ch1 100%
on ch1 33
off ch1 33

Because the bend width is 2 semitones, then this note strikes exactly the same pitch as B on attack and on release. There should be no audible difference from:

off ch1 33
...
on ch1 35
off ch1 35

If this isn't done right, then fretlessnes doesn't work right. Fretlessness is the foundation to getting correct *great* pitch handling. It's the only way to get microtonality working right as well, because real-world microtonality requires actual fretlessness (not MIDI tuning tables). Choral singers do NOT actually sing in 12ET and violinists don't play in it; they will tend to Just and Pythagorean intervals (which arise by the laws of physics, not man-made music theory) in the absence of other instruments providing the constant reference tones to keep that from happening. This is even more true with ethnic instruments like Sitar. It's also true that every instrument has characteristic intonation differences that helps to identify it.

So, these pitch issues are not something to be taken lightly as 'it never happens in the real world'. This is part of the reason why electronic music is still having trouble getting past the 'uncanny valley' into being convincing. The ability to start solving this problem is one of the more exciting things about using an iPad (or any continuous surface) as an instrument.

MIDI Channels

Omni is provided as a way to have a synth render MIDI that was originally being sent out across multiple channels. But what seems to typically get done is to just substitute the original channel number with its own before interpretation, as an easy hack that mostly works. Like this:

on ch1 33
bend ch1 +25%
on ch2 33
bend ch2 -25%
off ch1 33

The expected result is two instances of the same note where one is bent up a quartertone, and the other is bent down a quartertone. At the end of this the note bent down a quartertone is still playing. So, if we are doing omni, just treat it all like it's channel 1:

on ch1 33
bend ch1 +25%
on ch1 33
bend ch1 -25%
off ch1 33

This is wrong, given what the original channels were, and it's what most synths will do. First, we end up with silence at the end of the sequence, a dropped note (almost as bad as a stuck note), because we told the only note 33 on ch1 to be silent. The second thing is instead of chording two of the same note at detuned values (ie: chorusing), the first note 33 just got bent to the value of the second instance of 33.

So specifically, you must pass this test:

on ch1 33
bend ch1 +25%
on ch2 33
bend ch2 -25%
off ch1 33

The note should still be playing, detuned a quartertone here. This means that there are two separate instances of A, simultaneously played, chording, possibly out of phase, etc. This is no stranger than playing A and A# together. It should work, and this happens all the time on string instruments.

New Note - New Channel

You don't know if a note will be bent in the future, so it must have its own pitch wheel. If you run out of channels and have to share a note with another channel then you will get artifacts if one of the notes on that channel needs to be bent.

As a compromise, you don't have to monopolize all 16 channels for this. You just need a reasonable number, which if you are emulating a string instrument, is generally going to be the number of strings, or maybe just the size of the largest playable chord.

Adding more channels does help if you can afford it in performance, because a note being turned off is not the end of its life; releases are still happening when the note is turned off.

Single Patch "Omni"

Just because there is only one patch at a time, it should not mean that the MIDI data should be misinterpreted like the above situation. The channels specified in the original MIDI output exist to keep the various expression parameters separate. It is so important that the pitch wheel is kept separate; this intent should never be violated on pitch, even if it has to be violated for other parameters.

As a hack to make the separate pitch bends work right, a typical way to do this is to setup a multi-timbral synth to simply duplicate the same exact patch and settings across all of the channels (or just a subset of them). But just because you have a synth that can only load one patch should not change the meaning of the bend messages.

Overlaps

One of the issues created by MIDI channeling is overlaps. You must not get a silence when note on/off pairs for same note overlap. This happens when you try to play more notes than there are channels available to play them on:

on ch1 33
on ch1 33
off ch1 33

This should play two notes and then silence.

on ch1 33
on ch2 33
off ch1 33

This should play two notes, where you can hear two of the exact same note playing, then one of them cuts off. You should not hear silence yet. This happens because on a string instrument, you will trill on the exact same note to play very fast, like guitar picking:

on ch1 33
on ch2 33
off ch1 33
off ch2 33

Note: The MIDI specification states that it is up to the implementer what happens when note off and note on are not exactly balanced. But as with internet protocols "Be conservative in what you send and liberal in what you understand".

http://www.gweep.net/~prefect/eng/reference/protocol/midispec.html

"
If a device receives a Note On for a note (number) that is already playing (ie, hasn't been turned off yet), it is up the device whether to layer another "voice" playing the same pitch, or cut off the voice playing the preceding note of that same pitch in order to "retrigger" that note.
"
This is a disaster in the making. It is up to the implementer, but you will get a stuck note if the client and server choose different ways. Don't take that statement to mean that you should create a synth that will leave a stuck note when the notes don't balance.

Most synths take the (on, note, vol) tuple to be setting the state for a note, not pushing an instance of the note onto a stack, to be popped by a note off. If you are liberal in what you send, then you will only add to the number of synths that get stuck notes. If you need to play a voice twice, you should be using the channels. This is consistent with the practice of new note new channel that we are using here.

Think about the fact that (on, note, 0) is an alias for a note off. (on, note, vol) leaves vol as a variable. you can have balances get undone if the vol is equal to zero even though the intent wasn't to turn the note off.

Release Time

When a note is turned off, that's not the end of its life. That's the beginning of the release phase only. The frequency that it releases on will change if you mess with the pitch wheel for that channel before the sound is gone. It will be an annoying artifact for instruments like hammered dulcimers with special tunings (Indian music, etc).

Because every new note down picks a new channel of the 16 available, we need to carefully cycle through the available channels evenly. This will maximize the amount of time that a channel has been dead before we steal that channel to play a new note.

Note Tie Non Registered Parameter Number (restated from previous post)

It is not impossible to bend MIDI notes to any width you want at fullest possible resolution. the problem is that there is no defacto or dejure standard on how this is done. Imagine a piano player trying to simulate a bend, and it's on our channel cycling instrument....

bend ch1 0%

bend ch2 0%

...

bend ch16 0%

on ch1 33

...

off ch1 33

on ch2 34

...

off ch2 34

on ch3 35

...

off ch3 35

on ch4 36

...

So he's playing chromatics to simulate the bend because that's the best he can do. But if we are on a synth that inserts bend messages, the synth can at least bend from one chromatic to the next like this:

bend ch1 0%

bend ch2 0%

...

bend ch16 0%

on ch1 33

bend ch1 20%

bend ch1 40%

bend ch1 60%

bend ch1 80%

bend ch1 100%

off ch1 33

on ch2 34

bend ch2 20%

bend ch2 40%

bend ch2 60%

bend ch2 80%

bend ch2 100%

off ch2 34

on ch3 35

bend ch3 20%

bend ch3 40%

bend ch3 60%

bend ch3 80%

bend ch3 100%

off ch3 35

on ch4 36

bend ch4 20%

bend ch4 40%

bend ch4 60%

bend ch4 80%

bend ch4 100%

So, this would be a smooth bend, except we hear the note retrigger every time we reach the next chromatic. So let's say that we have a special message that notes that there is a note tie coming and that it's done when the next note on appears.

bend ch1 0%

bend ch2 0%

...

bend ch16 0%

on ch1 33

bend ch1 20%

bend ch1 40%

bend ch1 60%

bend ch1 80%

bend ch1 100%

tie ch1 33

off ch1 33

on ch2 34

bend ch2 20%

bend ch2 40%

bend ch2 60%

bend ch2 80%

bend ch2 100%

tie ch2 34

off ch2 34
bend ch3 0% #note that from tie to note on, we expect bends and a note off to happen

on ch3 35

bend ch3 20%

bend ch3 40%

bend ch3 60%

bend ch3 80%

bend ch3 100%

tie ch3 35

off ch3 35

on ch4 36

bend ch4 20%

bend ch4 40%

bend ch4 60%

bend ch4 80%

bend ch4 100%

We can continue this from the lowest note on the keyboard to the highest for a super-wide bend. It is at the full pitch resolution as well because we aren't playing tricks with the MIDI bend width. It is also the case that if we broadcast this both to a piano that can't bend, and the synth that understands, we get a similar result. It degrades gracefully on the piano, and sounds perfect on the synth that understands. We can use this to track up to 16 fingers at arbitrary pitches (in MIDI range of course!) bending in whatever wild directions they need.

The NRPN looks like this in our code:

#define TRANSITION 1223

static inline void sendNRPN(int ochannel,int msg,int val)
{
//B0 63 6D
//B0 62 30
//B0 06 100
int lsb = msg&0x7f;
int msb = (msg>>7)&0x7f;
//midiPlatform_sendMidiPacket7(0xB0+ochannel, 0x63, msb, 0x62, lsb, 6, val);

midiPlatform_sendMidiPacket3(0xB0+ochannel, 0x63, msb);
midiPlatform_sendMidiPacket3(0xB0+ochannel, 0x62, lsb);
midiPlatform_sendMidiPacket3(0xB0+ochannel, 6, val);
}

static inline void retriggerNewMidiNote(int finger,float midiFloat,int vol,int expr)
{
int channel = midiFingerUsesChannel[finger];
if(channel >= 0)
{
int ochannel = midiChannelOChannelSent[channel];
sendNRPN(ochannel,TRANSITION,midiChannelNote[channel]);
}
stopMidiNote(finger);
startNewMidiNote(finger,midiFloat,vol,expr);
}


Let us know if there is something unreasonable about that message. I haven't used NRPNs before, and since we write both ends of it, they could both be 'wrong' and work just fine between our synths.

Bend Width RPN

Just in case it isn't completely obvious what happens when we change the bend width slider, here's the code that tells the synth our assumed bend width when we move that slider:


static void bendRangeSliderMoved() {
// Check the current bend range with the previous. If it has changed, send the appropriate RPN.
static int previousBendRange = -1;
int bendRange = bendRangeSlider.scaledValue;
if (bendRange != previousBendRange)
{
NSLog(@"bend range has changed from %d to %d", previousBendRange, bendRange);

previousBendRange = bendRange;

// Activate the pitch-bend sensitivity (bend range) RPN.
SoundEngine_sendMIDIPacket3(0xB0, 101, 0);
SoundEngine_sendMIDIPacket3(0xB0, 100, 0);

// Send the new bend range value.
//BUG: earlier versions sent 2*bendRange, which was wrong
SoundEngine_sendMIDIPacket3(0xB0, 6, bendRange );
SoundEngine_sendMIDIPacket3(0xB0, 38, 0);

// Deactivate the pitch-bend sensitivity RPN.
SoundEngine_sendMIDIPacket3(0xB0, 101, 127);
SoundEngine_sendMIDIPacket3(0xB0, 100, 127);

// Flush the MIDI queue.
midiPlatform_flush();
}

// Update the bend range with the sound engine.
SoundEngine_midiBendsInSemitones(bendRange);

// Flag that the texture needs to be updated.
bendRangeSlider.needsNewTexture = YES;
}

53ET 5-Limit in Geo Explained

Just Intonation
Just intonations are tuning systems in which pitches follow physically relevant ratios. The mechanics of making instruments has, for a very long time, dictated the use of equal tempered instruments. Without equal temperment, retuning adjustments will need to be made when playing in different keys. Equal temperment that uses a small number of pitches also makes the mechanical instruments easier to build because there are less parts.

But with electronic music, the justifications for only sticking to the 12 tone system no longer hold. Now it is just an issue of what people are in the habit of playing and listening to. With the rise of touchscreens as controllers, fretless controllers that can do dynamic intonation will become more common. Pythagoras started out as an attempt to do dynamic intonation. But when I got things working, 53ET became a defacto feature of it that emerged on its own. I eventually gave up on actual harmonics based intonation and just went with 53ET because it was so close, within the exact pixel in almost all cases, to the real spectrum locations.

I don't aim to expand upon this too much into the realm of academic microtonality. This is something that is designed to be practical. So instead of learning by generating big lists of floating point numbers and ratios for you to ponder and bamboozling you with equations, I am building geometry into an instrument that allows these ideas to be used and tested in practice. There is a little bit of multiplication going on, because that's easier than giving every little ratio a weird name for you to remember.

There is only so much you can do with timbre to make an instrument sound interesting. Getting the correct intonation is the first thing that almost all MIDI based instruments are giving up on. The pitches need to go in correct, and cannot be corrected by cleverness in the engine. Two waves in perfect ratios form a standing wave of a new shape, and do not come out as two separate notes. The octave is the only interval that we are used to hearing completely correctly, and the fifths and fourths are reasonably close. All the other intervals could use improvement by exploring the spectrum outside the normal 12 notes.

Relating Actual Harmonic Geometry to 53ET

I have posted before about the 53 note per octave scale, 53ET, with some lower quality images of its pattern. It is a highly special tuning system that captures the behavior of the spectrum in a practical way. It is explicitly used in Turkish Makam, a classical music system. A subset of this scale shows up in some form just about everywhere that flexible tuning instruments such as violin, sitar, and voice are used. The 12 tone system that we are all familiar with is interesting in its own way, but there is very little about that system that readily connects with the reality of the spectrum in an obvious way. The more you dig into 53ET, the more you feel like you have stumbled upon some ancient Magick. It really is a special system, and is much deeper than the 12 tone system we are used to.

Starting with a clean slate. Forget all about the twelve tone scale. Just think about a reference tone at some middle frequency, like a drone note. It doesn't matter what it's actual frequency is, because microtonality is about well defined relationships in a relative sense.In the top figure, we have a note that's a reference note of 1. It is in some rows that are stacked by pure fourths. Every note emits overtones that can lock with notes up above it. So the lines going up out of a note are the overtones that it emits. The lines going down are notes that can have an overtone that goes into our note. The unmarked line at an angle of about 2 o'clock is actually marked 2, the octave overtone.

The pitch ratios are simple numbers in accordance with the harmonic series, (N+1)/N. That is to say, 2/1, 3/2, 4/3, 5/4, 6/5, ... 16/15, ... 81/80,... etc. In this system moving up to a string multiplies the pitch times 4/3, because we are tuned to the pitch ratio of a Just fourth - which is 4/3. It happens to be 22 frets of 53ET along the same string. So we treat going exactly up to be the same as going 22 frets to the right. (It's the same concept as a bass guitar, where going up 5 frets is the same as jumping up to the next string.) So this second diagram shows whole tones as the most important unit, which is roughly two chromatic notes. ie: A to B for example. I managed to forget to note the fact in the drawings anywhere, but that whole tone is the ratio 9/8 ((N+1)/N, so that means that it's a special part of the harmonic series).

So any notes that are in the scale have some short path along these lines where you pick a note and draw out harmonics from it as the first diagram does. So, if we start at pitch 1, take the route 4/3 by going straight up, then the path 3/2 by taking that diagonal, these numbers multiply to give us the number 2. That is the definition of octave; twice the original frequency. 4/3 is called a "fourth", and 3/2 is called a "fifth", 2 is "octave", 1 is "root". And note that we can get a semitone of sorts by following 4/3 up, but travel down to 4/5 to land on 16/15. This means that if you chord the notes 16/15 and 1 together, 16/15 has major third overtone at 4/3, and the 1 has a fourth overtone at 4/3, so they are consonant because they link up by a short path. Notice also that even though we reached 16/15 by navigating that it's (N+1)/N, which means that it shows up on the harmonic series if we go far enough, as does 9/8.

And I do in fact mean to chord two notes so close to each other, an "A" and a Just semitone of "Bflat" 16/15 will sound like trash on a piano. It's a reasonable chord in this system because it's 1 versus 16/15 rather than 1 versus 2^(1/12), which isn't very close to any reasonable integer ratio.

So this whole system ignores the really high harmonic ratios and uses the ones that you can create from nothing more than octave, fourth, major third relationships. It happens to be that octave (2) and undertone (1/2) are related, as are fourths (4/3) and fifths (3/2), and the major and minor thirds. What is more is that if you pick out all the prime numbers in their ratios, then this system is everything you can make only out of the numbers involving 2, 3, 5; the first three prime numbers. This gives a way to "measure" how closely related two pitches are. There are no note names here, just relative relationships by pitch.

The real enlightening moment for me was when I had fixed a bug in the drawing that had caused major and minor third to not hit exact 53ET frets, that everything locked together very tightly. I could hear this and knew this long before I had the drawing right; I let my wrong math overrule what was obvious to my ears. So, here is an explaination of *why* major and minor triads are special.


When you play a minor third interval, there are 9 steps to the first wholetone (9/8) which are reached by 3/2 * 3/4 = 9/8. But from the fifth (3/2), you can take a major third down by 4/5 to reach 1 * 3/2 * 4/5 = 6/5. So the ratio 6/5 is the minor third as noted in the harmonic series above. 6/5 has a minor third harmonic coming off of it, and the root has a fifth coming off of it and they meet at the same point.

Similarly if you play the root and major third together, you can see that there is a minor third harmonic coming off of it that meets at the same fifth that the root meets. These two chords are simply stacks of a major and minor third, but in a different order. These three notes together along with the octave relate: 1/1, 2/1, 3/2, 4/3, 5/4, 6/5 all simultaneously. They form a rich set of harmonics in that sense. That's why triads are special.



One thing to keep in mind about 53ET though is that it's very close to a complete system that captures all octave, fifth, major third relationships. It is not exact in a mathematical sense. But it is very close, much closer than the 12 tone system, in the sense that you will only be able to discern larger ratios to a reasonable degree of accuracy and it's a very practical approximation to the spectrum.

Note: The scale I am showing up above is not using the Pythagorean choice for the third and the sixth. A semitone is generally going to be 4 frets rather than 5 when in those cases. Pythagorean scales are generally a span of notes that are adjacent in the Just circle of fifths (or fourths... same result), which causes third and sixth to come out one fret flatter if done that way. So my scale is considered wrong if the goal is Pythagorean tuning, or might be acceptable if you favor being closer to Just major thirds and Just minor thirds. Transposition is easier if the Pythagorean choice is taken. There is a short harmonic path between both choices, so it's really a matter of what it is you are trying to do. (I am getting authoritative information from somebody that actually does Makam, and I am trying to make sure that I am doing things right. :-) )


Another interesting thing about 53ET is that it contains the interval that is the difference between a pure major third (5/4) and two whole tones (9/8 * 9/8 = 81/64). This difference creates a lot of controversy in tuning systems. There is a tension between wanting something that is simple and tunable by ear using only fifths and octaves, and the close to major third tone that a simple Pythagorean tuning gives. So, 81/80 shows up in a lot of places related to tuning, in Indian music, etc. This interval causes a situation where the scale may not want to follow the simplest ratios created by following the fourths and fifths. In a minor-like scale, there will be a tendency to make minor third, minor sixth and the seventh one fret sharper than you might guess if you did the geometry without listening to the result.

A Note On Accuracy

If you go through this and verify the intervals (as my Makam playing friend did), you will find that you have to be careful about how you interpret the geometric picture. If we assume that going up is multiplying by exactly 4/3, then the path that you compute to an interval will be exact. It's a simple matter of multiplying ratios.

But that doesn't mean that it lands exactly on a a 53ET fret. But you can tell from actually playing Geo Synth that it is close to a high level of accuracy, and see exactly how far off it is when it's not exactly to the pixel on the screen. When doing all navigation as pure 3-limit, you will always get exact numbers (ie: 2, 1/2, 3/4, 4/3 are the navigational options, all else is derived) if you don't treat the circle of Just Fifths as closed, and make sure that you only navigate a span of 53 fifths without trying to wrap around at the point where it almost closes. However, it still applies that the 53ET frets themselves are an approximation, as 2^(1/53) is an irrational number.

It gets even more complicated when we declare that the interval 00 05 is a minor third and 00 08 is a major third (see tablature below). You can have different paths that reach the same fret, but don't come out to the exact same ratio. But this is ok, because if you subtract one ratio from the other you get a very small number that can be considered zero for all practical purposes; especially because we pick *shortest* path to determine how a ratio is created.

This problem was proposed to me:

I am at A. Where is F?

... navigate down two whole tones
8/9 * 8/9
= 2^6 * 3^-4

... navigate up four fourths then two octaves back down to get into range

(4/3)^4 * 2^-2
= 4^4 * 3^-4 * 2^-2
= 2^8 * 3^-4 * 2^-2
= 2^6 * 3^-4

These are the same because they use 3-limit intervals. But when we use paths involving the 5-limit, they don't come out identical. I was given 405/512. So I try to factor it into a path:

405/512 =
810/1024 =
81 * 10 / 1024 =
81 * 5 / 512 =
2^-9 * 3^4 * 5 =
5/4 * 2^-7 * 3^4 =
(3/2)^4 * 5/4 * 2^-3 =
(9/4)^2 * 5/4 * 2^-3 =
(9/8)^2 * 5/4 * 2^-1

up two wholetones, up a maj third, down an octave. But subtracting these intervals:

64/81 - 405/512 = -0.0000892168...


They are practically the same value, so both landed at the same fret. So, if you mix in 5-limit , then don't use high powers of them and accuracy will not get out of control. The use of this is that you can do the geometry in your head, and if you do it carefully it will be exact. But you can use approximations without the number crunching via 53ET.






Tablature


When I was faced with writing tablature for 53ET, I thought that the idea of notating with large fret numbers was ridiculous. Instead I took a different approach, that is compatible with 12ET tablature once you get used to it.

The dark blue lines every 9 lines in my grid are the markers for whole tones (9/8). In 53ET, whole tones cannot be split into exact semitones because they are split into 9 subdivisions. So the first numerical digit notes which dark blue line (marking a whole tone boundary), while the second digit denotes the lighter line above the whole tone division.

One interesting property that this has is that if you pretend that these are actually decimal numbers with a decimal in between them and multiply them times 2 (ie: "15" x 2 = 30 ... think of as 3.0); the numbers round off into the same locations as their 12ET tablature counterparts.

Ex: 00 06 15 becomes... 0.0 1.2 3.0 ... or 0 1 3 in guitar tab, where it's obvious that the middle note is intended to be sharper. But the notation is completely unambiguous. It tells you exactly what frets to put your fingers on, without making up strange symbols to try to add to the standard music system. (Note: I am targeting guitarists. Almost no guitarists fluently read standard music notation, and it's not really necessary for this.)


Geo Synth (Pythagoras)

So the main idea that got Pythagoras started (now called Geo Synth) was geometry. I wanted to avoid any equal tempered system, or anything artificial that didn't come from the spectrum itself. But when I draw out the 3-limit, which is the closure of all fourths and fifths (up to some reasonable distance) then it became 53ET. It was never my plan to support 53ET. It came to me. And it wasn't until recently that I realized that my ears were right and my drawings for the thirds were wrong, and fixed the drawings. Now the major and minor lines draw straight through 53ET frets. It is now clear that even the ratios 7/6 and 8/7 which are up next are not too far from 53ET as long as you only navigate that ratio once.

Here is a quick video picking out some of the obvious intervals:

http://www.youtube.com/watch?v=eViP4cJYvvE

The idea behind the interface was as explained above, to show all of the harmonics coming off of a note. You line them up until they lock visually. When they lock visually, the sound is obviously consonant.

So I have the normal twelve pitch system drawn over it, as well as quartertones (ie: 24Et which includes the normal 12 tones).

24ET is roughly the official Arabic intonation system. It doesn't line up so well with the harmonic series. It's main point is to take a core pentatonic scale, and don't play semitone stretches. Whenever you encounter a minor third, the note in the dead center between them (median second) is what you want to play. These are what I would call "red notes" on a piano. If you had red keys where black keys are missing between E/F and B/C on piano, they would have to be quartertone "red notes".

I think touch screens will usher in some initial experimentation with these kinds of scales at first. But after a while, it will sink in that Phrygian scales and Harmonic Minor scales are bad imitations of the Arabic system (especially Phrygian sharp fourth), and people will start adopting actual Maqam scales for certain scenarios. Then people will realize that with 53ET you can do this, while also getting even better chords out of it than we are accustomed to getting.

MIDI Instruments on iOS

MIDI Best Practices Manifestos

There is a MIDI Manifesto on Best Practices that is getting a lot of attention at the moment. They are focused on getting developers to write apps that cooperate to make CoreMIDI work really well on iOS. You can read it here:

http://groups.google.com/group/open-music-app-collaboration/browse_thread/thread/939e4f2998a8bc

This focuses on interoperability issues. It is very much from the perspective of developers playing piano-like instruments to drive traditional synthesizers and beat machines. This is very welcome news, and it could not have better timing for this to happen. Up until a few days ago, I did not know it was even possible to have a sound engine run in the background while a controller runs in a foreground app, communicating over CoreMIDI.

I am not an electrical engineer or a sound engineer, so the mandatory internal engine that I had to write was always something that was an especially time-consuming part of my app that yielded less than ideal results for all the work that went into it. I have always wanted to focus on making a controller with world-class virtuoso playable potential. The achievement above, using virtual MIDI ports, promises to free me up from having the sound engine be such a huge focus in what I do.

This achievement won't matter if the sound engines get the pitch handling wrong. So here are some issues for sound engine creators to think about, which you will encounter as soon as you plug in something that's not a piano.

The Skill Mismatch

Frankly, I think there are a lot of synthesizers on the iPad with great sound, but playability on the device itself seems to be unconquered territory. I don't think it's possible to make a synth on iPad that resembles a piano, and be able to consistently produce people that can play it better than a real piano. The small rectangular dimensions defy the layout, and the continuous surface fights against the discrete-key design of the instrument being emulated. A piano is wide and doesn't need to be so tall. As a result, most synthesizers are something like 80% controls and 20% playing area. This makes for a really small octave range. There is a lot of demand for synths to "support MIDI" because of this. The truth is, it's mostly so that you can at least plug in a 2 octave keyboard so that you can play at a reasonable level of skill on the thing. It is not so that you can take your iPad and play a piano-like app to drive a Korg Karma. Nobody in the real-world does that.

There is also the skill mismatch that bothers me. If you look at any rock band, you will notice something about their makeup. It is usually 4 or 5 guys. The keyboardist is optional, but he shows up more often than he used to. There is always a drummer. There's always a guitar player. There is almost always a bass player. In a lot of cases, there are two guitar players. The ratio of guitar players to piano players is very high. If you look at all the music stores in your area, you will notice a gigantic guitar section, and a smaller piano and keyboard section. I think that there are something like 5 guitar (including bass) players to every piano player.

The electronic instrument industry is out of balance in this regard. They don't get it.

TouchScreens versus MIDI

Code that actually implements these ideas (the AlephOne instrument). It works very well. But it pushes all of the complexity of MIDI into a client library, and forces the synth (the server) to be as dumb as possible so that the client can get whatever it wants:

https://github.com/rfielding/DSPCompiler

It is no coincidence that iPads and iPhones are posing a challenge to the instrument industry at this time. iOS devices are essentially rapid prototyping devices that let you make almost anything you want out of a touchscreen, audio, midi, accelerometer, networking, etc combination. iOS developers are becoming the new instrument manufacturers, or are at least doing the prototypes for them at a very high turnover rate.

Multitouch has a unique characteristic of being tightly coupled to the dimensions of human hands. It does away with discrete keys, knobs, and sliders. If you put your hand on a screen in a relaxed position with your fingers close to each other without touching, you can move them very quickly. Every spot that a user touches should be an oval a little bit larger than a fingertip. If you make the note spots any smaller, then the instrument quickly becomes unplayable. So, lining up all notes besides each other to get more octaves does not work. The awkward stretch to reach accidentals at this small size is equally unhelpful.

A densely packed grid of squares that are about the size of a fingertip is exactly what string instrument players are used to. So, a simple row of chromatics stacked by fourths is what we want. We can play this VERY fast, and have room for many octaves. It's guitar layout. Furthermore, the pitch handling can be much more expressive because it's a glass surface. We know exactly where the finger is at all times, and frets are now optional, or simply smarter than they were before. An interesting characteristic of a stack of strings tuned to fourths is that there is a LOT of symmetry in the layout. There are no odd shapes to remember that vary based on the key that is being played in.

Transposition is simply moving up or down some number of squares. This even applies to tuning. Just play a quartertone lower than you normally would and you are playing along with something that was tuned differently without actually retuning the instrument. It is a perfect isomorphic instrument, with a layout that's already familiar to 80% of players. This is the real reason why this layout has a special affinity for the touch screen.

This link below on a *phone*, and it's ridiculously easy to do. I could play at a similar level on Mugician 1.0 only a few weeks after I got it working. The layout matters. It matters more than a dozen knobs and sliders. You can fix a simple sound with external signal processing, but if you can't play fast, or if the controller drops all the nuances, or it has latency, then that can never be fixed by the sound engine later.

http://www.youtube.com/watch?v=FUX71DAelno&feature=channel_video_title

This layout works, mostly because of the way that pitch handling is done.

Fretless MIDI Messages

Synths that take in MIDI messages are only getting the things right that a piano exercises. They seem to consistently get everything else wrong in various ways. So without any more ranting, here is how MIDI must behave on touch screens:

A MIDI number is a number such that 0 is a very low C note. That low C note is the reference frequency, and I don't remember or know how many hertz it is at the moment. So we will just speak in relative terms to "midi note 0":

frequency = c0 * 2^(n/12)

I believe that A 440hz is Midi note 33. It has this frequency:

c0 * 2^(33/12)

But what happens when the MIDI note is not a whole number? That's a MIDI note with a bend. If we bend midi 33 up with 1/4 pitch wheel with the default wholetone pitch wheel setting, we get midi note:

33.5 ( frequency = c0 * 2^(33.5/12) )

"A quartersharp" if you must name it. So, when we want this pitch, we send midi messages like:

bend ch1 +25%

on ch 1 33

So imagine for a moment the coordinate system that normalizes all touches to fit in the rectangle <0,0> at the bottom left corner, and <12,6> for the top corner. This is an instrument with 6 strings stacked by fourths, with the lowest note in the bottom left corner. You don't have frets yet, but you know that at = <2.5, 0> that you are on the bottom string, half way between fret 2 and fret 3. The pitch is 2^(2.5/12). Every string you go up by adds 5 frets, and therefore multiplies by another 2^(5/12). There is a pixel-to-pitch mapping. This generates the exact frequencies that we are trying to represent as (note,bend) pairs in MIDI.

We bend up to 50% and get an A sharp note....

bend ch1 +25%

on ch 1 33

bend ch1 +50%

Bend up another semitone:

bend ch1 +25%

on ch 1 33

bend ch1 +50%

bend ch1 +100%

So we are up at the top of the bend. We can't do a vibrato now! We will exceed the bend width. So, if the synth was set for the bend to mean "percentage of an octave up or down", then we can do this...

bend +0%

on ch1 33

bend +(3/12)

To bend 3/12 of the way up to the note C. So now, we can do whatever we want as long as we don't get up to an octave. Because users will simply drop their fingers on the glass and actually drag the finger up an octave, this MUST be supported. This drag CANNOT happen on a piano, which is why it doesn't seem that this scenario is ever getting tested.

The core problem with MIDI's model of bends is that it assumes that there is one pitch wheel for the whole instrument. You have to bind together multiple monophonic instruments as one if you want every note to have its own pitch wheel. A finger dropped on the glass is a pitch wheel for that finger. These per finger movements are what gives string instruments their special character, which you cannot fix later in the sound engine after the gestures have been rounded off and lost. Here is an example of two of the same note being played and slightly detuned from each other:

bend ch1 0%

bend ch2 0%

on ch1 33

bend ch1 -1%

on ch2 33

bend ch2 +1%

off ch1 33

off ch2 33

This isn't possible on a piano. But this always happens on a string instrument. We have two concurrent instances of the same note. They are bent in different directions. But they are on the same instrument. Here is another challenge, where note overlaps will cause trouble for synths that don't correctly interpret multiple channels:

on ch1 33

on ch2 33

off ch1 33

on ch3 36

On synths that replace every channel with the same number before interpreting, not only did it get the bends wrong, but it also gets the above scenario wrong. We should be hearing ch2 33 and ch3 36 sounded together as a chord, not just ch3 36 by itself. As for the next scenario, of the problems when you have more notes to play than channels available, just make sure you don't get a stuck note on this:

on ch1 33

on ch1 33

on ch1 33

off ch1 33

This should sound three times with exactly one note, and no stuck note at the end. But this:

on ch1 33

on ch2 33

on ch3 33

off ch1 33

off ch2 33

This will leave ch3 33 still playing. At the time that ch1 and ch2 had notes on, it wasn't one note either. It was two instances of the sound playing, most likely out of phase from each other.

Channel cycling is a requirement to get the independent pitch wheels. Because note off message isn't actually the end of the note, we have to give the notes release time. This is something done in the controller, but synth engines should be aware of why this is being done. We want to maximize the amount of time that a channel has been dead before we steal it to play a new note.

So when we make a fretless instrument, we spread it across a channel span, such as channels: 4-8 for a 4 channel instrument. That allows for 4 notes down with each note having its independent pitch wheel. It means also that if you play 4 notes per second, then an old channel gets stolen (along with its pitch wheel) at 4 notes per second. This means that there is a speed limit to playing without anomalies in MIDI because of the small number of channels.

Chorusing For Real Men

Note that if you are playing fretlessly on an instrument that allows note duplication, that you don't need no stinking chorus effect. Just play the line simultaneously at two different locations:

bend ch1 0%

bend ch2 0%

on ch1 33

bend ch1 -1%

on ch2 33

bend ch2 +1%

off ch1 33

off ch2 33

You can switch between chorusing and chording as you play as well. With microtonality turned on, you can get exact pitch ratios such that the pitches don't sound like distinct notes, but end up being different wave shapes as well.

Note Tie Non Registered Parameter Number

It is not impossible to bend MIDI notes to any width you want at fullest possible resolution. the problem is that there is no defacto or dejure standard on how this is done. Imagine a piano player trying to simulate a bend, and it's on our channel cycling instrument....

bend ch1 0%

bend ch2 0%

...

bend ch16 0%

on ch1 33

...

off ch1 33

on ch2 34

...

off ch2 34

on ch3 35

...

off ch3 35

on ch4 36

...

So he's playing chromatics to simulate the bend because that's the best he can do. But if we are on a synth that inserts bend messages, the synth can at least bend from one chromatic to the next like this:

bend ch1 0%

bend ch2 0%

...

bend ch16 0%

on ch1 33

bend ch1 20%

bend ch1 40%

bend ch1 60%

bend ch1 80%

bend ch1 100%

off ch1 33

on ch2 34

bend ch2 20%

bend ch2 40%

bend ch2 60%

bend ch2 80%

bend ch2 100%

off ch2 34

on ch3 35

bend ch3 20%

bend ch3 40%

bend ch3 60%

bend ch3 80%

bend ch3 100%

off ch3 35

on ch4 36

bend ch4 20%

bend ch4 40%

bend ch4 60%

bend ch4 80%

bend ch4 100%

So, this would be a smooth bend, except we hear the note retrigger every time we reach the next chromatic. So let's say that we have a special message that notes that there is a note tie coming and that it's done when the next note on appears.

bend ch1 0%

bend ch2 0%

...

bend ch16 0%

on ch1 33

bend ch1 20%

bend ch1 40%

bend ch1 60%

bend ch1 80%

bend ch1 100%

tie ch1 33

off ch1 33

on ch2 34

bend ch2 20%

bend ch2 40%

bend ch2 60%

bend ch2 80%

bend ch2 100%

tie ch2 34

off ch2 34

on ch3 35

bend ch3 20%

bend ch3 40%

bend ch3 60%

bend ch3 80%

bend ch3 100%

tie ch3 35

off ch3 35

on ch4 36

bend ch4 20%

bend ch4 40%

bend ch4 60%

bend ch4 80%

bend ch4 100%

We can continue this from the lowest note on the keyboard to the highest for a super-wide bend. It is at the full pitch resolution as well because we aren't playing tricks with the MIDI bend width. It is also the case that if we broadcast this both to a piano that can't bend, and the synth that understands, we get a similar result. It degrades gracefully on the piano, and sounds perfect on the synth that understands. We can use this to track up to 16 fingers at arbitrary pitches (in MIDI range of course!) bending in whatever wild directions they need.

The NRPN looks like this in our code:

#define TRANSITION 1223

static inline void sendNRPN(int ochannel,int msg,int val)
{
//B0 63 6D
//B0 62 30
//B0 06 100
int lsb = msg&0x7f;
int msb = (msg>>7)&0x7f;
//midiPlatform_sendMidiPacket7(0xB0+ochannel, 0x63, msb, 0x62, lsb, 6, val);

midiPlatform_sendMidiPacket3(0xB0+ochannel, 0x63, msb);
midiPlatform_sendMidiPacket3(0xB0+ochannel, 0x62, lsb);
midiPlatform_sendMidiPacket3(0xB0+ochannel, 6, val);
}

static inline void retriggerNewMidiNote(int finger,float midiFloat,int vol,int expr)
{
int channel = midiFingerUsesChannel[finger];
if(channel >= 0)
{
int ochannel = midiChannelOChannelSent[channel];
sendNRPN(ochannel,TRANSITION,midiChannelNote[channel]);
}
stopMidiNote(finger);
startNewMidiNote(finger,midiFloat,vol,expr);
}


Let us know if there is something unreasonable about that message. I haven't used NRPNs before, and since we write both ends of it, they could both be 'wrong' and work just fine between our synths.

What Is It Useful For

There is a very practical use for this: Violin! Oud! You could even do an accurate rendition of the human voice to MIDI without auto-tuning it. Most real-world instruments exhibit this because the spectrum itself is actually microtonal, with the exact 2^(n/12) adjustments being something that can't actually be achieved in practice on an acoustic instrument. They are real-world resonating bodies afterall, and will resonate in whole tone ratios and have real harmonics. Acoustic pianos often use a stretched octave tuning to deal with this problem.

This opens up the door to rendering music outside the 12 tone tempered traditions as well. MIDI should be a rendering format that doesn't break the capability to do what you mean without injecting its prejudgement of what should be disallowed. MIDI itself, like auto-tune, seems to be one of the key factors in keeping electronic music sounding more like it was produced by a machine than acoustic instruments do. There has been a lot of progress in getting good timbre out of instruments, but this is meaningless if you round off all the nuances in pitch that are what really give the acoustic instrument its character.

I am working on a new project, and if you are already a tester or jailbroken, then you can download it here (it's very minimal, with the purpose being more to produce reuseable code than to ship something right now):

http://rfieldin.appspot.com

This is a similar post that puts it in perspective with more general instruments like this for iOS:

http://rrr00bb.blogspot.com/2011/09/multitouch-midi-recommendation.html

53ET Geometry fix in Geo Synth (Pythagoras)

This thing is getting very close to being shipped. Kevin Chartier (Jordan Rudess' programming partner at Wizdom) is doing a lot of user interface changes. We are working on at least making a stable and feature frozen version that's shippable in principle to have something ship by a well defined date, or just go ahead and ship that version and keep going with an update.

The entire time using 53ET in this synth, there was a minor bug in the geometry that is now fixed. The consequence was that even though 53ET emerged as a consequence of the fourths and fifths without being programmed in (or even known at the time I first saw it!), the major and minor thirds conspicuously didn't seem to line up very exactly with any of the 53ET frets; in spite of the fact that the maj fret slightly flat of 12ET and min fret slightly sharp of 12ET sounded astonishingly like correct Just intervals.

So now that this minor numerical fix was made, you can easily navigate visually around while playing to discover the Just Intervals that are so closely approximated in 53ET. This numerical problem I always had also affected the 12ET tuning very slightly, something that I noticed when working on the MIDI, but had written off as just floating point inaccuracies.

This is kind of like discovering an exact representation for Pi and having all of the goofy engineering inaccuracies disappear so that you can actually understand what you are looking at. Now that this is in, major and minor triads actually form well defined triangles visually. I should think about making those an explicit part of the visualization at some point. Playing around with the spectrum rather directly like this is a bit of a game on its own.

There will be more to come. I will ship this in the store soon, hopefully with Wizdom as planned; but some version of it will ship no matter what.

http://rfieldin.appspot.com

Pythagoras MIDI

The new instrument is undergoing some pretty big changes. I am still working with Wizdom on this (the guys are busy, so I am trying to get a lot of progress made until we meet again), and it will come out with a different name (because Pythagoras is taken). But I stumbled upon some code that completely demystified MIDI over WIFI and decided to give it a try. Within a few hours, I had something that sucked, but kind of worked.

Pythagoras won't really ever be a "MIDI controller", and still has a MIDI-Guitar mentality. The real sound engine is the audio engine, and it's finger and frequency oriented. What I mean by this is that there are no note up/down or note name/numbers. You can have multiple instances of the same note being played, and bend them all in different directions. It's fretless to the core. Here is a recent sound check (just before I got per-string legato and mono mode working in MIDI). It puts GarageBand's BigElectricLead patch in left channel versus the native sound on the right. The timing being a little off is due to WIFI latency/jitter versus my sound latency/jitter. Some of the note drops on the MIDI side are the issue I mention in detail at the bottom.

http://soundcloud.com/rrr00bb/soundcheck88

MIDI on the other hand forces you to sucessfully model an instrument as a box of buttons and sliders. It sounds possible to do correctly until you start looking at the zeroes and ones of the MIDI protocol and see its true limits. Specifically, you can model it as 16 independent boxes of buttons with their own pitch wheel. This *sounds* like enough to make microtonality and polyphony work well together, but the devil is in the details. The per channel pitch wheel defaults to only a lame whole tone bend width. You can set it higher, like an octave, but it's a very non-portable thing to do. If I am stuck at whole tone bends, then the note must retrigger if I exceed it. Because I don't have an actual pitch wheel, when I am at the top of my whole tone then an accidental note re-trigger is only one pixel away (because you bend with finger position of the note, not with a separate wheel that has a top and bottom).

The other issue is that even if I write all of my MIDI messages completely correctly spread out across 16 channels, some DAWs will gleefully mess up all my work and mash all the channels back down into one channel at playback (My GarageBand OSX!). When that happens, you have to write different MIDI (omni-mode MIDI versus single channel) out to the stream to account for overlapping instances of the same note so that notes don't stutter and cut each other short. (This isn't supposed to happen, but what your MIDI stream says to do, and what the brain does are always going to be two different things. :-( This is less true as you get more expensive gear, but people will give me a 1 star "worst instrument ever" if it doesn't work with some MIDI brain that doesn't do it right. A lot of people won't understand why MIDI does keyboard just fine, and this layout doesn't quite fit the mold.)

#90 means note on channel 0

#91 means note on channel 1

# second number in row means note number

# third number in row means volume

90 32 55

91 32 46

90 32 00

...

This is two fingers going down on the same note, something that's impossible on a piano, because they are mechanical keys. So one finger comes up, and if they are truly on different channels this sounds right. If GarageBand forgets that 91 means that that's a different copy of the same note, it will just turn that whole note off and I get silence when there should still be a note playing. I know the workaround and will write it in, but what you see is correct MIDI, and I have to write it wrong to make some cheaper brain happy while possibly getting a wrong result on higher end hardware. If both of these notes go down and are slightly bent in different directions, then they are *not* the same note and I can't use tricks to count how many times down and up and compensate.

Since MIDI is all about making keyboard controller (boxes of buttons and pitch wheels) manufacturers happy, they think of microtonality as tuning tables, and think that this problem is basically solved. (The microtonal google groups insist that MIDI is just fine with microtonality because of *.tun files, etc). But fretless instruments have no tuning tables. Each channel (ie: finger) is one long volume and expression changing bend. I only need an octave bend in practice, but fretless instruments by their nature can bend from the lowest to the highest frequency with no problems.

The thing about this instrument that feels so different is that even with the frets on, its fretless nature is always apparent. You can refret between 12, 24, 53, or 665 notes per octave and blur the lines between what is and is not a fretted instrument, especially when you set the snap speed (speed to drift to the closest fret) very low.

So, MIDI is in Pythagoras. I am actually *excited* about it because I can double the internal voice, which does expressiveness pretty well, with MIDI voices that do other things in a world class way. The main trick will be keeping complexity away.

I do NOT want to attract a lot of very high maintenance users that won't ever be able to agree on any simple controller configuration as an acceptable one. I am going to shoot for something that is expressive and simple. I know that MIDI kind of opens the gates of hell for a million new controls to play with.

Tablature For My Instruments

I don't make these instruments just to put yet another iPad instrument into the store. I am taking the high risk approach and trying to build up a community. Communities need a common language. So here's some tablature.

The new instru

ment is at least as playable as guitar. It is special in that it uses automatic octave switching. But that makes fingerings different. If you have Pythagoras octave rounding turned on, these are the basic rudiments. The fingering is indicated by {L,R} for which hand, with fingers {1,2,3,4} being first finger to pinky. finger 0 would be thumb, but I am not using it here.

There are no frets, so the notation is relative to whatever center you like. That's why negative numbers show up in a few places. This notation doesn't handle microtonality yet, but I I think that will work out well when I get it figured out. There are also a variable number of strings, so this notation just shows the number that are in use. With octave rounding, there isn't a lot of reason to go beyo

nd 4, with 3 being enough for describing common things.

Play these with smooth rhythm. And where the double bars are, repeat as often as you are in audible range, then go on to the next bar.

I haven't figured out how to notate legato techniques made possible by the polyphony slider. When you stack up multiple notes in mono mode, pulling your finger up will play a note as if you hit it. It's better than a string because you can do these legato techniques up or down, and not just up the string as a real string instrument.

Here is another example that starts to make the point of why I need to think about a custom notation. When you notate by 12ET frets, it becomes ambiguous if you really meant to specify nearby locations that don't really land exactly on the fret. In 53ET, the fourths that the strings are tuned to are Just fourths. The seconds that you play are Just seconds. (Well, both are approximations, but very much closer than 12ET is). So, if we define 9 of these steps to be a Just whole tone, then 6 whole tones is 9*6 = 54. We overshoot the octave by 1 fret. When playing Pythagoras, you can see the whole tones very clearly in the layout. So we index first digit by whole tone, and second digit by the small fret in the whole tone. The beauty of this is that it looks kind of like the base 10 number. It's close to it. If you mentally "double the numbers" and think of 05 as "1/2" and 1 as "2", and 15 as "

3, then this 53ET notation is exact, yet it is easily rounded to the 24ET and 12ET locations. So, you can use one notation that's unambiguous (rounding is unambiguous, guessing with hints on what you really meant by starting at 12ET and saying stuff like "a little more than slightly sharp", etc.... that's ambiguous). Anyways, this is my 53ET notation, and I find it more useful than writing out guitar tablature with marks to qualify exactly how sharp/flat, etc. I need to write some code to generate prettier tablature.

But at the moment, I am crazy wrapped up in MIDI. Part of the MIDI task was to get the internals right so that I can record a transcript of the original gestures, to emit some of this 53ET tablature. But I really get it from the response that having a good MIDI implementation is required to get anybody to pay attention long enough to what Pythagoras can do that MIDI instruments don't do well. More on the MIDI stuff later... It's getting there!

53tet, Makam, and Pythagorean Tunings

The use of 53 frets per octave in Pythagoras' primary microtonal mode was a bit of an accident. I set out to tune strictly to the harmonic series and avoid any equal tunings, but got pushed into it. If all you do is line up your strings to just fourths (ie: 4/3 ratio rather than 2^(5/12) standard tuning), then that tiny little offset sets off a chain of consequences. I just drew lines straight up and down for 4/3 coming out of each octave, and the Just 4th offset caused the 53 frets per octave to appear. A Pythagorean tuning is by definition, merely a walk around the circle of Just Fifths. You can stop at 12 and pick those as piano notes, and just realize that when you try to modulate, the primary keys will sound excellent, but some keys will sound bad because the piano pitches are fixed but not equally distributed. You can go around the circle in both directions to get distinct sharps and flats for the different situations and add 5 more sharps to get a 17 note piano with duplicate (but slightly differently tuned) sharps and flats. You could go around the circle of fifths a little more to pick 22 notes and get a set of notes that is roughly the Indian Sruti scale.

The point is that this is something being dictated by Physics rather than musical numerology. Pitches are important in this case, so notes should have something to do with the spectrum itself. My hope is to get people out of the habit of thinking of tuning and pitch as something that you hire some piano tuner guy to think about (or just let a computer dictate it all for you). This is because it isn't really the exact pitches that are important in microtonality, but the very distinct Timbre that gets created by perfect ratios in chords and sympathetics, especially ones that sound new because they can not exist in the 12 tone system.

But only a few days ago I really understood what this meant. Besides saying that if you tune your strings using the harmonics, then if you had a 53 fret per octave guitar you would be basically perfectly in tune after that procedure. Not perfectly in tune in the sense as being in tune with an electronic piano, but with the harmonic series which the piano tuning tries to approximate (somewhat badly); octaves, fifths, fourths and major thirds specifically are very much in tune.

But the other thing that it meant is that this system which matches so nicely with the spectrum is very much like the Turkish Makam system. A Just "Whole Tone" is the pitch you get by going up two Just Fifths and back down one Just Octave. You can very nearly use an equal division system to deal with it, but it isn't as simple as a whole number of Just Whole Tones in an octave. This is what that "Wolf Interval" story you hear about tuning is all about.

In chromatic scales, a whole tone is two chromatic "frets". A Just Whole Tone is slightly larger. 6 chromatic whole tones make exactly 1 octave. These chromatic whole tones are "unphysical" in that their sound physics justification is very rough. The spectrum kind of roughly breaks up into 12 equal-ish parts when you take all combinations of fifths and octaves, but it is really rough. It doesn't line up all that well.

The real Just Whole Tone if it were split into 9 equal parts would have about 6 of these whole tones in an octave. Except six Whole Tones overshoot the octave by an about that is very nearly equal one of these 9 equal parts.

Six whole tones:

6 * 9 = 54

That's 54 "frets". There are 53 frets per octave. So we fretted the fret just above the octave on that sixth Just Whole Tone. When playing Pythagoras, seeing the simultaneous overlay of the 12tone system over the 53tone system will really reset what you know about music theory. So it's a walk around the Just circle of fifths plus one more step. Now the Wolf Interval story is just common sense.

Just as a side note to give you numerological nightmares. The next closest approximation to Just 5ths requires a much larger, but very curious number. After 665 steps of a Just 5th, the difference with octave is totally imperceptible on its own, let alone when the error is spread across 665 frets. So if you go one fret beyond that, you are at 666 Just Fifths. But you don't need to go there, because the approximation is so close that there is no practical reason to keep going. The extremely tiny interval that makes up this fret step is called "The Satanic Comma" for obvious reasons.

Every few days I pick it up to play, I get a new insight that I didn't have before I started.

To me, learning is whole point of playing music. It's why I would prefer to write an instrument than to just be a consumer of it. It's why I roll my eyes at the sort of musicianship that has no patience for doing anything from first principles. It's why I enthusiastically learned theory when I played guitar, and am enthusiastically trying to come up with something new with respect to pitch now that people are starting to play fretlessly on touch computers.

See the similarity to this article http://en.wikipedia.org/wiki/Makam. When I saw the image, which I posted at the top, I didn't have to read any more (as it's mostly ethno-musicology stuff and jargon that doesn't interest me). The day before, I had decided that my instrument had a natural coordinate system of ( (9/8)^x, (4/3)^floor(y) ), where 9/8 was chosen because it was the closest line in terms of the distance of the interval from the circle of fifths. When I saw this Makam article, I finally went to make Pythagoras widen so that I could hit every interval exactly, which I needed to run on the phone. Compare it to the picture of Pythagoras, which emerged as a happy accident of my attempt at drawing the full 3-limit on the screen.