An OSC / ChucK Windows8 instrument

OSC

Up until now, I had avoided OSC because getting synths that understand it setup correctly was very inconsistent, if not difficult.  At the time, I was wringing all I could out of MIDI, or rather unhappily building internal audio engines - knowing that the result would not be as good as a battle hardened synth.  I have tinkered with Pd (hard to do non-trivial programming in brute-force visual spaghetti code), SuperCollider (a rather horrible language, but more complete programming capability), and ChucK (a little weird at first, but a great language - but performance is not necessarily great).  The other main issue was that before I found myself on an x86 tablet, the GPL license for SuperCollider and ChucK were problematic.  On iOS, you end up having to bake everything into a monolithic app.




But I really wanted to offload all signal processing into one of these environments somehow, and found out that ChucK does OSC really nicely.  It's pointless (and ridiculous) for me to spend my time implementing an entire windowing system because the native (or just common) toolkits have too much latency, and it's just stupid for me to try to compete with the greatest synthesizers out there.  So, I offloaded absolutely everything that's not in my area of expertise or interest.  The synthesizer behind it?

Here is a ChucK server that implements a 10 voice OSC synth with a timbre on the y axis (implemented a few minutes after the video above).  It's just sending tuples of (voice, amplitude, frequency, timbre):

//chucksrv.ck
//run like: chuck chucksrv.ck
"/rjf,ifff" => string oscAddress;
1973 => int oscPort;
10 => int vMax;
JCRev reverb;
SawOsc voices[vMax];
SawOsc voicesHi[vMax];
for( 0 => int vIdx; vIdx < vMax; vIdx++) {
  voices[vIdx] => reverb;
  voicesHi[vIdx] => reverb;
  0 => voices[vIdx].gain;
  440 => voices[vIdx].freq;
  0 => voicesHi[vIdx].gain;
  880 => voicesHi[vIdx].freq;
}
0.6 => reverb.mix;
reverb => dac;
OscRecv recv;
oscPort => recv.port;
recv.listen();
recv.event( oscAddress ) @=> OscEvent oe;
while( oe => now ) {
  if( oe.nextMsg() ) {
    oe.getInt() => int voice;
    oe.getFloat() => float gain;   0.5 * oe.getFloat() => float freq => voices[voice].freq;
    2 * freq => voicesHi[voice].freq;
    oe.getFloat() => float timbre;
    timbre * 0.025125 * gain => voices[voice].gain;
    (1 - timbre) * 0.0525 * gain => voicesHi[voice].gain;
   
    //<<< "voice=", voice, ",vol=", gain, ",freq=", freq, ",timbre", timbre >>>;
  }
  //me.yield();
}
while( true ) {
  1::second => now;
  me.yield();
}

The two main things about ChucK you need to decipher it are that assignment is syntactically backwards, "rhs => type lhs" rather than the traditional C "type lhs = rhs"; where the "@=>" operator is just assignment by reference.  The other main thing is the special variable "now".  Usually "now" is a read-only value.  But in ChucK, you setup a graph of oscillators and advance time forward explicitly (ie: move forward by 130ms, or move time forward until an event comes in).  So, in this server, I just made an array of oscillators such that incoming messages will use one per finger.  When the messages come in, they select the voice and set volume, frequency, and timbre.  It really is that simple.  Here is a client that generates a random orchestra that sounds like microtonal violinists going kind of crazy (almost all of the code is orthogonal to the task of simply understanding what it does; as the checks against random variables just create reasonable distributions for jumping around by fifths, octaves, and along scales):

//chuckcli.ck
//run like:  chuck chuckcli.ck
"/rjf,ifff" => string oscAddress;
1973 => int oscPort;
10 => int vMax;
"127.0.0.1" => string oscHost;
OscSend xmit;
float freq[vMax];
float vol[vMax];
for( 0 => int vIdx; vIdx < vMax; vIdx++ ) {
  220 => freq[vIdx];
  0.0 => vol[vIdx];
}

[1.0, 9.0/8, 6.0/5, 4.0/3, 3.0/2] @=> float baseNotes[];
float baseShift[vMax];
int noteIndex[vMax];
for( 0 => int vIdx; vIdx < vMax; vIdx++ ) {
  1.0 => baseShift[vIdx];
  0 => noteIndex[vIdx];
}
xmit.setHost(oscHost,oscPort);
while( true )
{
  Std.rand2(0,vMax-1) => int voice;
  //(((Std.rand2(0,255) / 256.0)*1.0-0.5)*0.1*freq[voice] + freq[voice]) => freq[voice];
  ((1.0+((Std.rand2(0,255) / 256.0)*1.0-0.5)*0.0025)*baseShift[voice]) => baseShift[voice];
  //Maybe follow leader
  if(Std.rand2(0,256) < 1) {
    0 => noteIndex[1];
    noteIndex[1] => noteIndex[voice];
    baseShift[1] => baseShift[voice];
  }
  if(Std.rand2(0,256) < 1) {
    0 => noteIndex[0];
    noteIndex[0] => noteIndex[voice];
    baseShift[0] => baseShift[voice];
  }
  //Stay in range
  if(vol[voice] < 0) {
    0 => vol[voice];
  }
  if(vol[voice] > 1) {
    1 => vol[voice];
  }
  if(baseShift[voice] < 1) {
    baseShift[voice] * 2.0 => baseShift[voice];
  }
  if(baseShift[voice] > 32) {
    baseShift[voice] * 0.5 => baseShift[voice];
  }
  //Maybe silent
  if(Std.rand2(0,64) < 1) {
    0 => vol[voice];
  }
  if(Std.rand2(0,3) < 2) {
    0.01 +=> vol[voice];
  }
  if(Std.rand2(0,1) < 1) {
    0.005 -=> vol[voice];
  }
  //Octave jumps
  if(Std.rand2(0,4) < 1) {
    baseShift[voice] * 2.0 => baseShift[voice];
  }
  if(Std.rand2(0,4) < 1) {
    baseShift[voice] * 0.5 => baseShift[voice];
  }
  //Fifth jumps
  if(Std.rand2(0,256) < 1) {
    baseShift[voice] * 3.0/2 => baseShift[voice];
  }
  if(Std.rand2(0,256) < 1) {
    baseShift[voice] * 2.0/3 => baseShift[voice];
  }
  //Walk scale
  if(Std.rand2(0,8) < 1) {
    0 => noteIndex[voice];
  }
  if(Std.rand2(0,16) < 1) {
    (noteIndex[voice] + 1) % 5 => noteIndex[voice];
  }
  if(Std.rand2(0,16) < 1) {
    (noteIndex[voice] - 1 + vMax) % 5 => noteIndex[voice];
  }
  //Make freq
  27.5 * baseShift[voice] * baseNotes[noteIndex[voice]] => freq[voice];
  xmit.startMsg(oscAddress);
  xmit.addInt(voice);
  xmit.addFloat(vol[voice]);
  xmit.addFloat(freq[voice]);
  xmit.addFloat(1.0);
  35::ms +=> now;
  <<< "voice=", voice, ",vol=", vol[voice], ",freq=", freq[voice] >>>;
}

What is important is the xmit code.  So, when I went to implement OSC manually in my windows instrument, I had to work out a few nits in the spec to get things to work.  The main thing is that OSC messages are just as simple as can be imagined (although a bit inefficient compared to MIDI).  The first thing to know is that all elements of OSC messages must be padded to be multiples of 4 bytes.  In combination with Writer stream APIs that don't null terminate for you, you just need to be careful to pad so that there is at least a null terminator with up to 3 extra useless null bytes to pad the data.  So, OSC is like a remote procedure call mechanism where the function name is an URL in ASCII, followed by a function signature in ASCII, followed by the binary data (big-endian 32bit ints and floats, etc).

"/foo"   //4 bytes
\0 \0 0 \0   //this means we need 4 null terminator bytes
",if"          //method signatures start with comma, followed by i for int, f for float (32bit bigendian)
\0              //there were 3 bytes in the string, so 1 null terminator makes it 4 byte boundary
[1234]    //literally, a 4 byte 32-bit big endian int, as the signature stated
[2.3434] //literally, a 4 byte 32-bit big endian float, as signature stated 

There is no other messaging ceremony required.  The set of methods defined is up to the client and server to agree on.

Note that the method signature and null terminators tell the parser exactly how many bytes to expect.  Note also, that the major synths generally use UDP(!!!).  So, you have to write out things as if messages are randomly dropped (they are.  they will be.).  For instance, you will get a stuck note if you only sent volume zero once to turn off voice, or would have leaks if you expected the other end to reliably follow all of the messages.  So, when you design your messages in OSC, you should make heartbeats double as mechanisms to explicitly zero out notes that are assumed to be dead (ie: infrequently send 'note off' to all voices to cover for packet losses).  If you think about it, this means that even though OSC is agnostic about the transport, in practice you will need to at least design the protocol as if UDP is the transport.

OSC Ambiguity

So, the protocol defines little more than a verbose RPC syntax, where the names look like little file paths (where parents are the scope and bottom most file in the directory is the method name to invoke).  You can make a dead simple instrument that only sends tuples to manipulate the basic voice spline changes (voiceid, volume, frequency, timbre).  It will work, and will be a million times easier than trying to do this in MIDI.  Everything, including units is up to you (literal hz frequencies? floating point 'midi numbers' which are log frequencies?, etc.).  That's where the madness comes in.

If you use OSC, you must be prepared to ship with a recommendation for a freeware synth (ie: ChucK or SuperCollider), instructions on exactly how to setup and run them, and an actual protocol specification for your synth (because some synth you don't have doesn't know your custom protocol).  This is the biggest stumbling block to shipping with OSC over MIDI.  But I have finally had enough of trying to make MIDI work for these kinds of instruments.  So, here is an OSC version.  It really is a custom protocol.  The OSC "specification" really just defines a syntax (like Json, etc).  Implementing it means nothing for compatibility, as it's just one layer of compatibility.  But if you plan on using SuperCollider or ChucK for the synth, it's a pretty good choice.  You can scale OSC implementations down to implement exactly and only what you need.

Thought Experiment: Forget MIDI and OSC

 

Thought Experiment: Forget MIDI and OSC

MIDI

Trying to wrangle MIDI into handling polyphonic bending is so difficult that almost no synths get it right.  If you disagree with this statement, then you are surely spending all of your time on discrete-key instruments; a small subset of the kinds of instruments you can make on a touch screen.  If you are using MIDI for something that is polyphonic and fretless, then you will notice that very few synths can do this correctly.  Most of the capability is there if the synthesizers were to simply behave like multi-timbral synths, even when there is one voice assigned to all channels; at that point, it's an issue that setting up bend width is problematic from the point of view of ubiquity.  But MIDI's biggest problem is that it thoroughly vexes end users when there is a requirement to span across all MIDI channels to make a single instrument.  I do nothing but deal with support email for the 90% of synths that don't handle much of anything outside of piano controllers correctly; even if I document in detail what is supposed to work and what you should not try to make work (ie: Arctic vs Animoog, etc.).  MIDI's biggest problem is that the first problem that any music protocol should solve is to simply express pitch and volume correctly.  Maddeningly, MIDI just can't do this because it's very much note and discrete-key oriented.  MIDI is strangling progress on touch screens just as much as it helps progress.  Music notes do not belong in protocols, as they are a premature rounding off of the data.  We must deal with frequencies directly, or everything turns into a mess when handling real-world scenarios from touchscreens.

http://rrr00bb.blogspot.com/2012/04/ideal-midi-compatible-protocol.html

Read that link if you want to know what an unholy mess MIDI can be when you try to do something as simple as get correct pitch handling; where the situation is untenable when going to microtonal and fretless MIDI.

OSC

OSC on the other hand could readily solve all of my technical issues because it can easily represent what I am trying to do.  The problem is that almost nothing implements it.  And where it is implemented, only the syntax is standardized.  It's very much like the situation where you open a *.xml file, and haven't got the faintest clue as to what its semantics are, or what program is supposed to consume and create this format.  Even worse, most "OSC" implementations transport over MIDI.  This is like tunneling floating point numbers over integers; doing things conceptually backwards.  It's a lot of useless indirection that simply guarantees that nobody implements anything correctly.

The Simplest Possible Protocol That Can Possibly Work

So, what happens if I just forget about all standard protocols, and design the simplest thing that could possibly work?  I am convinced that it would be easier to get that standardized than it would be to subset the complex protocols we already have.  Observe the following:

• The music instrument industry currently has almost nothing to do with the new music instruments now.  The music instrument industry is mostly just chugging along in the same direction it has been going in, using tablet devices for little more than weak signal processing brains, or re-configurable knob/slider surfaces.  Everything they announce is just another piano-controller with a bunch of knobs/sliders, and a brain.  It isn't say... guitars that can do what all synths and all guitars can do (correctly!  MIDI can't do the basic pitch handling.)  It isn't say... violins... either.  It isn't microphones that can do the opposite of auto-tune and take a human voice and play instrument patches at the exact frequencies sung into the mic (even if none of the notes are close to the standard 12 notes).  MIDI is the root cause, because the protocol forces you into a discrete-note oriented mindset.  It's a mistake to think that the music instrument industry is really relevant here; we need to do what is best for tablet apps first.
• Almost everybody using tablets is reluctant to deal with MIDI cables or Wireless connections anyhow.  The reasons vary from latency concerns, to setup complexity, to a kludgy CameraConnectionKit way of getting connected.  We are standardizing on MIDI only because that was an easily available low-latency pipe.  It's weird that you need to use MIDI protocol just to use the pipe.
• Since the tablet is both the controller and the synthesizer, there is no reason to let obese hardware oriented specifications slow us down.  Presuming that you needed to fix something, you would get a result an order of magnitude faster if you simply get things working and publish the protocol and wait for the hardware vendors to follow the lead of a popular app that implements it, than to get the MIDI or OSC groups to make a necessary change for you.

So the main thing I need (what kills me about MIDI) is stupidly simple.  I just need to control independent voice splines, with continuous updates.  There is no need for a complex protocol to do this.  I write my own so that it's easy enough that I can describe it to any developer.  So 90% of it looks like this:

//When controller sends messages to the synth, the most basic method just
//sets voice properties.
//
//64 bits, all voice properties are set separately
struct SetProperty {
  int16 voice;  //there are no notes, just voices with multiple properties
  int16 property; //common stuff: phase,amplitude,pitch,timbre[i],etc.
  float32 val;
}

//Define standard mappings that everything is going to understand.
#define PHASE_property 0 //val==0 means sample begin, 0..1 is the period corresponding to freq
#define LOGFREQ_property 1 //val==33.5 would be pitch of 'midi' note's pitch 33.5
#define AMPLITUDE_property 3 //val==1.0 would be full amplitude for wave
#define TIMBRE_0_property 16 //first timbre control - assume it was mapped
#define TIMBRE_1_property 17 //next timbre control - assume it was mapped

This would handle everything that Geo Synthesizer and Cantor can do.  This is enough to handle polyphonic instruments that may be samplers or ADSR synths, because we explicitly set the phase.  This is because when a finger goes down, it maps to a voice (NOT a note!).  That voice will have its phase set to zero (SetProperty message 1), then its frequency set (next message), its timbres (next messages), then amplitudes (next message) set.  Then as the note is held, the pitch can change as the finger moves around or when amplitude must be changed.  Just send new SetProperty values to do aftertouch affects.  This is totally unlike MIDI, which treats aftertouch as special cases.

Note that timer stuff is not in the protocol.  That's because we presume to send the message at the time we want it interpreted.  Having timestamps in the protocol only adds latency or helps when things are in big batches (a different and more complex protocol that we should stay away from for now).

Negotiation

Building in a simple negotiation from the beginning helps to ensure that synth and controller are never sending unintelligible data to each other.  MIDI handles this situation very badly, where you end up having to enumerate synths in an ever growing list (assumption of central control).  As an example, presume that the controller and synth simply exchange lists of properties that they send and understand.  We re-use SetProperty, but use some well known values to note that we are simply exchanging agreement:

#define IUNDERSTAND_voice -1
#define ISEND_voice -2
//ex:  controller sends 64 bit SetProperty messages:
//  (-2,0,0.0),(-2,1,0.0),(-2,3,0.0),(-2,16,0.0),(-2,17,0.0)
//       which means "I send: phase,logfreq,amplitude,timbre0,timbre1"
//if we don't send any "I understand" messages, then the controller knows that this is
//a one-way conversation, and will not try to send messages to the controller.
//
//if we get "I understand" messages from the synth, then we must not send messages
//not in its list.
//

The whole point of this process is to simply recognize that rather than announcing a vendor or product id (which the other side may have never heard of), we announce what we expect to handle gracefully instead. 

Proxying Controls

The other thing besides controlling voices that we would need to do in a instrument controller is to have some mechanism to proxy knobs/sliders, etc of the synths in the background.  This is really important on iOS because we can't have the kind of setup mess that a typical hardware synthesizer user would deal with.  Because we have a negotiation mechanism, we can safely include any baroque sub-protocols that we need.  Presume that we have a message to say to start a blob of bytes.  We can use this to send a string.  The synth would like to name the controls (rather than moving them around...the controller uses fixed locations, and the synth may want to rename them):

#define BLOB_voice -3 // (-3,24,0.0) means to expect 24 bytes out of the stream (strings)
#define RENAME_voice -4 //(-4, 16, 0.0) means to expect a blob and use it to rename timbre_0

ex: Rename timbre_0 to 'Filter' (6 byte name):  (-4,16,0.0) (-3,6,0.0) 'Filter'

A synthesizer may want to expose the options to the controller.  So it would need to send a list of mappings to properties to strings, and a mechanism to remap standard properties to synth-specific ones.   Say that the synth set messages like:

#define IHAVE_voice -5 //(-5, 100,0.0) (-3,24,0.0) 'Filter'
#define REMAPFROM_voice -6 //(-6,16,0.0)(-7,100,0.0)
#define REMAPTO_voice -7

ex Remap standard timbre_0 to synth's 'Filter' at 100:
  (-6,16,0.0)(-7,100,0.0)

Where we knew to use 100, because earlier we got a IHAVE that gave us the name of the control that we showed to the user.  Once remapped, this;

  (3,16,0.5)  //voice three's timbre_0 set to 0.5

Has the same effect on the synth as:

  (3,100,0.5) //which is the same as 'Filter' set to 0.5

Carbculus

Carbculus

I am a newbie at doing any kind of detailed carb and insulin counting.  Previously I just tried to count from labels and make my meal hit specific targets like 60g and keep the insulin constant.  I have no idea how much any of this lines up with reality, so I am experimenting with it to see if I can find out what these actual values are (in as much as the equations line up with reality).  Obviously, there are always confounding factors like exercise.  But the point of this is near the bottom where I try to figure out how many carbs I ate by looking at the difference between what I tried to do, and what happened.

U is the units for insulin
CIR = Carb To Insulin Ratio (carbgrams / U)
SIR = Sugar To Insulin Ratio ((mg/dL) / U)
T = Target sugar reading (mg/dL)

R = Sugar reading just before eating (mg/dL)
F = Food (carbgrams)
I = Insulin (U)

Example values that mostly stay the same:

CIR=4, SIR=15, T=120

Example values that you plug in every time you eat:

F=45, R=93

Basic Insulin Calculation, of how much you need to stay on target while eating:

InsulinTotal = InsulinForFood + InsulinForCorrection
I = F / CIR + (R-T) / SIR

So I always divide food carbs by 4 (CIR) and how far off I am by 15 (SIR) and try to reach 120 (T).  If I eat what I thought was 45g of food and I am at 93.  This is simple enough that you can routinely do the calculation in your head:

I = 45/4 + (93-120)/15
  = 45/4 - 27/15

Roughly 10 insulin required.  But one thing I never hear anybody talking about is trying to actually measure how many carbs you actually ate; because all of the carb amounts are guesses unless you revise the estimates with actual measurements.  The problem is that all the variables that we plugged in were exact except for F which is a guess; one that could be very wrong.  If you actually did eat 45g of carbs, then you should be at 120 later; which is the whole point of taking the insulin.

So, I take another measurement when I expect it to be 120.  I got a new variable

L = Landing sugar reading (mg/dL)
G = Actual food (carbgrams)

So, you inject the insulin with a guess of F carbs at reading R, then measure your sugar later and call it L, and find the actual carbs G.  And I rewrite the basic insulin calculation to plug in landing sugar instead of target sugar and solve for food.  In essence, I write it as if my landing blood sugar was intentional and figure out how many carbs I ate:

I = G / CIR + (R-L)/SIR

Which you manipulate to get:

G = CarbsWeInjectedFor + CarbsPerSugar * HowFarOffWeWere
G = CIR * I + CIR/SIR * (L - R)

So say I landed at 130 rather than 120.  You do it in your head like this:

G = 4*13 + 4/15 * (130 - 120)
    = 4*13 + 4/15 * 10        #off by 10 sugar, injected 13
    = 4*13 + 8/30 * 10
    = 2*26 + 8/3
    = 52 + 8/3

Roughly 55 carbs.  You can memorize CIR/SIR (ie: 8/30), so that this isn't any harder than the basic equation.  There doesn't seem to be a lot of point in getting highly precise because carb counting involves guesswork, as does the way that you are going to consume it in your body.  But this tells me that I can go over all the numbers in my pump and look for systematic carb counting errors (ie: I assume that I am always trying to reach 120 and I have before/after meal readings).  So, you get the number I from the pump, which will take in F and R as parameters each time you eat(with all of the others set to constants), and plug in L 2 hours later as a check.  A little python program that I haven't used yet (I just came up with the carb regression a little while ago... So I will see how it goes):


CIR = 4.0
SIR = 15.0
T = 120.0

def EatAt(records,n, F, R):
  records[n] = {}
  records[n]["F"] = F
  records[n]["R"] = R
  records[n]["I"] = F/CIR + (R-T)/SIR

def Regress(records,n,L):
  R = records[n]["R"]
  F = records[n]["F"]
  I = records[n]["I"]
  records[n]["L"] = L
  records[n]["G"] = CIR * I + CIR/SIR * (L-R)

records = {}
EatAt(records, 0.0, 60.0, 80.0)
Regress(records, 0.0, 130.0)

print records[0]

Also, when my sugar is low, I would like to know how many carbs to reach my target without injecting (ignoring Basal for the moment):

I = F / CIR + (R-T) / SIR

0 = F / CIR + (R-T) / SIR

(T-R)/SIR = F/CIR

CIR/SIR * (T-R) = F

4/15 * (T-R) = F

So again, memorize CIR/SIR, which is 4/15 for me.  If I am 50 points under (ie: 120 is Target, and 70 is Reading), then I want 4/15 * 50 carbs.  4/15 is roughly one fourth, so I need 13 grams of carb.

4/15 * 50 is about 13

http://www.github.com/rfielding/BloodSugar

Co-Routine Refactoring

Co-Routine Refactoring

Say that you are given an arbitrary function in C code that typically runs in a blink of an eye and returns a value, and you need to be able to run it incrementally or partially for some reason.  And you want to do this without making large structural changes to the original function.  You could be implementing concurrency in a single thread, stepping through the function, implementing a generator, or making a generalization of a generator such as a symmetric co-routine.

int fun(int a, int b, int c)
{
  int d=0;
  int e=42;
  int f=2;
  int i=0;

A:

  foo(&d, &e, &f, &i);
B:
  for(i=0;i<10 font="font" i="i">
  {
    int g;
C:
    bar(&d, &e, &f, &i);

    baz(&g);
D:
  }
E:
  buxx(&d, &e);
F:
}

So this is our original function.  We want to be able to stop and inspect it at the points that are labeled.  If we want to be able to exit at D and resume at D by just calling fun, then we would have to live with the restriction that foo can't depend on anything that gets modified in the loop.  Eventually, this restriction is likely to get violated if this is a complex function and the transformation will have to be done anyhow.

The first part of this transformation is to pack all local variables and parameters into a structure to completely save the state of where we are:

struct fun_task {
  int a,b,c,d,e,f,g,i;
};

The next step is to figure out a state machine that gives us the stopping points that we need.  We can inspect fun_task at this point, which happens to be every single piece of state that is going on in this function:

//A steppable version of our previous function, 
// where you can use the source of this function
// as documentation on how to drive the 
// lower level set of functions:
int fun(a,b,c)
{
  int ret;
  int done=0;
  fun_task t;
  fun_init(a,b,c,&t);  //Up to A:
  do {
    //fun_loopBody decides what to 
    //do next based on its internal state.
    //it could be an FSM or more typical code, etc.
    done = fun_loopBody(&t); //C to D
  }while (!done);
  ret = fun_end(&t); //E to F
  return ret;
}

So, the single function fun is replaced with a bunch of functions that trace the life of the fun_task.  Where this gets interesting is when you add more parameters to the task functions.  If you add a parameter to loopBody called is_printable, and add debugging statements conditional on this parameter, then is_printable can vary per iteration rather than having to be the same after each iteration.  loopBody can *set* variables on each iteration as well.  So, this really is a co-routine with the ability to yield with parameters, resume with parameters, and to even let the caller inspect and manipulate state.  One tedious task that has to be dealt with because there is no language support for restoring context (ie: autos and parameters), is that these little functions reference the context like this:

  bool fun_loopBody(fun_task* t)
  {
    bar(&(t->d), &(t->e), &(t->f), &(t->i));
    baz(&(t->g));    
    int done = !((t->i) < 10);
    i++;
    return done;
  }

There is no construct in C to pass in a parameter to bind a struct to the autos like this fictitious item:

bool fun_loopBody[fun_task]()
{
  bar(&d,&e,...);
  ...
}

So we end up doing it manually as above.  Also note that yet another approach to this would be to mark the function somehow to automatically generate the function prototypes for stepping through the task, so that the original code can continue to look like a simple function.  (That would mean that you would also have to mark it up with parameters for yielding and resuming as well at these points)

OOP Digression and State Machines

Notice that this is kind of similar to OOP, in the sense that all state for this task is created inside of a struct and used for the duration.  The fun_* methods would just be methods in that case.  But there is a huge difference in that the methods have a very definite legal grammar on their calling order, and this aspect of these methods is far more important than any of the other aspects of OOP.  In fact, the type system can be used to document this order by taking a different perspective on OOP.  The first thing would be to separate the idea of the type of the state object from the set of methods that are allowed.  Conceptually, the id of the object (its pointer) would stay the same, though the interface would change based on what state it's in.  Ie:

//FlushedFile can only read/write/close
FlushedFile f = File("/etc/passwd").open();

//We can't close until it's flushed 
//- notice that the type *changed*
UnflushedFile f = f.read(&buffer);
//f.close() isn't available until it becomes a flushed file

FlushedFile f = f.flush();
f.close();

So, this concept is called "Protocols" in languages that implement it, like Sing#.  In Erlang, message passing order is probably the most important thing to document, which is why they don't think too much about type safety beyond preserving the integrity of the run-time system.  Anyways, it either passes around state from object to object when implemented as OO, or changes available methods based on known state (or possible states known at compile-time) if that is supported.

A major problem with the OO way of thinking is that the type system generally only documents what the function signatures are.  It's even more messed up when the concept of 'messages' gets warped in a concurrent context (where the messages should behave like an incoming mailbox like Erlang where work happens inside the object in a way that looks single-threaded to the object, rather than executing in the caller's thread like C/Java do).  But in any case, the concept of messages is the most important thing to get right, and it's basically neglected.  Beyond constructors and destructors, it's quite a rare thing for the type system to provide good hints as to the intended calling (messaging) order.  But a well defined specification should at least do these things:

• Document exactly what the object requires, generally through dependency injection where real dependencies are just on interfaces.
• Document legal calling orders (messages) that imposes enough constraints on the state machine that the code so that you don't end up having to support wrong usage that worked in earlier versions of the calling code, or write too much documentation on how to do it right.
• Try to have functions/methods per state so that the state machine is not so damn implicit.  When you look at a stack trace, the names of functions on the stack should be obvious when naming states.  The (nested) FSM that you would draw on the markerboard should bear some resemblance to the actual code.  This is a little more difficult to do in the absence of tail calls (where switches often emerge in their place), but it's important to do this where possible.

Back to C Functions and Refactoring In General

So, the concept of coroutines/tasks in C is basically just one of taking a function and breaking it into pieces by pulling all of its state (autos and parameters) into a struct.  It is more general than an object, since it is up to you to decide if you want to segregate the state into multiple pieces.  This is a more useful pattern than it may seem to be at first because of what happens when you try to refactor C code.

I judge the goodness of a language (and development environment) largely on how easy it is to turn crappy code into good code over a long period.  One major weakness of many languages when it comes to refactoring is that a lot of languages don't support multiple return values.  If most languages did support multiple return values, then function extraction would be trivial to do automatically in an IDE.  For example:

int foo(int x)
{
  int a=0;
  int b=0;
A:
  a = 1 + b;
  b = 2 * c;
B:
}

Extract A to B as a function if you have multiple returns:

int foo(int x)
{
  int a;
  int b=0;
  int c=0;
  a, b = doAtoB(b,c);
}

With multiple return values, it is very clear that b,c are in values, and a, b are out values.  We didn't have to pass in a pointer to a and wonder if doAtoB uses the original value of a, which is uninitialized.  With only single values returns, you would need to return structures (which are allocated in their own chunk of memory somewhere), or use the function pointers where it's not clear if the value is read and written, or just written.  Google's language Go actually uses this feature to lay down the law that the last parameter returned is an error code, and you are never to try to mix actual return values with error codes (ie: a comparator that returns a possibly negative integer, yet returns -1 when something went wrong, which can cause infinite loops in sorting algorithms by creating inconsistent sorting orders where a < b && b < a, etc.)

But the pattern of pulling all autos and parameters into a struct makes it very easy to break a giant function into a bunch of pieces without having to worry about changing how it behaves.  The alternative to this is to break into a bunch of smaller functions with a *lot* of parameters.  But the big downside to this is that these parameters can expose private types in the method signatures, or change the overall semantics of the function when it becomes unclear how to plug parameters back in when going from step to step.  In a big struct, some state may exist for a little longer than is strictly required, but pulling loop bodies into a single line function is now a very mechanical process.  Then perhaps with a little re-ordering of the code in the function, the parts can be given sensible names and put into functions that require no further commentary.

Rates and Latency

This Google talk is about latency, and it is relevant to making music apps.


And here is something being done by John Carmack, where the issue is ensuring that a head-mounted display tracks the image closely.  The immersive feel is totally due to getting low latency so that when you move your head around, you don't detect lag in movement.



Carmack recently lamented in a tweet that typically, you can send a transatlantic ping faster than you can get a pixel to the screen.


Rate Hierarchies

So, there are some analogies here that are relevant.

•  Render an Image Frame (width*height pixels) at 60fps.  60fps is the Screen Refresh Rate.  Frames are rendered on a VSync, which is a period at which you can draw to the screen without image tearing artifacts.  If you miss this deadline then you will get choppy video.  Color depth is generally going to be 8 bits per color, or 32-bits total for red,green,blue,and alpha.
• Touches come in from the OS at some unknown rate that seems to be at around 200fps.  I will call this the Control Rate.  It is the rate at which we can detect that some control has changed.  It is important because people are very sensitive to audio latency when turning touch changes into audible sound changes.
• Like the Image Frame rendered in a Screen Refresh, an Audio Frame is made of samples (an analog to pixels).  At 44.1khz, about 5ms requires 256 samples.  5ms is a rate of about 200fps when you think of sending a finished frame off to the audio API like an audio VSync.  If you miss this deadline, you will get audible popping noises; so you really cannot ever miss this deadline.  The bit depth is going to be either 16 or 32 bits, and the internal processing before final rendition is generally going to be in 32 bit floating point arithmetic.  And analogous to color channels, there will generally be a stereo rendition which doubles the data output.

 With video, the framerate is about "smoothness" of the graphics.  Taking the analogy to audio, you can think of it in terms of frequency.  At 44.1khz, most audible frequencies can be represented.  Even though the Image Frame rate of 60fps is sufficient for "smoothness", it may not be sufficient to give a notion of instantaneous visual response.  It is unclear if there would be any benefit to ensuring that graphics run at the control rate.  Anyways, given the number of pixels rendered per-frame, a limit of 60fps does not sound unreasonable.

In the same way, audio frames could run at 60fps (1024 samples), and the "smoothness" of the audio is not an issue.  But taking 23ms to respond to control changes feels like an eternity for a music instrument.   The control rate really needs to be somewhere around 5ms.  Note that the amount of data that would be emitted per Audio Frame is roughly the same amount of data as 1 line of the display frame in most cases.  But we need to render them at a rate of 4 to 8x as often.

CSound is a very old and well-known computer synthesizer package, and one of the first things you put into the file when you are working on it is the control rate versus the sample rate.  The sample rate (ie: 44100 hz) is generally about 10x to 20x the control rate (220 hz) and the control rate is 4 to 8x the screen refresh rate (60hz).

GPUs and Rates

So, with this heirarchy of rates in use, it is fortunate that the higher rates have less data that is output.  A GPU is designed to do massive amounts of work during the period between VSync events.  Video textures are generally loaded into the card and used across many Image Frames, with the main thing being that at 60fps geometry must be fed in, and in response (width*height) pixels need to be extracted.

To use the GPU for audio, it's only 4 to 8x the rate (to match the control rate).  And because the sample rate is going to be on the order of 200x the control rate, then we expect to have to return merely hundreds of samples at the end of each audio frame.  This way of working makes sense if there is a lot of per-sample parallelism in the synthesis (as there is with Wavetable synthesis).

http://www.music.mcgill.ca/~gary/307/week4/wavetables.html

Similar to video textures, the wavetables could be loaded into the GPU early on during initialization.  At that point, making control changes and extracting new audio buffers is going to be generating all the traffic from the GPU to the rest of the system.  If there were an audio equivalent to a GPU, then the idea would be to set the sample rate for the Audio GPU, and simply let the card emit the samples to audio without a memory copy at the end of its operation.  Currently, I use vDSP, which provides some SIMD parallelism, but it certainly doesn't guarantee that it will run its vector operations across such a large sample buffer as 256 samples in one operation.  I believe that it will generally do 128 bits at a time and pipeline the instructions.

In fact, the CPU is optimized for single thread performance.  Latency is minimized for a single thread.  For GPUs, latency is kept low enough to guarantee that the frame will reliably be ready at 60fps, with as much processing as possible allowed at that time (ie: throughput on large batches).  An Audio optimized "GPU" may have a slightly different set of requirements that is somewhere in the middle.  It doesn't emit millions of pixels at the end of its completion.  But it could have large internal buffers for FFTs and echoes, and emit smaller buffers.  In fact, the faster it runs, the smaller the buffers it can afford to output (to increase the control rate).  In the ideal case, the buffer is size 1, and it does 44100 of these per second.  But realistically, the buffers will be from 64 to 1024 samples, depending on the application.  Music instruments really can't go above 256 samples without suffering.  At a small number of samples per buffer, the magic of this GPU would be in doing vector arithmetic on the buffers that it keeps for internal state so that it can quickly do a lot of arithmetic on the small amount of data coming in from control changes.  This would be for averaging together elements from wavetables.  The FFTs are not completely parallel, but they do benefit from vector arithmetic.  It's also the case that for wavetable synthesis, there is a lot of parallelism in the beginning of the audio processing chain, and non-parallel things come last in the chain; at the point at which the number of voices is irrelevant, and it's generally running effects on the final raw signal.  

Wavetable Synthesis and Parallelism

Cantor's sound engine is the first of my three apps where I am experimenting with parallelism and the complexities of handling audio aliasing (rather than taking a "don't do that" approach).  I never got into instrument making thinking I would get sucked into actual audio synthesis; but it's unavoidable because MIDI has far too many problems with respect to setup and consistency.  The fact that I insist upon fully working microtonality almost makes MIDI untenable.  I don't have the electrical engineering background to compete with the great sound engines in the store; I got into this because the playability of iOS instruments is totally laughable at the moment.    I think anybody that can write a simple iOS app can absolutely run circles around the current crop of piano skeuomorphisms that rule the tablet synthesizer world today; if you judge by the playability of the resulting instrument.  So on this basis, I'm not giving up because of the Moogs and Korgs of the world putting out apps of their own.  :-)  But for now... because MIDI still sucks for this use case, I still need an internal engine; most users will never get past the setup barrier that MIDI imposes.

So, one of the more surprising revelations to me when I started into this was that you can't avoid multi-sampling, even if you aren't making a sample-based instrument, and even if you only have one sound.  The idea that you can just speed up or slow down a digital sample to change its pitch is only half-true.  The same sample played at higher than original speeds is actually not the same wave shape as the original sample.  You not only need to drop half the samples to go up an octave, but you need to cut out frequency content that it can no longer accurately represent.  Because this instrument doesn't have a discrete set of notes, and is inherently fretless, it also needs perfect continuity when bending from the lowest to the highest note.  So the function that defines a wave cycle not only takes phase into account, but the maximum frequency at which you might play it back determines its shape.

And also, because I have a life outside of iOS instrument making, I am always looking to tie-ins to things that I need to know more generally to survive in the job market.  I am currently very interested in the various forms of parallel computing; specifically vector computing.  Ultimately, I am interested in OpenCL, SIMD, and GPU computing.  This matches up well, because with wavetable synthesis, it is possible in theory to render every sample in the sample buffer in parallel for every voice (multiple voices per finger when chorusing is taken into account).

So with Cantor, I used Apple's Accelerate.Framework to make this part of the code parallel.  The results here have been great so far.  I have been watching to see if OpenCL will become public on iOS (no luck yet), but am preparing for that day.  The main question will be whether I will be able to use the GPU to render sound buffers at a reasonable rate (ie: 10ms?).  That would be like generating audio buffers at 100 fps.  The main thing to keep in mind is the control rate.  The control rate is the rate at which we see a change in an audio parameter like a voice turning on, and how long until that change is audible.  If a GPU were locked to the screen refresh rate and that rate was 60fps, then audio latency will be too high.  200fps would be good, but we are only going to output 256 samples at that rate.  The main thing is that we target a control rate, and ensure that the audio is fully rendered to cover the time in between control changes.  It might even be advantageous to render ahead (ie: 1024 samples) and invalidate samples in the case that a parameter changed, which would allow audio to be processed in larger batches most of the time without suffering latency.

Todo:

One thing I am not doing explicitly so far is handling interpolation between samples in these samplebuffers.  I compensate by simply having large buffers for the single-cycle wave (1024 samples). The single cycle wave in the buffer is circular, so linear interpolation or cubic spline between them are plausible.  There is also the issue of the fact that as you go up an octave, the wave is oversampled by 2x because you will skip half the samples.  Because of this, a single sample represented in all the octaves could fit in a space that's about 2x the original sample size (n + n/2 + n/4 + n/8 +...).

And finally, there is the issue of control splines themselves.  It is currently an ad-hoc ramp being applied to change the volume, pitch, timbre for a voice.  It's probably better to think of it as a spline per voice.  The spline could be simple linear or cubic (possibly), or the spline could limit changes to prevent impulsing (ie: more explicit volume ramping).

Post Processing:

The part that generates the voices is easily done in parallel, which is why I am thinking about GPU computation.  In an ideal world, I load up all of the samples into the card, and when fingers move (at roughly the control rate), the kernel is re-invoked with new parameters, and these kernels always complete by returning enough audio data to cover the time until the next possible control change (ie: 256 samples); or possibly returning more data than that (ie: to cover 2 or 3 chunks into the future that might get invalidated by future control changes)  if it doesn't increase the latency of running the kernel.

I am not sure if final steps like distortion and convolution reverb can be worked into this scheme, but it seems plausible.  If that can be done, then the entire audio engine can essentially be run in the GPU, and one of the good side effects of this would be that patches could be written as a bundle of shader code and audio samples, which would allow for third-party contributions or an app to generate the patches - because these won't need to be compiled into the app when it ships.  This is very much how Geo Synthesizer and SampleWiz work together right now, except all we can do is replace samples.  But the main question is one of whether I can get a satisfactory framerate when using the GPU.  I have been told that I won't beat what I am doing any time soon, but I will investigate it just because I have an interest in what is going on with CUDA, OpenCL anyhow.

OSC, MIDI, Control:

And of course, it would be a waste to make such an engine and only embed it into the synth.  It might be best to put an OSC interface on it to allow controller, synth, patch editor to evolve separately.  But if OSC was used to eliminate the complexity of using MIDI (the protocol is the problem ...the pipe is nice), then the question of whether to use a mach port or a TCP port becomes a question.  And also an issue of what kind of setup hassles and latency issues have now become unavoidable.

"Cantor" for iOS

The app formerly known as "AlephOne" was submitted to Apple last night.  It may or may not get approved, as it is using the finger-area sensing that a few other apps are using; though it has been very difficult to get absolute clarity on whether finger-area sensing gotten through dynamic, but public, APIs is admissable.  It was once a very important part of the feel of the app Geo Synthesizer's playability, which suffered mightily when we pulled that feature out to be on the safe side (it wasn't my account that was at stake).  I now believe that it totally is admissable (as I have been informally told), and that most developers are just not trying it out of little more than fear.  I decided to just jump into the app mosh pit.  It is generally good advice to never release "Beta" software into the app store, but that assumes that you have unbounded time to work on apps and that you won't go work on something else in any case.  I never uttered the name "Cantor" and used AlephOne because when I had used the name "Pythagoras" for my last app, that name was taken by the time I tried an allocate a spot for it in the store.

The test builds are here:

http://rfieldin.appspot.com

It is being shipped somewhat early (If it was "done" then there would never be any updates!).  So, where is Cantor?

• Wavetable synthesis engine - but it demands that you span MIDI channels to use it internally.  Using Waveable synthesis allows me to address per-finger filtering and anti-aliasing issues.  There is not a lot of sound variety right now.  But that's why it's a simple $2 app for now.  I have some testers that think that the MIDI support alone is very valuable and will never use the internal engine, but there will always be people that will never try the MIDI.
• Like Geo Synthesizer, it has octave auto, for very fast soloing.  This instrument is just as playable on the phone as it is on the iPad if you play it that way.  
• MIDI with polyphonic bending (note ties, channel cycling, etc).  A lot of synths won't be able to deal with this.  But Kontakt4, Korg Karma, Nord (N2?), ThumbJam, SampleWiz, Arctic, etc all work with it to some degree because they at least do real multi-timbral pitch handling.  ThumbJam, SampleWiz, Arctic specifically recognize the note tie that I added custom to MIDI to work around legato and bend width issues in MIDI.  Generally, you create many adjacent channels (more than 4) with identical patches, and set their bend widths very high, to either 12 or 24 semitones.  It will channel hop around to get all the pitch bending right.  MIDI does not "just work".  It's horribly broken for almost every non-piano instrument scenario when it comes to pitch handling.  This works around that, but it's not pretty.
• Looper - Audio copy doesn't work yet, but I have it half implemented.
• Moveable frets to let you define your own scales, which includes exact pitch locations and the number of frets per octave.
• Common microtonal scale shapes already setup:  Diatonic, Pentatonic, 12ET chromatic, Pseudo Maqam Bayati (ie: 12ET with a quartertone and the quartertone above and below it by a fourth - not quite the unwieldly full 24ET), Pythagorean, Just (Pythagorean with perfect maj/min thirds as a subset of 53ET), 19ET, 31ET, 53ET, or just fretless.  None of these are named, as you simply pick a shape and use the circle of fifths to center it on one of the 12ET base notes.

As Geo Synthesizer was a rewrite of Mugician (that was Pythagoras for a while), Cantor is a rewrite of Geo Synthesizer (and was called AlephOne for a while).  The videos of it in use are here:

http://www.youtube.com/rrr00bb

This app has nothing at all to do with beat making, sequencing, DAWs, or iPad-only workflows.  I'm a microtonal metalhead.  It's a pocketable instrument that's as playable as a guitar or a piano for a lot of uses.  It should be reliable and simple, for you to plug into an effects processor or a computer *like* a guitar or hardware synth.  I tried initially to ship this without any audio engine at all, as a pure MIDI controller.  But the current state of MIDI with respect to continuous pitch instruments is still bad.  I only implemented an internal engine to demonstrate a full exercise of the playability parameters.  For example: this is ThumbJam in Omni mode on the JR Zendrix patch (it's different from most Omni-modes on most synths... it's done correctly with respect to pitch handling):

If you write a MIDI synth and have issues getting Cantor to control it with full expression, then send me an email.

Please keep your money if you will only give it a low rating based on a missing feature.   See the videos and see what it does; and pass on buying it if you think it needs to do a lot more.  Generally, I cannot respond to feature requests via low ratings like "2 stars when X is added".  I wrote this for my own needs, in my spare time, to solve my own problem of needing a microtonal instrument, and don't sell enough to service any kind of hassles beyond that.  But please let me know if you use it for real and something is actually broken, or about things that are actually impossible to do correctly instead.  Specifically: I won't add in uneven tunings like guitar because they don't solve a problem that I have, and I cannot add in a mini multi-track on account of reliability and memory consumption (though I might get export of current loop to clipboard working in an update), and I won't spend a lot of time on the sound engine when MIDI is available.  If you need these things, then what you are saying is that MIDI itself is broken (ie: in all apps), which I can't spend more time on, (ie: other apps can already record MIDI output, and MIDI is recognized by background synths - but interpreting MIDI gives sonically unreliable results, etc.). 

Why "Cantor"?

Geog Cantor was a mathematician who essentially went insane contemplating infinity, specifically, contemplating the continuum.  Though history regards him well, he was regarded as a crank by a lot of his peers:

http://en.wikipedia.org/wiki/Georg_Cantor

Coincidentally, a "Cantor" is kind of like the lead musician in a church's music setup; an interesting meaning when thinking of an instrument designed to deal with intonation issues that weighed heavily on the church organ builders in the time before they all gave up on correct sound (with respect to numerology and aligning music with physical laws) and went to tempered instruments.