13 january 2015
new oct 2012
a sequence, repeated, becomes a pattern ... the endlessly-repeating sequence of arbitrary color rotating past the sensor-arm is stochastically mapped to musical notes. we hear evolving melody, where there is none.
pareidoliac is a machine/system that translates color to MIDI notes; a color sensor on the end of the arm samples whatever color is under it, four times per revolution. this information is massaged by various controls and processes then sent to the computer that modifies it further and issues the result as MIDI commands, all in real-time.
but that is not what makes this machine interesting.
first, some background is needed on what the machine does.
there is a knob on the machine that sets the key and mode. key is the lowest note, defining the scale used, and the mode is either major or minor, determining the intervals between notes (tone or semitone). a separate knob sets octave. the machine presently outputs notes only within one octave.
color is mapped to note in the following way. "brightness" information is extracted and the color normalized (eg. dark red is the same as bright red). colors deemed "monochrome", eg. the red, blue, green components are nearly equal, is mapped to a rest (silence); for the rest, the color range of zero to millions is reduced to 1 of 12, and this number is mapped into the note. for example for key=C mode=MAJOR, the color wheel of millions of colors, in numerical order -- which has no meaning visually really -- is reduced in range to 1 to 12, such that 1=C, 2=C#, 3=D, ... 11=B, 12=C. this works because the emphasis in this machine is the sequence of colors/notes, not whether red "means" C#, or whatever.
last brightness information is used as MIDI note "velocity"; bright colors play "harder". there is a knob that munges this value with other factors to produce variable note note duration.
a sequence repeated becomes a pattern, ... more or less. with an arbitrary image rotating on the turntable under the sensor, a repeating sequence of notes is output (assuming no human meddling). however the sequence is not often fully "fixed"; subtle stochastic conditions and choices exist between the paper and MIDI note.
| stochastic: a probability distribution or pattern that may be analyzed statistically but may not be predicted precisely. |
there are two major sources of indeterminacy at play; color determination and rotational sample time. the first set of complex indeterminacies involve interplay between the halftone-printed paper medium, the vagaries of ambient light and shadow, the physical lumpiness of the revolving paper, vibration, etc. where people think they see an obvious recognizable color, the sensory experience of color vision is in fact exceedingly complex and subjective. the color sensor is much simpler, tallying photons of various energies into it's three sensors, red, green, blue. where people automatically adjust for changing ambient light over a huge range, ignore/cannot see grain, noise, "unimportant" variation, the sensor is wildly affected by all of that. the second major factor is the sample time, 1/4th of a rotation, and the finite time that the sensor takes to see (integrate) light bouncing off a moving object. 1% repeatability equally means 1% jitter and variation; given that the sensor is approximately a millimeter square it means that given a series of rotations, each note position, 1, 2, 3, 4, varies by approximately 1% in physical position along the circle described by the rotating turntable.
therefore, the sum of this variable light and jitter means that each color sample time (and therefore position) has a probability that approaches, but does not quite achieve, 1.00. (actually, the first note of the sequence does not suffer from sample-timing jitter, being simultaneous with turntable sync. musically, this makes the downbeat more stable.) in practice, large solidly colored areas come arbitrarily close to 1; most colors map to the same note, every time. some however are "on the edge" of determinism; for example, an area of adjacent blue and yellow might look blue, yellow, or green (the sum) in successive samples. this is why this is stochastic, and not random; given a "map" of color::note mappings, the variations are not entirely random; the randomness involved selects adjacent color::note pairs, and not ones chosen entirely at random.
i apologize for the long-winded explanation above; but it is necessary to understand the most important part: why this produces musicality and not noise or glitch. musically-speaking, i'm uninterested in glitch or noise. i like melody, out of fashion or not.
the image is rotating on a heavy steel platter, whose rotational speed is synchronized with a master MIDI tempo clock (BPM) from the computer. psynaesthesiac takes four color samples per platter revolution, at equal intervals, and the first sample is locked to the "top" of the musical measure as specified by the MIDI master computer. you can picture the samples taken around the platter like a clock face; the first at 12 o'clock, second at 3 o'clock, etc.
synchronization between the machine and the MIDI master is fairly tight, about 1% error. while that is good, it is not zero. timing error manifests in two ways: turntable speed variations, which are slow (large fraction of a second), and color-sample-timing jitter, which is fast (millisecond range). given that the color sensor's active spot is very small, approximately 1mm square, the actual location of the sample taken is probablistic; imagining the rotating area under the sensor as a 1mm wide line drawn as a circle, the each sample will jitter from it's ideal position by a millimeter or two or three.
most color image pages i've been playing with were ripped from color halftoned newsprint. halftone itself is a series of colored overlapping dots of various sizes. given the sample scheme and the arbitrariness of images printed in magazines, each sample-spot is of various color and darkness consistency...
i think of this repeating sequence of four color/notes in terms of their stabilities: very often, three of the four are very stable, and one of them is flipping between two or three notes, such as in the adjacent blue::yellow dot example above. assuming that the sensor arm is plunked over rotating arbitary sheet of colored newsprint ripped from a newspaper, let's assume that the sequence of color::note mapping is A B v D, repeating, meaning first note A is stable, second note B is stable, third note v varies (yellow, blue, green) and fourth note D is stable. this may produce the note-rotation sequences of:
| solid RED | solid ORANGE | mixed BLUE/YELLOW | solid VIOLET | |
| A | B | G | D | rotation 1 result |
| A | B | F | D | rotation 2 result |
| A | B | G | D | rotation 3 result |
| A | B | A | D | rotation 4 result |
| A | B | G | D | rotation 5 result |
| A | B | G | D | rotation 6 result |
| A | B | A | D | rotation 7 result |
the accumulation of indeterminacy in the third sample, here due to adjacent yellow/blue dots sampled arbitrarily as yellow, blue, or green, selects randomly between adjacent, not random, notes. the distribution of probability follows the statistical normal curve (eg. a Gaussian); notes further away are possible, eg. E or B, but become increasingly unlikely.
what makes this all work is the human ear's obsession with hearing patterns. there is a musical pattern; (A, B, v, D), (A, B, v, D), (A, B, v, D), ... where v is the/a varying note in a sample sequence. this sort of variation in a repeated sequence is a fundamental component of human musicality. i have found that i almost cannot not hear some of the better ones as melody, even knowing the entirely deterministic process that is unfolding within the machine.
i've done a lot of work with and research into randomness in the last decade. i've come to the conclusion that like the concept of time, random is a bodily sensation, and not a feature of the external world. in mathematics it essentially means that all redundancy has been removed; nothing [is perceived to contain] repeats. there exist rigorous tests for randomness that do in fact mean "no pattern found". a pattern can be thought of the opposite of random; it is entirely or partially predictable, it is highly redundant.
other machines i've built that work with randomness are the Atomic Number Generator, which utters random numbers, and the more theoretically rigorous Gas Tube Number Generator, a true hardware random number generator based upon the 1948 RAND Corporation machine that produced the digits used in the book A Million Random Digits with 100,000 Normal Deviates (as well as my 2009 MFA thesis Rocks and Code.)
the name psynaesthesiac comes from the original idea for this machine, long since abandoned -- sound sequences were to be drawn as color onto blank paper. i made a couple of test lash-ups, and needed something to test with -- and tore off the side off a brightly colored NAPA automotive air filter box. i never made blank discs, and i'm still using that yellow, blue and black test disc. so much for intent.
30 may 2014: a decently musical example, a sheet of advertising torn from an LA Weekly, spindled, driving Native Instruments' REAKTOR set to 120bpm. the preset uses a [feedback] delay:
technically, the pareidoliac device is locked to a MIDI master clock signal (from Reaktor, Ableton Live!, etc). turntable rotation is locked to global tempo, and currently produces a fixed four notes per rotation. there are straightforward controls (knobs) for tonic and octave. there's a knob for... note shape; a combination of velocity and note duration, a sort of all-in-one piano-forte knob, a logarighmic function centered on a eighth note. there's a knob for automatic back and forth sweep of the arm, and one currently unassigned. the turntable portion handles the physical stuff and a Processing program manages the functions and MIDI stuff.
most fundamentally important here is that notes are output in realtime. psyn specifically does not "scan" or read a whole rotation then decide what notes to output. this is a fundamental constraint. this is a prosthetic musical instrument, with abstract software augmentation. augmentation not atemporal function.
the subject of quantization is interesting here. rotation is locked to MIDI tempo; this is unlikely to change. all "physical" computation is done in the turntable by the main arduino (the other one handles pots and lights); note timing eg. exactly when a note is issued, is determined here. Processing simply maps them to tonic, octave, etc.
currently the 4-note measures are very "square", but unquantized. it woudl be easy enough to quantize notes, but it doesn't sound better (worse, actually). turntable rotation is long-term precise, and short-term (per-rotation) well under 1% error (currently about 0.5%, using Proportional Integral Derivitive (PID)). seen on an oscilloscope, or allowing Ableton to record over and over, the jitter is visible and natural (sic) sounding; there is some periodicity to the jitter (almost unavoidable with PID) but it sounds and looks random (eg. well-enough-distributed noise).
the sample method is, the first note is output at the top of the loop/rotation/measure; that first note is zero jitter by definition. subsequent notes are issued at per-rotation calculated times; given that rotation speed isn't perfect (that 0.5%) the other (currently) three notes jiggle a bit. it would actually be easier to quantize them (in Processing) but that's not where i'm heading...
the one-measure-per-rotation makes physical, bodily, sense. it makes intellectual and musical sense. i'm sticking with that.
but the next likely development path is to look for color transitions, and output when color changes (beyond some threshhold). the thinking problem here is that this seems to require "reading ahead" the entire "track" to decide which transitions to output (likely there will be many, and arbitrarily placed). in fact the problem (i use that word the way mathematicians use it) is: how to "specify" (choose, select, whatever) a rhythmic motive? the fucking beat! dah dah dah DAH ...
i think this is where the disused fifth pot comes in. news at 11.
i started by working up a kludge prototype to test out the basic idea; the first pass was a simple intensity to frequency converter chip taped onto an ordinary turntable, driving simple Processing (java) test code. The goal was to see if the scheme might work out from an aesthetic/"feel" point of view. Here's a brief video of the test lashup in operation. (sound here is via the minim library; sawtooth into a low-pass filter; no attempt at musicality yet.)
from experiments and experience there i built the "final" hardware, shown below. the platter is 3/16" steel, about 4lbs/2kg, shimmed for zero runout. speed is controlled with an optical encoder and PID closed-loop software, speed-locked to MIDI tempo data, and accurate to under 2% (final goal is under 0.5%). (the poor motor was designed to run at 1800 rpm, i'm driving it about 60 - 200, it doesn't like it.)
the color sensor is a breakout board product from Sparkfun.com (ADJD-S311-CR999) which is fairly a pain to use. the arm is clear polycarbonate, and there is an RGB LED inserted at the far end that emits the current "color note" sample under the sensor.
here's video of it just barely working in "final" mode... not quite evident here yet is that the stepper-driven arm contains force-feedback such that you work with the software by physically manipulating the arm; a differential pair of optical sensors between the arm and the stepper are the sole control of arm position. Tapping one side will cause incremental motion; holding it in place for a moment will stop it. This, and one lit pushbutton, will be the sole hardware controls.
here is a first-pass "tech demo" sound example, when i'd just got it all working:
and last, though it is likely obsolete by the time you download it, here is the arduino and processing code. this likely isn't enough to reproduce or run it, at the very least the required libraries are missing, though they should all be found on the net.