Convert colors to sound

Generating sound with SuperCollider and ‘analog Not analog’.

Convert colors to OSC messages

Image data of the animation runs through several filters to create the graphic effects. In the last filter the image data is analyzed at two positions. Per position there is a small square area that is sampled. A single point would be too affected by image noise and rounding errors. For each area, the averaged color, saturation and brightness are calculated. These values are then sent via Open Sound Control. If the animation runs with 25 frames per second, 25 OSC packages are per second are send.

Where to scan the image.

Per frame, the positions at which the image is to be scanned can be recalculated. The size of the scanned area can also be redefined. For position and size different inputs can be considered. For example the axes of a gamepad, the sliders of a MIDI controller or these values are calculated from the frame number by formula.

Two areas are sampled at the same time. So you get separate values and can use them for left and right channel of a stereo sound.

Paloma Kop describes in her work FROM MATERIAL SPACE(TIME) TO ELECTRONIC SPACE(TIME): INTERFACES, SIMULATONS, HYBRIDIZATIONS that she taped a light-sensitive photo resistor on the screen. Changes in brightness in the animation then influence the sound. In doing so, she works in analog.

aNa is a digital / discrete system. It is possible to move the simulated photo resistor algorithmically across the screen. It is also possible to use separate simulated photo resistors per audio channel. If the C code is extended, also an 8 channel system could be supplied. The disadvantage of the digital system is that the sound synthesis is indirectly coupled to the discrete frames per second pulsing of the animation.

(ns lv.demo)
(use 'lv.core)

(layout "grid" [img0 img1]
               [ras2])
(image "flower-pages/*.png")

(texpr
  ; use nanoKONTROL2 als input
  (t0 (- 1 (* 2 ks0)))
  (t1 (- 1 (* 2 ks1)))
  (t2 (- 1 (* 2 ks2)))
  (t3 (- 1 (* 2 ks3)))
  ; use a game pad als input
  ; (t0 lx)
  ; (t1 ly)
  ; (t2 rx)
  ; (t3 ry)
)

(rgb :img0 [c0 c1 c2]
  (kaleid (+ 3 (sin (* 0.0067 f))))
  (rot (* 0.0071 f))
  (kaleid (+ 2 (sin (* 0.0061 f))))
  (rgb (mix (mix c0.rgb
                 c1.rgb
                 (usin (* 0.051 f)))
            c2.rgb
            (unoise (* 0.007 f) x)))
)

(rgb :img1 [c0 c1 c2]
  (rot (* -0.005 f))
  (rgb (mix (mix c2.rgb
                 c1.rgb
                 (usin (* 0.023 f)))
            c0.rgb
            (unoise (* 0.031 f) y)))
)

(rgb :ras2 [img0 img1]
  (rgb (* 1 (mix img0.rgb
            img1.rgb
            (unoise (* 0.019 f) r))))
)

(inspect [ras2]
  "color2sound/analyse_dualhead_color.cl"
  (x0 t0) (yl t1)
  (r0 (* 0.2 kp0))
  (x1 t2)  (yr t3)
  (r1 (* 0.2 kp1))
)

Convert OSC messages to sound

This is a script for SuperCollider. The SynthDef AudioHead is instantiated twice. On the left and the right audio channel the same synth is running, but they are supplied by the two “photo resistors” sending slightly different values.

s.waitForBoot({
  ~fadeout = 0;

  SynthDef(\AudioHead,
    { | out = 0,
        hue=0.5,
        saturation=0.5,
        brightness=0.5,
        fadeout = 0.0 |

        var lhue = hue.linexp(0, 1, 0.25, 0.75);
        var snd1 = DynKlank.ar(
                `[[800*lhue, 1071*lhue, 1353*lhue, 1723*lhue],
                  nil,
                  [1, 1, 1, 1]],
                PinkNoise.ar(saturation.linlin(0, 1, 0.001, 0.05)));
        var snd2 = Resonz.ar(
                snd1 * BrownNoise.ar(0.5),
                brightness.linexp(0, 1, 200, 800),
                saturation.linexp(1, 0, 0.15, 0.005));
        Out.ar(out, Mix.new([snd1, snd2])*0.2*brightness*fadeout);
  }, \kr!5).add;

  0.3.wait;
  ~left  = Synth.new(\AudioHead, [ \out, 0 ]);
  ~right = Synth.new(\AudioHead, [ \out, 1 ]);

  // fade out if there are no more osc messages
  t = Task({
    loop {
      ~fadeout = ~fadeout * 0.9;
      ~left.set(\fadeout, ~fadeout);
      ~right.set(\fadeout, ~fadeout);
      (1.0/25.0).wait;
    }
  }).play;

  // receive osc messages
  h = OSCFunc({ |msg, time, addr, recvPort|
    ~fadeout = 1.0;

    // msg.postln;
    // osc message received from aNa
    // [
    // /f_hsb4,             0 osc path
    // 2                    1 channels == 2 -> data from a dual head sampler
    // channel 0
    // 0.289,               2 x0 channel 0 x coordinate
    // 0.102,               3 y0 channel 0 y coordinate
    // 0.051,               4 r0 channel 0 size
    // 0.3435945212841,     5 channel 0 hue
    // 0.075323805212975,   6 channel 0 saturation
    // 0.497931599617,      7 channel 0 brightness
    // channel 0
    // 0.523,               8 x1 channel 1 x coordinate
    // -0.101,              9 y1 channel 1 y coordinate
    // 0.041,              10 r1 channel 1 size
    // 0.14372645318508,   11 channel 1 hue
    // 0.096923530101776,  12 channel 1 saturation
    // 0.902570962905886   13 channel 1 brightness
    // ...
    // two more channels
    // ]

    ~left.set(\fadeout, ~fadeout);
    ~left.set(\hue, msg[5]);
    ~left.set(\saturation, msg[6]);
    ~left.set(\brightness, msg[7]);

    ~right.set(\fadeout, ~fadeout);
    ~right.set(\hue, msg[11]);
    ~right.set(\saturation, msg[12]);
    ~right.set(\brightness, msg[13]);
  }, '/f_hsb4c');

});