Visual Patching

Week of Patching

Paloma Kop launched the Week of Patching. “Week of Patching is a playful invitation to spend a week, in parallel with other people across the internet, engaging with a patching-based practice.”

https://palomakop.tv/week-of-patching/

A description of my setup.

1

Jean is used to to animate groups of simple shapes. Additionally bitmap files can be displayed as background images.

Groups of moving shapes and static images are organized in layers. These layers are stacked. There are different blend modes to merge the content to a final image. This is similar to blend modes used in image manipulation programs. Merging down to a final image is done in real time, because of the moving shapes in some layer.

The shapes belong to two different species: cursors and calves. When cursors are moving they send messages with there current position and velocity. Calves are objects waiting to be touched by a cursor. Then they send messages on hit and release. Technically these messages are Open Sound Control (OSC) packages.

Calculating the movement of the shapes and the overlapping situation is done with simple equations. This is not a 2D physics engine.

Jean has no input channel for visuals stream. Only a control input channel for tweaking some parameters. It is a pure image generator.

Working with blend modes and equations for movements is done via live coding.

2

“analog Not analog” (aNa) consist of video filters and a routing (or transport) system for visual streams. It can freely reconfigured: a) which filter to use, b) how a filter effect should work in detail, c) how the filters are arranged in a network. This is the patching effect of my system.

External sources used as visual data streams: the content of an other window (for example the Jean generator). USB camera movie files image files Most importantly, filters can process the visual output of another filter. This makes feedback loops possible.

Patching the signal chain and tweaking the filters is done via live coding.

3

The user interface of this system is live coding. So we need a programming language that is able modifying programs while they are running. Without restart, without loosing data and nearly without flickering during reprogramming.

It’s Clojure, a modern Lisp. Lisp has had this dynamic aspect from the very beginning. They called it REPL read-evaluate-print loop. After 68 years, the terms from the Lisp world sound a bit unusual. Unfortunately, the consistent notation is also rarely used anymore. But it is still the most flexible and outstanding programming environment.

A

Capturing the content of an other window: Using X11 made it easy to capture the content of an other graphical window continuously. What is helpful for visual patches may be not a plus for security and privacy.

B

Within aNa most video filter run on the GPU, They are backed by double buffering of RGB float values. One special buffer is very large but only for RGB values with reduced precision. It can be used as a delay for visuals coming from upstream. Because the graphics card I use has 11GByte of video RAM there can be several ( and not only 4) filters running simultaneous. The filters of the patching area run with 30 frames per second. This is approximately 30 milliseconds to finish all calculations needed for a single video frame.

*1 background processes

Clojure is responsible for transforming the live coding to a format that is appropriate for Jean and aNa.

SuperCollider, an algorithmic sound synthesizer.

*2 internal control channels

The Pulsar editor is connected to a not visible Clojure REPL. Changes made while live coding are preprocessed and send to Jean or aNa.

*3 sound control

There is a OSC connection between Jean and SuperCollider. This is used to start or manipulate sounds in real-time.

*4 external control

A keyboard connected to the Pulsar editor. A nanoKontrol2 MIDI controller to tweak parameters of the patching area.

*5 live input

A USB camera.