feedback patterns

Patterns from the loop / visual feedback

Inspiration

70s: A TV monitor displays the video stream of a VHS camera. What happens when you point the camera at this TV monitor?

How can such effects be reproduced digitally?

How can we create similar (or altenate) effects with algorithms?

Role models

analog TV equipment
- a camera that takes its own picture.
analog modular synthesizers for sound
- voltage controlled oscillators
- no-input mixer

Terms

Loop / Feedback

Apply a process repeatedly to its own results.

Signal / Data flow

Image data that runs through a filter chain.

Internal / External

Internal: Manipulate image data with algorithms directly on the graphics card.
External: Monitor - camera - loops.
External: Additionally there are mirrors, glasses, foils in the optical axis.

“Studio”

Two screens, the camera is aligned to the left-hand screen. You see various software tools.

Monitor - camera - loop and a simple effect filter.

Monitor - mirror - camera - effect filter - loop. A nanoKONTROL2 MIDI controller to change the parameters of the effect filter.

The record player as a turntable, camera from above, different light sources, sensors.

A small “Infinity Mirror(s) Room”. Inspired by the work of Yayoi Kusama.

The Software i work with

“analog Not analog” aNa

Most of the examples in this workshop are created with aNa.

aNa is a video synthesizer and manipulator.
create shapes and color gradients algorithmically.
for prototyping the intended visual data flow.
can be controlled via Open Sound Control.
is capable of live coding.
is complex to install, steep learning curve required.

“Glitch camera / Loop camera / Blend camera”

Made for experimenting on the go.

Three websites that manipulate the camera stream.
Also runs on smartphones.
Easy to use.

Components of the workflow

A physical setting for the photographic shot
plus a chain of effect filters
plus the visual feedback loop
optional: algorithms for geometric distortion, perlin noise, edge detection, …
optional: optical glasses, mirrors, …

Data flow

Direct forward data flow with effect filter

Rotate colors / using the HSV color model.
Incorrect sampling of the color channels from the input stream.
Pixelation.
Distorted geometry.
Artificial mirror axes, emphasize edges, …

A simplistic and very colorful example:

(ns lv.demo)
(use 'lv.core)

(layout "vertical" [mov0 ras1 ras4 ras6 ras8])
(render)

(movie "4movies/*.mp4")

; movie as input stream
(rgb :mov0 [c0, c1, c2 c3]
  (sy 1.5)
  (rgb c0.rgb)
)

; color shift
(hsv :ras1 [mov0]
  (h (+ ks0 mov0.h))
  (s (* 2 ks1 mov0.s))
  (v (* 2 ks2 mov0.v))
)

; trichrom
(rgb :ras4 [mov0]
  (r mov0.r)
  (ty (* 0.3 ks4))
  (g mov0.g)
  (ty (* 0.3 ks4))
  (b mov0.b)
)

; pixelate
(rgb :ras6 [mov0]
  (pixelate-x (* ks6 1024))
  (rgb mov0.rgb)
)

; distored geometry
(rgb :ras8 [ras4]
  (tx (snoise x y))
  (ty (snoise y x ))
  (rgb ras4.rgb)
)

; ks0    0.44531
; ks1    0.57812
; ks2    0.66406
; ks4    0.4375
; ks5    0.58594
; ks6    0.4375

Mixer

Mixer for blending two image signals.

Similar to double exposure on an analog film.

Similar to blend modes between layers in a photo editing program.

(ns lv.demo)
(use 'lv.core)

(layout "vertical" [mov0 mov1 ras2 ras3])
(render)

(movie "4movies/*.mp4")

; movie as input stream
(rgb :mov0 [c0, c1, c2 c3]
  (sy 1.5)
  (rgb c0.rgb)
)

; movie as input stream
(rgb :mov1 [c0, c1, c2 c3]
  (sy 1.5)
  (rgb c3.rgb)
)

; mixing by multiplication
(rgb :ras2 [mov0 mov1]
  ; (rgb (mix mov0.rgb mov1.rgb ks0))
  (rgb (* mov0.rgb mov1.rgb))
)

; distorted geometry
(rgb :ras3 [mov0 mov1]
  (tx (* ks1 -0.3 mov0.g))
  (ty (* ks2 0.3 mov0.r))
  (rgb mov1.rgb)
)

Keys as “switches”

This is directly inspired by analog TV.

luma key: Where signal 1 is darker than a threshold value, signal 2 is passed on.
chroma key: Where signal 1 has a certain color, signal 2 is passed on. (green screen)

Here is a “luma key” example. The brightness per pixel controls the selection of which signal is send to the output.

(ns lv.demo)
(use 'lv.core)

; (layout "vertical" [mov0 mov1 ras2 ras3])
(layout "vertical" [mov0 mov1 ras2 ])
(render)

(movie "4movies/*.mp4")

; movie as input stream
(rgb :mov0 [c0, c1, c2 c3]
  (sy 1.5)
  (rgb c2.rgb)
)

; movie as input stream
(rgb :mov1 [c0, c1, c2 c3]
  (mirror)
  (kaleid 5)
  (sy 1.5)
  (rgb c3.rgb)
)

; luma key
(hsv :ras2 [mov0 mov1]
  (hsv (pos? (- mov0.v (* 0.3 ks0)) mov0.hsv mov1.hsv))
)

; ks0    0.26562

Internal feedback loop

The feedback takes place directly in the memory of the graphics card.

There is at least:

One input channel to stream image data into the system. (initiator or seed)
An additional image memory.
A feedback of the image signal.
A mixer or key.

There can be optional:

One or more effect filters.

Initiator / Seed

A data stream must somehow enter the system and be fed into the feedback loop.

Initiator in digital systems

Get the stream from the USB camera into the system,
or generate a simple pattern using an algorithm.

Initiator in analog systems

Get the stream of the VHS camera into the system.
Use the thermal noise of the transistors of an analog audio mixer. no-input-mixer

Algorithm as initiator

Example of a digital feedback loop. A simple striped pattern is transformed into a complex pattern by the feedback loop. Inside the feedback loop, the coordinates of the pixels are rotated, scaled and shifted. (Affine transformation)

Interaction of the components:

ras0 Generate the simple pattern.
ras1
- memory for feedback
- geometric transformation
- and mixer
ras3 a downstream effect filter

(ns lv.demo)
(use 'lv.core)

(layout "grid" [ras0 ras1] [ras2 ras3])
(render)

(hsv :ras0 []
  (v (round (usin (+ (* 0.11 f)
                     (* ks0 33 x)))))
)

(hsv :ras1 [ras0 ras1]
  (rot (* ks1 0.05))
  (sy (+ 0.95 (* ks2 0.1)))
  (ty (+ -0.05 (* ks3 0.1)))
  (v (mix ras0.v ras1.v 0.999))
)

(hsv :ras3 [ras1]
  (v (if? (> ras1.v (+ 0.45 (* 0.1 ks7))) 1 0))
)
; ks0    0.6875
; ks1    0.67188
; ks2    0.47656
; ks3    0.36719
; ks7    0.59375

Example with colorful stripes.

Camera stream or video as initiator

Caution: these patterns cannot be reliably reproduced. They are rather random finds.

Red blue internal feedback loop.

A camera stream is feeding an internal feedback loop.

Interaction of the components:

mov0 Live stream of the camera.
ras1 Memory for feedback + geometric transformation.
ras3 Downstream gamma correction.

(ns lv.demo)
(use 'lv.core)

(layout "grid" [mov0 ras1] [ras2 ras3])
(render)

(movie "4movies/*.mp4")
; movie as input stream
(rgb :mov0 [c0 c1 c2 c3]
 (sy 1.5)
 (rgb c2.rgb)
)

(rgb :ras1 [mov0 ras1]
  (rot (* ks1 0.3 (snoise x y)))
  (sy (+ 0.95 (* ks2 0.1)))
  (ty (+ -0.05 (* ks3 0.1)))
  (rgb (mix mov0.rgb ras1.rgb 0.99))
)

(rgb :ras3 [ras1] (rgb (pow ras1.rgb (vec3 ks7 ks7 ks7))))

; ks1    0.46094
; ks2    0.92969
; ks3    0.64844
; ks7    0.32031

Droste effect

The possibilities are significantly increased by the fact that there is now one (1) additional memory in the system. Here, for example, is the Droste effect.

Place a copy of the image in the same image.
Use this strategy repeatedly.

This is one of the effects made with the loop camera.

https://metagrowing.org/loop?filter=F022

Mehrfach-Verkleinerungs-Kopier-Maschine

Die Säulen der fraktalen Geometrie: Rückkopplung und Iteration

Die Mehrfach-Verkleinerungs-Kopier-Maschine

in: Bausteine des Chaos von Peitgen, Jürgens, Saupe; circa 1992

The downsizing and duplication process provides the attractor.
The original geometry converges towards this attractor.
The colors remain approximately the same.

https://metagrowing.org/loop?cam=user&filter=F010

IFS/Flame with camera or video input

Combination of different ideas

Map points of the plane repeatedly onto the plane. Execute a ℝ² -> ℝ² function repeatedly. Iteration
Original iterated function systems with only affine mappings as a starting point. Barnsley Farn, Sierpinski triangle ….
Allow more geometric transformations. As in Fractal Flames by Scott Davids allow trigonometry, exponential functions, polynomials …
Take the colors from the real world.

Technically: Switch from the step-by-step serial calculation of the original iterated function systems to a highly parallel implementation. 1 million points are transformed by only a few steps per frame.

Then the ℝ² -> ℝ² mapping fits exactly with OpenGL arithmetic: vectors as complex numbers.

The geometry is determined by the function that is executed repeatedly.
The colors remain approximately the same.

(ns lv.demo)
(use 'lv.core)

(layout "grid" [img0 igen1]
               [ras2 ras3])

(image "flower-sampler/*.png")

(rgb :img0 [c0 c1 c2 c3]
  (rgb (mix c3.rgb c2.rgb (usin (* 0.031 f))))
)

(expr :igen1 [ras3]
  "r2/r2image.cl"
  (p0 (* ks0 3))
  (p1 (* (usin (* 0.007 f)) 3))
  (p2 (* (usin (* 0.011 f)) 3))
  (p3 (* (usin (* 0.013 f)) 3))
  (p4 (* (usin (* 0.023 f)) 3))
  (p7 (* ks7 3))
)

(rgb :ras2 [img0 igen1 ras2]
  (rgb (mix (* 3 img0.rgb igen1.rgb) ras2.rgb 0.95))
)

(rgb :ras3 [img0 igen1]
  (rgb (+ img0.rgb igen1.rgb))
)
(render)

float2 sinusoidal(float2 v, float amount) {
    return amount * sin(v);
}

float2 spherical(float2 v, float amount) {
    float r = length(v);
    return (amount * 1.0f / (r * r)) * v;
}

float2 expotential(float2 v, float amount) {
    return (amount * exp(v.x-1)) * (float2)(cos(M_PI_F*v.y), sin(M_PI_F*v.y));
}

#define a_index(ix,iy) (SX*iy + ix)
#define g_index(ix,iy) (2*SX*iy + 2 * ix)
__kernel void set_zero(
__global int* countersR,
__global int* countersG,
__global int* countersB
)
{
    const int ix = get_global_id(0);
    const int iy = get_global_id(1);
    const int SX = get_global_size(0);
    const int i_here = a_index(ix, iy);

    countersR[i_here] = 0;
    countersG[i_here] = 0;
    countersB[i_here] = 0;
}

__kernel void map_r2(
__global int* countersR,
__global int* countersG,
__global int* countersB,
__read_only image2d_t image_in,
__global float* gaussian,
__global float* uniform,
const float p0,
const float p1,
const float p2,
const float p3,
const float p4,
const float p5,
const float p6,
const float p7,
const float dt0,
const float dt1,
const float dt2,
const float dt3,
const float dt4,
const float dt5,
const float dt6,
const float dt7
)
{
    const int ix = get_global_id(0);
    const int iy = get_global_id(1);
    const int SX = get_global_size(0);
    const int SY = get_global_size(1);
    const int i_here = a_index(ix, iy);

    sampler_t smp = CLK_ADDRESS_CLAMP_TO_EDGE | CLK_FILTER_NEAREST;

    const float px = (float)ix / (float)SX;
    const float py = (float)iy / (float)SY;
    const float w0 = M_PI_F;
    float2 pos = (float2)((2.0f*px-1) * w0,
                          (2.0f*py-1) * w0);

    const float ga0 = p7 * 0.004;
    const int g_here = g_index(ix, iy);
    pos += ga0 * (float2)(gaussian[g_here], gaussian[g_here+1]);

    for(uint l=0; l<7; ++l) {
        pos = sinusoidal(pos, p0);
        pos = spherical(pos, p1);
        pos = expotential(pos, p2);
    }

    float2 out_pos = 0.5f + pos / ( w0 * 2);
    int ox_pos = SX*out_pos.x;
    int oy_pos = SY*out_pos.y;
    if(ox_pos >= 0 && ox_pos < SX
       && oy_pos >= 0 && oy_pos < SY) {
        const float4 color_in = read_imagef(image_in, smp, (int2)(ix, iy));
        const int red =   (int)(256*color_in.x);
        const int green = (int)(256*color_in.y);
        const int blue =  (int)(256*color_in.z);
        const int o_here = a_index(ox_pos, oy_pos);
        atomic_add(&countersR[o_here], red);
        atomic_add(&countersG[o_here], green);
        atomic_add(&countersB[o_here], blue);
    }
}

__kernel void write_image(
__global int* countersR,
__global int* countersG,
__global int* countersB,
__write_only image2d_t image2
)
{
    const int ix = get_global_id(0);
    const int iy = get_global_id(1);
    const int SX = get_global_size(0);
    const int i_here = a_index(ix,iy);

    float red =   countersR[i_here] > 0 ? tanh((float)(countersR[i_here]/256)) : 0;
    float green = countersG[i_here] > 0 ? tanh((float)(countersG[i_here]/256)) : 0;
    float blue =  countersB[i_here] > 0 ? tanh((float)(countersB[i_here]/256)) : 0;
    write_imagef(image2, (int2)(ix, iy), (float4)(red, green, blue, 1));
}

Conformal Map

“Preserves angles” during mapping of the plane. Note: This image is created without a feedback loop.

https://metagrowing.org/glic?filter=F028

External feedback loops

Why i use some additional external feedback loop?

Can lead to optical effects that are very complex to generate algorithmically.
A scene of real objects can be changed very easily.
Creates many visual details with little effort.
Purely algorithmic looks too “clean”.

Tools

Objects and glasses in the optical axis as an additional filter.
Light sources as initiators / seeds
- LEDs in the optical axis.
- Illumination of the scene from outside. Because lighting exclusively through the computer monitor does not work.
The algorithm implementing the effect filter serves as an initiator / seed. The monitor illuminates the scene.

Monitor mirror camera loops

Examples created with an external feedback loop

Making of “light, glass and mirrors”

There are several ways of capturing the image.

What the camera - which is integrated in the loop - sees.
What the effect filter sees.
What an additional camera sees.

External versus internal feedback

This is simple when using an external analog loop:

Optical refraction of light
- Prisms, glasses …
Drawing with light
Liquids

This is simple with internal digital filters:

Change the color angle
Pixel Push / Pixel Sort algorithms
Audio-reactive
Delay loops
Motion detection

“Material” from the real world

Explore the material to find visual style features.

Materials:

Glasses
- Exploit optical refraction in glasses
- Colored glasses
Light sources
- Lightning direct inside the optical axis
- Indirect illumination
Liquids
- Oil paint on glass support
- 2 different colored liquids that do not mix.
- wave pattern in the liquid
mechanical movements
…

Stylistic characteristics

Style features analog and digital

Overflow in the color channels

With analog and digital feedback, the color channels quickly overflow.

Using a RGB system

1 channel overflow: red or green or blue dominated.
2 channels overflow: yellow or cyan or magenta dominated.
overflow in all 3 channels: white

Glitch Camera with 8 Bit unsigned integers per RGB color component.

Not always reproducible

Digital systems that contain feedback are not significantly easier to reproduce than analog systems with feedback.

The initial state in the feedback buffers influences the result.

In analog systems, electromagnetic coupling, temperature of the electronics, parasitic capacities, … also play a role.

WebGL

Style features using analog equipment

Camera Monitor Rotation

... piece was created using camera/LCD screen feedback ...
... rotated ...
.. camera was deliberately out of focus ...

Paloma Kop

FROM MATERIAL SPACE(TIME) TO ELECTRONIC SPACE(TIME): INTERFACES, SIMULATONS, HYBRIDIZATIONS

Page 20 https://palomakop.tv/mfa/

Soft shapes

Analog equipment used in feedback loops is suitable for creating soft and sometimes organic patterns.

I processed the footage using Signal Culture's analog video
equipment, including a camera feedback loop, and the Jones
Colorizer, which was created by Dave Jones.

Paloma Kop, “Weird Things”, Analog equipment

https://palomakop.tv/art/weird-things/

https://signalculture.org/

Color deviation

When looking at it, the first impression is that it is a bright white light. But there are small blue traces. It is not clear to me what causes the blue color deviation.

There is the answer from Paloma Kop: “i think this is due to the lcd monitor i was using for feedback, combined with the specific camera, they have a bias towards cooler color balance, this is amplified in saturation by the feedback loop to produce the blue color”

Paloma Kop, Analog equipment, projection on a spherical weather balloon

The technical basics of analog TV become visible.

A TV signal consists of frames. Frames consist of lines. In this respect, TV image processing is not completely analog. The fact that there are individual scan-lines is a discrete aspect. This can be exploited by deflecting the electron beam of the TV monitor with magnets while it is drawing the scanlines.

Digital style features

“Wrong” sampling the texture unit

“Mis-sampling” in the texture unit as a stylistic style. Here Pixel Push made without an additional buffer. Pixel Sort is possible with additional buffers. Glic-Camera WebGL

Rounding error with 8 bits per RGB color channel.

Bluish color errors can be seen in the upper left quadrant. The buffer for the internal feedback uses 8-bit integers. Loop-Camera WebGL with 8 Bit per color channel.

Geometric transformations

Many geometric transformations of the image plane are possible: rotation, translation, affine mapping, conformal maps, function with complex numbers …. aNa as a kaleidoscope with 32-bit floats per color channel.

Edges

The digital workflow produces sharp edges and (unwanted) aliasing.

Made digitally, it often looks very “clean”. This overly clean look can be avoided: Use material that was recorded in the real world as input. Or by projecting onto a structured surface. aNa with 8 Bit per color channel.

Quellen

Paloma Kop “Weird Things”

https://palomakop.tv/art/weird-things/

https://videos.scanlines.xyz/w/a7Act8yFoP9g38jq1x6Hq9

Paloma Kop “EDGE OF LIGHT”

https://palomakop.tv/art/edge-of-light/

https://videos.scanlines.xyz/w/uJTa1bchqGCByWpQqDE7SS

Limits

Technical limits

Color depth
- Overflow of the color channels
- 8 bit unsigned integer / 32 bit float per color channel
Flickering
- Monitor tearing / vertical sync / V-RAM write while the image is being transferred to the monitor.
- Inverting filters can lead to strong flickering.
Is the algorithm parallelizable?
- For porting a CPU parallelizable to a general purpose GPU: Do the algorithms fit into the “single instruction multiple data” (SIMD) scheme? Time required for the calculation
- 60 Hz -> 16 milliseconds per frame
- 30 Hz -> 33 milliseconds per frame
How often does the camera deliver a new image, with what color depth / resolution?

Unwanted effects

LEDs are operated in pulsed mode. -> Pulse wave modulation versus regulated direct current.
Camera with automatic exposure or auto focus that can not disabled. -> Hint: Use an external light source to avoid flickering.

Software

DIY Software

analog Not analog / Clojure + openFrameworks

About analog Not analog https://metagrowing.org/ana/

aNa cookbook https://metagrowing.org/cookbook/

Glitch Camera / Smartphone

Starting the Glitch Camera:

https://metagrowing.org/glic/

Howto:

https://gitlab.com/metagrowing/glic

Loop Camera / Smartphone

Starting the Loop Camera:

https://metagrowing.org/loop/