I would like to introduce myself by sketching a line of events through my life. I’m well aware of the multitude of other lines that could have been chosen or re-constructed, connecting other important moments in different ways, highlighting other kinds of meaning. But, here we go.
I discovered the wonders of electricity at an early age. Disassembling a lamp in my sister’s doll house and fixing a battery, a small switch and a bicycle lamp to a piece of wood, I created a signaling flashlight. My older cousin happened to stay at our house during the holiday and he provided the necessary technical explanation. At dusk he took the contraption to the other side of the block and I balancing myself on a chair so I could reach the attic window and see the flashes he produced in the distance. We both had a copy of the alphabet in Morse code and I was fascinated. A simple binary code of long and short flashes could be used to carry a true message and meaning. While my nephew‘s interest was probably more with the girl that lived on the other side of the block, for me the process was magic.
A few years later my knowledge of electronics had grown a bit and was applied to another project. Again the switch was the central part, in this case a row of 12 modified clothing pegs that served as a musical keyboard. With the help of a Philips kit my mini organ produced simple tones that were amplified through a disassembled radio with its valves glowing a mysterious orange. I brought that desk-size contraption to primary school to give a presentation on electronic music and ended my story with a weird sounding reverb-like sound that I could evoke from the device by reaching inside and turning the amplifier off while the organ was still producing signal. This time my reach was less accurate and I got a 220 volt shock which I bravely tried to hide, acting as if nothing had happened. But the seed was planted that day: maybe I could build and contribute something worthy to society, even though my soccer-skills were lousy.
Skipping many years and projects, we arrive at Duel, an installation for real time musical interaction that I created with a friend. Here the computer was to take the role of the conductor. Two percussionists stand back to back, like in a duel, each playing on a woodblock a repeating rhythm they could freely choose. They were wearing yellow ear protectors and didn’t hear each other (as partners in conflict do). The computer listens to both and, after discovering the repeated patterns, starts to play along with each player on a tiny earplug inside the ear-protector. The computer then steers the percussionists through a series of phases (aligning tempo, then dropping common beats, then adding notes that one player performs to the other) and when each stage was achieved, it signaled the performers to take a step away from each other. In the final stage, le moment supreme, the duelists turn to shoot, but discover they are playing the same rhythm, the conflict turns out to be non-existent. This was all pre-MIDI, the first version we played on mechanical switches (yes again), and the conductor program ran on a (yes again) desk size PDP-11. Next versions were performed by really good musicians at the Computer Music Conference in Cologne. The setup allowed complex rhythmic material to slide over each other, like the in pieces of Steve Reich, and interesting musical hocket structures to emerge.
In this project the computer had to detect repeating patterns, track the tempo, and judge when a performer was following the subtle nudging, for short, the computer had to be able to listen. It turned out that constructing the listening process artificially was very hard. It made us aware of the complexity of these cognitive tasks that we can perform ourselves so easily. Everyone is able to tap a beat to a simple rhythm, yet to have a computer do it is really hard.
This brought us to the scientific study of music cognition, mainly focusing on aspects of musical time (rhythm, beat, tempo, timing, meter) and the interplay in music of two timescales: the discrete and symbolic as notated in the score and the flowing and continuous as in individual expressive timing and rubato. We gained funding for a large ‘Music, Mind, Machine’ project and spend many years experimenting and detailing computationally how the process of performing and listening works. These tasks are performed by the mind, but they have to make use of the brain architecture as implementation.
As traces of this brain activity can be captured with electrodes and sensitive amplifiers I slowly moved from the more abstract study of music cognition to the neuroscience of music. In a special cabin we placed a cap on the skull of our subjects and a (yes again) desk size piece of equipment amplified the so called EEG signals and stored them. Older theories of music perception and beat induction often start from a perceived event, a note, but exciting rhythms often contain a syncopation, an expected but missing note. Its absence almost becomes an event, a ‘loud ‘rest. As it turned out, in the brain signals we could clearly detect that the brain is responding to such a missing event. To investigate this more thoroughly we had our subjects listen to 5 short rhythms repeatedly (hundreds of times). To further the missing note research, we were curious what would happen if we left out all notes. What would the brain signal be when there was silence but the subject had to imagine the rhythm? This task we can do very well (think of the annoying melody-stuck-in-your-head phenomenon). After months of developing the proper signal processing I was able to detect which rhythm the subject was imagining, it was far from perfect but it worked. This was a turning moment when I suddenly realized that pure brain activity can be used to control something and would allow paralyzed users to communicate with the outside world.
Slowly these so-called Brain Computer Interfaces became a topic of research worldwide. There are many kinds, e.g. based on detecting imagined movement or selective attention. I applied for a large subsidy and subsequently lead the project called ‘BrainGain’ which allowed many research institutes and companies to elaborate techniques for BCI. At that moment still all known methods were slow and unreliable. This caused a lot of frustration, trying to design a device for patients, as every component needed to be perfect, otherwise the whole processing chain did not work at all. There just was not enough signal in the noise to allow a reliable, fast and robust detection. I turned to telecom technology, where cellphones are able to reliably pick-up the specific signal intended for that one phone even in the context of weak signals and interferences. The carrier of the message in these cases is itself noise-like, apart from that it is precisely known to the receiver. It’s broadband character allows for a robust transmission, with interference having little effect. And luckily this approach worked also for BCI where the brain becomes the transmission channel. Noise tag BCI as we know it, with its very fast flicker patterns on the buttons or keys, became a fact.
From that first demonstration in a lab prototype it was still a long route towards founding a spin-off: validating performance, applying for patents, attracting grants and investors, getting a team together, work on better electrodes, and make the system run on smaller machines. At the moment we are in the middle of this, testing with patients at home, finding a business model. As CSO I’m employed halftime by the university and halftime by MindAffect. I feel very proud of the team and what already has been established.
From a flashing light seen through my attic window to transmit a message in Morse code I am now engaged in brain control that works with flashing lights and allows communication of messages. And so in elaboration story line turned into a circle.