Behind - In Motion for mobile phone gyro and pure data

I developed a pure data patch that is installed on the mobile phone and the gyro signal sends to the computer for audio processing, then sends to the PA system for performance. By moving the phone, the gyro picks up the three-dimensional acceleration signal, pitch, yaw, and roll, in meters per second (m/s). There are three important elements: two patches, one running on the phone, one on the computer, and a GUI interface on the phone to control the computer remotely. 

The phone side patch

The patch first receives the gyro signal, the measurement sends to the phone GUI for me to monitor. The acceleration signal (m/s) is in small digits. I use mathematical expressions to scale up the digits to translate into midi note numbers.

 

The basic construction of a gyro (pic. 1)

Since it is measuring the acceleration, when I stay still, the digit is around zero and it is scaled up around midi note number 45. When there is movement,  the gyro detects acceleration signals, more vigorous movement causing higher pitch. If the acceleration digit is too large and the pitch is higher than the midi note number, there is also an expression to fold back the digit within the midi note range. After that midi note numbers are packed and sent via wifi network to the computer side.

The expressions scale up the gyro signal to the audible midi range.

The speed limiter controls the frequency of receiving new signals (refresh rate), creating a sense of tempo. It is controlled by the Graphic User Interface (GUI)

Speed limiter patch

There are objects in this patch that control the audio processing on the computer side. GUI controls the parameters in the patch and sends via wifi network.

The phone side GUI

The application that I use for the phone is called MobMuPlat. It can run a pure data patch and build my Graphic User Interface to control the patch. The GUI that I design on the phone provides instant monitoring of the gyro signals and intuitive control of different audio processing parameters. The GUI controls the pure data patch which transfers my input to the computer. I did not put the control of volume on the phone side to prevent accident change of volume, damaging the PA system, or shocking the audiences.

On the first page, from top to bottom, the first object displays the gyro signal. Under the gyro signal display, there are different controls for audio effects. The first slider controls the amount of reverberation, to create a special effect of the sound. The second slider controls the room size of the reverberation, the decay time of the reverberation. The refresh rate controls the frequency of picking the gyro signal, as the tempo of playing new notes. There are two bottoms for resetting the parameters of the next page, harmonic and equalizer, in case of any failure. 

First page layout

On the second page, the multi slider, controls the volume of individual harmonics up to 8. As timbre is mainly determined by the harmonic content of a sound, it can also control the tone colour. The harmonic content also affects how we perceive loudness. With more harmonics and the louder above, the loudness is higher effective. Under the multi slider, the XY slider controls the bandpass filter, X axis for the center frequency; Y axis for the effectiveness, the q factor. It can isolate a certain range of frequencies, to further manipulate the timbre with better agility.

Second-page layout

Network setting

The signal is sent via a wifi network, my original network setting used my second mobile phone as a hotspot, however, the hotspot function is not stable enough. I then used the university wifi. It took me a while to understand the configuration. First, both devices have to log in with the same wifi. Then, check the wifi IP address at the computer side; on the phone side, it is important to set the correct broadcast address to the computer in the MobMuPlat application.

The computer side patch

The computer patch mainly served as the audio processing and output. The top layout of the patch receives the signals from the internet. Then, the number boxes display the midi number sent from the phone, (There are expressions on the phone side to scale up the acceleration signal from gyro) served as the monitor of knowing the signal is well received. The “mtof” objects translate the midi number to frequency. 

The pd harmonic patches generate the harmonic series of the input frequencies, and each volume of individual harmonic can be controlled by the GUI, corresponding to the multi slider; the highest harmonic partial is limited at the 8th. Although partials in the harmonic series, in theory, are unlimited, the loudness decreases as it goes up and has less effective. Limiting to eight is a balance of effectiveness. The combination of the volume generates enough variation timbres in the music.

The patch for generating harmonics up to 8th partials

After the patches generate the harmonics, it comes to the audio processing stage. There are two paths for the audio signal. One is a dry signal, one is the signal that goes through reverberation – the wet signal for spatial effect. The echo of the reverberation is controlled by the phone. Both streams have their own fader to control the level, mixing and balancing the audio wet dry level. I can remotely control the mix via my phone. 

After the reverberation, the combined signal goes through various equalization before feeding to the bandpass filter can isolate a certain frequency, corresponding to the harmonics to alter the timbre. The high pass and low pass filter cut off the extreme frequencies that sound too loud.

The last part of the patch ends with a compressor and a limiter, as a protection for the audio system, to set a ceiling of the loudness.