RPi DJ Loop

12/06/2017
A Project By Hao Wu (hw622) and Boyi Xu (bx64)

Demonstration Video

Introduction

A Raspberry Pi DJ loop was built as our 5725 final project. The DJ loop serves the similar functions as the DJ loop or launch pad in the market. Users can pre-install any sound resources into the Raspberry Pi and play with the sounds. Users can record any lengths of the sound and replay the recorded one in a loop. While looping the sound, users can still play other sound but not record it, or choose to add them into the existing loop. If users are not satisfied with any one of the recorded part, they can delete it without affecting the whole loop. To enhance the visual effects, a 32*16 RGB LED matrix is used as an audio spectrum.

Project Objective:

1. Main functions including playing, recording, looping, and deleting any parts of the sound resources.

2. Triggering the sounds with IR sensors and IR LEDs. The functions listed above can be triggered with pushbuttons along the side of RPi display screen.

3. A 32*16 RGB LED matrix displaying the audio spectrum according to the sound out.

Design

Our RPi DJ Loop is composited by three main parts: ① Raspberry Pi 3 -- the central part which receives player’s inputs, send sounds output and signals to control RGB matrix display, ② IR sensors -- the input part which contains IR LEDs and IR Photologic CMOS, ③ RGB matrix -- which is composed by 16×32 RGB LEDs and can display the sound wave magnitude to give some visual effects. We will describe what we design on each part.

1.Raspberry Pi 3

As strong as a personal computer, Raspberry Pi 3 provides convenience in designing billions of embedded systems. We designed our Loop system on RPi using Pure Data -- which is a very useful and easy-learning programming language for dealing with musics and sounds. The functions of Pure Data is illustrated in the next section. For a DJ Loop, there are three main functions, the first very basic one is to let creators play their music pitches to create fantastic music rhythms. The second one is to record the music pitches and replay them. The third one is that players can add new music pitches onto the former ones to make diversified music.

For the music play function, our idea is to use the sound samples ( which can be obtained from Internet or created by players themselves) saved in RPi’s memory part. For the record and replay part, we let the system to save the sound waves into memory according to player’s operation based on time, and let the system replay the sound waves for the same time period. For the diversified music, we allow players to save 20 sound samples to play, and build the separate record and replay systems for each kind of samples. All the functions above are designed on Pure Data’s extension version -- pd-extended on Raspberry Pi, which can be installed on RPi following this guidance (https://puredata.info/downloads/pd-extended-0-43-3-on-raspberry-pi-raspbian-wheezy-armhf). More details about the loop system design will be explained in Pure Data part.

For the player’s convenience, we also designed a GUI using pygame and displayed on PiTFT screen. Players can choose which kind of sound sources they are going to play, and press buttons near on PiTFT to elect the time for recording and replaying.

2. Pure Data

Most of the efforts have been dedicated into programming with Pure Data (PD). It is a graphical programming language, which is programmed by drawing function blocks and link them together following your logic. For our loop system, the real-time ability is emphasized, so the [table] object from PD is the most important object in our PD patches ( a PD patch is where the graphical programs live) because it put data into memories of RPi, which lifts up the processing speed for adding multi-kind music. For how to program in PD, please reference the tutorial in PD’s official forum and the video notes from YouTube.

2.1 Music play:

The first function for programming is getting the sound waves from sample files into PD tables. Using [soundfiler] object combining with message “read -resize your_sound_wave_file.wav your_table_name”, PD can read the sound waves into a memory table (very easy and fast !). But, just save it in a table cannot play it out, we need [tabread4~] object combining with [line~] object to read the table in a speed determined by the original sound waves. The [tabread4~] object is used to read the content in a table, in our case is the wave of a sound, we need to limit the speed of system to read the the wave out because sound waves are usually sampled in 44100 Hz, which is much lower than Raspberry Pi’s processing speed. The [line~] object creates a ramp that different time point responds to a different digital value. So, to play a normal sound wave, we use [line~] to receive the time - value pair from the table, and send it to [tabread4~] object that it will read the value out according to the time sent to it. This is the most important structure in our system and we really appreciate “radiostarkiller” in Pure Data’s forum who inspires us.

Now, we need to record the sound waves combination from our player’s creation. We use [tabwrite~] object and connect it to the output from [tabread4~], so that the new table can write sound waves from sample tables, one value one time according to the original sounds. We set a timer for the record process, and send the end time of recording to another timer, which use for replaying this new recording table. When the replay timer hits the recording time, restarting the play of the recording table, then the music pitches can loop. The [tabplay~] object is used here to play the recording table. For diverse music pitches, we use the same structure to create several sub-patches for each kind of sound sample, like drums and electronic effects. Connecting them to the [dac~] object, which can transfer digital signals to audio signals, the music can be heard when you insert your headphones or speakers into the 3.5mm jack on RPi.

2.2 Controls:

2.2.1 GPIO-Python-PD:

In PD, it can receive keyboard input from standard input keyboard. However, according to our design, we need GPIOs to wait for a player’s input, and PD cannot provide us with the GPIO object. So, we used Python to communicate with PD, using the [netreceive] object in PD and os.system(‘echo …’) in Python. Since our design is based on Linux system, we can use terminal sending commands to an opened PD patches over socket. The [netreceive] is used for receiving messages from a socket port, which can be defined like [netreceive 3000]. Then, if we type “echo ‘1’ | pdsend 3000” in the terminal, we can receive the ‘1’ in PD patch. So, we design the GPIO callback functions in Python file, let it waiting for players’ input. When a player makes an input in a GPIO, the corresponding callback function will send ‘pdsend’ command to terminal and the PD patches will receive the message to do something, like playing a single sound out, recording and replaying.

2.2.2 Selection of sound categories:

In our pygame GUI, players can choose the sound categories. In Python programs, we still use the ‘pdsend’ to give control message to PD. In PD, we use [spigot] object to help us shield the GPIOs signals to other categories. The [spigot] object is much like a gate, and when the right side receives a high value, the left side will allow control signals pass through. By sending logic ‘1’ to related [spigot] when related screen region is touched, the related sample category can be chosen.

2.2.3 PD-Arduino:

The RGB matrix is connected and controlled by Arduino board, but the patterns displayed on the matrix should be controlled by the sound output (the amplitude of the sound wave). Since our Arduino board is connected to our Raspberry Pi, we decide to use serial communication between PD and Arduino to transmit the amplitude information. In PD, we use the [rfft~] object to apply Fast Fourier Transform on the sound output, which can help us obtain the amplitude by square rooting the summation of its left output’s square (the real part) and its right output’s square (the imaginary part). Then, we use an object called [comport] in PD, by which we can connect PD to the pointed serial port on Raspberry Pi. Then, simply connecting the FFT output to the serial port connecting Arduino, we can now send amplitude data to Arduino. And in our Arduino program, we set the height of our displayed rectangle can be set by the data read from the serial port, so our RGB matrix can now display the sound amplitude.

3. IR Sensors

To play a sound, we need an IR LED to trigger the falling edge of the IR sensors. We chose GPIO 26, 19, 13, 6, 5 as the input pins of 4 IR sensors and 1 pushbutton for 5 different sounds in one music category. We used a voltage level converter to convert 5V output voltage from IR sensor to 3.3V voltage into the GPIO pins so that it would not burn the pins. The connection picture is shown below.

4. RGB LED Matrix

The setup of 32*16 RGB LED matrix panel went smooth thanks to the tutorial from Adafruit website which is included in the reference part. We first tried to connect the panel with the Raspberry Pi GPIO pins. However, the panel requires 16 pins and the GPIO pins are not enough with IR sensors connections. In that case, we used Arduino Uno specifically for controlling the matrix and connected Arduino with RPi for higher level control. To program the LED display, at first, we tried to program the matrix to display a certain pattern for a certain sound sources. The result was extremely satisfying since the designed pattern has lots of eye-blasting features. However, the problem occurred when we tried to loop the sound. Since the LED pattern can only be triggered with the IR sensor, there were no patterns occurred when looping the sound. To solve this problem, we first tried to edit the python control file to “record” the sequence of IR sensors detections and “replay” the sequence into the Arduino controlled LED matrix. This solution is technically possible but we met many problems with timing issues which were difficult to solve. We had to look for another solution due to deadline pressure. We analyzed the problem and realized that it would be better to display the LED pattern based on the output sound. Therefore, we searched different functions in PD since there is a final sound output port. We found out the “rfft” function whose input is the form of audio signal and output is the form of digital signal. We then looked into the FFT graph of the output sound and chose the frequency with the most significant change of amplitude to be displayed in the LED matrix. The sampling rate is set to 50ms so that the LED pattern can change on time. PD in RPi would communicate with Arduino and the Arduino would send the control signals to the matrix panel.

5. Python Files

There are two Python files in RPi. Both of the files displays a pygame screen on the piTFT screen. The first Python file “start.py” was added to rc.local so that it would be executed at boot. “start.py” has two buttons: start, quit. When touching the “start” area, the second Python file “control.py” would be executed and the PD file “loop.py” would be executed in the background. “control.py” would detect 8 GPIO signals, 4 from pushbuttons and 4 from IR sensors. All of the signals are generated at falling edge. When the signals are received, some data would send to “loop.py” to generate sounds. “loop.py” would generate audio signals to the speaker and digital signals to Arduino Uno to control the LED matrix.

Testing

We tested our project first by using a monitor and a keyboard. At first, we tested the loading of the music resources in PD. After the sound was successfully played out from the speaker, we implemented the replay function and delete function. Afterwards, we added the category features so that by choosing different categories, different sounds would be loaded with the same keypad entry. After finishing most of the fundamental music functions, we took a break and shifted our focus towards the RGB LED matrix panel. We firstly loaded some examples from Arduino LED library downloaded from Adafruit. The panel had a fantastic display when testing with the examples. As stated above, the pattern was unique for different sounds initially but this method encountered many problems. In that case, the pattern was changed to be the real-time reflection of one of the frequency bin’s amplitude. After the implementation of LED, we added the IR sensors to the GPIO pins and created the python file to receive the falling edge of the IR sensor input and output to the Arduino and the speaker. The python file was executed in the python shell to test. By clicking different music categories and pressing different keys in the keypad of the keyboard, different sounds were played out of the speaker. The functions of record, replay, and delete were also tested and they worked successfully. Finally, we edited the Python code so that the pygame screen could show up on the piTFT screen at boot of the system. However, when the second Python file was loaded, the PD file could not be loaded, ie., there was no sound playing. After searching for the problem, we realized that the PD could not be loaded without GUI when the command option “-nogui” was not entered. The problem was fixed when adding the option.

Result

At the final stage, the music can be successfully loaded and played when triggering the IR sensors with IR LED. Different categories can be selected by touching the specific area on the piTFT screen. Different sounds can be played with different categories. Moreover, the LED matrix worked well when playing different sounds. The record, loop, and delete functions also worked as we expected.

Conclusions

It was really exciting when we discovered that all the functions worked as we expected after a lot of dedications. There is only one thing that does not work really well. The IR sensors are hard to trigger and the solution can be found in Future Work section.

Future Work

1. Even though the LED matrix works well, we still would like to implement a better visual effect of LED matrix.

2. Since the IR sensors did not have a good performance, it should be much better to change IR sensor to launchpad soft buttons.

3. The performance could also be improved a little when we changed the triggers from falling edge to rising edge.

4. Sometimes the sound may not be changed if users do not touch the right area of the sound categories on the screen. Therefore, it is necessary to add an indication for different sound categories on piTFT screen.

5. Even though the wood appearance does bring a style to DJ loop, we would like to paint the wood or use another material to equip the RPi DJ loop with high-tech feeling if we had time.