Robot Framework Design

Home

Software Interface

Less

For the software user interface, we wrote a python script that utilizes the pygame and the RPi GPIO library. The pygame library serves the purpose of controlling the aesthetics of the interface as well as gathering user input via the touchscreen to transition the program between different states. The RPi GPIO library is used to add event listeners and callback functions to detect the pressing of the physical buttons on the PiTFT to also transition the program between different states.The way we incorporated the touchscreen and physical button transitions can be seen in the figure 1 below.

figure 1

We will first start by explaining how the main functionalities our program leverages on before delving into the details of the code. The main functionalities we used are rendering elements on the screen, detecting touchscreen events for the touchscreen buttons, detecting GPIO events for the physical buttons and sending PWM signals to control the servos.

We composed the different rendering helper functions for displaying the background, the various touchscreen buttons and texts. We wanted to keep the code modular and helper functions in this case would be of great convenience since we want the user to move back and forth between the different pages and will need to render the background, buttons and texts more than once.

For the background of our program, we searched for a photo we like on the Internet, resized it using the Paint program such that the photo is 320x240 pixels in size i.e. the same size as the PiTFT screen, and slightly modified the color tone of the picture such that it is light enough to make the touchscreen buttons clearly visible. For our render_background helper function, which loads this background image onto the screen, we first cleared the screen by filling it with black, using the command screen.fill(black) where both screen and black have been previously declared as global variables. We then load the image using the pygame.image.load function and passed it the background image file name. We get the enclosing rectangle for the image using the get_rect function and combined the background with the enclosing rectangle with the workspace screen using the blit function. This only updates the workspace screen but not the actual screen. To update the actual screen with the workspace screen, we use the flip function.

Other elements displayed on the screen include texts and buttons. Buttons are simply texts that respond to touchscreen events and transition the program to another state but the rendering process itself is the same across all of them. For rendering a text, we first set the size of the text using pygame.font.Font function. Then we indicate the text we want, whether we want the characters to have smooth edges and the color of the text using the render function. This generates a surface and we can get the rectangle enclosing the surface using get_rect just like the background case, except this time, since the surface will not be 320x240 and fill up the entire screen, we pass into the function the position of the rectangle’s center coordinates. This will place the text at that position on the screen. Finally, we update the workspace screen using the blit function. The rendering of all the touchscreen buttons follow the exact same logic.

For detecting touchscreen events, we used the pygame.event.get function and a for loop to retrieve the event that occurred. If the event type is MOUSEBUTTONDOWN which means the user pressed down on the touchscreen. At this point, we do not do any processing but waits until the user releases the touch. At this point, we get the position where the user pressed on the touchscreen using pygame.mouse.get_pos(). This returns the x,y coordinates where the touchscreen event occurred. To turn the text into a button, we experiment with the boundaries of text and set the condition such that when the x and y coordinate retrieved from the touchscreen event is within the boundary of the text, we transition to another state by rendering a different page, thus making it appear like the text is a button.

For detecting GPIO events, we have three options: polling, polling but without missing an event, interrupts. We did not want callbacks as we wanted to reuse some physical buttons for different functionalities depending on the state we are in. This means having to add the callback and removing it when the state changes which we found to be troublesome. We did not want to use polling as we want the user interface to respond to events quickly and miss as few events as possible. Thus, we went with an approach similar to polling except even if the physical button was pressed not at the time the poll happened, the event will still be kept and the system will still respond accordingly. Firstly, to listen for an event, we use the GPIO.add_event_detect functionality. We set it to listen for each of the physical button when there is a falling edge and set a bouncetime of 300 so if the user pressed the button for a bit longer, we will not treat it as two button presses. To check if the event has occurred, we used the function GPIO.event_detected which returns true if the event happened and false otherwise.

For controlling the servos, we used software PWM. This means first using GPIO.PWM on the pin where we want to send the PWM signal and passing into it the frequency of the signal. Then we simply call start then subsequently ChangeDutyCycle to set and modify the duty cycle of the PWM signal. The information for the frequency and duty cycle needed to control the micro servos and the two continuous rotation servos can be found in their respective datasheets. To stop a servo from turning, we can just modify the duty cycle to 0.

Having explained the major components used by our design, we will now explain in greater detail on what the various helper functions do and how the different states are connected together.

The script starts off with some initialization and setup of the PiTFT screen such that when the program launches, it will start up on the PiTFT screen instead of desktop. We also need to initialize pygame by calling the init() function and setting the mouse visibility to false.

We also setup the GPIO pins for input or output. We chose the pin numbering system such that they will be referred to using the "Broadcom SOC channel" number. Since we have 6 micro servos, 2 continuous rotation servos and 4 physical buttons on the PiTFT, we need to set up a total of 12 GPIO pins. Since we are using the Pi to send signals to control the servos, the corresponding pins need to be declared as outputs. Similarly, since we trying to detect whether the physical buttons have been pressed by reading the GPIO input value, the corresponding pins need to be declared as inputs with internal pull up resistors.

We initialized the PWM signals for the micro and continuous rotation servos. For the micro servos, we used software PWM to control their motions. We looked at the datasheets for the micro servos and the frequency of the PWM signal is 50Hz which we set using the GPIO.PWM function. One issue was that when the servos are first connected to the power supply and no signal has been supplied, they are in an undefined state and thus jitter a little. This is very problematic in our case since small movement of the micro servos can cause the rotating cover to turn and the ingredients to start spilling. To solve this problem, at the start of our program, we called start(5), which sends a PWM signal with duty cycle of 0.05, and this turns the servos to the extreme left position. This extreme left position corresponds to the position when the rotating cover is closed. Only when the program is running do we turn on the power supply for the micro servos. Since there is a signal already being output from the GPIO pins connected to the micro servos, the servos are now in a defined state and will not jitter randomly. For the continuous rotation servos, we also used software PWM but with different frequency and duty cycle compared to the micro servos. Specifically looking at the datasheet, the frequency for these servos are all around 46.5Hz. Initially we do not want the bowl holder to turn so we set the duty cycle for the PWM signal to 0. Although when the power supply for the servos are turned on, the continuous rotation servos also jitter a little, this is not a problem since they do not affect any functionality of our cooking robot and we designed the framework such that we can easily reset the position of the bowl holder even if the jitter caused the bowl to be slightly misaligned with the first shaker can.

We setup some global variables which are shared across states and functions. Since we have the amount page for the individual ingredient where the user can input in the amount but we also want this amount to be reflected in the menu page after the user pressed Ok, we had a global dictionary that gets updated to the amount entered by the user and whenever we need to read the amount for the individual ingredient, we index into the dictionary. After each dispense, we want our program to return to the home page and since we want the user to use the cooking robot more than once on each run of the program, we have a global variable new_start to keep track of if one round of dispensing has completed and the user can now start a new round. Other global variables include the size of the screen, setting up the screen workspace, and different colors using the RGB values.

The render_background function is as aforementioned which basically loads the background. We did not call the flip function as we have other buttons to render and only want to update the actual screen when all the elements have been written to the workspace screen. The render_quit, render_cancel, render_ok helper functions are used to render the various buttons and works the same way as how texts are rendered as explained earlier as well. The render_dispense function simply renders the text “DISPENSING” on the screen.

For rendering_menu function, rather than rendering each ingredient name text individually, we created a local dictionary that uses the names of the ingredient as keys and the position the button will be placed as the value. We then simply go through the dictionary, get the key and the corresponding value and pass it into the render and get_rect functions respectively. For rendering the amount, we need to retrieve the amount from the global dictionary since there are two cases where the menu page is displayed: (1) when the user pressed start from the home page (2) the user has just entered an amount for a particular ingredient and pressed the Ok button. For the second case, the amount for a particular ingredient might have no longer been 0 and hence, we cannot simply just render a fixed text but rather needs to get the amount the user actually entered. We created the global dictionary holding the ingredient names and amount such that its keys are the same as the local dictionary holding the ingredient names and positions. Thus, when going through the local dictionary, the key we retrieved can also be used to index into the global dictionary. This will return the amount for that particular ingredient and we simply convert the number to a string and pass it into the render function. We want the amount text to be aligned with the ingredient name text along the x-axis but lower. Hence, using the position we retrieved from the local dictionary, we further get the x-coordinate which is index 0 and add 30 to the y-coordinate which is index 1 to place the amount text lower than the ingredient name text.

The render_amount and render_ingredient helper functions work together. The helper function render_ingredient takes in the name of the ingredient and listens to see if the the physical +, - and Ok buttons have been pressed using the GPIO.event_detected functions as explained earlier. We left the rendering of the +, - and ok texts on the screen to the render_amount helper function which the render_ingredient calls. If either of the + or - buttons are pressed, we need to use the name passed into this function as key to index into the global dictionary to retrieve the current amount before adding 1 or subtracting 1 from it. We then simply call another function render_amount that also takes in the name of the ingredient and use it as key to index into the global dictionary, retrieve the current amount and render the text on the screen. It Ok is pressed, we simply return. We do not render anything here since we want the start_h helper function, which called the render_ingredient helper function, to do the rendering process.

The render_menu and render_ingredient helper functions are all called within the start_h helper function. This function essentially connects the menu page, amount page, dispensing page and return back to home page together. In start_h, we first set the duty cycle for all the micro servos to 0. Recall from previous explanation, the duty cycle was set to 0.05 previously to prevent the jitters but is now no longer required since we have already ensured when the micro servos are powered up, they are in a defined state. Start_h helper function is entered when the user presses the start button on the home page. On entering start_h, we first render the menu page. Then we need to both listen for if the user wants to dispense, return back to home page, or selected a particular ingredient and wants to set its amount.

If the user pressed a touchscreen button corresponding to an ingredient name, we simply call our helper functions for render_ingredient and render_amount. When the render_ingredient has returned, this means the user has finished selecting the amount and we simply render the menu page again.

If the user pressed the physical button for dispensing, we simply render the dispensing page which just displays a text to indicate dispensing. Then we need to perform the actual dispensing. Since we already know the PWM duty cycle needed to turn the micro servos clockwise or counterclockwise from the datasheet, we simply read the amount value from the global dictionary and using a for loop, open and close the micro servos for the corresponding number of times by sending the right PWM signal. However, we cannot continuously change the duty cycle of the PWM signal as that would cause the micro servos to simply jitter, we used the sleep function in between changing the duty cycle to allow time for the micro servo to be turned fully before turning it back.

When an ingredient is being dispensed, the bowl will be aligned below it. The bowl’s position is controlled by the bowl holder which is in turn controlled by the continuous rotation servos. We want the bowl to be completely in place before starting the dispensing process and similarly for the dispensing process to completely finish before rotating the bowl. Hence, we asked the program to sleep for a second before and after the dispensing process to ensure little spillage. After a particular ingredient has finished dispensing, we need to now rotate the bowl holder so the bowl is now aligned below the next shaker can. To do this, we need to turn the two continuous rotation servos attached to the bowl holder. This is performed by the next_bottle helper function. Essentially this function sets the duty cycle for both the continuous rotation servos such that they turn in the same direction. The two servos need to turn in the same direction for the bowl holder to turn about a pivot point. When setting up the framework, we had ensured the shaker cans are placed 45 degrees apart. This helps us ensure that when moving the bowl between shaker cans, the duration the servos need to turn for should be the same. In this case, after experimenting, we found that asking the program to sleep for 0.48s is just right for the bowl to reach the next shaker can.

After all the ingredients have been dispensed, we want the bowl to return to the initial position to avoid the wires from being all tangled up and to make it easier for the user to retrieve the bowl. This is done by the helper function new_round. This helper function simply turns the servos in the opposite direction compared to the direction taken during the dispensing process. Since we have 6 shaker cans and the bowl is aligned below the first shaker can at first, this means the bowl moved 5 times with each time being 0.48s. Hence, to turn the bowl back to its initial position, we technically need to only let the program sleep for 0.48*5=2.4s. However, after experimenting, we found 2.7s to be more accurate. After returning the bowl to its initial position, we simply return from the start_h function. This gives control back to the main function which called start_h when the start button was pressed.

In our main function, we set up all the event detection for the GPIO pins. Then we render the home page which comprises the background, start and quit buttons. If the start button is pressed, we enter start_h. Once start_h returns, this means the current round of dispensing has completed and we render the home page again, reset the global dictionary so the amount are all 0 again, and set the global new_start variable to 1 to indicate ready for a new round. If a touch event is detected on the quit button, we clean up the GPIO and return from the main program.