Introduction
Our project is a Raspberry Pi based tracker which tracks a moving object and autonomously follows the moving object. The system has two main parts: (1) tracking system (2) controlled vehicle. The PRi camera captures the image of moving object and sends the image frame back to the RPi. After obtaining an image frame of moving object, the RPi runs a tracking algorithm that detect the object in the frame and returns the position and size of the object by catching the contour of the object. The control algorithm will control the direction and speed of the vehicle by analyzing the position and size of the object.
Overview & Objective
The major drawback in today’s surveillance systems rests on the involvement of human operators which can easily be distracted. To mitigate this problem, our team aims to establish a system that can autonomously monitor regions continuously, make decisions when identifying unwanted or obnoxious things, and respond accordingly. At first, our team wanted our system to track any interested moving objects, but this idea highly depended on the OpenCV3 boosting tracking API, which involves numerous calculations. By that, RPi3 would not be able to get the current frame immediately. Therefore, we decided to track specific objects like a tennis ball, which decreases the calculations of tracking algorithm, and makes the tracking process more precise and efficient.
Design
Tracking Algorithm
TLD Tracker
With the initial idea of tracking the object with a given bounding box efining the object of interest in a single frame, our goal is to automatically determine the object’s bounding box or indicate that the object is not visible in every frame that follows. We first chose to use TLD tracking algorithm which is a machine learning based long-term tracking algorithm. TLD stands for tracking, learning and detection. This tracker decomposes a long-term tracking task into three short term components. It follows objects frame to frame. The detector localizes all appearances that have been observed so far and corrects the tracker if necessary. The learning estimates detector’s errors and updates it to avoid these errors in the future. We implemented this algorithm in our own PC, the accuracy of this tracker is satisfactory in general. However, there are some disadvantages: (1) It involves a large number of calculation, which will decrease the tracking speed. (2) The output of this tracker tends to jump around a bit. First of all, TLD is machine learning based tracker, which tends to learn every frame in order to avoid errors happen in future. As a result, the tracking speed will decrease dramatically. Secondly, the tracker will sometimes track the wrong object temporarily, so it not the best choice due to the large amount of false positives.
Boosting Tracker
Boosting tracking algorithm is based on an online version of AdaBoost which runs smoothly on Raspberry Pi. AdaBoost classifier needs to be trained at runtime with positive and negative examples of the object. The initial bounding box supplied by user, the user can draw a bounding box as their region of interesting. And many image patches outside the bounding box are treated as the background. With the implementation of it on PC, we found that the calculation speed is fast, however, there are some advantages: (1) The increased tracking speed will sacrifice the tracking accuracy, and it may trace the wrong object and not learn from errors. (2) Boosting is a not self-adaptive algorithm, and the size of bounding box draws at the first frame will not change frame by frame. Since the vehicle needs to respond based on the tracking feedback, the boosting is not able to provide accurate object information. Thus, we discarded it.
Template Matching
The template matching method is used as a simple method to track objects or patterns that we want to search for in the image frame. It recognizes a segment with the highest correlation as a target, but it is not as adaptive as TLD and boosting. It will miss the object when jumping to the next frame, so we discarded it as well.
Tracker combined contour with color
The algorithm we came up with is the combination of object’s contour and color information. It highly relies on the color information, so we decided to track color unique object compared with background. The color of tennis ball is unique compared with our laboratory environment. After catching the color information, the algorithm will draw a contour around the ball and send the other information to RPi, such as the size of the contour and the center of the ball. With these information, we are able to determine the distance between ball and vehicle to send the command to vehicle through RPi. Overall, the algorithm can be divided into two parts: (1) color detection, (2) draw the contour around detected colors, then extract the size and center of contour information. In order to keep the color matching with tennis ball, we have to adjust green color boundary every time when we turn on the system, due to its sensitiveness to environment. After constructing a mask of color “green”, we performed a series of dilations and erosions to remove any small blobs left in the mask. As the green mask be constructed successfully, we implement ‘findContours’ in OpenCV to find the center coordinates and radius of the contour. By gathering these information, we compare it frame by frame, then send the command to vehicle through RPi.
Servo Adjustment based on tracking result
The sevor we used in this project is the same one as we used in Lab 3. The two servos were calibrated before being used. The servos are powered with PMW signal output from the GPIO pins. In order to avoid excess current draw from pins, we added a 1k resistor between each servo’s PWM wire and its corresponding GPIO pin. We chosen pin 5 and 6 for PWM control of servos. We control the speed of our vehicle by controlling the duty cycle and frequency of PMW signals outputs from the GPIO pins. Considered the small stagnation during the movement of the vehicle due to the speed limitation and quality of servos, we control the frequency and duty cycle change every two frame. The direction and speed depends on the mean of the two frame which can reduce some sudden change due to detection error.
Testing
TLD Tracker testing
TLD is a machine learning based algorithm that can decompose a long-term tracking task into three short-term components: tracking, learning and detection. The algorithm is self-adaptive and it can learn errors from the previous frame and then correct in the next frame. We first tested it on our PC, and the tracking accuracy went well. However, due to the computation complexity, the tracking speed was very slow compared with other algorithm. Then we implemented it on RPi, but the algorithm could not send back the correct location of object on time due to the limitation of the calculation capability of RPi. In order to update the location precisely, we transplanted TLD into PC and sent back the updating object information to RPi through WIFI. However, as we mentioned before, TLD may track wrong objects temporarily, so at the initial frame, the bounding box may jump into other objects. Since the ‘cv2.imageshow’ function could not show the image frame continuously if there was large delay between the previous and current frames, we were not able to get the updated tracking image in our PC. Without it, the servo adjustment could not be implemented as well. Therefore, we decided not to use this algorithm.
Tracker combined contour with color
The tracker combined contour with color works well in the specific object tracking. It searches the predefined color inside current frame, finds the matched color pixel, and then draws a contour around these pixels. This color matching based algorithm filters out background noise effectively. If we narrow the lower and upper color boundary, the tracking result can be very precise. One thing should be noted, the OpenCV only reads HSV color code, so the color boundary should be adjusted based on HSV.
Result
Though the PiCamera is highly sensitive to the lightening of the environment which affect the color of the ball, our algorithm works well after adjustment to the color parameters of the tracking algorithm. Our tracking algorithm tracks the ball very well: the contour of the ball is tracked accurately when the ball is moving which provides accurate position and radius information of the ball to the control function. The vehicle runs following the ball smoothly. If the ball is out of field, the car will stop. Once the ball is in the field, the tracking would be resumed.