Real-time object detector
Last updated
Last updated
The original project had the goal to implement a real-time object detector on the PYNQ-Z2, making use of the Zynq-7020 FPGA to accelerate the inference time. I'm proud to say it was achieved, although the frame rate isn't impressive. Anyway, this implementation might be perceived as a starting point for future developments on this matter.
To make this real-time object detector possible, you will need to have a camera, of course. The choice here was a USB webcam, as it interfaces directly with the PS side of the board, making it simpler to deal with.
Another future application might be making use of an HDMI webcam. It would certainly improve the processing time because PYNQ already have an HDMI overlay we could use. It means the HDMI processing could resort to the PL, making the image acquisition way faster.
To plug the USB webcam to the board, you follow these schematic:
After this, to interface with the webcam we can simply use the OpenCV package which is very simple to do.
The c++ program we used to detect objects on a single image is definitely the starting point to make the real-time application. The basic structure stays the same, and we just make changes on the main function. Here, we will want to implement something like this:
As you can see, we preserve the basic structure of the YOLO on the PYNQ-Z2, but we add an infinite loop. Not so infinite of course because one thing has got to give: either you press 'q' or the program crashes... In any case, on the loop, a picture is taken by the camera and then gets turned into an image (goes through the pre-processing). After that, the image is processed just like on the original program and the result will be the image with the detections on a window. This process repeats, making it "real-time".
You can check the yolo_video.cpp or the tiny_yolo_video.cpp on the Deployment folder to see how that thought process was implemented on the code.
As you might know by now, the makefile is responsible for giving the PS all the specifications for the compilation process of the c++ program. There needs to be some minor changes on the original Makefile you have on the Deployment folder of the repository (It's the same on YOLO and on tiny YOLO). We only need to point the name we want the executable to have, the object file name and the c++ program we want to be compiled. This is very easy, and you can do it just like this:
This might give you a better idea how these makefiles work. Note that you need to keep the files organized as they were because otherwise the makefile will not work.
Save the changes to the file and open the yolo_pynqz2 folder on the console. To compile the desired yolo_video.cpp program, you just execute this command: make .
An executable file will be created. This file will be called "yolo" and there is no extension to it.
To finish this off, you just need to execute the executable file. Simply type this command:
The program will take some seconds to start-up and will be prone to crash. If it does crash, just execute the command again and again until it stabilizes. A window will pop up with the video captured by the camera. There is a considerable delay and the frame rate is miserable, but it gets the job done.
I have a little demonstration video so you can have an idea of the expected results: https://bit.ly/YOLOv3-PYNQ-Z2
If you notice the console and not my bike, you can see that the frame rate appears to be more than 1 FPS, but that is incorrect. When the recording was done, there was a small mistake on the FPS counter. It is fixed now and should display something like 0.6 FPS on average
