Opencv contains more than 2500 functions, and almost any mature algorithm you can think of can be implemented by calling Opencv’s functions, which is super convenient. This post combines Python and Opencv to detect cars in surveillance video and check whether they commit a violation (crossing the double yellow line).
System environment: Ubuntu16.04LTS+OpenCV3.3.0+Python2.7.12
Background extraction
For surveillance the position is relatively fixed, so the background will not change considerably over a long period of time. We can therefore consider using background subtraction to roughly extract the background.
Here I use the KNN algorithm built into Opencv; this algorithm uses the first small portion of the surveillance video as training frames to train a model and obtain an approximate background image.
Of course OpenCV already implements several very easy-to-use algorithms. For the specific differences see: background subtraction
# coding = utf-8
import numpy as np
import cv2
cap = cv2.VideoCapture('filename')
#specify the number of training frames as 30
trainframe = 30
#KNN algorithm background subtraction
bgsubstractor = cv2.creatBackgroundSubtractorKNN(detectShadow = True)
Preprocessing
The video frames are median filtered according to the video resolution. For different video resolutions, different kernel parameters are used. For example: for a video resolution of 240X320, the kernel parameter chosen is 5X5.
After median filtering the image is converted to grayscale for Gaussian blurring, and a suitable kernel parameter can be found experimentally. This is followed by edge detection, where the lowest and highest thresholds are also found through repeated experiments. Finally morphological processing is performed, such as erosion and dilation; the default kernel is used here with 2 iterations.
#median filter the original frame
gray = fg_mask.copy()
gray = cv2.medianBlur(gray, 11, 0)
#image preprocessing
thresh = cv2.threshold(gray, 244, 255, cv2.THRESH_BINARY)[1]
thresh = cv2.erode(thresh, None, iterations = 2)
thresh = cv2.dilate(thresh, None, iterations = 2)
Moving object detection
Use the findContours() function in Opencv to obtain the detection boxes in the corresponding video frames.
Note: a copy of the video frame should be used
The contour retrieval mode chosen is cv2.RETR_EXTERNAL, which returns only the outermost contours. The contour approximation method is cv2.CHAIN_APPROX_SIMPLE, which returns the minimum number of key points of the detected box.
#obtain the detection boxes
_, conts, _ = cv2.findContours(thresh.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
Filtering moving objects
This part could use the cascade classifier in Opencv for recognition, but due to the lack of a sufficient sample set and time, a method based on the contour area is used here to distinguish objects from non-objects.
For details see: object detection with the Opencv cascade classifier
#object extraction
#obtain the approximate list of vehicles carlist
carslist = []
cars = []
cars = sorted(conts, contoursSort)
for car in cars:
area = cv2.contourArea(car)
if area > cv2.contourArea(cars[0])/2 and area > 400:
carslist.append(car)
In the code snippet above there is a function contoursSort(), which is used for sorting.
#contour area sorting rule function
def contoursSort(c1, c2):
return cv2.contourArea(c1) > cv2.contourArea(c2)
Marking the double yellow line
The Hough transform can be used to obtain the position of the fairly close double yellow lines.
#Hough transform
rho = 2
theta = np.pi/360
threshold = 15
min_line_length = 40
max_line_gap = 20
line_frame = np.copy(frame) * 0
lines = cv2.HoughLinesP(masked_edges, rho, theta, threshold, np.array([]), min_line_length, max_line_gap)
#draw the lines
for x1,y1,x2,y2 in lines[0]:
cv2.line(frame, (x1,y1), (x2,y2), (0,255,0), 2)
Collision handling
First obtain the coordinate points of the current frame box: use the boundingRect function to get the coordinates $(x,y)$ of the top-left corner of the frame box and its width and height $(w,h)$.
Record the position of the double yellow line as: the $x$ coordinate of the left yellow line $x_b$, and the $x$ coordinate of the right yellow line $x_c$
Collision cases: Moving forward and crossing the double yellow line: $x_b < x < x_c$ or $x_b < x+w < x_c$;
Stationary or straddling the double yellow line: $x < x_b$ and $x+w > x_c$
After a collision is detected, draw a box around the violating vehicle. Compute the centroid coordinates of the contour in the $x$ and $y$ directions separately, and based on these draw a rectangle with a suitable distance.
#mark the violating vehicles (red box)
#position of the double yellow line
x_b = 155
x_c = 160
for car in carslist:
x, y, w, h = cv2.boundingRect(car)
#detection value for crossing the double yellow line while moving
k1 = (x_b < x < x_c) or (x_b < (x + x+w)/2 < x_c)
#detection value for being stationary or straddling the double yellow line
k2 = x < x_b and (x + x+w)/2 > x_c
if k1 or k2:
M = cv2.moments(car) #compute the various properties of the contour
cx = int(M['m10']/M['m00']) #centroid in the x direction
cy = int(M['m01']/M['m00']) #centroid in the y direction
cv2.rectangle(frame, (cx-10, cy-10), (cx+20, cy+20), (0, 0, 255), 2)
Saving the video
Usually we want to save the processed video, which can be done by constructing a VideoWriter class in advance.
prename = raw_input('File name (.avi): ')
name = prename + '.avi'
#obtain the frame rate of the original video
fps = int(cap.get(cv2.CAP_PROP_FPS))
#obtain the size of the original video
size = (int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)), int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT)))
#set the video format
fourcc = cv2.VideoWriter_fourcc(*'MJPG')
out = cv2.VideoWriter(name, fourcc, fps, size)
Summary
The above is my implementation process for detecting traffic violations in surveillance video. This process was very hard going, but thanks to my persistence the whole idea gradually became clear and my inner restlessness subsided quite a bit. From being completely lost about using Python-Opencv at the beginning to writing this beginner-level article now, I have learned a lot — not to mention all those morphological transformations, edge detection and other image and video processing algorithms, my Baidu/Google skills have also advanced by leaps and bounds. Judging from the detection results, this program is not entirely satisfactory and there is plenty of room for improvement, for example adding a shadow removal algorithm.
Reference
Thanks to the almighty Baidu/Google.
Below are some references:
OpenCV official tutorial (Chinese version)
Qin Xiuli. Shadow removal algorithm based on YUV color space and graph cuts [D]. Wuhan University of Technology. 2010
Wang Zhiyong. Design of a traffic violation monitoring system based on video sequences [D]. Zhengzhou University. 2015
Luo Yurong. Research and implementation of a seat belt recognition system [D]. Beijing University of Technology. 2008
Wu Peimin. Research on video-based face detection and recognition [D]. Nanchang University. 2011
Finally, a few blogs/websites: