Pothole Detection and Avoidance: Driving Autonomy to the Next Level

In order to achieve full self-driving, a system would have to be able to handle a wide range of different scenarios, including different weather and road conditions.

Then there are potholes. It can sometimes be difficult for even human drivers to handle, and some people find it improbable that self-driving systems will be able to appropriately navigate them. However, computer vision and machine learning can help solve this issue by detecting potholes on roads and informing the vehicle about its position.

Pothole detection and avoidance may be considered similar to other obstacle detection and avoidance, except that the potholes are depressions rather than extrusions from a surface.

Potholes are formed due to wear and tear and weathering of roads. In particular, potholes can cause more problems in the rainy seasons. We can see an example in the image below. Damage compensation and repair costs due to potholes results in loss of a lot of money.

They cause not only discomforts to citizens, but can also damage cars and may even lead to life-threatening accidents. If a vehicle passes over a pothole at high speed without noticing it, the vehicle may break away from the driving lane, which may pose a threat to the driver’s life. The US itself records more than 2000 fatal accidents per year due to potholes and bad road conditions.

Hence, a fast and accurate pothole detection system is a necessity for realizing a self-driving car.

Introduction

Many companies like Google’s Waymo are planning to launch autonomous vehicles in full swing. and object detection is a key software and a fundamental task for an autonomous driving system. Computer vision and machine learning algorithms are important tools for detecting obstacles around the vehicle like other cars, pedestrians and even potholes.

One of the many annoyances countless drivers face are the potholes on roads. It does not matter how shallow or deep these are, driving over them is a huge pain. These little gaps in the road can interrupt the flow of the commute and are a major cause of axle and suspension failure in cars. They can form as a result of bad road design and building as well as severe weather and poor drainage.

However, it seems that while the process for repairing these won’t get any easier, the technology enabling drivers to avoid them could get better.

Detecting potholes using deep learning would help the autonomous vehicle to move smoothly without getting stuck in the potholes, which can in turn solve problems like accidents or slowing down the transport system.

To achieve this, cars generally have sensors such as LiDAR (which pulses lasers at objects and measures reflections to calculate their distances). However, these sensors are very costly. Also, potholes may be deep and can be relatively small for Lidar systems to recognize them.

So, detection using cameras can be a reliable alternative to sensors such as Lidars, as they cost less and using deep learning models like CNNs, they can provide very fast and accurate results. However, they may face difficulties in poor visibility during bad weather or night time. Hence, sensor fusion is generally used in most self-driving cars.

Apart from Self-Driving Cars also there are numerous use cases of a pothole detection system. For example, a civic authority can detect and locate potholes and assess their magnitudes so that they can plan for repairs.

Let’s begin by understanding more about what potholes actually are.

What are Potholes?

A pothole is a depression in a road surface, where traffic has removed broken pieces of the surface. It is usually the result of water in the underlying soil structure and traffic passing over the affected area.

In simple words, a pothole is a hole in the surface of a road that is formed by traffic and bad weather.

But how do they form? Pothole formation generally requires two factors to be present at the same time: water and traffic.

Water first weakens the underlying soil, traffic then fatigues and breaks the poorly supported asphalt surface in the affected area. Continued traffic action ejects both asphalt and the underlying soil material to create a hole in the road. We can see an example of potholes in the image below.

In areas subject to freezing and thawing, frost heaving can damage a pavement and create openings for water to enter. In the spring, thaw of pavements accelerates this process when the thawing of upper portions of the soil structure in a pavement cannot drain past still-frozen lower layers, thus saturating the supporting soil and weakening it.

Potholes can grow to several feet in width and if they become large enough, damage to tires, wheels, and vehicle suspensions is liable to occur. Serious road accidents can occur as a direct result, especially on those roads where vehicle speeds are greater.

Now that we know what potholes are, how they are formed and what damage they can do, let’s try to understand how we can detect potholes.

How is Pothole Detection Done?

The task of detection of potholes is very similar to any other object detection task.

Due to the recent progress in Artificial Intelligence (AI), various CNN-based Deep Learning models have been proposed for detection of potholes.

To achieve the task, the model is generally fed with lots and lots of images of potholes. Looking at pixels (or groups of pixels) in the image, it then learns features of potholes like shape that can be used later to identify a pothole. When an image is shown to the model later, based on its learnings, it can then detect potholes in the image.

At a very high level, these steps are followed:

1. Data Acquisition and Preparation:

We need labeled images to train our model. There are two ways you can achieve that: You can try to find a good labeled dataset and use it for your training, which is very rare. Or you can download lots of unlabelled images of potholes and label them yourself using tools like Labellerr. You can see an example in the image below where pothole images are being labeled.

Labellerr is a tool that can help you label raw data in image, text, and audio formats according to specific techniques such as bounding box, segmentation, polyline, etc. to prepare high-quality data for ML model training. You can read more about Labellerr from here.

After you've prepared your labeled data, divide your data into two different sets: Training Set and Test Set.

The Training Set will be used to train the ML model and the test set will be used to test the performance of the ML model.

2. Model training and evaluation:

After having all the images labeled and separated into training and test sets, you are ready to train your model.

But first you’ll need to choose a model to train on. There are lots of options available like the SSD algorithm, RCNN and its different variants and the YOLO algorithm.

The Single Shot Multi box Detection or SSD algorithm offers reasonable accuracy with greater speed compared to other algorithms such as RCNN or its variants. Another advantage of SSD is that it can detect smaller objects in the image. YOLO also offers great accuracy and speed.

During the model training phase, the model will try to fit the best combination of weights and biases to a machine learning algorithm to minimize a loss function over the prediction range.

After choosing an algorithm, and training using your training set on it, you’ll need to evaluate the performance of your model.

Model evaluation is the process of using different evaluation metrics to understand a machine learning model’s performance, as well as its strengths and weaknesses. Model evaluation is important to assess the efficacy of a model during initial research phases, and it also plays a role in model monitoring. Examples of such metrics are accuracy and precision.

3. Model deployment for real-time detection:

Now your ML model is ready to be used and deployed in real-time applications. You can combine it with the dashboard cam on your car and detect potholes in your way. An example of pothole detection can be seen in the image below.

However, building a reliable pothole detection model is not as easy as it sounds, there are various challenges you might have to face.

Challenges

1. Lack of data is one of the biggest concerns for any deep learning model. You need a lot of images for all possible scenarios, without which your model may fail in certain scenarios.
2. Small potholes or potholes filled with water or soil with similar color as of road might make it difficult to detect the pothole.
3. Another challenge is that the vehicle moving ahead might cover the road, and there might be very less time for your vehicle to detect the pothole and react to it. Such cases can especially arise on highways, where drivers drive very fast.
4. Poor visibility due to bad weather like rain, fog, or snowfall and night time could make it difficult to detect the pothole too.
5. Even if the driverless cars could detect potholes, which may not be possible in all weather conditions, it may still be dangerous for them to avoid them, as they might crash into some other object during the process, or leave their lane which might be dangerous. So, it is more likely that they will simply drive slower.

Or even better, it will be able to change the configuration of the suspension system in real-time to better handle hitting the potholes it can’t avoid. Before the right front wheel hits a pothole that the system sees coming in 1.5 seconds, the shocks on that wheel will prep for it. The car will raise and lower itself and make other ride quality and efficiency changes in response to actual road conditions it can see coming.

Summary

In order to realize a safe and completely autonomous vehicle, we need to overcome several challenges. One such challenge is the detection and avoidance of potholes on roads.

Potholes are holes in the road usually formed by traffic and bad weather.

If we fail to detect them, the car may fall into one of these which might cause damage to the vehicle or the passengers.

Deep learning models like CNN can be used to build a fast and accurate pothole detection system.

However, several challenges like poor visibility make it difficult to detect the potholes. And even if we successfully detect them, avoiding them could be dangerous as the car might crash into other obstacles or leave its lane in the process.

For self-driving cars to move efficiently and safely, our entire road network may need to be upgraded at a cost of billions, otherwise driver-less vehicles may all be forced to crawl along in ‘proceed with caution’ mode.