The 5 Core Components of Any Self Driving Car
First Of All, What Is An Autonomous Vehicle?
An autonomous vehicle is a vehicle capable of sensing its environment and operating without human involvement.
Today we will be talking about self-driving cars. A self-driving car has 5 main components.
1. Computer Vision
Computer vision is the identification and classification of objects using cameras. It is split into three stages.
Stage 1: To identify and classify objects, we use an algorithm called the You Only Look Once algorithm (YOLO). This algorithm divides images into smaller squares and each of these squares is run through a Computational Neural Network (CNN).
A CNN is a deep learning algorithm which can be trained to recognize certain features in a photo.
The result is a class probability assigned to each square along with predictions of bounding boxes in all sorts of sizes.
Stage 2: At this stage, all the bounding boxes are present, and none of the possibilities are ruled out. Our goal is to find the most accurate bounding box for each object. We can do this using a method called Non-Maximum Suppression (NMS).
NMS is a method used in nearly all object detection algorithms. It suppresses the bounding boxes with the lowest confidence scores, and it keeps the boxes that have the highest confidence scores.
This is why this method is called Non-Maximum Suppression! It suppresses the boxes that don’t have the maximum probability!
Stage 3: After performing YOLO and NMS on an image, we obtain our result!
This method of identifying objects in images is applicable to the identification of humans, other cars, traffic lights, and everything else in between.
Now that we’ve identified the objects, let’s figure out where those objects are! We will be using a process called Sensor Fusion to achieve this.
2. Sensor Fusion
Sensor fusion is using multiple sensors to augment the vehicle’s perception of the world. We will be using sensor fusion to track objects. Sensors include: Radars and LiDARs
Before we begin discussing how to merge Radar and LiDAR data, we must study the sensors.
Radar
What is Radar? Radar stands for Radio Detection and Ranging. This type of sensor uses radio waves to determine the distance, angle, or velocity of objects. It is the type of sensor police officers use to catch speeding cars!
In our case, we will be using radars to estimate the velocities of objects.
LiDAR
What is LiDAR? LiDAR stands for Light Detection and Ranging. This type of sensor uses light pulses to generate accurate 3D information about its surroundings.
In our case, we will be using LiDARs to estimate the distance of objects.
Now that we have identified our goal and gained sufficient knowledge of RADARs and LiDARs, let’s move on to fusing the output of the sensors. We’re going to focus on late fusion which is not fusing the raw data together, but combining the output of radar with the output of a LiDAR.
To combine the respective outputs of RADAR and LiDAR, we will be using Kalman Filters. Kalman filters are unimodals and use Gaussians to represent state and uncertainty. Kalman filters rely on probability and a “predict-update” cycle to gain a better understanding of the world.
What Does Unimodal Mean?
Unimodal means that we use a single peak to estimate the state of an obstacle. Put differently, the obstacle is not both 11 meters away with 90% and 8 meters away with 70% certainty; it’s 10.7 meters away with 98% certainty or nothing.
What Are Gaussians?
Gaussians are characteristic bell-shaped curves which are implemented in Kalman Filters. They are used to represent state and uncertainty.
Now that we’ve learned how a Kalman Filter represents state and certainty, let’s learn how the filter works. Let’s do this by understanding a key concept.
Key Concept: Bayesian Filtering
Bayesian Filters are a type of filter used in Kalman Filters. They operate by Predicting and Updating continously.
Prediction: We use the estimated prior states and present states to predict the future state and uncertainty.
Update: We use sensors to correct our prediction and we consider the previous result while making the next prediction.
and repeat…
Let’s review what we learned:
- our goal is to locate the objects and track them
- sensors we will use are LiDARs and Radars
- we will be fusing their respective outputs, not the raw data (late fusion)
- we will use a Kalman Filter to do this
- which are unimodal and uses gaussians to represent state and certainty
- and uses predict/update cycles
next…
Radar LiDAR Fusion Cycle
The cycles is as follows: every time we get a new measurement, we update the system and we make a new prediction. This brings us back to the start of the cycle, and the cycle continues.
After each cycle, the predictions are more certain. This is due to the update formula implemented in the Kalman Filter. Click this link to learn more about Kalman Filters and update formulas.
Everything we’ve talked about up till this point is that the assuming objects move in a straight line. But we both know that the real world is non-linear.
LiDARs use linear-cartesian values, but because radars measures angles that are non-linear, everything we talked about does not work and is wrong. So how do we solve this?
Non-Linear Kalman Filters
The world is non-linear. Not all sensors work the same way. Thus, we must use a specific type of Kalman Filter to linearize radar output:
- Unscented Kalman Filters
Unscented Kalman Filters are used to linearize a nonlinear function of a random variable through a linear regression between n points drawn from the prior distribution of the random variable.
Considering this, here is our final sensor fusion cycle:
The cycle is not so much different from earlier except that there is an additional step for the radar output to be linearized. Now, let’s look at the results!
Results
In the video above, we track the vehicle’s posterior using radar and LiDAR sensors. We use an Unscented Kalman Filter to fuse the radar measurements (blue) with the LiDAR measurements (red) to estimate the vehicle’s current location (green).
This method of tracking is what we will use to track the objects we’ve identified in the beginning…
Now that we’ve found a way to identify and track surrounding objects, let’s figure out where the vehicle is in the world. This process is called localization.
3. Localization
Localization is determining the vehicle’s precise location on the map. By doing this, it can determine where other elements are, such as the center of the lane, the distance from the vehicle to the curb, or the distance to the next intersection.
Using GPS to do this seems like an easy solution, but there’s a problem. GPS is only accurate up to 1–2 meters. Imagine if your car was 1–2 meters off the right or the left, it would be disastrous wouldn’t it? For this reason, we need to use a different method.
There are two commonly used methods to localize a vehicle:
APPROACH #1:
We can identify specific landmarks (e.g. poles, mailboxes, curbs) and measure the vehicle’s distance from each landmark. Then, with those measurements we can use triangulation to calculate the vehicle’s location.
APPROACH #2:
Matching. We can match the point cloud the vehicle sees to the point cloud in the map.
What are the advantages and disadvantages of each approach?
The second method is more computationally intensive, but what happens if you are driving through a road with hundreds of mailboxes? Will the sensors be able to differentiate between mailboxes?
Cost…
Mapping out the streets is frightfully expensive!
Every street needs to be scanned with lasers and stitched together! Just when you think every street is scanned, the government and other companies will be out there changing the roads! Tearing it up, building new roads and repaving existing roads.
Imagine how much a change in the road could impair a vehicle’s decision making capabilities.
This is why autonomous vehicles are launching first in urban areas! The cost of scanning and re-scanning the roads is more evenly distributed among the large urban customer base than the rural area.
Now that we understand how a vehicle determines its precise location, let’s move on to path planning, the brains of the operation.
4. Path Planning
Path planning is figuring out how to get from where the vehicle is now, to where it wants to go. The vehicle uses path planning algorithms to find different routes and their feasibility.
The algorithms convert the urban environment into a digital configuration that enables it to search for and detect space and corridors in which a vehicle can drive.
Here are a few popular algorithms used in path planning:
(a) Voronoi Diagram Algorithm
This algorithm generates paths that maximizes the distance between the car and the obstacle.
Advantages of this algorithm is that it generates paths with maximum distance to obstacles. Disadvantage is that it is limited to static environments.
(b) Occupancy Grid Algorithm
This algorithm is similar to the Voronoi Diagram, but this algorithm calculates risk and feasibility through presence of obstacles and road/lane boundaries.
Advantages of this algorithm is that it has fast discretization and is computationally inexpensive. Disadvantage is that there may be problems with vehicle dynamics, and there may be errors in the presence of obstacles.
(c) Cost Map Algorithm
Much like the Occupancy Grid Algorithm, this algorithm uses grids consisting of positions of obstacles. However, the difference in this algorithm is that the obstacles are more intensely represented if they take up more cells (higher cell cost).
Advantages and disadvantages are similar to the Occupancy Grid Algorithm.
(d) State Lattice Algorithm
This algorithm uses repeated squares or rectangles to discretize a continuous space. Over the squares, lattices are constructed by regularly repeating primitive paths that connect possible states for the vehicle.
Advantage of this algorithm is that it is efficient without increasing computational time. Disadvantage is that when using this algorithm, motion is restricted and there may be difficulties dealing with restrictive maneuvers.
(e) Driving Corridor Algorithm
This algorithm uses SLAM (Simultaneous Location and Mapping) to generate continuous collision-free spaces. These corridors are bounded by lanes and other obstacles between which the vehicle is expected to drive.
Advantage of this algorithm is that it generates collision free spaces in which the vehicle can move. Disadvantage is that this algorithm is computationally expensive.
With the algorithms above, autonomous vehicles are now capable of considering their environment in a way that enables them to search for a path.
When choosing which path to take, we want to consider the safety, convenience, and efficiency of each path.
Autonomous vehicles can consider each path using three principles:
- First, the fewer lane changes, the better.
- Second, the larger the distance to the moving object ahead, the better the score for maneuvers.
- Third, the greater the velocity of the object ahead, the faster the car can drive in the lane.
Paths with the highest scores are then used to build the expected trajectory.
Results
In the video above, we see a simulation of a vehicle using path planning algorithms to generate and evaluate possible paths. The green line shows the final expected trajectory…
After trajectories are defined, we can move on to controlling the car to follow the planned trajectory.
If you want to read more on path planning before moving on to control, you can follow this link
Now that we can classify and track objects in our environment, localize ourselves in the world, and generate paths to navigate that world, let’s move on to control.
5. Control
Now that the vehicle knows where it is in its surroundings and the path it wants to take, it must put it into action. The goal of the control module is to generate an angle for the steering wheel and control acceleration in a way that allows you to follow your predetermined trajectory.
When a car is travelling in a wrong direction, the error of that path is either above or below 0. If the car was travelling on the correct path, the error would be 0. So the question is, how can gradually drive the error to 0?
We will be talking about a control algorithm called the PID in this section…
PID (Proportional Integral Derivative)
The PID control algorithm is used for applications requiring continuously modulated control. We will be implementing it into a vehicle so it can smoothly follow a path.
The PID is made up of 3 terms. Proportional. Integral. Derivative. Now, let’s look at the first term!
P
P stands for Proportional. The P term adjusts the steering in a way that the correction would be proportional to the error.
The P component should look something like this:
y = highest/lowest point of error
P = -(y)
However, the correction would too drastic that the car overshoots our reference path and when it tries to steer back it overshoots again.
Clearly it’s not going to be very comfortable to sit in this car, so how can we decrease the steering angle as we near the reference path?
PD
This leads us to PD (proportional-derivative). The main idea of PD is that as the car moves closer and closer to the reference path, the car counters the steering so it gradually eases itself into the reference path without overshooting.
The PD component should look something like this:
y = highest/lowest point of error
y2 = previous error
PD = -(y)-(y2-y)
Sure this works and all, but what if there is a glitch in the system and your wheels keep shifting to the opposite direction? Will PD be able to overcome this? The answer is no.
If a car’s wheels keeps shifting to the opposite direction, PD will not be able to overcome the inherent bias.
PID
To resolve this issue, we will need a third term. This leads us to PID (proportional-integral-derivative).
By adding the Integral term, the car avoids the chance of oscillating somewhere far away. When it sees that we are not moving closer to our reference path, the steering angle will gradually increase to overcome the bias.
To add the Integral term, we will need add few lines of code. Those lines of code will command the steering angle to be increased after a duration in which there is no progress to correct the error.
With the same steering bias, the PID controller is now able to overcome it and move itself towards the reference path.
Results
In the video above, you see a simulation of a vehicle using the PID controller. Although the PID controller is the simplest and most common in the world, it cannot adjust our maneuvers according to the size, mass and dynamics of the vehicle.
For a more robust control system, we can use the MPC controller (Model Predictive Control). This controller is very powerful and is known to integrate the physics of the vehicle. Although it is very difficult to implement, the time spent developing this regulator is worth the result. Driving is more enjoyable than in a PID controller.
And That’s It
Now that we know how an autonomous vehicle tracks and classifies objects, localizes itself in the world, plans trajectories and executes them, we’ve effectively covered all 5 components of self-driving cars!!!!
TL;DR (Too Long Didn’t Read)
- An autonomous vehicle is a vehicle capable of sensing its environment and operating without human involvement
A self-driving car is made up of 5 core components
- Computer vision is the identification and classification of objects using cameras and algorithms such as YOLO and NMS
- Sensor fusion is using multiple sensors and tools such as radars, LiDARs, Unscented Kalman Filters, and prediction/update cycles to track objects
- Localization is determining the vehicle’s precise location on the map through the identification and localization of nearby landmarks
- Path planning is using path planning algorithms such as Voronoi Diagram algorithm and Cost Map algorithms to figure out how to get from where the vehicle is now, to where it wants to go
- Control is using control algorithms such as the PID to generating an angle for the steering wheel and control acceleration in a way that allows you to follow your predetermined trajectory
___________________________________________________________________
Hey! If you’ve gotten to this part in the article, you’ve probably gotten a good look into The 5 Core Components of Self-Driving Cars 🔥🔥! Give yourself a pat on the back for all your reading 🙌💪!
If you enjoyed this article, and you want to continue learning about various emerging technologies, you should follow me 🤯🤯!!!
Until next time… 👋👋
Anson Lee, signing off
…
Hey! I’m a 15-year-old student interested in emerging technologies & self-growth. For queries or discussions, you can contact me via LinkedIn. To stay updated with my projects and progress, you can sign up to my monthly newsletter here.
___________________________________________________________________
We’re providing opportunities for the next generation of student thinkers, inventors, and learners, to publish their thoughts, ideas, and innovation through writing.
Our writers span from all areas of topics — from Growth to Tech, all the way to Future and World.
So if you feel like you’re about to jump into a rabbit hole of reading these incredible articles, don’t worry, we feel the same way. ;)
That’s why students x students is the place for getting your voice heard!
Sounds interesting? Why not join us on this epic journey?
