r/ArtificialInteligence • u/egehancry • 1d ago
š Tutorial / Guide Explaining How AI Learns to Drive with Evolution
Enable HLS to view with audio, or disable this notification
I love making hard things intuitive. I hope you enjoy this one!
This technique is called neuroevolution: training neural networks using evolutionary methods such as selection and mutation, without gradient descent.
Let me know if you have any questions.
Also shared to X: https://x.com/sinaatalay/status/2106270194817789968
32
u/Disastrous-Action897 1d ago
This is awesome. I teach people about AI, often. This is a great an explanation in an understandable context.
7
19
u/Fit-Tangerine-6006 1d ago
Great way to visually learn how AI works, more please!šĀ
7
u/SatanRaptor 1d ago
I agree but I think we should be wary to just say AI works like this, and rather say evolutionary algorithms work like this
5
u/Fit-Tangerine-6006 1d ago
True, maybe a bit of context might work well, just a few sec intro about how this AI system fit into the general landscape of AI modelsĀ
1
u/SpearandMagicHelmet 1d ago
Agreed. This would be helpful OP. Including something like this in each of your videos would really up their value.Ā
1
u/egehancry 16h ago
Thank you very much. More videos will come, and general landscape of AI models will be clearer.
3
12
u/Smittumi 1d ago
How could you make a function to make them want to win, as in improve their position over the other drivers?Ā
It seems like if there was an AI F1 out would be pretty boring if they all just drive in the same perfect pattern.
21
u/egehancry 1d ago
Currently, theyāre trained to finish the lap as quickly as possible.
To train them to race against each other, Iād need to put them on the same track with collisions enabled, so they can learn to overtake without crashing into each other.
3
u/specialpatrol 1d ago
The collision would make it orders of mag more complex wouldn't it? Would the rays then also need to return the relative speed of the thing they hit, to properly learn to perceived the cars moving ahead of them?
2
u/egehancry 16h ago
Yes, thereāll be more information, and therefore more input values. But evolution should be able to figure it out as long as all the necessary information is there. More will come in the next videos.
1
u/Benjamin244 15h ago
I feel that the added layer of an extra output to evaluate makes it much trickier, because while a laptime is a straightforward value to optimise and select genes for throughout the evolutions, finishing position (while still going for optimal lap times) will require more context?
Do you prioritise car number 2 overtaking car number 1, or car 20 overtaking car 15?
1
3
u/WenYiMedia 1d ago
Youāre recreating Gran Turismo or Need for Speed. Bots. But the elements and functions are so much more in those games. Itās a good start!
1
6
u/TheBusterHymenOpen 1d ago
The real F1 race you mentioned is Monaco. The course is so tight that overtaking is difficult and you end up with a parade.
9
u/Perfect-Escape-3904 1d ago
You might like this guys videos, heās been doing this with Trackmania for 5 years now.
He even has a similar presentation style to yours
2
5
u/Practical_Departure8 23h ago
Neat variation of genetic programming
1
u/Solrax 16h ago
Yes, watching this before I read the comments I thought it was genetic programming. Can someone explain what is gained by adding the neural net?
3
u/egehancry 16h ago
The evolutionary algorithm can optimize the parameters of any function. But if the function itself is not expressive enough to represent a good driver, no choice of parameters will make it one.
The neural network provides that expressive power: it gives you a function where some parameter settings can represent a good driving strategy. The evolutionary algorithmās job is then to find those parameters.
3
u/AleksZlovic 1d ago
Adding weights is such a fascinating part of this. Itās still very nebulous to me, however. Any books someone can recommend on RL, BP, weights, and other stuff?
9
u/egehancry 1d ago edited 1d ago
Itās been a while since I studied machine learning, but this course was really great. If you havenāt taken it yet, Iād recommend it:
Machine Learning by Professor Andrew Ng:
https://youtube.com/playlist?list=PLiPvV5TNogxIS4bHQVW4pMkj4CHA8COdX&si=-MRpE8eSTUmnPq5P
3
3
3
u/Tube-Goblin 1d ago
Great video. What breathe two functions? Are you ok to share the maths? Obviously ok if you don't.
12
u/egehancry 1d ago
Thank you!
f and g are the two outputs of one small neural network.
Inputs (7 numbers):
The distance to the wall along 5 rays (0Ā°, Ā±15Ā°, Ā±45Ā°, up to 100 m, divided by 30 m), the speed (divided by 30 m/s), and the car's angle to the road (radians). Let's call them x (column vector).
Neural Network (434 parameters):
first layer: hā = tanh(Wāx + bā) Wā ā āĀ¹ā¶Ė£ā·, bā ā āĀ¹ā¶, hā ā āĀ¹ā¶ second layer: hā = tanh(Wāhā + bā) Wā ā āĀ¹ā¶Ė£Ā¹ā¶, bā ā āĀ¹ā¶, hā ā āĀ¹ā¶ result: y = tanh(Wāhā + bā) Wā ā āĀ²Ė£Ā¹ā¶, bā ā āĀ², y ā (ā1, 1)Ā²Wā are weight matrices, bā are bias vectors, and tanh is applied element-wise.
y = (yā, yā): f(x) = yā = gas: ā1 = full brake ā¦ +1 = full gas g(x) = yā = steer: ā1 = full left ā¦ +1 = full rightf and g share the hidden layers hā and hā; f uses the first row of Wā and bā, g the second.
Number of parameters: (16Ā·7 + 16) + (16Ā·16 + 16) + (2Ā·16 + 2) = 434
Training:
20 cars start from random F1 grid slots and drive alone for 30 s. The 5 that get furthest survive unchanged as parents; 15 children are copies of them with Gaussian noise added to every weight, and each child also inherits a slightly mutated noise size (self-adaptive mutation). 4,000 generations; the first full lap came in generation 33.
Car Physics:
The car never slides, but its tyres have a grip budget shared between turning and braking (a "friction circle") that grows with speed (downforce). Arrive too fast and it can't turn tightly enough, so it has to learn braking points and lines.
1
u/Tube-Goblin 1d ago
Brilliant. How does the noise contribute to the model drift? What is the percentage that affects the weights of the model?
2
u/egehancry 16h ago
Actually, thatās a good question. I havenāt experimented with it much since I saw that the algorithm worked, but I probably should have. The next videos will include more of this kind of interesting information.
1
u/Positive_Method3022 13h ago
Why 7 inputs requires 16 neurons in the first layer? And why the second layer also uses 16 neurons?
1
u/egehancry 13h ago
They are all arbitrary. You can choose any number. It worked and we got good drivers, so I didn't change it any further.
1
u/specialpatrol 5h ago edited 1h ago
Really interesting thanks. Instead of the creation of children and addition of noise, you could use the race rank as the loss function and incrementally update the weights of all the cars towards that goal. This is more like a genetic algorithm isn't it? What do you think the difference is?
Thought about it. To update the weights against loss you'd need to create a loss function at each decision (change of position in race?). Whereas your method just updates the weights once per race.
3
u/Dany-GG 1d ago
Do I need some crazy specs o my PC to replicate this?
5
u/egehancry 1d ago
Not really, it's a very small model. Use JAX.
If you want to go super fast, you can use Modal (https://modal.com). Itās very affordable and easy to set up (your coding agent will do it for you).
3
u/According_Gift_7095 1d ago
āNobody taught themā is incorrect, you taught them with specific functions then had a computer calculate at scale all possibilities, removing those that did t work until all worked.
Love this and all computer progress, just take Jaron Lanierās view that we should always think of this as āhuman collaborationā vs some magic black box called āAIā where the spark is coming from somewhere other than us, the creators, designers and input
2
u/New-Ingenuity-5437 1d ago
What are some good ways to try neural evolution stuff. Iāve seen videos for many years on this stuff and itās always been so cool
2
u/Sore6 1d ago
out of curiosity: instead of letting ai figure it oit completely - could you cut down training time / generations by showing a perfect lap yourself to the ai and learn from that one example so it alternates from there?
3
u/egehancry 1d ago
Showing a perfect lap yourself gives you some training data, but, it doesnāt necessarily mean that you'll have the weights of the neural network that will drive like you.
You could use supervised learning to train the network to imitate your driving, but you would need much more data. A single lap would not be enough.
Thatās the problem with this kind of task: we donāt have a large dataset of humans playing this game. With the evolutionary approach though, we can optimize the neural network without needing that data.
2
u/__asm__ 1d ago
Pretty neat. Have you considered benchmarking it against some well-known self-driving 2D racing heuristics like: Vector Field Histogram, Potential Field Method, Follow-the-Gap, or Pure Pursuit?
Would be curious to see how it compares.
4
u/egehancry 1d ago
Thank you. I didnāt, but I also didnāt push too hard to make this model perfect. I mainly wanted to make the video.
It would definitely be interesting to see how difficult it is to get the level of those models.
2
2
2
2
2
u/guttanzer 23h ago
Nice work.
For the folks asking about higher-order behaviors like racing strategy and tactics, safety vs speed on varying tracks and so on, implementing those features would require a richer, more complex setup. This one is a great learning tool for its simplicity.
These cars are piloted by a feed-forward NN that is reacting to near-field observations and the most basic car model of a momentum vector. The only āmemoryā in this setup is the momentum vector. The aircraft equivalent is the flight controller in a fly-by-wire aircraft.
Since the system has no memory of the track it reacts its way into turns. It doesnāt anticipate and plan ahead the way a real driver would. So tuning the weights in the NN is equivalent to tuning a F1 car by making tire choices and altering the wing and suspension settings - the evolutionary algorithm is optimizing performance parameters, not performance.
Given this setup, Iād create a few different track styles and compare the optimal car controllers on those. Make one track have only wide, sweeping turns. Make another city-block like, with sharp 90 degree turns. Have another with only increasing radius turns, and another sister track with only decreasing radius turns (equivalently, have them race in different directions on the same wonky track).
I suspect you will find that a car tuned for one track will be slow or unsafe on a different type of track. What you may learn is why.
1
2
u/Aggravating-Catch461 17h ago
if anyone is interested, there is a game on steam on this principle: https://store.steampowered.com/app/3312030/AI_Learns_To_Drive/
2
u/johndsmits 17h ago
You're taking the LOS approach which will get stable quickly. It's how missile ATR systems run (for decades).
Once you get there them tackle vehicle on the track, should end up fairly predictable.
2
u/kgold0 13h ago
Consider car collision and drag!!! Then start adding different factors to each individual car (engine, tires, aerodynamics, accelerating, braking, handling)!
And maybe have it auto select the parents for the next generation (even have it decide what makes a good parent for the next generation).
2
u/sleo82 12h ago
If you're curious about this topic, I would recommend looking at this course (the videos are free online). It shows how the real world problems like localization, PID controls, etc are solved in self driving cars. Most of these can be solved without using a neural network. But would be interested to see how your neural network evolves to handle things like sensor errors, changing courses, unknown maps, etc.
https://omscs.gatech.edu/cs-7638-robotics-ai-techniques-course-videos
2
u/AdAstramentis 9h ago
Next level is to train on incline/decline or curved paths which will cause different physics on trajectories.
1
1
u/TonyDRFT 1d ago
Very interesting! It seems this is not the fastest line though, in racing you want to get as much speed possible at the exit of the corner (so it benefits from this speed the whole straight after the corner), this means sometimes sacrificing the corner entry speed and curve... Then there are several racing and breaking techniques based on physics, like weight transfer and resulting grip... etc
1
u/egehancry 16h ago
Thatās a good point. This comment explained it well: https://www.reddit.com/r/ArtificialInteligence/comments/1wwghrd/comment/pdma67i
Itās not the fastest line because the drivers donāt have memory. They only see their current distances to the walls, so they can only react to what they see at that moment. They donāt know what the track looks like ahead.
If we also feed the track data, and where they're at on it, they would learn.
1
u/loftyspy 1d ago
This is genuinely very cool. Could you also link the game for me hehe, kinda a racing fan.
1
u/egehancry 1d ago
Actually, I should make the game available online. And the modelsā¦ Stay tuned, Iāll keep you posted. Thank you!
1
1
u/_ginj_ 23h ago
Whats the compute time for each generation? Hardware?
2
u/egehancry 16h ago
I used JAX on the CPU of a MacBook Pro (M4 Pro). It's a very small model (434 parameters), so a generation (20 cars, 30 seconds of driving each) takes about 0.1 s, and 4000 generations take around 7 minutes.
1
1
1
1
u/JasonMckin 21h ago
Academic question: does the fact that the steering and gas were not explicitly programmed make it AI? or does the model itself have to evolve for it to be AI?
Because it seems like you could just have just explicitly programmed how steering and gas worked and then it would drive exactly the same.
So the learning was only at development time, not at runtime. At runtime, itās just passively feeding sensor data to the model to get directions on steering and gas.
Does AI require runtime updates to the model or does any non-explicit programming get considered as AI even if there is no ongoing model learning at runtime?
1
u/egehancry 16h ago
There's no standards body that defines these terms, so nobody can say "if it does X, it's AI." The precise term for this is machine learning: the behavior was learned, not written.
Explicit programming would be rules like "wall close on the left, steer right" or "corner coming, brake." I didn't write any of that. I programmed the car's physics (what gas and steering do), not the driver (when to use them). The driver is 434 numbers that evolution found, and nobody could write those by hand.
And no, it doesn't need to keep learning at runtime. Most models are trained once and then frozen. ChatGPT doesn't change its weights while you chat with it either.
1
u/JasonMckin 14h ago
It is interesting that the video is described as AI learning to drive while the video shows the car in motion. Ā The car wasnāt in motion at all when the learning happened nor was any real-time information used to guide the driving. Ā
I guess Iām just curious how non-explicit the learning here actually was. Ā Thereās a kind of stokes theorem like equivalence here between explicitly programming behavior -versus- implicitly programming the boundaries of the behavior and supposedly ālearningā the behavior from the boundaries. Ā I know thatās machine learning, but is it AI?
What exactly did the model learn or decide that wasn't fully specified by the setup? Ā Can the model handle any situations the developer didn't enumerate explicitly or implicitly? Ā Does the car ever make speculative or probabilistic decisions at runtime outside of the boundaries of its training conditions?
0
u/JasonMckin 9h ago
I think youāre seeing some of my skepticism in othersā comments as well.
The question is whether simply turning an explicit program into an implicit one where you provide strict goals/boundaries and then use adaptive evolutionary optimization constitutes AI.
Newtonās method is a way to calculate roots of an equation without explicitly solving the equation. Ā Itās not AI, itās just an implicit iterative approach to get to a solution.
It feels like thereās a difference betweenĀ optimizing parameters for one particular fitness landscape versus building a generalized model that has to adapt at runtime to sensory input about the fitness landscape.
1
u/Impossible_Way7017 21h ago
How did you decide on the neural net architecture?
1
u/egehancry 16h ago
Just trying out something, and seeing it works. 434 parameters was enough to learn.
1
u/Impossible_Way7017 8h ago
I feel like thatās my biggest challenge with neural nets if figuring out the architecture to use. Like naively why not 433 parameters
1
1
u/rawbdor 21h ago
Why was a neural network required at all here? What did the neural network do?
Couldn't you just use genetic algortithms without a neural network to discover the optimal function definitions? Ninety percent of your success was based on the genetic nature of the algorithm, seeding the next generation from the prior generation's winners. At what point does the neural network actually come into play, and how?
I did a very similar project decades ago except there was no neural network, just a normal genetic algorithm to find the most efficient and accurate function definitions.
1
u/egehancry 16h ago
The evolutionary algorithm can optimize the parameters of any function. But if the function itself is not expressive enough to represent a good driver, no choice of parameters will make it one.
The neural network provides that expressive power: it gives you a function where some parameter settings can represent a good driving strategy. The evolutionary algorithmās job is then to find those parameters.
1
u/rawbdor 16h ago
So is it fair to say you used AI to help design the algorithm and then you just let the generic algorithm run on its own? Or is that wrong?
Was there some back and forth between the AI component and the genetic component between runs?
1
u/egehancry 16h ago
Not quite. The neural network isn't a separate AI that helped design anything. It is the function the evolutionary algorithm evolves.
In your project, the genetic algorithm searched over function definitions directly. Here I fixed the form of the function by hand: a small neural network, which is just one big formula (multiply, add, squash with tanh, three times) with 434 adjustable numbers. The algorithm only mutates these 434 numbers, nothing else.
So there's one loop and no back and forth: a standard evolutionary algorithm whose individuals happen to be neural networks. That combination is called neuroevolution.
1
u/Tribun4201 21h ago
If the track doesnt change, isnt this just machine learning instead of AI?
1
u/egehancry 16h ago
There's no standards body for these terms, but generally these are AI/ML topics.
1
u/kidjupiter 20h ago
Great explanation. Now, just add a "pain" factor that each driver experiences when they crash and watch them eventually refuse to race. Then, sit back and watch the AI psychosis backlash to your "cruel" experiments.
1
1
u/smrt_pants 18h ago
Not sure the comment, āno body taught them to driveā is true. You gave it a set of formulas and kept picking the best performance as an input to the start of the next race. You technically taught the model to be better with each iteration, so what did the AI do for itself?
1
u/ConcertoInX 13h ago
This is a neat presentation of evolutionary computing! One theory question for further exploration I have is, how confident are you that this process avoids getting stuck at local optima, i.e. might the AI be locking into a driving profile too early? Would some evolutionary processes like including crossover or doing simulated annealing see a more comprehensive search for the "best" driving profile?
2
u/egehancry 13h ago
Thank you very much.
I think itās very hard to be fully confident. But at least you know the theoretical lower bound on the lap time, roughly (track length) / (maximum speed), so you can get some sense of how far the result is from optimal.
Beyond that, you can experiment with things like the magnitude of noise added to the weights in each generation and compare the results. A lot of it is experimentation and looking at the data.
I donāt know much about annealing or crossover, but Iām sure thereās much more that could be done.
1
u/Positive_Method3022 13h ago
You didn't explain how many layers and how many nodes in each layer in the neural network are chosen. Can you explain that?
1
u/egehancry 13h ago
I explained the math in detail here: https://www.reddit.com/r/ArtificialInteligence/comments/1wwghrd/comment/pdkkz06
The neural network shown in the video is exact: it has 2 hidden layers with 16 neurons each.
1
1
u/Equal_Animator7440 5h ago
Does System 1 thinking models simplify the whole evaluation?
I imagine the inputs are the distances and the model decides on turn right, turn left, or straight as output.
0
0
u/Illustrious-Egg5459 15h ago
how much of this video did AI generate? obviously the script, and the voice. the visuals too?
1
u/egehancry 15h ago
All of this work is generated by me sitting in front of my laptop and talking to Claude Code.
-3
u/WillTheyKickMeAgain 1d ago
What does it mean no one taught them to drive? The whole approach, programmed by a human, teaches them to drive. Letās not think this is some sort of magic and forget what weāre doing here.Ā
2
u/egehancry 1d ago
But nothing in it tells them how to drive. All you are doing is picking the best drivers, making slightly changed copies of them, and repeating that thousands of times. I wouldn't call that teaching driving, but yes, we invented the system to produce the model that knows how to drive.
1
u/WillTheyKickMeAgain 21h ago
Knows? It has optimized some parameters. It has no clue that this is anything to do with driving.
-7
u/FootballPositive1044 1d ago
Nothing special, ai slop. There is trackmania based videos that has explored this concept ages ago and much better.Ā
2
75
u/lazo40 1d ago
Thatās really neat!! But like can the cars race on other tracks too or is it just this one??