Comprendre la formation en descente de gradient Ppt

So you calculate the slope where you're standing, then step in the opposite direction to go downhill. It's like rolling a ball down a hill to find the bottom - you want the steepest path down. Step size is crucial though. Go too big and you'll bounce right over the valley you're trying to reach. Too small? You'll be there all day inching forward. I always tell people to try it on a simple parabola first - you can actually watch it work and see the math click. Once you get the intuition, the whole thing feels pretty natural.

So gradient descent is how ML models actually learn - they minimize the loss function by tweaking parameters bit by bit. Picture rolling a ball down a hill to find the bottom. The algorithm figures out which direction reduces errors most (that's the gradient), then nudges parameters the opposite way. Works for pretty much everything - neural nets, regression, all that stuff. I always start with 0.01 for learning rate, then mess with it depending on how the training curves look. Honestly took me forever to wrap my head around it at first, but once it clicks it's pretty straightforward.

So batch gradient descent looks at your whole dataset before updating weights, while SGD updates after every single example. Batch is way more stable but painfully slow with big datasets. SGD's much faster and the randomness actually helps it break out of local minima - though it gets pretty jumpy. Most people just use mini-batch anyway since it's the best of both worlds. Oh, and definitely start with mini-batches around 32-256 samples. That's what I always do and it works great. SGD alone can be a bit chaotic honestly, especially when you're starting out.

So the learning rate is basically how fast your model learns - bigger steps vs smaller ones when updating weights. Set it too high? Your model goes crazy and overshoots everything (been there, super frustrating). Too low and you'll literally fall asleep waiting for it to train. Start with something like 0.001 or 0.01 - those usually work pretty well. You can also try learning rate schedules that gradually decrease over time, which honestly makes a huge difference. It's all about finding that balance where it learns quickly but doesn't go haywire.

Ugh yeah, high learning rates are the worst - your model just bounces around like crazy instead of actually learning anything. The loss will jump all over instead of going down smoothly. Think of it like trying to hit a target but taking massive steps each time, so you keep overshooting. I've seen it blow up to infinity which is... not ideal lol. Try 0.01 first, maybe 0.001 if that's still too jumpy. Just watch your loss curves and adjust from there - they'll tell you everything you need to know.

So momentum basically gives your optimizer some memory - instead of just reacting to whatever gradient it sees right now, it builds up velocity from previous steps. Picture a ball rolling downhill that doesn't just stop when it hits a bump. The momentum carries it through small dips and smooths out all that annoying zigzagging you get with noisy gradients. Usually you'll want to set the coefficient around 0.9, so it keeps 90% of its previous speed. Honestly works great when your loss is jumping around like crazy or getting trapped somewhere.

So the loss function is like your GPS for gradient descent - it measures how badly your model is screwing up right now. Gradients from that function show you which way to nudge your parameters to cut down the error. Picture yourself blindfolded on a hillside, using your feet to feel which way slopes down. Steeper slope means you take bigger steps. Honestly, without a loss function, gradient descent is just wandering around aimlessly with zero clue what "improvement" looks like. Oh and seriously - don't rush picking your loss function, that choice can totally make or break everything.

Oh man, non-convex problems are a pain. SGD or mini-batch gradient descent works way better than regular batch methods - the randomness actually helps you jump out of local minima. Adam and RMSprop are solid choices too since they build momentum to push through weird saddle points. Start with a high learning rate then let it decay gradually. Honestly? Half the battle is just messing around with different hyperparameters until something clicks. I always run the same setup from multiple random starting points and cherry-pick the best one. Your initialization can make or break everything in these messy optimization landscapes.

Oh man, local minima are such a pain! Try momentum first - it basically lets your optimizer build up speed and punch through those shallow valleys. Adam's usually my go-to for this reason. You can also restart from different random points or add some noise to gradients (sounds weird but totally works). Learning rate schedules help too. Honestly, regular stochastic gradient descent does half the work for you since those mini-batch updates are naturally noisy. That randomness actually kicks you out of bad spots. I'd just start with Adam though - saves you from overthinking it.

So mini-batch gradient descent is basically the perfect middle ground - you process small chunks of data (like 32-256 samples) instead of everything at once. Your GPU loves this because it can actually parallelize those smaller batches efficiently. Plus you're not sitting there forever waiting for updates like with full-batch. Honestly way better than stochastic GD too since that's just all over the place with its updates. I'd say start with batch size 32 or 64 and you'll probably see your training speed improve pretty quick!

Dude, just use Adam - it's way better than regular gradient descent. The thing automatically adjusts learning rates for each parameter, so you don't have to babysit everything and watch your model either crawl along forever or completely overshoot. RMSprop is decent too, but honestly? Adam combines momentum with those adaptive rates, which works great across different problems. Start with the default settings and you'll probably see faster, more stable convergence than SGD. This is especially true if your data's noisy or sparse. I mean, there's a reason everyone uses it now.

Honestly, the biggest pain points are badly scaled features and getting stuck in local minima. Your algorithm will zigzag all over the place if one feature ranges from 0-1 while another goes 0-1000. Local minima trap you in "pretty good" solutions instead of the actual best one - super annoying with non-convex functions. Saddle points mess things up too since gradients get tiny but you're not even at a minimum. Higher dimensions make this nightmare worse because there's way more saddle points everywhere. Normalize your features first, then experiment with learning rates or try Adam optimizer. That usually helps break free from these annoying spots.

Dude, you NEED to scale your features or gradient descent will crawl at a snail's pace. What happens is the big-range features totally hijack the whole process, making your algorithm zigzag around like it's drunk instead of heading straight to the solution. Scaled features let it find the minimum way faster - we're talking 10x speedup easily. I learned this the hard way when my first model took like 30 minutes to train what should've been 3. Just normalize or standardize everything beforehand, especially with linear regression and neural nets.

Honestly, just start with loss curves - they're dead simple and you'll immediately know if things are broken. Contour plots are pretty sweet though, you can actually watch your algorithm navigate the loss landscape like it's rolling downhill. 3D surface plots look impressive but good luck reading them clearly. Loss curves track how your error drops each iteration, so you'll catch stuff like slow convergence or weird oscillations. Oh and learning rate viz is clutch for tuning - shows exactly how different step sizes mess with your path. But yeah, loss curves first since they're the most straightforward.

So here's the thing - initialization matters way more than people think. Start with weights too big and you'll bounce all over the place, missing your target completely. Too small? You'll be waiting forever for it to learn anything useful. I usually just go with Xavier initialization for sigmoid/tanh networks or He initialization when I'm using ReLU - they actually factor in your layer sizes which is pretty clever. Random small values around zero works too, but honestly why not use something designed for your specific setup? Deep networks especially hate bad initialization since gradients either vanish or explode.