Clever Geek Handbook

Getting intuition from a gradient descent update rule

Gradient descent update rule:

enter image description here

Using these values ​​for this rule:

x = [10 20 30 40 50 60 70 80 90 100] y = [4 7 8 4 5 6 7 5 3 4]

After two iterations using a learning rate of 0.07, theta p> 

-73.396
-5150.803

After three iterations of theta:

1.9763e+04
   1.3833e+06

So it seems that theta gets bigger after the second iteration, which suggests that the learning rate is too fast.

So, I installed:

iterations = 300; alpha = 0.000007;

theta is now:

 0.0038504
 0.0713561

If these theta values ​​allow me to draw a straight line of data, if so how? I've just started trying to understand gradient descent, so please point out any errors in my logic.

source:

x = [10
    20
    30
    40
    50
    60
    70
    80
    90
    100]
y = [4
    7
    8
    4
    5
    6
    7
    5
    3
    4]

m = length(y)

x = [ones(m , 1) , x]

theta = zeros(2, 1);        

iterations = 300;
alpha = 0.000007;

for iter = 1:iterations
     theta = theta - ((1/m) * ((x * theta) - y)' * x)' * alpha;
     theta
end

plot(x, y, 'o');
ylabel('Response Time')
xlabel('Time since 0')

Update:

So the product for each x value multiplied by theta produces a straight line:

plot(x(:,2), x*theta, '-')

enter image description here

Update 2:

How does this relate to the linear regression model:

enter image description here

Because the model also outputs the prediction value?

+3


source to share


1 answer


Yes, you should be able to draw a straight line. In regression, gradient descent is an algorithm used to minimize the cost (error) of a function of your linear regression model. You use the gradient as a path to navigate to the minimum of your cost function, and the learning rate determines how fast you move along the path. Too fast and you could pass the global minimum. When you hit your desired minimum, plug these theta values ​​into your model to get an estimated model. In the one-dimensional case, this is a straight line.



Check out this article for a good introduction to gradient descent.

+3


source






More articles:


All Articles