image.png

Recommended book: Mathematics for Machine learning (free online): https://mml-book.github.io/

image.png image.png image.png image.png

Simple linear regression

Simple linear regression is a simple term.

Loss function:

Baby linear regression

Consider slope only model:

The loss:

image.png

How to find .

Baby gradient descent

image.png

Have a initial guess, based on the derivative. We can guess which direction to go.

Update rule:

Here is the learning rate.

It's tricky to find global minimum for non-convex function.

If you learning rate is too large:

Baby gradient descent cont.

We need to compute the derivative of the loss:

We get:

Baby analytical solution

At the minimum:

We obtain:

Back to simple linear regression

Introducing partial derivatives

You first fix .

Update rules for each parameter:

In vector form:

Compute the gradient

We need partial derivatives:

Reference

https://www.youtube.com/watch?v=lWGdFeMsjzg&feature=youtu.be