Logistic Regression

2020-12-11 本文已影响0人 Kevin不会创作

Overview
- Logistic function
- Model
- Loss function
- Cost function
- Gradient descent

Overview

Logistic regression is a statistical model that in its basic form uses a logistic function to model a binary dependent variable.

Logistic function

Logistic function is a common example of a sigmoid function.

$S(x)=\frac{1}{1+e^{-x}}=\frac{e^x}{e^x+1}$
Model

$z=wx+b$

$p=sigmoid(z)=\frac{1}{1+e^{-z}}$

$\hat{y}=\begin{cases} 1, & p\geq t \\ 0, & p<t \end{cases}$

where $t$ is the threshold.
Loss function

Cross-entropy is used as the loss function in Logistic Regression.

$L(y,\hat{y})=-(y*log(\hat{y})+(1-y)*log(1-\hat{y}))$
Cost function

Just sum up $L(y,\hat{y})$ you can get the cost function.

$J(w,b)=\frac{1}{m}\sum_1^{m}L(y^{(i)},\hat{y}^{(i)})$
Gradient descent

You can use gradient descent to find the optimum parameters $(w,b)$
1. Initialize $w$ and $b$
2. Compute $dw$ and $db$ using chain rule
  
  $d{w}=\frac{\partial{L}}{\partial{w}}=\frac{\partial{L}}{\partial{\hat{y}}}\cdot\frac{\partial{\hat{y}}}{\partial{z}}\cdot\frac{\partial{z}}{\partial{w}}$
  
  $\begin{split} \frac{\partial{L}}{\partial{\hat{y}}}&=-\frac{\partial(y*log(\hat{y})+(1-y)*log(1-\hat{y}))}{\partial{\hat{y}}}\\ &=-\frac{y}{\hat{y}}+\frac{1-y}{1-\hat{y}} \end{split}$
  
  $\begin{split} \frac{\partial{\hat{y}}}{\partial{z}}=\hat{y}(1-\hat{y}) \end{split}$
  
  $\frac{\partial{z}}{\partial{w}}=x$
  
  So we can obtain
  
  $dw=\frac{\partial{L}}{\partial{w}}=(\hat{y}-y)\cdot x$
  
  Similarly,
$db=\frac{\partial{L}}{\partial{b}}=\hat{y}-y$
1. Update $w$ and $b$
  
  $w=w-\alpha\cdot{dw}$
  
  $b=b-\alpha\cdot{db}$
  
  where $\alpha$ is the learning rate.
2. Repeat step 2&3 until the improvement drops below a threshold or it reaches the maximum number of iterations.

Logistic Regression

Table of Contents

Overview

猜你喜欢

热点阅读