Linear Regression & Logistic Reg

2018-10-18 本文已影响0人 asuka_19d5

Expected Value
$\mu = E(X) = \sum^n_{i=1}x_ip_i$
Variance
$\sigma^2 = Var(X) = \sum^n_{i=1}(x_i - \mu)^2p_i$
Here X means random variable

Bernoulli Distribution (two-point)
$P(X = x) = p^x(1 - p)^{1 - x}, 0 <p <1, x = 0, 1$
p 为 x=1 的概率
Binomial Distribution
n independent trials
$P(X= x) = C^x_np^x(1 - p)^{n-x}, x = 1, 1, 2, ..., n$
p 为试验成功（1）的概率，x为成功次数
P 描述在n次独立的实验中成功x次的概率

probability density function (PDF)
- the probability is given by the integral
- the entire space is equal to one
  
  image
Expected value
$\mu = E(X) = \int^\infty_{-\infty}xf(x)dx$
Variance
$\sigma^2 = Var(X) = \int^\infty_{-\infty}[x - E(X)]^2f(x)dx$
Normal Distribution (PDF)
- given $\mu$ and $\sigma$ , the form is determined

exmaple: use random(5) to get random(25)
answer 5*random(5) + random(5) （quinary 五进制）
Background:
If a random variable reflects a large number of independent random factors, and each single factor does not play a significant role in the intermediate stage of the comprehensive influence, the random variable generally follows the normal distribution. 若一个随机变量反应了大量相互独立的随机因素综合影响，而每一个单独因素在综合影响中期的作用不显著，则这种随机变量一般都服从正态分布。
analysis:
- the limit of sum of independent variables with the same distribution is normal distribution
- because the sum can approch $\infty$ , we consider their normalized forms. Then it is a normal distribution
no matter these factors have any forms of distribution, when n is very large, the distribution of their sum and the sampling distribution are close to normal distribution

Least Square: (argument mini) 从距离的角度建立目标函数 $argmin_{\beta}\sum_{i=1}^{n}\epsilon^2_i = \sum_{i=1}^{n}(y_i - \beta_0 - \sum_{i=1}^{m}\beta_jx_{ji})^2$
Maximum Likelihood Estimation (MLE): 从概率的角度建立目标函数
- Definition: find the parameter values that maximize the likelihood of making the observations given the parameters
- assumption for linear regression:
  p(y|x) is a Gaussian distrubution with $mean = \mu = ax + b, variance = \sigma$
  $p(y) = \frac{1}{\sigma \sqrt{2\pi}} e^{-\frac{1}{2\sigma^2}(y- \mu)^2}, -\infty < y < \infty$ $p(y_1, y_2, ... , y_n|\sigma, \mu) = p(y_1|\sigma, \mu)(y_2|\sigma, \mu) ... p(y_n|\sigma, \mu)$ use $\theta$ to denote $(\sigma, \mu)$ , we have $\theta_{ML}(Y) = argmax_{\theta}p(Y|\theta)$
- For a sample Yi, its PDF: $P(Y_i) = \frac{1}{\sigma \sqrt{2\pi}} e^{-\frac{1}{2\sigma^2}(Y_i - \beta_0 -\beta_1X_i)^2}$
  Because all the Yi are independent, the MLA function becomes: $L(\beta_0, \beta_1, \sigma^2) = P(Y_1, Y_2, ... ,Y_n) = \frac{1}{\sigma \sqrt{2\pi}} e^{-\frac{1}{2\sigma^2}\sum(Y_i - \beta_0 -\beta_1X_i)^2}$ $lnL = -nln(\sqrt{2\pi}\sigma) - \frac{1}{2\sigma^2}\sum_{i=1}^n(Y_i - \beta_0 -\beta_1X_i)^2$ $max L -> min \sum_{i=1}^{n}\epsilon^2_i = \sum_{i=1}^{n}(y_i - \beta_0 - \sum_{i=1}^{m}\beta_jx_{ji})^2$

How to create a function to fit discrete values?
step 1: Discrete Y -> Continuous y(p)
step 2: Continuous y(p) -> X
then we can get X <-> Y
p -> [0, 1]
odds = p/(1-p) -> [0, +infty]
log(odds) -> [-infty, +infty]
thus, x= log(odds) <-> p

assumption for logistic regression:
p(y|x) is a Bernoulli distrubution with $P(Y =y | X) = \frac{1}{1+ e^{-ax-b}}$
Step 1: Choose Model $P(Y = y_i) = p^{y_i}(1-p)^{1-y_i}, 0 < p < 1, y_i = 0, 1$ $p = h_{\beta}(x_i) = \frac{1}{1+e^{-\beta_0-\beta_1x}}$
Step 2: Calculate loss function - MLE
$L(\beta_0, \beta_1) = P(Y_1, Y_2, ... , Y_n) = \Pi_{i=1}^n h_{\beta}(x_i)^{y_i}(1 - h_{\beta}(x_i))^{(1-y_i)}, y_i = 0, 1$ $min log(P) = \sum^n_{i=1}[ - y_ilog(h_{\beta}(x_i)) - (1 - y_i)log(1 - h_\beta(x_i))]$