交叉熵数损失推导

2022-07-07 本文已影响0人 leon0514

交叉熵数损失推导

伯努利分布

伯努利试验说的是下面一种事件情况：在生活中，有一些事件的发生只有两种可能，发生或者不发生（或者叫成功或者失败），这些事件都可以被称为伯努利试验。

那么其概率分布称为伯努利分布（两点分布、0-1分布），如果记成功概率为 $p$ ，则失败概率为 $1-p$ ，则认为概率质量函数为：

$P(X=y|x) = p^y(1-p)^{1-y} = \begin{cases} p & y = 1 \\ 1-p & y = 0 \end{cases}$

似然估计 - 使用sigmiod函数计算概率

$L(\theta) = \prod_{i=0}^n{P(y^{(i)}|x^{(i)};\theta)} = \prod_{i=0}^n{h_\theta(x^{(i)})^{y^{(i)}}}*(1-h_\theta(x^{(i)}))^{1-y^{(i)}}$

对数似然估计 - 取对数

$ln(L(\theta)) = \sum_{i=0}^n{ln(h_\theta(x^{(i)})^{y^{(i)}}) + ln((1-h_\theta(x^{(i)}))^{1-y^{(i)}})} = \sum_{i=0}^n{y^{(i)}ln(h_\theta(x^{(i)})) + (1-y^{(i)})*ln((1-h_\theta(x^{(i)})))}$

多元线性函数

$g_\theta(x) = \theta_0x_0 + \theta_1x_1 + \theta_2x_2 + ... + \theta_nx_n,(x_0 = 0) \\ g_\theta(x) = \theta^Tx$

Sigmoid函数

$h_\theta(x) = \frac{1}{1+e^{-g_\theta(x)}}$

交叉熵损失

$L(\theta) = -\sum_{i=0}^n{y^{(i)}ln(h_\theta(x^{(i)})) + (1-y^{(i)})*ln((1-h_\theta(x^{(i)})))}$

化简得

$L(\theta) = -\sum_{i=0}^n{y^{(i)}ln(\frac{1}{1+e^{-g_\theta(x^{(i)})}}) + (1-y^{(i)})*ln((1-\frac{1}{1+e^{-g_\theta(x^{(i)})}}))} \\ = \sum_{i=0}^n{y^{(i)}ln(1+e^{-g_\theta(x^{(i)})}) - (1-y^{(i)})*(ln(\frac{e^{-g_\theta(x^{(i)})}}{1+e^{-g_\theta(x^{(i)})}})}) \\ = \sum_{i=0}^n{y^{(i)}ln(1+e^{-g_\theta(x^{(i)})}) - (1-y^{(i)})*(-g_\theta(x^{(i)}) - ln(1+e^{-g_\theta(x^{(i)})}})) \\ = \sum_{i=0}^n{g_\theta(x^{(i)}) - y^{(i)}*g_\theta(x^{(i)}) + ln(1+e^{-g_\theta(x^{(i)})})} \\ = \sum_{i=0}^n{ln(e^{g_\theta(x^{(i)})}) - y^{(i)}*g_\theta(x^{(i)}) + ln(1+e^{-g_\theta(x^{(i)})})} \\ = \sum_{i=0}^n{ln(e^{g_\theta(x^{(i)})}+1) - y^{(i)}*g_\theta(x^{(i)})}$

对 $\theta_j$ 求导得

$\frac{\partial_L}{\partial_{\theta_j}} = \sum_{i=0}^n\frac{x^{(i)}_j*e^{g_{\theta}(x^{(i)})}} {e^{g_{\theta}(x^{(i)})}+1} - y^{(i)}*x^{(i)}_j \\ = \sum_{i=0}^n{x^{(i)}_j* (\frac{e^{g_{\theta}(x^{(i)})}}{e^{g_{\theta}(x^{(i)})}+1} - y^{(i)})} \\ = \sum_{i=0}^n{x^{(i)}_j*(h_\theta(x^{(i)}) - y^{(i)})}$

交叉熵数损失推导

交叉熵数损失推导

猜你喜欢

热点阅读