Deep learning 三大神Review

https://www.cs.toronto.edu/~hinton/absps/NatureDeepReview.pdf

本文是Nature杂志为纪念人工智能60周年推出的深度学习综述, 看一下三个作者 , 瑟瑟发抖啊
Hinton(BP算法用于训练DL的始作俑者之一, 17年又提出胶囊算法, 探索AI新的形态),
LeCun(LeNet),
Bengio(learning deep architectrues for Ai的作者, GAN的作者, 开创了DL做NLP的先河 , A neural probabilistic language model, 在DL_NLP上的贡献可以和LeNet比肩)

不夸张的说, 这三位一起造就了2006年始的深度学习的复兴(Review中提到的2006 年, CIFAR的工作)

有些概括的观点直击灵魂, 可以用一下

Deep learning allows computational models that are composed of multiple processing layers to learn representations of data with multiple levels of abstraction
深度学习可以让那些拥有多个处理层的计算模型(神经网络架构)
来学习具有多层次抽象的数据的表示
我理解多层次就意味着数据可以按照抽象程度分解, 比如图像/语音/文本等这些都 可以看作是多层次抽象的数据

Deep learning discovers intricate structure in large data sets by using the backpropagation algorithm to indicate how a machine should change its internal parameters that are used to compute the representation in each layer from the representation in the previous layer
DL能够发现大数据中的复杂结构, 它是利用BP算法完成这个发现过程的, BP算法指 导机器从前一层获取误差而改变本层的内部参数, 最终让目标的cost function最小化
(loss function和cost function
loss function是单个训练样本上的, 就是算一个样本的误差
cost function是整个训练集上的, 就是所有样本误差的平均)

Deep-learning methods are representation-learning methods with multiple levels of representation, obtained by composing simple but non-linear modules that each transform the representation at one level (starting with the raw input) into a representation at a higher, slightly more abstract level.
representation-learning这个词高度概括了DL, DL本质就是一种特征学习方法, 把原始 数据通过一些简单但是非线性的模型转换成更高层次, 更抽象的表达, 在CNN上, 这 就是空间上的信息越来越少, 而深度上的信息越来越多的过程。而DL作为特征学习 方法就是和conventional ML的最大区别, 之前的ML需要大量的先验知识来设计一个 合适当前命题的特征提取器(比如HOG来检测行人, DSIFT来识别人脸, ORB来做空间 匹配)

The hidden layers can be seen as distorting the input in a non-linear way so that categories become linearly separable by the last layer
隐藏层的作用可以看作是使用一个非线性的方式来扭曲输入空间, 使得输入数据对 应的类别在最后一层变得线性可分

For smaller data sets, unsupervised pre-training helps to prevent overfitting40, leading to significantly better generalization when the number of labelled examples is small, or in a transfer setting where we have lots of examples for some ‘source’ tasks but very few for some ‘target’ tasks
对于小的dataset来说, 无监督的预训练可以防止过拟合, 当带有标签的样本很少时 , 这个方法可以带来更好的泛化性能, 这个是大神们在神经网络未兴起时的方法(用 auto-encoder先去预训练参数, 再用带标签的样本训练), 其实DL当然还是数据驱动的 算法, 最后一句也说明了, 如果数据量足够大(比如我们现在已经有了ImageNet了), 我 们当然可以直接用数据来进行监督训练。

All units in a feature map share the same filter bank. Different feature maps in a layer use different filter banks.
这是CNN的基本结构, 就是在一个特征图中的全部单元享用相同的过滤器, 不同层的 特征图使用不同的过滤器, 使用这种结构出于两方面的原因
First, in array data such as images, local groups of values are often highly correlated, forming distinctive local motifs that are easily detected. Second, the local statistics of images and other signals are invariant to location. In other words, if a motif can appear in one part of the image, it could appear anywhere, hence the idea of units at different locations sharing the same weights and detecting the same pattern in different parts of the array
局部值高度的相关性, 所以能够将局部多个像素抽象为相同的局部特征
不同位置局部统计特征的不相关性, 在一个地方出现的某特征, 也可能出现在别的地 方, 所以不同位置的单元可以共享权值

Although the role of the convolutional layer is to detect local conjunctions of features from the previous layer, the role of the pooling layer is to merge semantically similar features into one. Because the relative positions of the features forming a motif can vary somewhat, reliably detecting the motif can be done by coarse-graining the position
of each feature.
Conv层的目的是检测上一层的局部链接, 池化层是要在语义上把相似特征合并起来 , 这是因为形成一个主题的特征的相对位置不太一样, 也就是说在pool的窗口滑到不 同的位置, 形成的主题也会不一样
同时, 池化也保证了CNN一定的鲁棒性, 当输入数据在前一层中的位置有变化时, 池 化通过slide window可以保证输出的主题差别不是太大

---大神对深度学习的未来展望
Human and animal learning is largely unsupervised: we discover the structure of the world by observing it, not by being told the name of every object.
期望无监督学习越来越重要, 因为unsupervised是人类主要的学习方式
Systems combining deep learning and reinforcement learning are in their infancy, but they already outperform passive vision systems at classification tasks and produce impressive results in learning to play many different video games
深度学习和增强学习的结合, 可以用来解决更复杂的视觉问题(比如行车壁障等)