Deep Learning:Pushing the New Fr

刘铁岩

• 微软亚研院副院长，首席研究员

Key Technical Area

• Computer Vision
  ImageNet 11-12年深度学习
  ResNet(Residual Network)技术
• Speech
  语音识别
  16年年底，微软把speech recognition word error rate 降低到5.1%（Magic Number，人类的错误率）
• Natural Language
  机器翻译水准尚还低于人类，但距离不远
  翻译准确率的量化？N-gram可以粗糙地衡量
  业内认为一年后，可以超过同声传译的专家
• Games
  Alphago

Key Industries

Security

公安领域，交通领域
技术：人体分析，车辆分析，行为分析
Industry Trend
资本流向技术
博鳌亚洲论坛——Face++安保

Autonomous Drive

Google,Baidu,Mobileye,Tesla,Benz,BMW
最主要的问题：复杂路况，道德，法律条款
无人驾驶的车撞人的责任？
Industry Trend
Baidu:阿波罗计划
Google:200w mile路测数据
Mobileye:3000wkm路测数据
Tesla:16年与Mobileye停止合作

Healthcare

最数字化（早已经是计算机辅助的技术，血常规，CT..）

1. 基于大数据（CT，核磁共振）的辅助诊断系统
2. 医疗知识图谱
3. 智能医疗顾问
4. 基因工程
5. 制药、免疫

Deep Learning

An end-to-end learning approach that uses a highly complex model(nonlinear,multi-layer) to fit the training data from scratch.
做Genomics不需要先学几年生物
LightGBM
速度快于XGBoost

Basic Machine Learning Concepts

• The goal:To learn a model from experiences/data
  Training data
  model
• Test/inference/prediction
• Validation sets for hyperparameter tuning
• Training:empirical loss minimization
  Loss Function L
  1.Linear regression
  2.SVM
  3.Maximum likelihood

Biological Motivation and Connections

Dendrite 树突
Synapse 突触
Axon 轴突，输出信号

Perceptron

Feedforward Neural Networks

有界连续函数可以被深度神经网络完美逼近（要有隐层）Universal Approximation Theorem
Hidden Unites: Sigmoid and Tangenth
Sigmoid: f(x)=1/(1+e^(-x))
Rectified Linear Units

Loss Function

交叉商

Gradient Descent

GD肯定可以收敛，计算量很大
SGD(随机梯度下降法），过程快很多，是对整体的无偏估计
SGD也有问题：可能方差非常大，掩盖收敛过程的小的抖动，不能保证收敛性
定义一个Learning Rate，平方阶求和收敛
实际上使用的是折中的办法——Minibatch SGD
以上的都是基本方法
现在用了很多技巧和改进
比如Momentum SGD,Nesterov Momentum
AdaGrad
Adam

Regularization for deep learning

Overfitting
Generalization gap
DropOut:Prevents units from co-adapting too much
Batch Normalization:The distribution of each layer's inputs changes during training带参数的归一化
Weight decay(or L^2 parameter norm penalty)
Early Stopping

Convolutional neural networks

局部连接
模拟人的模式识别的过程
卷积核：SGD学出来
Pooling:Reduce dimension
An example:VGG

• Gradient Vanishing
  深层神经网络，梯度求不出来
  Sigmoid求导数小于等于0.5,深层求导相乘，会变得很小
  解决：Residual Network(ResNet)
• What's Missing?
  Feedforward network and CNN
  However, many applications involve sequences with variable lengths

Recurrent Neural Networks(RNN)

We can process a sequence of vectors x by applying a recurrence formula at every time step
记忆上一层的输入

• Many to One:输入序列，输出单一标量
• One to Many:输入单一向量，输出序列（例如：看图写话）
• Many to many:Language Modeling （联想下一个词）.Encoder-Decoder for Sequence Generation.

同样的问题：网络过长
解决：Long Short Term Memory

Deep learning toolkits

1. Tensorflow(Google)
2. Caffe(UC Berkeley)
3. CNTK(Microsoft)
4. MAXNET(Amazon)
5. Torch7(NYU/Facebook)
6. Theano(U Mnotreal)

图像分类：Caffe Torch
文本：Theano
大规模：CNTK
丰富性：Tensorflow

Advanced topics in deep learning

Challenging of deep learning

1. Relying on Big Training Data
2. Relying on Big Computation
3. Modify Coefficients
4. Lack of interpretability
   黑盒子？白盒子？
5. Lack of Diverse Tech Roadmaps
   NIPS，ICML越来越多的论文是Deep Leaning
6. Overlooking Differences between Animal and Human
   解决的是函数拟合问题，离真正的智能还很远

Dual learning

• A New View:The Beauty of Symmetry
  Dual Learning from with 10% bilingual data (2016 NIPS)

Lightweight deep learning

Light RNN

Distributed deep learning

Convex Problems
Universal Approximation Theorem只是存在性命题