Neural Networks and Deep Learnin

if the basic technical idea is behind
deep learning behind your networks have
been around for decades why are they
only just now taking off in this video
let's go over some of the main drivers
behind the rise of deep learning because
I think this will help you that the spot
the best opportunities within your own
organization to apply these to over the
last few years a lot of people have
asked me Andrew why is deep learning
certainly working so well and when a
marsan question this is usually the
picture I draw for them let's say we
plot a figure where on the horizontal
axis we plot the amount of data we have
for a task and let's say on the vertical
axis we plot the performance on above
learning algorithms such as the accuracy
of our spam classifier or our ad click
predictor or the accuracy of our neural
net for figuring out the position of
other calls for our self-driving car it
turns out if you plot the performance of
a traditional learning algorithm like
support vector machine or logistic
regression as a function of the amount
of data you have you might get a curve
that looks like this where the
performance improves for a while as you
add more data but after a while the
performance you know pretty much
plateaus right suppose your horizontal
lines enjoy that very well you know was
it they didn't know what to do with huge
amounts of data and what happened in our
society over the last 10 years maybe is
that for a lot of problems we went from
having a relatively small amount of data
to having you know often a fairly large
amount of data and all of this was
thanks to the digitization of a society
where so much human activity is now in
the digital realm we spend so much time
on the computers on websites on mobile
apps and activities on digital devices
creates data and thanks to the rise of
inexpensive cameras built into our cell
phones accelerometers all sorts of
sensors in the Internet of Things we
also just have been collecting one more
and more data so over the last 20 years
for a lot of applications we just
accumulate
a lot more data more than traditional
learning algorithms were able to
effectively take advantage of and what
new network lead turns out that if you
train a small neural net then this
performance maybe looks like that
if you train a somewhat larger Internet
that's called as a medium-sized internet
to fall in something a little bit better
and if you train a very large neural net
then it's the form and often just keeps
getting better and better so couple
observations one is if you want to hit
this very high level of performance then
you need two things first often you need
to be able to train a big enough neural
network in order to take advantage of
the huge amount of data and second you
need to be out here on the x axes you do
need a lot of data so we often say that
scale has been driving deep learning
progress and by scale I mean both the
size of the neural network we need just
a new network a lot of hidden units a
lot of parameters a lot of connections
as well as scale of the data in fact
today one of the most reliable ways to
get better performance in the neural
network is often to either train a
bigger network or throw more data at it
and that only works up to a point
because eventually you run out of data
or eventually then your network is so
big that it takes too long to train but
just improving scale has actually taken
us a long way in the world of learning
in order to make this diagram a bit more
technically precise and just add a few
more things I wrote the amount of data
on the x-axis technically this is amount
of labeled data where by label data
I mean training examples we have both
the input X and the label Y I went to
introduce a little bit of notation that
we'll use later in this course we're
going to use lowercase alphabet to
denote the size of my training sets or
the number of training examples
this lowercase M so that's the
horizontal axis couple other details to
this Tigger
in this regime of smaller training sets
the relative ordering of the algorithms
is actually not very well defined so if
you don't have a lot of training data is
often up to your skill at hand
engineering features that determines the
foreman so it's quite possible that if
someone training an SVM is more
motivated to hand engineer features and
someone training even large their own
that may be in this small training set
regime the SEM could do better
so you know in this region to the left
of the figure the relative ordering
between gene algorithms is not that well
defined and performance depends much
more on your skill at engine features
and other mobile details of the
algorithms and there's only in this some
big data regime very large training sets
very large M regime in the right that we
more consistently see largely Ronettes
dominating the other approaches and so
if any of your friends ask you why are
known as you know taking off I would
encourage you to draw this picture for
them as well so I will say that in the
early days in their modern rise of deep
learning
it was scaled data and scale of
computation just our ability to Train
very large dinner networks
either on a CPU or GPU that enabled us
to make a lot of progress but
increasingly especially in the last
several years we've seen tremendous
algorithmic innovation as well so I also
don't want to understate that
interestingly many of the algorithmic
innovations have been about trying to
make neural networks run much faster so
as a concrete example one of the huge
breakthroughs in your networks has been
switching from a sigmoid function which
looks like this to a railer function
which we talked about briefly in an
early video that looks like this if you
don't understand the details of one
about the state don't worry about it but
it turns out that one of the problems of
using sigmoid functions and machine
learning is that there these regions
here where the slope of the function
would
gradient is nearly zero and so learning
becomes really slow because when you
implement gradient descent and gradient
is zero the parameters just change very
slowly and so learning is very slow
whereas by changing the what's called
the activation function the neural
network to use this function called the
value function of the rectified linear
unit our elu the gradient is equal to
one for all positive values of input
right and so the gradient is much less
likely to gradually shrink to zero and
the gradient here the slope of this line
is zero on the left but it turns out
that just by switching to the sigmoid
function to the rayleigh function has
made an algorithm called gradient
descent work much faster and so this is
an example of maybe relatively simple
algorithm in Bayesian but ultimately the
impact of this algorithmic innovation
was it really hope computation so the
regimen quite a lot of examples like
this of where we change the algorithm
because it allows that code to run much
faster and this allows us to train
bigger neural networks or to do so the
reason or multi-client even when we have
a large network roam all the data the
other reason that fast computation is
important is that it turns out the
process of training your network this is
very intuitive often you have an idea
for a neural network architecture and so
you implement your idea and code
implementing your idea then lets you run
an experiment which tells you how well
your neural network does and then by
looking at it you go back to change the
details of your new network and then you
go around this circle over and over and
when your new network takes a long time
to Train it just takes a long time to go
around this cycle and there's a huge
difference in your productivity building
effective neural networks when you can
have an idea and try it and see the work
in ten minutes or maybe ammos a day
versus if you've to train your neural
network for a month which sometimes does
happened
because you get a result back you know
in ten minutes or maybe in a day you
should just try a lot more ideas and be
much more likely to discover in your
network and it works well for your
application and so faster computation
has really helped in terms of speeding
up the rate at which you can get an
experimental result back and this has
really helped both practitioners of
neuro networks as well as researchers
working and deep learning iterate much
faster and improve your ideas much
faster and so all this has also been a
huge boon to the entire deep learning
research community which has been
incredible with just you know inventing
new algorithms and making nonstop
progress on that front so these are some
of the forces powering the rise of deep
learning but the good news is that these
forces are still working powerfully to
make deep learning even better Tech Data
society is still throwing up one more
digital data or take computation with
the rise of specialized hardware like
GPUs and faster networking many types of
hardware I'm actually quite confident
that our ability to do very large neural
networks or should a computation point
of view will keep on getting better and
take algorithms relative learning
research communities though continuously
phenomenal at innovating on the
algorithms front so because of this I
think that we can be optimistic answer
the optimistic the deep learning will
keep on getting better for many years to
come
so that let's go on to the last video of
the section where we'll talk a little
bit more about what you learn from this
course