Video Gallery

What is a Neural Network Deep Learning

Go to Video Gallery Added Nov 25, 2020 •

Copy this URL:
Embed code:	<iframe class="edlio-embed-player" type="text/html" width="640" height="360" src="//www.thecije.org/apps/embed/?v=301570" frameborder="0" allowfullscreen></iframe> Change dimensions

Go HD

Show Transcript

this is a 3 it's sloppily written that rendered at an extremely low resolution of 28 by 28 pixels a moment to appreciate how crazy it is the brains can do this so effortlessly I mean this this in this are also recognizable as threes even though the specific values of each pixel is very different from one image to the next the particularly sensitive cells in your eye that are firing when you see this three are very different from the ones firing when you see this tree but something in that crazy smart visual cortex of yours results as representing the same idea why was the same recognizing other images as their own distinct ideas but if I told you hey sit down and write for me a program that takes in a grid of 28 by 28 pixels like this and outputs a single number between 0 and 10 telling you what it thinks the digit is all the test goes from, Klee trivial to dauntingly difficult unless you've been living under a rock I think I hardly need to motivate the relevance and importance of machine learning and neural networks to the present into the future what I want to do here is show you what a neural network actually is assuming no background and to help visualize what it's doing not as a buzzword but as a piece of math my hope is just that you come away feeling like the structure itself is motivated and if you like you know what it means when you read or you hear about a neural network quote-unquote learning this video is just going to be devoted to the structure component of that the following is going to tackle learning what we're going to do is put together a neural network that can learn to recognize handwritten digits this is a somewhat classic example for introducing the topic stick with the status quo here because at the end of the two videos I want to point you to a couple good resources where you can learn more and where you can download the code that does this and play with it on your own computer there are many many variants of neural networks and in recent years that's been sort of a boom in research towards these variants but in these to introductory videos now you're just going to look at the simplest plain-vanilla form with no added Frills this is kind of a necessary prerequisite for understanding any of the more powerful modern variance and trust me it still has plenty of complex defrost to wrap their minds around they can learn to recognize handwritten digits which is a pretty cool thing for a computer to be able to do and at the same time you'll see how it does. Have a couple hopes that we might have for it as the name suggests networks are inspired by the brain but let's break that down what are the neurons and in what sense are they linked together right now when I say it in Iran all I want you to think about is a thing that holds the number specifically a number between 0 white one it's really not more than that for example the network starts with a bunch of neurons corresponding to each of the 28 x 28 pixels of the input image which is 784 neurons in total please hold the number that represents the grayscale value of the corresponding pixel ranging from 04 black pixels up to 1/4 white pixels this number inside the neuron is called its activation in the Imaging mind here is that each neuron is lit up when its activation is a high number to all of these 784 neurons make up the first layer of our Network no jumping over to the last layer this has tender on each representing one of the digits the activation in these neurons again some number that's between 0 and 1 represents how much the system thinks that a given corresponds with a given digit there's also a couple layers in between called The Hidden layers which for the time being to just be a giant? For Honor this process of recognizing digits is going to be handled in this network choose to Hidden layers each one with 16 neurons and admittedly that's kind of an arbitrary choice to be honest I chose two layers based on how I want to motivate the structure and just a moment and 16th all that was just a nice number to fit on the screen and practice there is a lot room for experiment with a specific structure here the way the network operates activations in one layer determine the activation of the next layer and of course the heart of the network as an information processing mechanism comes down to exactly how those activations from one layer bring about activations in the next later it's meant to be analogous to how and biological networks of neurons some groups of neurons firing cause certain others to fire another Network I'm showing here has already trains to recognize digits then let me show you what I mean by that it means if you feed an image lighting up all 784 neurons of the input layer according to the brightness of each pixel in the image that pattern of activation some very specific pattern in the next layer which causes some pattern in the one after it which finally gives some pattern in the output layer and the brightest neuron of that output layer is the Network's choice so to speak for what digit this image represents before jumping into the math for how one layer influences the next door how training Works let's just talk about why it's even reasonable to expect a layered structure like this to behave intelligently what we expect what is the best hope for what those middle layers might be doing when you were I recognize digits we piece together various components and 9 has a loop up top and a line on the right and also has a loop up top but it's pair another loop download a for basically breaks down into three specific lines and things like that on a perfect world we might hope that each neuron in the second to last letter corresponds with one of these subcomponents that anytime you feed an image with save a loop up top like a 9 or 1/8 there's some specific neuron whose activation is going to be close to 1 and I don't mean this specific group of pixels hope would be the any generally loopy pattern towards the set off this neuron that way going from the third layer to the last one just requires learning which combination of subcomponents corresponds to which digits of course that's just kick the problem down the road because how would you recognize this up golden Spore even learn what the rate subcomponents should be and I still haven't even talked about how one layer influences the next but run with me on this one for a moment recognizing a loop can also break down into subproblems one reasonable way to do this would be the first record the various little edges that make it up similarly a long lines like the kind you might see in the digits 1 or 470 that's really just a long Edge or maybe you think of it as a certain pattern of several smaller edges so maybe our hope is that each neuron in the second layer of the network corresponds with the various relevant little edges maybe when an image like this one comes in it lights up all of the neurons associated with around 210 specific little edges which in turn lights up the neurons associated with the upper Loop and a long vertical line and those light up the neuron associated with the nine whether or not this is what I find network actually does this is another question 1 I'll come back to once we see how to train the network but this is a hope that we might have a sort of goal with the layered structure like this you can imagine how being able to detect edges then patterns like this would be really useful for other image recognition I need to Beyond image recognition there are all sorts of intelligent things you might want to do that break down into layers of abstraction person speech for example involves taking raw audio and picking out distinct sounds which combine to make certain syllable which combine to form which combine to make up freezes and more abstract thoughts to Tetra but getting back to how any of this actually works picture yourself right now designing how exactly the activations in one layer might determine the activation next the goal was to have some mechanism that could conceivably combine pixels into edges or edges into patterns or patterns into digits and zoom in on one very specific example let's see the hope is for one particular neuron in the second layer to pick up on whether or not the image has an edge in this region hear the question at hand is what parameters should the network have what dials and lobster you be able to tweak so that it's expressive and the potentially captured this pattern or any other pixel pattern or the pattern that several edges to make a loop and other such things is a sign a weight to each one of the connections between our neuron and the neurons from 1st Lair these weights are just numbers then take all of those activations from the first layer and compute their weighted some according to these weights I find it helpful to think of these weights is being organ I think we're little grid of their own and I'm going to use green pixels to indicate positive weights and red pixel to indicate negative weights where the brightness of that pixel is some loose depiction of the weights value if we meet the weights associated with almost the pixel 0 except for some positive weights in this region that we care about them taking the weighted sum of all the pixel values really just announce to adding up the values of the pixel just in the region that we care about and if you really want pick up on whether there's an edge what you might do is have some negative weights associated with the surrounding pixels then the sum is largest when those middle pixels are bright but the surrounding pixels are darker cute awaited something like this you might come out with any number but for this network what we want is for activations to be some value between 0 and 1/2 a common thing to do is to pump this weighted some into some function that squishes the real number line into the range between 0 and 1 and a common function that does this is called the sigmoid function also known as the alergista curve basically very negative inputs into posted zero very positive inputs in the post one and it just deadly creases around the input 0 to the activation of the neuron here is basically a measure of how positive the relevant weighted sum is maybe it's not that you want the Neurontin set up when the weighted sum is bigger than zero maybe you only wanted to be active when the sun is bigger than say 10 that is you want some bias for it to be inactive what we'll do then is just adding some other number like -10 to this weighted some before plugging it through the sigmoid specifications function that additional number is called the bias so the weights tell you what pixel pattern this neuron in the second layer is picking up on and the bias tells you how high the weighted sum needs to be before the neuron starts getting meaningfully active and just wonder on every other neuron in this layer is going to be connected to all 784 pixel neurons from the first each one of those 784 connections has its own weight associated with it also each one has some bias some other number that you add on to the weighted sound before squishing it with the sigmoid and that's a lot to think about what does hidden layer of 16 neurons that's a total of 784 * 16 wait along with 16 biases and all the connections from the first layer to the second the connections between the other layers also have bunch of weights and biases associated with them all said and done this network has almost exactly 13 thousand total weights and biases 13,000 knobs and dials that can be tweaked and turn to make this network behave in different ways when we talk about learning what that's referring to is getting the computer to find a valid setting for all of these men many numbers so that it'll actually solve the problem at hand one thought experiment that is at once fun kind of horrifying it to imagine sitting down and sending all of these weights and biases by hand purposely tweaking the numbers so that the second layer picks up on edges the third layer picks up on patterns Etc I personally find the satisfying rather than just read the network is a total Black Box because when the network doesn't perform the way you anticipate if you've built up a little bit of a relationship with what those weights and biases actually mean you have a starting place for experimenting with how to change the structure to improve when the network does work but not for the reasons you might expect digging into what the weights and biases are doing is a good way to challenge your assumptions and really expose the full space of possible solutions the actual function here is a little cumbersome to write don't you think let me show you a more notational a compact way that these connections are represented this is how you see it if you choose to read that more about neural networks organize all of the activations from one layer into plums as a vector thin organize all of the weights as a matrix where each row of that Matrix corresponds to the connections between one layer and a particular neuron in the next layer what that means is that it weighted sum of the activations in the first layer according to these weights corresponds to one of the terms in the Matrix Vector product of everything we have on the left here Play the song machine learning just comes down to having a good grasp of linear algebra so for any of you who want a nice visual understanding for matrices and Matrix Vector multiplication means take a look at the series I did on linear algebra especially chapter 3 instead of talking about adding the bias to each one of these values independently we represented by organizing all those biases into a vector and adding the entire Vector to the previous Matrix Vector product then the final step a sigmoid around the outside here and what that's supposed to represent is that you're going to apply the sigmoid function to each specific component of the resulting Vector inside once you write down this week Matrix and these vectors as their own you can communicate the full transition of activations from one layer to the next in an extremely tight and neat little expression and this makes the relevant code both a lot simpler and a lot faster since many libraries optimize the heck out of nature multiplication remember how earlier I said these neurons are simply things that hold numbers of course the specific numbers that they hold depends on the image you feed in it's actually more accurate to thing you turn around as a function when it takes in the outputs of all the neurons in the previous layer and spits out a number between 0 and 1 really the entire network is just a function when it takes in 784 numbers as an input and spits out 10 numbers as an output it's an absurdly complicated function one that involves 13,000 parameters in the forms of these weights and biases that pick up on certain pattern and which involves iterating many Matrix Vector products in the sigmoid switch occasion but it's just a function nonetheless and in a way it's kind of reassuring that it looks complicated I mean if it were any simpler what would we have that it could take on the challenge of recognizing digits does it take on that challenge how does this network learn the appropriate weights and biases just by looking at data that's what I'll show in the next video and I'm also dig a little more into what this particular Network we're seeing is really doing that was the point I suppose I should say subscribe to stay go to find about win that video or any new videos come out but realistically most of you don't actually receive notifications from YouTube do you maybe more honestly I should say subscribe so that the neural networks that underlie YouTube's recommendation algorithm are primed to believe you want to see content from this channel get recommended to you anyway they posted for more thank you very much to everyone supporting these videos on patreon I've been a little slow to progress in the probability Series this summer but I'm jumping back into it after this project so big you can look out for updates there pickles things off here I have with me Lisa Lee who did her PhD work on the theoretical side of deep learning and who currently works at a venture capital firm called amplify Partners who kindly provided some of the funding for this video to Leisha one thing I think we should quickly bring up is the sigmoid function as I understand it early networks use this to squish the relevant weighted sum into that interval between 0 and 1 you know kind of motivated by this biological analogy of neurons either being inactive or active relatively few modern networks actually use sigmoid anymore it's kind of old school easier to train their unit Nike Air Max zero and a where a is given by what you were explaining in the video and partially by a biological algae with how neurons would either be debated or not and so if it passes a certain threshold it would be the identity function but if it did not then it would just not be activated to be zero so it's kind of a simplification using Sigma training or it was very difficult to train at so quiet and people just tried to call you and it happened to work very well for these incredibly deep neural networks amplify Partners in early-stage VC sing technical Founders building the next generation of companies focused on the applications of AI if you or someone that you know has ever thought about starting a company someday or if you're working on an early stage one right now yeah but would love to hear from you leave and set up a specific mail for this video three blue one Brown at Amplified partners.com so feel free to reach out to them to that

Video Gallery

What is a Neural Network Deep Learning

Popular Videos