0:21 Hello and welcome to this full course on machine learning. In this course, we will guide you through everything you need to know to get started with machine learning from basic to building real project. So what is machine learning? In simple words, it's a way to teach computer to learn from data and make decisions without being directly programmed for every step. Let's understand this with an example. Meet Ria, a machine learning engineer. One day, her team wanted to find out if customers were happy or upset. Just by reading their review, instead of going through thousands of review manually, Ria built a machine learning model. She collected review data, cleaned it, and trained a model using Python and libraries like pandas and scikitlearn. She used a simple algorithm called logistic regression to help the computer learn what a good or a bad review looks like. Now here's the exciting part. You can do this too. We will teach you the core ideas like supervised and unsupervised learning features and labels and how to choose the right algorithm. We will also use popular tools like Jupyter notebook, numpy and scikitlearn. So without wasting any 1:22 time, let's look at the very first topic on our list. I went through hundreds of courses, watched countless tutorials and compared the most popular learning paths for machine learning. And you know what I found? Everyone says something different. Some tell you to start with Python. Others throw complex math at you right away. It's no wonder so many people feel confused and stuck before they even begin. But here's the truth. You don't need to learn everything. You just need to learn the right things in the right order. And to help make your journey easier, I took a deep dive into how some of the top universities in the world like Stanford, MIT, and Oxford actually teach machine learning. I studied their course structures, the topics they prioritize, and how they build your understanding step by step. Based on that, I've created a simple and well structured road map just for you. This road map will guide you on where to begin, what topics to focus on, and it even includes free resources to help you start learning the smart way. All right, so without wasting any time, let's get started with the very first step in your 2:24 machine learning journey. That is to build a strong foundation in math. And by strong foundation, I don't mean you need to be an expert in every topic. Rather, you just need to be comfortable with a few key areas. Start with linear algebra, which deals with vectors and matrices. These are used in almost every machine learning algorithm. So, understanding how they work is essential. Then move on to calculus. It helps you understand how models learn and improve, especially when it comes to concepts like derivatives and gradients which are used in model optimization. Next, focus on basic statistics. It will help you understand your data, compare results, and measure how well your models are performing. Finally, learn some probability. This is important for understanding how models make predictions, especially in classification problems. If you're looking for resources, there are free courses by MIT on all these topics. But if you want to learn these concepts in one place and in a simpler manner, I recommend the 6our statistics for data science full course by Intellipath. This free course will equip you with all the 3:24 fundamentals of stats needed for machine learning algorithms. Step two, learn programming with Python. After building your math foundation, the next step is to learn programming. And Python is the best place to start. Python is simple, beginner friendly, and widely used in data science and machine learning. Its clean syntax makes it easy to learn even if you're new to coding. So you can begin with the basics like variables, loops, functions, and object-oriented programming. These are the building blocks that will help you write clear and efficient code. Once you're comfortable with Python fundamentals, start learning the most important libraries for data science and machine learning. First is NumPy. This helps you work with numbers and arrays. Then comes Pandas, which is great for organizing and analyzing data. Mathlib and Seaborn help you create graphs and charts to visualize your data. To get started, Google offers a free Python class that's perfect for beginners. And if you want to learn the libraries I mentioned, I have created full courses on numpai, pandas, mattplot liib, and seabon. You can find them directly on YouTube or on 4:24 Intellipath's channel. Step three, get started with supervised learning. Once you're comfortable with Python, it's time to step into the world of machine learning. Starting with supervised learning. This is the most common type of machine learning where the model learns from label data. In simple terms, the data you feed into the model already has the answers and the model learns to predict those answers for new data. You'll start with basic algorithms like linear regression and logistic regression. These help you solve simple prediction problems like predicting a number or classifying data into categories. After that, you learn about regularization techniques such as L1 or lasso and L2 or ridge which are used to avoid overfitting and improve your model's performance. Another important part of this step is model selection. You'll need to understand how to balance bias and variance to choose the best performing model for your data. And then you also need to learn how to evaluate your model. Evaluation can be done on some key metrics like mean squared error, MSE, which is for regression 5:25 problems and accuracy, precision, recall, and F1 score for classification problems. These metrics tell you how well your model is doing and help you compare different models effectively. to learn these topics. I'm going to tell you guys about the free resource after the next step of our road map. So, step four, explore advanced supervised learning techniques. Once you're confident with the basics, it's time to move one step further and explore some advanced machine learning algorithms. These models are more powerful and can handle complex problems better. Start with Nive Base, a simple but effective algorithm that's great for working with text data like detecting spam emails or classifying news articles. It's fast, easy to understand, and works surprisingly well in many real world tasks. Next, learn about support vector machines or SVMs. These models are especially useful when you're working with data that has many features. These models are especially useful when you're working with data that has many features. SVMs try to find the best possible boundary between different classes, which makes them very effective 6:26 for classification tasks. You'll also come across decision trees, which are known for being easy to interpret. They work like a flowchart making decisions based on conditions in the data. While simple on their own, they become much more powerful when combined in ensemble methods like random forest and gradient boosting. These techniques combine multiple decision trees to improve accuracy and reduce errors. You can check out UNMBC supervised learning course on edex. Here every topic is explained with clear visuals and simple language making even the hardest ideas easy to grasp. Step five, understand unsupervised learning. Unlike supervised learning where models learn from labeled data, unsupervised learning is all about finding hidden patterns in unlabelled data. This means the model tries to make sense of the data without being told what the correct output should be. It's useful for tasks like clustering, customer segmentation, and simplifying large data sets. A good place to start is with clustering algorithms. Begin with K means clustering which groups similar data points together based on 7:28 their features. Then explore hierarchal clustering which builds clusters step by step either by merging or splitting groups based on how close they are to each other. After clustering, move on to dimensionality reduction techniques. These methods help you reduce the number of features in your data while keeping the most important information. Start with principal component analysis or PCA and factor analysis. both of which are commonly used to simplify complex data sets and improve model performance. You'll also come across more advanced tools like gshian mixture models, GMM and the expectation maximization algorithm. These are useful when you want your model to work with probabilities and uncertaintity. To dive into these topics, you can check out Stanford CS229 lecture series. Or if you like UNMBC's course on supervised learning, they also have a similarly structured course on unsupervised learning. Step six, dive into neural networks and deep learning. This is where machine learning gets really exciting. Neural networks are the backbone of modern AI and deep learning models are capable of solving problems 8:29 that were impossible just a few years ago. Start by learning how basic neural networks work. Understand perceptrons, activation functions, and the back propagation algorithm. Once you're confident, move into deep learning concepts like CNN's for image task and RNN's for sequence based data. You can also explore autoenccoders for representation learning and more advanced models like retrieval augmented generation or RA which are used in modern language models. Some amazing resources here include IBM's deep learning with TensorFlow and KAS course. There's also a free book called Deep Learning with PyTorch if you want a more technical deep dive. Step seven, explore advanced topics in AI. At this stage, you can branch out into specialized domains within AI. If you enjoy working with language, natural language processing might be the perfect fit. You learn about tokenization, sequence modeling, and state-of-the-art models like BRT and transformers. If you're interested in decision-m and robotics, reinforcement learning is another 9:30 exciting area. You'll explore mark of decision processes, Q-learning, and policy gradients. Basically, teaching agents how to learn through experience. You can also look into learning theory to understand what makes models generalize well. And finally, it's important to learn about AI ethics. How to build responsible models that are fair, interpretable, and aligned with human values. Stanford CS24N for NLP, David Silver's reinforcement learning lectures on YouTube, and Mosilla's ethics and AI initiatives are all great free resources to guide you here. Step eight, work on real projects. The best way to learn is by building. Start with simple regression and classification projects using open data sets from platforms like Kaggel or the UCI machine learning repository. As your confidence grows, start working on more complex projects like image classifiers, sentiment analysis tools, or even reinforcement learning agents that can play games. Document everything on GitHub. Share your work on LinkedIn and maybe even write blogs about what you've learned. This not only solidifies your 10:30 understanding, but it also helps you stand out when applying for jobs. So what we going to do is we are going to um start with machine learning using Python. All right, let's start. Okay. So what is machine learning? Let's start by saying that you know machine learning is all around you. When I say why it's why it's all around you, you are already seeing machine learning here and there around you. Let me give you one example. So we have this uh everyone must have bought from Amazon or Flipkart or any e-commerce website right so we would have bought from there so how does how does Amazon uh implements machine learning so imagine if I am going today and trying to buy uh you know a phone there let's say I I'm a big fan of uh OnePlus and also Apple uh So what I'm trying to do 11:32 is I'm trying to buy a phone. Okay. So what do I do? I go to Amazon. I search different variants. I search different things and then I'm like okay I am going to purchase this 64GB this this this and this is in my budget. This is my phone. What does when you go to the uh when you're looking at the product page, don't you see a small thing there saying that you know there is uh what do you say there is something that you would like to buy a phone cover charger probably some accessory that comes with mobile right do you see that let me try and show you that let's go to Amazon 12:23 So when I Okay, I'll have to login. I don't don't want to log in. Let's say if I search for owns. 12:42 Okay, let it load. Okay, let's say this. If I if I go and click on one. Okay, they'll come and tell you see more products details here and you'll also have frequently bought together. Right, this is around you. What are they saying? You are purchasing a phone. You would also like to purchase a cover, would you? Right. And then they are also recommending that related products with this everything. So, so you see how how Amazon is recommending this is machine learning. This is machine learning. Let me close this. So, when Amazon tells you that you know you are purchasing a phone, you would also need a uh cover. Then when once you put the cover there the next thing what they are going to tell that you would need a screen guard right 13:45 that's what so that is a type of machine learning that is around you okay that is a type of machine learning that almost everyone must have experienced here and there and in some e-commerce website be it Amazon be it anywhere else you must have definitely uh seen this okay and then let me give give you another example how uh you guys are already implementing also u this uh machine learning. Okay, let me come down. Okay, let's try with this. Okay, so let's say you are you you are uh what do you say uh a you know a player. Do you like sports? So let's say you are watching um football. You're watching a football match and what is the what is the goal? The goal is that you need to put something 14:47 in into the net. This is the ball and this is you. Trust me I am a you know I born pretty late otherwise I would have given competition to MF Hussein and you know people like Dawaini. So trust me my drawing is absolutely best. [Music] So uh the thing is that we need to what do you want to do? You want to hit the goal right? This is where it belongs. But what happens you have just started you know playing football. What you do is you first try to hit it here and the ball goes here. That's your first aim. Okay. Now what do you contemplate? What do you see? Okay. I have you know pushed something to very right. I need to put it somewhere here. This is the gap that I see. I need to put the ball somewhere here so that it goes inside the goal. Then this is what you try. This is the second attempt and you score a goal. You score a goal. 15:49 So what happens is what is this that you have seen? What is this? This is nothing but this is what you have learned. You have learned this. This is your assessment that you know you need to be this much closer to the goal. Your ball should hit not outside the goalpost. You need to put it towards the left so that it goes inside the goal. So what you're doing here is essentially you're learning. This is how machine learning also works. So rather than you learning this is how you're implementing machine learning in your real life right now not even playing not even while playing you know we also watch matches so a cricket match there was a world cup right now right so we were all guessing yeah we need to take you know wicket of class so that we win the world cup and everyone was like sitting there and saying that you know we were waiting anxiously that you know when goes out 16:51 people they're putting the ball really in front so that he's hitting sixes and fours. Why you're giving bowling to you know spinner he's hitting them really well feeling. So that is where we are judging and we are learning from a mistakes that the last ball he hit was a six and it was right in front for there to be hit. So the baller also learns. We are also saying that it shouldn't be there. And how are we learning? We are learning from our mistakes. That's key key part here. You know we are learning from our mistakes. Okay. So when we are learning from our mistakes what are we learning? We are learning the delta. We are learning that we have made a mistake and we have not reached the right task in goal also. And when we were getting belted by class N and everyone from the South African team in the final we were also missing that right. Key thing is that we are learning from mistakes. We are learning from our 17:53 past experiences. Okay. So that's how we were learning. Now the other thing that that keep this in mind. Okay. Keep this in mind that how are we learning? The other example is Alexa. So have you have you seen Alexa or have you ever seen Siri or Kotana from Microsoft? What do they do when a person speaks? When a person speaks, Alexa automatically Yeah, exactly an it follows our instructions, right? It follows our instructions. I say, "Alexa, wake me up at 7:00 a.m. tomorrow. It will immediately go there and you know put an alarm of 7:00 and at the morning 7:00 I don't get up. It will it will not stop till you say that you know okay I am up or stop or shut up. So this is how Alexa it's learning from you. It is around you. And 18:53 also a very simple example even before Alexa and everything came you know how uh I don't know if people use it quite uh you know uh popularly today but there is an option in your keyboard you can press the mic and speak and it will write the message for you right so why why machine learning is here is because you know it is it is understanding from people voices there are so many people there are so many people who have different voices who have different tones. Okay. They have different uh what do you say uh textures in the voice. A female voice and and a male voice you know it would be like hey how are you doing or or a female would be hey how are you doing? So you know all those things are there. There's pitch there is also the shrillness in the voice. So Alexa is able to understand everything and we are Indians. Alexa is being used out as well right? So some uh someone 19:56 from Britain would say hey mate can you show me a uh you know a beautiful place to go and someone from uh the other side or you know a Japanese or Chinese they'll have a different accent so Alexa is able to recognize all these accents they are able to you know nullify all the pitches male female different languages different you know cultures different accent and still able to understand what you're Correct. So, Alexa is having that voice recognition. That's why it is able to do that. Okay. Let me put this on the side. And then yes, what else? We have Netflix. This is my favorite. So, how many of you have Netflix accounts or have, you know, interacted with Netflix? We have gone to Netflix. And do you know what it does? As soon as you log in, as soon as you log in, as soon as you log in, it will 20:59 ask you what is your genre, what are your favorite things that you like, right? It will it will ask you know, do you like comedy? Do you like thrillers? Do you like uh you know English movies, Hindi movies, do you like uh romantic movies? All those things it will ask it will just ask for four things. And as soon as you watch a movie one or two or three I think the cutff is four once you see four or three movies it will automatically start you know recommending you the movies that you would like. Have you seen that? How how is it able to do that in the background? So this is also a case of machine learning. This is called recommendation system. Movie recommendation. It is recommending you the movies. You have not even gone there and you know rated the movie that yes this is five star this is one star this is four stars but based on that it is still able to you know tell you that what what what you 22:00 might like in the future so how does how how do they do this what they what they implement something is called as let's say there is an okay he watches Netflix. He goes and watches four movies. Okay. And what are the characteristics of Anil? When he logged in, when he, you know, took the account of uh Netflix, they say that okay, Anil, tell me what's your age, man. He's 42 years old. And Netflix also knows he is a male, right? And also, which part in India do you live? Delhi. Okay. So these are these different things that you know they know you have already given to Netflix for free. As soon as you log in, as soon as you you know create your card and everything your account there, you've already given that uh knowledge to Netflix for free. Okay. So what Netflix 23:03 does is Netflix will try to look at all the people within Delhi. Okay. all the people who are you know male and all the people who are in a certain range let's say 30 to 50 range okay it will try to see what all movies these these people are look watching what what what movies or TV shows these people in Delhi or NCR are watching what movies are all the males watching what movies are is this group watching And there is also one peculiar thing that's when you you know uh become a data scientist eventually you'll try to understand this is very simple. The other very peculiar thing is your watch time. Your watch time. So are you watching in the evening? Are you watching at night? 24:04 Okay. Are you watching in the afternoon? this they will automatically get from their platform on platform from Netflix itself they they don't need to come to you so what they are going to do is they are going to look at all the other people also which watch in this in evening at night and afternoon whenever they watch what are they looking at and they also have preferences anil has also told I like comedy movies. Okay, they are going to do this matching based on all these characteristics and they are going to say that these are the four movies that people in this area, people in this with this with this traits are watching. So what are they going to do? They are going to go and recommend all these movies to Anil. 25:07 Okay. So this is very peculiar and this is a type of u machine learning as well which is called recommener system. We call them in industry we call them in short rexes recommener systems. Okay. All right. This is one very peculiar example and very popular one. Trust me guys. I have taken interviews and I have also gone to so many interviews. People will ask you how does Netflix you know recommends movies. This is a quick uh from my experience I'm telling you you can note this down. You need to have at least this basic understanding. Even if you are not applying for any role that requires recommenders systems, you need to know how it works. It is very important from the interview side. Okay, I will mark a star here so that 26:09 all of you know that this is something that you should at least know at this level whenever you're sitting for interviews whenever you're going for you know a role that you're applying be it data scientist machine learning and anyone that you should know at least what is recommend system and how does at least Netflix works and you know once you know this the beauty is the beauty is that you also know how Spotify works, how Spotify works, how Amazon works, right? And there are so many so many places YouTube works, Instagram works, all these all these e-commerce sites they are recommending you something or the other, right? You watch one reel which is you know related to let's say uh dance 27:11 and then the next following reel is also around that. Is it a coincidence? No, they know that you would like this. Whenever you see a video on YouTube, they you're looking something you know that how to do this in Excel, how to do this in PowerBI. The next on the right hand side, you will see some of the videos related to PowerBI because you know that you're looking at this they you might need this. They're recommending you. Okay, these are subscription based. These are freebased. your YouTube and your IG this they they have a slight difference but mostly it's it's around the same concept okay we'll go into the detail what happens in detail but it is around the same okay perfect so we know recommener system and Netflix you won't believe was one of the first companies who who started doing this I'll tell you if you want uh some 28:11 you know background around is I'll tell you what what you know m uh this Netflix used to do Netflix was a company who used to sell CDs go and look at the internet they used to sell CDs and then they used to keep the data of you know who's buying this CD who's buying this CD and even then they used to recommend CDs to M that you have seen this you might see you might like this as well. So they were one of the first companies who implemented this uh they they you know bet a lot on data science on recommener systems one of the first companies. Okay, cool. Let's see what do we have the next example. 29:13 Google maps. Now tell me everyone would have uh you know uh used Google maps here and there. I live in Bangalore and trust me my life is hell without this Google traffic prediction. Okay. What does it tell you? As soon as you say that I want to go from here to here, right? They are going to tell you that this is the time that it is going to take. It is going to take 1 hour. It is going to take 30 minutes. It is going to take 15 minutes. And over there also they'll tell you the route. Which one is taking less time? They'll give you an alternate route. And also it takes it next level. How does it take it next level? It tells you that if you travel between 9 to 11, it is highly congested. It is the most time-taking route. If you want to save your time, travel between 7 to 8. It will tell you this is the traffic prediction what Google is doing. 30:13 Okay, this is also around you. Next, okay, let's start machine learning. Let's get into it. So, machine learning. Now, I will do one thing. I want to start with a few things that are not in the PPT and not this thing but is coming straight from my experience and trust me I need your 100% attendance 100% attention here if you're looking at a phone if you're distracted by someone trust me if you learn this in the next half an hour what I'm trying to explain you will get the entire cusp of machine learning I will be able to cover this in future But not in this detail. Okay. So please pay attention. Let's do Okay. So machine learning, what is machine learning and how is it different from your programming? Has anyone uh made a program? Can you tell me that we can write a a code? 31:14 Okay. To say that which number is even, which number is odd. Very simple, right? We know that all these things are odd. Okay. And we also know all these are 2 4 6 are even. Correct? So we do this very basically. So we can write uh a program say to identify this. Okay. Now tell me one thing. Let's look at another example. The other example would be email. Em email spam. Okay. Now tell me one thing. You also have Gmails any any Outlook or any any uh you know this email system you have you have your account there you might notice a you know uh a folder called spam there 32:16 it tells you that these are not required or you know unnecessary. unnecessary. Okay, you need not pay any attention to this. So what happens there is how do we do this? Tell me one thing. How do we do this? How is Google able to or Gmail able to I'll take an example of Gmail because most of us might be using Gmail. How is it able to tell whether it's a spam or a non-spam? Imagine if you are uh uh you know you don't need any coding for this. Just think with me logically. Let's say you have this data set like this where you have your emails. Okay. And we tell you whether the content of it. This is your content 33:18 and let's make one another column whether it is spam or no. Okay, this is your email one. This is your content one and this is spam. This is your email two, content two and this is spam. This is email three, content three and this is not spam. and so on. Okay, let's say you have 100 emails and you have all the content with them, all 100 of them. And this is also non-spam. For example, how will you how will you guess whether it is spam or not spam? If you are given this opportunity that you have to guess which one is spam, which one is not spam, how do you how do you guess? Have you gotten any messages on your phones that you know you have won this lottery? 34:19 KBC phone, you know, you won 1 cr and you need to deposit, right? KBC say you have won you you are the lucky customer and you won 1 cr rupees. Deposit 2,000 rupees and Okay. All these messages are there, right? So these messages what they do is they have this is the keyword here. Let me close this for a moment. The keyword here is your keywords. 34:57 You are able to tell with the presence of keywords whether this could be a spam or this could not be a spam. What are these keywords? What are these keywords? These keywords would be lottery, right? It could be prize. It could be lucky. Another thing would be win. Right? All these are your keywords which can help you identify whether a male uh email is spam or not spam. So now once you have understood this now this is where go we go into the difference between your uh classical learning as well as machine learning. How is machine learning different? So if there was a normal programming, okay, let's say this is classical programming 35:57 approach. 36:05 Classical programming approach what they will do is they will have one program. Okay, a software engineer or or a developer. Let me say that you know this is a software developer engineer. Okay. What they will do is they will write a program and that program will have what? It will have two things. It will have rules and it will have your input. Input is what? Input is email. Right? And the output is going to be whether it is spam or not spam. Correct? But imagine how are these rules coming [Music] rules. If you see if you see a a male h 37:07 an email having these words lottery prize and words if you see that and if you notice that a ma email has close to 70% of these keywords of these keywords then you will say that it is a spam. Not spam. Correct. If there may be so many if else is if else. If else is that you say if lottery present then do this else if it is not present else do this. So they will have to look at each keyword in the email. Correct? It is so tiring. It is so tiring. It is so tiring and this is not fruitful 38:09 as well. This is not the right approach. You will not be able to capture all the keywords and imagine code run. So that's where ML comes in. That's where ML comes in. ML approach. What is ML approach? We have this again the same data set. Okay. These are your emails. This is your content and this is your output. Okay, this is E1, E2, E3 and so on till E 100. And similarly and you say this is spam, 39:10 this is not spam, this is spam and this is not spam. Something like that. What does ML approach says? ML approach says we don't have to find the keyword. What we will do is we will ask our ML to recognize the pattern. To do what? Recognize the pattern. Machine learning is only this. You have to I will write this again. You have to recognize the patterns in data. Okay, this is ML. What is the ML approach? Now, we will not write you know those programs where you have rules, you have inputs and then you know it will have an output. What ML will do is ML will take this data okay 40:14 and it will pass it to what do you say as a function. This is an MLGO. Okay, this is an ML algorithm. It could be knife base. It could be a classification base. It could be uh random forests. It could be anything. It could be logistic regression. You might have heard these words. We going to we are going to cover it in this module itself. And trust me, be with me. We are going to do it from scratch. Like absolutely no iota of doubt should be there in your minds. What is ML algorithm? We have these emails. Okay, it will be passed into this ML algorithm. ML algorithm will do what? ML algorithm will make a function. This function is also sometimes called a hypothesis function. 41:18 Okay. And what's the job of this function? Hypothesis function. It will find patterns. It is going to find patterns for you. And it is going to say based on the data that I have, this is my input. Okay, this is my input. And this is my output or my target variable. What is this called? Target variable and is also called label. Okay. So what it is going to do is it is going to learn from this data. How it's going to learn? There is a concept of training and testing. Okay. So what happens is it 42:18 will divide all these 100 emails 100 emails into train and test and test how it is going to do. So the popular approach in the industry is what even I personally use is 7030. I use 7030. What do you mean by 7030? The other approach is also 8020. How how how do we do that? So I have these 100 emails. Okay. I will create one data set which is called train. Okay. And I'm going to use the 7030 split. It is going to keep out of those 70 100 emails, it is going to keep 70 emails here. and is going to say that you know this is my training data set 43:21 this is 70 and you are going to keep another test on this which is going to be 30 emails. Okay. Now why do we do this? What the ML algorithm is going to do? How it is going to find the patterns? It is going to look at the strain and is going to look at the content of all of these emails and rather than noticing those each keyword and putting if else's it is going to say that okay I have learned that whatever you have spam and non-spam I'm looking at both the both the options right it is also looking an email which is spam it is also looking as uh an email which is not spam correct so it understand it will find that pattern pattern. It might be keywords. It might be the language. It might be something else. It might be the timing of it. It 44:22 might be the urgency. Click on this button. Uh else you know there's only 2 hours of time to do this. There's only 10 minutes of time to purchase this. You have won this lottery. All these things are fed into this machine learning in the training data set based on these concepts. It is going to learn that. It is going to automatically learn that and that is when your model is created. Okay, this is what your ML model. So what are what is the test for? Why test? model is created and model has identified the patterns. Then why do we need test? Okay, model is trained but we need to know whether the model that we have trained is working correctly or not working correctly. Is it saying that spam mails are spam only or is it saying 45:24 non-spam mails also as spam? Is it putting everything in spam or is it putting everything in non-spam? Correct. That is why you need test. And an analogy for this, an analogy for this is you need test because you need to meet the requirements. validate your results coming from model. Correct? This is why you need test and analogies. All of us has gone have gone through colleges and you know even even the uh what do you say the schools we used to study everything from the book but then we had exams right? We had exams. What was the motive of the exam? The motive of the exam was that we need 46:24 to test whatever we have learned whatever we have read. Do we understand it properly or no? Correct. That is how you are understanding. You are testing your model on this 30% and saying okay I've given you most of the data that I have. I have given you most of this data to learn to find the pattern. Have you done a good job to find the pattern I need to validate in my model? That is why you need test. This is the basic concept that going forward each of the algorithms will use. So what happens is let's say let's take the same uh example of email and spam. Okay. Now you have 100 emails. Okay. You have 100 emails. Each one of this email has been given into your model. Okay. You have first 47:25 split it into train and test. You are given 100. Okay. You are Anju and you are given the responsibility to tell whether an email is spam or not spam. Spam meaning is it harmful or unnecessary. Non-spam means it is a general email and you know we can read it and there is no suspicious links in that so that you can get fraud or something like that. So you are given these 100 emails. One way is that you would read all these emails and find that okay these keywords are present in my spam and non-spam. How am I able to tell okay this is spam or not spam? You are able to identify those keywords. Correct? This is the original way or a brute force way but since we are data scientists we are aiming and you know learning machine learning we'll do things smartly. 48:26 We are going to do things smartly. How? We are going to leverage machine learning. Correct? We are going to leverage machine learning. So machine learning says that you have been given these 100 emails. You will divide these 100 emails into a train set and a test set. You will you will break these 100 emails into this two. parts. One is train and the other is test part. In the train part, you will randomly select 70 emails, not top, not bottom, randomly. Okay, you will randomly select these 70 emails and say that okay, these are my emails. You will pass those contents to your ML algorithm. You will pass those emails to this ML algorithm which will try to find the patterns 49:27 which will try to find the patterns in the contents automatically. It will try to find the patterns automatically. There are many algorithms for this purpose. These are called classification problems. Okay, we will learn. And one most important algorithm or the basic algorithm in classification is logistic regression. Okay. So that is going to find the patterns and is going to give you probability whether the probability of this male of being a spam is 70% 80% or not spam is 70% or 80%. So we have that threshold. It gives a probability. It will not say 1 or zero until and unless you code it that way. You provide that instruction that way that anything above 50% or 70% say it is a spam. It is learning from here. Okay. And once you have learned your 50:29 model you have to test it whether it has learned properly. Can has it learned it properly or no? You need to test it right. You cannot say okay I looked at these 100 emails and I split them into 70 and I trained it on all 70 then uh I'm good to go I can put it in Gmail and Gmail can start using it. No no no no no you cannot do this mistake. Correct? You can never do this mistake until and unless you have validated your results. So you how will you validate? You have these 30 emails. You have these 30 emails. Okay. Now what you're going to do? You will validate it. All these 30 correct. This is spam. This is actual. And this is your prediction. You have said this is spam. This is spam. You have said this is spam. This 51:29 is not spam. But you have said spam. This is also not spam. But your model has said spam. So you need to validate if it is saying everything is spam. Is this a good model? It has not learned anything. It's just saying that everything is spam. It has not learned anything. So that is why you need to split the data into train and test and test it on the test set whether your model has learned it properly or not. if it has learned it properly. There are certain metrics that we look at. Okay, there are certain metrics that we look at but that is in in in in other sessions we'll be covering what metrics are looked at. All right, let's move forward. So once you have done this once you have created train and test then you have your model. Now one important thing is when was the model created? Was it 52:30 created on 100 emails or was it created on train data set? Did we also use the test data set for creating our model? Creating model. Did we use 100? Did we use 70? Or did we use 30? This is train plus test. This is your train data and this is your test data. So we only trained our model on this. That is why the name training. Okay. Now quick uh definitions or terminologies that you need to know whenever you're you know sitting into interviews or whenever you're talking to anyone you are going to become future data scientist you're going to become future machine learning engineer you're going to become AI engineer you need to know 53:32 certain terminologies so this train and test are called train data set okay this is training data set is the same data set but you're dividing into training and testing. So this is your test data set. I am putting 70 here and 30 here. What is 100? This is your complete data set. Correct. Okay. And this thing dividing into train and test is popularly known as train test split. Train test 54:32 split. I will write industry here. Okay. These are good jarens. When you're talking in your interviews, when you're speaking in your interviews, you need to say training data set, test data set, complete data set. And the concept of dividing this is called test train split or train test split anyways. Okay? Doesn't matter. And the other important thing is thumb rule. Thumb rule is either you divide it into 7030. Okay. Either you divide it into 8020. How do we decide it? It it is decided based on data. The the crux is the the crux is that we need to follow we need have one rule for 55:34 this thumb rule that we need to provide as much data as much data for training. Now can you tell me why why do we have to give as much train as much data as for training? It is because your model is being built on train data. Correct? If you are going to pass less data for building your you know uh this thing model it is not going to learn all the patterns in building your model or creating your model. You need to identify all the patterns. You need to give as much data as you can. It has to cover all the factors. It has to look at everything right. And how is this 7030 or 8020 split done? It is done randomly. 56:39 It is done randomly so that if there are a few uh you know whenever you get data it is recorded somewhere. So there is already a sense of you know date wise it is either date wise or it is sorted by you know who has sent it or something or the other. So you want to do it randomly so that you cover all the factors or cover all patterns in your train data set. Does that make sense? Okay. So these are some key terminologies and key things that you need to remember whenever you are going for a job interview and also keep this you know coded in your mind because this is going to be your bread and butter. This is going to be your bread and butter. Okay. 57:41 All right. Now the thing is now we have understood this basic concept. Okay. We've also understood how ML versus classical programming works. Then from there we have also learned how we ML works. Train finding patterns. Then you know testing it on test data set. This we have learned. Now the other thing that we need to you know look at it is types of ML. types of machine learning. Okay, there are majorly majorly two. Okay, there are three but for now trust me there are two. This is supervised and unsupervised. Guys, we are going to just cover supervised and unsupervised. 58:42 And trust me, this is a 100% question. 100% interview question. You need to know this. Even if it is not in interview, you need to know this no matter what. This is the basic. Okay. So what happens in supervised and this is close to 90% of the problems that we see. I'm coming from industry. I've worked on so many projects in my life. Okay. In last 9 and a half years 90% of the task is supervised. 90% of the tasks that we get or ML algorithms that we build or projects that we do in the company they are dealing with supervised type of machine learning. 59:44 Okay. So in this what happens? Okay. Tell me the the above example that we saw of email Spam not spam. This is supervised. How is this supervised? What does supervised mean? Okay, let's start from from you know understanding the word. What is supervised? There is a person who is supervising the ML model. We are looking at those inputs. We are looking at the ML when it was creating that email uh model to classify whether it is an spam or not spam. We were we chose the training data set. We tested it on 30 emails. Okay. So that in future 1:00:44 we have the model built in future if any number of emails come there are another hundred of them coming you already have the model and it is a good model out you know testing it and you have fine-tuned the model any number of emails might come you will just pass it into that ML algorithm and it will tell you whether this is spam or not spam so you have to do that effort once of supervising the model supervising the entire higher thing. So one key thing that it has is label. Correct. Our label here was spam or not spam. Correct. We had this data set. There were emails that we had and this was our output whether this is spam or not spam. spam or not spam. This is your target 1:01:48 or it is also called label. So the data is labeled data. Whenever you are going into supervision or supervised type of models, you will always have label data. Always always you will have 100%. you will have label because if you don't have the label you cannot learn correct if you were not given spam and not spam you will be looking at keywords only no and even looking at keywords you will not be able to say okay I'm looking at keywords but I have identified these two keywords prize and lottery but I don't know whether this email was spam or not spam right so you will be always given this label And in unsupervised what happens is you will never have this u label data. This is going to have unlabeled 1:02:51 data. Okay. Is going to be unlabeled data. Now you will say you should tell me if it is unlabeled. You just now said that how will it learn? Then how will it learn? Any example that you can think of where unlabelled data can be used by machine learning. Okay, I I'll give you one example. All right. Now, this example would be fruits, correct? So, you have an apple. You have an orange and you have a banana. You can club all the apples together. Okay, you can club all the apples together. You can club all the oranges 1:03:52 together. There's no orange. Okay, let me put it this way. You can club all the oranges together. And you can put all the bananas together. You can club them together. How will you club them? How are you going to club them? You are going to club them bases on color. So these are some vision models in machine learning. Okay? and they can look at the color or you have been given a data set. Let me do this way. Let's not go into uh this. Let's say you have you've been given this uh data set. You are not told that whether it is apple or orange. You are given color. Okay? And then you are given its shape. 1:04:56 these two factors only. Let's consider only this. There could be other thing also like length and width. But let's consider only these two color and shape. So anything that is red and is kind of you know oval and this orange would be what? Orange color. And it is what shape? round and banana would be yellow color and it would be long. Correct. Like this. You are given 100 rows. You are given by just looking at these two color and shape. Can you say this is apple, this is orange and this is banana? How how will machine know? Machine will know that yeah the color of this is different from this. Correct? The color of this is 1:06:00 different from this. And the color of this is different from this. Then I have another you know red color. I have another yellow color. I have another orange color. And the shape of this is different from this. Whenever I see red color I see oval in the shape. Whenever I see orange, I see round. Whenever I see yellow, I see long. So what it is doing? The keyword that I'm looking for, what is it doing? To club them together. Features. Yes. But what is it looking at? What does machine learning do? I told you in the beginning, it is looking at patterns. It is looking at patterns. is saying that whenever it is red I have oval and then there is a different type whenever I see orange there is round then a third type which is yellow and long then all the rows out of these 100 1:07:01 rows they are being you know having these three combination only red and oval orange and round yellow and long then I see red and oval orange and round so I can divide them into three cate categories. I can divide them into three categories. This is where the concept of clustering comes in. Okay. So without looking at the data, without even looking at the labeled data, have I told you this is apple? Have I told you here? No. Right? Even if this was label, I don't have the label. I don't have the label. But still just looking at the patterns, you can say the red, orange, yellow, red, orange, yellow, red, orange, yellow, oval, round, long, oval, round, long. There is 1:08:01 a pattern. I can divide these. I can pick these three. Correct? This is your type of ML. 1:08:18 This is an another type of MN. Okay. Okay. So, let's make it small so that it's easier to explain. Okay. So, we have two types of ML majorly. Okay. We have two types of ML. One is supervised, the other is unsupervised. Okay. Here in this example you know what is apple, what is orange, what is banana. But imagine a project where you're told that okay you remember the u initial what do you say um the initial example of Netflix where we took Anel's example you don't know when you're saying that you know everyone in Delhi everyone in uh you know in the age group of 30 to 50 everyone you know uh watching in the evening or at Right? They are related to each other or no, you don't know. You are just blindly 1:09:19 following this model and saying that okay it is good at you know identifying the patterns. So it would have done the right thing. So clustering is really important. It is used where it is used in customer segmentation when you have to segment your customers. Right? You have to do fruits like this example. I I'll I'll explain you with another example. Okay? Let's say let's say you you are given some photos. Okay? You are given some photos. I'm not saying what but this is one photo. This is another one. 1:10:21 I told you guys I I'm a competitor for MF Hussein. So if you're not able to understand what is this, it is not my fault. Huh? This is not my fault. Okay. And then we have this. 1:10:47 Can you can you identify? Let me make this change so that it is easily distinguishable given my So you have this 1:11:07 H. Trust me these are dogs and these are cats. Okay, trust me when I say you're not able to understand my drawing, but this is what they are. How how how would you identify once I've told you this is cats and dogs? Are you able to say that all the cats are having these whiskers, this mooch, mustache and the dogs, they are not having that correct? So even model your ML algorithm will out of all the photos that you have is going to identify the patterns. It's going to say I have one set of photos which doesn't have this hairs on on the face and I have one set of photos which have this hairs on the face. Correct? The other example. Okay. Tell me one thing. Do you uh do 1:12:10 you guys have your accounts on Facebook or Instagram where photos are shared? Do you have it? So do you remember that initially when Facebook rolled this data out right when Facebook uh was rolling this feature out where it could automatically tell that you know you and your friend let's say uh Tulsi and Sarana you are together in in that photo it would ask you first initially it would ask you to tap it that where is Sarana where is that where is that it will automatically identif Identify your mothers and father your mother and father in in a photograph. So it is trying to club those clusters. Okay. It is trying to identify that okay you are always with them. You're always with them. But since it is a a more you know advanced kind of uh algorithm where ML where you are using vision you're using 1:13:10 photos. Correct. So in that case you will have to give it some kind of you know input. But here in these simple examples like fruit like you know when the data is tabular keyword is tabular data keyword is tabular data. This is your tabular data. When the features are given to you in terms of tabular data you can implement clustering. Okay. And whenever you are given features in a tabular data it will say what will it say here the data would be like this. Here the data would be viscous. Okay. Um and here would be sample let me say serial number because you don't know I'm just giving you an example with one feature in the end there are multiple feature that goes in 1:14:12 but let's say these are 1 2 3 4 we have four examples then you have whiskers yes you have no you have yes and you have no so are you able to looking at this you can say there There are only two output here whether fis whiskers are available the facial hairs are available or no based on that you can club these one and three together and you can club what two and four together can you or can you not just looking at the data you don't know what it is this side of the picture you don't Correct. So that is what these are your clustering algorithms. It will automatically do that looking at the data. Okay. Perfect. So now we have 1:15:14 covered supervised and unsupervised. Okay. So we have already learned about clustering. Correct. And the example that we saw for clustering was it was fruits and then the other example was dog cat. Correct. This is clustering and it is unsupervised. 1:16:02 This is unsupervised and the data is unlabeled. This is what we saw. Okay. Two examples. Then we learned something. Now I'm giving a formal name to it. It is called classification. 1:16:29 Classification for classification. What example did we saw? We saw email spam and not spam. Okay. This is what type. This is supervised. 1:16:57 and it is on labeled data. Okay. Now the thing is guys if you look at it I did this purposely. If you look at the output of the ML model, is it not yes or no? Or it is not can we not say it is uh 1, two or three? Like in our fruits example, three three categories we were able to create, right? In dog and cat it was two. Correct? So we are having in both these we are having the output of the ML model is like what is integer correct or discrete. Now this is the term that you should refer. This is what industry follows 1:17:57 discrete. Okay. Meaning that you will either have a specific number of categories or a specific number of outcomes. It cannot be 1.2. Correct? It cannot be continuous. So the opposite of discrete is what? Is continuous 1:18:24 and to understand only for our understanding. We don't have to tell this to anyone. It is only for our understanding. These are the outcomes are real numbers. Okay, you understand what are real numbers? It could be 1.5, 2.3, 4.8, 100.8. Okay, floating. It could be floating. Let me put put this as float. Okay, let me put this as float here. Okay, I'm just telling these two. You don't have to tell to anyone. It is only for our understanding so that you can understand uh the this is the one that you need to speak continuous and here it's the discrete. This is only for our understanding so that we are able to understand the concept. Okay, nothing wrong in it. It's just that if 1:19:25 you will say continuous or discrete outcomes or continuous outcomes then it is people will know okay he or she knows what they're talking about correct with me just telling you out of uh experience follow this trust me you you'll not face any problems in fact you'll you'll have a good impression on the on the person uh in front of you okay be it your colleague be it your interviewer be it your boss anyone Okay. So there are two examples of um the continuous outcomes that I want to discuss. One is regression. Okay. Have you heard about regression? This is generally the first topic that everyone teaches. So regression is nothing but when you are predicting continuous numbers 1:20:28 continuous numbers or values. Now what could be the example? Example could be predicting 1:20:53 salary, right? So, if you want to predict the salary of someone saying that okay, this is the uh amount of years of experience he's having, he or she is having, these are the skills that they work in, this is the field that they work in, then we can predict the salary and salary could be any number, right? could be any number. Even the price it is not fixed that salary will be only 1,000 or 2,000. It cannot be you know uh 40,000 or 50,000. It can also be 52,000. It could also be 52,000 50 rupees. It can take any value. Any value is possible. Okay, we try to round it up in 500s or 100 but in logic it can take any value. Correct? Then the other also example is predicting house price. So if you are trying to 1:21:53 sell or buy okay a house that is also a regression. How will that regression happen? Now in this example again it is supervised. It is supervised. It will have label data. How will how will the data look like in this case? Just giving an example. So let's say the price is this. Okay. And you will have a area. Correct? What is the area of the house? number of bathrooms and let's add a few more. So the other one would be um locality correct. So this is how things will look 1:22:53 like and you will say okay this is you know 20 lakhs this is 30 lakhs this is uh whatever number of bathrooms 3 4 5 something like that correct so this is this value can take anything. So again the concept here would be same as we learned in classification this is supervised. So whenever there is supervised learning you will have to split the data correct because you want to learn you want to learn the patterns and then you also want to test your model. So this will again have that train test split everyone with me. Okay. Train test split amenities. Oh, obviously, obviously. You know, how many features does uh uh this 1:23:54 thing have? If you're selling a flat, whether in that society, how many things are there? You know, is there a swimming pool available? Is there uh let's say a children area available? And if it is not in an apartment, it is an independent house. Do is it a facing towards the park? How far is the park? You know, a a place where people play, a garden, something like that, right? amenities. Okay. So all these things you will learn from these patterns and you will say this is what I have. You will train test and then you will create a final model. So what will happen if if we till now we have this understanding that whenever there's a a supervised problem in front of us no matter whichever type it is whether it is you know classification or regression unsupervised label is present we have to 1:24:54 split it in train and test with me now in that once you have done that split you have to train the model on train that's why the name as training data set and then you have to test the data set on the test plate. Now you have found that after testing it the model is not right. Whatever actual values that you see let's say this was your train data set. Okay. And this is your test data set. Okay. Let's say whatever features that we had on top we are having here also. This is your test for example. Now this is your actual value. You're saying the actual value is 30 lakhs and the actual value is 40 lakhs. But your model prediction your model is predicting this is 100 lakhs. Okay. And it is saying this is 200 1:25:56 lakhs. It is very bad, right? It is very bad. So what will you do in that case? You tested it for some reason. You know it is not working good. Will you go and tell it to you know what uh 99 acres for example that take this model you'll be able to you know uh do well with it or you will give it to yourself or your friend saying that I have built this ML model. It will it was trained on 70% of the data or 80% of the data. It is predicting what should be the price of the house. If you're looking for property, use my model. Is it useful if you're getting these predictions? No. No. Right? No, you cannot use it. If your house price in actual was 30 and you're predicting 100, okay, you'll be at loss and people who are using your model will be at loss. The only person who would be laughing or would be you 1:26:57 know saying is good is the one who selling the house not the people who are buying it. People who are selling it they'll be like this model you know that this guy has made is really well is giving me profits profits it's a raining money here and there. Okay. So whenever you test your model you have to improve it. You have to improve it. improve it. And when I say improve it, what do you do in industry or in you know everyone in the field in data science and machine learning? They call it as finetuning. Okay, they call it as finetuning. So keep this word in mind that till till now take my word for it we will go into fine-tuning how it is done 1:27:58 what are the different techniques there are hundred of techniques for each of these type of MLS also we will cover each one of them I will leave no stone unturned that you know you don't get 100% of this not 100% 200% out of this module but for the time being till we go there. Just trust me this is the process that you train the model you test the model you on testing if it is good you pass the model you give it that it can be used in real world if your model is not good you have to fine-tune it and when I fine-tune it imagine it this way or understand it this way that you are going back to your training data set and trying to make some changes so that your ML model is a new version of the model and is an improved version. Okay. And then you test it again. Then you say okay this is better than the previous one or worse than the previous one. If it is better then you try it 1:29:01 again. So it is a cycle. It is an iterative process. Correct? Okay. How we do finetuning? What are the techniques? There is feature engineering. There's feature selection. All those things are there. will cover it one time at a time at the right place. Okay. So here keep this in mind till then that we fine-tune the model. Okay. And regression is continuous based. Okay. Prediction predicting your salary, predicting the house price, everything that is related to real numbers is this thing. Then there is a different case. Okay. uh people very often I've seen many people uh they get confused with this and that is forecasting 1:30:03 what is that forecasting why they get confused and what do they get confused with guys I need your 100% % attendance and attention. This is a key concept which is you know people it is very close there is a thin line and people get confused in this a lot and this is what you know people are looking at to reject you okay they don't know this only okay so please pay 100% attendance here and attention so forecasting is also continuous. Okay. And is also prediction is also done on labelled data which in turn means it is supervised. 1:31:07 How is it different from regression? Then my pen is not working properly. How is it different from regression? There is one very key aspect, very key aspect which is why this forecasting is different from regression. This is the key thing anyone asks you this is the first thing that you should tell it is having a component of time. time component. This is the key. This is the main thing how forecasting is different from regression. Okay. Now tell me one thing. What are the examples? I I'll say the examples and you'll get to know what are the examples for forecasting. Forecasting example is 1:32:10 weather forecast. Have you ever said heard anyone saying weather prediction, right? It is weather forecast always because there is a time component to it that what was yesterday and what is today what was 10 days back what has been happening every day or every month based on what kind of forecasting you're doing. Is it weekly forecast? Is it a weekly weather forecast? Is it a monthly uh forecast? Is it a daily forecast? Or it is a hourly forecast. Okay. So this is third and the key component is time. Here how would the data look like forecasting can be done only with the label also. You know why? You know why? So if I tell you that this is your data set 1:33:12 the let's say temperature. Okay temperatures are pretty high right now but I'll I'll try to be lenient. uh 30°, 32°, 33°, okay? And so on. You can put in anything 40°, 50°. This is given for one location, let's say. Okay, this is what your label or your target variable. You don't need anything to forecast this. All you need is if you tell me when this occurred, I can tell you when it is going to occur in future. Let's say this is date. This is date. This is day one. This is day two. This is day three. 1:34:12 Day four. Day five. Okay. So we can do anything. If I tell you these were the five days and these were the temperatures I can I can get it now. I can tell you that this was the forecast. This was the temperature in you know last 3 days or 4 days. Even you you think of this very logically. If you know that you know winters are coming. How do you know winters are coming? Because you keep it in mind that whenever November or December it starts winter are going to start it's time based right whenever you feel that you know the the temperature temperature is going up you have that in your mind that okay it is only during these part of days or this part of years where you know temperatures are high which is called this thing summer. So the only thing we'll cover this in detail we'll cover this in detail but I wanted to 1:35:14 show this to you that there are different types of ML okay and different implementations. So it is very seasonal. This season is what time based. So this is the key thing time based. Okay, never be confused for now. Just say that it is time based and we do we do a prediction or forecast of a continuous variable. Okay, cool. The fifth thing. Okay, we've already covered it. What was that? Rexis or recommener systems. 1:36:16 What was the example that we discussed? We discussed about Netflix and then we discussed about Spotify. It suggests you the music based on your taste. Correct? Okay. So these are the five types. These are majorly five types. And there is one more thing which is you know I told you in the beginning that majorly it is supervised learning and unsupervised learning. Correct? It is supervised or unsupervised. But there is another thing called as reinforcement learning. Okay, there's another thing called as reinforcement learning. We'll look at some examples. I think with the examples it will be very clear how uh you know they have. Let's move in guys. This is very very important. Let's get to it. So we had left the PPT here on introduction 1:37:17 to machine learning. I think what machine learning is it's a subset of artificial intelligence this is all you know a formal definitions but main for your understanding is machine learning learns patterns based on data okay then we test it so that we test its performance okay but formal definition is that it is a subset of artificial intelligence artificial intelligence is a huge field there are so many things into it like robotics like you have uh you know machine learning there's so many things into it you know RPA all those things are also you know anything that helps uh you know a a program or or a robot or anything to do something that has some artificial intelligence to it you know and intelligence into that system is there that is artificial intelligence okay uh machine learning is a special field 1:38:17 inside artificial intelligence. Okay, which has been gaining great traction these time and trust me there are so many jobs and it's it's really an exciting field to be in um data is the key and learning algorithm. Okay. So that's we've already understood. Then we also understood what is machine learning. And yes, this is one thing that what is machine learning? So there is a meme on this that you know a teacher is there is a class of robots and everyone is doing their job and there's one person or one robot in this case who is misbehaving. So the teacher says that Robbie stop misbehaving else I will send you to data cleaning and the class that they are going through is machine learning. Okay. So the pun here or the joke here is that as a data scientist okay or as a data analyst or or even you know anyone in this field machine learning engineer the 1:39:20 most of the task that you'll be doing the life cycle looks like this. in machine learning. So you have data correct? You will first get the data. That's your first step. Get data or it's also sometimes called load data from somewhere wherever it is available from the client from you know uh someone needs to build an app. So anything on the internet you need to get the data first. Then the other thing is we need to transform it. We need to you know do multiple things so that it is fit for machine learning. So we'll go into in details what is the difference between transformation loading extraction ETL jobs all those things data engineers are the ones who do that. Okay this is more of a I'm clubbing everything together. This is ETL. Okay. And majorly 1:40:25 is the data engineers who do that. Okay. Majorly it's not that data scientists cannot do that or an ML engineer shouldn't do that. They do it. It depends from uh company to company the size of the company. Okay. I'll come to that. The other thing is uh data processing. So this is where you know 80% of the task is done 80%. Data processing or cleaning I will say. So whenever you get a data in real life imagine I was I was showing you email one content one email two content two and I was showing you the data for house prices right amenities and all that. In real life it is not so clean right? People who are noting out that data will make mistakes in noting. Now 1:41:26 people who are capturing the data they will not listen to one thing and they will type in some other thing. There are always anomalies into the data. There are always some things that you need to clean in order for it to fit for machine learning for the model. Once you have once you get the data you clean the data so that you know you've removed all the values that are missing all the values that are suspicious you've removed them you've cleaned them then is when you you can work on the ML ML part of it dividing it into train and test split based on whether it is you know supervised task or perform you know unsupervised task like of clustering this comes only after doing data processing and cleaning This is 80% of the job and trust me this is is is very time consum consuming tedious and people don't like to do this job and 1:42:27 I generally call this as grunt work grunt work what do you grunt means something you don't like doing okay so it is the major part of uh machine learning life cycle. Then third is your ML, model building. Now in model building there are many things like there are EDA once you have cleaned the data. Okay, EDA can be done while doing processing also. Okay, so that you understand the data. EDA what what do I mean by EDA? It is exploratory data analysis 1:43:28 exploratory data analysis. What what do I mean by this? You are exploring the data. Okay, you are exploring the data and saying that okay after exploring the data I found these these relationships first you will explore the data then only you can ask the model or decide which model to put right you cannot close your eyes and just say that okay this is I have the label I need to do this once you have the label you need to know now whether you want to do classification or regression or forecasting so and which are the fields that are important content. Okay, we should remove the outliers. Outliers are also called anomalies. If you have a a data set which is of all the cars which are being sold in you know cars 24 or spinny for that matter. If all the cars 99.9% 1:44:29 of the cars are between the range of three lakhs to 20 lakhs. If in the data set when you receive it you see one car for 100 crores 100 crores or other way around one rupee what is that that is anomaly someone has made a mistake be it a human most of the cases it is a human who makes that mistake while noting down that data while collecting that data sometimes it could happen due to some software issues also correct so all this thing comes into data processing and cleaning that is really important but is also not a fun task to do. It is you know uh very difficult part but very essential part. Okay, that's where the meme is coming from. Okay, that if you misbehave in the machine learning class, I'll send you to a data cleaning class. Okay, cool. 1:45:30 Let's go ahead and then uh how does machine learn? We've learned it. It learns through patterns. This is what this slide is also talking about. So it is it is a really good example and even to eyes you know it looks very uh disturbing. So this is a breed of dogs which is chichua. Okay. And this is a muffin. So if you you know some even your brain and eyes get puzzled especially looking at a few of them right like this one looks like a almost like a dog but it's it's a muffin correct and even this one this example so there is this doodle labrada doodle breed but you might get confused with a chicken like this one it's a chicken this is a chicken this is a So at the end of the day, things could be very messy. It could be very 1:46:32 confusing. In real life, things are not as simple as black and white. There might be a gray area. Okay? It's not white and black ever. There's a gray area. And that's where machine learning works on the concept of probabilities. it will say that I am there's a 100% probability or there's a 70% probability that this is a dog and 30% probability that this is a muffin that is a wrong model. You need to train it again fine-tune a few things so that it improves. Yes, that's the entire process. You need to be able to fine-tune our model and it will always give you a probability. It will say this I am 99% sure this is dog. I am 20% sure this is dog. So it is always one minus one game in machine learning especially in classification. Just note this is it 1:47:33 is always one minus one. Meaning if you say that there is a 20% possibility that this particular thing is is a dog okay then 80% of it is it is muffin correct it is always 1 minus one game in machine learning but the underlying concept is probability it will never say I am 100% sure for anything and everything it will always give probabilities whenever you have these task as classification. Okay, cool. Let's go ahead. Yeah, this is another example of classification only. So now the a curious uh or an interesting thing about this is that you see capture, right? Whenever you try to go and log into some uh you know a secured website or a portal, you get this capture. They want to identify 1:48:34 whether you are a human or you are u this thing uh a robot who's trying to you know get to that data and try to you know hack or scrape things. So you are labeling it as what is the fire hydrant. So you are labeling the data for them. Labeling of the data is the tedious task. People need labelled data so they can make these models. 90% of the things are supervised. But if there is no one who is labeling it, there's no use of supervised learning, right? It is as good as your label data. So whenever you go to the website, next time think it this way. You're not telling the website or you're not saying that you know I am a human, let me enter. It is in fact you're doing labeling for them for free. You're doing labeling for Google for free. Okay. Cool. All right. There are a few myths uh that we'd like to you know address. Now this I broadly touched upon 1:49:36 that you know artificial intelligence is a whole umbrella under which we have machine learning and even niche of machine learning a subset of machine learning is deep learning. Okay. So here you would say there are things like robotics and in machine learning there are a few things that you can do with machine learning only and there are a few things that you well you can use machine learning and deep learning both but there are cases or instances where only machine learning will do better. You have a gut feeling based on experience. You learn over time and you know for a fact that deep learning will do better here rather than machine learning. Okay. So an idea to that is as and when you get huge data as in and when your data grows. Okay. That's where deep learning is really powerful. It's really powerful and really better than machine learning. When your data is 1:50:36 absolutely huge, humongous, then deep learning is what has the most chances of working better than machine learning. You can still do the same task with machine learning but you know you'll get okayish results. Your model will be okayish but deep learning when the data is really huge it will do wonders for you. Your your boss and everyone will be very happy if you implement deep learning model and perfect. Okay. So we'll touch on this one by one. So artificial intelligence is basically I told you you know just imparting a way of uh intelligence into a machine as a whole is a huge field is artificial intelligence. Okay. And then machine learning which is a subset of artificial intelligence is the ability to you know train a machine to 1:51:38 perform and learn without explicitly coding it. Remember we talk about classical programming versus uh the ML approach the email spam. If I have to do this in classical programming, if I have to write a code for it, program for it, I'll have to look for keywords in each main and say that okay this is what it is I am writing a code it is really difficult to do that and how many combinations will you keep at the end of the day correct so that's where machine learning you're giving the data you're saying this is your label and this is where you need to learn from it and then once you are trained once you are working good on my test data. Now you're ready to go into the world and tell the world that this is spam and this is not spam. Perfect. Okay. Deep learning. Now deep learning is slightly different than machine learning. Okay. Now in machine learning there are very little nuances that you have to take care of like when 1:52:39 we were talking about uh the house prediction case. Okay. There are certain features. When I say feature, there are columns or information that are important for your model to predict. But there are some things that are uh not really important for deciding the price of the house. Number of bathrooms, yes. Okay. Number of floors, yes. Size of the house, yes. It will tell you higher the size, higher the cost. Obviously, right? Or not obviously, I don't know. depending on the locality. So you cannot say only one factor but other factors also. Okay. If I tell you with all the house prices, if I tell you which agent is selling it, agent as in which broker is selling it, will that be of any any value? Will will it be you know useful in predicting the house? If I say person A is selling house A and person B is 1:53:41 selling house B, will you be able to say that okay the person selling A house will be less price and B will be high price or vice versa? Will you be able to tell? No. Why? Because that variable or that information is absolutely of no value for your model. If you put that variable into your model in training data set, your model is only looking at noise and is going to become corrupt. Okay, it will try to find some patterns unnecessarily. Deep learning on the other hand, so in machine learning you have to decide which variables, which information I need to put in my training. But in deep learning in the other uh sense is very smart. Why it is very smart? It works like a human brain. There are small small small small small neurons. These neurons are nothing but one simple nodes 1:54:41 that are a part of an algorithm. It will process all the data that you have and it will automatically decide that you know you don't need person A, person B agent data, agent information or broker information for house prediction. It will automatically do that. So this thing is selection of features and passing which feature or variable or information which information is necessary for the model which information is not necessary for the model is called feature engineering or feature selection in machine learning. But that kind of concept is not required in deep deep learning. Okay, that kind of concept is not required in deep learning. So uh that is deep learning. It works like human intelligence. You have neurons which learn from each other. That is why for it to automatically tell that you know uh this this information is not useful and this information is 1:55:41 useful. For that to happen you need to have huge data. You need to have huge data. That is why now you understand why deep learning is better for data where it is you know really large and ML could work on data which is even small or medium sale. I don't say that you know deep learning cannot be implemented on medium data set. It can be implemented but you know the effort and the uh you know the the kind of results that you'll be getting between machine learning and deep learning are not going to be very different. Okay, you can make distinction between deep learning and machine learning. Okay, cool. Now the other myth. Do you feel your job is under threat because of AI? Yeah, so this is a myth guys. Trust me, this is a myth. I've been working on models like GPT. Okay, me myself have 1:56:42 trained many models like chat GPT. Okay, if you guys don't know chat GPT, you should definitely know about it. Go back after this class, experiment with chat GPT. But for those who know it, it is a kind of generative AI. Okay, it is dumb. It there's no other uh way I can put it. It is dumb. It needs a human for it to tell this is right, this is wrong. This is where you're doing wrong, this is where you're right. Whatever you see right now is you know uh if I have to give you u an example chat GPT like models right if if it breaks this myth I would like to share this data with you guys chat GPT I don't remember the exact uh count but it was trained on trained on data 1:57:45 worth two lakh years. Imagine this. Imagine this. The data that a person you know we don't live even for two lakh years. It was trained on a data which is worth of you know uh two lakh years of data. If I start collecting the data now or start reading or understanding the data now, it will take me two lakh years. Will I be even there till two lakh years? Humans are way more you know uh understandable and way more intelligent. A kid of 12 years old, 10 years old starts to learn that you know this is right, this is wrong. only in 12 years, 10 years, 8 years and the after feeding in this two lakh years of data also to Chad GPT and other you know enthropic and llama all those 1:58:45 things they are still dumb. So trust me guys, this is an absolute myth. Don't be worried. You need, you are required for AI to grow, not the other way around. Okay? You are never going to lose your job at least for another 100 years. I can give that in writing. But there's one condition to it. You need to be very uh what do you say? You need to know how to use AI. That is one different thing. Okay. So, you need to know how to leverage this versus you know uh or use this versus worrying that you know it is going to take my job. It is you know I'm going to be replaced. No, that's not going to happen. I I can give you in writing. Okay, that's not going to happen. It is not in the current state. It is not in that uh league. Automation is replaced. That's what automation is. But it's a robotic 1:59:48 process, right? So all the grunt work, all the things that are repetitive in nature, it can be replaced. But you still at the end of the day, you still need a person to look at it. You cannot blindly trust AI, right? So you need to know how to use AI, how to fine-tune, you need to be expert in your field and you need to know AI. It's not that you can blindly uh give this thing. Okay, tell me one thing that if if you're going into a operation theater, will you be comfortable a robot doing your surgery rather than a doctor? Correct. It has no emotions. It will do something some stitches somewhere it has to put and it will put somewhere and it is emotionless. It will not be very soft or gentle. a robot and you know things can go haywire. So we are really at a stage 2:00:50 where is which is very far okay from us being replaced. Okay. All right. So we covered types of machine learning supervised unsupervised. Okay. Now uh the other thing is before we move on to other things I want to capture a few examples that are in the PPT. Okay. And yes, this is something that I would like to touch more in detail which is uh AI versus machine learning and deep learning. You know what? Let's do this. Let's do this. Then I think uh we have everything that is there in PPT. Okay. Now what is the concept? This is your AI. Okay. This is your ML and this is your deep learning. Okay. Now, artificial intelligence also 2:01:51 has a field of robotics. Correct. Any slight intelligence that you put in is called artificial intelligence. Okay? Whereas machine learning could be unsupervised learning, could be supervised learning. And the third type that we have to discuss in detail is reinforcement learning. Correct? Okay. Now deep learning is also same but it has a few additional things like NLP like video geni. Okay. So let's try to divide AI is a broader field within which ML is a subset and deep learning is even a subset a very special case of machine learning. Okay let's discuss what is the difference. So if I say I have ML here, okay, and I have deep learning here. Okay, before we 2:02:53 go into that, what all type of data can we have? What data do you see around you? I'll give you one example. Text data. Okay, let me write this. We are going to be data scientists, machine learning, data engineers. So uh okay let me write a few examples so that you get to know. I have text data. Okay. I can have video data. I can have image audio. Perfect. Image audio. You have done clustering or classification here rather you have done classification here. Why? Because there is no other type of data that you can get. Think 2:03:54 each and every type of data can be classified into one of these categories. Either it could be textual data. Okay, when I say text, number and all those things. Okay, or either it could be video data because video is slightly different. Image data and audio data. Audio is kind of your signals frequencies. Okay. Now there is another nuance to it. Okay. The other nuance is though we know this data there are two types. It could be structured, okay? Or it could be unstructured. When I say structured, what is that? It is a tabular data. Okay? Whatever you get in a table format is tabular data. It is structured. It 2:04:54 doesn't mean that you don't you you you can skip data cleaning. You can skip any part of it. You can skip out treatment or missing value treatment. We'll touch upon that. But you still have the data in a format that is structured. But is there any structure in in video? Is there any structure? image, right? Audio and a special uh you know category of text which is natural language. If it is written, if it is spoken, whatever any case of textual data which is in the format of natural language, how we speak, how we interpret things is unstructured. Everyone, this is really important for us to understand deep learning and machine learning. We understood AI is a huge field. Okay, 2:05:55 within that there is a small part which is ML. Okay, which is this. And then within that we have a few things like we have AI, we have ML. Okay, ML has unsupervised learning, supervised learning and reinforcement learning. And then we have deep learning which is working on NLP, generative AI, some of the features or some of the you know implementations I'm talking about. Okay. Like uh image processing, identifying which image it is, digit recognition, whether it is one or two okay five or a four all those things that is under deep learning. So at the end of the day everything is about pattern finding patterns. Okay. Now I asked you here what are the different types of data that we get in a real world. Okay. Not data types of a variable. Okay. But the kind of data 2:06:59 that we can get. Okay. If if this is confusing, let me change it to kind of data available. Okay. In the world, what is the kind of data that we have? It is either structured. Correct? It is either structured or it is unstructured. When we say unstructured, it is when we say structured is it is tabular data. Okay. And unstructured would be that you know there are so many things which a computer cannot understand or your software cannot understand but we as humans can interpret it. You're watching this video right? You can interpret this but how will your system how will a machine understand this? How you are able to understand is through your brain through your neurons. 2:08:01 But you know your system doesn't understand what is a video. The way the machines talk or they understand things is in terms of binary in bits in zeros and ones in numbers. They don't know what what is K what is a here with me at a very grassroot level. So unstructured data is that data which we can understand but is not in a tabular format for or you know for uh the machine to understand. Okay. It is unstructured. It has to be structurized. So that is video, image, audio and natural language. Give me an uh an example of natural languages. the thing that I'm talking now okay if it is okay a better example would be uh you go to doctors they write prescription right whatever that you have does it have any format is it in a table can you directly pass it into uh into a software or into 2:09:03 a machine and say that okay tell me what what all you know this was my when when you go to the doctor on top they write okay this is what the complaints are and at the bottom they write this is my medicines. It could be one medicine. It could be 10. Correct? There is no structure. Can you directly pass this language to your system to your machine and say that okay tell me for these these complaints 100 patients I'll give you data for these these complaints. Can you tell me which medicine should I take? No. Right. Machine cannot interpret it as we have as we can interpret it. We have to do some things that is natural language. Okay. So, so now the other question is now you know this is you know kind of data that you have the differential between ML and deep learning is really easy very easy okay I told you one thing that 2:10:08 ML works good on small data right or medium but deep learning can work on medium data also but the uh effort that you have to put the price that you have to put for a deep learning model on a medium data will not be worth it. You might get a 0.001% 001% improvement or even you know.1% or 1% 2% depending on what your domain is right you might get that much of improvement or you might get deterioration or it might be exactly the same but the amount of money that you'll be spending on uh putting deep learning there because deep learning runs on GPUs it needs additional processing so that's why we prefer this on more data or large data, huge, humongous data whenever it is there deep learning, close your eyes, go to deep learning. Okay. The other thing 2:11:09 why what is the difference between ML and deep learning? It is feature engineering or before that now tell me since we have studied this already, I'll tell you and then we'll talk about feature engineering as well. So here uh you can work on structured data and you can work on unstructured. Understood? That is why I wanted to clear this thing. What is your unstructured data? What is your structured data? You cannot use video as it is for your machine learning. Deep learning is a better bet for image processing, for audio processing. Deep learning is better. Machine learning will only understand numbers. There are no features for you 2:12:10 to play around with. Deep learning is your bet. This is one of the reasons why deep learning is gaining traction in the market because machine learning can work only on tabular data. Most of the companies they have tabular data only. They have records, they have you know sales, they have stocks, all those things. But as and when we are now producing so much of data, people are producing so much of data. data is now being accumulated in unstructured data as well. We are having videos, we are having images, we having audio, right? And also earlier we used to not have those capacity of running. We did not have GPUs, we did not have you know these big big databases, data warehouses to store the data. Now we have them. That's why your deep learning is gaining traction and very clear distinction. Now you can confidently go and say that okay 2:13:11 this is where I will apply deep learning this is where I will apply machine learning is very clear now okay whenever the data is unstructured whenever the data is large deep learning whenever the data is small or medium but has a structure to it prefer ML okay okay now let me touch upon feature engineering okay and with that the other difference This is the third difference which is feature engineering is required in machine learning. Okay. And feature engineering is not required in deep learning. This is the third example. We'll touch this in detail now. But keep this in mind. Feature engineering is required. You have to do it in machine learning. In deep learning, there's no need. It will do it on its own. and you'll understand why it does it on its own. We'll connect everything. I will connect everything for you. Okay, 2:14:12 cool. Now, let's talk about feature engineering. So, this is feature engineering. 2:14:28 Okay, what is feature engineering? Let's try to understand our data set better. Okay. No one will tell you. So guys, this is something that you know is very standard in the industry but no one teaches it. No one teaches it and you you understand it but you know your understanding is kind of con convoluted between these other other things. Okay. So understand this with absolute attention. Okay. move away your phones and move away from you know the reals that you're watching. Okay, cool. Now for understanding the data, this is how you get the data. Correct? You have many columns and there is one and there is one target variable. 2:15:28 Okay, this is your target mostly. This is how you get it. Okay, so how do we define this? Now this is your output. Okay, or label. This is called label or it is also called target. Okay, this is called your features. 2:16:02 This is called your features or your variables. Okay. All the information that you want to do is called that. Let me try to explain this with an example now. Okay. Our house a house prediction only. This is your price. Okay. This is your area area of the house. This is your number of bathrooms. This is your locality. Okay. This is number of floors. Okay. And what all other things. Okay. So these are your features everything in the blue and red is your target variable or your label. Okay. And now pay attention guys. 2:17:04 Pay attention to this. This is X I. Okay. This is one instance. What is this? This is one instance. It is also called a data point. Okay, everything in your features in your variables one single row X I is and all the features in in coding and everything a standard practice is you represent your features with X. Okay. And your target variable is Y. Okay. This is a standard uh nomenclature that everyone follows. So whenever you go ahead and even read some of these uh PPTs and also uh if you're reading on the internet or looking at someone codes this is very standard okay that's why I'm trying to 2:18:06 clear this standard okay cool and this is yi for i if row this is the actual actual output or actual value. 2:18:29 Okay, for if row I'll write this if row, keep this in mind. These are very small things. No one teaches it, but they are really important for you to understand all the following uh you know concepts and even when you're looking at some of the formulas and everything, it will be very clear. Okay. Now there are other uh you know terminologies or some this nor nomenclature as well. So whenever you have yi okay it is actual value actual value of iat row or iat data point. I hope that is clear. Okay. And this is how they say you put this, you know, cap on top of it. You must have understood this in maths, 2:19:31 you know, somewhere. If not, but trust me, you don't need more than that of maths. This is ycap. That's how they call it. And this is so that you can distinguish between actual values and predicted values. This is your predicted value. Okay. of IRO. Okay. Now feature engineering is what? Feature engineering is what? Now let's say one of the feature that was given to you was broker name. Broker name. who is showing you that house. Okay. Where or is the broker if you're selling a house? A broker is a person who is middle middleman who takes commission from both the sides. If if no one knows that, okay, he 2:20:33 is responsible for selling it. Now tell me one thing. Do you need the broker name? Will it have any information or will tell you to decide the price to predict the price of the house? Does it have any value? If you pass this to model, you know, let's say Amar is selling one, Akbar is selling one, Anthony is selling one. Does it tell you anything? No. Right? It is a useless feature. So the act of you deciding the act of you deciding that this feature will not go in the model this feature will not go in the model is feature engineering. The act of deciding which feature 2:21:35 to pass to model is called feature engineering. Okay, there is another thing to this which is you can create additional models also additional features sorry creating additional features. How is that? Let me give you an example which everyone can understand. Okay, let's say you have this uh thing where you have to predict the age, right? Age. Now you are given this problem statement that predict age. Okay. So you are given what? You're given height of the person and you're 2:22:37 given the weight of this person. Based on that you can guess the age. Okay. Simple, very simple. But what you can do there is something called as BMI, body mass index. Correct? It is some uh combination of if someone can tell me it is something you know height over weight square or something like that. There is some formula for this. So it is a calculated field. Calculated field which takes calculated field which takes both height and weight inside it. Correct? Okay. So this is your calculated field. So this is a new column that you're creating. Okay. So you have either you can pass both of them or you can pass all three of them or you 2:23:37 can pass only BMI to predict the age. That process of modifying your features, adding, creating, deleting all that thing is called feature engineering. Okay. And now we can also understand this point again. So in machine learning you have to do this. In deep learning it will automatically do that. If you have 30 features, 30 columns, 30 variables and only five of them are important. You have to do that selection in machine learning from five to from 30 to 5 based on certain techniques. You need to come from 30 to 5. We'll talk about those techniques what it is but logically understand this. In machine learning you have to come from 30 to 5. In deep learning it will automatically come from 30 to 5. So feature engineering is required in machine learning. In deep learning it's not required. All right. Cool. Then let me take this 2:24:41 question. What is categorical 2:24:50 and numerical? Okay. So now you know that in this format that I told you in the above. Okay. This is one of your feature. This is one of your feature. We've understood this. It is also called a variable. Right? It is also called a variable. Now these are its type. One feature category or numerical. That feature is either numerical or categorical. When I say categorical, it will have discrete values. Okay, it will example would be if you have a column saying gender. 2:25:50 Okay, male or female. Correct. Right or wrong? If you have a feature called fruit, it could be in our example apple, orange or banana. Just be careful here guys. Here we are passing this as a feature. We are not categorizing it. Whenever you have categories, right? That variable when it is in feature is called categorical feature. When that same categories go into your label, you have to do classification. Understood the concept why it was necessary to distinguish between this and this. Right? When you have categories in your label, you have to do certain tasks like classification. You have to say okay this is yes or no. It is email is a spam or non-spam. 2:26:54 Correct. You looked at that example that becomes your that will give you a hint of what type of model you have to put, what type of learning you have to do. But when it is on the other side, when it is in your features that you have to use, you have to handle it differently for your model and data cleaning. Okay. Okay. Cool. And this numerical numerical is any continuous variable. 2:27:33 Okay, any continuous variable. Now our example here was area. Are area of the house. Okay. Area of the house. Then what else? In our above example, it was weight. Okay. And when this continuous variable comes into your red part which is your label. When it becomes label, it becomes what kind of problem that will give you an idea what kind of problem you have to do? You have to do either regression or you have to do forecasting. Forecasting you will do only when it is time based. you can apply forecasting there. Okay, I hope this is clear. So what I'm saying is we understood here that this this is the uh terminology that we understood here, right? Everything on the left is 2:28:35 your features. Okay? And this one is your label. Okay? And then that's the only thing that you need to remember as of now. Now there are types of variables. Okay, types of variables or they are also called data types. But I would like you to understand it type of variables. Okay, now variables are what? Everything that is on the left side which is in this blue blue line. Okay, these are variables. And for one column, one column only we are talking about this. If this column which is your variable is categorical meaning it has only few uh entries in the row. Example it could be the owner is male or female. It is a gender kind of a feature or a variable. Okay. Number of bathroom could be 2:29:36 categorical. Right? In in our previous example, number of uh uh this thing bathroom could be categorical. You could have 1 2 3 4 5 6. Right? Hear with me? Locality also could be categorical. Saying that locality is Adashnagar, locality is Yelanka, model town only three or more than that you can count it. It is discrete value. Okay. But whereas continuous variables or numerical variables have continuous value. It is continuing 1 2 3 4 5 6 7 8. There's no end. So your uh area could in keep on increasing. It could keep on reducing. Okay. Weight could keep on increasing. Height could keep on increasing. It can take any value. It is not that it will take only uh you know integer value or 2:30:40 only limited number of values. Okay. Continuously changing as in it could take additional value any other value. There's no limit range that it will take only one or two. All right. So I think we've covered this. Now ML we know is of three types. We've understood super supervised learning in detail. Correct? We've understood unsupervised learning in detail. Right? There's one thing that I told you that we will touch in detail is reinforcement learning. ML has three parts SL, UL, RL. SL stands for supervised learning. We've looked at it in in detail. UL is unsupervised learning. Looked at it detail. RL is reinforcement learning which we will look at now. Okay. Now reinforcement learning 2:31:43 I will first explain it from the PPT. Okay, which I feel has a good example really good example. Let me show that. So we've covered this AI versus ML versus deep learning. AI is not taking your job. I told you yesterday also it is really dumb. It is trained on two lakh years of data. Human people, small kids with only 12 years of data, 10 years of data, they know what is right and wrong. Okay, is not going to take your job. Supervised learning we looked at it. Unsupervised learning we looked at it. Now let me go back to these are examples. Okay, Apple, banana, email spam and you know clustering of these. We've covered all this voice-based assistance, Netflix recommendation. This is where I want to talk now. Okay, reinforcement learning. So, reinforcement learning in fact is better to be explained with this. So, if you have seen a dog being trained or a cat being trained whenever you know uh 2:32:45 there is two things before conditioning and during conditioning and after conditioning. Okay, stick to this word word conditioning is meaning that you are in the process of doing something. Okay, you're changing something that learning in you know machine learning that learning is conditioning. Imagine imagine it this way understand it with this way. So before learning okay what used to happen is when you put a food in front of dog the saliva will start coming in. It's a response to stimuli. This is his response by looking at food. It feels hungry. Okay? And whenever you press a bell, you know, you say that okay bell is ringing. It has no response. It has no absolute no response. You know the dog will feel like why are they ringing bells? Have they gone mad? Okay. But during conditioning 2:33:48 meaning during learning when the dog is being trained what you what trainers do or what people do okay when they condition a dog they say that whenever I ring a bill bell and I will give you a food they do this 10 days 12 days 13 days 100 days whatever dogs are very smart they keep they learn it very quickly but what they're trying to do is in are learning they are trying to associate the ringing of bell with the food. That's what they are trying to teach the dog. Okay. They will every time they bring the food they will also ring the bell every time. This will happen for a few days. Then after you know that is conditioning. That is conditioning. After conditioning, what happens is whenever you ring a bell in front of the dog, you will get the same response of saliva coming out. Meaning it thinks 2:34:49 that for the last 10 days, 12 days or whatever x number of days whenever the bell used to ring, I used to be served food. So it will have the same response by listening at the bell as it used to have for looking at the food. Okay. So this this is after conditioning. You are reinforcing something in the learning. Okay. You are reinforcing that ringing of the bell is associated with food for the dog. Okay. The other thing let me now go back and explain you next example and we'll connect. Imagine you have a maze. Okay. And your goal is you have a toy car. Start to end is what you want to go. 2:35:54 Okay, that is your goal. This is the start and this is the end. Okay, and let us make us more interesting. So if uh Lena and Anvesh are friends, okay, and Lena says, "I'm building a toy car, okay, that can run on its own, that can drive on its own." And Anves is Lena's friend and says that okay, how can I help you? So Lena says, "Build me a maze on which I can train my car." Okay. So Lena says, "Build me a maze on which I can train my car." So Anvesh being a good friend, he builds a maze. Okay, I hope you guys know what what maze is. Okay, something or the other. Let me show you what a maze is. Maze image. 2:36:55 This is a maze. Maze, sorry. Okay, you start from here and you get to some other place. You start from here and your goal is to go here. Okay, so what happens is this is an absolute gem of a maze that Anvesh has built. Now what happens is Lena says that okay I will not look at the maze because I want my car to learn autonomous driving to automatically reach from start to end. Okay, let's open the space here. I've made a closed maze. So when the car goes here, okay, this is the car. When the cars go here, since it there is no remote control, nothing, it will first crash here, right? And then it will try to go here and then again it will crash here, then again here. And ultimately it will try to go out of 2:37:58 here. Correct? You are with me? That it will first crash. It will not go on its own. Correct. Perfect. Now reinforcement learning is this concept that listen to this very carefully guys. If you are able to understand this one example, life is set. Reinforcement learning is done. Okay. So what Lena has done is she has given points to its car or rather Anves and Lena they are tracking whether the car is good or wrong. They have given 100 points to the car. Every time it crashes it is given a penalty of minus 10 points and every time it takes right turn. Right as in correct it could be right or left. Correct turn it is given plus 10 points. Okay. Now 2:38:59 the model is such that it will try to optimize this number. It will never try to reduce this number. So it will try to avoid crashes. Correct? And try to gain as much as points that it can. If it crashes also, it will try to gain 10 points by making a right turn or correct turn. Okay, is that clear? Similar to the dog example, similar to kids example, you're given toffes, chocolates for good things, scoldings for bad things. Similarly, this is reinforcement learning. Okay? And there are certain terminologies to this. Let me put that in perspective as well. There are certain terminologies. Now this yellow maze is called environment. 2:40:03 Okay. This entire maze on which it is learning is called environment. Okay. And this car which is learning is called agent. agent vinod not agent vinod that's not the agent that I'm referring it's the agent in this case it is car okay to whom we are asking to learn okay then we also have goal this goal I have already told you this is the goal move from start to end okay this is what This slide talks about okay this is your agent and the environment in our case the maze was the environment what it does it observe if it crashes it gives you 50 points okay and this is your reward and penalty + 10 and minus 10 2:41:03 action is basically whatever happens you know uh if it crashes you have to update the policy in that case they are calling it policy So this is your action policy meaning you decided it will be + 10 or minus 10 + 10 for correct turn minus 10 for crash that is your selection of using policy. Okay. Then action meaning actually whenever it uh crashes minus 10 and takes the correct turn + 10 that is your action and after every action you update the policy based on reward or penalty meaning 100 become 90 or 90 becomes uh 100. Okay based on crash or turn perfect and iterate on this entire process. Now once Lena and Anvesh has successfully made this car or agent learned 2:42:04 it can now go and do the same thing for any number of mazes in the world and they have made an autonomous driving toy car. You train it only once or twice on the environment you know depending on the case. This is the concept that your autonomous driving class uh cars are also following or in other words it's also called driverless cars. Right? This is the concept behind that. Okay. The other thing is that reinforcement learning guys is very research oriented. Okay. In my experience, if your company is into research and 2:43:05 development or if you're looking for PhD, this is important. Okay? Otherwise, if you do supervised learning and you do unsupervised learning, meaning coming from machine learning or deep learning, this is what your jobs will require you to do. 99% of the jobs will be this. Okay. So understanding reinforcement learning at a high level and understanding what is happening is important that can be asked but on the jobs I'm coming from experience. I'm talking from experience there's hardly any use case that you'll be working on. Yes, if you work for you know a company who's working on autonomous cars, drivers lay cars then you would need reinforcement learning there. Okay. 2:44:08 Now, what we going to do is a really cool stuff. We're going to do something really cool. What is that? Yeah, this we have done. Okay. So, let me just close this uh introduction thing. So if anyone asks you these kind of questions can you clearly and confidently say it is a machine learning problem one and what type of machine learning problem okay that's that and then yeah these are some examples I think uh we've covered most of it okay we've covered anomaly detection is nothing but fraud in transactions you know uh anomaly that is out of the Okay, enough of uh this thing. Let's get to something that is interesting. Okay. Now there's also one um assignment that 2:45:08 I would like to give you a homework which is you need to go to the internet okay and tell me what is semi-supervised learning okay Now we are going to do something which is called linear regression. 2:45:49 Okay. So this is where everyone starts teaching or starts learning. This is the basic uh you know thing in machine learning that you need to know linear regression and trust me guys I will teach you in first this is how we are going to do we will look at some of the theoretical concepts okay we'll be doing that and then we'll be going into practical or hands-on approach This is how we are going to learn every concept. Okay. Why theory? See to be honest all of us we don't uh you know we we don't like theory that much. Practical is where you know everything happens and you know our brains open. We have a minute to minute understanding of things. But trust me 2:46:50 the theory that I'll teach will be in really interesting format. Okay? You'll not feel like you're learning theory. Okay? And we'll only see what is important, what is required and whatever I can cover with the practical hands-on knowledge. I will do that without compromising on the quality. Okay? You have to trust me on that. Okay? You have to trust me on that. All right? This is how we'll do it. Okay? And trust me I am going to teach this in a manner that you will not feel like you have you know done something a lot you'll feel like yeah this is so easy this is so easy okay we'll do it that way done okay so now before we go into linear regression okay we briefly touched upon what is regression Right? We know what is regression. House prices 2:47:52 prediction of that. Okay? Then your salary prediction meaning it is continuous variable or your output or your target variable is continuous. Okay. Output is continuous and then we have we need label of course in in label here we call it as target or label. Okay, this is regression. Now essentially regression is nothing but it it essentially what it does is how are we predicting these? How are we predicting these? It is trying to find relationship. The word that I told you to keep in mind for machine learning was finding patterns. Okay, regression is in those 2:48:56 patterns you're finding relationships. Okay, you're finding relationships. This is the key word. What are you finding relationship between? You're finding relationship between your target and your features. Now when I write features, everyone is clear what I mean by features and target here. Correct? You are trying to find relationships between your target variable and your features. Okay, cool. Keep that in mind. That is that method that type of learning where you're finding relationships is called regression. Now regression could be of two types. Okay. Topic is linear regression. Okay. Linear regression. 2:49:57 Okay. But there could be something called as nonlinear regression as well. 2:50:09 Okay, based on this relationship word that I told you what is linear and nonlinear. Okay. All right. Let me say. Okay. So whenever if you replace this regression with relationships it means linear relationships and nonlinear relationships instead of regression if you choose this word that I'm telling you that you trust me you trust me say that regression is relationships only okay for our understanding this is not the formal definition but for our understanding we need to be first clear then we can you know pick up all the formal jargon and everything. So when linear regression means it is a linear relationship. When we say a linear reg uh relationship it is this. Okay. So as and when this is your age oh sorry this is your salary right? 2:51:14 Okay. And this is what you are predicting here. Okay. And you have only one feature. You have only one feature here and that is experience. Okay. What do you feel will happen? As your experience increases, your salary increases. Is that a fair relationship to keep in mind? As experience increases, your salary also increases. Is that fair to assume? So the relationship here is linear. Okay, it is linear. So what I'm trying to do here is I'm trying to plot all the people here. What is this yellow point here? It is people in data set. In our data set. Okay. And what what what what does it 2:52:17 explain? This tell me this tells me the experience of this person. Okay, this person's experience. This x-axis tells me and the yaxis will tell me this is the salary of this person. Correct. Everyone with me? This basic we know about the graphs. Okay. Cool. All right. So now when when we have linear regression, okay, when we have linear regression, you see this is a linear relationship. I can fit a line here which is straight line. Okay, I can fit a line which is straight line. This is your linear regression. Okay. Whenever there is a linear a line type focus on this word 2:53:19 in linear linear regression is only telling you it is a line type and regression I have solved it for you. Regression means relationship. So when there is a line type relationship linear relationship is called linear regression. Okay. Perfect. Now, now with this what is nonlinear relationship or nonlinear regression? Okay, whenever your data is like this, okay, there is no pattern here. It is not something that is you know you can say that it is uh linear. It is all over the place. Correct? Let's say this is y, this is x. Can I fit line like this? No. Right. So the best fit would be something like this. Okay. This is nonlinear relationship. 2:54:22 Okay. So what we are going to study and what everyone studies is linear regression. And this is what is the basic of machine learning. This is the utmost minimum you should know. Okay. Now let's get into the details and see what actually is linear relationship or linear regression. Okay, what is there in linear regression? Now let's take an example. I told you this is theory but we will try to do it with examples. Okay. Now the problem statement is problem statement is that you have to predict the price of the secondhand car. 2:55:26 It could be you know something like cars 24. Everyone knows what is cars 24 spinny etc. There is a person okay or there are so many people who want to sell their car okay but cars 24 okay cars 24 and spinny they are those companies which buy these cars from the people in the secondhand market. So let's for example I have one car okay I I want to sell it for some reason I don't know where I can sell it to some person right who I know who contacts me but a good chance in today's world in today's India what is happening is companies like cars 24 and spinny they have opened their shop and I can straight away go there without you know waiting for a response on ox or eBay something like that or if there is no 2:56:26 one in my relation who wants to buy my car. I can directly go to cars 24 showroom, spinny showroom and say that guys this is my car. I want to sell it. Will you purchase it? Now let me create a data set. Okay, this is the data set that we'll be using to understand the entire concept of linear regression. Okay. Now let's say I this is the data that I have. 2:57:10 So what all we have? We have mileage, right? We'll have mileage. We'll have the age of the car. Age of the car. When was it bought? Correct. We will have the manufacturer of the car. Correct. Mileage is nothing but average. Okay. We Indians, we love ma average. Maximum average car is the best car. But there are a few who like Ferraris. They don't absolutely care about average. Okay, it's zero or one. Either we will want the one with best mileage or we would want one the that zoops goes 0 to 100 in you know split seconds. Cool. And the other would be engine. 2:58:10 Correct. And let me put this in a different color. Let this be the price. Okay, now we know what what is this? This is your features. Okay. And in Okay, let me first clarify this. And this is what this is your target. 2:58:44 Okay. In linear regression, okay, this is also formally defined as something else. All the features are called independent. Independent. This is also called variables, right? And this is also called target variable. They are independent. And this one is called dependent. Why is that? Now you're finding a relationship. Okay, you have to find a relationship between dependent one who depends on something and independent. Meaning your target variable with your features. In other words, if something is dependent on some other thing, then only you can find the relationship. No. 2:59:45 If they are independent of each other, what will be their uh relationship? Correct? Everyone with me? Everyone understands this? Features are also called independent variables. Target variable is also called dependent variable. Okay, I'm keeping it this for now. Okay, I'm keeping only this. You can add in as many features as you want. Now, let's say the mileage is 20. Okay. Age is meaning the car is already 8 years old and manufacturer is let's say Maruti. Engine is 1,000 cc. Okay. And price is price is let's say five lakhs. Okay. Whatever. Then five lakhs seems to be too much but okay this is example for an 8-year-old car which is 1,000 cc only 3:00:46 and okay the average is 12 age is 7 okay and uh it is let's say Hundai and it is 1200cc this is let's say 4.5 lakhs Okay. Then we have another which is let's say uh the mileage is 9 and the years is it is only 3 years old. This is let's say Kia. This is 1,500 and this is around 10 lakhs. Okay. And so on. I just wanted to give you a glimpse of the data. Okay, this is how it is. Now if we have to understand linear regression. Okay, now if you look at it, your task is to 3:01:47 consider all these columns 1 2 3 4. Okay. to predict your price. You you're given four variables, four independent variables to predict one dependent variable which is target variable. Okay. Now it is your choice to either take one of them or take two of them or three of them or four of them. Your choice but your choice will be driven by some mathematical functions. Okay? Some mathematical concepts. But at the end it is your choice and that choice that uh you know act of deciding which feature to put in which feature to not put in to create a new feature that is called feature engineering. Correct? We just studied it. It is feature engineering. Perfect. So now I will start very simply with one feature as of now. Okay. I will write also 3:02:49 considering only one feature. Okay. Or let me say one independent variable. Okay. We are going to look at only one feature now and I am choosing it to be age since it is very easy to understand because you know as and when the a the car gets older the price goes down. Okay, everyone understands as and when your car gets older it is supposed to it is supposed to have low price. Okay. Now the other fun in this the other thing while plotting this we would have studied this earlier also in school that whenever we are plotting whenever we are plotting okay we have independent 3:03:52 variable as x-axis okay and we have dependent variable dependent variable on y-axis okay keep this in mind So whenever you're creating a graph, this is a very novice uh mistake that some people make on the job and also during interviews. So always keep it in your mind that independent feature independent variable will always come on x-axis and your dependent variable will always be on y-axis. It's it's not something that is told but it is something you know which is intuitive to humans. We can understand a line flying we with respect to y-axis when you have a have a graph like this with respect to y-axis it is visually or to the eyes it is easy to understand okay that's why we put it like that and that's why we are even 3:04:52 taught in our schools also like that okay that's the logic you can do ulta also there's nothing wrong in that but this is how I would prefer you guys to do it okay and that's how even people in industry refer it. So now if we have to create a real uh let's plot the data considering only one variable and I will consider age. Okay, I will consider age. So now in my data I will have only one which is age and I will have price. Okay. So the first example was 20 and this is five. This I am putting in years. This I'm putting in lakhs. Okay. So I am removing this L here. Okay. Cool. And the other one was um okay 8 73. 3:05:54 I'm just trying to copy 8 73. And this was 4.5. And this was I think 10. Yeah. And there are few more. Okay. There are few more. It is four. And this is around three. And it is you know five. It is around six. Okay. Now there are a few exceptions here. Right. There are a few exceptions here. Right. This one it is only 3 years and it is 10 years 10 lakhs correct because others are in the range of 4 5 7 8 okay but anyhow we will still consider it you are not going to do anything with it since this is a dummy example okay now when I try to plot this don't you think I'm not considering the 3:06:55 numbers but you get the idea of Don't you think this will be your age and this will be your price? Okay, this will be your price. Lower the age. Lower the age, higher the price. Correct? Then lower the age, higher the price. Lower the age, higher the price. Then as and when your age increases okay the price will keep on coming down. Correct. There might be some outliers as well. Correct. Is that fine? As age increases of your car the price will come down. Correct? So this will have relationship like this. Is that fair to understand? 3:07:59 Okay. Now instead of age I take average or mileage. Let me do that here as well. I'm doing this for mileage. This one was for age. Okay. This is your mileage and this is your price. Now what what will this relationship look like? Same as age correct. Okay. So now these type of relationships okay where it is you know uh age is increasing this in terms of line in terms of line this is having a negative slope correct it is moving like this 3:09:02 it is it is a negative slope correct if you have a graph like this where A line goes like this. It will have a positive slope. Correct? So this is negative slope and this is positive slope. Everyone understands this? Okay. So now why I spoke about slope? Do you know why? Do you know why? We have linear relationship and it is a form of a line. Correct? And the way how you can define a line is nothing but y = mx + c. Okay, this is your slope. This is your intercept. Okay, this is your independent variable. You see how y is 3:10:04 dependent on x? That is why it is your independent variable. Okay. And this is your dependent variable. Okay. How y is dependent. Okay. Look at this. Now y is being defined. You can you can get the value of y if you put some value of x here. m and c are going to be constants. Correct? For a line, for a line, this is some slope of it. Let me explain it separately. For a line, okay, this is the slope of it. It is going to have some fixed value. It is going to be fixed. Correct? And this is the intercept. This is the C value. This is also fixed. Okay? Now to get any y value to get any 3:11:05 y-value on the y-axis if you keep replacing some value of x here you will get the corresponding y- value correct. Similarly here if let's say this x is equals to 5 and you plot it here and then you project it on y-axis you will get the value of y. Correct? Y is defined like Y = MX + C. In other words, if you have to, you know, understand it this way, these are mathematical functions. We always say y is equals to f ofx. Correct? It means y can be explained or defined as a function of x. Okay, there is some function you put some value of x here and you will get value of y. Okay. So this is also nothing but a function of x. A function of x which defines y. That is why it is dependent. 3:12:08 It is depending on your x. And didn't I tell you that in industry we keep all the features as x and all the target variables as y. Right? What is intercept? Wherever your x value is zero, x is equals to 0. And wherever it crosses your y, that is your intercept. Without any value of x, you will have minimum this much of amount. In our car example, in our car example, okay? No matter what, no matter what, you will at least have to pay 1 lak rupees. This is your intercept. Okay, at least you'll have to pay if even if you know it doesn't have any engine, it doesn't have any manufacturer, it doesn't have any mileage and you you have to you know sell it and cars 24 are 3:13:08 bounded by uh purchasing it. They will still purchase it. If if one lakh seems to be higher, let it be 50k. I'm just for simplicity, for calculation, I'm saying one lakhs. It is the cost of scrap that you'll be getting. I'm just relating it with the example. But it's like if there is nothing at least we will see this value of the target variable or dependent variable. Okay. Okay. So this is desmos. Okay. Now here let me add a few values here. Okay. Let's say uh we had four and five, 8 and 10, 6 and 12, and this is 5 and 13. This is one three. Okay, these are your data points. 3:14:12 These are your data points. Okay. Now, linear regression is what I want to put more examples. Let me at least we will have something to talk about. Okay. So, let's say this is what we have as an example. Okay. Uh I'm not telling you any linear concept, linear regression concept. I'm just telling you an example of this thing line. How lines looks look like mx + c. Okay. Why is it going there? This is the uh equation of the line. Correct. This is the equation of the line. Okay. Now we have a basic thing set up. Correct. So now let me play with this. Let me try to see changing C. You see it is negative. What happens when it is negative? It touches your 3:15:14 line or it cuts your line when x is equal to zero at a value which is negative. Okay, we still have the slope as positive. Positive mean one. Correct? So we we understood that the positive will look like this and negative will look like this. Correct? So if I keep on decreasing it, you see how if I keep on changing the value of C is the direction of the line changing is just that where it is moving on Y-axis is changing correct correct okay now let's see let's see let let us put it to zero now this zero meaning it is still a it could still have a linear relationship ignore the points here. But if if there is a linear relationship, it could still have that linear relationship. But when x is equal to 0, y is also equal to zero. Okay, that is when c is zero. 3:16:17 Now if I change my m, if I change my slope, how do you think the the line will react? Let me make it more positive. You see is it ever crossing this line? It is still in this. It is moving like it is rotating. It is but never coming like this. Okay. This is for all the values of uh m and if I make it negative you see how how the rotation has changed when I am at m is equal to0 it overlaps your x-axis nothing but m is zero means there is no slope so it is x-axis correct now if I make it negative you see how it is changing now it has become is 3:17:18 correct. So lines can take any value. Lines can take any value on that plane. You can keep on changing your C. You can keep on changing your Y and you can have any combination of this. So M and C can take any values depending on the relationship. Is that clear guys? C is the intercept where it touches Y or the value of Y when X is equal to zero. Okay, that is the definition of C or intercept. M which is slope is nothing but it is the slope at which the line is formally it is the change in Y divided by the change in X. If you see this, okay, if you see this this triangle, this is the change. If I have to get the 3:18:20 slope, I will see that the change is this much and the change on x-axis is this much. It is the slope. Okay. So basically what happens is in linear regression your motive is you need to find the values of find values of M and C such that you get the best line. Okay, keep this in mind. We will come back to this. We need few more other concepts to understand this. And this is best line. Best line is one that will perfectly fit your data. Keep keep this in mind. Okay, for now keep this in mind that best line is that will fit your data to the best of its ability. Okay, 3:19:22 now we will go into what is this best line? What is this? Okay. Now what happens is let's take this example again. Let's take this example. We have all these points. Okay. We have all these points and you are trying to fit a line. Correct? Why not this line? How do you say which is better? Okay, what about this line? Okay, now I'm telling you, you have these options. This is line number one. Okay, and this is your line number two. This is line number two. And this is your line number three. Tell me which is the best line. 3:20:22 which identifies the relationship between your data sets really well. So there is some confusion between one and two. Some confusion between one and two, right? But there is absolutely no confusion about three. This is worst. Correct? Everyone understands this. This this is the worst line. It cannot happen. Everyone with me? But we are confused between one and two. One or two? We don't know. How will we know? How will we know? We'll know about it by checking the errors. Okay, by checking the errors. 3:21:24 Now when I say error, what what do I mean by error? Okay, let me take another example to explain this first clearly. Okay, let's say this is your data set. Okay. Now I'm doing this with a positive correlation or sorry positive uh slope so that it is easier to explain. Okay, the same thing applies for the negative slope also. Okay, now this is my line for example. This is the value of this is one line that I feel can fit my data really well. I don't know yet. I don't know yet. Okay. Now what happens is what I say that every point on this line is the prediction because my function is mx + 3:22:24 c. Correct. Whatever value I put in x I will get the value of y. Okay. Let's say this is 1 2 3 4 5 6 8 Okay. 1 2 3:22:54 Okay, this is the graph. Now for this point for this point the one that I'm highlighting okay do you see that it is far from this line correct and this one is also far from the line this one is also far from the line but this one is the the red line here is really small the red line here is small here probably there is no red line And there are a few points which are on my line which are zero. The red line is not present only for them. Correct? What are these red lines? These red lines are nothing but your errors. Why? Why are they errors? Because if you notice it correctly, yellow point is 3:23:55 your actual value. Yellow is your actual value and all the points that are on line because you're saying that this is your you know uh function or expression of the line. You're saying all the ones that are on this line. Line is a combination of multiple points together. So all these lines are your predicted values. Correct? Every point you are saying for this value of X this should have been the Y value. This is your predicted value and Y cap. Okay. But in reality this is the point related to this. This is the point which is your actual value. So this becomes your actual value. 3:24:58 Your actual value is this. This is your y. And the difference which is coming from this which will come from this is nothing but error. Does that make sense? Does that make sense? So error is nothing but error is nothing but your actual minus predicted. Correct? Red line is not standard deviation. Red line is how far is your actual value from the predicted value. How far is your actual value from predicted value. So you have predicted this value. Okay, for x that is this. Okay, but in reality the actual value is this. Correct. So you have predicted it to be 3:26:00 this value. Okay. But in reality it is this value. This is some value of let's say 8 and you have predicted 6.5 for example. Then 8 - 6.5 is what? 1.5 is the error. You are off by the actual value. Your prediction is off from the actual value by 1.5. Correct? Okay. Now there's one little thing I want to just highlight. One thing that you need to make sure. So your motive is that you have to get the lowest error. The line which has lowest error is your best line. Is that clear? If your predictions are very close to your actuals, don't you 3:27:00 feel your error will be the lowest? And if the error is lowest that's your best line that's the line. So here between 1 and two the one which has the lowest error we don't know. We have to calculate it. We don't know the line which gives the lowest error is the best line and is the regression line. Everyone clear with this? You know intuition should be good visually what is happening. We'll go into the practical example. We'll type it out. You know running linear regression is only three lines job in code. It's only three lines. Not these lines but three coding lines. But understanding what it is doing internally is really important. Okay. All right. All right everyone. I hope this understanding this intuition if it is clear no for linear regression. I promise you it is going to be a 3:28:01 cakewalk. I promise you. Go back. Okay, I need only two things from you. One, trust. The other consistency. Go back. Refer these notes. Refer the original notes. Okay, the Ppts. Refer the internet. Come back with more clarity. Come back. Ask me more questions. And that's how we are going to learn together. Okay. So imagine we have this graph again. Okay. I will plot some lines with other color. 3:28:56 Okay. And now let's say let's say my predicted line is this. Okay, my predicted line is this. Now what does this say? This says that for this is my x-axis. Let's assume this is your age only. What age will now I have created the other thing. Let's take it as mileage because it has positive slope. It is having an increasing relationship. So I'm not putting my x-axis as h. Okay, perfect. And this is your price. Okay, so now what does this say? Now this is your actual value, right? This is one of your actual values. Now what is your prediction? Your 3:29:57 prediction is this. So for this data point for when your mileage is let's pick one H. So when your mileage is let's say this is 100 here. 100 is too much. Let's take 10. Okay. When your mileage is 10, your actual value says this. Correct? If I plot it here, this is your y. We looked at the uh you know certain things that we need to keep in mind. Y tells me actual value. Okay. And ycap tells me predicted value. Correct? You've covered this. You've covered this. But you have said this is your line. So given this value of X, 3:30:57 this is what you're predicting. Correct? And if you plot it here, you need to know the price of the car, right? You need to know the price of the car. So this is your Y cap. Okay? And this line or in other words here, this is your error. Everyone with me? So error is what? Error is actual minus predicted. I will write it here also. Actual minus predicted. Okay. Now don't you think that all these this is one error. This is one error. Now you might have this error also. This error also. this this this and there are some errors on the opposite side as well. this this right? So you're making errors on each 3:31:58 prediction. Correct? You're making errors on each prediction. All the errors that are here will be positive and all the errors here will be negative. Can you tell me why? If you look at the y-axis for this one, the value let's say is 1,000. Okay, I'm just saying 1,000 and predicted would be 900 because this is y-axis, correct? And for another one, let's let's take this example. So your predicted was this. Oh sorry this is your actual and your predicted would have been where it touches the line your yellow line and where it touches your so let me put this as two let me put this as one and one what would this value be this is a higher value ycap is a higher value here 3:33:00 let's say this is 50 this would be 40 correct Okay. And what is your error? Actual minus predicted. So in the first case, actual is 1,000. I'm taking first example. Okay. Let me put it here. 1,000 minus 900. This would be 100 of an error. Let's call this error one. And this one which is second would be what would be 40 - 50 - 10. This is your error 2. You understand that we are getting positive and negative errors. This is key concept. Are you able to understand? We could get positive errors. We could get negative errors. Let me explain in short how can we get predicted value and the actual value. Please check my question above. Some says le square 3:34:03 method. What is ycap? Okay. So we covered this summer but we didn't cover this huge in a huge uh you know it's a small concept in notation in in whatever material you study outside you know even if you have to refer technical documents or notebooks or different books by different authors this is the notation followed anything which is y is your actual value anything which is you know this y cap this this you know on the top there's a cap a hat it's also called y hat and we we used to study this in in our schools that is uh referred as predicted value okay that's what I'm mentioning here okay let's say there is one equation okay this is the equation y mx + c this is the line internally that is what we are doing we need to know m and 3:35:06 see correct that's the whole purpose what we are doing is we are trying our best to understand internally under the hood how is the algorithm working how it is able to calculate M and C okay so given you are able to identify M and C let's say M is 2 and C is 10 for example okay now forever for whatever value that you put in X you will get one value of y correct you will get one value of y so your equation would be this now here it will become prediction that's why ycap so it will be 2 * x + 10 if your this is your mileage for example okay this is your mileage now if if you say your mileage is what 10 so 2 into 10 + 10 meaning what? 30. So you're saying 30 3:36:07 lakhs would be your prediction. In actual you don't know what is the actual value probably the actual price. Actual price which is y actual price of the car with you know uh mileage of 10 would be 25 lakhs. So this is your actual value. This is your predicted value. All your predictions will be on this line. All your predictions will be on this yellow line. These are all your predictions. Every point corresponding to an xaxis lies on this line. To know its value, its price, you need to project it here. You need to project it on the y-axis. That's that's how you will know the price, right? We have price here. Okay, cool. So now we know that we might have 3:37:10 positive errors, we might have negative errors. Now don't you think we know that your best line is the one which is going to have lowest error. Correct? Best line is the one which is going to have the lowest error. Yes or no? So that lowest error. Why? So that if my line is able to overlap all the actual values. Let's say if your actual actual value was like this. If you had data like this, you know like this. This is your actual value. And if your line perfectly crossed all of them, will the error not be zero? It will be zero. Right? And that will be the perfect perfect line that all your actual values lie on this line. 3:38:14 Correct? But in reality that doesn't happen. In reality it's absolutely you know difficult to find such a beautiful data that everything is you know uh overlapping with the line. So that's why you will always have some error. So that is why the whole idea is that we reduce the error that we reduce the error. The whole idea is that we reduce the error. Okay. lowest the error that is your best line. Okay. Now, don't you think if there are some positive errors, there are some negative errors, they will cancel each other out? Yes or no? If there are positive errors and negative errors, will they not cancel each other out? Correct? So, that is why what what what can you do as a solution? 3:39:16 What can you do? You need to understand why we took root mean square error. Why? There is a why to it. Once you understand the why, then you know remembering that thing is going to be piece of cake. So there are two things that I uh I'll tell you that comes to my mind when you know you haven't uh read this at your end. One way is take absolute Correct. Take the absolute value. Take absolute value of y and take the absolute value of uh this thing. Or in other words, let me write it this way. Take absolute value of actual minus predicted. Correct? Now this is for one value. This is for one. If you do a sum of it, it will be for all the values, all the 3:40:18 data sets. Let's say all all the data points. Sorry. Let's say you have how many points? You have I is equal to 0 to n 1 to n. Correct? One error is this. One error is this. The other error is this. The third error is this. You have to calculate the best line based on the entire data set. Correct? So the errors have to be summed and there there have to be a a concept of averaging it out. So what you're doing here? You will divide it by total number of values that you have. Okay. What is this called? This is your mean. This is your absolute. Correct. Mean absolute. And what is this inside? This is error. 3:41:18 So this is mean absolute error. Okay. Okay. This is your mean absolute error. Now what do you think will be mean squared error? It's very easy. Now if you understand you're able to identify all three types of errors very easily. Now the other option was instead of absolute you take square that's what Lakshmi also said. So instead of this absolute what will happen it will become 1 / n summation why are we doing summation for all the data points and we will do actual minus predicted square this is what this is your MSE mean squared error what's the difference absolute square instead of absolute if you take square you'll get mean square 3:42:19 Okay, everyone with me? The third one which is very popular and mostly that's what in the industry people use is RMSSE. Now you already know MS MSE part in this RMSSE correct. So what are we going to do? First we are going to calculate the mean squared error actual minus predicted square and then take its root. What is this? Now will you ever forget absolute error mean absolute error mean squared error RMSC will you ever forget? No. Right? It is very intuitive. People who have made this they also thought like us. It's just that no one tells us that what they were thinking 3:43:19 when they devised this error could cancel each other out. So there are only two ways either take either you take you know absolute or you take the squared error. Okay. So the story starts from here. Now what could happen is based on whether the x x point the actual value is above the line or below the line you could have positive error or negative error. Correct? This is what we looked at. This is what we looked at. This is the second data point. This is the second data point and this is your first data point. So the first one is going to have positive error. Why? Because we are assuming that the actual value was 1,000 and the predicted value was 900. How we say y was higher than ycap or yhat is because it is looking like this only. Right? When you 3:44:19 project this point on y-axis and the point on line on y-axis, you will get these values. Yes. 0 and it is increasing. And similarly for the second point which is here this point if you plot the actual value is lower than the predicted value. Correct? So you'll get a negative error. Now the reason behind summing it is now you're looking at the errors one by one. Error 1, error two, error three, error four, error five, error six, error n. And how do you know what is the value of n? the number of data points each row each observation is your row and is your data point. So as you need to find the overall error of the data set not only for one observation correct. So you need to find the error of all the observations. So you need to take a sum 3:45:21 and with the idea of taking that sum for one value then second value third value fourth value there is a chance that these errors will cancel each other out. So your idea of finding the best line will be flawed. Why? You will get zero error. This is the line that you're looking for. Correct? This is your best line. But if you don't take the positive errors, if the you know uh things will cancel each other out, what will happen is this data set and this line this will be your best line. Or in other words, if I have to give you an example of the slope, if this is your data set and this is uh you know how it is looking like, this will be your best line because the distance here and the distance here is 3:46:22 exactly the same. It will cancel each other out. This is not your best line because you're predicting every time less or high. Correct? So this is the flaw. That's why you have to take sum and you have to take the you have to nullify this cancellation. You have to nullify this cancellation by you can either take the absolute value of the error. That's what we've done here. Or you can take the square. Okay. actual minus predicted square it will not be negative. Can square be a a negative value? No. Right? And then in industry we use RMSC. What is the why we have put this root? You need to understand we understood absolute why it is there squared may Y square is there. Now we need to understand why was if you already had 3:47:24 this MSE why did you wanted to you know put this root on top of that. So the idea was when you take squares now the error is so high and and a value which is really par let's say your value is here and if you take the error right it contributes really high when you're finding that value when your line is trying to adjust what will what it will do is it will try to fit a line like this so that the error on this item is really less but that does not necessarily give you the right line that is an outlier. Okay. So to adjust all those values you are taking a RMSSE so that it is comparable. Okay. Perfect. So now we have understood the errors. Now we have understood the errors and the whole idea is that we 3:48:27 need to minimize our error of the line and we have to tilt it this way. So we had looked at this right. 3:48:43 Right? So this is your line. If you change M, it rotates. Correct? It rotates. And if you change C, it moves up and down. And when you change m it rotates. So the whole idea is that you need to rotate or that rotation and that cutting of y-axis should be in such a way that you minimize the errors. Okay. So that we don't give so much value to some values. So the entire mean squared error will be really high because of only one observation for example. So you need to find a way to nullify that a a error coming from a really one value to nullify that you're doing root. Okay, perfect. And m here is slope and this is what it does. Let me 3:49:45 change this value so that it is in the window. And this is your m. Okay. Now I'll tell you if you look at this no you'll be like yeah linear regression so simple linear regression 3:50:11 H see are you able to see this what is it doing it is changing the C and it is changing the M based on these distances. Correct. Let me try to zoom in so that everyone is able to see this properly. Okay. 3:50:50 Hm. See this is the iteration it is doing based on your data set. It will rotate and it will fix at that value where the error is minimum. This distance is D1 D2 D3. We were marking it as what? Um error one error 2. Right? Let's look at the other one. Is there a GIF here? H see it is looking at the mean squared error. This is a better example though. You see the linear relationship. The line will start from there. You see the mean squared error is going down is going down. Right? Then when it reaches its minimum. Now at this point if it crosses if it crosses there the slope changes more the error will start increasing. 3:51:50 Okay everyone with me? Now you understand with one variable only how you will find the M and how you will find the C. Correct? with one variable. Is the intuition of linear regression clear? That's how you're going to do linear regression. That's how all these packages inside the algorithm that's how they work. Computer has power. So you can say that this is my data set. Now fit me a line. It will fit that line in such a way that you get the best M and best C. And when I say best M and best C, it means those values of slope and intercept when your error is minimum. There could be multiple lines. We looked at line one and two. Now you know how to decide which one is better looking at these metrics. 3:52:50 Okay. So look at this. I think it is easier and understandable from here. Now let's say the error is reducing right when this line is going. Now this is your final error. It will stop somewhere where the error has reduced. Correct? Now if this red line tries to go somewhere here, okay, it is moving like this. When it starts moving like this, the error on this bottom part will increase. Let me try and explain it here. So let's say this is your graph. Okay. And all the points are like this. 3:53:47 Okay. When this is your line, let's say this is the line best line. Okay. Now if this is one here, this was your second line. The error here would be more. This one, this one, this one, this one would be more correct with two. Now if the line moves here, if this red line moves here, what will happen? It will look like something like this. It will look something like this. Correct? Now the error will still start increasing. It will increase. So in both the directions there will be some errors. You need to find this red 3:54:48 line where you minimize the error. All right. Okay. So now we know how to find the best line with one variable. Okay. Now I'll tell you one story. All the statisticians and all all uh these people they have one uh thing in common. Can you can you imagine a a 4D graph or a 5D graph, 10D graph, 10 dimension, seven dimensions? No. Right. The max that we can visualize is three-dimensional. Correct? So now let me explain you this with two features. Now this is going to have your dependent variable price. Okay? And this is going to be your age dependent uh independent variable. And this is going to be your mileage. 3:55:50 Now in 2D we were fitting a line. Correct? Now the values could be somewhere here. Correct? Now instead of fitting a line you'll be fitting a plane. Now you cannot fit a line. You'll have to fit a plane right in 3D where all these points lie and the error is really less. It could be any plane. In three dimensions, it is it is going to be a plane. If you go to more dimensions, it the shape of that you know line or plane is going to change. So we say that it is hyper plane. We don't know it is a threedimensional, fourdimensional, fivedimensional. And how do you decide this dimensions based on your input features or your 3:56:51 independent features? If you have 10 independent features and one target variable, target variable is going to be only one. Okay? If you have 10 features, you have to minimize the error in 10 dimensions. One is age, the other is mileage, other is engine. then manufacturer then some other value kilometers driven someone was saying imagine can can you even visualize no right no so that's why people call it as hyper plane it is more than 3D 3D you can say 3D plane yeah you know this is cuboid or cube somewhere but above that it is a hyper plane. Okay, everyone says it as hyper plane. Now tell me one thing since this is not a line. Now your ycap which is your prediction 3:57:53 will be what? So now there comes a concept of weights. This is beta 1 x1. I'm saying beta you can assume it b also. This is beta_2 x2 plus beta kn. This is nothing but your intercept. Assume it that it is intercept. But this is something that without any uh input any value of x this what is x? x1 is age x2 is mileage and there is some weight of the age and there is some weight of the mileage which is beta 1 and beta_2. Okay, you need to now find the value of beta 1, beta_2 and beta 3 so that you have the best plane. And how do you find it? Concept is same error. Reduce the error, minimize the error. Sometime in text they also refer to this as weights. 3:58:54 So you might also see some uh text some uh you know uh posts or blogs saying this as weight 1 x1 plus weight 2 x2 plus weight zero. Okay, this is with two independent features. And when this is n, there are n features in your data set. N independent variables, then this line will become what? X2 + X3 + W N XN + W. Both these uh notations work either beta. So I will write it here. Beta, 3:59:57 w or b. All of these are nothing but weights. And in other words, technically they are called coefficients. 4:00:19 Whenever you go into the interview, you can either say weights but it is good if you say coefficients. Okay. All right. So coefficient or weight tells what is the contribution of that uh feature. Okay. Let's take a hypothetical uh equation. Let's take hypothetical equation which is this is your prediction. This is your prediction and I'm assuming only two features. Okay, two features. One is age and the other is mileage. Okay. One is age, the other is mileage. And this when I say uh two features it is independent okay two independent. Now let's say this is uh W1 into H 4:01:24 okay * H plus W2 * mileage plus W kn if you have to understand it very intuitively hear to me what I'm going to say next okay this is not that important but it will help you in understanding to derive when you're in in the projects to explain your concepts to a person who doesn't know linear regression. That is part of the job. Trust me, that is part of the job. If you are going to explain this to someone, this is going to help you and is going to help you leaps and bounds. Listen to this with undivided attention. Okay? Undivided attention. 100% focus here. Let's say W1 is 100, W2 is um 200. Okay. And W not is 4:02:31 50. Okay, it's 50. Now your equation will look like this. 100 * h + 200 * mileage + 50. Correct? Now if you change your value, you keep everything constant. You change only the value of h. Change only age. Keep everything constant. What is going to happen? Your y new will increase by a factor of 100 is directly proportional to 100 * h. 4:03:36 Correct? I'll give you an example. If your age was 2 years, okay, this should have been negative. To make it sense, I'm changing it negative. Okay, to make more sense, I'm changing it. You will understand why I'm changing this to negative. If your age is two and if your age, you increase the age by one. Here I'm keeping everything constant. Here y would have been 200 - 200 and here it would have been -300 correct. So the increase in age increase in age we know the slope is going to be negative right and this is going to be positive. This is where the sign is coming in. If you increase the age from 2 years to 4:04:37 3 years, the price the impact of that increase on the price is going to be a factor of 100. It is going to have an impact on your prediction or the price of the car by 100. Okay? If it was let's say 500 at 2 years of price 1 year down the line 3 years it is going to become 400 assuming everything is constant understood how do you decide the coefficient is 100 or 200 it's is the slope right is the slope so if the relationship is negative here let me draw So this as and when your age increases your price is going to go down. As a age increases in your secondhand 4:05:39 market as and when your car gets older the price of it is going to go down. Correct? So the relationship is negative and the slope is negative. Relationship is negative. And if your mileage increases, if your car has a higher mileage, the price is going to be more. So the relationship is positive. Here it is going to be negative. That is why these values positive for mileage, negative for age. Okay. So independent variable is in our case independent variable is your target variable. It is uh sorry it is your dependent variable. I'll repeat that 4:06:41 dependent variable is your target variable. It depends on certain other features. So your price of the car depends on other uh some other features like age of the car, manufacturer, kilometers driven, engine it has the mileage it gives. Okay. So mileage according to your problem now is independent for prediction of price. So you can predict the price of the car given you have mileage. You can predict the price of the car given you have age. Okay, all these are called independent variables and your target variable is called dependent. Okay, how can we reduce error in the uh in whatever happens in in the system right in algorithm generally it follows brute force. Generally it follows brute force. brute 4:07:42 force as in it will try to do all the iterations and then it will find the absolute value or the correct value. But since we are doing machine learning, we are smarter than you know executing every all the iterations all the 100,000 of them and doing it in once in machine learning. That concept of finding the best error is gradient descent algorithm. in small number of steps or in an efficient way you can find that error. The minima or the maxima of the curve. Okay, that is gradient descent. Gradient descent is something that helps internally underlying everything that this is how you know you can find the next error, next error. It is basically differentiation. Okay. So these things you can go and explore and I will definitely recommend that go and explore. The more you read, the more you read, the benefit it is 4:08:43 going to have on you and your learning is going to be really good. My job here is to give you the easiest explanation so that you understand the concept. Once you understand the concept, you've taken, you know, you've created a base. Now you want to uh create a fourstory building or five-story building or a singlestory building that is up to you. Okay. You want to create an antila go read, explore, burn your curiosity. Okay. So this will confuse a lot of people uh who don't have maths background. Okay. So gradient descent basically uh I am touching upon some mathematics. So guys if you're not clear with this this is not required. Okay this is not required. I am highlighting it. If you're not able to understand this don't worry. 4:09:45 So in general what happens is 4:09:55 this is how the errors look like. Okay. In in any graph even forget about error. Okay. In reality all the graphs look like this. This is your maxima. Maxima meaning max value. This is your minima. This is again a maxima in this region. In this region in its locality it is maxima. Correct? Even though it is lower than this. Okay. That's why we call it as local maxima. In its locality it is maxima. In its locality it is minima. In its locality it is minima and so on. This is local minima. This is local maxima. Now this since in its locality and 4:10:57 overall also this is maximum value. So this is called global maximum. Okay. And this value here is the minimum value is the minimum value. This is called global minimum. Okay. Now the idea is for algorithm the idea is that you need to reach here. You need to reach here if you are trying to reduce the error. If your problem is like that that you need to reduce the error, you are looking for global minima. If your problem is that that you're trying to maximize it, like for example, uh you are trying to maximize your uh car price when you're selling it. Correct? Your motive is to maximize it. When you're trying to increase your profits, it is you're trying to find the 4:11:59 maximum. Correct? That is global maximum. But when you're trying to reduce your expenses, your expenditure, you're trying to find this global minima. Okay, this is clear. Now let's look at now let's zoom in only at this part. Let's zoom in only on this part. Okay, I am pointing it here that we are zooming in. Okay, how will this look like? If you zoom in, you're going to look at this like this. This is how it looks like. Correct? You are now in pursuit of this local minima. So what happens is you start arbitrarily from here. You start from here. You set the values in the gradient descent. You set the values of your 4:12:59 learning rate in such a way that after one iteration, this is one iteration. Second iteration, third iteration, fourth iteration, third, fourth, fifth, sixth, you try to reach here. When you have done this, when you have done this, you have achieved your motive. Okay, this is in short that I can explain. You can imagine it this way also that imagine you are um coming from a hill down. Okay. On the top when the steep is really sharp, the uh the mountain is really steep. Your small steps you know that you know you are going down. So your steps are going to be really huge because this is the change in error. This is the error that you're looking at. This is your error. Okay. So the idea is that you need to 4:14:01 find this and that's how you do it. How you do it? This is the gradient descent algorithm. It has two parts. It has two parts. One is the direction where it should go. The other is steps. Step meaning it should be long steps or it should be small steps. Okay. The direction is you need to go here or you need to go here depends on that. Okay. Steps is also sometimes called learning rate. The idea is that you need to find the global but in real world what happens is you are not able to find the uh global minimum or maxima every time. Most of the time you'll be stuck in some locality. You'll be stuck in here or 4:15:02 here. Okay? So you try to find the local maxima or the local minimum. Okay. And that's that's the beauty of maths and that's the beauty of the data that even if you find this the effort that you have to do from reaching this local minima to this global minima is going to deteriorate or is not going to be that much of effort then you're going to get the benefit out of it. Okay. So everyone is happy with this. This might give you 90% accuracy. This might give you 90.002% accuracy. And the effort that you have to do from reaching to here is really a challenge. I'm not saying it is not doable. It is doable. But in reality, u it it is you know kind of overfitting. So in future if your data changes 4:16:05 this will do a better job rather than this. Okay, no more going into deep. This is pure mathematics. That's the best that I could explain in short. Okay, I will write it here so that no one confuses when you're looking at this GDA. This is not required. Okay. 4:16:49 Okay. You can understand it logically, but if you need to go in detail, you need to cover the mathematics around it. Perfect. All right, guys. So this is what this is linear regression and so on. Okay. Perfect. All right. So I have uploaded my data set here. Now let me So what do we do? First thing comments make it on the go while you're typing in because no one comes back. You might think that you know I'll put in the comments no one comes back and it becomes difficult in industry follow it. I also follow it as in when I'm writing at least for my clarity I forget a lot. So I write the comments as it is on the go importing 4:17:52 dependencies. So what do we import generally pandas as PD and I typically import these three without any this thing u without even thinking about it mplot lib dot pip plot ht okay I end up using these three every time. So I you know load it on the top and for this session at least since we are going uh you know uh bit by bit I will try to import the dependencies where they're required. I try to then you know once I do that I try to keep everything together. I'll show you how I do this. It's a good practice to follow at least for myself that's how I follow it. Everything that I'm loading is in 4:18:55 one place and everything that I'm exploring in is in one place. So let me start with this first. Okay. Now since we have this already, let me put in a few cells. H take. Now what are we going to do? We are going to first read the data. Where is your data? Your data is this. Correct? So PD dot 4:19:25 my keyboard is giving me a problem one is the name CSV. Then I try to do as soon as I read it to make sure that I have read it. Uh we need to run this first and then this first. Okay. So now this is your data set. I'm just making sure everyone is on not having this thing. Yeah, virtual will take some more time. Data imported and visible. Perfect. So now let's understand what this data set is. Okay, this data set is basically the problem is let me try to give you the idea of the problem. Okay. So the problem is we have to predict the our favorite problem house price. Now this house price what we had looked 4:20:27 at during our examples was you know for explanation it was an easy to understand data set. This is real life data set. Okay. So this is being maintained by uh by us u it it is the actual data that they have covered and recorded for you know their purposes and even for these purposes. So they have they've kept it online. Uh let me show you what is this. So Boston data set wiki is what we are looking at. H this will also work and before you start any problem. Okay. Look at this. This is Boston. This is a region in um United States 4:21:27 Massachusetts. Okay. So there are multiple towns inside this Boston. This is how it looks. Okay. And this is your Charles River. This is the river that flows through it. And it is the Charles River. This one. Okay. Such a beautiful city. Right. Now, now since you've done that, it's always good to you know have that external factor. Machine learning machine will also do. But have that idea as a human what are you doing? Okay. And this is a real data set. Okay. That's that now let's look at these columns. Now I've gone through the description of of this data set. Okay. This is where you can look at the data set. What is it doing? Okay. Now this crime is telling you the the crime rate of the of of the 4:22:28 Boston area. Okay. Now imagine when we were looking at the prices, we were finding number of bathrooms, area size. Yes, they are there. But in reality there are so many things correct. You need to know the crime rate. It it might affect the price, right? Don't you think an area which is having high crime rate people will not buy there probably the price will be less. Okay. Then this is u another proportion of the zonal uh the lands which are above 25,000 in the zone. The number of those lands which are owned by people which are above 25,000 ft². This is industry telling that you know how uh proportion of non- retail business per acre town. Okay. So what is the proportion of the industry think it that way in the town. Then you have chas it is a simple column 4:23:29 zero or one. Okay. Which is telling you that is it close to the Charles River. I showed you the Charles River here, right? Okay. So I showed you the Boston anything that is closer is is on the the area which is on the banks of the Charles River is your um one else it is zero okay that's how it is and let me uh this one any area which is closer to that is one and this area which is not closer to it is zero okay then we go into Nox now this is a proportion of nitrogen oxide in in the atmosphere that is harmful. So they are noticing this is 53% 46% and so on. Okay. Then RM what is RM? RM is the number of rooms. We had also considered this. Yes. Age age of the house. Then 4:24:29 this is distance. Okay. This is the distance from the five uh employment regions in the uh Boston. What is its weighted distance? Weighted as in what is the it's a weighted average distance. Okay, that's that. And then this is radial distance. The tax that is paid. Then P ratio is basically uh PT ratio is basically per people how many teachers are there? What is the ratio? Okay, 15 is to1. So every 15 people 15.3 people one teacher is there and this um now this data has been recorded way long and you know they used to keep this track number of black population or black black people in in a specific area. Okay that's the count and this L stat is uh let me refer what is this LTA. Huh? Okay so people with lower income they are calling it as Lat. Okay. And medv this 4:25:33 is your target variable. It says median value of owner occupied homes. Okay. What is the mean uh median value of houses owned by people? Okay. Now this is in $1001,000. So 24 here means 24,000. Okay. Cool. Everyone understands the data set. And this is our variable target variable medv. And all these are our independent features and medv is our dependent feature or target variable. Correct? Now good thing to do every time you load in the data is first understand the explanation of each column because that might be helpful um not only from the mathematical sense that you know this is categorical or this is you know continuous not in that sense but also 4:26:35 kind of u you know business-wise what would be more important what would be not more important okay It is always to have that. It is not necessary to do that but I like to do that. So I would like if you also followed that. Okay. Just have a human from a human perspective try to understand the practicality of the data set. Okay. Cool. Now uh we see this is 0 or one kind of a flag or a categorical variable. This is also an index. I think radial is is not distance as I said is the index. Okay. So index of accessibility meaning it it would be like the one if if it has value one it is uh really accessible. If it has two it is somewhat accessible. Five would mean you know really not accessible. Something like that. It is an index. Okay. Imagine it to be something like high low 4:27:36 medium kind of a thing. Okay. Then rest others I think we have ratios, we have counts and all it's a mix of a data set. So what I try to do generally is I try to look at the head we've already done. I like to do this describe. So what it does is it in a nutshell gives me a lot of information about the data set. Okay. Now this tells me 506. 506. Everything is having 506. Correct. This tells me the count of uh values that this has or the rows that that column has. Crime has 506. What is the mean value? It is 3.6. Okay. What is the standard deviation? 8.6. Minimum value is this 25 percentile. 50 percentile 75 maximum 4:28:38 value. Okay. So in a nutshell it tells you a lot of things about the data set. Okay. What I can quickly gauge out of this is that mean is around three. So on an average the crime rate is around 3.6. Okay. And there are a few let's say 50% of the houses 50 percentile means 50%ile or 50% of the houses in this data set are in a locality which has really low crime rate really low.25 average is 3.6. Okay. So it tells me that 50% of them are having really low crime and 50% of them are having really high crime. Okay, above 75% meaning close to uh the maximum value is this 88.9. This is 77 uh 75 percentile. It means the third quartile. You must have learned it in 4:29:39 your statistic classes. 77 75% of the values in the data set are below 3.67. Okay. 50 percentile tells me 50% of the values are lower than 0.25. Okay. Perfect. And if I look at my target variable, the average price is around 22.5 in 2000. Okay. And the maximum value that it can take is 25 uh 50. This is the maximum value. 75% of the houses are having what price lower than or equal to 25. Okay. So this gives a lot of information about the data set and even prior to this people also tend to do this data dot shape to find how many rows and columns are 4:30:41 there. It tells 506 and 14 14 columns are there. Okay. Now what I do is I look at this so I get an idea that okay 500 six are there and I can count everything here but it will tell you only the value of continuous variables. You notice that you don't have the variable Charles here. Correct? You notice it is not here because it cannot calculate the average value of 0 and one. No it cannot tell 50% 25% all standard deviations all those things it cannot tell. So that's why you don't have that even if it has it is not making any sense here. Okay. So when even even if it is there don't look at it with real value. You're not going to get much value out of it. So this is what we do. Shape 506 rows 14 columns. Okay. This is how the data set looks like. This is you print to see that. 4:31:43 Okay. first five rows head you know on on uh by default gives you 10 five you can also do this let me run it on a separate cell so that I don't read it again so if you want to look at 10 rows you can do this and it will give you 10 rows okay if you want to do look at from the bottom if your data is having a time aspect to it and it is sorted You can either sort it or you can simply print the tail. Okay, cool. This is the summary. And the other thing that I also try to do not every time but is a is a good practice is you know doing an information. So what it tells is it tells me each column okay how many non-null values are there. So looking at this you'll find if there are any missing values. Okay, it also tells since I deal with really huge 4:32:45 data sets, I also need to know this what is the memory usage, what is the you know if it is really huge data if I'm working with image data it's typically in GBs. So this helps me to whether I need to parallelize my things. I need to you know work on the cloud or will this it work or do do I need to you know put in the GPU or something like that. All those things it will tell for each column it will print out its data type whether it is a float integer object what not. It will also tell whether how many nonn values are there. Now you know that there are 506 entries. Okay, if any any column says 500, so you know six of them are missing values. They're blanks. Okay, they're null values. That's one. And then this also tells me 4:33:45 a summary that out of 14 11 of them are float type and three of them are integer type. Okay. Very uh in in one line you get a lot of information about the data set. Okay. All right. Cool. Now once you have once you have the data set ready and you've loaded it in, what do you want to do next? Now we are here to do what? We are here to do linear regression. Right? So the other thing that you can do is um the first thing is that you need to know the dependent and uh independent variables. We know right data cleaning is not required here. Okay. And we are going to do simple linear regression first so that we cover uh linear regression also understand it. I want to do two things the way I explained you with one feature and then with multiple features. Okay. In industry this is called linear 4:34:47 regression and multi features is called multiple linear regression. Okay keep that in mind very intuitive but just in case. Okay cool. So now what I will do is I will try to create my data sets. I will do data I want to keep only two variables as of now. one is I will keep L start this I've done and so what we are doing is linear regression with one variable one variable is what independent feature is equal to L stat okay and dependent dependent feature is dependent variable is median value. Okay, 4:35:48 cool. So this is what I'm doing. I am let me change this to 4:36:06 okay. So now you have this you have one variable and one target variable correct we are trying to understand linear regression with one feature first and then we are going to do with all the features okay so that you you are doing this for the first time right we learned linear regression how with one line single line correct and then we replicated it for all the features planes hyperplanes So for you to digest it better, I'm first doing it with one feature so that I can show you some graphs and also explain you how things are looking like. Okay. Okay. Perfect. Now the other thing is once we've done this, we should also plot it. Let's see. We were always plotting everything. Let's also plot this. Okay. So what we'll do is now we have already plotted the mattplot lip. Now we are going to plot the uh relationship between LS stat 4:37:08 and uh med median value med. Okay, cool. So it is 4:37:30 what do we have in x-axis? I'll start correct dependent variable on y-axis. So we will have med v. Okay. And then I want to do what? I want to plot the x label. X level is I'll start n dot we are giving the labels. Okay, this is the median value of house and then we have plt dot show. Okay, so this is how it is looking like. It shouldn't be looking like this. 4:38:35 Yeah. Okay. So, this is how it looks like. Scatter plot. Yes. Okay. Cool. So, let me close this. If this is this guys is not visible to you, let me close this. So, you see how how it is changing. Correct? If the percentage of people with lower status percentage of people with lower status is high, the price is really less. And if the percentage of people with lower status or lower income in that manner, if you want to understand it, the price is really high. Okay. All right. That's you. You see the clear you see the clear uh division. You see the relationship now the best line would be somewhere here. Correct? Let's try to find it. So now you have this data set. What do you want to do next? 4:39:37 So we will separate out our uh dependent variables and independent variables. Okay. 4:39:52 segregate the dependent and independent variables. Okay, by you know uh by notation anything which is x is your dependent variable. So what we are going to do is we are going to create this data frame where your what value is your dependent variable your independent variable sorry x is independent y is dependent so x is lstat here right l x is lstat correct so let's put in the lst here so what was the name it was df_1 feature 4:40:48 So we are going to pass here lat and in y we are going to pass what? We are going to pass our target variable which is nothing but med. Okay. Now we have that. Now the next step is now you have your dependent and independent features separate out. What did we learn in supervised learning? What what do we have to do now? The next step is to create the model. To create the model what do we want to do? So this is what we have done right now. Now imagine what did we do? Divide the data set into you have to build your model. What do you have to do? Divide it into train and 4:41:49 split. Correct? You have 500 rows. You have to split it into train and test. Divided into train and test. Now you have divided the columns first into dependent and independent. Now you need to divide your rows one on the training the other on the test. So that once you create your model on the training you need to sorry you need to test it on your validation or test data set how how your thing is working how your model is working. Okay perfect. So for that what happens is we have to import from skarn dot model selection. Now sklearn and scikitlearn are the same thing. When I started learning first of all when I started learning Python wasn't there uh in the picture for machine learning it was our language. Uh but now Python has 4:42:50 taken over. So what happened is uh I I started looking at what is skarn what is skarn what is skarn and in the internet I used to get scikit scikitle learn scikitle learn scikitle learn so both the things are same okay it's just how people are calling it so I need model selection from it and from model selection I'm going to import my train and test split Okay, this function is going to help me divide my data set into train and test. Okay guys, perfect with me. Done. Okay, now what it does is it will give me these four things. 4:43:52 This is the train array test split and what it takes as an input. This is the output that it gives right from Python you know packing and unpacking right. Whatever this gives as an output it has four outputs and you are giving each out each value to this X train X test Y train Y test. Okay. Whatever value comes from this. Okay. Now what it takes is what what it takes inside it is it takes inside the x value dependent features. It takes your uh independent features. Okay. And then you can also look at it this this definition you can look at it. Okay. Train size. Then you have the random state. And you also have other values in this whether you want to shuffle or not shuffle. How do you want to select things? Okay. Do you want to stratify? Stratify is 4:44:53 meaning if you have three classes apple, banana and orange and you want to take in the same ratio. Let's say 50% of banana is there 20% is apple something like that. Then you take it in the same ratio. Okay. And divide it into 70% or 80%. So what what we did was this is that and then the split. So I will do 80% here. So this test size what is the test size I'm passing here random state 42 okay all right so what happened here you're passing all your x here and you're passing all your y's here okay you're saying my test size should be 0.2 two meaning 20% in the test 80% in your train and this random state is telling 4:45:53 me what is telling me so that I can repeat this so some things happen randomly right I'm trying to take randomly 20% out as a test and 80% as train so that you guys also are able to repeat this as your at your end every time randomness is constant so if I provide this random state it will you know uh repeat the same set of results for everyone. Okay, cool. Now let's print it here. How are my things looking like? How are my uh you know X train X test all those things are looking like. So I am going to do this. I hope everyone knows frings. 4:46:53 X and shape is this and I will repeat it for let me put it this way. Let's do Y train so that you know how Y train is looking like. Then we visible for everyone and then I want to also look at my test set. So I'm going to put here test test and here instead of test it is going to be or train instead of train it is going to be test. 4:47:56 Okay. So now what we have done we have in my train X train we have 404 values and similarly I have 404 actual values the label Y is label and your test is 20% of 506 okay it is 102 502 in your dependent variable and 102 in your Y test. Okay, cool. So now you have the train and test. What is the next step? Okay, we will now create the model. Okay. And which model we are going to create? We are going to create linear regression. So linear regression comes from your linear models. Okay. And it is again from skarn 4:49:00 model linear. 4:49:10 It it gives suggestions when it wants. Okay. All right. Let me write this properly. From this import which which algorithm are we going to do? Linear regression. Okay. So we are going to import linear regression from here. Now what I do is we have to first instantiate we have to first instantiate this linear regression model. Correct. And then what we are going to do is we are going to fit or let me do one thing first. What did I do? I created I instantiated this linear regression into model. Now my model when I print model I will get this. It is a linear regression type. It is an 4:50:12 instance of linear regression. Okay. Everything of linear regression is now held in model. Now what you do is now you fit the model. Fit the model. Here our model is what? Our model is linear regression. Now what I will do is I will model dot fit. It needs to fit on what? It needs to fit on X and Y. But X and Y are on train. Okay. And I will save this as it will be saved where in model. So all the learning that it is doing will be saved in model. Now till here model is only linear regression. Here it is going to fit. It is when I say fit fit on my X train and Y train and find the patterns. This is what is happening internally. Okay. Once you find the patterns, I am 4:51:14 going to test you on this. When I say patterns, it is fitting a line there. Linear regression, this thing has that concept already built in that it will find the best one. Okay, cool. So now let me run this. Okay, model.fit happens. And then what what we have al also seen that we have model dot. Now if you do this you will see all these things that are in inside this model or inside this linear regression. Okay we want to see the coefficient. Coefficient was what? M or w or beta. So for lstat this is the value. And you notice the negative value here. You notice the negative value. Why negative value? Negative slope. That's why negative value. Magnitude tells you how 4:52:16 big an impact. And positive or negative tells you whether it will reduce your prediction or increase the value of your target variable. Okay. Then there is other thing also which is inside model. What do we have the other thing? Intercept. This is your intercept value. Okay. So we we we uh this linear regression will help us find what? M and C with one feature. M is your coefficient. Okay. And C is your intercept. And you see the values for it. Okay. Cool. Now you see the value for it. Now what's the next turn predictions correct? Now you know the line, you have uh weights, you have the betas, you have 4:53:16 the coefficient, you have the intercept, you have a w knot or beta not whatever name you want to call it with, you have it. Now predict get the ycap get the predictions and then compare predictions with your test. So simple. Okay, that's how you you will know your model is good or bad. Correct. So what we going to do is we are going to make predictions and predictions will be done with model dot predict function. And yes in the predict function what are you going to pass? You are going to pass only test. you are not going to pass Y test because if you will give the Y test to it you're giving already the actual values you need to test right so this is what we are doing with Y bread I am now also creating for 4:54:18 it to be visible clearly I'm creating it into a data frame okay and let me give the value of the columns to be its value as predicted. 4:54:40 Okay, y dot 102 predictions. Let me also do this. Let me show you what you have predicted. You have predicted this this this this for 102 values because your tra test had 102 values. Correct? Now, how do you know whether your model is good or bad? How do you know whether your model is good or bad? We learned about three errors. Which were they? mean absolute error, mean squared error and root mean square error. Correct? Let's do that. Now, for that also, you don't have to memorize all the formulas and everything. You already have it. Okay? So, this is your prediction. This is your test values. 4:55:41 Let me show you. So, in reality, it is 28.2. But what you have predicted is 27.37. Okay, this is 27.69 that you have predicted. In reality, it is 23.9. So you see there are some errors. You can already see there are some errors 16.9 and 16.6. So now what we are going to do? We are going to find those errors. Okay? And those errors are in another skarn dot matrix and I am going to print it directly with the f string. Okay, this is um which one is the first one mean absolute error. I will also need numpy since I've already imported it. I'm not doing it 4:56:42 here. I'm saying m ae mean absolute error is what is matrix 4:57:01 dot mean 4:57:09 python is such beautiful you know it you need to find mean absolute error You provide only that name. Okay. And what what did we matrix? Okay. It is it is saying us quickly line because we have not run this till now. Let me run this uh from skarn import matrix. Okay. From skarn we are importing metrics and we are putting this and we need to pass the actual value and we need to pass the predicted value. This is mean absolute error. Similarly, MSE and you change this value to mean squared. Mean squared error and then 4:58:13 RMSSC is what? 4:58:22 is what guys is just the root of what is just the root of your mean squed error. So what will I do? I will do np dot sqrt. Is that clear guys? So now you know mean absolute error, you know what is the mean squared error and you know the root mean squared error. Okay. Now what I do generally is I keep everything that I have imported in one cell. This way I'm able to load everything in one. 4:59:22 R square. We can do R square also. It's nothing but if you do instead of here if you do R2 I think it is R2 square uh score unmatched unstring it will tell you R square also okay now you have I will cover R square later But it's very simple. It is another metric only. Okay. Now we are going to go into multiple linear regression. So in single linear regression what did we do in single linear regression? We created a data set of only one dependent variable and we had the one dependent 5:00:25 variable and one independent variable. Okay. Now here what we are going to do is we are not going to do that. We are going to run it for all the features. So all my features are where are in uh data are in data. Okay. So what I'm going to do is I'm going to copy this and now I am going to say dependent all features I am going to say data data all the columns dot iO Look. 5:01:38 Do you understand what I did? So from data okay let me print it out there data dot head from my head the last column is medv so I'm saying all the rows start to end from start to minus one meaning the last one is never captured right in in the list slicing so I'm saying from start to minus1 one. Okay. Take all the columns and put it in X and in Y take only the one minus one the last column. So that I have what? I have all my dependent features in X and all my all my independent features in X and dependent feature only in Y. Okay. Now what was the next step we did? Next step was to split the data. 5:02:39 We will do the same thing. Okay, we are going to do the same thing here. Again, it has run. Let us do the same thing. Whatever we did for one feature has to be done for every feature. Right? The concept is same. So now we are going to do what? We are going to print our X test and Y test. X train and Y train. Now the only difference is the number of dependent features. Independent features. Why am I saying dependent? Independent features are 13. Earlier it was only one. And your dependent feature is only one which is your target variable. Okay. Then what did we do next? What did we do next? We created the 5:03:40 model. We created the model. We'll create the model. And then what did we do? We fit the model. We fitted the model. You see why I am doing this copy paste on purpose. We've already did it once, correct? We already did it once for one feature. Is there any change that you that I'm doing except for just passing the features that I wanted to pass here. This is the only change. All the other steps are same. Train and test split. Finding the you know X train and Y train. X test and Y test only the difference is number of features here the model I created it's the same model 5:04:41 it is telling you the same thing I'm fitting the model in the same manner correct I am also predicting it in the same manner look at this okay I will show you this is kind of important okay now earlier you had how many intercept cept one intercept right now also you're going to have one intercept but this time you had how many coefficients how many weights you had how many coefficients you had only one coefficient now how many are you going to have you're going to have 13 of them one coefficient and one beta value, one w value for each feature, each independent variable. Okay? And then what you do is you do the 5:05:43 predictions, 5:05:54 you do the predictions. Okay, this has your predictions and then we are going to evaluate its performance. 5:06:12 Guys, did I do anything different? Did I do anything different from one feature to multiple features? No. Right? You can see the values have changed. It's in the understanding. Now instead of one feature, you are having 13 features. The main absolute error was five earlier. Now it is three. This was around I think I don't know what the number was here. It was six. Here I think it was 26 or something 46. Remember see you have done so much of improvement. And what did you do with one feature to predict it is difficult with multiple features it is easy to predict. Okay. So that's the whole idea. Let me save this. Okay. Cool. Okay. So with multiple features it is basically a plane of the line. Right. 5:07:14 Cool. Now part. So is nothing but Okay. How do I explain this? Let me pick this here. Now, what is this doing? 5:07:40 I log is nothing but it tells a location or data to pick up the location. First tell me this guys. Uh if I say um a is equals to 1 2 3 4 5 6 7. Okay. Now if I print a you know it is printing this to this 1 to 7. Okay. Now if I say three, what does this thing say? This is a list. Okay. What is this going to print now? It is going to print only 1 2 3. Why? Or let me do this 20. This is Python concept guys. Um revise it. I am explaining it but revise it. You'll be much confident. Okay. 5:08:42 So it is nothing but it is slicing. This is your starting part. This is your ending part. So it is saying the counting starts the indices starts in Python from zero. So 0 1 2 3. But the caveat is the starting part is included. Ending part is not included. So it is going to print from 0 to 2. Index is 0 to 2 not 3 and 0 1 2. So that is why it is printing 0 1 2. If I say print till five, if I say 0 to 5, this is nothing but a blank there means zero. Okay, this let me explain this. I I said this zero 0 to three zero element to third element but third element is not included. So it will include till two. It is going to give you the same result 10 20 30. Okay. And here if I do this 5:09:42 five, it is going to give me first four. Five is not included. So 0 1 2 3 4. This is what it is going to return. Okay, it returned till 50. Now minus one. If you say minus one, what will it do? it will give me the this number from the last if this is zero this is minus one okay it gave you the last value correct now this I'm explaining on list if you do on data frame so now you understand this is what I did basically this to this minus one so now you understand this is columns and this is rows Since I am now talking about data set, it has rows and columns. 5:10:44 So this first part is telling you rows. This is blank. This is blank. Meaning all let me explain this here also. If I do a and do not pass anything, it means pass me the entire list 1 to 70 1 100 to 70. Okay. So this part tells me all the rows and this part tells me all the columns except minus one. Okay. But in data frame you need to put this when you're giving indices you have to give me you have to use this iO index location so that data frame understands that it is the index location that I'm passing here. Okay. >> Just a quick info guys. Intellipad offers an executive post-graduate certification in AI and machine learning in collaboration with iHub, IIT, Rurki, and Microsoft. This comprehensive course is designed to make you an expert in AI 5:11:44 and ML in just 11 months by working on over 50 realtime projects and industry case studies. With this course, we have already helped thousands of professional and successful career transition. You can check out their testimonials on our achievers channel whose link is given in the description below. Without a doubt, this course can set your careers to new height. So visit the course page link given below in the description and take a first step towards career growth in the field of AI and ML. >> Now the other thing is now we know how linear regression works. Okay, we've also done the hand job. There are certain things two minute things which are really important from your interview standpoint. Okay, whatever I have explained you, if you understand that, okay, you know how uh linear regression works internally. Okay, whenever an interviewer asks you a question, can you explain in linear regression? You can start from the scratch from the start to 5:12:44 the end you can explain what linear regression is. Okay, correct. And if they ask you can you do a hands- on can you implement uh linear regression here you'll be like give me a data set I'll do that okay hands on is clear now there are certain other uh variations even I as a recruiter go and try to ask this more from a overall understanding of the concept okay so in the overall understanding of the concept you need to know it's working you need to uh do a hands-on on it then you need to also have need to know certain theoretical aspects of it. Okay. Now, what are these theoretical aspects related to linear regression? So, with respect to linear regression, there is this R squar. Okay, I will mark a star and I will say interview. 5:13:45 Okay. What is R² 1? Then assumptions for implementing linear regression or people generally try to ask this as assumptions for linear regression. Okay, these two are really important apart from what I have already told you. the entire understanding the intuition of the algorithm is really important then the hands-on part okay now we'll cover squared R squar in a moment assumptions of linear regression are basically you as a data scientist as a machine learning engineer or data analyst anyone in this industry you need to know when to apply linear regression and when not to that is really critical ical it doesn't mean that you are given 5:14:47 a data set and blindly you will say that I have learned linear regression I will run linear regression no there are certain assumptions meaning assumptions on the data that you're taking that if the data follows these assumptions I'm making these assumptions on the data set if they are present then linear regression will give me a good result or it is something that I can implement code will still your Python notebook all those things will still run. It will still give you an error. But is that the right thing to do or not do? That is also a part of the job. Okay, that comes with experience. But assumptions are the first step that you need to look at. Okay, so we'll cover that. Now, what is R²? Let's look at the first thing. Now r 2 as a really easy concept. 5:15:50 This is the formula. What is this? This is nothing but this is sum of squared. Sum of squared. Okay. Let me also first explain this. your errors that we looked at, okay, are also called residuals that is left, right? So residual is something that is left. So the error is basically between your uh actual and predicted, right? That is something left. So residuals is another name given to errors. Okay? So this is something that you need to keep in mind. Now here you're telling the residuals sum of squared errors of residuals and sum of squared errors of total. 5:16:54 Okay, this is the formula. Now let's deep dive. How did they come into this formula? Okay, why did they create this? Now tell me one thing. Tell me one thing. If you have a um plot, okay, let's say you have a data set like this. 5:17:31 Okay, you have a data set like this. What you're doing is you are finding the best line. Correct? This is your best line. But how do you say this is your best line? You need to have some benchmark, right? Some comparison. Correct? Till now we were looking at errors. Till now we were looking at errors. But there is another benchmark that you need to have that is you know statistically also proven. Let's say I have this average line. Okay. What is this? This is average line. You need to compare your line with the average line. Okay. Meaning if I used to if I predict let's say this value is uh 100 okay if no matter what your features are you 5:18:34 keep on predicting blindly the average value of of the entire data set of the entire pricing in our example that median value how good is your this regression line as compared to the average line. Okay, that is what R² is. It is also called R² is also called coefficient of determination. I will put it here. Coefficient of determination. You're determining whether this is your good line or bad line. Okay. And how are you determining how good does your regression line is better when compared with the average average line? You need some benchmark, right? To compare. So this is what it is 5:19:35 doing. So you need to tell if your model is right or wrong compared to the average. And this is the formula for it. Here what does this tell me? Numerator tells me my sum of squared errors of my regression line. Okay. And this is nothing but sum of squared errors of my average line. Everyone clear? Now you understand the formula why it is that you're taking a ratio. You're saying how better is my regression line compared to the average line. Okay. Now what values can R square take? This is really important. What values 5:20:38 can R² take? Okay, now think it this way guys. Look at this formula. If the error between your uh if if the errors of your regression line let's say is um what? Let's say 100. Okay, if this is 100, let me put it here again. This is case one. If your S S is equal to 100 and your sum of squared for your uh regression line or average line is also 100 and you know the formula is 1 minus S S over sum of squared of the total which is nothing but 1 - 100 / 100. This will cancel out and this will come out to be 1 - 1 0. Right? 5:21:39 When do you think your regression line and your average line will have the same errors? Only in one case when they're overlapping each other. Correct? When they're overlapping each other. So this case one is that your model is model is average line only. If your best line or whatever line that you have you know either through logic or anything either by hand you've drawn if they if it is exactly like average line you're just predicting the average only on the data set you will have R square as zero is that what we want to do no right we want to do better than the average anyone can simply Why are we learning machine learning? If if all we want to do is look at all the prices and just say blindly this is the average. Whatever 5:22:39 features you pass me crime rate, lower status, okay? Whether it is a cacing, it is not seacing, number of rooms, disregard all the features and blindly say this is the average price of the house. That is your average line and this zero is not good. We want to move away from zero. Okay? Because 0 is average. Anyone can do that. Now the second case, what is the second case? 1 minus S S total you have some error. Okay. In average you have some error. Let's say 100. Now we are saying S S is zero is the best line having zero zero error. Okay. And your SS for the 5:23:40 average line is still 100. Now what is this going to be? It is going to be 1 minus 0 over 100. 0 is what? 0. So this is going to take what value? 0. Correct? This is the perfect line. Okay. So the idea is idea is idea is you need to be closer to one and far from zero. Is this idea clear guys? Correct. So R² is clear. So with this you can compare different lines. If your R square is 6 and your R square is8, which 5:24:44 is the one which is better 6 or8 R², which is better? We need to be closer to one. So closer to one is 8. So the line which is giving you R square of8 is better. Okay cool. Now you understand why it is also called coefficient of determination. See my style is that you need to be able to know the entire story. Okay. You need to be able to connect the dots. To connect the dots this in-depth knowledge is required. Why they are saying it as coefficient of determination you're determining which is your best model with the help of R square. Okay. And how you're determining you're determining comparison with the 5:25:44 average. No matter what line you predict, no matter what line you fit, if you are able to look at the R square, you are able to say that okay this is better than average by 60%. 6 means 60% better.8 8 means 80% better. Okay. Perfect. Perfect. All right. R from zero. Okay. Cool. So this is R². Now what's the next thing? The next thing is assumptions. 5:26:30 This is a very plain uh theory guys. I could have uh you know told you that you guys go and explore it at your end. But uh I think it's better at least we touch upon this. It's uh you can read about it. Uh there are five assumptions and you know they're very straightforward also. So I'll cover it in short so that if there are any queries around this we clear the linear regression from the root from after we have completed it. What is the pattern that you would be following for your preparation of interviews you have learned linear regression in the sessions. Okay, go back, explore, do hands-on, and then what you do? You locate the interview questions, complete the interview questions, then you're ready for linear regression. That's how you do it. You will not do it like okay, I've learned linear regression, I will do logistic regression, I will do this, I will do 5:27:30 that, and once the entire module is complete, I will look into the interview question. N complete linear regression once and for all. Okay, cool. So, what is the first very simple? Let me do it this. It should have linear relationship. It works on linearity, right? Linear regression. So, it should have linear relationship. Your independent feature with your dependent feature should have linear relationship. Correct? Second multicolon multi- colinearity shouldn't be there. 5:28:20 No multiolinearity. Okay. For the ones uh who are not from the ma maths background and for the others who must have forgotten this. What is colinear? collinear. If a point lies on the same line as that of the other, this is the line. Okay, these two points. These two points are collinear. Okay, they're lying on the same line. So when I say multiolinearity it shouldn't be present meaning if you have feature one you have feature two you have feature three you have feature four and so on till feature n they should not be giving the same 5:29:21 information meaning they shouldn't be collinear amongst themsel you are predicting target one using feature one to feature two feature This is your and you're predicting target. The relationship between feature one and feature two shouldn't be linear. Okay? Meaning they shouldn't be on the same line. What is the example in the car example that we were looking at? when you're trying to go and sell your secondhand uh your trying to sell your car in the secondhand market. If I say car's age, okay, and number of years since you bought the car, isn't they the same thing? Correct. They're telling you the same thing, right? If you purchased a car 5:30:23 purchased in what 2020 was a really wild year to purchase a car since we were in CO but anyways if someone did okay what would the age be? Age would be four and number of years since you bought would also be four. Correct? So they're giving the same same it is basically just a different name but they're giving the same information to the model. No additional information is being given. Okay cool. Then the third third example is that your errors okay I'm saying errors but when you look at the literature they will say residuals and you know what errors and residuals are they are one and the same thing 5:31:24 residuals should be or should be forming a normal distribution. Okay. If you go and plot your errors, okay, they should follow this kind of a curve. Okay, they should be normal. Very simple, right? Cool. Then what is the fourth one? This is something that people ask. Okay, that's why I'm putting a star mark. Not because um it is different. It is very easy. Is just that the name is really different. Okay. And they try to test whether the person has this uh knowledge or no. Okay. So it is what that your error should be 5:32:28 no hetroasticity. 5:32:37 Okay. No heteroscadacity in errors or in other words it should be homo scadastic. You get the gist of it. Okay. Now why they ask is because this heteroscadasticity and the homoscadastricity doesn't you know really uh understand uh people don't understand it really well. So what happens is you understand the word hetro and homo. Homo means same hetro means different okay homogeneous mixtures and heterogenous mixtures. Now your if you fit a line okay your variance your variance should not be like this of your data set. This is your data set. Okay 5:33:44 let me make it more clear as and when this is increasing. Okay, this is increasing. This is also increasing in different proportion. The variation is really different, right? This is not required. This shouldn't be the case. This should be the case. 5:34:14 Okay. It should be equal on both the sides. Okay. The this is this is heteroscadastic scadasticity and this is homoscadasticity. See it's difficult to say also this is what you're looking for. Okay. And the last one is autocorrelation. When I say autocorrelation there shouldn't be any autocorrelation or no autocorrelation. This is mostly related to the time series kind of data because that tells you or that was if you remember in the first session I told you there is a difference between regression and time series which is different and is very minute. You need to remember it. It is time based and it is not. Regression is not time based. Time series data is time based. Okay. Now here what do I mean by 5:35:15 autocorrelation? Correlation you understand there is some correlation if you break the word auto in this context means with itself. So previous values should not have relation with future ones. In other words, in other words, future values this I'm talking about the target. Okay, target your future values of the target should not depend on previous values or previous or past values. 5:36:19 Okay, with this linear regression is complete. There is any number of you know text or literature that you can refer. There are many different types of uh regressions as well. Linear regression itself. Go and explore it. Okay. I highly recommend that you go out of your comfort zone. Go and recommend it. You have the base now really strong with these sessions. Your base is really strong. The more you you know explore the more it will become beautiful. Your knowledge will become beautiful and the interview will be like yeah this guy knows basics so good and then you know he knows all the varieties also. There are small small minute changes and they call it with a separate name. Internally this is what is happening. Okay, perfect. So this was linear regression. 5:37:20 Now we are going to start with logistic regression. 5:37:35 Okay. Now what is logistic regression? It is a classification type. It is classification type of an algorithm. Okay. Target variable is how target variable will have categories. Okay, it is either zero or one. Example would be spam or not spam. Okay, this is an example. Okay, now let's look at some theory around it. Let me introduce you to this. 5:38:36 Yeah. Now we looked at machine learning. We talked about the third also which was what? Reinforcement learning. Right? But I told you 90% of the times you'll be working on supervised. Okay? Now we looked at the supervised first. We looked at regression and we looked at linear regression. Right? Now we are looking at classification. In supervised when I said supervised it is based on supervised is based on label data okay it should have label. So in classification there are two things one is logistic regression which is what we are going to study now and there is another which is called SVM. SVM is nothing but very popular classification model and has been since for a long time. I have personally used it a lot and it stands for support vector machines. Okay, really beautiful concept if you go into the depth of it. really mind-blowing how how you know these 5:39:37 statisticians and these uh people who were doing research they came up with these uh examples and these thought processes it's easier to understand but when you have a blank slate and you have to come up with this you know it takes some courage that's why they are there and we are not there okay but we are learning from them so logistic regression is a classification model based on statistics Okay. Deals with categorical target variable or dependent variable. Okay. Could be binary or dichotomus. Now binary you understand binary. Dichotoous is same as binary. If you don't know the meaning of it, it comes from uh biology kind of you know perspective since they were researching it. So it's just stuck with this algorithm. Just keep that in mind. It it is it works on dichotoous variables or target variables when the output is zero 5:40:37 or one. When I say zero or one, not literally zero or one. It could be one or two also. Depends on how you're coding your target variable. Okay? But there should be only two categories. Okay? There should be only two categories. Multinnomial, I'll touch on that. But just say that you know it it could follow any anything it could follow anything multi there could be multi- uh this thing you know nominal and this thing there there could be many just ignore that point for now we'll touch on that okay now takes both continuous and discrete input data meaning your in independent features can be either continuous or can be discrete okay there wasn't any any uh discrimination in linear regression also correct you could do anything and here also there is nothing in independent features there are certain algorithm why 5:41:38 it is necessary to put that out there are certain algorithms that work on all these kind of things they work on discrete also but the way that you have to put that information inside the model needs some modification needs some when I say modification needs some encoding. Okay, like for example XG boost, we are going to uh look at decision trees, random forest. Okay, some of them they need different kind of modification of the discrete type of values before we apply the model. That is why it is kept there. Okay. So, regression is the one where you are predicting your uh continuous variable. So regression your target variable will be continuous 5:42:41 meaning like predicting predicting what house price it can take any value know 100,000 10,000 1 lakh 20 lakhs, 20.22 lakhs, whatever. Predicting car price. Okay. And what was classification? You need to classify it either this or that or out of many things. So remember we looked at these two examples. Your target variable is discrete. When I say discrete, it has one or the other. Meaning you'll understand from the example uh spam and no spam. Either it is spam or not spam. Correct. The other example was where you 5:43:41 guys looked at my beautiful drawing fruit example. Now we looked at in the concept of clustering unsupervised but you know there are three apple banana and orange you have only what three categories you're providing the features and you're also labeling given the label is there then it becomes a classification problem. The other example okay the other example would be now you have cars correct I'm relating it with this so that you you are able to understand you have cars here you're predicting the car price but if you had to predict the type of the car now the car can be what car can be an SUV higher segment then it could be a sedan 5:44:45 Okay. It could be a hatchback. Correct. So these are different types. When you are given this data, car type and let's say you are given engine. Okay. Uh what else? seater ground clearance meaning how up it is ground clearance whatever okay and you are told this is an SUV this is uh a sedan this is a hatchback and whatn not this is your target variable okay so your target could be depending on all the features that you have it could either be SUV or a sedan or a 5:45:46 hatchback. You have only three options. Options is not the right word but for understanding I'm saying three options but you should always say three categories. Okay. So now you understand what is classification. But your logistic regression works on only binary binary binary output only two logistic regression will work when you have only two categories in your target. Okay, I'll have to slightly digress, but let me explain this. This is uh important for understanding. Now, tell me one thing. 5:46:47 Tell me one thing. when you say hatchback or when you say SUV or when you say sedan and you pass it to your algorithm you know which is on your laptop okay it will be like what are you saying will it understand hatchback SUV sedan no right what does computers and everything that work on digital they understand based on numbers bits binaries that English is our language Hindi is our language this is our language these things they don't work on machines they don't understand that okay so what happens is on the same data set you are encoding it 5:47:49 Okay, I'm not putting in the additional columns, but you get the gist of it. Let's say this is car type. This is hatchback, then sedan, then this is SUV, then hatchback. Okay, what you are doing is you are saying whenever I see a hatchback, I will put zero. Whenever I see a sedan, I will put one. Whenever I see SUV, I will put two. Okay, this is this concept is called what? Encoding. You are encoding it for the machine to understand it. It will not understand what is hatchback, SUV or sedan. It will just say that all the zeros are separate, all the ones are separate, twos are separate. Okay, these are three categories. Now when we do it on label it is called label encoding. 5:48:50 Okay. There are different types of encoding also. We'll cover it into the future. But there is something called as one hot encoding. Okay. Then there is TF term frequency and inverse document frequency. This is this is the concept of NLP. Let me relate this with a really uh good example which everyone must be using. Okay. You uh everyone has looked at chat GPT or understands what chat GPT does. Correct? You are typing in words. You are typing in words but it doesn't understand the words internally. What it's doing is it is doing this encoding. It is doing this encoding and over there the concept is called embeddings. 5:49:51 There is way further or a much detailed uh you know concepts there which also say tokenization and embeddings and there are many things but did you understand it is not non- numeric okay it is not non- numeric you are no matter what you're getting you are doing some kind of encoding for it to be understood to your machine. Okay, perfect. One hot encoding it is this one hot this all concepts are coming from uh the English speaking community. So one is one hot is nothing but zero 1 zero. Okay, it is nothing but 1 0. This we learn in decision trees and um random forest but this is how you are encoding. 5:50:54 Okay. Perfect. So we just introduced logistic regression. Okay. It works on binary output. Then classific it is a classification type. Your target variables will have categories. Actually in fact these two should be same only but I'm just connecting it here. Okay. And then one this is your example. Okay. Then we looked into a difference between regression and classification. We covered it here. Then to answer a query around non- numeric data to have that intuition and understanding correct. It is not uh only non- numeric. It is also you know we pass all the non- numeric or anything textual thing by encoding it because that's how the machines understand. Okay cool. So here some of the use cases. Let's look at some of the use cases. Okay. So now Olympics is going on. 5:51:59 Correct? Olympics is going on. So let's take an example from Olympics. Let's say neither Chopra will win or no. I hope you you know who Nidaj is. He's a guy, okay, with long hairs who throws javelin. Okay, this is my javelin and this is the field on which he throws. Okay, he's he's an athlete, Indian athlete. He won gold last time. So we are saying what? We are saying win or lose. 5:53:01 Right. This is what 01 correct. The other example World Cup. Will India win? Yes or no? There is no nothing in between, right? It's clear. Either we will win it or we lose it. We are not going into the rankings that they'll reach semi-final or no. Our motive is only to say whether we are going to win or lose. Yes or a no. Okay. The other example we have been looking at it. Email spam. We say spam not spam. If you ask a question, is it spam or not spam? Again, the answer is yes or a no. 5:54:03 Correct. Fourth, is the cat fat? Yes or no? Outcomes in all these use cases. Whenever your outcome is either this or that binary output binary output that's when we use logistic. Okay. Now some of the real world examples. Imagine. Imagine okay you are the owner of uh geio cinemas for example okay you're pretty rich but still you are the one who is um responsible 5:55:05 for getting the profits okay now you need to know using machine learning this is an actual real world example can you predict whether a customer is going to churn or not churn. Customer churn. So this is uh how churn means whether uh they are your members. Okay, you've paid you are you've taken the prescription subscription sorry of of that you know geio cinemas, Netflix whatnot. Okay, are you going to churn? Meaning will you leave? Will you stop your subscription? That is called churn. And when you do this for the customer, you mean to say whether you're going to churn or not churn. Okay? So this is for anything. Could be Netflix. 5:56:09 Okay? Could be Jio. Could be Hot Star. anything. Okay, this could also mean for a for a um in in US there are these Walmarts. Okay, here we have demarts. Walmarts and demarts. You need to know if your customer is going to churn or not churn. Okay? You must be using some Wi-Fi, whatever brand, let's say AEL or Geo Fiber. What else? Who is the ACT? Correct. All these folks they need to know whether their customer is going to churn or not churn. Meaning are they going to stop their 5:57:11 services and not renew it? This is a big problem guys. And if you are able to if you are able to identify this with high level of accuracy, you can stop the customer from churning. You can give uh an additional discount, an additional something. Okay, that will stop the customer from not renewing or going off your platform. The other example would be Swiggy or Zumato. So, Swiggy would like to know whether a person who is ordering from me today, tomorrow, from next month onwards, okay, is he going to churn Mean is he going to stop ordering from me? Not completely. Probably the what do you have that Swiggy Super is there? No. 5:58:12 And Zumato Gold is there. So is he going to stop using that and switch to Zumato or not going to switch to Zumato? I don't care. All I care about is is he or she going to stop my service or not? Is he going to ch is he going to switch? Why is this such a big problem? Because big problem because customer acquisition is really high. Customer acquisition cost is high. They want to so to make you the customer for the first time they have spent so much of money. They have given you marketing on the TVs, on your phones. Okay? They would have, you know, have one person uh calling you not directly for you, but they're spending on their ads, right? You must be seeing 5:59:14 uh these people they uh they put really big banners on the roadside, correct? They also give uh some uh TV commercials, TV ads. Then they also sponsor some of the teams in tournaments. Correct? So they're spending for what? So that people see that what is if there is a person who doesn't know what is Swiggy. Okay. And really likesh food. If they know that there is through these uh you know advertisements they know that okay there is something called a swiggy. out of curiosity they'll be like yeah I'm seeing this everywhere who is this sweetie if they search and they they're like he's giving me uh free home delivery and all I sitting at my home he'll give me you know food that I have ordered and I have so many options all these things are on convenience right it 6:00:15 is for the customer's convenience you're getting food at your home you are able to order something which is 20 kilometers from your house, okay, without going there and you're still getting that whenever you want when you're ordering. So everything of this is on convenience. So if a new customer looks at it, they most probably will find it convenient and they will enroll into Swiggy and then it is Swiggy's different game of marketing. First thing is to acquire the customer and to acquire the customer once it has become my customer now there is a different uh game altogether it is called CLTV which is customer lifetime value there is a journey that a customer takes in the business okay read about it it is an interesting concept uh how people think about it read about it whenever you're 6:01:17 free okay so This customer churn is really important. So they need to find a way that I can predict which customer is going to churn is going to churn. And when you say churn, ML models, they don't work on 100% guarantee, right? Even with linear regression we knew we are predicting and there could be some errors. So they work on what probability this the in the first class I told you it is one minus one game correct they tell you this is the probability of a customer to churn let's say if the probability is8 meaning 80% 6:02:18 it is going to churn the not it that that person is going to churn customer is going then can I also say 20% no churn there's a 20% probability that it will not churn correct it is 1 minus one game it works on probability okay now these are some examples that we looked at now in linear regression what we were doing we were fitting a line now if you look at a data for classification how will it look like? Okay, let's take an example of email spam and not spam. Okay, because it is an easier example to understand and same is you know we can apply to anything. The idea is to understand the concept. Okay, this I'm saying I'm I'm going to say 6:03:19 that this they will they are spam and here on my uh y-axis I have all the ones which are not spam here. Okay. And there are a few keywords. Remember it is going to look at the patterns and keywords would be what? Keywords. If if in the mail content these keywords come you are able to say whether it is a spam or not spam. If it says lottery, correct? Win prize, jackpot. If these keywords, these words are present in the content of your email, this means that you it is it has a high possibility of being a spam. More these words, higher is the pro probability. If 6:04:20 in an email I see there are you know three of them coming three of these keywords coming in one email. One email three keywords present versus another email in which one keyword is present. which has a higher possibility or a probability of becoming a spam or being a spam mail. Let me change this to 3 to 8. If eight keywords are present versus one, right? The one which is having more number. Correct? Everyone with me? Right? So, this is going to have higher probability. This is going to have a lower probability. This is going to have a higher and this 6:05:20 is going to have a lower. Okay. Now let's say here are your number of keywords. Number of these keywords on your y-axis. Now let's say this is 1 2 3 4 5 seven. Okay, this is how it is. Let's say an email that has four or five keywords. Let's say five. It has five keywords. Am I not going to say that it is a spam? Correct. The one with six can also be a spam. Seven will also be a spam. Okay. There could also be instances where 6:06:21 when I see only one keyword and it is not a spam. Two keywords it is not a spam. Okay. Sometimes there might be three keywords and still it is not a spam. And vice versa can also happen. There are two keywords present there. Someone is really sharp. They've taken only two keywords but still it is a spam. Okay. An example would be hey you have been shortlisted by KBC. You have won lottery of 10,000 1 cr. I used what one and lottery but this is a typical spam emails now these days they send you a lot but do you do you agree there are some exceptions someone can be really sharp and just use only one word so that even they know that someone who's reading this content the whole idea is that they 6:07:25 should click once they click enter their details then gone money is gone. Your hard-earned money is gone. So this thing can happen. Similarly, there could be you know a genuine email, a genuine email which has five keywords but still is not spam. Your friend has sent you this email saying that hey u you know what? Hey Anil, I am really happy to say that you know I have won a lottery and the prize money that I'm getting from it is uh you know let's say 50 lakhs. I think I have hit a jackpot in life. I've used all the words but this your friend has sent you. He's not asking you to you know press on any link or anything. So it is a genuine email. So this case and this case you understood right there are exceptions. 6:08:27 Now since we are able to do this whether this is a zero or one okay now let's say I am saying when it is a spam understand the concept I told you that when uh you know you have to encode it I'm saying the spam is one and not spam is zero. Okay. All the spam in your y-axis you have either zero or one. Earlier you were having what? Continuous variables, continuous values. Now you are having only 0 to one. And there are only two values that it can take either zero or one. That we know for logistic regression that is the case. Now the concept is same. We have to fit the line. But I but can I fit this line? Can I fit this line or this line? No. Right. I cannot. 6:09:28 Which is the line that I need to fix. Okay. The line that I can fit could be something like this. Could be something like this. It looks like an S. Correct? So that you're able to classify all these correctly. This one, this one, this one, and this one, this one, this one. You're able to classify it correctly. This one you were not able to. This one you were not able. Here the line is not going to be a straight line. It is going to be a curvy line. Okay. So now you have a data set. Now you have a data set. One one email. This is one email. This is one email. This is one email. Okay. You are trying to fit the line on the entire data set. 6:10:29 Over there in linear regression also we were trying to fit the line in the entire data set. This is your entire data set. All the emails together. How many do I have? I have eight. Let's say you have eight emails. Okay. The best line with which you can reasonably fit is that you can correctly identify three here and correctly identify three here. Three in spam, three in not spam. But you're not able to identify this correctly, this correctly. Why? Why? because same concept as that of your um linear regression. Now if I go this is my line this yellow is my line. Let me draw this properly. Okay, this is my line. Now whenever you are trying to whenever you are trying to in in in 6:11:31 linear regression also what we were doing when I have to look at something which is at having two keywords I used to plot it on my plot it on my line here and then look at its value on the y-axis correct this is what we used to do right here this value is telling me that it is closer to not spam. So I'm going to say that it is not spam. But this in reality this is the one and this is the actual. So this is an error or here here the errors are called mclassification. 6:12:23 Everyone with me? Do you understand mclassification? This one and this one are your mclassifications. Correct? And this one and this one all these are correctly classified. What we are trying to achieve here is we are trying to fit a line such that our misclassification is less. Okay. In this case when there is a zero and a one output you cannot fit a straight line. It has to be a curvy line. Now the curvy line is this type of a line that we are able to fit which is S shape. Okay. This is what S shape. Okay. And the motive is similar to linear regression that your error should be minimum. Here in this case the 6:13:25 mclassification should be minimum. Okay. Cool. The other example, the other idea is that um here on the y-axis you have 0 to 1. Okay. Doesn't it tell me that it is the probability somewhere in between this is 0.5. If there is a value that is coming here from out of this this one anything greater to greater than 0.5 is going to be span and less than.5 is going to be not span correct based on this curve based on whatever plot you do here anything if I plot it on my y-axis if it is greater than.5 it is going to be spam because it is closer to spam and Anything which is less than 0.5 is going to be not spam. The chance of it being a not spam is 6:14:27 more because it is closer to the one where it is not spam. Okay. Now this this 0.5 is also called a threshold or a criteria. This you have in your hands. Depending on use case to use case you can either take it as.5 or 7 or 0 4 or 6 I don't care depends on the use case you have this in your hand looking at the data okay and all the values here in in between 0 to 1 can take any value now you understand since this is 0.5 it can take even 0.2. It can also be.112. It can also be 234. It can also be 67. This is nothing but this is probability. 6:15:27 This y-axis is a probability. You have this data. You have this data. Okay. Okay. Now you have this data set with you. Is the same thing correct? Let me clean it a bit. It is the same thing that we plotted above. Now tell me one thing. Are you able to see that in this yellow line this is not present and in this yellow line this is not present. If you try to plot let's say I have seven here. I had seven here. Right? If I plot the number of keywords in an email, if seven are present, if I try to plot it here, will I not touch the line here? So, what does this value have here? It has what? It has one, which is nothing but spam. 6:16:30 Okay. So now here when you are at two you are trying to touch the line your line that you have fit and when you try to plot it on the y-axis it is what it is let's say 01 here is your zero and here is your zero and this is what this is not spam correct and you're going to say this is 0.5 your criteria is any probability in this line if it falls anything between this I'm going to say it is spam meaning anything if the probability is greater than 0.5 it is going to be spam and if it falls in this region which is less than 0.5 it is going to be not spam. So this one rather than calling it as an 6:17:30 error you are not able to classify it correctly now it is spam but you are still saying that anything which has two keywords is going to be not spam but this is spam in reality. This is your error and error in classification is called mclassification. 6:17:56 Hm. Okay. So in general since this is probability in general threshold as a thumb rule is chosen to be 0.5 50% here 50% there. But depending on your use case depending on your use case you can change that value. It is in your hands. It is in your hands depending on what you're seeing in the data. Let me give you another example so so that you understand this threshold thing. Okay. Now imagine another use case. You are working with HDFC. Okay. And I come to you and I say that can you give me a loan? So what do you do? What is your problem? Understand this. Your task is should I be given? Okay. 6:18:56 Since you are going to root it uh read it at your end. Should you give loan that is your problem correct? That is your use case. That's the uh you want the solution for this. You mean the other way other way to look at it is if I am coming to you okay let's say this is who is asking for loan okay asking for loan you need to say whether he is a he or she is a fraud or no this is What? Another example of 01? Yes or no? 6:19:57 Correct. So here again logistic regression will come. But tell me one thing out of let's say you receive 100 applications. You receive 100 applications. How many of them are fraud? Only 1%. Or in imagine we look we we are living in a really bad country. Even then even then this would not be more than 5%. terms and conditions applied on the country as in depending on the country it might be 10% also but in in my experience this is really huge even 1% is really huge okay now this is also called as imbalanced data set 6:20:58 imbalanced data it here. Out of the 100 out of the 100 95 95 are not fraud. Okay. And only five of them are fraud. Now in this case do you think 0.5 will be a good choice? No. Right. No. Right. Now you understand how you have to you need to put it really tight. That threshold should be really tight. Meaning that 95% of the times I don't get a uh fraud application. There's only 5% chance. So this line here, this threshold here will not be 50%. Correct? it will be somewhere here. 6:22:00 Okay. So that's how you change your threshold and that's why you have it in your hands and that's how it depends from use case to use case. Understood? Is that clear? All right. Okay. Now, now you know this and now you know that it is an S shape. Now, did anyone tell you or do you know why S shape? 6:22:33 Why only S shape? Why? Why cannot I put a Z shape? Why cannot I put a C shape or an invert C shape? Right? All these are possibilities, right? If I put a graph here, all these are possibilities. Y-curve isn't this something like a parabola if I just twist it parabola is like this right if I just change it will it will this not be expressed in mathematical concepts mathematical equation it will be right so all the statisticians in the world all the data scientists machine learners machine learning engineers research scientists They looked at all these curves. You and me, we don't have to. They looked at all the possibilities in the world and they said this Sshaped curve is the only way that we can solve this problem. Imagine 6:23:36 we looked at the uh curve, right? We looked at how we interpret it. So imagine this is your keyword at two. Okay? And this is at three, four. Now at four if you try to plot it on your curve will it not give three values? Will it not give you three values here? Correct. This is spam. This is not spam. So for one input one input you're getting three output. You're saying there's a possibility of this being not spam and this is the possibility of being spam. This is the possibility of being spam only in one input. Will that solve our problem? No. Right? For each observation I need to tell whether it is spam or not spam. For each email I cannot say that you know uh there is this possibility, this possibility and this possibility. Then what is the use 6:24:38 of machine learning? Here also you'll get two points. Correct. So the idea was all these people, all these research scientists, they looked at everything and they found Scurve to be the best Scurve to be the best which could help us solve this problem or Sshaped and could easily be converted to mathematical explanation. Okay. Cool. All right. So now you know why. Okay. And you know how we are fitting a S S shape. You know 0 to 1 probabilities. Now I need to take you through one really important concept inside this for you to have further 6:25:40 understanding in the logistic regression. Okay. which is called odds. Okay. And how it is different from probability. Okay. This is a good concept. Now the idea is let's say let's say you have five red balls okay and you have three everyone from the non uh maths and non techch background we are revising a concept I'll take you from the basic to explain you this odds concept which is not really uh easy to understand Okay. So, play pay 100% attention here everyone. Okay. This is a really good concept and really a a base concept for 6:26:40 you to understand logistic regression. Okay. Now, when we used to understand our what u probabilities, correct? So probability was what? The definition of probability was number of events when it was successful. When success you have okay based on what you have over total number of events. Correct? This was the definition. Now your you have to give me an answer out of this. You have to tell me what is the probability of picking a red ball. 6:27:49 Tell me look at this. This is the definition 5 by 8. Correct? 5 by 8. How did we reach to five? This is the formula for probability number of events when success. So you'll be successful when you when one of these balls are in your hand which is one of the five. So you can be successful out of five in five events. Correct? Out of total number of events. Total is what? Five here and three here. So 5 + 3 8. This is the probability. Correct. Now odds. Odds is slightly different. Okay. What is odds? Odds tells you the likelihood 6:28:50 the likelihood of success with respect to failure. Okay, this is the very crude and very simple definition of odds. It is the likelihood of success with respect to failure not with respect to total events. Understanding? Okay. So in the same example in the same example okay if I say what what are the odds of picking a red ball? What are the So in this there is another concept of favorable and infavorable. I am just saying whatever I'm saying it is favorable. Okay. What are the odds 6:29:52 of picking a red ball 5 by3? Why? Imagine this to be success number of success events and number of failure events. This is your odds. Meaning in this case it would be five and you could fail in three of them when you pick one of the yellow balls. So this is your odds. Okay. So what you're doing is you are doing eight. Eight is total events. You only need to count failures. You're saying how likely am I to be successful compared to my failure. Not compared, not probability of me becoming successful. There's a very slight 6:30:53 difference but once you go through this concept now once again revise it, you'll be able to understand it. Okay. It's slightly mathematical. I've tried to, you know, simplify it to the core of it. Okay. Perfect. Cool. Now you know what is odds and what is probability. To concrete this concept further, let us understand a few more examples. This okay is fine. We've already looked at it. Where is logs? Huh? Okay. So this is one concept. Let's say this guy went for fishing five times in a week. He caught a fish on two times and he failed three times. Okay. So the probability is what? Probability is chances where he was successful over total chances. Okay. So he was successful in catching the fish twice out of how many times did he go? 6:31:55 Five times. So 2x five is the probability. But odds what is the odds? Odds is number of times when he was successful to catch the fish and number of times when he was not able to he was he failed to catch catch a fish. Okay. So these are your odds. Another example ignore this logit and everything. Another example is u so the this is the predictions okay of a rainy day and a sunny day. Okay. Now the odds of catching a sunny day is green means yes, red means no. Okay. The odds of catching a sunny day is 2x3. 2 over 3. Okay. And the odds of catching a rainy day is 3x2. Red is yes, green is no. Okay. 3x2. There there was rain on three times and it it didn't rain on 2 days. So 3x two. Now you could 6:32:58 also take a ratio of this 2x3 divided by this is what they are doing here. Okay, you could also take a ratio of this. Now this ratio is called odds ratio. This ratio is called odds ratio. Okay, let's do one thing. Why am I teaching you this odds concept? What is the reason behind this? The reason behind this is that whenever you are doing logistic regression then we when we were doing here okay we understood the curve has to be fit in an S shape but at the end of the day I am comparing my spam with not spam right I need to know an email is spam how likely it is to be a spam and how likely it is not a spam they have to compete against each other not out of how many I have spam. For one email I have only two 6:34:00 decisions to make spam or not spam. So I have to compare the these two. I need to compare the odds. I need to compare what is the likelihood the key word is likelihood of an email being a spam and a not spam. That doesn't come from pro probability. It comes from odds. Okay, that's why it is very similar to probability. So it is easy to get confused but that's the minute difference. I am deciding for an email or a 01. Okay, win or a loss, fraud or not fraud. Whatever example you take, I am competing with the other one. So I need to know my chances of success compared to my chances of failure. In other words, my chances or my likelihood of catching a fraud person versus my chances or likelihood of not catching a 6:35:00 fraud person. Understood? Okay. Key word is likelihood. So it's not plain probability, it is odds. Perfect. Now you know this. Now you already know this. Let's move into the logistic regression further down. Okay. Now look at this very clearly. Okay. Can I say odds is equal to probability over 1 minus probability? Can I say that? Let me give you an example why I can say that. So in the red marble and the yellow marble example that I gave you. Okay. Over there the probability was what? Probability of catching a red ball was five out of 6:36:02 eight. Correct. And let me put this in this way. Red. It was catching a red one red ball was five out of eight. Okay. And catching a yellow ball was what? 3 by 8. Everyone with me? And what was the odds? Odds was 5x3. Now you understand where I'm going. Can't you say 5 by 8 over 3x 8? This 8 and 8 will cancel will it not left? You will have 5x3 which is same as odds. Probability of getting a red ball over probability of not getting a red ball. Isn't that odds? If you're successful 6:37:03 that is probability and over not or or being failure can I not say failure is probability of not being successful. Correct. Okay. So now, now let's derive a few things. Okay. This will be very just follow me. We are not doing really high rocket maths. It's very simple. Okay. two things I need my uh non- tech guys and also people from tech background and mathematics background to remember or revise this when you have to take a log of a by b. What is this? This is nothing but log of a minus log of b. Correct? If you have not studied this, this is the property. Okay, this is the property of logarithm. I have not 6:38:05 derived this. It is already there. Okay, now we know odds is equals to probability over 1 minus probability. This is your first step. Second step, take log on both sides. 6:38:30 What will happen? This will become log of odds is equals to log of p / 1 minus p. Correct? Okay. And I'll tell you what is the equation of sigmoid function. Sigmoid function is nothing but 1 / 1 - e to the power min - x. This is your sigmoid function. Okay. Why I'm telling you this is because in in linear regression in linear regression our equation was y = mx + c. Correct. In logistic regression 6:39:37 it is this 1 y = 1 / 1 - e to the power min - x. Okay that's why we are going to reach from here from here what we learned odds to probability to this equation. How do we reach there? We going to derive it. And trust me, this is the line that we'll be working with. This is the line that we have been working till now in linear regression. Okay. Now, we already know it is an S shape. Don't take my word for it. It is the S shape that in logistic regression we have to use. And how do I say it is going to follow this? It is going to be this. How do I say this? Let's prove it. Okay, we'll go into mathematical derivation also, but I want to show you this as well. Okay, this is desmos. It's 6:40:38 nothing but a chart. Right? Now what happens is if I put an expression here which is y = 1 / e to the power - x. Okay. Don't you see this is this is an sshaped curve. This is the equation guys. y is equals to this. Okay, if I zoom in, whatever we have studied till now, it cuts exactly at 0.5 and it does not go whatever the value of x is, it does not overlap one. Basically, it is asymtotic to one and on the negative side, it is asmtotic to zero. What have we plotted here? We have plotted this 6:41:39 Okay, this is the best curve for us to say whether it is zero or one. Correct? S shape and this is the equation. I'm proving it to you right now. Showing you visually at least. Are you able to understand that this is the equation for us to get this kind of shape an S shape? No matter is it going above one? No. Right? It is not. Is it going below zero? No. Correct. Okay. This let's say y = to e to the power x. Okay. This is the graph and y = log of x. This blue line is log of x. Okay. And this is your e to the power exponential. Opposite of exponential is 6:42:40 logarithm. Okay. So this is the graph of logarithm and this is the graph of exponential. And I'll tell you why why we need this. Okay. I'll show you. Here is why we need it. So for example, y logs. Okay. Um let's say you have 3 ^x is equ= to 9. Okay. Can you not write it 3 to the power of x is equals to 3 ^ 2? This is 9. 3 ^ 2 is 9. Right? So with this can't you say x = 2? Correct? That's why for these we take exponentials. 6:43:40 This is nothing but this is exponential. Okay. And sometimes when these uh you know things are really complicated, we take logs because log is nothing but ulta of exponential. Okay. It is exactly the opposite of exponential. Okay. Now, homework. I need you to look at these two things. Okay. Homework one, adjusted R square. In linear regression, we studied R square. We need to look at adjusted R square. Okay. Then there is something called as VIF. I cannot cover everything. Okay? I'll try to cover things that are important from the interview side and for you to understand and I'm going to give you some homework that is going to improve 6:44:43 your understanding overall understanding. Okay? You need to tell me what is VIF? It is variance inflation factor. Inflation factor. Okay. And I'll tell you where you will see this. We looked at multiolinearity. You'll find it there. Okay. This will enhance your understanding of multiolinearity. Okay. And the third thing is log and exponentials functions. Just look at their equation. Just look at their graph. A few properties. Graphs and properties. That's it. Cool. If you are able to do this, trust me 6:45:45 guys, logistic regression is going to be a cakewalk. Okay? And first two are for you for linear regression. Third, if you do with me, logistic regression is going to be a cakewalk. Not only the explanation, the intuition, also the hands-on. Okay, these two are really base linear regression for regression and logistic regression for classification. Okay. Okay. So, cool. Now, what is adjusted R square? Let me take this up. First one what is adjusted R square. So now R square we understood right it helps us to tell how better is our model with respect to the average line basis of the errors and more the value of R the R square better the model is the idea is to be around one. So now guys 6:46:49 they notice that there is a slight problem with R square. So adjusted R square came because R² will increase with increase in number of independent features. Why was that the case? So let me give you an example. So imagine you are trying to predict your exam score. Okay, you're trying to predict your exam score. Uh imagine you're you're going into the exam, okay, and you want to predict your uh marks that you'll score out of 100 bases the number of hours that you have studied. Okay, so the R value, so I'm writing the problem here. Predict the exam 6:47:50 marks based on number of hours studied. Okay. So you want to predict your marks based on or or you know for a subset of students you want to predict their uh marks based on exam score based on the number of hours that they have studied. Okay. Let's say you get an R square equal to8. Okay. Meaning that 80% 80% of uh you know the the marks can be predicted. It is the model is better 80% better and you know they can be predicted based on your marks uh the number of hours studied. Now imagine I I give you another feature. I give you their favorite color along with number of hours studied. I give you 6:48:51 the favorite color of a student and I tell you that you know now go and predict your this thing favorite color of a student your marks. Now R square will definitely increase R square will increase but does this feature have any any importance? Will it tell you the favorite color of a student and will predict its uh his or her performance in the exam? No. Right? No. So to penalize the R square, so R square always increases with the number of features that you pass okay to your model. To give some sort of penalty to R square, they came up with something called as adjusted R square. So adjusted R square penalizes your R square. Internally it is the same thing. It calculates the same thing. There is some relationship between adjusted R square and R square. That is up to you 6:49:52 to figure out. But in that formula you will notice that it it penalizes your R square. It penalizes R square based on predictors. Okay. It also has based on number of predictors. It gives more importance to the ones where the feature is actually important for your model. So in the above case here adjusted while the R square will increase. Okay, but adjusted R square will most likely be same or it will reduce. Okay, that's the concept of adjusted R square. Now, what is the next one? Okay, VIF. So, I also told you that it is coming in multiolinearity. Okay, I gave 6:50:52 you a hint. So, let's look at the other example VIF. Now VIF is variance inflation factor. It is used to detect the multicolinearity in your data set. Okay. The concept is very simple. The concept is very simple. Imagine you have four features. Feature one, feature two. Okay. Feature three and so on and feature four. These are your independent features. Independent features. What VIF will do while doing that linear regression or finding a uh you know multi-olinearity, it will internally try to plot feature one with feature two. It will try to do linear regression between feature one and feature two then feature three with feature or it will do the same thing but just to 6:51:53 maintain the consistency it will do it like this. It will do with feature one with feature four and we'll try to see if there is a linear relationship or no. If there is then the VIF value will become very high. It will become very high. If there is a multiolinearity, okay, it will become really high and you will have to whenever you're running your linear regression, you have to remove the uh feature with really high VIF. Okay, remove it and run the VIF again. run the linear regression again. If you still see some relationship, some feature having a VIF really high, remove it. Do it again. That way you'll be able to eradicate your multiolinear. Okay, 6:52:57 all of you with me on both these concepts. So this you don't have to do. It will automatically calculate VIF. It will automatically calculate VIF. there there's there's these uh you know things in the uh linear regression but you need to know how to implement it how to even think about vif okay so that was the idea okay cool cool so the other thing is what was the other thing log and these functions exponential ial graph. Okay, we'll go to the images. This is your exponential graph and there should also be exponential and log. 6:54:00 Yeah. Yeah, this one should be fine. Yes. Okay. So this one is a to the power x which is like this and this is log with base a to x. This is how the graph looks like. Okay. Two things that you have to remember for the log is in this graph log one is equal to 0. Okay. When log one log one is equal to 0 and log of 0 is minus infinity. Okay. And what will this value be? And this is the value of e to the power x or a to the power x here. A to the power x is also similar. Okay. Cool. Okay. So I will remove this three. We've already covered. Now let's go into logistic regression in deep detail. Now the idea is that we had this graph correct. We had this graph which had a 6:55:03 line like this and we had a few males here. Okay, which were spam and then there were a few here which are not spam. Okay, cool. So there were a few here and here was your keywords. Keywords high security keywords we looked at them lottery prize and all those. Okay. So this I will categorize as one. This I will categorize as zero. So we looked at this. Now how did we come to this? Okay. The idea we talked about log of odds correct we also said this is 0.5 here somewhere in between so the idea is that we looked at now this is one value and 6:56:05 this is zero originally I also told you that we have to be slightly different from probability we have to move towards odds because we have to compare whether it is a spam or a not spam okay out of not the total we have to consider the odds So we had looked at this formula that odds is equal to probability over 1 minus probability. Okay. So we if we take the log on both the sides this will become log of odds and this will become log of p / 1 minus p. Okay this is what it is. Now what what I also told you the the property of log is of log that you need to remember is log of a divided by b is equals to log of a 6:57:05 minus log of b. Okay, this is one property. Now what happens is let me pick this here again. So now we have this. Okay. What if let's look at two two cases. What if case one case one probability is zero. Okay. If the probability is zero, will it not be this that I will be my log of odds will become what? Log of 0 / 1 - 0. Correct? I'm I'm I'm putting just the value of P here. Okay. So, what will this become? This will become log of 0 6:58:06 minus. Now we are taking advantage of this property and we're going to say log of 1. Okay. And just now from the graph what did we see? What did we see from the graph? Log one is what? Log 1 is log 1 is zero. And what is log 0 is minus infinity. meaning it will become what? Minus infinity. So this this point when I'm saying it is zero this is log of odds right. So if it is zero can I say can I say let me put it here. Okay so we're looking at this case one. Let's also do the case two. Let's also do case two. Case two when probability is equal to 1. Now in the same formula log of odds I'm writing it again would be what? It 6:59:10 is P / 1 minus P and P's value is zero. Oh sorry one here in in this case in case two then it will become what? This is log of this. Then this will become log of 1 / 0. Correct? This will in turn become log of 1 minus log of 0. Look with this with this property. This property we are changing log 1 by 0 to log 1 minus log 0. So log 1 is what? Log 1 is 0 and minus of and what is this log 0? it is minus infinity. So minus minus is plus infinity. Okay. So what is happening is I'll tell you geometrically how you have to intuitively understand this. You have to intuitively understand this with this concept. Let me paste it here. 7:00:14 Okay. So this this graph imagine it was here. We were here between 1 to 0. Okay, we have looked at 1 to 0. And now will this these values which were at one with the concept that we proved here when it is zero it will go here. These ones will go to minus infinity. Imagine I have this this with me. Okay. Here it is minus infinity. Here it is plus infinity. Okay. Now all my green all my green not spam ones will be somewhere in the infinity minus infinity. Okay. And all our positive ones which are one will be somewhere in plus positive uh plus positive infinity at positive infinity. Meaning now this line this scurve line 7:01:16 is somewhat like this. Can I say that or can I not? It is somewhere like this. Why? How did we reach here? How did we reach here? We know this is log of odds. Okay. This is log of odds this axis. Okay. Now what we are trying to do is we are trying to put the values of probability. Now if we do a probability we know that our zero whatever males that we have as not spam which is zero okay will go to minus infinity and the ones that we have at one which is you know spam here will go to positive infinity why how will they go what I have done I have just replaced the values in the formula okay meaning that we can move this graph. We can move this graph from 0 to 1 to a minus infinity to plus infinity problem using log of odds. 7:02:21 We want 0 and one. This is the outcome that we want between this but originally it is like this. So these values so what we have done uh we have just expanded our y we have not done anything to your x. So here there were keywords here also there are keywords more the keywords it is going to be spam. Less the keywords the likelihood is that it is not going to be spam. Okay. So we have just expanded this y. We have just expanded the y. we have not expanded your x-axis. Okay. So basically the question that I was trying to ask or I was expecting that logistic regression has a word regression in it but it is a classification algorithm. Correct? It is a classification algorithm. Then why regression? It is 7:03:23 because of this. Okay. So what happens is since logistic regression has all the essence all the mathematical fundamentals of linear regression it is called logistic regression. So nothing but you have the data like this from minus infinity to plus infinity. Okay, there might be different values for these spam and not spam. But since you want this minus infinity to plus infinity, you want to squash the y value between 0 to 1. Squash as in you want to contract this infinite y-axis between 0 and one because that is meaningful for your problem. Okay. So the all the concepts of fitting the line and everything is very similar and also the mathematical concepts are very similar to linear regression and logistic regression. That is why it is called in the name it is logistic regression but 7:04:24 internally what it does it is a classification algorithm. Okay. The underlying concepts are like linear regression but the way we want to solve it is with scurve. We cannot solve it with a linear line but internally if you look at it it is a linear line but we are somehow changing this linear line to an scurve using some function and that function is called sigmoid. Sigmoid sigmoid function. Okay. And it is generally said that now your ycap is nothing but sigmoid of zed. Zed is nothing but your value or your line equation. This is nothing but sigmoid. In linear regression, what was it? In linear regression, your ycap was 7:05:28 w + w1 x1. Let me write it again. So in linear regression, your ycap was what? W + W1 X1 plus W2 X2. And depending on how many variables you have, it was wn XN. Correct? But in logistic regression this is your equation. And this zed is nothing but this zed is nothing but it is some combination of linear line only some combination of this linear line. Everyone with me? What you're doing? What you're using this sigmoid function is to squash the output 7:06:30 between 0 to 1. This function is just doing that. So basically you're getting some line in logistic regression. Okay, let's compare linear regression and logistic regression. In linear regression, we were fitting a line. We were looking at the errors and we were finding the best error. That's your best line. Okay, that is what you were doing in linear regression. Now, in logistic regression, what you're doing is you're essentially also doing the same concept, but then you are wrapping that everything inside a sigmoid function. Sigmoid function which helps you to get the output between 0 to 1. Okay. And sigmoid function works on what? Log of odds. How you can do that? You can do that based on log of odds. Okay. Now with an example. So let me take the same example of churn. 7:07:30 Churn example. Now imagine you have this. You are trying to predict which customers are going to churn. Okay? And you have this you have discounts you have discounts in your x-axis and you have number of calls of calls that the customer had to make to customer support to c support. Okay. So what would you see the number of people who have lesser discounts and have more problems meaning they are reaching out to customer support will be somewhere like this right and the people who do not call that much to customer support for their problems and also have 7:08:31 high discounts will be somewhere here. Correct. Let me put in some more values, some more data points. This is how in real world you'll get the data, right? This is how you will get the data, right? Your idea is to find a line that separates this. This is your logistic regression. Idea of logistic regression is to club all the people who are going to churn all the red dots separately and all the yellow ones separately. Is this line not doing that perfectly? This is what logistic regression is right. It is trying to fi fit a line that perfectly classifies your examples or your data sets between yes or a no, 7:09:33 between red or a yellow. Okay. Will this line tell me will this line not fit or will this line not tell you perfectly? Perfect. Now one very stark and a very powerful explanation that will clear all your concepts in one in just one example in this one visualization. Okay, which is what what is the concept of linear regression? What is linear regression going to do here? linear regression will try to fit a line like this, right? Because over there the motive is to reduce the error to fit a line that reduces the error not separate them. Correct? It wants to fit a line so that it passes through almost all the data points if it can. If it cannot then the error should be less. Everyone with me? 7:10:33 Now you understand the difference between regression and classification also. So logically thinking people who have really high problems with the platform and who are not given really huge discounts they are more likely to switch right. So those are your red dots and the people which are your yellow crosses or the yellow people they are the ones who are having least number of problems with the platform. They don't have to reach out to customer care again and again and they are being given really good discounts, really high discounts. So they'll be happy, right? They will not churn. So here red one churn probability is high and here not. Let me write it that that way. not shown. Okay. The idea of logistic regression is to separate the red and the yellow 7:11:35 points completely out. And for the linear regression, the idea is to fit a line so that the errors from all the data set is really less. There is some linear relationship between number of calls to support customer care and discounts. Okay. Cool. Perfect. Perfect guys. So now this out of the way, this out of the way, let us also do some derivation. Let us do some derivation. I will now mathematically prove it. I have proved it visually and geometrically that you know this is this this is this and I've also given you an example how it works. We are just you know solving something really under the layer of logistic regression. We are going into real details that why it is called 7:12:36 regression. one even though it is classification how does it fit my data okay and what does sigmoid do sigmoid is nothing but it is just a wrapper on this line it is just a function on this line which will now help me to say whether it is zero or one it will give me probability whether it is between 0 or one okay that's the only thing now with derivation I'm going to prove that okay so Now this will not take much time guys. Um log uh sorry odds we already know that this is this. Now taking log on both the sides. This is first step. This is second step. Taking log on both sides. It's log of odds 7:13:37 log of your 1 minus p. Okay. Then we go into now with the example that I showed you. I showed you that you know this this curve this scurve can be converted to a problem which is from plus infinity to minus infinity using this right using log of odds. So what I can say is in other words if I instead of this I put the equation of line the equation of line here which is w + w1x is equals to log of log of p 1 minus p. You understood why why I put this value equation of line just now we saw it visually right now the other property this is the third 7:14:38 okay this is coming from here proved already and then the fourth line is property now this is for all the non contact people and people who have not looked at exponential. Okay. Property is that whenever you have to get rid of log you you take an exponential. So the property is log of to the base a for log of x. Let me put n here. Log of some number and if its value is some feature. This can also be written as n is equ= to a to the power x. I am not doing anything here. It is the property I I'm just putting it here so that you can quickly revise and all for my non tech people as well. Okay. So we 7:15:39 are going to do the same thing here with this. We are going to do the same thing here. So this will become what? E to the power W + W1X is equals to probability 1 minus probability. Okay. And now fifth step is that now I will just solve this problem. I will take this here. Okay. Chain rule. I will say something like this. Just be with me guys. It is going to be multiplied by 1 minus p. Okay. And now I'm going to take this e ^ w + w1x inside it is going to become e ^ w + w1xus p * e ^ w + w1 x1 7:16:40 is equals to p. Now I'm going to take this p here again. Okay, it is going to become e to the power w + w1x is equals to p plus this minus will become plus there plus p * p * ew + w1 x1 and now if I take common P out of this. Seventh step is E to the power W + W1X is equals to P 1 + Now anyone who can see where I'm going with this. The last step is if I bring this here this side what 7:17:44 will it become? It will become e w + w1x over 1 + e to the power w + w1x is equals to p. Have you seen this equation somewhere? Have you seen this equation somewhere? Since you are you have not seen it, what you can do is you can divide. This is an optional you know step. This is eight. So we'll go into ninth step. Ninth step is you divide both of these with e to the power this this term. Okay. So this will become e to the power w + w1 x1 divided by e to the power w + x1. 7:18:47 Here in denominator it will become 1 + e ^ w + w1 x1 this entire thing divided by e to the power + x1. In a nutshell I will not go into the maths. In a nutshell this will become this and this will cancel out. Okay, this and this will also cancel out but this will come here. So it will become 1 / 1 + e to the power minus of w + w1 x1. And if you say w + w1 x1 is equals to ycap. This is nothing but 1 / 1 + e to the power minus ycap. 7:19:49 What is this? This is sigmoid. Didn't we talk about this? This is the equation of sigmoid. Okay, this derivation no one is going to ask you. Okay, no one is going to ask you. But I wanted to show this to clarify that mathematically this is possible. This can I not say can I not say that this is nothing but some verification some function of your sigmoid which is given on ycap. Okay. So sigmoid will give you the probability. What did we have here? We had probability here. Right? So it will give you the probability here when you're trying to look at all the values. It will convert this linear line to a value where it can be looked at as a probability that how much probability it 7:20:51 is to go to spam, how much probability it is to go to not spam. No one is going to ask you this derivation but it is very important to know at least from the understanding that how did people go from this scurve or you know started with linear line go to an scurve. Okay. Perfect. All right guys. So I think we have covered this. Now let me see what I wanted to Okay. We have already talked about the balanced data and the imbalanced data. 7:21:33 imbalanced data. Okay, we have already looked at the fraud problem. Out of 100 100 applications, 95 of them are genuine and 5% of them are fraud. Now if you are a very smart person, if you are a very smart person, okay, why you will think that I'm getting all the applications if I make a model or if I tell that whatever application you're getting just say just predict that it is what genuine 7:22:34 If you just make this, if you train your model just to don't pass anything, just constantly say that it is genuine. Whatever loan that application that you're getting is genuine. Then how accurate will your model be? You have 100 emails out of which 95 of them are genuine. Five of them are fraud. So this I will say not fraud. This I will say fraud and if your prediction is your ycap is 100 of them are 100 of them are not fraud for all of it you will say that this this is your prediction what is the accuracy accuracy is 95%. This is called dump model. Anyone can make this dump model. No brain. 7:23:34 But is this 95% right? Even though it looks like 95%. It looks like a really huge number. Is it doing what we need to do? If tell me bank processes the loan for these five people you know how many people have gone out of India right they have fraud they are frauds how much money can they take away with them all these five frauds or four frauds or whatever these people are if you end up giving loan to them they will never repay they will run away with bank's money and it is lost to the bank right even though your model is 95% accurate. Is your mind blown? And you are simply saying you have created a very stupid model that whatever application you are coming giving to me just say process the loan for this guy. It is always saying not fraud and 7:24:34 process the loan. That's not right. So the important thing for here what is important here the important thing is that we do not process loan for even a single fraud person or a fraud application. That's how that that is what the banks want, right? That is the ultimate goal of the bank. So here with this example, you know that accuracy 7:25:35 is not the right measure for classification. For classification problems 7:25:58 or let me put this very this is going to you know something that People generally ask in their interviews that why accuracy is not used. Looking at alone accuracy is not the right measure for classification problems. Everyone with me? Even a dump model can tell you that you know with 95 accuracy 95% accuracy you're not building any model there. You just put one line that whatever in the code your code will look like this. Just say or predict there's a constant value there which says not fraud. Still it is having 95% of accuracy. This is why data scientists, machine learning engineers, AI engineers like you are paid really high because anyone can 7:27:01 make that 95% accurate model but you are the one who need to go in detail and see that what is the ultimate goal of the problem. The goal is I shouldn't okay I can process for all all 95 of them but more important for me is I need I shouldn't be processing the loan for these five people especially okay I can reject seven of them I can reject seven of them but out of all these 100 two could be people who were genuine but I still ended up predicting fraud for them but I need to make sure all of these seven uh applications that I'm rejecting. They should be at least closer to fraud. They should have all these five loan applications which are absolutely fraud. There is some collateral damage by rejecting loan for another two people or three people which might or are at a 7:28:03 risk of fraud based on the probability. Okay. So that is why accuracy is not the alone looking at accuracy is not the right measure for your classification problems. Okay. Especially in imbalanced data set even in balanced data set never look at accuracy alone. In classification we have something called as confusion matrix. 7:28:36 confusion matrix. Okay, we have something called as confusion matrix. Why the name? Because it confuses a lot of people. It confuses a lot of people. That's why the classific uh the name is confusion matrix. So you need to be extra careful when you're doing classification. Okay. So, so we know that the outputs can be only 0 and a one, right? The outputs can only be 0 and one. So, what will you have? You will have this. Okay? You will have this H. So here, imagine Okay. So imagine this is your Y cap meaning predicted. 7:29:38 Okay. And here is your Y meaning actuals. Okay. Now you could be predicting. I'm going back again to the spam and not spam because it is easy to understand that and we've been talking about that example really well. So this is not spam. Okay, you have predicted it is not spam. All the values on the columns are your predicted and you have predicted it is a spam. Let me do it this way. And the actuals are it is actually a spam mail. Okay, it is actually a spam mail. dot spam. Let me put that in green. And in actual it is not a spam. These are the four instances that can happen 7:30:39 that you have predicted. What does this this block say? This is one 2 3 what else? H four. Okay. What does one tell us? One tells us one is ycap is let me put this as zero zero. Okay. Prediction is not spam. We already know I'm not marking it here. Okay. Not spam is zero. Spam is one. So here it is zero. Your y predicted is zero. And ycap is your y actual is also zero. So this is what this is correct right you have correctly predicted that it is not spam and in reality also in actual also it is not spam I'm talking about this one okay let go to the let's go to the second number in this box what is 7:31:41 this is you have predicted it is a spam meaning ycap is equals to 1 and what is your y actual y actual is zero. This is wrong. Right? This is wrong. Right? You have predicted that it is a spam but in reality it is not a spam. This is this is the reality here. It says that it is not spam but you are saying that it is spam. Okay. Now let's look at the third one. Third one is what have you predicted? Prediction is not spam. In reality what it is in reality it is spam one. Y is equals to 1. Y cap represents or this this this hat or cap. Okay. Represents your predictions. So you have predicted zero which is not spam. Okay. And in 7:32:42 reality it is spam. So this is also wrong. Right? This is also wrong. you have missed. And the last one which is the fourth one four here you have predicted you have predicted it to be a span meaning you have predicted it to be one and in reality it is one. So y is equals to 1. So this is correct. So these two are wrong but these are right correct. So you need to get the maximum value here. 1 and four should be high because they are correct. But two and three, the numbers two and 7:33:44 three here should be low as much as possible because they are your errors. Correct? I am just uncomplicating or unconfusing this confusion matrix with you. So just go with me step by step. Okay? Because if you come back the next week or even the next day, you'll get confused which one is too positive, which one is too negative, what is that, what is not that. So I'm just it everyone in in the industry confuses on this a lot. That is why the name is confusion metrics. Okay? So just stay with me. I am making it unconfusion metrics for you. Okay? Now if you look at it, if you look at it you know logically let me copy this to save time. Okay. So the idea is if I remove this H the 7:34:44 idea is all of these they have two values. They have two values here. I want 100% attendance here. If you look at this. Okay, if you you know see understand what I'm trying to explain, you will never forget in your life what is confusion matrix. And trust me guys, this confusion matrix is really important and is one of the best pet questions of your interviewers. Everyone asks this, everyone in one way or the other, everyone asks asks this question. Okay. Now you have to fill these two blanks here. What are these two blanks? Okay. One tells you what you have predicted. Okay. 7:35:45 The other tells you whether you are right or wrong. You are true or false. Okay. So here let's say not spam is what is negative right let's say this is negative and spam is positive because we are trying to get which is spam so for us it is a one so it is going to be positive for Okay, it is going to be positive for us. Now what you have to tell me is you have to tell me what value will come here. We'll first fill the second second blank. This is spam. What have you predicted? This is negative, right? You 7:36:46 have predicted negative and this is not spam. And in reality also this is not spam. So is this true or a false t right? So we are doing true here. What should I fill here in in the second blank? What is your prediction? Spam. Spam is positive or a negative? It's a positive. And you have predicted it is spam. But in reality it is not spam. So this is true or false? F right here again. Here what what what should I put here? You have predicted what? Positive or a negative? You have predicted what? Positive. And this is you have predicted spam. It is also spam. This is true. And what will this come? This is you have predicted negative you have predicted not spam and it is 7:37:47 spam. So this is what false congratulations guys you have completed logically you have completed or filled confusion metrics and trust me I'm coming from the industry I take interviews and the amount of people who get confused between TN FP FN TP where it should be where it should not be the amount of people who fail into this and are not able to answer this are startling very high. This is such a basic but this is where you create the impression. If you're not able to answer this you know interviewer will be like you know who it doesn't know anything this is just basic. Okay. So now these are called your type one 7:38:49 errors. Type one errors and these ones are your type two errors and trust me guys even I get confused even I get confused. Okay. So this is the way that you can instead of memorizing logically you can understand we'll talk about what are these two errors type one and type two but now can you tell me what is your ideal model ideal model based on the spam example only that I gave you fraud example fraud example of 100 fraud or 100 applications. So based on this what we have studied. 7:39:56 Okay. Can we say that these two numbers have to be the maximum? Okay. This has to be not spam. This is not spam. And this is spam. This is spam. This is your YCAP. This is your actuals. Okay. So these two numbers have to be really high. So in ideal world, the ideal mode uh the ideal model you had 95 of them which were not spam or you know not fraud. Let me in fact change this to fraud problem. This is fraud. Okay. And this is not fraud. How many of them were not fraud? 95. Right? In reality also it is 95. And you should you would want to your perfect model will also want to predict all 95 7:40:57 of them as not spam. So this will become 95. Correct? And five of them were fraud. In reality, your perfect model or ideal model would want to predict all five of them correctly and you would want to make these type one and type two errors zero. Correct? This will give you what? 100% accuracy because your errors are zero. This is the perfect model. The uh example of the fraud, what did we do? We say that we have 100 loan applications. Okay. And out of them we know 95 of them are not fraud. We know. Okay. They are not fraud and five of them are fraud. We know this. So your job as a person who is working 7:42:01 as a top level data scientist with some bank let's say JP Morgan Chase or HDFC or City Bank excess anyone mashrek bank from Dubai and all even in Swiss bank imagine you know even Swiss banks you are given the responsibility to make a model that is effect. How will you make it? Your goal is that you should be able to predict all 95 of them correctly. You should you should make them as u you should predict that they are not fraud for all 95 of them and should you should also be able to predict that these five are only the fraud. That is your perfect model will have zero errors because we have already seen that this false positive and false negative are type one and type 7:43:01 two errors. There are some sort of errors. What type of errors? We'll get into it. So your idea is that these two values should be zero and these two values should be absolutely correct. So your model should predict it is not fraud. And in reality we know 95 of them are fraud not fraud. In reality in actual we know 95 of the applications are not fraud. Your model should also say 95 not fraud for these 95 applications. So I'm putting a 95 here. Okay. Your prediction of fraud should be exactly the same where these are fraud. In reality, if it is a fraud, your model should also say it is a fraud. Okay. All right. So now what we have seen is we have seen the confusion matrix. We 7:44:03 know how to fill the confusion matrix. We know what type one errors and type two errors are. It is nothing but some sort of incorrect right. So they have a false in front of it. False positive is your type one errors. False negatives is your type two errors. And then we also looked at what is the ideal model you know. So this one would have 100% accuracy. Okay 100% correct. Let me put it as accuracy also. And here the accuracy is also good. Why? Because your type one and type two errors are zero. Okay. So we are not looking at only accuracy alone. We are looking at the errors also. Okay. Now formally what happens is 7:45:04 there is something called as precision. Okay. So there is something called as precision. 7:45:18 Okay. Let me talk with an example. There are two cases in your spam and not spam. One is you receive a spam mail in your in your mailbox in your inbox. Meaning it wasn't classified as uh spam but it is a spam. And what is the other thing? The other thing could be that you that your I hope this doesn't happen but imagine if if this would happen how how bad it would be. Your offer letter that came through an email mail mail goes to spam. [Laughter] Tell me now guys, which one is more dangerous here? This thing is harmless, right? 7:46:20 This thing is harmless. You are receiving a spam mail in your inbox. It is there in your inbox, but action is not yet taken by you, right? But you interviewed somewhere and you you got the job. The offer letter did not reach in your inbox ever. It was wrongly classified as a spam mail. So that is more dangerous to you. Right? Now you're thinking that you are not placed. You did not receive any feedback, any offer letter from the company. But in fact, you have already been shortlisted and given the offer letter. So this is dangerous, right? So this is dangerous. This is more dangerous. Now let's think about this. What is this? This is a false positive or a false negative. What is the first one? We are just making our understanding even more concrete with these examples. 7:47:22 Okay. First is false positive FP. Correct? So remember we spoke about two fill in the blanks that we have to do whenever we are doing such things. So what did we predict? It is an actual spam. But what did have you predicted? You have predicted not spam. You have predicted not spam. That is negative. Right? You have predicted negative. And it is a spam mail but it came in your inbox. That is false. Right? Your prediction of negative is false. So this will be false negative. And this one what is it? What have you predicted it to be? You have predicted it to be a spam which is one zero or one. One is positive in our 7:48:22 case and that is also wrong. Right? So all those who said it's it's false positive. Do you understand why it is false negative and why the second one is false positive? So the first one is that you received a spam mail in your inbox. Okay, imagine that you have a model which is responsible to pulling pushing those emails which are not spam in your inbox and the ones which are in spam which are spam according to the model should go to the spam folder. But this one came to your inbox folder. Meaning what has your model predicted? Your model has predicted it is not spam. Meaning it predicted negative. Out of these two fill in the blanks, it predicted a negative clause for it. Meaning it is not a spam. And that decision is false because it is actually 7:49:24 a spam mail. So it is false negative. The other one, second example is that your offer letter has gone to the spam folder rather than coming to your inbox. That would have been the right thing. It has gone to your spam folder. So meaning your model has predicted that it is spam. So you have predicted what? Positive class for it. Spam. Okay. And that is false. That is incorrect. So it is false positive and in this case false positive is dangerous meaning here the idea is that false positive should be minimized. Correct? Correct or no? And true positive should be improved in this. If you look at this, if you look at this, this value should be minimized and this 7:50:26 value should be maximized. That is your precision. So precision formula is 7:50:40 precision formula is TP over TP plus FP. How precise are your predictions? How precise are your predictions? On the positive side, how many are you able to correctly say they are positive? Out of all the ones that you have predicted as to be positive. Okay. How precise is your output, your predictions? Then second is second is recall. Now for recall I have another example. I have noted it down. Huh? Okay. So now you have to tell me this. You have again two cases. The idea is that you have to predict whether a person who is having cancer you have to 7:51:42 detect whether it is cancer or it is not a cancer. Okay. You have to your model has to tell whether looking at the condition of the patient or looking at the graph or your images of probably chest X-ray or something like that you have to say whether it is the person is having cancer or he he or she is not having cancer. Okay cool. So there could be two cases. A healthy person is considered as cancerous. We are talking about only the errors. Okay? There are others as well. But I want it to be very clear for you which is this type one versus type two errors. Let me put it down here. 7:52:48 Type one versus type two. Okay. This is one situation. The other situation is what? A person is cancerous but you are saying that he is healthy. A cancerous person is considered healthy. Now first tell me which one is more dangerous to write. A person who is healthy will be starting taking the medications. Correct? There will be some side effects of the uh medications but it is not life-threatening and in subsequent tests it might there might be a possibility that it is detected that this person has become 7:53:48 healthy. So they would claim that the medicines have worked but initially it was like you know the person was healthy itself. Imagine it or relate this with your CO also a healthy person was considered to be COVID positive or a COVID patient that is more dangerous or a co patient in the test it comes out that he's he is healthy he or she is healthy okay related that way also so more dangerous is this Now tell me what is this and what is this? Which one is false positive and which one is false negative? What is the first one? Let's look at it. What have you considered? What have you predicted? You have predicted that a person is cancerous. Meaning one cancerous is one 7:54:49 non-cancerous is zero. when your motive is to find which one is cancerous earlier in the earlier example which one is spam. So what have you predicted? You have predicted a positive for this. You have predicted that the person is cancerous which is false because obviously the person is healthy. Now can you tell me what is the second one? Second one is false negative meaning you have predicted it to be a healthy non-cancerous person meaning you have predicted negative for it and that is false because in reality the person is having cancer so it is false negative. So here we saw false positive was dangerous. Here we see false negative is 7:55:49 dangerous. So depending on condition to condition things change. Okay. Depending on your data, depending on what is your motive and depending on what is more dangerous to have, which type of error is kind of acceptable, not kind of acceptable, but is less dangerous to the other one. So here the motive is that you need to that you need to reduce f negative correct here you need to reduce the f negative in this one. Okay you need to say that it is less negative. So the formula of recall is PP over PP + FN. 7:56:53 Okay. Here the motive is that we should improve our true positives. Meaning the ones which are cancerous we need to improve their prediction. It should we should also predict cancerous patients. Okay, you're recalling out of all the ones which are positive, you should be able to tell the ones which are positive correctly and you should have less people who are having who are false negative. Okay, true and false is based on whether it is right or wrong. So here this one these are the two blanks that you have to fill right in let me close this in this confusion matrix it is two blanks first one is what you have predicted okay positive is the case when what is 7:57:53 your goal so if you want to identify spam once that is positive if you want to identify cancer patients that is your positive Okay. And the second is whether what you have predicted is right or wrong based on actual. We looked at these, right? We looked at these 1 2 3 4. All the ones that are correct are having your true there in the first blank. All the ones which are wrong, predicted versus actuals, they are false. Okay. It's not easy to understand this concept but trust me if you are able to revise this I guarantee you you'll never forget the way I have explained it here it is very logical and also with examples you will never forget it given you understand this otherwise the other option is you can simply go and memorize 7:58:54 that you know what is TP what is FP what is TN fn and all those things. Okay, I would recommend you think it logically and as in when you keep doing this, no, it will become second nature to you. You'll do this like this. Don't have to think twice. Okay. So this is your metrics that we call. So we have looked at now what is accuracy? Let us also look at accuracy. What is accuracy? Accuracy is the ones which are true over total. Right? So accuracy based on your confusion matrix can be TP plus TN. The ones which are true over total total observations and total observations are TP plus FP PN plus FN. 7:59:59 Okay, accuracy is nothing but all the ones that you are correct with over total number of observations you're correct on one and three or one and four depending on whatever you are doing this these two boxes you are correct only here and here let me take it from here one and four is where you are correct out of total okay so that is your accuracy And there is this one thing called which is F1 score. F1 score. This is nothing but a combination because now I told you that somewhere you would prefer to have a higher precision, somewhere you would prefer to have a higher recall depending. So in the cancer problem you want to have a better recall. But in your spam and not spam, you want to have a better you want to have a better precision. So F1 score 8:01:02 combines both your PR uh precision and recall. Okay. And how it is related is your F1 score is nothing but twice the precision over precision plus recoil. Okay, which in other words, what was your precision? Your precision was TP plus TP plus FP. Okay. And what was your recall? Okay. Let's leave the maths. There's enough. I feel it is nothing but it is twice of TP over 8:02:04 TW of TP plus FP plus FN. This is your F1 score. You can either use this one the bottom one or the above one is one and the same thing. Okay. If you solve this, no, if you put the values of recall and precision here, you will get this only. It's one and the same thing. Perfect. Awesome guys. Is everything clear now? Logically everything should be clear now. Okay. Why we need to look at accuracy in regression? Why we cannot look at only accuracy in classification? Okay, we looked at all the examples. Let's move on to the hands-on part. Okay, cool. All right. Now, let's go on to your hands on. 8:03:06 Okay. So, here I have this data which I've already uploaded. It is the heart attack data. Okay. So, we we've been looking at this all along. So let me go ahead and first you know do all these. Okay. So let me first import everything. Import what pandas as pd. 8:03:46 Okay. I will also import seaborn because I want to show uh one thing in seaborn also. I hope you have covered that. And then there were some others also that we were looking at in linear regression. Idea is same guys. Idea is same. This is as okay. Get the data understand the data. Okay. Then separate your target and uh train variables. Target as well as your uh test as your uh train. Okay. Do the split. Get your independent features and your uh dependent feature separately or your target feature separately. Then do the split. After you do the split, train it. After you train it, predict it. After you predict it, evaluate it whether it is correct or wrong. So here I am importing. 8:04:52 Okay. Then uh let me go ahead and at least for a few lectures I will not add all these things that I want in one go in the first cell like I like to do it. But here what did we do? What do we have to do? We have to read in your data frame which is let me say this as CSV. What is what is the name? R1 dot CSV. Okay. And then we will also do a head of it. Oh. [Music] Okay. So this is our data. Okay. Now what is the data set? What have I told you that once you get the 8:05:53 data set? Try to understand whatever you want to understand of the data set. So that at least from making the model site it is not important but when that's the difference between a great data scientist or a you know principal data scientist a great machine learning engineer AI engineer data engineer all those things that's the difference between the one who is great and the one who is just getting the job done. Okay, get the data. Try to understand even if it is not in your you know comfort zone. Now this data set is basically the data set of the people who get heart attack. Okay. Now this is your target variable. It has already been given. Okay. I'll explain a few of these. Okay. Because most of it is from the biology side based on the test. But this is the age of the person patient. Okay, this is the 8:06:54 sex male or a female. CP here tells whether it is chest pain and it is of four types. 0 1 2 3 I know I've looked at the data so I'm telling you it is four types. So it is basically whenever you get a heart attack you more less or likely it is that you will get a pain in your chest first. Some pains are angina anginina related uh pains atypical angina typical angina. If you go into these uh you know terms you'll get to know about it. But for our understanding these are different types of chest veins. Some are related to heart attack some are not. Some could be for example could be due to gas. Some could be due to something X factor some other reason apart from heart. Okay. Imagine it that way. Then this T rest BPS is nothing but your resting blood pressure. Call is cholesterol. This is fasting. uh fasting sugar, blood sugar. Okay, this is 8:07:56 another ECG test is done in the heart resting how it is looking like. These are some other tests. Okay, old peak is when it is peaking slope and all these are all you know technical terms from the medicine side but at least the ones that you're able to understand you'll be able to get to know that okay these are some of the conditions that are noticed in the patient what are and some of the values of the tests that we are seeing here and the target variable is one when it is a heart attack zero when it is not a heart attack okay now let us look at and describe the data. What are we going to do? We are going to Okay, let me first do this. It has 303 rows and 13 columns. Okay. So, patient is 303. We have the data for 303 patients. Now, we 8:08:58 are going to do df do.escribe We will also do info. Okay. 8:09:21 And then okay. So what does the describe tells me? It tells me all the ones which are 303 the count looks like all of them are having right values. Okay. And then the mean age on an average the patient age that we getting is 54 which is likely right people as in when they get older the risk of getting them a heart attack is more compared to someone who is young. Correct? So this connects with you. Then uh there is some ratio that we seeing the mean average is this. We cannot make anything about this because this is 0 and one. Okay. Categorical feature. The only thing that we can understand from this is close to 25% of the people are zero. Okay. Meaning if it is a female it is zero. If it is male it is one. Okay. So more than 50% of the data is from males. 8:10:24 Then chest pain what is the again categorical value here so we are seeing 25 percentile 50 75 is 012 okay it is very equally distributed then PP BP BP readings okay so here on an average the BP reading is 130 which is kind of in the normal range right and some people have an extraordinary high BP it's really high 130 120 is considered to be normal cholesterol value. See a person who is having this high cholesterol. Cholesterol is nothing but as in when you keep on eating oil it uh you know or fat the wrong fat it it accumulates in your body. Okay. It obstructs the blood to flow from your veins arteries and veins. Okay. That could lead to heart attack. So a a general normal value is below 200. But we are saying all the patients at least 25% of them or more 8:11:27 are having value 211. So high co cholesterol is related to heart attack. Even without looking at anything, even without you know predicting anything we have it looks like you know high cholesterol high cholesterol above normal value is related to heart attack. Okay. And similarly you can look at the others. Cool. Now info we are saying that okay all are null and I'm getting one float value and I'm getting 12 integer values. Okay. And the data set is pretty small since it is uh this thing only 300 rows. Okay. So let's let's go ahead. What what can we do next? So I can show you the value counts meaning how many uh how many of them are zero and how many 8:12:28 of them are one in my target column. In my target column out of the 303 people I have 165 of them have a heart attack and 138 of them are not having heart attack. Zero is not having heart attack. Okay. So people also like to visualize it. So let me do a count plot using Cbond. X is nothing but my target variable and the name of my target variable is this. My data is DF and my palette would be palette is nothing but uh a color scheme okay that that you want to uh choose. So there are many color schemes that you can choose. I'm using sticking to the one that I have been using a lot. 8:13:31 Okay. So, what error are we seeing? Pallet spelling. What is the spelling of the pallet? Pallet. Pallet. Huh? 8:13:48 What are we seeing here? Let me just check what is going wrong here. Data is equals to DF only. The spelling is wrong here. Okay. So now you see let me minimize this. Now you see you have plotted around 138 of them are zero and close to 168 of them that we have seen this number 165 of them are one. Okay. So we have just plotted a simple plot here. Okay. Now what is the next step? What is the next step? Next step is to create your data frame or creating your um X and Y separately. Okay. So how are we going to do that? 8:14:55 Everyone understands this eyelock right? We have covered this. So all the features, all the columns except the last one is your independent feature and only the last one is your target variable. Correct? So let's try to see x dot head. Let me not take much space here. So you have all the columns except the last one. And in your Y you have only one column which is your target. Okay. Once you have done this, what did we do in linear regression? Then we did splitting of the data. Okay. Now, how did we split the data? 8:15:55 From sklearn dot models model selection import. What did we import there? Train test split. Okay. And then what did we end up doing? We end up creating my X train X test. Second value is that. Then the third value is Y train. Then we have Y test. And what do we do? We call this function which is your train test split. And what all values does it take? It takes your X, it takes your Y and it takes your test size. I'm going to keep it again as 20%. And the last thing that we looked at random state I'm going to say one any number if 8:16:57 you want to repeat it any number. Okay let me run this. Now once you have this let us look at s x train dot shape. It is 242 and X test dot shape. You know what? Let me print this neatly. Give me a second guys. Extreme shape is equals to this. These are F strings, right? I hope you know. We also looked at it last time. And I want to say Y train here. I will change it to Y 8:18:02 just for creating some separateness so that it is easy to understand within this class. I'm just printing this. So this will be my test. This will be my test. Test test test. Okay. Let's run this. Okay. So 303 20% goes to test. So 30 and 30 close to 61. 30.3 and 30.3 close to 61 goes to my test and the remaining goes to my train and I have chosen it to be 8020 because my data set is pretty small here. Okay, it's only 300 patients data. Cool. Now what do we do next? What do we do next in linear regression? What did we do? Create the model. Absolutely. Model 8:19:03 fitting. So next we will do what? even fit the model or build the model. Okay. Now over there what did we do over there? We imported from skarn dot what was it? Linear. It was linear models right. Linear model we imported there what? Linear regression. What are we doing here? We are doing logistic regression. So we will import logistic regression. Okay, cool. Now what are we going to do? Over there we created a model which is model. Now I'm just to make it separate. I am creating a instance of that class which is your logistic regression. Okay. So I am creating this instance of 8:20:04 the class in log model. So that you know you are able to understand it is a log model. So log model and what will my log model do? Okay let me first show this. Let me run this. Now here earlier we were getting linear regression. Now we getting logistic regression. Right? So log model dot fit is what we have to do. And we have to do fit on what? We have to do fit on train period. Correct. Perfect. So we will do fitting here. Then this is nothing but warnings. Okay. Data conversion warning and everything. Now what is the next step? predicting with the model and everything that you have learned all the patterns all the relationships that you have learned which is log model. So using log model what are you going to do? You are going 8:21:05 to predict and you're going to predict on what? You are going to predict on X test and you will save all your predictions in Y bread. So now you have your predictions, you have your test. Let me show you this is your prediction. Okay, let me print a few values. Let me also print a few values from your actual data set, your test. It's nothing but 0 1 0 0 0. Here you are saying 01. Is there some change? No. Looks like you're predict uh no is there no here it is one last third and second last are 0 0. Here you are predicting this to be one. Okay. But in reality it is zero. So you see one error there. Okay. Now we know what we have what we 8:22:06 have the predictions and we also have the actual values. Okay. We need another five minutes and logistic regression is done. So now what did we study after we predict the model? We have to evaluate the model whether the performance is right or wrong. So in linear regression we looked at the evaluation should be done using RMSSE MSE uh R square okay mean absolute errors but here what have we studied we have studied confusion matrix correct so we will import what confusion matrix so from skarn dot h let me do that here skarn dot matrix import should be from what are we going to 8:23:06 import what is the name confusion metrics okay Python is very intuitive and now to show you something but okay let me show that later. Let me first complete this. We're already 2 minutes up. So, ah, so here, let me print this and let me call this as confusion matrix = to what do we do? First value is what? actuals, right? First value is actuals and then second value is your predicted then we will print confusion matrix. This is your confusion matrix. Okay, let 8:24:08 me copy paste this. This is nothing but that uh uh that that thing that we were talking about. This is your what? TP TN PN. This is what FN FP. You're getting those values here. Okay. Now, another way that I generally try to use is this. Okay. I try to use confusion matrix display. 8:24:46 It takes in the confusion matrix itself so that it can print it properly. Okay, let me put this and then it has what? Display labels. What is the uh let me show you what I'm looking at. Uh whenever I am stuck, I look at skarn confusion metrics with labels. And this is the first one that I'm opening. And I go to the documentation. And there is more likely or no there is an example. So let me copy this. Okay. Your predictions label is equal to CLF classes. I'm putting that here. I wanted to show you this 8:25:51 but ultimately what it does is it shows you this I don't want to take up much time the same thing that you're seeing there it shows it in a pictorial manner okay so that it is intuitive to understand. Okay, let me delete this and let me there is another thing that you can do which is metrics which is your classification report. This classification report and then it does what? It is nothing but is the same thing. Just printing it. Classification report and you're passing in the same thing there. First value is actuals then second value is predicted 8:26:51 and you get all the things precision recall F1 support is another thing but accuracy weighted average macro average this is what you're getting right all the things that we looked at today are everything here okay so this is your logistic regression Okay, >> just a quick info guys. Intellipad offers an executive post-graduate certification in AI and machine learning in collaboration with iHub, IIT, Rurki and Microsoft. This comprehensive course is designed to make you an expert in AI and ML in just 11 months by working on over 50 real-time projects and industry case studies. With this course, we have already helped thousands of professional and successful career transition. You can check out their testimonials on our achievers channel whose link is given in the description below. Without a doubt, this course can set your careers to new height. So, visit the course page link given below in the description and take a first step towards career growth in 8:27:52 the field of AI and ML. >> Okay. So here now we have looked at all your precisions, all your recalls, we have looked at the evaluation metrics. But there is one more thing that I want to talk to you about. Okay? So I need you to understand this TPR and FPR. Okay. Can anyone guess what is this? Can anyone guess what is this? This is your true positive rate and this is your false positive rate. The rate at which you are having true positives and the rate at which you are having false positive. Okay. The idea is that if you plot if you plot here if you have TPR in your Y and FPR in your X. So false positive rate is you want it to be very less. The idea is you want to reduce this, right? And the idea is you want to improve your true positive rate. Correct? Everyone with me? Come on guys, 8:28:52 everyone with me. You want to reduce your errors. In this case it is false positive rate and you want to improve your positive predictions which is your true positive. Now what happens is if you try to plot this if you try to plot this it looks something like this. Let's say what what is the maximum value that false positive rate can take? It can take one right it can be 100% right? So 100% in terms of numeric is one. If if you are getting a 100% of false positive rate it will be one. And if your rate of true positive is what one that's the maximum that it can be and here it is zero okay so this this is the value that you can get correct I have just drawn a dotted line okay now in reality if you try to you know iterate on all the thresholds okay this is how things will look like this is how things will look like okay this is what are you measuring 8:29:52 you're measuring that on this let's say your uh false positive rate is 10%. 1 what is your true positive rate there? If you try to map this if you try to map this this is what you will get. Let's say this is 04. So your error is less 10%. But your predictions your true positive rate is also 40% only correct. The idea is that you want to maximize your you want to maximize your you want to maximize your true positive rate while minimizing your false positive rate. Okay. So in that case isn't this that value let's say this is2 and let's say this is85 isn't this peak that it is taking the best one because over there your false positive rate is 20% and your true positive rate is 85%. Correct. Uh.3.1 whatever. Okay. And please share the notebook notebook. Sure Leslie I will. 8:30:53 And there was someone who wanted me to do the image part again. We will move on to the uh collab only after this. So I'll I'll do that once again. Okay. Anyways, we will be plotting this this also in in the notebook. So I'll take it there. Okay. First just concentrate here and understand what we are trying to achieve. But can someone also tell me that the value if I try to take from here will be somewhere around 92 then why not choose this point I'm getting a better TPR right the true positive rate is better there but still I'm going with this 85 why but FPR is high exactly exactly okay so the tradeoff there is a trade-off if you try to plot it here this will also be somewhere around 65 right you you know we have to increase our TPR but we have to also keep FPR minimized with this tradeoff between 85 to 92 see how much of an error we are increased so that's why the closer it is 8:31:55 to FPR least FPR and maximum TPR is the way to go so this is not right and this is correct is that clear is that clear we want FTLS it is FPR okay Cool. So what you have studied guys is is called ROC is called R O and ROC stands for receiving operating characteristic. How this name came and why I have no idea. they have you know taken this concept from uh I can explain but I don't want to go there because it will be digressing from machine learning a lot but the concept is coming from your uh electrical field okay electrical engineering field where they look at you know how and you know it over there also we have something on the transmitters if I'm not wrong transmitters or transformers this is the same graph that we get there so that is why it is called the ROC Okay. Now, now since you have 8:32:58 understood what is your ROC, let's move on to another thing which is very simple. Which is very simple. Let's plot the ROC again. Okay. This is your TPR. This is your FPR. Okay. And this is one. This is one. And we will try map it in here. Okay. Now, this is one of the things models that you have tried. Okay. And there is another model which works like okay and there is this. Now you have to tell me one which is a better model. Let's say this one is M1. This one is M2 and this one is M3. Yellow one. M1 is M1. M1. Okay. Cool. That's right. M1 is the best model. How did you decide that? You looked at this value. You looked at the trade-off between your TPR versus your FPR. Here, this is the value of TPR, correct? And this is the value of FPR, correct? This is for M1, 8:33:59 M1. And this is for your M_sub_2, right? So obviously M1 is better. It has lesser FPR. So for M1, M1 is best model. Correct? But what is this M3? Remember in in linear regression we compared our model with the average model. This M3 is nothing but random model. Meaning if you don't do anything and randomly keep saying yes no yes no yes no yes no yes in a random fashion. You can go ahead and check this mathematically also. This is the line that it will give you. Okay. Okay. Now tell me one thing guys. If you have this another model which goes something like this. Okay. Let's say this is M4. What can I say about M4? 50/50. Yes. Red line is your 8:35:02 random model. It's 50/50. What can I say about M4? M4 is worse than random model also, right? Which is very sad. Even if you randomly do things ever ever if you have built a model which is M4 which is underperforming than your random model also which is M3 then that's not the right model. Correct? You are even performing worse than random model. What's the point? You are your model is predicting even worse than guessing. Correct. So M4 is a big no. Okay. Is a big no. Everyone everyone understands everyone understands that. Okay, cool. Now, how there is another metric to compare it is coming out of this only. Okay, now can I can I just copy this and somehow explain that? Let me do that. I don't want to let me try to do all this. Now, can I say can I say this is the 8:36:05 area under curve? Can I say this or no? For model M1, isn't this the area under curve? You get the idea. I'm not able to color it properly but you get the idea right for the green model what would it be for the green model for the green model I'm just removing this will this not be your area under curve right so can can I say that area under curve area M1's area under curve A is more than M_sub_2's area under curve right this is how you compare this is how you compare okay so if a model's area under curve in the ROC graph okay is more compared to the other then you say the model which is having the highest area under curve is better model okay and this will always be less than one I'll tell you why so let's take the same value same 8:37:06 thing this is your TPR this is your FPR here you have one here you have one when I join this remember I drew this in the beginning this is what can I not say that this is the total area under curve right and what is this what is this this is this entire width is one this is also one so area of square or rectangle is what 1 square is 1 okay so the maximum Area under curve is equal to 1 and probably your m_sub_2 is around.83 and your m sorry this is m_sub_2 and this is let me complete this first is around 63 m1 is that clear guys? So this is the better model. Okay. So this in short 8:38:08 this in short is together called R O C A U C which is nothing but this is your receiving operating characteristic and this is nothing but area curve. Cool. Okay. So this is how you compare the models. Okay. This is how you compare your models. So now let's dive into the code and see how can we get ROC there. Any queries before we jump in there. So now let's go into the hands-on and see that for our model that we have already created. Okay, for our model that we have already created, what does the ROC looks like? Okay, and we will also create this again. Right, we have stored the confusion matrix here in CM. Correct. Now let's go and do this again. Let me take my keyboard. Let's do this again. So from skarn from skarn dotmetric I'm doing that again but you 8:39:11 need not we've already you know imported it. import confusion matrix display. Okay confusion matrix display. Once you do that you call that confusion matrix display. You pass in your this is only for display. You have to pass in your actual confusion matrix here. You have to pass in your actual confusion matrix that you have stored in cm. Okay. So you will pass this cm and you will say dotplot. Okay. And here on the side it is saying no true label means the actual here it is saying predicted label meaning the predictions. So you have predicted zero but it is a one. You have predicted negative n and this prediction is also wrong. So this is fn. This is tn for roc curve. Skarn has made 8:40:13 it so easy. Skarn has made it so easy. It has done all the heavy lifting lifting for you. It has done all the heavy lifting for you. So what you'll do is from metrics you will import ROC curve. Okay. And you will also capture the ROC score. Okay. After you do this, after you do this, now pay very uh you know full attention here. I'm guessing there are no queries. I'm just looking. Cool. All right. So once you do this, once you do this, what you have to do is you have to there is this function called ROC. Now whenever you don't know I'm just showing this how do I add a tab when this is not there I want to add a new tab but I am unable to because this thing is coming in front of me okay so whenever you don't know what what is happening 8:41:15 okay go to ROC curve okay just take a guess guys will ROC curve here. Let me go back to there. Will ROC curve will ROC curve accept what values as arguments? TPR, FPR. Okay. What else? Any any other guesses that it might take the FPR and TPR? But let's see what does it take? Go to the scikitlearn documentation. What does it take? It takes your Y true and Y score. Okay, this is not saying y predictions. This we understand is your y test. But what is this y score? Let's look at this probability estimates. These are your probability estimates. So how do we get the probability estimates? So I just want to look at we have the log model, right? I want to see if it's already there. Yes, it's there. So here 8:42:17 let me show it here itself. So when we when we wanted to predict the label right what did we use we did y test correct correct or no and what did it do model doy y it's x test right we passed in the matrix and it gave us what zero and one correct it gave us 0 and one there is another thing which is log model dot predict dot roba. Yeah, what will it give us? It will give us the probabilities. Let me save it here. Probabilities is equal to and we have to pass in the x test. Okay, let's look at these probabilities. Rather than having this zero and one, it is giving me probability. Till now we 8:43:18 have understood all the probabilities in uh logistic and everything right. So this one is this first one is zero second one is one correct. So that's why this target is becoming zero because there's 99 pro probability. Thus this will become a label zero and there's only 0.5 and here it's 79% probability that it will become label one and this is 2 that it will become zero. That's why you see one. It is automatically doing that for you. So if you have to look at the probabilities you have to look at predict proba but if you want it to only be what uh the labels based on 0.5 you will do only predict. Now you know how how the logistic regression is putting the probabilities. Okay. Now this is what we need to enter here the probabilities. Now let's do 8:44:19 that. Okay. So here and what does it give us as an output? It gives us it returns FPR, TPR and the thresholds. Thresholds are the thresholds that we have to decide. Okay, either 54.3 which one is better. So we will do this. We will first is FPR then TPR then we have thresholds then we are going to call ROC curve and we are what was the first input the first input was Y test and then the other one was probab probes probabilities and what was the other one labels and all okay that's fine we can run it with this also okay but Now your probabilities is is a is what is having two columns right is a list with having two indices. 8:45:19 This one first one is always related to zero. The second one is always related to one. So in general everyone is trying to do one. So your case is that you need to detect spam which is one. You need to detect people which are having cancer is one. Okay people, what example did we look at? Fraud. Are they fraud? One. So this is what we are going to pass in probabilities. We are going to pass it the probabilities of whether it is a spam or a not spam. Okay. So we are going to pass all the rows but the first column 01. And let's see what it let's run this first. Okay, now we know what is FPR. Let's print it out. There are so many values, right? For each threshold, it is trying to do that. For TPR, so many values, correct? Now, the point is that 8:46:20 we need to plot the ROC curve. How do we plot it? We will plot it using plot dot figure and then we will do plt dot plot. You have to pass in your FPR and TPR. Then the other labels and all I will show but I think let's do this first. Do you see the ROC curve? Let me you know put in these labels so that it is much more clear. True positive rate and let me put what title? Title ROC. Okay. So is this clear now? This is your true positive rate. This is your false positive rate and this is your ROC. Okay. So in our case the problem that we have solved isn't this the one that you're looking for if your FPR is what 40%. So that will come from business that what is an acceptable range of SPR or based on your use case you would know that this is not acceptable. This high 8:47:22 is not acceptable. Anything between 04 or less than 04 is acceptable. Anything which is less than 0.1 is acceptable. Anything beyond that is not acceptable. Okay. So you will get this. Now let me also plot this plt dot plot. Let me also plot in. So what is this? The value that your red line the random model can take what x-axis is 0. The first point will be 0 comma 0 and second point would be 1 comma 1 right? Second one would be 1 comma 1 right and we will make it as red dash line. So this is your random model. Okay. Now we will also look at the area under this curve. So the area under this curve would be what? ROC A score. And then we will pass what? We will pass Y test. And then this is my predictions. 8:48:24 My predictions were Y bread 73.6. So you know the area under curve. Now is everyone clear now? I know I'm making so much of typos. I don't know. I've got this new keyboard probably because of that if ROC covers more area then model is good. Yes. Yes. And generally the way things are done you know you will always have this peak somewhere. Okay. For all all the lines that you'll be drawing here you'll be having that. Okay. Cool. So now we've covered ROC. We've covered ROC under the curve. And now we'll move on to what we have finished logistic regression. What are the limitations for ROC curve? See I cannot cover 100%. But in ROC there are a certain limitations. Okay there are certain limitations that's why we go ahead we also somehow look at uh it's 8:49:26 not limitation it is requirement. Okay, I'm covering this but is there any alternate? Yes. Yes, there is. There is. Let me go back and explain this. Okay, so here what happens is now we've looked at ROC under curve. Anywhere under ROC we are looking at TPR and we are looking at FPR. Are we looking at looking at the negative predictions? No. Right? We are not looking at true negative predictions. The true negative rate meaning we are looking at the emails that were correctly specified as spam. Right? We are not looking at uh your uh you know cases where how well are you doing when you are not predicting the non-spam as not spam also that is your true negative rate. Okay we 8:50:27 are not looking at that but in some cases some cases when that is also a metric to look at we are only concerned about if my true positive rate is better. Okay, I'm able to identify the fraud. I'm able to identify the spam. But there is a very small very small uh you know idea that you in some cases rarely you would also want to look at how well am I doing on the other side also on the other label also which is your true negative rate that's when your ROC A will not you know kind of come in handy then you have something called as PR curve and this is nothing but this is your precision ision recall curve. All the concepts are exactly the same. All the concepts are exactly the same. Same as ROC. Okay. Here you will have precision. Here you will have your recall. Okay. And the area or or this is 8:51:31 how in that case things will look like. It's something looks like this. Okay. You need to find something which is which is having precision also high and recall also high re also high. Okay. Every concept of ROC is exactly the same. It's just that your y-axis and your x-axis has changed and this is a mirror image because over there precision will be higher recall will be high. That's the motive. Okay. Okay. But this is not good model. Exactly. So if you if you had looked at the accuracy if you had looked at the accuracy did we print that? Let me see. Okay, let me print in the accuracy also here. So the accuracy of the model was so the accuracy of the model is also coming from skarn dotmetrics we do what score okay or it's already there. So let me do log model dot store 8:52:34 and I will pass what my this is the if you look at the data set this is the accuracy that you'll be getting this is the accuracy that you you'll be getting okay in this case what has happened is your AOC curve is also similar to this but if you look at some other cases you know your accuracy especially when the data is imbalanced your accuracy will be very high like 95% but your area under curve will be less your precision will be less here. Look at the precision. Where is that? Here. Look at the recall. Okay. Look at the F score. So you have to look at all these things together. Since our data set was balanced, it was an imbalance. You are not seeing very different things, right? Model is bad. The thing is you need to go ahead and do some changes in your model. Now you need to try to remove some columns. You need to try put in some more uh features for it to improve. Okay, that's the idea. We 8:53:36 we are doing one pass but you know what the next step is how we can make good model in this. Go try remove a few columns, do some data exploration, try to do some manipulation. Give it a shot. Okay, that's how we learn. See, I'll tell you what. Practice is what is, you know, very necessary. I can keep on doing these things, okay? And you would feel that this is very easy. This is very easy. I'm telling you an honest thing, okay? But when you try to do it, you'll find hundred of errors. you'll find hundred of questions. That's where the learning is happening. That is where learning is happening. I showed you how much time do we take for you know making this model. There are steps that we already know. How different is linear regression with logistic regression. It's the same steps correct everything is already done in that library. So until and unless you practice guys 8:54:37 things will not be 100% clear. I can clarify all your doubts. I can clarify all your logic. I can show you the hands-on. We can do this together. But maximum learning will happen when you do it at your end. You need to do things at your end. Okay? Take at least two data sets. Take at least three. There are so many data sets on the internet. Try running linear regression. Try running logistic regression. We've done both of them, right? Make some changes in the data sets. make some changes, do something new, right? That's where the art is coming into the picture, right? You can create the model, you can evaluate the model, but then next how to improve the model that takes skills of different level and that comes with practice. Okay, cool. See, uh, an gradient descent is an optimization, right? So internally what it will try to do is it will try to reduce the error using the optimization 8:55:39 algorithm right this thing that we are looking at this thing that we're looking at for increasing model accuracy huh but see gradient descent will work if you have given I will explain this with an example so if you have given four columns four independent features okay and you're trying to find the best M or C in linear regression. Okay, for those set of input features, those set of independent features, it will give you the best M and C. Correct. But it is your job to go back and see that what if I give only three columns. What if I give only five features? Then when you pass it, then gradient descent will look at those five and give you the best M and C in linear regression. And then you need to compare that adjusted R square exactly is better than what you were getting with four features. Okay, gradient descent is 8:56:42 working underneath already but you need to give a different input for it to work differently to it to further reduce the error. Okay, perfect. Which is naive base. Okay. So this is used for classification. Okay. And these two words this base comes from B theorem. Everyone who knows B theorem in probability or has heard about it? I don't expect you to remember it. But have you heard about B theorem? Not for and naive. What does naive mean? Does anyone know what does naive mean? So in Hindi if everyone knows that it is bulah. Okay. So this is the meaning of naive. Okay. Lack of experience or judgment. Very innocent. I would say innocent. Okay. Is innocent. 8:57:47 Okay. If you know this term or innocent very naive you know is beginner kind of thing. Very simple. So now let's understand this why it is called so. Okay, base theorem. So basically all of this is under probability. Okay, we looked at logistic regression. Okay, we looked at odds, we looked at log odds, we looked at probability, how things are different. But B theorem is specially based on the uh knife base ispecially based on base theorem. Okay. and this innocent I'll come to this why it is called this naive but that is one of your interview questions okay this is an important question what is naive about naive base classifier okay this is something that you need to know guys 8:58:47 this is something that people generally try to ask okay if if your uh interviewer is in the right state of mine this might be one of the first questions to test your you know basics around Nbase we'll come to this we'll answer this I want to give you the opportunity to look around this but let's go more onto this technical term which is you know base theorem so all my non-math background friends and non- tech background friends u just trust me and what I'm writing here are pure simple formulas pure simple uh you know properties that have been defined. Okay, we are not going to derive anything but it is something that is already existing in mathematics. We cannot learn unfortunately we cannot learn nine ways without going into maths okay because the whole fun is around probability and mathematics okay so I'll tell you I'll 8:59:48 I'll try to simplify as I can okay the more I can I'll try but this base theorem okay is told as if your probability of getting this is what this is probability ility we know this is telling me some probability correct this is and what is this is event A this is event B correct what is this what is this what does this mean unrelated fe uh kind of that but yeah I'll tell you we'll capture it so this in short is called how you would read this is probability of a given b has occurred. Okay, meaning what is the probability of A if B has already occurred? Okay, that is what this tells me. A is event A, B is event B, p denotes probability. This 9:00:49 sign here is not of division. Okay, this is not of division. This is how you would read it. Okay, probability of A given B has already occurred. Let's take an example. Imagine you have five red balls, okay, and four yellow balls, okay, in a box, okay? And now you have to tell me what is the probability of getting a yellow ball. What is the probability of getting a yellow ball? And what is the probability of getting a red ball? So this is five, this is four and total is 5 + 4, 9. What is the probability of getting a yellow ball? 4 by 9. 5 by 9. Yes. Okay. So, this is four. We have four here and total is 9. So, it is 9. Okay. And this is five. Total is 9. So, 9. Okay. Now, what happened is one person has already taken 9:01:50 out a yellow ball. Let's say so Lakshmi has already picked one ball and that ball is yellow one. She has already picked from here. Okay. Now you have to tell me what is the probability of getting a yellow ball given this event. Let's call this event as D. Okay. Given D has already happened or let me write it in a cleaner way that Laxshmi has already picked one yellow ball. What is the probability of getting a yellow ball in that? Probability of getting a yellow ball when Lakshmi has already picked one yellow ball out of the blue bag. 3x 8. 3x 8. Good. You guys are smart. It is 3x8. How did we get here? How did we get here? Let's try to do this step by step. Okay. So, let me copy it here. This is your step one. Step one, this is what you have. Correct? Then what is step two? What happened? Laxshmi came 9:02:52 and she I didn't realize I'm skipping a word or skipping a letter. Laxshmi came and she picked one yellow ball. When this happens, how will your system look like now? How is this bag going to look like? Isn't this system going to look like this? That she has already picked one ball. So, how many balls are now left in yellow? Balls are left are three. And how many total are left now? Eight. Correct. So what is the probability now? That's what you were saying, right? Three out of total eight. So this is the probability. Okay, everyone with me? I've tried to explain it as easy as I can. You have considered that one event has already happened. Lakshmi has already picked one yellow ball. So your blue bag will have now three yellow balls and will have the remaining five balls because remaining five red balls because nothing has been done to that and now your total will be one less because one ball has been taken 9:03:55 out. Okay. What will be the probability of getting a red ball when Lakshmi has already picked one ball already picked one yellow ball? Can you guys help me with this? What is the probability? Correct. Correct. 5 by 8. Right? Because now you have five balls and total is 8. Okay. So that's why it is 5 by 8. Now hold on to your seats. Okay? Hold on to your seats. So base theorem is nothing but this is a formula. Okay. This is a formula for base theorem. Base theorem says okay that probability of A given B has occurred is equal to is equal to probability of A okay into probability of B given A has occurred over probability of B there's nothing fancy in this this is the formula of base term okay for you to predict or to get a probability that A has occurred after B 9:04:57 has already occurred You can calculate it by using the probability of a individually into probability of b happening when a has occurred divided by probability of b. Okay, this is the base theorem. Okay, everyone with me? I know you will not remember this formula but at least for now you saw the formula, right? Correct. Some 11, 12th, 10th marks maths is coming up to your mind. Even ninth standard, 10th standard probability okay used to solve this. So now if I say how do we do we take this concept for our machine learning data set. How do we apply this here? This is really the crux of it guys. Okay. This is really the crux of it. You need not spend any other video or any other literature on this. Just pay 100% attention and it is going to be super clear. Okay. So imagine you have this data set. Okay. This is your y which is your target variable and this is your x 9:05:59 which is your independent. Okay. And you're having feature one, feature two, feature three and so on. Okay. And you have only one target variable. Correct? So how if I want to know the probability of y given x probability of y which is your target variable given x how will you calculate it? How will you calculate it? Now you know you have to somehow get a probability whether this comes in zero or y or a one which is your y given x has occurred meaning some feature feature one feature two feature three given these are the values what is going to be my y can I understand this as that let me write that what is the probability of y okay Given it has seen features f_sub_1, f_sub_2, f_sub_3 and 9:07:00 so on which is nothing but your x. Can I call it like this? Right? And from base theorem you know that probability of y given x has occurred. When you say occurred meaning the data is already there in heart attack data you know the cholesterol was 240 for example. You know the age was 61. So it has already occurred right? So in this case it would be probability of y into probability of x given y over probability of x correct acts like target variable correspondingly. Yes. Yes. Okay. Now what is this x? We know x is nothing but is a combination of feature one, feature two, feature three and so on till n features. Correct. Okay. So can I write this above equation like this? Let's say this is one. Okay. This is two. Can I write one as this? That probability of Y 9:08:03 is nothing but when feature one, feature two, feature three so on till feature N has occurred. Can I say this? And just imputing the values of X. Can I say this? It is nothing but probability of y into probability of feature one, feature two, feature three so on till n features till n features given y has occurred over probability of feature one, feature two so on till feature m. Have I done anything different? Have I done anything different? I have just used this and put this value here. I have just used this and put the value of X there. Right? I hope that's clear. That's very simple. Third, now the other property other property of probability is if A and B are independent. If A and B are independent, then probability of 9:09:04 probability of A, B is equal to probability of A multiplied by probability of B. This is the property. This is the rule. If A and B are independent, correct? Meaning occurrence of B does not influence across of B Accurrence of A then these two are mutually independent. Okay, they're independent. So this is how you say. So now we know our features one, feature two, feature three are independent. And that is where naive base is kind of naive. Okay, it assumes it assumes that all the features are independent. Okay, it assumes without looking into anything very innocently very innocently it will say whatever features that you're giving they are I am assuming that it is they are independent to each other okay which may or may not be the case that is a different thing okay that is the assumption that knife base works with that my features are independent so when 9:10:05 they are independent can I now break the above equation like This given feature one, feature two so on till feature three is nothing but probability of Y multiply by probability of feature one given Y has occurred multiply by feature 2 given Y has occurred and so on till probability of feature N given Y has occurred divided by probability of feature one into probability of feature two and so on till probability of feature. Is this clear guys? Right, we are using the just this rule and we are saying we are imputing all the values that probability of feature one, feature two, feature three, feature four, feature five, all of them occurring together given Y has occurred is equal to probability of feature one when Y has occurred into probability of feature two when Y has occurred into so on. Okay, 9:11:05 this is how you apply knives base to your data set. That's done. It's done. Knive base. In knive base, if you understand the base part of it, there's nothing crazy about knives base. It just works on probabilities. And this will now tell you what whether your target variable it will give you a probability whether given all the features your y is.33 or 78. If it is 78 let me give you an example. Okay. If this comes out to be 0.33 or this comes out to be case 1, case 2, this comes out to be 78. Then what is your prediction going to be? In this case it is going to be zero. In this case it is going to be one. Correct? Right? Remember the spam and not spam. Remember this graph that if it is 0.5 this is going to be 1. This is going to 9:12:05 be zero. Given the threshold is if it is greater than 0.5 it is going to be one. If it is less than 0.5 it is going to be zero. Remember this that's what we are doing. Have we done anything new? Right? This is what your prediction will be. No. And ultimately what you have to do you have to predict the label only. Even while doing logistic regression we looked at predict probabilities. It was having probabilities right? So what did it have? it has if the probability of having one is 78 probability of having a zero is 1 minus 78 okay is that clear guys internally internally I showed you in logistic regression when we were doing okay I showed you let me go back there this is for our understanding okay internally I showed you the probabilities where are the probabilities ities show you know uh I told you the 9:13:08 probabilities is.99 05 so if you sum these two it will become one so the probability of it being.99 is zero okay correct if it is that then you're going to say zero okay so this is where this is slightly different how things are calculated and how it is there. So I explained you with 0.5 that is what you have to remember but mathematically what is happening is it will tell you the probability of it having a zero is this probability of it having a one is this. Okay got it. That's why we are saying even if it is 33 we are saying it is zero. So we will say this is nothing but this is nothing but this is nothing but you always predict the probability of probability 9:14:10 of one. So this is telling you probability of it being y= 1 which is.33. So what is the probability of being a zero 1 minus.33 right? And this is also telling you probability of it being a spam or y is equal to0 is 78. That's why you're going to say it is one. This small minute difference this 33 tells you its probability of being a 1 is.33. So what is the majority prob probability? The majority probability which is 66% or 67% is of it being a zero. That's why we say it is a zero. Okay. Little confusing. Yes. So you have to not confuse. I'm telling you here now so that you can clarify it here. Okay. So that you can clarify it here because 9:15:10 when you I'm already assuming and treating you guys as data data scientists and machine learning engineers. You're just learning on concepts. Okay, I'll explain it once more. So whenever we when we did predict proba, it gave us what? It gave us probabilities. Correct? It gave us probabilities and the probabilities were looking like this. This is zero and this is one. This was what.99 and this was 01. Correct? And then what about the next one? Let's say let's take our exam 33.33 and this is 67 and this is 0.78 and this is 0.22. Okay. This 33 is nothing but this. This 78 is nothing but this. So when it is very low 67 is zero right? So it will be zero. When 78 is more greater than.5 9:16:10 just remember greater than.5. Okay, it is one. Okay, cool. Third case anyone would have asked me could be that it is exactly 0.5. What would you do in that case? What if it is.5? Nothing. You It's a 50/50, right? It's a 50/50. So, no one is going to question you whether you're saying that.5 should go to zero label or.5 should go to one label. It is your choice. Okay? And it is also the business choice because it has equal likelihood that it could be a zero or it could be a one. It could be a spam, it could be a not spam. It could be a fraud, it could not be a fraud. So depending on the situation, it could be a cancer, it could be a healthy person. depending on the situation. If you are more conservative meaning you want to you know say that even if you know someone gets 50% chance that of that person is a cancer patient you will say 9:17:10 that it is label one okay so in that case you will say.5 is one so it depends it is 50/50 it is right on the edge it is that's why it is 0.5 right 50% 50/50 okay that's why I'm not talking about it but it is widely accepted it can be either here either Perfect. Now this I have just said here to connect it with predicted probabilities. How it comes in the code? Comes in the code or skarn. Okay, this comes from logistic regression example. I've just given you an example that you shouldn't be like you know you're saying 01 and you're saying that it should be label zero it should be but you should be looking at only this then you shouldn't be looking at probability of zero and saying that it is.99 and I will mark it as one that is the confusion I wanted to clarify you 9:18:12 need to be very clear that you're looking at this value you're looking at probabilities of zero given y has occurred or you're looking at probability of one if y has if all the features has happened that you need to be very clear here we are looking at probabilities given y is equal to 1 given it is positive ppr okay so that's what I wanted to clarify cool now we have done this let's do one example which is what practical so I have already already created a data set so that it's easy. Okay. So we will do one one practical example. We will do one practical example how knife base works. Okay. Of how knife base classifier works. Let me put this example here. Yes, I 9:19:16 have it. Okay. This I have created a data set. This is your feature one. Let me call it X1. This is your feature two. Okay. And this is your target variable. Okay. So it says how is the outlook of that day? Meaning it is rainy, it is overcast, it is sunny or something else. Is there anything else? No. Right. And how is the temperature? Temperature is hot, mild or cold? Okay. And given these two features, will you play or not play? That is given here. No means no. There it's not a good time to play, not a suitable conditions to play. And yes means it's a suitable condition to play. Data set correct. Okay. Now we have to do what? Now you have to tell me how many we are going to look at outlook first. We are going to look at outlook and this is sunny. How many unique 9:20:17 values are there in the outlook? Come on guys, I need you to be very interactive. I have not done this. I want to do this with you. Rainy. And what is the other unique value? It is overcast. Correct. And how many yeses are there here? When it is sunny. How many nos are there when it is sunny? Tell me. So when it is sunny, how many yeses? Three, is it? And how many nos? How many nos? Two. Okay. And when it is rainy, how many yeses? So when it is rainy, how many yeses do I see any? I see 1 2 right. Okay. Similarly, rainy and no. How many? Rainy and no. 1 2 3. Okay. Overcast and yes. Overcast and yes. Three or four. Weird, right? Data set is weird. When there is overcast, people are playing. And when 9:21:20 overcast is there, how many nos do we have? Zero. Correct? Now, if this is your attribute, let's say this is your attribute, outlook attribute. Okay? Can you guys tell me based on what we have looked at till now, what is the probability of a yes given an attribute has occurred? Meaning what is the probability? Let me hide this. What is the probability that it is yes when it is sunny or before that? Let me do this. Let me do this. Let me do total. What is the total? 3 4 7 and 2 9. Okay. Yeses are nine and nos are five. Correct. Now what is the probability of a no? Will this probability yes and this box would not mean 3 over 9 sunny and a yes correct? No. Sunny is no and total 2x 5. 9:22:22 What about rainy? 2x 9 overcast 4x 9. Tell me this guys probability of rainy probability of no here in this box when it is rainy 3 by 5 because no is a five. Okay, let us now fill this 0 by 5. Okay, and this is 100%, this is 100%. Okay, now to make things quick, we will also have to do this for temperature. Okay, we will also have to do this for temperature. And temperature is what? Hot, cool, mild. Okay, here we have yes, no, then we will have probability of yes and probability of no. Okay. To do things quickly to not waste time in this, we have done one. I have it here. So, let me pick this, this, and this. And let me copy. Let me come back here. Okay. So, we have everything here. This 9:23:23 is what we saw for Outlook. There is one mistake that I that I'm noticing now. This is three and this is two. This is 3x 9 and this is 2x 5. Correct. Correct. Right. That's what we calculated. And I'm putting the temperature one here. And this is nothing but probability of yes or a no. Total yeses are nine. Correct. Out of total how many are this? This is this is 1 2 3 4 5 6 7 8 9 10 11 12 13 14. You can do a total sum from here also. But I wanted to show you guys. Okay. Is the data set clear? Perfect. Now I have to find now you know whatever we did earlier we have to do it here. Okay. So I have to find probability of playing. Yes. Is a plane. Okay. 9:24:26 Probability of playing given it is sunny and hot. How can I do this? You understood the problem right? Y is nothing but play golf. Are you able to understand? Is this making sense? How knive base is working on your data set? You have to find the probability that it is likely to play given it is a sunny and a hot day. Use base theorem. Exactly. So we are going to put base theorem into place. So we are going to say probability of Y given sunny and hot is nothing but probability of Y into probability of sunny given it's a yes multiplied by probability of hot given it's a yes divided by probability of 9:25:28 sunny over probability of Not. Is this clear guys? Is this super clear? What you have to do now? You have the probability of Y. You have the probability of sunny. You have the probability of sunny and a yes. You have the probability of hot and a yes. This one and this one and this one. Can't you put the values here and solve it? Correct. Will you be able to will you be able to do this? Okay. And similarly, you can also calculate probability of not playing given it is a sunny and a hot day. This only, right? It's just that y will now become yes. Y will become no. Correct? And you'll take the opposite values. Okay? After you put in these values, 9:26:31 this thing will come down to be somewhere around 46% or 45%. Okay? And this will come down to be somewhere around 55% or so. Okay? Or in other words, it is going to be 045 and this is going to be 55. Okay? Depending on what is your target variable you are going to predict what this is more right probability of not playing so you will predict zero you will predict don't play which is zero one was play right cool perfect you have by hand manually calculated n base with an example with a life example This is what the model is doing internally. Understood? So once you understand linear regression and logistic regression, you understand the concepts, 9:27:33 then things are simple and especially this NYN base. NY base is so simple that it is so mathematical that you can calculate it. Okay, clear guys? Okay. So I think we are going to start something really cool. You'll enjoy. So we are going to start new family of algorithms which is nothing but treebased algorithm. Treebased algorithms. Okay. And here particularly we will be looking at decision trees. Okay. and we'll also be looking at random forest. Okay, these two are very similar in terms of context and in terms of how things work. So we'll look at decision tree first. From there on you know explaining random forest will not be will not be uh difficult to even understand. It's just a broader concept of decision trees. So let's start with 9:28:34 decision trees. Okay. Now what do you understand by this term decision tree? Tree is nothing but a structure, right? something like this. Correct? It is symbolic of a structure and this is something that helps you to decide. Right? So this I can say some rules. Okay. Rules that help you to decide. Okay. Now interesting thing about these is in fact all the treebased models is it can be used for both classification and regression tasks. Isn't that cool? Isn't that cool? It can be used for both. It can also be used for uh classification. It can also be used for regression. Predicting yes or no depending on one by one model. Okay. And then break down a problem into smaller. So we can say the same thing for logistic regression right or knife base also. It's breaking down the 9:29:34 problem into smaller ones or you know depending on one by one we are trying to you know say whether it is yes or no. So very broad in in general this is what decision tree is about. Okay. Main thing to take away from here is rules and this structure. Mhm. Okay. And the m main thing the most important thing is that it can be used for both classification as well as regression. Just give me a second guys. Yeah. Okay. So that's the main thing. That's why you know these all decision trees, random forest, all your treebased algorithms are so popular because it's so intuitive. It's so easy to understand when you're working on a problem in in in the real world in the industry. When you're working on a problem, it's very intuitive in terms of explaining to the customer or your business or you know for whom whomsoever you're doing this work. Be it your boss, 9:30:34 be it someone who doesn't understand machine learning given the output. Okay, it is easier to explain to them that this is how you're getting the results. But for you and me uh you know who are in this line of work which is machine learning engineers, data scientists you know or AI engineers all these people we need to know a bit more right we need to know more than a normal person knows right that's that's when we can explain it in a in a normal fashion right so that's what we are going to cover here we are going to uncover some of the things that most of the people don't understand Okay. But why the name trees? Why the name trees? Okay. So, it comes from nothing but you know if if uh some of you are from computer science background or even the tech background you might know there is a data structure called trees right you know there are some data structures that are tree based 9:31:37 you know like the binary tree red and black trees. Okay. Then you have n array trees all these. Okay. But that's that's why that's why how decision tree works is based on the tree. Okay. So that's why the name comes as tree from there. Now to understand this tree you have this one. Okay. And this is how the tree looks like. Okay, this is your root node. We call it as root node. Okay, and everything which is yes, you will have some rule here. Let me write it with you will have a rule here. Okay. And anything that follows that rule. Okay. Anything that satisfies that rule comes in your left 9:32:38 in your yes. Yes is always on the left. Okay. And this one which is which doesn't satisfy that rule is on your right. Okay. This is an example of a binary thing. That's how things look like. Okay. Now let me understand now once we know this let us understand more from the intuition side. Okay. Tree is this but how how does it work in reality? Okay, let me create a data set and from there we will you know look at how to go about things. Let's take an example. Okay, now the problem is you need to find uh employees in your in your uh company that are going to leave. Okay, imagine you work in a really big MNC. Okay. And you want to find people who are going to leave 9:33:40 your organization or are going to switch or are going to do something, you know, change their jobs. How will this help? This is going to help if the employee is really good and you want to hold them, you want to, you know, keep them back, you will probably, you know, give some increment, some bonus or something or the other. Okay, that's how and and also it will help you to plan that okay I'm going to see a shortage of let's say 10 people who are who are going to leave I need to hire 10 more people right that's the problem so this thing is called employee attrition this thing is called employee attrition basically any uh employee the rate at which the employees leave over the time that thing is called employee attrition and you need to find people who are going to leave. Okay. And this concept is called Okay. So tree structured model. Yes. Used for decision making. 9:34:42 Exactly. Clustered index uses binary data structure. Clustered index uses binary data. So it's slightly different from there in here. Okay. You you'll get the idea once we look at this example. So let me plot a graph for the same problem that we looking at. Okay. Now the issue is that we are looking at age on the x-axis and we're looking on overtime. The number of hours that a person has to do overtime. Okay? And that determines whether the person is going to leave or not leave. And also probably the age, right? Probably the age as well. Okay, let's have it here. And then let me quickly plot the graph. Let's take this. So, okay. So, these yellow dots are the ones that are going to 9:35:43 leave. Okay. And I will have it somewhere here also. Okay. Based on the people who are having a high age, they are also going to leave. And the people who have really high overtime they also will leave. And then let's also plot set of people who will not leave. Okay. So let us plot it here all these people. Okay. Now tell me you you get the idea. You need to find people who are going to leave. This is just a similar example like attrition is an example like customer churn. Okay, you need to find people who are going to leave. Now this is a classification problem. Okay, it is a classification problem. Yes or a no problem? Correct. It is a yes or a no problem. 9:36:44 Yes or a no. Right. And I'm going to say all the folks all the people all the people which are with green I'm going to say that you know they are going to stay. Okay they are not going to leave. And all the people which are yellow dots here in our example they are going to leave. You understand the problem? We have to find attrition rate. So this we will keep as one and this we will keep as zero. Okay. Now tell me one thing. It is a classification problem. Okay. Let me give you a hint. Okay. I feel if I give you this hint, you can come up with something. Let me put this as 35. Okay. Let me put this as 29 and this as 3 hours. 9:37:47 Okay. This is 29 years of age. This is 35 years of age. Okay. There is one predict probability. But how? Yes. Yes. You're on the right track. We'll product predict probability. But until now until now whatever you have learned right you have learned that there should be a linear relationship right if I had a problem if my end I cannot fit a line here like this right this will be a logistic regression line correct I cannot do that I cannot also split something like this that everything on the left is stay and everything on the right is leave because then we'll be you know mclassifying so many points right that's how logistic regression will work even if we try to put something here it will 9:38:49 not work right and in linear regression in logistic regression we cannot do this also right we cannot put a line put put uh uh something like this which is nonlinear understanding understanding the problem now correct thresholds no thresholds will not also work so basically this is where your treebased algorithms you know are really good they're really good why because you see that there is no way that you can draw one single linear line there is not a linear relationship between out over overtime and age, right? As as the overtime increases, you see some people who are still staying back. Okay? As you see your overtime increasing, let me put a line here. You still see that some of the people are 9:39:50 staying which are green and some of the people are leaving, some of the people are leaving. Correct? And here there is no definition of age also that anyone who is above 35 will leave. That is also not only true. You also have people who are less than 29 they also leave. But less than 29 also there are some people who stay and some people who leave. Okay. So in these cases okay what what can I say? I can can I say can I try to make a you know tree here let's make a tree here which is if I say my uh age is greater than 35 okay my age is greater than 35 then I will say it is what on the left hand side what it is it is yes 9:40:51 Right. I will get what I will get leave. Let me write it with yellow so that it is I will get all the people who will leave. Correct. But then if they are less than 35 it is a no. I cannot still say that they will definitely leave. Correct. This we have identified with this. Yes, this we have already done, right? Anyone which is greater than 35, they will leave. Correct? Everyone with me till now? Okay. Then on the no part, when the age is less than 35, when the age is less than 35, can I say that people with less than age 29? Can I say people with age less than 29 and having an overtime of 9:41:52 greater than 3 years or 3 hours are also going to leave. Then we will be able to identify this area. No. Correct. Understanding the rule, understanding how we are reaching that area. It's very intuitive, right? It's very intuitive. Easy to understand. Correct. But how do we put this in in the tree? How do we put this in the tree? So and we also need to identify these people also. Correct? So how do we do that is in the no part there is another question that we can ask. We can say people with age less than 29. Age less than 29. Correct? People with age less than 29 we know it is yes is going to be on the left. But can we say something about people with no like greater than age 29? Can we say something about this region? We already checked greater than 35. They are no 9:42:55 here. Meaning they are not belonging here. They are greater than 35 and less than 29 or is here but are greater than 29. Less than 35 and greater than 29. Isn't that this area? Right. What can we say about these people? We can say that they are going to stay. Can we or can we not? Okay, I hope this is now much better so that we make the tree also with along with looking at the data, right? Not leaving. Correct. Less than 29 will leave due to overtime. Yes, that's intuitive also, right? People who are new into the job, they're young, okay, they can do overtime. But if the overtime is really huge, okay, greater than 3 hours, they will leave. And anyone who has aged a lot, okay, they they are going to anyways, you know, they have spent so much of time in in the company, they're going to leave, okay, but between this range between, 9:43:56 you know, 29 to 35, they don't mind doing overtime also above 3 hours. That is what we are seeing from the data, right? So here we have got all the people who will stay. But what about this? What about this region? We are still left with this region, right? This entire region. And how do we classify them? How do we distinguish between them? We say we have another feature which is overtime. If overtime is greater than three then and no this is no. This is yes. If the overtime is greater than three what do we have? People will leave. If the overtime is less than three and they're less than 29 years then they will stay. Correct? Isn't that right? So essentially essentially these are the rules and 9:44:57 these purple lines that I was showing you is a stepbystep criteria that we created right at the end of the day at the end of the day you are going to look like your your your uh this thing is going to look like your final or your line is going to look like this. This is what we wanted right in fact correct now you are able to identify everything with this tree right and this is nothing but I can say this is a tree correct and what have you noticed till now it's nothing but rule based we made so many rules we made rule which is this age should be you know greater than 35 we can have some people which are going to leave and some people we cannot decide about. Then we created another rule which is this one. Okay, which is this one. And we're going to say that 9:45:59 you know that the age is going to be less than 29. If they are greater than 29 and you know less than uh 35, we can still say that they are going to stay this period this this region. Okay. But then in this region we can put another rule about the overtime and say that you know if it is going to be greater than three they're going to leave if they are going to be less than three then they're going to stay. Everyone with me do you understand this now? Right? Clear key things that or you know okay let me put it as key things. What did you notice here? All these white lines these are axis parallel. These are axis parallel meaning either it is going to be parall to your either it is going to be parall to your either it's going to be parall to your x-axis this one and this one or it's going to be parall to your y-axis this one or this one. This one is also parallel. Correct? You cannot fit a line 9:47:02 which is you know at some angle with the axis. It has to be parall to either x-axis or yaxis. Okay. because it's very intuitive. These conditions you can only put if you're looking at either y-axis or your x-axis, right? Both of these are your features. Okay? You cannot say that okay if you know when the age is this do this when the age is this do this. You cannot create a line like this. You can create a line you can create a boundary for classification like this white line based on multiple rules in multiple nodes. Okay. So the this is the first thing that it is access parallel and the other thing is isn't this similar to many if else conditions? Isn't your tree like many if else conditions here right? If this if age is greater than 35 then do this. If age is less than 35 then we have already reached all the set of people we have classified them well. No 9:48:04 need to classify them further. Okay. Then on the right hand side if age is less than 29 we have not been able to classify all of them then we need another condition right isn't this like your if else yes it is okay so these are your key things and you can create here a nonlinear model I asked you when we looked at this initially can you create a logistic regression or kn base kn still you can work with the probabilities and somehow get but logistic will not work here right logistic will not work here because it will try to fit a single line single linear line which whereas your data is not having any linear relationship correct. Okay. So these are the no need to worry about to keep the dependent independent variables in X and Y. No no it is required. It is very much required here. What was your here? What was your target variable? Your target variable was this. Your target variable was attrition and outline oh sorry the 9:49:06 overtime and the age both of them are your features. Okay both of them are your features. You're creating some rules on your features. Okay cool. You'll you'll get a hang of it. If you have understood the logic if the intuition is clear that you know how things are working I think you'll get the hang of it. Okay. No need to worry. Let me see what I'm Okay. So now let me let me do one thing. Since this is if else I can create another now you understand this I can create another uh tree like this this is your tree one correct since this is if else conditions and you can write if else conditions in multiple ways right having that thought in your mind having that thought in your mind okay let's create another tree of the same prop let's create okay now we are going to create another tree this is your tree two now instead of starting from 35 can I not say in my first age less than 29 and I will have all the yeses here, all the nos here. Okay, when 9:50:09 my age is less than 29, I can create another here saying that you know if overtime is greater than three, correct? If it is greater than three, what do I have? If it is greater than three, then it is leave, right? And if it is less than 3 and a is less than 29, we are in this region. Correct? So we are going to say that they are going to stay. What about this less than 29? We figured out and greater than 29. Greater than 29 we can have another node here. Same which is age greater than 35. Here all the nos or all the yeses and all the nos. When the age is greater than 35. When the age is greater than 35 sorry all the yeses it's going to leave people are going to leave and when the age is less than 35 which is no part here they are going to stay same rules same rules but the structure is different structure is different right same problem same 9:51:12 solution but your if else conditions are different now now you started with age 29 earlier you started with age 35 correct correct different no same features 32 is also with the same features. It is nothing like it is like let me explain this with a if else condition. If if age is or what is here greater than 35 if age is greater than 35 okay then what will you say then leave right and then what will you say? Else if else if age is less than 29 what was it here? It is less than 25 right? If it is less than 25 then it is going to be another nested if else right inside this there's going to be nested if else. If overtime is less than three or greater than three, what have we put? I just want to greater than three. Okay. If overtime is greater than three, then leave. Then else stay, right? Then we 9:52:13 come in this part. Then we come in this part. This is the first level. Okay. This is the first level. This is inner nested. If else correct, can't we can't we say that here? We are going to say what? Here what are we going to say? that age is greater than 25 29 and less than 35 then they're going to else let me write here stay else right we can write if else again this is based on tree one based on tree two we can write the if else again instead of keeping 35 here we can put 29 first and we can say if age is less than 29 okay then we go inside the else loop and we nested if else and we are going to say if over time is greater than three then we say that it is leave else it is going to else people are going to stay correct and then for this one the another part would be the another part would be based on this age it's going to say else if 9:53:15 age is greater than 35 then people are going to leave else they are going to stay this is t2 and this is t1 Same problem, different trees we can make based on which rule comes first. Correct? Right? Okay. Understood? Now, keep this thought in your mind. We'll we'll revisit it. Keep this thought in your mind. Okay. Before we do that, I need to tell you a few things. Okay. Before we do that, I need to tell a few terminologies of your tree. Let me copy this out. Any tree we can look at. I'm choosing to take this tree. Okay. To explain the terminologies. Let me make this bigger. And we are going to look at terminologies. So this is called your root node. Okay. This is called your root node. Let me try to put it with a different color. 9:54:17 This is your root node. This is what you call it. The first node of your tree. Okay. And these all these which are in the end they are called your leaf nodes or simply leaf. Okay. These are called branches. These are called branches. And what are these called? What are these called? This one and this one. These are simply called nodes or internal nodes. Internal nodes or simply nodes. Okay, this is your terminology of the tree. First n first is called root node. Then these are called your branches. These are called your leaf nodes which cannot be you know drilled down further. You have already got to the final outcome. You have all the leaf uh you know yellow dots here, all the stay green dots here 9:55:17 separated and all the green dots here. All the yellow dots here separated. You cannot further divide it. That's that is called your leaf node. Okay. This imagine this to be a tree opposite. If you have to remember this, imagine this to be a tree which is inverted. Here we have roots, right? And we have leaves on top. Correct? At the last just imagine this to be opposite rotated. You have the root on top and leaves at the bottom. Okay. Then no fruits in our tree. Okay. So now we have looked at what is decision tree, how we can make a decision tree, what are the terminologies of the decision tree. Okay. But tell me one thing. What were we trying to do? Let me try and copy this back again. What were we trying to do when we were trying to whenever we 9:56:19 are trying to do classification in in as a matter of fact whenever we are trying to do classification or any other problem we are trying to do what here exactly tell me we are trying to create a leaf which is homogeneous. We are trying to create a node, a leaf which has all the yellow points together, all the green points together. There shouldn't be any mixture, right? It should be a pure node. Correct? So the goal was goal was to create homogeneous node. Okay? Or a pure node. Correct? That was the goal. All yellow together and all green together. That was the goal, right? Because that's what you want to do. You want to separate out the greens from the yellow ones. Correct? So the idea is that you need to reach to this pure node. You need to have a homogeneous mixture. Homogeneous is same type. Heterogenesis is different 9:57:21 types. So how do we work from here? There is something called as entropy. Okay. There is something called as entropy. Entropy is nothing but measure of randomness. Okay, I'll I I'll tell you an example. I'll tell you an example. Let's say let's say you have this tree. Okay, and you have you have 200 your data has 200 samples. Okay, n is 200 out of which out of which positive is 100 and negative is 100. Okay, imagine it to be any any example but for now just assume that you have out of these 200 you have 100 in positive class 100 in negative class. Okay, you have this and your you have some feature four. You have made some uh rule. Okay, this is 9:58:22 feature four. some rule you have made and you get to these you get to these which is here you have here you have let's say you have 60 positive okay let me put it with a different color so that and 40 negative 40 negative and here you will have 60 negative correct and let me also change this negative we are representing by green and you will have 40 positive data points here. These are all your yellow points. Okay, here you have more yellow points which are positive and here you have a few green points. Here you have more green points. This is how your split is. Okay, this is some feature feature four assume. Is this better? Okay, is this better or another tree which is giving you this? Let's say this is some other feature now 9:59:24 feature three because that's how we will get a change right. So instead of having this now you have 10 positive here. 10 positive and you have 90 positive here. Let me do because the dots are more there. 10 positive here and you have 10 negatives here and you have 90 negatives here. Which one is better? Which one is better? This is your tree. Let's say this is T1 and this is T2. Which is better? T1 or T2? Which is better? Anyone who thinks T1 is better. How did you decide T2 is better? Because it has more of it can clearly say that in this part 90 of them are belonging to yellow and only 10 of them are green. Correct? Here 90 of them are green and here 10 of them are yellow. But in T1 it was 6040. Right? Here I can be 90% confident. Let 10:00:27 me write that I can be 90% confident that it is positive class. Correct? And here I can be 90% confident it's negative class. Correct? Here if I have to say the same thing I can be only 60% sure that it's that it's negative. Correct? So I'm more confident in T2. Right? So T2 is better. T2 is better. How did we decide that? We said that the randomness the mixture in T2 is less compared to T1. Correct? Right. That's how we decided. You only gave the answer, right? That T2 is better. How did we decide? We decided that the randomness, the mixture of yellow and green is more clear in T2 compared to T1. Ideally, it should be 100 and a zero. But in real life, you will not be able to do that. Right? So 9010 is really good. 9010 is really good in this 10:01:29 first level. Okay. So that's where your entropy comes in. Now this is your entropy. Now formula of entropy it will seem like a bit huge formula but stay with me. We'll take it through okay and it will be really easy. Now entropy entropy which is also defined as h of y is nothing but summation of i is equals to 1 to k probability of yi. Okay log to the base 2 probability of yi. Okay. This is the formula of entropy. Okay. This is the formula of entropy. What is p here? P is probability. Okay. What is K here? K is your number of features, number of categories. Sorry. Here in our case, in our case it was two in in the above case it was two 10:02:33 categories. One was positive, the other one was negative. Okay. Log to the base 2 multiplied by probability of Yi. These two terms are same and we have to summit for all all the categories. Okay, we have to summit for all the category. Stay with me. Okay, stay with me. It will be super clear. We need to No, no, you don't have to remember this formula. Sometimes our interviewer might ask but you don't have to buy hard this. Okay, you only need to know what entropy is. Okay, what entropy is you only need to know that it is a measure of randomness and measure of randomness meaning mixture yellow and green point just remember this example and you'll get to know okay but since we're learning it it's my job that I explain to you such that that it sticks in your mind you understand the concept and for that we 10:03:33 need to look at the you know formula let's say let's say let's consider an example example here for entropy let's say you have a bag okay in there you have three red balls and three yellow balls okay this is S1 and let's copy this okay we'll do that later but let's calculate the entropy now what does entropy say what is the value of K here K is two categories red and yellow Okay. And how many points do we have here? How many points do we have here? Six. Right? So K is 2, N is equ= to 6. And what is the formula? Formula is this. Let me write that here again so that it is easy to remember. Okay, let me keep that here. This is your entropy 10:04:35 formula. Now, how do we create this? We are going to sum this. We are going to create the entropy of S1 in this bag. What is the entropy? The entropy is my bad guys. There is a negative here. See, even I forget there is a negative here. Okay. So, we are going to keep this negative outside and we are going to create the probability of what is the probability of yellow balls? What is the probability of yellow balls? 3x 6 correct is 3x 6 3x 6 * log of 2 * log of two of probability of probability of yellow this is your yellow part okay yellow category plus what is the other red is also the same 3x 6 log of 2 3x 6 right this is nothing but and this is your red category 10:05:36 red balls. So this will come down to 1x 2 log of to the base 2 1x 2 + 1 by 2 log of 2 1 by 2. Okay, this is going to come down to log to the base 2 is nothing but log to the base 2 is nothing but what? Look at it. Okay, it is nothing but it is this is what you will get. at the end. Okay. And what are you going to get out of this is you're going to get a + one. This entire term is going to come down to minus1. Okay? So it is going to be minus/ and minus/ this is going to be + one - plus. So the entropy of hs1 is going to be + one means maximum. If we do the same thing. Okay, if we do the same thing here and I will 10:06:38 say and I will say this to be s_2. This is going to be s_ub_2 and we are going to have all the six balls as all the six balls as red. Then what how do you think this will change? This is going to be s_2. It is going to be what? minus probability of 6 / 6 log of 6 / 6 plus is there anything here? No. Right? Category yellow is 0. So 0 multiplied by anything is going to be zero. So this is going to come out as minus and log one is what we looked at in the previous sessions. Log 1 is nothing but zero. So again 1 into some 0 is going to be 0. Okay. So you see here we have a homogeneous mixture and the entropy is zero and here the homogeneous mixture is half half. It's 50/50. The entropy is + 10:07:38 one. So so the graph of entropy is nothing like this. If we have the probability here let this be a probability of one. Let this be a probability of zero. Okay. And let this be.1.5 okay 50% probability and this is your entropy when we looked it was 0.5 probability was 0.5 here right 50/50 then your entropy value was one correct and when the probability is one either all of them are one then it is going to be zero and if it is zero meaning no probability of yellow it is again zero. So the graph looks something like this. So the graph let me okay I don't want to draw this again. Let me change this to be one. This is the graph of entropy. Okay. The idea is that you need to reduce the entropy. Okay. Correct. We 10:08:40 need to remember no. Okay. I've already 100% read only. Exactly. So minus one neck is going to be zero. So whenever you look at entropy just remember this graph. It will help you. So the maximum value that entropy can take is one and the minimum is zero. Okay. Okay. Now let's take an example. We are going to create a decision tree from scratch. For you to understand how the algorithm works, it is always good to do a manual working. Okay? It is always good to do a manual working. So imagine you have this data set. Okay? You have few features here. This is your feature one, feature two, feature three, and feature four probably. Okay. And then you also have a your target variable, which is why, right? Okay. Let's say the n is the n is 1,000 points. You have 1,000 rows in your data frame. Okay. 500 of them 500 of them are positive and 500 of 10:09:43 them are negative. Okay? Let it be 0 1 or whatever. I'm saying positive or negative. Okay. Now tell me one thing. Tell me one thing. We've already looked at the feature which tells you that let's say there is this feature four. Okay. There is this feature four which tells me this that I have this is having 1,000 data points here in the beginning. Okay. Which is nothing but 50/50. Okay. I've used it to get to this uh situation where now I have 800 total data points here. Okay, I have 800 data points here out of which 300 are positive and 500 of them are are negative. Then using feature four I am able to say that in this 200 of them are positive 10:10:43 and zero of them are this. This is one of my features. Okay using feature four. The other option that I have the other option that I have is of feature one where I say here at this end I get 600 of them. Okay. I get total 600 of them out of which 400 are positive and 200 are 400 are negative and uh 200 of them are positive. Here I get ulta here I get that 400 of them are positive and 200 of them are negative. Right? Now tell me one thing for the root node. For the root node, if you were to make your you know decision tree, will you pick feature four or feature one? The first decision which which is helping you to reduce the entropy which is able to 10:11:44 reduce your mixture. Is it feature one or is it feature two? I have to start making the tree. Okay. If I use feature four, this is what I will get. If I use feature one, this is what I will get. So should I pick feature four or feature one? Okay. So let's change this. So let it be and 100. Now it's balanced. Now it's balanced. Right? Which one will you pick? So whenever you're creating this tree, whenever you are creating this tree, how do you decide whether you take age 29, you take age 29 less than 29, age greater than 35, any feature can become your root node, right? How do you start? You start it based on entropy. Okay? You try to pick that feature that is reducing the entropy to the maximum meaning is reducing the entropy really good that is going to be the first 10:12:44 feature. So internally what happens is it is the the algorithm is going to look at feature one. Okay how much reduction in entropy do I get with feature one? How much reduction in entropy do I get with feature two with feature three? with feature four so that I'm able to you know divide my Y in a really homogeneous mixture. I'm able to reduce the randomness. I'm able to reduce the entropy. That is what we are trying to replicate here. We are trying to say if I choose feature four this is what I will get and if I choose feature one this is what I will get. Right? So we are going to choose feature four. Absolutely. Right. So feature four will give us this. Feature four will give us this. This is what it is going to give us. So now if you have to calculate the entropy, if you have to calculate the entropy, we had initially 500 positive 10:13:47 and 500 negative. We we got in here. How did we decide feature one and feature four? How is the computer able to decide this? How is the algorithm able to decide this? Okay, since we have already looked at this is 50/50, right? This is 500 and 500. So, what is the entropy here? Isn't the entropy here equal to 1 maximum? Right? Isn't the entropy here maximum? H is equals to + 1 because we have 50/50 case similar to what we saw here. It was a 50/50 case, right? We looked at the graph also at 0.5 it is going to be one maximum. Okay? Here also at the starting the entropy is + one. At the starting you have this the entropy is one. But if you choose feature four. Okay. And if you do this how do you get get to here? 10:14:48 The entropy here is what? The entropy here is zero. Entropy at this node is zero, right? And entropy at this node would be 3x 5 into log of 2 into 3x 5 plus 3x 8 sorry because we have total 800 points here, right? Plus it is going to be what? 5x 8 into log of 2 log base 2 5 by 8 some value it will be right. So that's how you choose your root node. How do you choose your root node? You choose your root node based on reduction in entropy. Reduction in entropy. And this is nothing but this is called information gain. This is called information gain. Okay. How much of gain in the information are you getting here? You are able to get more gain in the 10:15:48 information. You're reducing the entropy more here. Right? Here there is still some value here. There is still some value. What's the value of this? Homogeneous. Yes. So, okay. Let's do one thing. Let's do one thing. Can I create a new one from here? Let's try to create one here. New notebook. I need a new notebook. notebook. So we have a new notebook. I'll tell you why I'm doing this. Okay, numpy as np. Let's run this. And then what was our this thing? It was 3 by 5 multiply by NP dot log log of 2 3x 5. Okay. And what do we have? So this is how even you can you know calculate I don't want to calculate it using the calculator you already have 10:16:52 numpy so here I have 3x 8 right I have 3x 8 I have 3x 8 here as well then the other point was what 5x 8 and this was also 5x 8 let's calculate what is this I think there's something wrong. Shouldn't be negative. Shouldn't be negative. Numpai dot log 2. That's what we using, right? Ah, okay. I get it. So the entropy is minus, right? So we have minus in the front. I get it. So in our formula, we already have a minus in the front, right? So that's why you're getting this negative. Let me fix all the comma brackets so that each of them is considered separately and the board mass is not applied. 10:17:52 Okay, this has to be in a separate outside it has to be this. Yeah. So it is 0.5. Let let us go back here. So this value is nothing but this value is 0.95. Okay. So we had a negative in in our entropy right this negative. So minus.95 into -1 this one into minus1 the outside minus - becomes positive right. So that's how we are getting 95 here. Okay. Now if you do the same thing here and the same thing here. Let's do that. Let's do that. There's an important concept that I want to teach out of this. So, let's copy this. Okay. Let's put it here. And what is going to be here? It is total is six. So, it is going to be 4x 6 and 2x 6. Right. Right. 10:18:54 So we are going to do we are going to put that 2x 6 2x 6 and here it's also going to be 2x 6 here 4x 6 400 and 200 was the split right so for that node it's going to be 91 okay so it is going to be 91 what about this total is 400 right is 300 out of 400. So 3x4 of a probability of red is 3x4 and probability of yellow is 1x4. So let's do that and let's do this again. Let's put in here and this is going to be 3x 4 * log base 2 3x 4. Then this is going to be 1x 4 * log to the base 2 1x 4.81. Okay. So this is giving me 81. Right. So 10:19:57 now the thing is how do you decide that the entropy has reduced? That the entropy has reduced. What was the this is called parent. This is parent right and this is child. This is child. Both of these are child. Do we understand this? Right? Do we understand this? Right? We can say parent and the child. So how how do we calculate? How do we calculate? How do I calculate? Now I know the entropy here is 1 + one at my root node. Here the entropy is 95. Here the entropy is zero. But at this level, at this level it is one. But at this level how do I say? I have two children. Now what do I do? I need to get the total entropy. What do I do here? What is the reduction? The reduction comes based on you need to consider both the child. You can take an average, right? You can take an average sum of the child entropies. Yes, there 10:20:59 that is one example. That is one way. But how do we do this is we take a weighted average. Let me keep it here. Let me copy it. Let me put another one here. We take a weighted average. We take a weighted average. So rather than just summing them we take a weighted average. So the total entropy at this level would be or the reduction would be nothing but reduction in entropy is entropy of the parent this one minus entropy of the child. Right? And for the child since there are two child here we'll have to take a weighted average. How is weighted average calculated? You know, here you have 800 points. Here you have only 200 points. So the weighted average entropy is multiplied by the number of observations that it is responsible for. So here it is going to be one since we have already calculated 10:22:01 minus I'll keep it in in the bracket. The first child is 95, right? Multiply by multiply by how how many how many is it responsible for? How much is it responsible for? It's responsible for 800 over total 1,000 plus 0 into this one is responsible for only 200 out of total 1,000 that we saw in the parent. Right? So this is going to be zero and this becomes 1 minus what what what does this become guys? This is going to be 4x 5. Let's do this again to the numpy. This is 0.95 multiplied by 4x 5. What does it give us? 76.76. So the entropy is this. So the entropy is what? The updated entropy is 1 minus this is what 0.24. 10:23:03 Now after dividing this you are left with only so such a low entropy. This is called your information gain. Okay. Whatever that you are getting by using. So what it does is internally the model does the same thing what we did for only one feature. We did only for feature four. It does the same thing for feature one, feature two, feature three, feature four. So many features that you give. It will do it for that and it will keep on keep on keep on keep on try to you know find which one should come here. Then after going here okay then after coming here it will try to look at again feature one feature two feature three we'll try to say okay which one is giving me the reduction in entropy. Put that put that rule here. Then it will again expand it further. Then if we have a pure node, it will leave it. If we don't have a pure node, it will try to divide it again based on features. Understood? How the features are being selected? 10:24:04 Which one to pick first? Which one to put in the first here, then here, then here? Is that clear? It is based on reduction in entropy. And reduction in entropy is nothing but information gain. Okay, I know it is mathematical but understanding this logically logically we understood that you know how it works. It works on rules. It works on rules here. Right. Go back. Yeah, it works on rules. It is nothing but if else and it is access parallel. But then how do we decide which feature to keep? Feature four or feature three. The one that gives us the pure node or is closer to the pure node is the one that gives us better and is going to be the one that we are going to start with. Okay, this is what we use. We use entropy. This is the formula of entropy. Then we looked at how does a decision tree make itself. 10:25:06 First it will select on feature four because it gives me the best entropy or best reduction. Then it will go to some other feature. then the third, then the fourth until it gets everything sorted out. It will keep on creating this. It will keep on creating this. It will keep on creating this until it reaches a leaf node. Where it reaches, it will not do anything. But if it doesn't reach, it will keep on growing there. It will keep on growing there like this. And this is also called your depth. So for trees we say max you know we say height but here since our tree our decision tree is ulta is inverse so we called it as depth. Can we use this entropy concept to get which feature is useful? Not instead of adjusted R2. Adjusted R2. This concept, this entropy 10:26:08 I will when we go into the hands-on, you don't have to calculate it. It is already there in the algorithm. It tells you which feature is more important. It tells you which feature is more important. The one which is most important will be selected first then second then third then fourth. So you can use this for finding the features that are important for any problem. Okay you can get the importance of the feature but not instead of adjusted R2. adjusted R2. What it does is it tells that between this model and this model which is a better model. It has that value, right? The one which is having a better adjusted R2 is a better model. But to pass the features inside the model, you can of course use this concept and entropy is what is happening at a very minute level at a algorithmic level. it is calculating those you know information gains but when you're using 10:27:09 this algorithm in skarn in scikitle learn it will automatically give you the feature importance we'll look at it when we go into the hands-on if this is clear I promise to you guys if this is clear I promise to you guys decision trees and random forests are going to be super easy for you are going to be super duper easy for you okay cool we looked that reduction in entropy nothing but information gain. Info gain is nothing but h parent minus h child. Okay. But since you have multiple child, you'll be looking at weighted average. Then all this this comes down to information gain of which feature? Information gain of feature four which is 0.25. Okay. This is the entropy at this level total. This is the entropy at this level which is 1. So parent minus child is 76. 10:28:10 Now again it will try to reduce from 0.95 to other. Okay. So basically what do we get from this is that the feature this is an important thing the feature which gives the maximum which gives the maximum reduction in entropy right the one which reduces your entropy maximum is called your first it will be chosen for root node. Then doing all the other things also like if it is in one internal node from there you need to break in again it will look at all the information gains and the one that is giving you the maximum information gain it will put that okay and that's how you'll be creating your tree going forward and forward. Okay information gain is nothing but reduction in entropy. Okay this is 10:29:10 really important. Okay, this is really important. Okay, so what it will do? It will try to look at IG of where is our data set to solidify all the things. I am putting this once again. Okay, let's copy this to solidify your understanding. This is what we are going to do. We are going to it is going to look at information gain of feature one. Information gain of feature two, information gain of feature three, information gain of feature four. Then in our example, why was feature four chosen? Why feature four chosen? Because it gave the maximum information g. Okay, that's how things work. Now you know that entropy is used and what 10:30:11 information gain is and you know that based on the information gain you choose the feature that you want to split. Okay, splitting is nothing but this this question that if else thing you know you're here and then you're dividing it here. This is nothing but split. Okay, you're splitting the tree into two. Now what happens is we also looked at that this is your depth. Now what happens sometimes sometimes you have let me try to find an example if I can show that with an example. There was a beautiful example I looked at earlier. H there is it. So just look at it. Okay it tells us what is the root node? The topmost root. Okay, this is your decision node. You're trying to decide, you know, any if or else. And this is your leaf node. Okay, to indicate the final one. Then we've already read this. 10:31:12 You can, you know, we've already covered this. Now it says that probably this is your data set. It has oak fruit here, which is in orange. It has cherry here and it has apple here. Okay, everyone who is able to see the screen, everyone who understands uh what we are trying to solve this this dot the size is nothing but you know the diameter that it is trying to show. Everyone who can see it type a Y type a Y who can see this and understands the data set what we are trying to do. Okay. Okay. Cool. So now what do we do? We try to start splitting and we say that if the diameter in the first we had diameter less than 0.45 we say if the diameter is greater than equal to 045 we are able to identify all the oaks here all the fruits that are oak are in this orange region. Okay. 10:32:12 Then we will do some splitting. Additional splitting. Here we will do what? We will say our diameter is less than.32. Height is first we will do height is less than equal to 4.88. Here is 4.88. If it is less then it is cherry. Right? This this purple color is cherry. Okay. And this is still some space where we have some you know things which are between purple and green cherry cherry and apple right so here we are trying to do some more decisions we are trying to put some more criteria to split we'll do some more splitting how does it do it is saying if now the diameter this one again it is trying to put if the diameter is greater than is less than.32 2 less than equal to.32 then we have another decision to make 10:33:13 this is the region okay but if it is greater than.32 this is what we get we get apple then again it will try to break this height less than 7.714 this is 7.714 7.14 anything below that is cherry anything above that is apple right so you see How many things it is trying to do, right? You see how many how many partitions it has done. Outliers. Yes. Yes, there can be outliers. This one, this one you're not able to identify. But it doesn't matter. It will still try to do some splitting based on it. Right? So what does it say? The tree that we are seeing, this is the depth, right? 1 2 3 4 5 Five. This five. We don't want to go this deep. We don't want to go this deep. Why? Do you see the problem with this? Do you see the problem that we 10:34:14 will face if we go this deep? Depth is increasing. Yes, depth is increasing. But don't you feel that it is trying to create so many rules to find a pure node, right? To find a pure node, it is trying to do so many calculations, right? It is trying to do so many calculations to find the pure node. That sometimes leads to overfitting. That sometimes leads to overfitting. How does it leads to overfitting? Let's go and check this out. Okay, let's go and check this out. Let me go back and open this. So here, let's see what they have done. They have they have taken this entire thing right? Can I copy it? So basically what I would have done I would have not gone to so much of depth right I would have not gone to so much of depth. What happens is there is a concept of underfit and overfitit. Okay. If you 10:35:18 keep on increasing your tree right if you keep on increasing your tree because ultimately it is an algorithm. It is a mathematics right? It is going to uh look at a few things and say that okay I'm not able to do this. Let me go much deeper. Let me go much deeper. Let me go even more deeper. Probably going this deep is not right. You need to control this depth somehow. You need to control this depth somehow. I would have cut the tree here. You see it is having 1 2 3 4 5 6 7. I would have cut the tree somewhere here. Okay. Why? Because if I keep on going till here, it will lead to overfitting. It will try to say that all my cherry is 10:36:18 in all my cherries are together. All my apples are together. All my what was the other fruit? Oak are together. It will keep on dividing. It will keep on dividing. Even if your this node for example, this node for example has nine cherries and one apple. It will still divide it further but you can still stop there right you don't want 100% what it will try what is overfitting overfitting is to exactly replicate the pattern of the data right what is overfitting overfitting can I say overfitting is something like this if I have a data set if I have a data set which is like this and the line that I'm trying to fit or 10:37:21 whatever nonlinear thing that I'm trying to fit if it goes and tries to capture all the points and tries to capture all the points isn't this overfitting this is overfitting right this is overfitting correct that's what it is trying to do here with all these trees. And what is underfitting? What is underfitting? Underfitting would be something like if I have my dots like this, I'm getting a line which is like this or probably this. Again, I'm talking about all the nonlinear combinations. As of now, this is underfitting. It is not able to find the patterns only. That is underfitting. Overfitting is it is memorizing the patterns. It is fitting to perfectly. 10:38:22 Okay. The idea the idea of machine learning is that we create a model that is you know can be generalized. If your points are like this. Okay. If your points is like this then we are trying to build a model which fits the data like this probably this is an outlier. Okay. This is something that we would prefer general model. We are trying to create a model that is not affected by outliers. Is trying to fit the model really well. This is what we prefer and that is why we will try to cut this in between so that it does not overfits and this process of cutting is called pruning. Is called pruning. Okay. Why do we do this? Everyone understands underfit and overfit. Let me give you another example. Okay. Let me give you another example. We have three cases. We have 10:39:25 three cases. Okay. We have already fit the model. We have already fit the model. And this is your model one. Okay. This is your model 2. And this is your model 3. Okay. Then the train train accuracy is 100%. Okay. And your test accuracy is 72%. Or let me create it more realistic. Your train is 92%. Okay. And here your train is 52%. And your test is 44%. And here your train is 81%. And your test accuracy is 78%. Which one is a better model? Which one is underfitting? Which one is overfitting? Out of these model one, model 2, model 10:40:25 3, what is model one? M3 is good. Yes. But what is M1? Is it an overfit? Is it an underfit? Is it a general good model? M3 is good. We understand. Everyone who understands M1 is overfit. Okay. Lakshmi says M1 is overfit. M1 is overfit. Ganesh also says M1 is overfit. Anyone else? That's right. That is right. You see what it is trying to do the data set on which we trained which was the train data set and we try to you know look at the accuracy on the train data set only using the predictions on the train it is getting 92%. But when we are trying to test our model on a data set that it has not seen in the test data set, it is performing so badly. This is an overfit condition that it has learned so well on the training data set but is not doing 10:41:26 so good on the test data set that is that is unseen to it. Here in M2, it is not doing good on the train also and it is not doing good on test also. This is under fit. This is under fit. This is under fit. And this is your general model. You see train is yes there is always going to be some difference between train and test because test is something that you have not seen at all. Right? So it is general fit or the right fit. That is what you're trying to do. That is what you're trying to do. Imagine. Imagine you have created this. Okay, you have created this. Okay, let us go to this exam. Let's go to this exam. Now imagine you have you have to do predictions now. Okay, you have to do predictions. You have already trained your data set. You have already trained your data set. This is your new data 10:42:27 set. This is your new observation. How will you do prediction for this? How will you do prediction for this? Nothing just takes just take it its age take its over time pass it through here ask the question that is it less than or greater than 35. It is greater than 35. So come here. Okay come here. Then you will ask whether the age is less than 29. Is it no? Then it will again come here. Then what does it say? It will say stay. So you know that for this point prediction is so easy once you have trained the model that's how you predict when a new unseen data comes in suppose this is in your test suppose this is in your test you have to do the prediction that's how you will train right if your model if your model trains everything that is in in your training data set like this one then 10:43:30 what will happen if if a new prediction model comes in here, what will it do? It will perform very badly, right? That's not what you want. That's not what you want. That's why we will try to do some pruning on this. We will do pruning on this. We will try to cut cut the tree. We will try to control the depth. That phenomena is called pruning. And pruning is done to what? Pruning is done to avoid overfitting. Okay. To avoid overfitting. Cool. That skarn uses. And I'll tell you why what the reason is also be behind that. Okay. Let me just figure this out. So let's go to SK learn and let's pick up decision tree classifier 10:44:30 and yes I think it's this one. Okay now you see this is your decision tree classifier. Okay it uses something called as criteria. Okay and we looked at entropy. We looked at entropy to create the decision tree. Right? There's something called as genie. There's something called as log loss. Okay. And the default is genie. What is this genie? Okay. Then we also looked at depth of the tree. You have this something called as max depth. By default it is none. But you can decide what depth you want to keep. What depth you want to keep for the decision tree. Minimum samples. What is this minimum samples? This minimum samples split meaning how many samples should be there in a leaf node or a node for it to split again. Even if it says two two uh you know your data points are there it will split again to one and one ultimately 10:45:32 what it will try to do is it will try to put one and one in the leaf nodes. All the leaf nodes will have one. Okay. So that's what you don't want to do. All these things you can All these things you can modify. These are the samples. But let me speak about this genie. Okay. Let me speak about this genie because this is most probably an interview question that a person will try to ask. Okay. Your interviewer will try to ask. Let me put down this as interview. Okay. There could be many questions based on whatever we have studied. There could be many but majority of them could be what is entropy? Okay, that's what they will try to ask. What is uh information gain? We know what is information gain. Okay, we have already looked at it information 10:46:33 gain. Then what is genie? It is generally called genie index or genie impurity and some people you know some people say it is as genie or gimme okay so I call it genie I don't know why but I call it that's how I I was taught or that's how I learned so it's all these things you know they can call it as genie index guinea index They can call it as genie impurity or genie impurity. Okay, all of them mean the same thing. Now what happened? There's really an interesting story behind this. Okay, let's look at genie index or impurity. Let me write it completely so that you guys are not confused that it is genie index and impurity are the same thing. It's genie index or genie impurity. Okay. Now we 10:47:36 have already studied entropy which finds the randomness correct and we looked at the example that you know the formula is I equ= to 1 to k and it is nothing but probability of y i multiplied by log to the base 2 probability of yi. What happened? Now all these algorithms this decision trees all of these they have been laid out way earlier in 60s50s7s 80s okay they are being relevant now why because then we did not have the computation power now we have clouds we have such fast GPUs CPUs earlier we never had those things so you know why people came up with genie is because they thought computing this was little expensive. 10:48:37 Computing a log of a probability took more time, was tedious, was you know uh more expensive. So what they said that let us define a another metric which is genie. Okay, let me put it as genie impurity and it is nothing but it is very similar to entropy. It is 1 minus summation of I = 1 to K. First part is same probability of P Y I to the square. Instead of calculating this log what people did they said we will calculate the probability twice now and multiply it. Do you see h how this is easier to calculate compared compared to log two? All of the concepts that we have studied for entropy are same for genie impurity. Nothing changes only apart from this 10:49:39 formula. Nothing changes. If you have understood entropy, same thing works with genie impurity. Nothing changes. We have to reduce the genie impurity that is called information gain. We calculated our entropy. We will also calculate our genie impurity for selecting the uh you know root node. The one feature which gives us the maximum information gain or reduces the genie impurity to the maximum. That is the feature that we're going to choose. Okay. To solidify this nothing changes. Nothing at all changes. It's just that the formula is different. It's just that the formula is different. To solidify our you know uh understanding let us take this example. Let us take this example and try to calculate the genie for this. Okay. We had these two same as that. Let me even 10:50:43 I feel you know calculating genie is easier because we don't have much we have to only calculate you know probability once and then you know just put it in place. So example for calculating genie impurity or genie index what was the issue here? So we have to calculate 1 minus I'm taking this inside it is 3x 6 probability squared + 3x 6² right what does it become 1 minus this will become this will become 1 right why because it is 1 minus probability of red square plus probability of yellow square means 1 - what does this become? This is going to be 1x 2² + 1x2 square which is 10:51:43 nothing but 1 - 0.25 and 0.25. This will come out to be.5 right and here what will this become? This is going to become let me write it properly here. So the genie impurity for S_sub_1 is 0.5. Okay. From here and genie impurity here will become what? For s_ub_2 1 minus what is this probability? What is the probability here? 6 on 6 square plus 6 on there's nothing here zero right? Probability of red yellow is 0. So this becomes 1 - 1 square nothing but 1 - 1. So this gene impur here is for s_2 equals to zero. Okay. So what did we notice? What did we notice? We noticed that entropy can take a range from 0 to 10:52:46 1 whereas genie impurity can take a value from 0 to 0.5. That's the only difference. Rest everything is same. Rest all the things are same. Let me try and see if we can get an example. It is entropy and genie graph. Why is my there you go. Okay. So this is your graph for both genie as well as probability. So what did we notice here that here the maximum value that it can take is 0.5 and what is the maximum value that it can take is one for what this is 0.5 okay and this is only probability and this is only impurity measure all the working is same is just that that your uh scikitlearn 10:53:49 uses g genie impurity as a default Whatever we studied for entropy, entropy is what was used earlier. And why did they come up with genie? Because it's easier to calculate, right? And it's also easier to remember. So probably this is something that you can expect in your interview. Can expect this thing in your interview. You can also expect some of the other things is that what did we look at? How splitting is done. How splitting is done. Okay. Then uh how root node is selected. All the terminologies. All the terminologies that we looked at meaning what is root node? What is leaf node? Okay. Then uh what is information gain? What is gene index that we already have put in there? Uh what is under fit? Okay. Then uh what else? What what else did we look at? All the things that we 10:54:51 have studied together here is everything that is important. Everything is important. Whatever we have run you know study together because trust me guys in the industry these decision trees and random forest are used heavily are used heavily and they are really good. Okay. So you need to be very clear with this concept. Okay. Thank you. >> Hi guys. Here we have just covered the theoretical part related to decision trees due to the time constraint. We have covered the decision tree implementation part in this separate video. Please check it out by clicking on the I button right now. Moving on, we'll take you through top 15 machine learning interview questions right away. >> So let's begin with our very first question which is what is machine learning and why is it important. Now as you can see in this diagram, machine learning is a part of artificial intelligence. Right? Under machine learning, we have concepts like neural 10:55:51 networks and deep learning. Right? Machine learning basically consists of dozens of different machine learning methods or algorithms which uh we will be covering a few in this particular video throughout our questions. So machine learning is basically teaching computers to learn on their own. All right. So what machine learning basically does is it will show the computer hundreds of pictures of say apples and oranges. Okay. So machine learning is basically going to pick these two fruits and show them to the computer. Now what machine learning is doing here is basically it's not telling that apples are going to be red and round. Similarly for oranges that they're going to be orange in color and they're also going to be round. Instead of giving certain rules, it is going to let the computer figure it out by itself. It is going to go through the hundreds of images and going to figure out a pattern by itself. So this way whenever a new 10:56:53 kind of apple or a new kind of orange is shown to the computer, it is going to be able to figure them out. So this is the basic idea behind machine learning, right? So it basically reduces the repetitive manual work like so say you want to filter out the spam emails, right? So you do not need to do that manually rather machine learning is going to do that for you based on certain keywords. So basically if an email contains words like free and offer so keywords like that. So based on that machine learning is going to help you identify that these emails could be spam. So machine learning reduces repetitive manual work. It helps businesses make datadriven decisions. It powers personalized recommendations like Netflix and Amazon. So you guys must have noticed uh every time you open Netflix you will be suggested some movie or the other or some series or the other based on your previous watches. So that is again machine learning behind it drives innovations like self-driving cars and medical diagnosis. So again it 10:57:54 is not just limited to the basic applications we see like Netflix. In fact it is seen everywhere like self-driving cars and medical diagnosis as well. So our second question is differentiate between supervised unsupervised and reinforcement learning. So first let's go to supervised learning. So as you can see over here we have some raw data and whatever algorithm we apply it is going to have label data. This is basically supervised learning over here. This is the major point that we're going to have labelled data and then we're going to have a predictions. So now let's look at that in detail. So supervised learning one of the three types of machine learning models is when the model is given label data that is your input and the correct output you have it labeled your algorithm knows that this is the input I'm getting and this is the output that is supposed to be uh predicted right it is going to learn the relationship between the input output to predict the future results right it is used for task like say classification and regression 10:58:55 classification is uh say your uh spam or not spam emails and your regression could be predicting your house prices. Right? So basically what supervised learning does is it learns from the examples and it predicts the answers for new unseen data. Right? Another example you can consider is uh we give labeled photos of cats and dogs. Right? We give photos and we label them as cats and dogs. Now it is going to learn to identify animals as either cat or dog in the future pictures. And since we have already told the algorithm when we were giving our data that this is a cat and this is a dog. This makes our data labeled. Right? So this is all about supervised learning. Then we have unsupervised learning. Now whenever we give a raw data it is going to be unlabeled. So this is the major difference between supervised and unsupervised learning. Let's look at it in detail. So basically unsupervised learning is uh if you were to consider a 10:59:57 real life analogy a student solving problems without a teacher. All right. So it is doing everything on its own. So no answers are provided. The student or you can say our algorithm has to find patterns or similarities on its own. So basically unsupervised learning is when the model is given data without labels and must find patterns or groupings on its own. Now in this case we just have given some random pictures of animals and now our algorithm that is our unsupervised learning method has to figure out on its own which one's a cat which one's a dog or maybe it is going to group similar pictures together and then we can identify a pattern between them. So it works like that. So the model basically doesn't know the answers beforehand. It tries to group similar data together like clustering customers by their buying habits. Right? So it is going to find a common pattern between uh the data and it is going to try and group them together. That is what unsupervised learning basically is. 11:00:57 Moving on to our third type of machine learning model which is reinforcement learning. Now reinforcement learning works like say you're teaching your uh pet dog some new tricks. So whenever the dog is going to do something right or it is going to uh get the trick right, it is going to get a treat and if it is not able to perform that particular trick, there will not be any uh treat or you can even uh introduce some sort of penalty. So that is the same way reinforcement learning works. So whatever agent you have, it is going to learn through interacting with the environment. Right? So if the agent gets the action right, there's going to be a reward or there could be a penalty for getting anything wrong. Right? So let's look at it in detail. So the model learns through trial and error train software to make certain desired actions. Right? So you can say like a robot who is left out in the open is going to interact with its environment and through trial and error it is going to learn what actions lead to rewards and what uh which ones lead to 11:01:59 penalties. Right? It is going to learn by interacting with the environment. It gets feedback in the form of rewards and penalties. So that is basically how reinforcement learning works. So we have a table over here differentiating the three based on data, goal and an example. So you can say supervised learning is labeled data. Unsupervised has unlabelled data. And reinforcement learning learns by interaction with the environment. Right? Then there's the goal. So supervised learning has the basic goal of learning the mapping between the inputs and the outputs. Unsupervised learning finds patterns and groups the similar data. Whereas reinforcement learning maximizes the rewards by trial and error. Then we have our example. Supervised learning could be classifying emails as spam or not spam. Unsupervised learning could be clustering the customers based on a particular habit or behavior. Then reinforcement learning would be training a robot to walk. Right? So moving on to our third question. What 11:03:00 is a model in machine learning and how is it used? So a machine learning model is like a recipe you can say or a set of rules that basically helps the computers to solve the problems. It's built by learning from data and figuring out the patterns. So basically you have your training data, you extract patterns from the data and then you use those patterns to predict the results. That is basically how you evaluate whatever you have trained your model on. Right? So a machine learning model is as we discussed a set of rules or instructions created by learning patterns from the data. It is used to train a data to make predictions or decisions. Right? So say for example you're training a student. So when you give that student a textbook, so the student is basically going to study from the textbook and answer the questions based on that. Correct? So that is similarly how machine learning models also work. They basically uh study from the training data that they receive and then they implement whatever they have learned 11:04:02 whatever patterns they have figured out from the training data set they implement it on their uh test data set or whatever new data they receive and then go ahead with the predictions. Right? It is used to understand relationships, classify things or predict outcomes. So basically there are two steps. First is training the model. The model learns from examples or the data to understand patterns. This is your training data. For example, feeding the model pictures of animals labeled as dog or cat. That is your training data, right? You're giving your algorithm, your model some sort of data and you're telling that this particular image is of a dog. This particular image is of a cat. Now, it is your duty to understand it to understand why this is a dog, why this is a cat. And then you test it or you make predictions, right? You are going to show a new picture that the model or your algorithm has not seen previously and then test it whether the model is able to predict the right answer or not 11:05:02 whether it's a cat or a dog. So that is basically how a model works. Right? Moving on to our fourth question. Explain overfitting and underfitting with examples. Now let's first look at these diagrams over here. So basically what these dots are are your data points, right? You can see all these dots. These are your data points. Now in your first graph you have a straight line going through the data points but not really covering or not really touching a lot of them. So this is what is called as underfitted or underfitting. Right? Then you have this image over here where your line is basically trying to touch every single data point that is there. This scenario is called overfitted. And then you have your middle image where your line is trying to uh find a middle path where it doesn't really touch every data point and it does not also exclude all the data points or is not too far away from all the data points. Right? So this is 11:06:04 called a good fit or a robust fit. This is what we ideally want. And these are the two conditions that you want to avoid. So now let's go ahead and look at them in detail. So overfitting basically happens when your model learns the training data too well. Now say for example you are preparing for an exam and whatever materials you have. So instead of understanding the concepts you try and buy heart every single example every single definition that is given to you in your material. Now if in the exam you get the exact same questions with the exact same numbers you are good to go right you are going to know every answer because you have by hearted you have memorized them. But in case you get some unseen questions that is the time you're going to suffer. So that is basically overfitting. What overfitting is doing here is basically it's trying to memorize each and every data point in your uh data set. Right? So in this case whenever a new data set is given to your model, it is not going to perform very well, right? It is going 11:07:04 to perform very poorly on unseen data. Why? Because it captures all the noise and the irrelevant details. So say for example you're training a model to classify dogs and cats. If it is overfitted, it is going to learn that cats only have a one specific background in the training images. It is going to capture that maybe all the cats are white or uh some unnecessary detail right and whenever you're given say suppose you give a picture of a white dog your model is going to classify it as a cat because it has learned or it has memorized that this particular image or this kind of image belongs to a cat. So that is what overfitting means. Then you have your underfitting. This occurs when your model is too simple. Like we saw in this diagram, when you have an underfitted model, your line is not really going to uh care about the data points as such. It is going to oversimplify the things and fail to capture the underlying patterns in the data. Right? As you can see in that diagram, the line is not 11:08:04 particularly capturing any sort of patterns. So this leads to again poor performance on both the training data and the new data. Now if we were to consider the example of a student again. So now in this case what the student is doing is he is basically skimming through the pages just reading the headlines and moving ahead thinking that he has learned everything about the topic. Right? So this is underfitting. Now a model which is meant to classify cats and dogs is going to be underfitted if it predicts every animal as a cat because it hasn't really learned the difference between features of dogs and cats. Right? So that is what underfitting means. So here we have a table differentiating overfitting and underfitting. So performance on the training data for overfitting is going to be excellent whereas for underfitting it is going to be poor. Performance on the testing data is going to be poor for the overfitting models and for the underfitting model it is also going to be poor. Now as you can see the underfitting models are going to perform poorly no matter whether it's a training data or a testing data. Why? Because 11:09:06 they do not really capture any sort of patterns right from the beginning. So even on the training data they're not going to perform well. Whereas overfitting is going to do exceptionally well on the training data but it is going to fail to perform well on the testing data or which whatever new data that is presented to the model. So then we have our model complexity. So overfitting is going to be too complex because as we discussed overfitted models literally try to learn every single detail of the model or of the data set. So it is going to be way too complex. Whereas underfitting is going to be way too simple because they barely capture any details or they barely try to understand any sort of pattern. So the solution is to simplify the model, use regularization or get more data. Whereas in underfitting models you can use a more complex model or tune the parameters. Right? So reading these points you must have got an idea that what is important is a balance between the two 11:10:06 to achieve a good or a robust model. Right? Your model shouldn't be overfitted or underfitted. That is it shouldn't be too complex or it shouldn't be too simple. It should be just midway between these two to achieve the best balance which is also known as the biased variance trade-off. Right? You can also mention this point because let me just write it down. So it is also called the bias variance tradeoff. Right? So moving on to our next question. What is a confusion matrix and how is it used in evaluating the models? Now a confusion matrix is basically a table that helps you to evaluate how well a classification model is performing. Right? So it compares the model's predictions with the actual outcomes. Basically whatever predictions your model is going to make you're going 11:11:08 to have some uh correct values, you're going to have some uh incorrect values or incorrect predictions. So all that is mapped in a very organized way and shown in a table which is called the confusion matrix. Right? So it is useful for understanding the types of mistakes your model is making. So this is what a confusion matrix looks like. You have your actual values that is say you have your training data and your testing data. Right? So for your training data you have your actual values that is say you have a column predicting yes and no right. So it basically goes like this yes no yes no whereas your testing data is going to predict something like yes yes no yes right. So as you can see over here this particular prediction is correct over 11:12:09 here whereas all these are wrong. Right? This was supposed to be no. This was supposed to be yes. This was supposed to be no. So all this can be mapped in a very organized way as you can see over here. So our actual values are here and our predicted values are here. So if our actual value was positive and our predicted value is also positive, we call it a true positive. So this is basically a correct value. But if our actual value was positive but our predicted value was negative, we call it a false negative. that means this value is incorrect. But if our actual value was negative and our predicted value was also negative, we call it a true negative. That means it is correct. But if our actual value was negative and our predicted value was positive, we're going to call it a false poser. That means it was incorrect. So this is basically how a confusion matrix maps out all your predicted and actual values to help you analyze how well your model 11:13:09 is performing and how much of your predictions are correct and where exactly your model is making mistakes. Right? So true positives uh we just discussed the model correctly predicts a positive outcome. Uh false positive is when the model incorrectly predicts something as positive. This is also known as a type one error. Then you have your true negative. The model correctly predicts a negative outcome. False positive is when the model misses a positive case and uh instead predicts it as a negative. So that is called a false negative. All right. So now that you have understood what is a confusion matrix, let us also see how it is used in evaluating the models. So for evaluating the models, we use terms like accuracy, precision, recall and the F1 score. Now accuracy basically means the overall correctness of your model. This is what accuracy is going to test. Precision focuses on how many of the predicted positives are actually uh positives. Recall focuses on 11:14:12 how many of the actual positives are correctly identified and F1 score is basically a balance between precision and recall. So you can give an overview of how uh these terms you can also go ahead and explain the formula for these particular metrics. Right? So that was all about the confusion matrix. Let's move on to our sixth question which is define cross validation. Now cross validation is nothing but a method that is used in machine learning to test how well a model will perform on unseen data. So how is it going to test? It is basically going to split the data into multiple parts. So say you have a data containing 50 rows. So what this cross validation is going to do is it is going to split the entire data set into say five rows each and it is going to create 10 parts of such five rows. Right? So it is going to train the model on some parts and test it on the remaining parts. So that is basically how cross validation works. Now this ensures model performs consistently. So it is going to 11:15:14 prevent overfitting. It is going to have a reliable testing and it is also going to maximize the data usage. Right? So as you can see uh cross validation has split this particular data into five parts. Right? So what cross validation is doing here is it's going to train the data on these four parts and then test it on this fifth part. And it is going to do that until and unless each and every part of it is tested excluding the remaining four. As you can see at the second time it is going to train the data on these four parts and then test it on the fourth part over here. Then it is going to test it on the third part then on the second part and then on the first part. Basically it is going to create a combination for uh all the number of parts that it has created for the particular data set. Now this is going to ensure that our testing is reliable. So this also prevents overfitting and it maximizes the data usage. Now consider this example where say you're preparing for a spelling B competition. Now you have a list of 100 11:16:14 words that you need to prepare for. Now instead of testing on the entire list of the these 100 words, what you are going to do is you're going to split these 100 words into 10 smaller groups. Right? So you're going to have uh 10 lists each having 10 words. Right? So now you're going to study these nine groups and then test yourself on the 10th group. Right? Now you're going to repeat this process until you have tested yourself on every group. The next time you're going to go ahead and uh practice on the first eight and the 10th group and then test yourself on the ninth group. So this is exactly what cross validation is doing in this case. Right? So make sure whenever you go in an interview, you give relevant examples and you are very clear with your explanation so that the interviewer understands that your concepts are clear. Right? So moving on to the seventh question. What is the role of precision and recall in evaluating a model? Now let's understand 11:17:15 this diagram first. So basically we have our spam emails over here in this section and we have our not spam emails in this section. Simple. Now what our algorithm is doing is it is predicting all these emails as spam. Now as you can see the green section is correct because it is correctly identifying these uh spam emails as spam. Now but the red section over here is that the not spam emails are also classified as spam. So basically what our recall and precision is doing is our recall is going to test all our true positive that means all the correctly predicted uh spam emails as spam based on our spam emails. So precision basically measures how accurate the model's positive predictions are. So in other words you can say that all the predictions the model made as positive how many were actually correct. Right? So basically it is going to compare these two sections. Now say for example you have uh 10 emails in total 11:18:17 and eight of them are actually spam. So in this case our precision is going to be 8 upon 10. All right. Now let's look at recall. So now recall basically measures how well the model identifies all the actual positives. So it is basically going to compare this entire section right as you can see in this image. So out of all the spam emails how many did the model catch? So say we have 20 spam emails. All right. And the model actually figured that eight of them were spam. This is our prediction. Right. So our recall is going to be 8 out of 20. This is what the recall is going to measure. So both these metrics work on true positives, false positives and false negatives. So again we will have to use 11:19:19 a confusion matrix in order to calculate these. Right? This is the formula for precision. True positives upon true positives plus false positives. uh basically trying to predict how many of the predicted positives are correct. Right? Then for recall we have true positives upon true positives plus false negatives. How many actual positives were identified correctly? Right? So a precision and recall basically give the insights beyond the overall accuracy. Now if your model has a very high precision your model is basically trying to avoid the false alarms. Basically that means very few false positives whereas a high recall is going to minimize missed positives that is very few false negatives. Now these criteria are very important. Let us quickly look why they are especially important when the cost of errors vary. Now consider a scenario of a medical diagnosis. Now missing a disease that is a low recall 11:20:21 is going to be worse than a false alarm. Right? uh in cases of medical diagnosis, you're going to prefer that if there is even a slight chance or a slight possibility that a person could have a particular disease, you would prefer a medical diagnosis in that case. Rather missing that particular disease and hence not going for the medical diagnosis is going to be worse in this particular scenario. Whereas another scenario where you are detecting spam emails. So in case you flag the genuine emails as spam. Now say for example you have a very important email and you flag that particular email as spam which is in the case of low precision. It can be annoying or it can even uh cost you say a missed opportunity. Right? So you need to consider these scenarios in mind and choose which metric uh it could be precision or recall which one you need to be higher in which scenario. Right? So moving on 11:21:22 to our eighth question. What is the F1 score and how is it calculated? So the F1 score is basically a metric that combines precision and recall into a single number. Now just as we discussed the two scenarios where a high precision or a high recall is going to be beneficial in one case whereas so the F1 score is basically a metric that combines precision and recall into a single number. So it it basically provides a balance between the two right precision and recall. Now just as we discussed in our previous slide the medical diagnosis and spam detection uh situation where keeping one variable low and the other high is going to be beneficial in one case whereas uh it could be harmful in the other. So in cases like this we need something that is known as the F1 score which does nothing but provide balance between these two metrics. So the F1 score basically ranges from 0 to 1 where one indicates perfect precision and recall. 11:22:24 So if you achieve your F1 equal to 1 that means you have a perfect balance between precision and recall. So this is the formula for your F1 score twice into precision into recall divided by precision plus recall. So F1 score is like we discussed very helpful when your precision and recall are in conflict. Right. Right. So that is all about F1 score. So moving on to our ninth question. What is the purpose of regularization in machine learning? So regularization is basically a technique that is used to prevent overfitting by adding a penalty to the model's complexity during training. Now it adds this penalty to the loss function during training. Right? This prevents overfitting. Now overfitting occurs when a model performs well on training data but poorly on unseen data as we already discussed previously. Right? So regularization is going to help the model generalize better on new data. So basically it's a technique that 11:23:26 is used to prevent overfitting. That is the main criteria. Now there are two types of regularizations that is L1 and L2. L1 is called the lasso technique and L2 is called the ridge. So as you can see without regularization we have an overfitting where our line is trying to touch each and every data point whereas after regularization we're going to have a better fit uh where our line is more generalized. It is near all the data points but it is not really memorizing any uh data points as such. So the L1 or the lasso regularization is basically when it shrinks some weights to zero. Basically it tries to remove the irrelevant features and it tries to make the model more simpler. Right? Whereas the rich uh regularization focuses on uh reducing the large weights. It encourages the model to balance the complexity without uh exactly 11:24:27 eliminating any features. Right? So basically uh regularization reduces overfitting that is uh it prevents the model from becoming too complex. Right? Or we can say it reduces the complexity. Reduces complexity. We've already seen how uh it simplifies the model. Uh it basically encourages more uh interpretable weights, right? It improves generalization that is it makes sure that the model performs well on unseen data as well, right? So the training as well as the testing data is focused over here through generalization. Then we have multi-olinearity that is the high correlation among input features. So say we have two features that is number of bedrooms and number of bathrooms. So if these two features have a very high correlation 11:25:28 there's going to be a problem in uh simple models like linear regression. Right? So this multi-olinearity problem needs to be handled and it can be done through regularization. So these are the benefits of regularization and I hope you guys understood it and we'll be able to explain it in an interview as well. So moving on to our 10th question. Explain the bias variance trade-off. So the bias variance trade-off is basically the balance between two sources of error in machine learning. Bias and variance. So we have already discussed uh underfitting and overfitting. Underfitting and let me write overfitting over here. Right? So underfitting basically occurs when we have a very high bias whereas overfitting occurs when we have a very high variance. Right? So now in order to find a balance 11:26:28 between underfitting and overfitting we have something known as the bias variance trade-off. Right? So as bias decreases variance increases and vice versa. So the this is how these two variables are correlated. All right, let me just clear my screen. Okay, so now the goal is to find a balance where both bias and variance are minimalized for best generalization. Now, we do not want any of these terms to have a greater effect on our model. So, we're going to have to minimalize both of these terms to achieve a better generalization. Right? So as I already told you uh if we have a very high bias our model is going to be too simple that it is going to be underfitted right and if we have a high variance that is our model is going to be too complex we're going to have a good accuracy on our training data but a very poor performance on our testing data so that is what high variance does or overfitting does right so consider an 11:27:30 example where you're hitting a target now if you have a very high bias is your arrows are going to land very far away from the target but in a very consistent or you can say systematic error right either they're all going to go say towards the right direction but in case of a high variance your arrows are going to scatter unpredictably right one arrow could go here one could go here one could land over here it is going to be very unpredictable some could even hit the target and some could miss it right But a well- balanced model is going to be a case where all your arrows cluster close to the target. Right? So consider a very wellbalanced model. And in that particular model, all your arrows are going to land somewhere over here, over here, over here. Right? All near the target. So this is when both your bias and your variance has been minimized. So that is all about bias variance trade-off. Moving on to our 11th 11:28:31 question. What is feature engineering and why is it crucial in machine learning? So you can see in this diagram we have raw data and then we have something known as feature engineering and then we come to our ML model. So now what feature engineering is it's basically uh involving refining or creating input variables to make them more useful for a machine learning model. Right? In order to implement a model on your raw data, you need to have a well- refined or you need to have your input variables that are very useful. So that is done through feature engineering. Right? So feature engineering basically involves steps like feature selection, feature transformation, feature creation to improve a machine learning model's performance. So feature selection is where you pick the most important features for your model. So say in a house price prediction model the area of your house is going to be more relevant than the color of the house. Right? Then 11:29:33 comes feature transformation. So this involves modifying features to make them more suitable for analysis such as you can say normalizing your numerical data or encoding your categories. Right? So basically you transform your data into a more useful format. Then you have creating input variables or features. So this is basically called feature creation. This is where you uh create new features from the existing ones. So say for example you combine the height and weight into a BMI feature or a body mass index. Right? So that is called creating your input variables. So the benefits of feature engineering are uh model accuracy improvement. So by providing the model with the right information, you improve the accuracy of the model. Right? Then comes uh reducing the overfitting or preventing overfitting. This is done by focusing on relevant features. Right? So once you reduce the noise or the irrelevant 11:30:35 information like color of the house, automatically your overfitting is reduced. Then comes simplifying models. Now when you eliminate the unnecessary complexity, your model automatically becomes simplified. Right? Then your final advantage is highlights the domain knowledge. Now when you perform feature engineering, it incorporates domain expertise. Right? If you have a good domain knowledge, you can make your model much smarter and you can tailor it to the problem. Right? So again feature engineering has a advantage of highlighting your domain knowledge. Moving on to our 12th question. Example of when a false positive is more crucial than a false negative and vice versa. So this scenario we have already discussed but let's look at it in detail. So we know a false positive is when a model predicts something as positive but it is actually negative. Example a fire alarm going off when there is no fire. Right? 11:31:36 Then we have a false negative when a model predicts something as ne negative but it is actually positive. So this is missing an actual fire when it occurs, right? So there is a fire but you missed it. You predicted it as negative like there's no fire but in actual there was a fire. Basically that's what it means. So now that we have covered what is false positive and what is false negative let's look at an example or very scenario where one is more important than the other. So let's look at the scenario when a false positive is more crucial. So in case of a spam email detection, what a false positive is going to mean is basically you classify a legitimate email that is an important email as spam. And now what the consequence is going to be is it is going to get missed by the user. It could be a job offer or a client communication letter. It could be any sort of email. And a false positive is going to mean is that that particular 11:32:36 that important email is going to get classified as a spam. Right? So this is where a false positive is very crucial. You cannot just uh blindly mark any email as a spam. Then you have your false negative. So in case of a cancer detection or any uh say medical diagnosis, right? So a false negative is going to delay the diagnosis of the treatment which could be life-threatening. We've already discussed. So the model is basically going to fail to detect cancer in an affected person. So this is where a false negative is crucial. Moving on to a 13th question. What does nivity stand for in knives bias? Right. So knive bias is again a machine learning algorithm based on the bias theorem. Now this is used for the classification task. Now what the knife uh stands for in knife bias? Basically 11:33:37 it refers to the assumption that all the input features in your data set are independent of each other which is again not true in the real world. Right? Now say for a house price prediction model you cannot have uh number of bedrooms and the number of bathrooms or number of bedrooms and the price feature as independent variables. They cannot be independent of each other because obviously each of these factors are interrelated and are going to affect each other. Right? If you have lesser number of bedrooms, your price is definitely going to be affected. Right? So it is not true in the real world but it is an assumption made uh in machine learning whenever we use knives bias that all the assumptions that all the input features are independent of each other. So basically this is how the bias algorithm works. It basically classifies all or basically groups the different items into different groups. And this is the formula for the bias theorem. 11:34:37 So we already saw NY bias assumes that changing one feature is not going to affect the others. This is the independence assumption. We have already discussed this. Now this works well with limited data mainly for one reason because it does not really need the complex relationships between features. Right? So it is again fast and efficient that is it makes the computation really efficient. Then again it is surprisingly accurate. Even though in the real world uh the assumption that the knife bias algorithm considers is not true, it is surprisingly accurate in machine learning. So basically even in the example we discussed. So even when the independence assumption is violated, it will still provide accurate predictions for many tasks. Right? So that is one advantage or one very important characteristics of uh knife bias. Moving on to our 14th question. Give an example where the median is better measured than a mean. So now we know 11:35:37 mean is basically when you add up all the numbers and divide by the total count. Right? So say for example you want to find the average of 2 4 and six. You're going to add 2 4 and six and since the total count of these numbers is three. We will divide by three and our answer is going to be four. Whereas median is basically finding a middle number when all the numbers are arranged in a particular order. Right? They need to be in a descending or in ascending order. And whatever middle number you find is your median. So the median of 2, 4 and six is going to be four. This is the middle value. All right. So now when does median outperform mean? Now when do we prefer using a median rather than a mean? So consider a scenario where you have five people and you're given their incomes. First is 30k. Uh then you have 35k, 40k, 45k and then you have 1 million. Now you can notice that these four numbers come in a very close range 11:36:38 whereas 1 million lies in a very far range. That is this particular number is an outlier. Right? So if you were to plot these numbers in a graph 1 million would stand out compared to the rest of the numbers. Right? So that is basically what outlier means. So if we were to find the mean of these particular values, we would have a number something like this. This this is not accurate because if we were to find the mean for these particular values or these particular incomes, we wouldn't have such a huge mean, right? Our mean has been uh changed or it has been influenced because of this particular outlier. So in cases like this where we have a heavily skewed data, this is when median is more efficient. Now if we were to find the median for these particular values it would be 40k and this would represent the income of these five people more accurately right so basically median is not affected by the extreme values of the outliers as we 11:37:39 just saw in this particular example right so it gives a more realistic picture whenever data is skewed like in income or in property prices right so whenever you have a high range of your data you're going to prefer median over Moving on to our 15th question. What are the key differences between bagging and boosting in ensemble methods? So let's understand what is bagging first. Right? So bagging basically stands for bootstrap aggregating. Now this creates multiple models by training them on different random subsets of the data. Now each model makes a prediction and their results are averaged for regression or quoted on for classification. Now example of bagging is random forest right now in case you're not familiar with ensembler methods let us quickly discuss ensembler methods. So ensembler methods are basically like building a team right multiple models or players they all come together to make better predictions. So 11:38:41 all these factors coming together basically reduces the errors and increases the stability of the model. So bagging or bootstrap aggregating. Let's consider an example to understand it better. So imagine you're dividing your classroom into groups and giving each group a problem. Now whatever answer each group comes up with, you average their answers. Now this method is going to reduce the variance and it is also going to make the final model more stable. Now random forest is a classic example of bagging. Then we have boosting. Now boosting takes a little bit of different approach. Now models are trained one after the other with each new model focusing on the errors made by the previous one. So instead of combining the models simultaneously, boosting is going to combine them sequentially. Here each model is going to correct the errors of the previous one. So the boosting algorithm basically uh corrects the errors or improves the mistakes as 11:39:44 the algorithm moves forward. Right? The examples of boosting algorithms are the gradient boosting or the add a boost algorithm. So let's understand boosting by an analogy of real world example where a teacher is helping the students who struggled the most in the last test. So now boosting is going to be a very great way of correcting the bias and weak models. Right? So whichever students uh struggled in the last test are going to be trained separately and help improve their performance in the next exam. So this is exactly how boosting also works. Let's quickly look at the difference between bagging and boosting uh methods. So bagging basically works in a parallel method where models are trained independently. Right? All of the models are being trained at the same time independently. That is they do not have any sort of uh connection between each other. Whereas boosting works sequentially. models depend on each other where one model corrects the mistakes of the previous 11:40:44 model. Bagging focuses on reducing the variance whereas boosting focuses on reducing the bias. Bagging treats all the data points equally whereas boosting focuses on correcting the errors only. Bagging is less complex whereas boosting is more complex due to the dependency. Bagging is good for stable models with high variance whereas boosting is good for weak models with high bias. Just a quick info guys, Intellipad offers an executive post-graduate certification in AI and machine learning in collaboration with iHub, IIT, Rurki and Microsoft. This comprehensive course is designed to make you an expert in AI and ML in just 11 months by working on over 50 realtime projects and industry case studies. With this course, we have already helped thousands of professional and successful career transition. You can check out their testimonials on her achievers channel whose link is given in the description below. Without a doubt, this course can set your careers to new height. So visit the course page link given below in the description and take a first step towards career growth in 11:41:44 the field of AI and ML. 11:41:54 [Music]
