0:02 [cheering] 0:09 We must stand together. Order. Order. This project will be done by [music] 0:27 This project will be done by statistical machine learning. [applause and music] You don't use a sword to cut an apple. You use the kitchen knife. [music] Case closed. Classes begin. Machine learning also known as statistical machine learning is still relevant as of 2026. I can say this with confidence because in our AI services company Atlake Technologies where we build projects for our clients based in US and UAE, we are seeing many parts of projects require statistical machine learning. And this is because machine learning models are lightweight, they are easy to train and cheaper. It is like you're driving in a crazy Bangalore traffic. Will you go on a big car or 1:29 will you go on a bike? When you go on a bike, it is lightweight, it costs you less and you can reach to your destination faster. Nowadays, people who are living in this big cities, they keep both the car and the bike and based on a given situation, they will either use a car as a vehicle or bike as a vehicle. Similarly in AI projects you can use generative AI for some parts of projects and for some other parts you can use statistical machine learning. Machine learning gets used whenever you have a label data set especially the numeric data and you want a very fast cost effective method for doing classification regression etc. In this crash course, we are going to teach you statistical machine learning from scratch. As a prerequisite, you need basic Python knowledge. For Python, we have another YouTube video. So once you 2:29 know basic Python, you should be able to follow along. We are going to cover all the concepts, all the theory, okay? Regression, classification, supervised learning, unsupervised learning, all of that along with code and exercises. So this is going to be very hands-on tutorial and at the end we will have a capstone project where we will do product return prediction based on certain features. So we'll take a use case of this e-commerce company where you want to uh help that company predict if the item will be returned by customer or not. In terms of technical stack, we are you going to use scikit learn along with logistic regression, xg boost, all of those models to train the model and then we'll wrap it up into fast API back end. We will use NodeJS as a front end. Of course, we'll be doing some vibe coding as well and it will be an end to end project. It will have an impressive 3:29 UI where you can enter all the product features and it will [music] tell you if the item will be likely returned by a customer or not. All right, it is going to be a full fun engaging crash course. So please get ready and let's get started. Let's understand what is machine learning and we'll understand the core principle behind it. We already use machine learning in our day-to-day life and that is your spam detection. If you're using Gmail or any other email, you will have this feature of spam detection. And companies like Google uses machine learning to build this spam detection mechanism. Now when you look at any email which is spam, you will notice some patterns. It will have suspicious email id. It will have some offers which are too good to be 4:30 true. It will have some kind of urgency. That offer is valid only for 2 hours. Hurry up, click on it. So these are all the different patterns that we see in the spam emails. Now let's imagine Google is a new company. They do not have engineers who can write ML program for that. Okay. So let's say Larry Pageige and Sergey started Google in a very early days and they want to build this spam detection feature. What they can do is they can hire a human. Let's say this particular person. Okay, looks like a familiar face. This person is hired at Google as a spam detector. So Google doesn't have too many users. So they get few emails every day. This person will manually examine those emails and will classify as a spam or not spam. Now the person has to go through the training first right because 5:30 the person doesn't know how to detect spam and non-spam. So Google provided him a data set of 10,000 emails which were spam, 10,000 emails which were not a spam and this person started training basically he started training his own brain. Okay. So he started looking at the patterns and he then derived all these patterns that spam emails have urgency, lucrative offers, suspicious email ID and so on. So while going through all the emails, this person extracts some patterns that whenever the email is spam, they will have urgency, lucrative offer, all these patterns. So the person trained himself on spam and non-spam. Just like how you can train the person to detect spam, you can also train a program, a computer program for uh detecting these patterns. And there are algorithms like linear regression, 6:30 logistic regression, SVM, random forest. We're going to cover all of that in this particular crash course. Using these techniques, you know, these are statistical mathematical techniques. The core algorithm is called gradient descent. These are all mathematical techniques using which you can train a program so that it can also capture all these patterns. Now, when you're training this computer program, what you have is a label data set. Label data set means those 10,000 emails which are marked as spam or non-spam. So if you look at the data set, see this is the first email which is marked as spam. This is the second email which is marked as not a spam. Usually you'll have many emails, let's say 100,000, 200,000 emails. Okay, machine learning requires huge volume of data for the training. So this data set is called ground truth or labeled data set. So you have a data set 7:31 which is labeled spam, not a spam. These are the labels that you have attached to each record. So you have all these emails then [snorts] the the label spam and not a spam and combinely input and output both is called a labeled data set. Once you train a model on this when you give a new email to it the train program you know the train ML program will say whether is spam or not spam. Okay. So essentially you are giving all this data this entire data set to ML model. Okay. And you run the training. And once the ML model is trained, this is the ML model. Now you can give a new email and it will say spam or not spam. So there are two phases. The previous phase where you were training the model is called training. This particular phase when you are actually making the prediction, it is 8:34 called inference. Okay. So training and inference. Now how does it do it? So so let's do this thought experiment. I am showing you this input output table and then I'm giving you input number as four. Tell me what will be the output. Literally pause this video. Observe this table and then tell me if the input is four, what will be the output? Okay, I think most of you got it right, the output will be 16. But how did you do it? As a human, tell me, pause this video, how did you do it? Well, the answer is you looked at the data set, you captured the patterns, you said okay 2 4 3 9. Oh, this is square. Okay, so this is y = x² you derived that pattern in your brain. Okay. And then when the new number comes in since you know the equation now you'll say 4² is 16. So y is 16. This is how you as a human uh 9:38 makes a prediction. So computer does the same thing. You are given the data set which is input and output. You give it to ML program. You use any of these techniques. Support vector machine, linear regression, SVM, KN&N. These are all different techniques. We are going to cover all of that and you train this program and then when [clears throat] you train it the output is the model basically. So the model has nothing but that equation y= x² and then when you give it a new let's say input it will tell you the output. What happens in traditional program is traditional program is given the input and equation. So you write code right you you you write code to square the number and then that is a traditional software program and it will produce the output. Okay. So the core difference here is in case of ML you give input and output and it will derive 10:39 the equation. Here you give input and equation and it will derive the output. Okay. So that's the core difference. ML program is given input and output and during the ML training it will come up with the equation or a logic. So this is training phase and in the inference phase you just give it an input and then using this logic that it has built it will produce the output. In case of traditional software you have input logic is whatever is your code and then it will produce the output. Folks remember this diagram super important. In traditional software you give logic and input you get output. In machine learning, you give input output, you train the model and the output, the output of that training is the logic. So just to define machine learning, it's a discipline in computer 11:39 science where we train machines on data so that they can make predictions without explicit programming. Folks, without explicit programming, this is super important. See here when I showed you these patterns and when I said four, how did you get 16? Well, without any explicit programming, you looked at input output. You just capture the patterns. You derived the equation in your brain and then you use that to make the prediction for the new uh input. Let's go over the entire AI family tree so that you understand where machine learning stands in this entire AI landscape. Artificial intelligence is this overarching multidisciplinary field where you train computers to do tasks which humans are generally good at. For example, driving the car, writing poem, you know, recognizing text, working with text, uh vision, computer vision. So AI is that bigger 12:40 field. Machine learning is a domain within it or it's a one type of technique that helps you do AI. In machine learning you have statistical machine learning. So the agenda for today's course is this. There is deep learning also. Okay. So in statistical machine learning you will have all these algorithms linear regression, decision tree, k means all of that. We are going to learn all of that today. In deep learning you have neural networks. Okay. So neural network is the fundamental or the basic building block of deep learning. And then outside machine learning uh you have regular expressions, robotics, rulebased systems. You can use all of that to make a system as good as humans. Okay. So this is the overall picture. Our objective today is going to be this component. Before we begin our coding, of course, you need to install certain things. The first one is very obvious. Python. Click on download Python and 13:42 install uh Python latest version. You can go for 3.14 or 15. Okay. Uh install a stable version. How do you do that? Well, I don't want to spoon fade. It should be pretty straightforward. Okay. Just just click on it. Install common sense process. Another thing you need to install is something called Jupiter Notebook. So Jupyter notebook is interactive environment where you can run the code. Data scientists use this to train their machine learning models. Okay. Uh to write their code. So you can just say pip install notebook. Once you have installed Python, go to command prompt, say pip install notebook. And that's it. Throughout the code, we will be using different Python libraries. Let's say you're using pandas. The way you install pandas is just by calling pip install pandas. pip install pandas. Let's say you need numpy. pip install 14:42 numpy. I'm not going to tell you uh all these commands as we go on. You have to use your common sense and install necessary things. Whenever you're facing any errors, take help of chatg etc. uh to resolve that error. All right. So I'm assuming you are done with installation and let's get started with our first algorithm which is linear regression. Before we go further, I would like to thank today's sponsor Dcope. Everyone is racing to ship AI agents right now and O is usually the thing that gets bolted on last if at all. That is a problem because agents that can call tools and talk to other services are basically new identities in your system [music] and most teams don't have a good way to authenticate them. Scope what they can access or manage their credentials over time. Descope build the agentic [music] identity hub to fix that. It is a dedicated identity provider for AI agents and MCP servers. On the developer 15:43 side that means drop in support for OOTH 2.1 PKCE and both major client registration flows DCR and CIMD. So you are not implementing MCP O from scratch. On the security side it means scope based access control at the tool level short-lived credentials with full life cycle management and policy controls that let you go on access by user tenant agent or server. Basically developers move fast security teams sleep at night. Link is in the description if you want to try it on your own MCP server or agent. All right, let's go back to our video. Now the very first topic we are going to talk about is linear regression. Let's assume you're working as an AI engineer in Swiggy. Swiggy is a food delivery company. And one problem statement that they have is based on the distance they want to estimate the delivery time. So when I order paneetikka masala or let's say samosa from my favorite restaurant I want to 16:43 know at what time it will be delivered and in Swiggy application the AI engineers who are working at Swiggy would have built a feature which will tell me the ETA estimated time of arrival. For this they will use machine learning. So let's see how they use linear regression as a machine learning technique. they would have some past data. Okay. So let's say for in the past there were few deliveries and these are the timings noted for those deliveries based on the distance. So in the past let's say somebody order something and it was 1 kilometer away and it took 16 minutes 2 km 13 minutes and so on. I know that actual time depends not just on distance but on other factors such as traffic, rain, um weekday, weekends etc. But we're keeping things simple for now. Okay. So let's say uh the time it takes 17:46 to deliver a food depends only on one factor which is distance. Okay. Now you want to know for a new order let's say which is 7 kilometer away. How long will this delivery take? So what you can do is you can plot all these points you know all the past data that you have you can plot it on this kind of chart where on the x-axis you have distance y-axis you have delivery time and you will get all these points. Now you can draw a best fitting line. So let's say you're drawing a line which best fits all this past historical delivery data. If you do that then for the new data point which is 7 km let's say if it is 7 km here you should be able to tell that this will take 32 minutes let's say okay you can make some kind of approximation if you can draw the best fitting line but 18:48 the question is best fitting line can be this it can be this it can be this how do you know which is the best fitting line Because these points are not like linear okay they are like scattered. So how do you figure out a best fitting line okay and for this we use linear regression. So what we can do is essentially we start with some random line and we can measure the error basically. So as per this line okay if let's say my distance is 2 km my actual data point if you look at this actual data point right this would have been whatever let's say 11 but as per the best fitting line this can be 12 right because see as per this line this is going to be 12 so the actual data point that occurred is is basically 11 we call it y and the estim Estimated data point 19:49 we call it yhat is let's say 12. So you measure the error basically. So see see this is the error. Let me use a red color for the error. So let's say this is my error and I call it delta 1. Then this one this is like let's say delta 2 right? This is delta 1. Then this is delta 3. Then this is delta 4. And using some mathematical technique you want to minimize these deltas essentially. Okay. So that is the algorithm. Linear regression tries to build the best fitting line that passes through all the points by reducing the total error. So the total error is what? So delta 1 plus delta 2 plus so on delta n. Let's say this is my n data point. So delta n and this is my total error. Okay. So this is my total 20:53 error. Now in linear regression you don't use just this delta. You actually use something called the mean squared error. So you do the square of this error. So delta 1 square delta 2 squared and so on and you take the average of it. Now why do you do square? Well, because of two reasons. The first is you want to make sure that the error is always positive. Because see this can be positive, this can be negative. If you add them as is then it is going to wipe out the error. It will give you a wrong picture of what is your error. So what you can do is maybe you can take the absolute well absolute that works. That's like mean absolute error. MAE. Okay. Mean absolute error mean absolute. So this is called MAE. This can work. But the technique that gets 21:53 used is mean squared error. M where you will do delta 1 squared delta 2 squared so on delta n² and you take the average which is you divide it by n. So mathematically speaking it is going to be this is the notation in math. So I = 1 to N delta I² 1 / N that is your mean square error and you try to minimize it. Okay. So when you're building this uh you will first start with some random line like this then you'll you will calculate all these errors you know mean square error and then you will use derivatives and a technique called gradient descent to find the best fitting line. So eventually through the iterations it will find this best fitting line and once you have the best fitting line you 22:54 should be able to predict the delivery time required for a new data point. So now let's say a new delivery comes in where the distance is 7. Now you should be able to say that this is going to be 31. So essentially once you have this particular line you will be able to solve this problem. So this process of finding the best line okay for that you use a mathematical technique called gradient descent okay and when you're doing that you're essentially training the model you're training the model to find the best pattern that can match or that can fit all these data points and once the model is trained. So when do you say model is trained? When you have found the best fitting line, that's when you say that model is trained. And once the model is trained, now you can use it for inference, which is any new data point comes in, you can use this line to 23:55 predict that this is going to take 31 uh minutes. Okay. Now, if you know the basic math, uh this line is essentially uh represented by two factors. one is slope. So the slope of the line is m and the second one is intercept. Okay. So the equation is y = mx + b. You should know this as a prerequisite. y= mx + b where m is slope and b is called intercept. Okay. In our case y basically this y is nothing but time and x is distance. So time is equal to m into distance plus b. So what is the intuition behind this? Okay. So let's understand the math and intuition. So here m is a slope and you can think of m as a extra minutes per kilometer. So let's say you have a totally linear 24:56 equation where for let's say every 1 kilometer it takes you let's say 5 minutes. Okay. So in that case your y is equal to 5 into this distance. So let's say if you're traveling 3 km then 3 into 5 it will be 15. If you are having let's say uh x is 5 km then 5 into 5 it's going to take 25 minutes. Okay. So that is simple math for this line. But there is fixed constant also. So let's say it takes the driver some preparation time before the driver even leaves and let's say that time is 10 minutes. So that is your intercept. That is the intuition behind the intercept. Okay. And training is finding the best m and b from data. We are going to write code which will find best m and b. So don't worry if things 25:57 are not clear as we write code as you digest this concept it will become clear and there is a technique called gradient descent. Okay gradient descent is a technique mathematical technique that will find you the best fitting M and B. If you want to learn about gradient descent and detailed math then we have this uh boot camp where we have explained all the maths. Okay. So if you look at this we have explained gradient descent why MSE not MAE all of that in detail. This particular video is a fastpaced crash course for ML. That's why I'm not going to go into too much detail of mathematics here. For now you can just remember that gradient descent is a mathematical technique that lets you find the best fitting line. And once you get the best fitting line you're getting essentially M and B. Now just some noming cle time is called dependent variable the 26:59 thing that we are trying to predict distance is called the independent variable. Okay so you have to remember in the conversation this will come up what are my uh independent variables and what is my dependent variable because time depends on the distance. So that's why it's a dependent variable and distance is independent variable. Okay. So linear regression is a process of finding the best fitting line. It establishes the relationship between dependent and independent variable. Okay. So it will establish the relationship between time and distance. How time changes as distance changes. Let's now train our first model. I have this CSV file with all my past orders where you have distance and delivery time in minutes. I have around 41 data points. This file is present in this particular folder CC code ML practice linear regression. And here I'm going to create a new notebook. So this is a CSV 28:00 file. Let me create a new Jupyter notebook. And I hope you know the fundamentals of Jupyter notebook. It's a way to run interactive Python code. And here I'm going to import pandas spd. Pandas is a library that lets you manipulate data, clean the data and so on. In machine learning, you will see pandas being used very extensively. If you want detailed tutorial on pandas, our channel has pandas playlist. So you can refer to that. Here you can read the CSV by calling read CSV method and simply just run df head. So it is loading the data from CSV file into a data frame which is just a table like data structure essentially. And now I'm going to plot this data to see how it looks. For this I will use mattplot lib 29:01 library. And we have separate videos for mattplot lib if you want to take a look. Okay. You can use AI coding, wipe coding to write all this code. Okay. So here I am drawing a scatter plot and see scatter plot shows that uh this is the distance versus delivery time graph and all these data points are individual deliveries. And what we have to do is now somehow draw this kind of line based fitting line that can then help us predict time for new deliveries. Now let's import the model from this scikitlearn library. Scikit learn library is the main library that gets used to train the machine learning models. Okay. And by the way at any point if you find module not find error let's say you are running mattplot lib and it says mattplot lib module not found in that case the fix is very 30:03 simple you will say pip install mattplot lib I'm assuming you at least know all these fundamentals as a prerequisite okay so pip install mattplot lib pip install scikit learn whatever libraries you have to install just use your common sense and use it and take help of AI. So here we have imported linear regression class from scikitlearn and we are going to create a model out of this and then we will say model do ffit. So this fit is actually running the training. See, so this thing is actual model training. You know, this is where it will find that best fit line. And here you need to supply both the independent variable and dependent variable. Okay. So here uh I have this data frame and in that data frame 31:06 I will use distance. See here let's see this is the name of the column right in my data frame see this is a data frame it has two columns and in the fit method the first argument is the distance it is the independent variable and the second argument is the dependent variable right uh which is delivery time minutes so let me just copy paste this and here the first argument is usually data frame because there can be multiple independent variables, right? The delivery time depends not only on distance but on traffic, on rainy conditions, so many factors. We will learn all of that in the future but for now just assume here you need to pass a data frame. Here you need to pass one column which is panda series or or numpy array. Okay, but here it's a data frame and that is the reason for these two 32:07 brackets. Usually you will have multiple columns okay like traffic things like that that's why you have this as a data frame. So let's do control enter and it is going to train the model. My model is now trained you see model is trained and once the model is trained for a new delivery now you can say model.predict and our let's say new delivery is 7 kilometer. Okay. So here I will supply 7 km. Okay. So control enter. It is giving me the error. Expected 2D array got 1D array instead. Expected 2D array because you might supply multiple deliveries as well. Okay. So I I want to know the time it takes for let's say 7 km delivery and let's say 34 km delivery and then it will say that for 7 km 33:08 delivery it's going to take 30 minutes folks congratulation we just trained our first model and also we ran the prediction for 34 km delivery is going to take 106 minutes. The better way to do this is to pass a data frame. See here I'm passing a two-dimensional list. But better way is to create a data frame. You have distance and you have this seven. Right? So you can say 7 and let's say 34. Right? I want to predict the delivery time for two orders. One is having 7 km distance. Second one is having 34 km distance. And these are the timings that it will take. Now let's understand what it did behind the scenes when we call model.fit. After we call model.fit, it would have created this coefficient. So coefficient is essentially your m 34:12 your slope. Okay. And the other thing that it came up with was intercept. See intercept and intercept is 10. So what this says is it is 2 81 minute per kilometer. Remember that y is equal to mx + b. So in our case y is distance is equal to okay y is actually time right time is equal to m into distance plus b which is let's say initial preparation time. So in our case let's say when you have a 7 kilometer delivery what's going to happen is let's say m into distance right. So what is m is this coefficient 2.81 81 correct? So 2.81. 35:13 So let me just type this into distance is what? 7. So 7 plus intercept is what? This write this. See this matches right? 30.61 30.67. I mean I'm not taking the whole number. I mean if you take the whole number it will match. See 30.67. This is how internally it calculates it. It's a simple math. While training it will come up with these two parameters m and b. And then it stores those parameters into coefficient and intercept. And then when you are doing predict function internally it is running this math only. And now let's draw the line best fit line using mattplot lip. I'm not going over this code because you can generate this code easily with AI and it is simple you know it is out of scope for this tutorial. We have a separate mattplot lib tutorial 36:14 series you can watch it but see look at this beautiful line. It is the best fitting line. You can draw the line this way that way but this line reduces your mean squared error. You get the minimum error average error by having this particular line. And this line is defined by two factors only m and b. And in our case m is 2.81 b is 10.94. So far we had one independent variable which is distance. But we all know life is not as simple as this. On the road there is traffic, there is rain, there is cockroach jenta party protest and so on. And when you have all these factors, the distance alone will not work as a independent variable. You have to take into account traffic, rain and any other factors that can impact delivery time. So your equation becomes something like this. 37:14 Time is equal to m1 into distance plus m2 into traffic and so on. Okay. So here m1, m2, m3. These are individual coefficients and you have one common intercept. This is called multiple linear regression. Initially we had single linear regression. Now you have multiple linear regression and this is a generic form of that same equation. So here we have many independent variables. Okay. So simple versus multiple. In simple what we just did is m into distance plus b. And in multiple you have multiple uh independent variable affecting your target variable or your dependent variable. And this is where what real world models look like. When you're working as an AI engineer in companies like Swiggy or any other company, your equation will look mostly like this. 38:15 you'll hardly have this kind of situation. The example that we saw was to predict time from distance. But when you want to predict house price based on your bedrooms, square foot, location, amenities, you can use linear regression. When you want to predict salaries of of employees based on their experience, degree and any other factors, you can use linear regression. Okay, linear regression can be used for number of problems. All right. In terms of code practice, I'm going to have a different file here and this time this contains distance as well as traffic level, raining, delivery time in minutes. Okay, so we have these many data points. We are going to load this file in our notebook now. Okay, so let's call it DF2 which looks something like this. And I will create model 2 using the same linear regression class. and then model 2 dot fit df and here we need 39:17 to supply all those independent variables. These independent variables by the way are also called features. Okay. So our features are these three. Traffic level is our first feature. Raining is the second feature. Actually the first first feature is our distance. Right? So this distance in kilometer. Okay. So independent variables are also called features because based on these features you're trying to predict something. It's like if you're trying to predict some human face let's say Ladam Mangeshwar or let's say Donald Trump you try to predict the face based on their features you know their nose their face the color the hair etc. These are the features that helps you identify a person. Similarly, these are the features that help you determine the time. Okay, delivery time. So here we will just say features. And what are we trying to predict? Once again, we are 40:19 predicting delivery time in minutes. So I'm going to run the training. Control enter. Okay. Okay. Okay. raining is not in the index because it has to be DF2. Okay, so this is DF2. All right, the model is train. And now for the new order where my distance is 7 km, traffic level is 8 and raining is 1. It is going to take how much time? Well, 39.8 minutes. Hooray. Congratulation folks, you just built a realworld model using multiple features or multiple independent variables. So here you perform training and here when you say predict, you are performing something called inference. Inference is basically 41:20 predicting it for a new data point. At this stage I would like to ask you a very simple question. Let's say you want to learn pickle ball and if I show you a YouTube video of pickle ball practice or coaching, this video has 6 million views. After watching the video, are you going to become pickle ball expert? No. If you want to learn it, you have to take the bat, go to the ground and practice it. Similarly, to understand linear regression, you have to practice it. We have given you this exercise folder. So down below in the video description you will find all the code. Okay. So ML crash course linear regression you have this exercise folder which has salary data. Okay. You see salary data. So based on education level years of experience certificates you are trying to predict the yearly salary lakhs perom. And for this linear regression problem 42:23 we have given you the problem statement. Okay. So if you open that you will find all the hints all the empty cells. All you need to do is fill out these empty cells and write the code. Only by writing this code you are going to understand it. So if you're not ready to write this code please stop this video, go watch some movie on Netflix. Previously we trained our model on all our data 61 records of these and then when the model is trained we deploy it to production and in production it will start making prediction. But how do we know the model that we have trained is a good quality model or not? What is that method to confirm before even I deploy it in production? How do I gain that confidence that this model is going to do well? Essentially, by training a machine learning model, you are trying to capture patterns in 43:24 the data set and then you train the model so that it can work on unseen data. Now, when you train your model on your entire data set, you run into one type of problem. Let me explain that problem by giving the analogy. Let's say you have a student name of the student is Lu. You want to train this person on mathematics and let's say you have some 100 math problems. Okay? So you have 100 math problems and you are training this student. Once the student is trained, you want him to appear for an exam because you want to evaluate if the student is trained properly or not. Now what you do is you take some questions, some math problem from the same data set. Okay? And when you have him appear for an exam, the person may do very good. They may score 44:25 95%. But is this a fair evaluation? Is this the right way to evaluate? Probably not because this student has already seen these questions. Okay, it's like a exam league. You have already seen these questions. So, it is possible that you have memorized them and therefore when it appears in the exam, you will score well. This scoring tells about students memorization capabilities. It doesn't tell you how good they are in math. The right way of doing this might be you have 100 questions. Okay. So let's say you have 100 questions. I'm going to draw it here. Okay. You have 100 questions out of which you take 70 questions and you train Lu on that. Okay. Luho has not seen these 30 questions at all. And when Lu appears in the exam, the exam questions 45:29 are based on these 30 questions that Lu has not seen. And when Lu scores, let's say 92%. You know that this is a fair way or fair scoring for Lu because through this 70 questions, Lu has actually learned the concepts. He has learned the patterns and concepts really well and he's using those concepts to uh solve those exam problems. Okay. And these 30 questions are something he has not seen. It's like unknown. And then he's using his fundamentals, his concepts to solve these questions. Okay. So same concept applies in machine learning as well. When you are training your model on let's say this kind of data set, what if we train the model on only 70% of records and we can randomly pick those records and then we evaluate the model on remaining 30%. It can be 70 30 80 20 46:33 it depends. Okay, there is no like fixed guideline on this. But when you evaluate this model on the data on which model is not trained, you will get a good understanding of how model is going to perform in production. We will use a method called train test split for doing this. So I have loaded the same data set here and then here I'm going to define our features. So you all know that our features are uh these. Okay, these three are features. This is our target variable. And what you will do is you will say x is df of features and y is df of this one your target variable. Okay. So this x is a data frame of features. Y is your target variable or your dependent variable that you want to do prediction on. And the reason this is capital X is because it's a data frame with multiple 47:35 columns with three columns and this is small case is because it is a single column. So let me run this and now this is see this is a data frame. Okay folks so it's just a you took this data frame you took this three columns that's what X is and if you look at X shape it has 60 rows three columns. If you take Y dot shape, pause this video and tell me what you will see in the output. Well, 60 there is no column. It's a series. Okay, panda series a single column. Now, let me use train test split method. You can import it from skarn.odel selection and here you will pass X Y and then test size. So let's say my test size is 20%, 80% records are actually the training. Okay. The return value of this will be a tuple. So there will be four variables. 48:35 You'll get X train, X test, Y train, Y test. Okay. So let's just run it. So the these things are split. And if I look at X train, it is essentially those three columns. But look at the row numbers. It's kind of random. And if you look at the shape it is 48. So 48 if you uh divide that by 60 that is roughly 80%. Okay. And if you look at X test here again it's the same kind of structure here but the shape is 12 rows. So 12 is 20% of 60 records. Okay. And when you look at Y train it's just a single column. Okay, these are the index but these are the actual times you know delivery times. So you can you can just check it out. The shape will be 48 and y test will be 12 rows. Once you do the 49:35 split you will train your model only on train data set. So your train data set is x train and y train. Okay you train your model. the model is train and then you evaluate this model on your test data set. So how do you do it? So you will just say model dots score X test and Y test. So this will return the R squared score. Okay, R squar score has some mathematics involved in it. But if you want to simply understand it, what it means is it's it's a scale from 0 to 1. The value is from 0 to 1. So let me let me just simply first measure it and then I'll explain. So test R2 came out to be 94. If it is one then it is perfect. It is doing perfectly good on all the data points. 94 is like you know student is scoring 94% in the exam. So if it is 50:37 closer to one it is good. If it is closer to zero the model is not doing good. Okay. on your test data set. You can also measure R squar on train data set. Okay. So the way you do that is you will measure the training R2. Okay. After the data is trained on the training data set itself. Okay. So on the training data set itself you can measure its accuracy. So it is not going to be 100%. Okay. But it will be around okay X train and Y train. See the accuracy is coming out to be same for everything uh for the test and train. Now one more parameter I want to you know show you is see right now let me just do this. So X train is note down these row numbers 26 35 48 and so on. Okay. So let me run it 51:38 again. See it is changing. You see because it is random. If you want some kind of reproducibility and if you want similar split then you will use this variable called random state. So random state is like a seed. It's a number that you pass. You can pass any number. Let's say I pass let's say 10. Okay. And when I pass that what happens is now my split will be same. See it will be 20 37. See it's not changing. I'm executing. See, control enter. I executed. Control enter. It is always the same. If you change the random state parameter 42 is kind of popular number that people use. See it changed. But now between the different runs, it will not change. So that is the reason people use this random state parameter. Okay. So let me now rerun this model again. And as you can see on training it got 52:38 96%, on test it got 88%. This is like student scores okay the student is scoring 88% in the exam when they got trained. So let me show you. So what we did is we had this split was 8020. Okay. So this split was 80/20 and when we uh take Lu's exam on this 20% which is unseen questions Lu is scoring how much? See here 88%. Okay so he's scoring 88%. But if you take Lu's exam on these 80 questions on which Lu is trained then they are scoring 96 almost 96% right so almost 96%. So that's what it is. You are splitting your data set into train and test. And then to evaluate the model performance, you always use test so that 53:39 you're testing the model on the data that it has not seen. And you can use this particular score as a confidence measure before deploying your model to production. Okay, this is a good benchmark that you can use. Now in model it's not necessary. It will give you 88% accuracy. It's not necessary because in production you will see different type of data and this is probabilistic model. Okay, the behavioral is probabilistic. So it may not score 88%, it may score differently. But this at least gives you some confidence that okay, I'm confident that I can now deploy this model to production. Let us now look at machine learning task types. We already looked at one category which is regression. In regression, you try to predict a number. The prediction is a continuous number. When you're predicting a house price, it is 100,000, 200,000, 250,000. It is some kind of a number. When you are 54:41 predicting the delivery time, it is 35 minutes, 40 minutes. So, it is one of the numbers. So, that is regression. The second type is classification. Here you predict a category. You have an image and you classify whether it's a cat or a dog. You have a transaction, you classify whether is fraud or not fraud. You have an email, you classify whether it is spam or not a spam. So here you are predicting a category. So regression in real life we have this Zillow which is a property website here in US and on this website you can enter any address and it will show you the home uh price the current price but it also shows you something called Zestimate which is an estimated price of a home. See this one. So here they are using regression most 55:41 likely. Okay. So to predict this kind of home prices you use regression. This is a very real example of uh regression. The second one is salary prediction. You go to these uh jobs websites and for any given company it will show you the median pay etc. See for IBM it is this mu sigma it is this and so on. So that is an example of a regression where you're trying to predict a salary based on number of factors. Stock price prediction is another one. So here they are saying this particular Len into bro you know the target price is this much. Then comes classification. So in classification let's say you have this transaction. This is a real example of my PNC bank transaction and it is showing that uh this can be a potential fraud. Now it will use bunch of logic 56:43 for doing it but essentially it classifies the transaction as fraud or not fraud. So that is classification. Previously what we saw was regression. Another example of classification is email is spam or not spam. uh image classification cat or dog you have another image which is a car image whether it's damaged or not damaged so we we in atlique we have a client who and for that client we are working on this problem okay they are in auto reposition finance industry where you have an image of a car and you want to classify whether it's damaged or not damaged another example is news categorization. So in Google news you have a bunch of news but let's say I want to read only business news then I can read that. If I want to read technology news then it will do that. So it is classifying the news into one of 57:44 these categories. So in classification output category doesn't have to be only two it can be multiple. In this case see the output categories are business technology entertainment and so on. So in classification there are two types of classification binary and multiclass classification. When you are classifying a given transaction as fraud or not a fraud it is binary classification because the output categories are only two. See fraud and not a fraud. Whereas for the news category classification it is multiclass classification because output categories can be more than two. So the real difference between classification and regression is what comes out. In case of classification you have fraud, not fraud etc. So the output variable is categorical. In case of regression it's a continuous number. Basically it's some number you know it's not like fixed defined categories. It can be any number. And here are the uh 58:46 task you know where you will use classification versus regression. So just take a look folks. This is a pretty simple concept. All right. So I hope your fundamentals on classification and regression are clear now. So just to summarize ML task can be divided into classification and regression. And in classification you have two categories binary classification and multiclass classification. Now that you have a good understanding of classification and regression, let's look at our first classification technique which is logistic regression. Let's say you are working as an AI engineer for an insurance company and you have this data where you have age and if the person has insurance or not. So you have conducted demographic survey, you're launching a new product and you want to predict based on age what is the likelihood that the person 59:47 will have insurance. Okay, pretty simple data. Now if you plot it on a scatter plot, it will look something like this. Okay, so here you have age, here you have insurance and you have all these data points. This is clearly a binary classification problem because the output is either zero or one and that is the reason you are having this kind of pattern. If you had a continuous data which is a regression problem then these points will be scattered throughout the chart. But here you have binary classification. So either is zero or one. Now let's build intuition for logistic regression. Whatever we learned from linear regression, we can apply certain concepts here. So what if we draw a best fitting line for this data set? A best fitting line may look something like this. So let's say you run linear regression, you come up with a best fitting line. And since it's a straight line, it is kind of hard to cover all 1:00:47 the data points, right? So this is approximate line that you can come up with while applying linear regression. And here one method you can use is see there is 0.5 right so here it is zero here it is one one means person has insurance 0 means they don't 0.5 is the midpoint. So what if I draw a line from 0.5 here and then this becomes my decision boundary. Okay. So my decision boundary based on this data set is let's say this is 43 age. So anybody who has age less than 43 will not have insurance. More than that will have the insurance. So now when a new data point comes okay let's say there is a person with age 44. Well, they will likely have insurance. So when I'm running my insurance campaign, I'm meeting my prospects. Most likely I will meet people who has age 1:01:49 more than 43 because the likelihood is more. Okay. While this seems like a reasonable approach, let me throw a cobell at you and introduce this new data point. And let's say you have this 90-year-old grandma who is having the insurance. So this is the data point for that particular grandma. You see she's having let's say an age of 90. Now when you draw a best fitting line your line may look something like this. Let's say you have couple of data points. You have some 100 year old grandfather 110 year old person. A couple of data points are here. So now your best fitting line might get skewed like this. And in this case when you draw this boundary at 0.5 you know you might get let's say a boundary at the age 47 for example. Now the issue with this is that some of your uh data points see these three data 1:02:51 points will be mclassified as per this line. Although they have insurance they will likely have insurance. My line is predicting that they will not have insurance. So let's say if I get a new prospect, new client, you know, whose age is 43, as per my line, the person will not buy insurance and therefore I will not spend any time consulting that person. Okay? So this is more like a step function folks. Uh if you know the mathematics, you know, it is like a step function like this. It's a step function where you have hard boundary at 47 and you're saying anything less than 47 they will not buy. Anything more than 47 they'll buy. Okay. And this kind of outliers can skew your line and uh reduce the quality of your prediction. Now let's take a pause. I want all you to think about a better approach. Please pause this video right now. Okay. 1:03:52 seriously and intuitively think what can be a better approach instead of drawing this straight line. Can we do something else? Okay, I hope you have thought about this. Well, another idea I have is how about we draw this kind of line. See, this is short of kind of like a best fitting line because it is fitting all these data points. Okay, it is not a straight line but that's fine. The line that you're drawing doesn't have to be straight. Uh this is just a best fitting curve or we can say and here when you draw a line at 0.5 even though you have outliers like this all these data points will not be mclassified. Okay here my boundary becomes 38 let's say. So anything greater than 38 they will have insurance. Less than that they will not have insurance. Okay. So what exactly is 1:04:54 this line? Well in mathematics this is called sigmoid or logit function and this is the formula for it. So what happens is when you give any number to this function. Okay. And by the way this e is some constant just like pi. It is uler's number. This is the fixed value. When you give any like a series of data points to this, what this function will do is it will convert it into a range 0 to 1. So try this out. Take your calculator, come up with some number, you know, whatever numbers you have and use sigmoid function. In our Python notebook, we'll do that. Use sigmoid function and any number you give, right? 47 38 any number you give here as an input it will reduce it down to number range which is between 0 to 1 and the reason is you are dividing one by a 1:05:57 number which is slightly higher than one. See 1 + some factor. Therefore it will always give you a range which is 0 to 1. So I can show you in the Excel file. So by the way this is the data set that we are going to use for our practice problem. So see 1 / 1 + e number is 2.72. Let's say you pass 47. Okay. See 0 whatever 3. Then you pass let's say any number. Let's say 2 0.1. You pass any number folks. I'm telling you it will reduce it between 0 and one. And that is the specialtity of this particular function. So essentially you have this linear regression line. So you first fit that line you get this equation y= mx + b and then you apply sigmoid function on top of it. Okay. So 1:06:57 see this y applying sigmoid function. Maybe I should call it z. Okay. Let's say this output is actually z y. and this is my y. Okay. So essentially this straight line when I give it to sigmoid function it will convert that into this kind of a curve. We will now train our machine learning model for this use case. As you can see we have around 46 records. I have loaded that data into my pandas data frame. And the first thing I will do is plot the scatter plot. So mplot lib library has this function plt do.catter. scatter using which you can supply your X and Y axis age and insurance and you can plot this kind of chart. Okay. Now we will train a model. Okay. So that it can fit these data points and we can see the outlier. See this person has age of 60 but they don't have insurance. Whereas 1:07:57 this person maybe you know kid of some rich person is 22 year old and they still have insurance. So you will see these kind of outliers and irregularities in a real life data set. I would not call it irregularities but this is how the life is you know there is no like a fixed age boundary for having the insurance. Now let's use train test split to split our data set into training set and test set. This is a standard practice in machine learning world folks. You always take your data set then you do actually data cleaning and feature engineering. We have not covered that yet. Okay. The data set we have is kind of clean. In real life you'll have messy data set. You will do data cleaning, feature engineering and so on. And then train the model. U we are just in the beginning of learning this. So that's why we have clean data set. Once you have clean data set, you will split that into train and test set. 1:09:00 Okay. And we use this function. Now what is our X? See capital X. Always remember that. So our X is age. You see this age. And what is our Y. So our Y is the insurance. Okay. Has insurance. This particular this particular column. Okay. And by the way in pandas you can use this notation or to refer to a column you can use this as well. Both the formats work. Okay. And then to train and uh test split you will supply X Y you will say uh test split test size actually you let's say 20% samples you want in test 80% you want in training and I will give this random state variable to ensure the reproducibility 1:10:01 so that if I run this cell multiple times it will create same kind of uh train test plate. Okay. And you'll get tpple as an output. So here you'll have train test train uh this kind of a thing. Okay. So let's uh print the shape of train and test data set. Okay. So see so 36 rows are in train 9 are in test. So 36 and 9 is like 45 total records. So if you look at X.shape see it is 45 total records. Now you can import logistic regression from skarn. So sklearn dot linear. If you hit tab by the way it will autocomplete. Sometimes this autocomplete doesn't work but majority of the time it will work. Okay. So linear model import logistic regression. See when I hit tab now it is 1:11:03 working. So it depends on it mood and then you are creating a model here and then model do ffit. So this is where you are performing your training. So x train y train control enter. Now see it just got train in a second. In real life situation you will have a huge volume of data set with so many columns. It may take some time. This is learning very less records. So the training is happening very fast. Now I want to perform prediction for couple of ages. So let's say I want to do a prediction for 90 year old. Most likely we'll get answer as one. Then 40 23. So here you need to pass twodimensional array. Okay, even that doesn't work. So X has three 1:12:04 features but logistic regression is expecting one feature as an input. So let me do it properly. Okay. So I will create a data frame with certain ages 22 45 and this and then I will pass this here. See for 22 year old it is saying they will not buy insurance. For 45 they will they will buy insurance. For 60 they will buy the insurance. Okay. So this is how you can do predictions once the model is trained. Other than predict there is another function called predict probabilities. Okay. So this is predicting the probabilities you'll actually get the probability. So this is the probability of them not having insurance and this is the probability of them having insurance. So a 22 year old guy has a 10% probability of having insurance, 89% of not having insurance. 1:13:06 Whereas person with 45 age there's a 66% chance they will have insurance and person with 60 age 92% probability. You can change this like let me see if you have 99 year of age the probability will be even higher. See 99 almost 99% uh probability that they will have insurance. Now let's look at the math inside it. Okay, so we have seen this in a linear regression tutorial that once the model is trained, it will calculate model coefficient and model intercept because see essentially it is using first linear regression only and then it is calling sigmoid function to train that model. So now I got this coefficient and intercept and I can define a sigmoid function. Okay. So this is how you define your sigmoid function. 1:14:07 You have x 1 + 1 plus mathematical exponents. So this is that ulers's number. Okay. - x and then my prediction function will be something like this. So y is equal to mx + b. Right? So m this is the slope. Okay. So let me just tell you our our function is y is equal to mx + b. Correct. So here m is your slope which is this. Age is your input variable. So which is a and b is b. Okay. Now since we are applying additional sigmoid function instead of Y we will say Z and then apply sigmoid on Z and then you will get Y. So that is happening here. Okay. So now let's apply predictction function. So prediction function on let's say 60 year old 1:15:09 person. See probability of them having insurance is 92. And of course this is a probability that it is showing. Okay. So let me show you here. See 92% here. So that's what it is getting. And if you try uh 22, you should get this kind of probability, right? So let's try 22. See 0.1089. So 0.1089 it is e to minus one. So 0 I know you know that much math basic math. Okay. So see internally when you train the model it calculated coefficient and intercept and when you are calling predict probability or predict function all it is doing is internally it is applying these functions and we just looked into the transparency of those functions. Okay. So I hope that makes sense folks. If you're confused, don't worry. You can pause this video. This is 1:16:11 not a rocket science. It's super duper easy. Please pause this video. Use AI. In case you want to develop deeper math intuition, you can do that. But this should be super clear. Now, let's evaluate our model. Okay? And for that, we can use this function called model.score, score which will measure the accuracy of our prediction. So I'm going to give X test. Okay. And what it will do is it will calculate the predicted values and then it will compare that with Y test which is the truth. Okay. So the score that I got is 77%. What this means is this. See, so I have all these test records. Okay. And it uh performed the prediction. So let me do that. X test. 1:17:12 So it said okay this person will buy by not buy etc. Okay. And then when you look at Y test see now compare the actual values. Okay. So see here I got it wrong. So for a person which is age 60 it said they will have insurance but in reality in the real data set they don't have insurance because we can have outliers right? 43 my function said one but actually it's not. So I got two predictions wrong. Okay. Now let's check remaining. So after two we have four ones. So four ones here it got it right and three zeros. So it got it right as well. So two predictions out of how many? So let's look at the shape of it. So out of nine two predictions are wrong and seven are right. So if you divide right see this is the accuracy I got. See 7778 something like that 778. So 1:18:14 this is how it calculated that accuracy. Now folks, accuracy is not the only way to evaluate your model performance because relying on accuracy can get you into a trouble. Let's say you are uh creating a model for predicting whether the transaction is fraud or not for your credit card transaction and we all know most of the transactions are actually not fraud. So let's say you're working in a bank as an AI engineer and out of 10,000 transactions let's say total transactions are 10,000 okay fraud transactions are only five. Now if somebody writes a very dumb function called let's say predict fraud okay and you just supply a transaction and let's say they return zero by 1:19:16 default they just return zero by default so most of the time it is saying not a fraud what happens is you have 10,05 predictions okay so total predictions are total prediction predictions are 10,05 10,05 and out of that the true predictions are how many see it said everything is not a fraud okay so true predictions are 10,000 so what is the accuracy now for [clears throat] these transactions which are not fraud well the the to the accuracy is 10,000 divide by 10,05 99% accuracy. Is this model good? No. It's a horrible model. You need to look beyond accuracy when it comes to cases like this. You have to 1:20:18 check uh other factors like precision and recall. And for that we will use a different function. So let me show you that function. So you will uh import from skarnmetrics you will import a function called classification report and in the classification report you need to pass bunch of parameters. So what are the parameters? Well let's look into the documentation. So the documentation says you need to pass your true data set and your predicted. Why true and why predicted. Okay. So y true is our y test correct. We need y pred. So y predicted is actually nothing but model dot predict on x test. Okay. And then once you get this let let me print the report so you will understand. 1:21:21 Okay. So what this has done is it will take your Y test it will compare it with Y predicted and it will give you this report. Now let me go over this report. So 0 and one are the true labels. So zero here means person is not having insurance. One is person is having the insurance. Now what is the precision here? So let's look into that. So for that I need to print Y test. Okay. And also Y predicted. So I converted Y test to list so that we can save some real estate here. So this is the actual labels. These are the predicted labels. Now of course there are some mistakes. See these two are zero. Our model is predicting them to be one. So these are the mistakes. So the way precision works for the class zero is you always look at your predicted labels. So in predicted 1:22:22 labels look for zero. In our prediction how many records did it say that they will not buy insurance and what is the accuracy of that? So see question is are these three correct predictions? So when you look at the true data set yes the last three are zero. So whatever it said, whatever people it is said they will not buy insurance will actually not buy insurance based on the truth data. That's why your accuracy is 100%. We made no mistakes here. What is recall? In recall you always start with truth. In precision you always start with prediction. Okay remember that. So in recall now let's look at number of zeros. So I have 1 2 3 4 5 0. out of which how many it actually predicted correctly. So out of five it actually correctly predicted three because the first two see first two it got it wrong. 1:23:24 So out of five it got five right. So that is your recall that is 60%. So essentially out of however many people did not have insurance in the truth data set out of those folks how many we captured in our prediction um by saying that they will not buy the insurance well three. So we had total five out of which we were able to capture only three. So 3x 5 is 60%. So that is recall. Now let's look at precision for class one. And this prediction categories are also called class by the way. Okay, this is class 0. This is class one. So in class one, how many uh predictions we had which were one. So see 1 2 3 4 5 6 out of that how many are actually correct. So okay we are saying six 1 2 3 4 5 6. So let's look at the first six. Well we 1:24:24 got four right? See these four are one these are zero. Okay. So we get four right out of six. So that is your precision for class one. It is 67. See and what is the recall for class one which is you look at the actual data set first and then you look into prediction. So in reality I had four people who had insurance out of which yes those four we said yes they'll buy insurance. So we did not make any mistakes and therefore the recall is one. Okay. And then the accuracy that you calculated here is shown here. See accuracy. Okay. And then you have F1 score. So F1 score is actually a single measure that can combine precision and recall both. It is kind of a single measure to represent your precision and recall both. So if you look at the formula it is 2 ult*lied 1:25:26 by precision into recall precision plus recall. So folks this is an exercise for you apply this formula on precision and recall and see if this number is correct. All right folks so in case of classification people use this classification report they also use this another thing called confusion matrix. So what exactly is confusion matrix? So once again let me just show you straight away so you understand. So in confusion matrix you will pass test and prediction. Okay same thing. So these are the true labels. These are the predicted labels and you get this confusion matrix. It looks something like this. Not very intuitive. Therefore we need to use this particular function to make a nice plot and then I will explain. Okay. So it's a see folks it's the same confusion matrix. We are just uh using a nice visual uh to 1:26:27 explain that. So what this confusion matrix means is on x-axis you have predicted label y-axis you have true label. Okay. So in reality there were three no insuranceances and you predicted them to be no insurance. In reality there were two no insurance which you predicted to be has insurance. Okay. So what are those? So see these two. So these two did not have insurance in reality because in reality is why it is test right? These two do not have insurance. Right? So no insurance too. But we predicted them to be having insurance. So see here this is a prediction. So we predicted to have the insurance. So these two we got it wrong. Like how many instances we said they have insurance but we said no insurance. Okay. So let's check that. So we have 1:27:28 four records which has insurance and we got all four of them right. So we did not make any mistake on that front. See these are the mistakes folks. So anything that you see on a diagonal are the right predictions. Anything you see in the other areas are the mistakes. So we made no mistakes when it came to people having insurance in true label. We predicted them all to be having insurance. That's why there's no mistakes here. But there were two records which had no insurance in reality but we said they have insurance. So these are the mistakes. Okay. And these are these two. They don't have insurance but we said they have insurance. And then there are four records. Four. This is like number of records which has insurance and we said they have insurance. So see so this is test right. So four they have insurance and we said they have insurance. Okay. So this is the confusion matrix. Just by looking at this matrix you can get a visual clue on 1:28:30 the mistakes that your model is making. Once again to summarize anything you see on a diagonal. Okay. And this is 2x2 by the way. This can be bigger. In the future lectures, we'll see a bigger confusion matrix. Anything you see on a diagonal. Okay, diagonal like this. These are true prediction. Okay, there are no mistakes here. So these three and four, seven data points, we made no mistakes. But with these two data points, we made mistakes. So far with insurance data set, what we have done is binary classification. Now let's look at multiclass classification where the output categories are more than two. We're going to use sklearn data set here. So if you Google it, you will find this functionality where you can access some predefined data sets from sklearn library itself. If you're studying machine learning in college etc in academics people use these data sets for training 1:29:31 all the time. So there are data sets for uh Boston home prices then there are data sets for California house pricing you see and then there is things like iris data set etc. We are going to use this particular iris data set for this tutorial. And here uh they have these three type of flowers. Sethosa uh versol and there is a third flower. So let me just show you. So versol virginica stosa. So there are these three types of flower. Now you can detect which flower is it based on saple and petal length and width. So these flower will have two kind of leaves. Okay. So the bigger one is sipple and this is the height and width. Smaller one is called petal height and width. Okay. I don't exactly know smaller or 1:30:33 bigger but they have two kind of leaves in the flower and based on the width and height of these two flowers you can decide whether it is versolor virginica or stosa okay so these are the three output categories it's a multiclass classification use case so I'm going to call this function load iris okay and Let's uh call this function load iris and that is your Iris data set and this data set will have multiple attributes. Okay, it has data feature names and so on. So let's look at those. So my feature names are basically sele width and so on. My target names are the names of those three flowers and my data. If you look at data, 1:31:35 it is basically number of flowers that you have. Okay. And this is basically sele width, petal length, petal width. They also give you a data frame for your easy access. And you can just say iris.frame. frame and if you do dotcopy okay and store that into a data frame then you're getting object has no attribute okay I think I need to say here as frame is equal to true then you will get the actual data frame. Okay. So let's see this is same. Okay. [sighs and gasps] So see now the data that you get is a data frame and you can also get that 1:32:37 data frame by saying dot frame dotcopy and that way you get additional target uh column as well. So target column has this unique value. So if you do target dot unique you will get 012. These three are the three flower types that we have. Now I will assign appropriate column names because there are spaces. I would like to make it underscore because that's a python uh snake case convention. And the way you do that is you'll say df doc columns is equal to this. And now see the column names are changed. And in the target instead of these numbers I would like to have the name of these three flowers. And the way you do that is you will say df target which is this 012 and you will map it to this dictionary. Okay. And what is the dictionary? Iris.target name. So you create a 1:33:37 dictionary from it. You map it and when you do that you get these pieces. So setho size zero. Then you have versol and virginica. Folks, if you are confused about this, you can use AI tool to understand the syntax. This comes under pandas fundamentals. We have a separate pandas tutorial series as well. You can check it out on our YouTube channel. Okay, my data frame looks almost ready. I just have this extra column which I can remove. So I can just say df do. Columns is equal to target. You can drop multiple columns as well. That's why it's an array. and just say DF. The reason we are doing it is because this is extra column. Okay, target because we already have species. Okay, now this data frame looks pretty clean. I have my independent variables uh these four and my target variable or dependent variable which is species. Now let's do some analytics on it. I can run this function from pandas which will give me a value 1:34:38 count for each of these flowers. Okay. So each flower has 50 records. Total number of records are 150. Now let me plot things using mattplot lip. So here I'm plotting petal length and width. Okay. So petal length, petal width and this is a scatter plot with different color for different species. And as you can see stosa flowers are easy to classify. They have a smaller petal length and width. Whereas Versicolor and Virginica has higher petal length and width and they have kind of like an overlap but still there is some differentiation right virginica has is on a higher side versol is little less. So I think we should use this feature to perform the classification and before I do that let me also plot sele length and width and see how that looks. Folks I'm not going into this mattplot lip code in detail. You can understand it using AI tool or look at 1:35:39 our Mattplot series uh on YouTube. See for Mattplot lab pandas we have separate tutorials. If I start explaining all of this then this ML tutorial will become like 10 hours long. I don't want that. That is why I'm skipping over some of this code. Okay. So sele length and width is is kind of see it's kind of overlap still satossa is very like clear. You see it's a clear cluster compared to other. All right. Now let's do our train test split. Before that you need capital X and Y. Right? So these are the features and if you supply the features you get X and your Y is this. Okay. So this is your X and this is your Y. So if you do X it's a data frame eventually. It has only features. It doesn't have target. And if you do Y, it has only the target. Now we can perform 1:36:40 train test split. And folks, we have looked at that code before. So I'm not going to write it all from scratch. Pretty simple code syntax. You pass XY test size random state. This can be any number. Okay, by the way, 11 is a random number. You can have 5, 10. The idea is you run this cell multiple times and it gives you reproducibility. All right, training rows are 120, test rows are 30. Now we can use logistic regression to train the model. So this is the model. You can say x train y train. [snorts] Okay. And by the way, you can use this model.classes classes to see the classes that the model learned. Now let me predict some values on the test data set that it has and we are going to call it y 1:37:43 predicted and I will create a data frame this time to compare the y predicted versus y truth. The way you can do it is pd dot dataf frame and here you have actual. So your actual is Y test correct and your predicted is Y bread and you can call it a compare data frame and let's print some 10 rows. Okay, y test here you have to say values pd is not defined import pandas spd and see actual virginica right prediction right prediction this is wrong uh this is right right okay most of it are right in first 10 at least uh there is only one which is wrong now folks let me print the confusion matrix to see how things look. 1:38:43 I can just use the code that we wrote before with some changes. Anything that you see on a diagonal, this are the correct prediction. Outside of that, you have mistakes. So, we have only two mistakes here. So, there were two virginica flowers for real in truth which it said are vers color. Okay. So, it made only two mistakes. Rest of it looks good. So when we print a classification report, we'll get pretty good results. Okay. So let's print a classification report, you import this function, you pass Y test and Y predicted. And this is how the report looks. See, most of it is precision is good. Uh for Satossa, see Satossa, we expected it to be giving a good accuracy because look at this. See satossa if you look at this satoss are blue color it they are very clear that they are separated from rest of the group so that's why precision 1:39:44 recall everything is one versol and virginica has some errors but still overall it's a pretty good result now see when somebody comes with a brand new flower you can make a prediction so let's say this is my new flower with sele length width petal length width and when I Say this say it will tell me whether it's what kind of flower it is. All right folks that's it. In the code that you find below you will also see exercise folder and this is the most important part of this tutorial folks. Exercise you must work on exercise otherwise how are you going to build the skills. Okay. So the exercise here is to predict the t-shirt size based on person's height, weight uh etc. Okay. So we are going to download the data. So here is the data set. Let me show you height and weight. Based on that you are trying to predict the t-shirt size. Okay. So that's the exercise problem 1:40:46 statement. You have a solution as well. But folks don't look at the solution. If you look at the solution before giving it your sincere try I have embedded some kind of AI virus into it. your computer will burn. Okay? You will lose your laptop if you cheat. So, make sure you are giving your first try. See, after I have given you so many hints, you see, you have to just type in code into all these cells and then prove me your worth as a student. Okay? So, once you finish working on the exercise in the comments below, post your results. Okay? how much accuracy you got for this particular problem statement. Post it. I want to see sincerity from all of you. So far, whatever data we have worked with is very clean. It's like this cute dog. But when you look at the real enterprise data, it looks something like this. It 1:41:48 is very messy, very complex, hard to work with. I'm saying this not just based on this image but based on my 20 plus years of experience working on enterprise scale data. When I was at Bloomberg, we used to deal with huge volume of financial data and we see all kind of problems. There are format issues, there are missing values, there are outliers, you know, you see all kind of issues and this picture becomes a reality. See, once you start working on real world data, you will soon pretty much agree with the image that you see here. Okay. Many times in ATL when we are working with clients in their initial meetings they will describe their data as this but when we actually get their Excel files txt PDFs and so on the actual data will look something like 1:42:49 this again it's very very messy hard to work with and the reality is nowadays in any AI project you know when you are building or training a statistical model data matters more than the model. This is the plain fact folks firsthand experience. We have seen this again and again in the projects that we work with in ATL for our clients. Okay, remember this data matters more than the model. And the unfortunate truth nowadays is that people want clean data. They don't want to work on cleaning the data. When you are working on training a statistical model as a data scientist or as an AI engineer, you will spend 80% of your time in cleaning the data. Model training fine-tuning will be very small part. Majority of the part will be 1:43:52 cleaning the data. So now we are going to look into a use case where we will go over various techniques for data cleaning. Here is the data set for loan approval. Okay. So we have applicant ID, name, age, income and so on. And this is the target column whether the person's loan is approved or not. This is also called credit risk modeling. It is a popular problem in finance companies. Companies which issues loan they use uh this kind of statistical modeling to figure out the creditworthiness of a person. So they would have huge volume of past loan data. They would train a model and for the new incoming loan application they will use the model to uh determine if the loan should be approved or not. 1:44:52 Now look at this data. You immediately see some major problems. The first one is the genders are not uniform. See male, M is capital, M is small. Then there is M and F. Some values are missing. Same with yes and no. See by no yes. See folks trust me this happens in real life all the time. If I'm working on enterprise project and if the data is too clean, I usually get very doubtful, you know, suspicious that there is something wrong because the reality is data will 100% have some kind of issues. Okay, look at credit score. See here it is some values are missing. Look at age. This person is 200 years old. Is that even possible? Then income. income uh there are some NAS okay NAS are okay u but it's a tax 1:45:52 column we have to convert it to numeric column okay so we see all these issues in this data like look at this question mark so whenever you have data it is a good idea to just scroll and just have your eyeballs you will immediately notice some patterns and then you can use pandas library you can use some statistical method to find out the actual issues because the actual data set might be million rows you know you won't have time to manually examine all the million rows here I have loaded the data into a pandas data frame and usually if you have worked with pandas if you have followed our pandas tutorial you'll use functions such as df info df describe etc df info will tell you the column information Okay, these two columns are string, this is float, this is string and so on. Then there is df 1:46:52 dot describe as well which will tell you uh the count the mean you know min max and so on. So see age mean is minus5 minus 5 age is not possible right? Then 200 that is again a problem. Then credit score 0850. Okay, that is fine. Okay, this is fine. You get all this percentiles, you get the count and so on. So that will give you some understanding. And then to get an idea on the missing [clears throat] values or NA values, you know, not available, you will run this particular pandas functions which says that each column has 18 rows where values are not available. Let me count the total number of rows. So out of 212 rows, 18 rows do not have age, 10 rows do not have 1:47:53 income, 17 rows do not have credit score. Okay, so by the way, this is also called EDA. Okay, exploratory data analysis. So we are just doing data exploration. We are doing data analysis to get a feel on the data so that we can apply cleaning techniques. We can clean the data and then we train the model. All right. So let's start our cleaning process. Okay. So let me print few rows to kind of see what kind of columns I have. Now see when I'm training my model the machine learning models work on numeric columns and whether person's loan is approved or not it doesn't depend on name or applicant ID. So these two columns seems like they are unnecessary. So let's first drop these columns which are not necessary. So what you will do is df.trop 1:48:54 you will say columns which columns you want to drop. Well, applicant ID and name and then you assign it back there. Okay, see this as is will not change the original data frame. You have to assign it back or you have to call in place true. You have to do one of these. Okay, so I'm assigning it back and let's look at I'm just printing first three rows. See those two columns are gone. Next thing I would do is find if there are any duplicate rows. And the way you do that is by having this function df.duplicated. When you give that to df, it will show you the duplicated rows. And check this. See age 28, income 78, 500. And see, you see these two are duplicates. Then there 1:49:55 is 26. So you will find 26 somewhere else. See 26 12,000 574 26 12,000 574. And when you look at the actual data frame, see this particular applicant ID 1 93 Lamali 28. See 113 Leam Ali 28. 78,500 is the income. Look at this. See, so there are duplicates that we have here. And to remove the duplicates, you can simply say drop duplicates. So it will check duplicates based on all the columns. So all the columns are matching. If you want to supply specific column then you can do that as well by passing subset. Okay. So when you do shift tab in Jupyter notebook on the function it will show you the function help and see here when you do drop 1:50:56 duplicates it will check duplicates based on all the columns. If the values are same in all the columns then it will drop it. But let's say you want to drop duplicates on a specific column then you just supply subset. If you want to supply more than two columns you can do that as well. Okay. So just read the documentation here. We'll keep all the columns. Okay. And then I will say see this is how you get number of rows. Okay. Before and after is after you drop duplicates. And then you can say print before. Okay. Before this is before after. So this is after we're using Python format string. So before it was 212 rows after it is 1:52:00 200 we dropped 12 rows which were duplicate. Now see this income is tax column. You see income if you look at here it is string but in order to make use of it you have to convert it into a number. So the way you do that is first you remove this dollar and comma sign then you get just the number and you convert that to uh an integer number or a float number. So first let's remove comma and dollar sign and the way you can do it is using regular expression. By the way to get this code you can use AI. Okay. Uh you're saying income as type string then you're replacing dollar and comma sign with empty. Okay. And you're doing the strip which means you're removing any leading or legging spaces. And then when you do this, you see 1:53:01 it looks clean. But once again the the column is still string. See income column is string still string. We need to convert that into number. For that we can use this function pd.2 numeric. This is the function that is used to convert entire column from string to number and errors is equal to course which means if you have junk value it will or if it cannot convert due to whatever reason right let's say there is a alpha numeric character here there is a alphabetical text here right 75 a something if it gets error while doing the conversion it will just simply put not available there okay so Let's now convert this. Okay. I mean you can clearly see it it is already number. So if I do info here income is float 64. All right. So that 1:54:03 part is done. Now which column should we handle next? Okay. How about we handle this gender? Because that column see if I look at all this. See it has M, F and so on. So let's get the values. Okay, let's do some analysis. So here DF dot gender dot value counts. And you see all these different values. Okay, we want to standardize it to male and female. So let's come up with two standard strings for male and female and let's convert all of them or let's normalize all of them into these two buckets. And the way you do that is by using map function on a column. You are saying that if it is this male with small m map it to this. If it is m map it to this. See now we are mapping it to male and female as a standard columns and then assign that back to 1:55:06 this column. Okay. All right. Now let me do this. Uh see it is now normalized. And same thing you will do for married column. Okay. So let's do df dot married value counts. Okay. Same problem. So what you should do is use the most powerful weapon which is copy paste. Okay. Marit married here. And I want to standardize everything to yes and no. So if it is Y, I want to say yes. If it is yes, I want to say yes with Y capital. Okay. So observe that. And then if it is N, you are saying no. If it is no, you are saying no. Okay, I hope that looks good. 1:56:11 Dot value count. Perfect. So this is now normalized to two standard values. And you will do same thing with employment status. Okay. So employeement status same problem you have bunch of non-standardized value you want to standardize them okay so this is how I would do it okay same logic folks and now look at this values are mapped to employed unemployed or self-employed now let's fix the outliers pairs and age see age should be in range 0 to 100 right normally here is 200 minus one clearly wrong and also let's see credit score credit score should be in range 1:57:12 300 to 850 so you talk to your business manager they said the range is 300 to 850 okay that's a standard range so you of course don't want to have any zero here. All right. And for age, what I would do is loan approval happens for adults usually greater than 18 age. So I will keep my range as 18 to 100. Okay. So when I run this code, see what's going to happen is if my data frame has any values in age column which is less than 18 or greater than 100 then it will put na value in that particular column. Okay. So let's do this and let's run df do.escribe. Okay. NP is not this is by the way numpy library. So import numpy as np [sighs] and look at this now my minimum age is 21 max is 64 remaining 1:58:14 whatever we had now it's all na and same logic for the credit score okay so credit score less than 300 greater than 850 we put any value in that column okay and see now my credit min and maximum is in some sane situation. Now see we have lot of NA values for age income and so on. Okay. So let's see df dot is na dot sum we have like 20 values in age which is basically see if you look at data frame here see the first row itself age is not uh available right na credit score is non okay so we have to substitute those values with something so for age what can we do okay think about a logic see there is no standard rule by the way there is no standard rule you have a common sense discussion 1:59:16 with your business manager and you you figure out in our case the business manager said that median is the right value okay so if I [snorts] do dfage dot median I get median age as 41 why we are not doing average because if you do average it will get 12.6 six and so on you know it will get floating point number I don't want that so I want to use median value and replace any values with medium values in age column and the way to do that is df age is df okay dot fill na so here you do fill na and whatever is the median value you supply that okay and we'll do it for multiple columns so why don't we just define all those columns here. So I'm just going to copy this and run a for loop and do this all for these columns in a for loop. So for C in 2:00:17 this okay this is C. This is my column and this is C. NDFC dot median. Okay. And then I'm doing this. All right. Perfect. See there is no NA left in age, income, credit etc. And by the way the age 41 right? So if I do age 41 I will see some of these records. The first row was NA now it has this 41. You can you can verify in your data frame. Now let's look at the NA values in employment status. See here we can use median but employment status is a text column. For text column you use something called mode. Mode is the column which is frequently appearing. Okay. So here if you do employment status dot value counts see you will get 2:01:18 employed is the mode actually. So most number of values are in category employed. So that's why if you do mode you will get employed as a result. So for all this text column we can use mode instead of median to replace the na values. Okay. So let's run the for loop here for gender employment status and married and we are using mode to replace those values and then you will do df doa dot sum. See all NA values are gone. Now folks at this point if you are bored then please think that in your real job you'll be doing this work only. This is boring. This is tedious. I understand it but that's where you are adding value. You don't have an option but to go through this grunt work. Okay. So please stay on. Few things are left. We are 2:02:19 almost done. Now folks our models understand numbers. they do not understand text. So we have to convert this textual values in the columns to numbers somehow. So what can we do here? So let's think about it. See approved is a category. Okay, rejected or approved. So we can maybe convert that to one or zero, right? Approved is one, rejected is zero. Okay, so let's do that. We have done this before. Map function pretty straightforward. And when you do this you can see see rejected is zero approved is one. What do you do with a gender column female and male? We can do the same thing right 0 and one. Well no you don't do that because when you're converting it into a number see rejected zero and approved one in the numbers there is some kind of ordering which kind of makes sense. But if you give order to let's say male and 2:03:21 female, you say female is zero, male is one, then it shows that male value is higher than female. No, that is not the case. They are equal. Male, female are equal. It is just that they are different categories. Same way employed, non not employed, na these are like categories. When you're doing news article classification, you have sports article, politics article and if you give numbers 0 1 2 3 4 etc. then mathematical model will think that this value is numerically higher and it will apply those kind of math assumption which is not good. So what we do is something called one hot encoding. So the way one hot encoding works is let's say you have this kind of data. Okay. When you do one hot encoding, you will create new columns per category in that particular column. So what you will do 2:04:21 is let's say person column remains as it is. Okay. So this is Lena, this is Moan, this is Rita. And for gender you will create two columns. Okay. So you will say gender_male and gender female and then for Lena male you will assign zero female you'll find one moan uh male you'll find one zero retita 01 okay so this is one hot encoding if you have three values four values then you will create those many new columns the way you do that using pandas code is you'll just say get dummies. It's called dummy columns. Okay. Uh I don't know why they use this term but get dummies. Then you mention the columns for which you want to apply one hot encoding. And uh there is this mathematical 2:05:23 principle that you always drop the first column. Okay. Uh which is if you have three columns. So let's say you have these two columns, right? You can technically drop this one column because by just looking at this value you can derive other column. It is usually complimentary. Let's say if you had three columns right three columns which are one encoded. If you drop one then by looking at these two values you can derive the third value. And there is a mathematical principle it's called dummy variable trap that is at play. And to tackle that problem, you will always drop the first column. So let me let me show you so you get an idea. So see here for gender male, gender female is gone because we dropped it. Okay. Then we have um employment status. Okay. So selfmp employed and unemployed. We dropped the employed because by looking at these two, see if these two are 2:06:24 false, it means the third one has to be true. So you can derive that third column and married was yes and no. So we had two columns two dummy columns right? Married yes married no. But because of drop first we dropped the the second one right? We dropped the first column. So now we have only one column. Now folks we have all these numeric columns. Okay let me just show you. We have all these numeric columns whose scale is different. See this this is in scale 0 to 100. This is in scale in thousands. Okay. And when you're training a model, if you can convert them into same scale which is let's say 0 to 1 or whatever you know minus1 to one. If you can convert them into same scale then the model will train better. So that is this mathematical principle of scaling. So what now we will do is we will convert these columns into the same scale using this standard scaler from okay let me 2:07:26 just put it here from skarn so skarn pre-processing has all these methods okay so we are using standard scaler you create a scalar here and then you say scalar fit transform on this numerical columns okay so when you do this see all these columns are converted into some kind of range. And what you can do is call describe to see what that range looks like. And here it is. Min is this, max is this. See, so you're converting them into a similar scale. See, now these values are on a similar scale. You know, minus1 to one, that kind of scale. At this point, my data is now clean and ready. Okay. So, if you look at my data frame, see all the values are numbers. We have of course dummy column. So, which is false 2:08:28 and true. So, it's kind of like number only. Okay. So, we have done one heart encoding. We have done scaling. We removed duplicates. We removed NA values. We actually took care of NA values. Okay. We handled any values folks. I know it was a lot of work. I know it's tedious but finally we are getting a reward. Now we can train our model. Okay. So for training our model I'm not going to write all the code from scratch because you all are smart. You have learned it already. I don't need to go through it again. And here approved is a Y. X is anything other than approved. Train test plate logistic regression max iter means uh when it is doing gradient descent you know that algorithm to find the best fit line it will do thousand iteration because we don't want to go into infinite iterations. So you can control number of iterations that you can run you can read more about it online by the way and then 2:09:29 you train a model and your accuracy is 65. Okay let me remove this iterations. Okay, even with that fine I mean it's okay accuracy can get lower that's fine and if you look at the classification report this is what you are getting okay folks the whole point of this exercise was to explain that data scientist and AI engineers spend a lot of time around 60% time in data cleaning and prep data collection is maybe 20% model training is 10% tuning and everything is everything else is 10% person. So this is a major chunk of your work if you are working on training statistical models. Therefore you need to develop all this data cleaning uh data preparation skills. Okay. So I hope you learned something good out of it. And at last I would say do not get bored and do not shy away from doing this dirty cleaning 2:10:31 work. Let's now discuss the next important algorithm called SVM support vector machine and we are going to use the example of iris flowers where we have petal and sample width and height as our features. Let's say you plot them in a scatter plot and if you want to classify them what you have to do is draw a decision boundary. It's a line that separates the two classes. The two classes are Satossa and Versic color. Now, which line is the best line? Like, is this the best line or can we draw maybe a different line? Let's think about that. This particular line looks good. It is clearly separating these two groups. But one thing I'm noticing is this line is kind of closer to this group. It's like you have two villages and there is a boundary and it is this boundary is closer to this particular blue village. So what happens is 2:11:31 tomorrow when you want to predict a new flower let's a new home comes in the village which is this then it will wrongly classify it because if you visually look at it this is more closer to here and this blue data point looks like it should be part of this blue cluster. But since the boundary was drawn closer to the blue group which is vericolor this kind of issue will happen. So what if we have an algorithm that can draw a line which is very far from both the groups. So the line has to be here. Okay. But it is kind of far from blue group as well as green group. In that case, you know, compared to this line, when this new data point comes, it is still in the bluer uh class. Okay. So, the whole idea behind SVM is that it just doesn't separate the classes. It 2:12:31 separates them with as much clearance as possible. So, you're drawing a boundary which is kind of far from this group and that group. So somewhere in the middle so that when a new data point emerges you know they're still kind of separated by this clear boundary and that is the whole purpose of support vector machine. Here the distance between the point and the boundary is called margin. This is basically the margins and in order to construct the decision boundary you don't have to consider all the data points. See all these far away data points you don't have to consider. You just consider the data points which are closer to the boundary. See these data points are closer to the boundary. So you use them to draw the margin. And here these data points are called support vectors. Basically these are the data points 2:13:33 which are helping in deciding the boundary because you're calculating the margin from only those data points. You don't calculate margin from these data points from the far away. Okay. So whichever data points are contributing in calculating margin and making ultimate decision on the decision boundary are called support vectors and the distance these distances are called margin. Now we looked at two-dimensional use case. Okay, two-dimensional it's easy to draw the boundary. In three-dimensional space, you can draw a plane, you know, a plane, a 2D plane. But if you have more than three dimension, what do you do? What is that object? Okay, pause this video and visualize. So for two dimensional, it's a straight line. Threed dimension, it is a plane. What will it be? Uh in n dimension, let's say your 100 dimension. How do you plot this? Well, let me tell 2:14:36 you. You try to visualize it. Okay, n dimensional is hard to visualize. But in mathematics, you can still compute it. In mathematics, you can have a hyper plane in n dimension that can separate the data points in n dimension. It's just that with our eyes, we can't visualize it with our eyes and brain. But mathematically, it is possible. Okay. So support vector machine draws a hyper plane. It is called hyper plane in n dimension in n dimensional space such that it maximizes the margin between classification groups. All you're doing is maximizing the margin between the classification groups. Now there are certain parameters at play gamma and reg regularization. When we write code we'll look at this parameter. Let's go over them pretty quickly. So when you have this kind of um data points you can have this line you know this curved line 2:15:38 which can separate them and these are the support vectors you know all these are support vectors which are basically helping in calculating the decision boundary and these are the margins. So this is one possibility for same uh thing what you can have is a straight line and basically more data points can contribute to it. See here only the data points which are closer are contributing to it. So how many are there? See 1 2 3 4 5 6 7 8 there are only eight support vectors. The only nearby ones. The other possibility is you consider extra data points not just nearby. Okay. So here if you count 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15. See we are calculating more. So there are like 15 support vectors here and here there are eight. Okay. So in the left side what we 2:16:40 have is high gamma. When you specify high gamma it will consider only nearby data points. when you have low gamma it will also consider more data points basically to draw the decision boundary. Let's look at other parameter. So here we have a decision boundary which is kind of going exactly through all these data points and drawing a precise boundary. Okay. And other use case you might have is like you don't want to be too precise and focus on too many details. You just draw a line uh an overall line you know a straight line maybe that can separate the two cluster groups. So on left hand side what we have is called high value of C. So C is the parameter we will use while we are doing coding and it is called weak regularization. On the right hand side what we have is a 2:17:41 strong regularization or it is called the lower value of C. So when you have strong regularization you try to generalize. When you have weak regularization you try to be very very specific. In machine learning language this is also called overfitting whereas this is just generalizing. Okay. Sometimes if you generalize too much it might result into underfitting. We'll we'll look into this parameters in a in a bit but just understand that gamma and c okay gamma and c regularization are the two parameters you'll specify when you're training the model. Now let me talk about one interesting use case. So let's say there is a company okay uh retail company which is uh selling groceries like Patel brothers let's call it ATM and they have all this crates you know food items and here is the temperature 2:18:42 and here is the value which says whether they are spoiled or not spoiled. So when the items are at very lower temperature they get spoiled. when they are at higher temperature they also get spoiled and this is a deviation by the way temperature not exactly temperature but the deviation okay so it's simple like one feature and one output or a target variable so you have deviation in temperature they get spoiled or not and we have total 16 data points okay now when you plot it on the chart essentially What you get is this. So this is a temperature deviation. Okay. And you have the x-axis. Blue is product. Okay. Purple is product spoil. Now for such a data set, how do you draw the decision boundary? Because there is no clear separation, right? So if you have decision boundary like this, where 2:19:42 do you draw it? So if you draw like this, then in this class you're getting this purple ones, right? See, it's not easy. uh in in some cases to draw a straight line. So what you do is you apply some transformation. So this is my x-axis which is temperature deviation. Okay, you just square it. If you square it, basically y is equal to x². So this is a deviation squared. Look at this label. So when you square it what happens is you are creating this polomial and it becomes easy to draw the boundary. Okay. Once again the problem was that with this kind of original data set it is harder to draw a boundary. So what you did is you apply transformation and you got y= x² whatever was your temperature deviation. Whatever was your 2:20:44 feature, you just squared it. You apply transformation. X square is one transformation. There can be other transformations as well. And then it becomes easy to draw the boundary. All right. So when we'll write code, we will use SVC class from scikitlearn and that will take argument called kernel. And in the kernel when you pass poly degree 2 is is basically y= x². So when you pass this kind of kernel value it will apply polomial transformation with a square. I I hope you know the basics. So folks the math math basics are part a prerequisite for this course. So I'm not going to go very deeper into polomial transformation and all that because otherwise this course becomes like 20our long video. I want to keep it scripts. So if you want to refresh your math fundamentals, you can use AI tools or refer to some videos. But polomial with degree 2 is essentially y= x². All 2:21:47 right. So this is what we did. Uh now let's uh look at the data set and we will write code. Now for the data set we are going to use uh UCI machine learning repository. We already saw Iris data set and many other data sets which are part of standard sklearn library are actually coming from UC Irvin's machine learning repository. Okay, this is University of California and we should thank UC Irvin for providing all these free data sets so that we can practice u learning uh machine learning. Okay. They say that gratitude is the best attitude. So as a human we should have gratitude and we want to thank University of California Edwin for this. There is another popular website which is called Kaggel. Kegel also provides so many 2:22:49 different data sets. Okay. Here you can search through the data sets, you can filter, you can explore notebooks and so on. Now what we'll do is we will do some filtering because there are so many data sets. I want to pick a data set for our SVM coding exercise. So here I will go to task and pick classification. We are working on classification. So it has now filtered only those data sets which are for classification. I will also use features. Features is basically number of columns. the independent variables I will make it maximum 10. I don't want to deal with too many features otherwise we'll spend lot of time in feature engineering pre-processing and so on. Okay. So let's look at this and the first data set that pops up catches my attention. So this is heart disease data set and the 2:23:50 features are age, sex, uh resting blood pressure, serum cholesterol, fasting blood sugar. Based on all these factors, you're trying to figure out whether the person will have heart disease or has heart disease or not. Okay. So let's download this data set. So just click on download. I have downloaded it already. So if you're downloading the source code from the video description below, you will get the Excel file ready. So you don't have to do this. But in general, if you are a machine learning practitioner, you should be aware about these two websites, Kegel and USA's machine learning repository. Here is the data set. So AHX uh cholesterol and so on. The target variable is the disease. So zero means person doesn't have a disease. One means uh they have some disease not very 2:24:52 severe. Two is severe. Three is very very severe and and I think there is four. So it's it's like 1 to four is the severity scale. Okay. Four means it's pretty bad. Now when I explore this notebook, what I find is these kind of question marks. So when they don't have values available instead of putting it blank they're putting question mark and this is what you'll see with real life data sets as well. It will not be just blanks it might be some weird characters and we need to handle this in pandas. So I have loaded uh the data into pandas's data frame and when you are doing read CSV if you say na values is equal to question mark it will treat that question mark as n okay so df is ready and this is the cell containing the metadata about all the columns CP means chest pain one is typical okay these are all medical terms 2:25:53 folks I'm assuming coming for is kind of these are like I think categories or types of pains. Okay. Um if you want to know more use AI tool to gain some domain understanding I at least know about resting blood pressure, cholesterol, fasting blood sugar all those things. Okay. And uh yeah here is the severity. Now let's do some data prep-processing. I'm going to first check how many values are na etc. And before that I will see okay total 303 rows 14 columns and then if you do df do is na dot sum this is panda's method it will tell you number of na values for each column so we have some na values as you can see here but there are only six actually so instead of handling them we can just drop it because we are not losing too much data 6 out of 303 three 2:26:53 is percentage wise it's not a big number so let's drop it you can call drop na and then um see we lost six values now it's 297 it is okay the next step is to normalize these values so right now the target is having a scale of 0 to 5 what we are interested in is to figure out whether the person will have disease or not. So we are interested in binary classification. You can classify it into five categories as well. No problem. But since data points are low, I want to keep things simple and do a binary classification. So let's create a new column. And before I do that, I want to know the value counts. Okay, 160 people with no disease and all these people with 2:27:55 disease. Okay, fine. So let's create a new column called target. And here you can say df num. If it is greater than zero then you will say true. If it is not false okay fine that will work. But we want we don't want boolean column. We want integer column. So I will say s type int. And when I do this you will see this target is zero or one. It is also a good idea to do scaling because all these features if you look at it see this is in range 1 and 0 1 1 4 2 etc. But this is in a higher range 145 160 this is 233 etc. See the numbers are in different range. And if you want your SVM to work properly, if they are in a similar range, then it will help. So we will use standard scaling. Standard scaling folks 2:28:56 is nothing but a process of converting a range into a standard range. So let's say you convert it into a range of 0 to 1. Okay? You convert it to a range of minus1 to 1. So there is a fixed range you are you are taking and you're converting the original numbers into that range. And if you look at the formula for the standard scalar, it looks something like this. So you take a data point, you subtract the mean. See this mu is the mean. And this is the standard deviation. So this process transform data to have a mean of zero and standard deviation of one. So you're essentially calculating the standard deviation of all these numbers. And you will get a range which will have a mean of zero and standard deviation of one. All right. So we are going to do transformation. But before that don't you think we should drop this? You know this num column is not needed because we have target. Okay. So let's do this. Let's create x first. So our x capital x 2:29:57 is obviously there is no target column. It has only the features. So from features we need to subtract both right? Num and target. And then y is going to be only target. And then you create an object of a standard scalar. And then you will say scalar dot fit transform. So fit transform is a standard function that you will call on the capital x. And what you get is xcaled. Now xcal when you look at it values will be okay. X scale. Okay. So this is not a data frame. So I'm going to print first five rows. It is a numpy array. Okay. So see the values are in now that standard range actually it's uh standard deviation basically which will be most of the data points will be under three standard deviation. So minus 3 to three. 2:30:58 You're essentially putting them in a narrow range already. Now we are going to do train test splits. Same method folks. I don't want to write code by hand this time. You all know how this method works. You know AI tools can write it. So simply you are splitting the data set into train and test. Test size is 20%. Stratify Y means it will see we have people with heart disease, people with without heart disease. And if there is imbalance, let's say 100 people are with heart disease, but there are 500 people without heart disease. Then if you do stratify Y, it will distribute people having heart disease and not having heart disease kind of equally into your train and test. You don't want that all the people with heart disease goes into X train and all the people without hard disease, right? you want some kind of equal uh uniform distribution. So that's the 2:31:59 purpose of this particular argument and let me print the size of train and test. So train is 237, test is 60. Now let's train our model. Okay. So from skarn dot SVM. So how do I know this? Well, I don't remember syntax. In today's AI age, no one remembers the syntax. You just use AI tools, you use Google to figure it out. Okay. So see uh here if you look at the example code for this you will find that you can import SVC from SVM. Okay. So from SVM import SVC and our model is SVC. This has an argument called kernel. You see kernel RBF is default. What are the other values? Linear means your decision 2:32:59 boundary is linear. It's a linear function. See this is linear kernel. Polomial. Polomial is basically you are applying this transformation and you are creating this kind of boundary and then there is RBF sigmoid. So I'm again not going into math of all of this. RBF is default. Okay. You can read more about our RBF if you want to. But RBF is the one that works the best. Even if you don't specify it folks, it's going to work. Okay. So now let's do model fit X test and Y test. All right. Model is trained. We have a very few data points. Okay, that's why it's not taking much time. And let's print the classification report. Our usual function you get y prediction and you print the classification report. Okay, these are the values precision recall you should look at both precision and recall because in disease diagnosis 2:34:02 recall is more important. If there is a person who has a heart disease and if your model doesn't catch that then that's bad. That's a low recall. Okay. So just remember that in disease prediction recall is more important than precision. So if you have u let's say these people okay so I'm just plotting all these people okay let's say you have three people with hard decision okay precision tells a recall tells out of three how many you are catching so let's say your model does something like this okay and it incorrectly says there is a heart disease for these people as well Here the recall is 100%. Why? Because in the original data set. Okay. So this is truth. Okay. So this one is a truth and this is prediction. Prediction in the truth in reality three people had 2:35:04 a disease and my model found this for three people. So we notified them that okay start the treatment. But our model made mistakes. People who did not have disease, it said that you have disease and of course this is bad. The these people will get tensed but when they do further test they will find out that they are fine. They don't have a problem. Okay. So it is okay when it comes to disease prediction to have a little bit uh less precision but your recall has to be very high. But let's say you have a different model. Okay. So this is model A, model B. So this this model is 100% recall. And tell me what is the precision for people with heart disease. So see it predicted that 1 2 3 4 five people have heart disease. But in reality only three has it right? So the precision is three out of five. See 2:36:07 there are two classes. Okay. No disease and disease. We are only talking about disease this one. Now let's take a different model. Okay. So this is model A. And now we have a model B where in our prediction we are not you know we are not these people who do not have disease. We are not saying okay you have disease. Okay. So fine we are not uh increasing the blood pressure of these people but the model also doesn't catch that one person or two person who had disease. So here what happens is see here my precision is this. So in this case my precision for people having heart disease. So I said two people have heart disease in my prediction and in reality these two people have heart disease. So my precision is 100%. 2:37:07 But my recall what is my recall? See there were total three people having the heart disease. I caught only two. So my recall is 2x3. This is is this good? Tell me. No this is not good because this person will go home. They will eat samosa, baja, all the fried item. They they will be like, "Oh, my heart is okay." Okay. They will not take enough precaution, not take medication. There will be enough trouble. So when you go for machine learning interview, they always uh ask questions on precision recall. So make sure you have a clear understanding. Okay? And then confusion metrics. Very straightforward folks. See, anything on a diagonal is all good. But we have two labels where they have disease for real and we are saying they don't have disase. This is bad actually. Okay. So we should fine-tune our model so that we 2:38:08 don't make this kind of mistake. We have absolutely high recall. See here uh for disease we have little less recall. We can fine-tune our model and make sure our recall is higher. It's okay to have less precision. You can try a different kernel as well. Okay. So, let me try linear. Usually RBF performs well. Linear is little lower, right? The 89. Okay. RBF. RBF. So, you have 9694 here. 9694 kind of. Yeah, linear is also giving good accuracy here. But if you have a complicated data set, usually RBF works better. Now folks, it's a good idea to add all these commands in the Jupyter notebook. Okay, to add a command, see in any cell, if you convert it to markdown and if you just say uh training for 2:39:09 example, see this is a markdown syntax. And if you add these kind of comments, your notebook becomes more readable. Okay, so make sure you're doing that. Now let's look at C and gamma. So C and gamma are the parameter you can pass C in SPC you can pass C and gamma. And what I have done is I have come up with different values of C and gamma. Okay. So see here you have uh these are the C values. Okay. 1.1 10 and so on. And you have values for gamma. So gamma can be uh small gamma means very smooth boundary. High gamma means most complex boundary. Okay. And when you run this code you will get different values of accuracy, precision, recall. See and we'll learn more when we 2:40:09 talk about hyperparameter tuning. So when we are kind of doing hyperparameter tuning here. So you are applying different values. These are like different dials in your model training. You're applying different values of gamma and C and you're trying to find out which model gives you the best performance. So in my case, this is recall is my highest priority. So here it's giving 79 I think. Okay, I I'll probably go with this one because this is I want this value to be maximum. Okay. So I will go with a value of gamma 0.1 and c equal to 10. So see gamma has all these values scale and auto and then c has this regularization parameter. Okay. The strength of regularization is inversely proportional to c. So if you use higher C it means weaker regularization. If you see higher C weak regularization strong regularization is lower C. Now we 2:41:12 will discuss a topic that gets asked often in interviews and it's a real problem that you will face when you are building real world statistical ML models and it is called overfitting and underfitting. We all know overfitting and underfitting when it comes to the wearing the t-shirt. In the first one, the t-shirt that I'm wearing is too tight. It's overfit. The second one is under fit. It's too loose. The third one is a balanced fit. Which one you all like? Usually balance fit, right? Why? Because if it is overfed and if my body increases, let's say if I eat a lot of home-cooked food prepared by my mother-in-law, she's here right now at my home. We just had a puran puli full of ghee. If I keep on eating things like that, my body will uh expand. Obviously, the t-shirt will not fit. So I don't prefer over fit. Under fit is also not good. Doesn't look good, right? And the 2:42:14 balance fit is a perfect approach where if my body increases or decreases by little, I will still be able to wear that t-shirt. Let's understand this using students analogy. Let's say there is one exam there are three students. Ravi Ravi is this person you know who goes through the syllabus at the very last day or just one day before the exam. He will go through the syllabus. He'll read few things and that's it. He's not very sincere. Then we have Priya, sincere student. She understands the concepts. She has practiced it. She has understood the fundamentals of whatever she's learning. Then there is Arjun. He is very good in memorizing. He has memorized 10 years of papers word by word but he's not understanding a concept. Okay, he's good in memory. Now what will happen when you have a practice test from the past papers the ones Arjun has memorized. So now there 2:43:14 is an exam and exam is coming directly from the practice papers which Arjun has memorized it really well. What is going to happen now? See Ravi is not going to score much because he's just reading at the very lastly. Okay. Not very sincere. Priya is sincere. She understands concepts but she has not memorized that exact taste paper which which was given during the exam. Arjun will score 100%. Why? Because the paper he has memorized the whole thing word by word. Now who is a good student? Pause this video and tell me. Okay, keep the answer with you. Let's take another exam. new exam where the test paper is from new questions. Okay. So now it is not the one that Arjun has memorized. So let's say it's a math exam and you are getting entirely new test paper based on the topics which were taught in the class. Okay. So it's not nothing out of syllabus. Ravi will still get 38% because again not a very 2:44:17 sensitive student. Priya will score well because she has understood the concepts. She's not memorizing but she understood the concepts. And then Arjun might score 45%. Because now things are not from his memory. The questions are new. He doesn't know how to answer it. Okay. So Arjun didn't learn the subject. He learned the answer key. He he just memorized things. Okay. Now let's map this analogy to our overfitting underfitting. So the past papers are called training data. When we train the model, you know, we do train test split and during the training the model, we are training the brain of which is the model and then when we evaluate the model, we evaluate it on the test data. Right? So this is your X train Y train that is your test paper and this is your X test 2:45:19 Y test. So that is a real exam test data you have never seen before. So in this case the behavior that Ravi is showing is called underfitting. Whenever model is using a simple approach not being very sincere not capturing enough patterns and let's say my actual model has to be let's have this kind of data set. So u underfitting will be like something like this. You are drawing a straight line. You are keeping a very simple approach. That is Ravi. Priya is a good fit. So Priya's training model will look something like this. So she's drawing a line which is a well fit from all the data points and Arjun is overfit. So what Arjun will do is from all the training papers he will try to memorize all the pattern like imagine a line that is passing through all the 2:46:20 data points correct but the problem that Arjun will face is when there are new data points like this Arjun will get this taste error whereas Priya's line is generic she will be able to perform well I know you all are smart you had already understood it Okay. So this is under fit. This is good fit over fit. In overfit you try to memorize the patterns from training data too much. So you get this kind of wiggly line. And now when you have new taste data points you know you get like big errors. See these are big errors. But if you have good fit and when you get new data points the error on the new data point is not very big. Okay. So formal definition the model which is memorizing the data noise and all instead of learning the pattern you're not trying to capture the true pattern and trying to find out the best fit line you are just memorizing the data that is overfitting underfitting 2:47:20 model is too simple to capture the pattern that is genuinely there. So see under fitting like you're just drawing a simple simple line. Now how do you spot it? When you have a low training score on both train and test that is underfitting. But if you have a high training score on basically training data but the score on test data is low that is overfitting. When you have high uh on both training test that is good fit. As a data scientist and AI engineer your aim is to get good fit. Okay this is what you want to deploy in production. And if you have a case where training score is low and test score is high, check your split. Some kind of split error, some kind of data error. Now why overfitting happens? First model is too complex. What I mean by that is you have once again this kind of data. Okay. And what you did is you try to draw a 2:48:21 line which is just trying to pass through all the data. This is what we mean by model is too complex. So let's say you need a linear equation. Now you're using polomial equation. So polomial is complex compared to linear. Then the second reason can be too many features. You know you have 60 features in your data set whereas in reality only five features are impacting your target variable. Now if you're using all all the 50 features for your training of course you [clears throat] will get overfitting. and then you have too few or weak features. So that's another reason and you're training it for too long. It's like the student is going through exam test papers so many times so many times that they will memorize things they will not learn the patterns. I remember my brother's example my elder brother he would just read one book so many times that he would almost memorize the whole thing. Now by memorizing he 2:49:23 has not learned the actual pattern. So if when he gets new kind of uh questions in the exam he will get confused. Okay. Underfitting. Why does it happen? When model is too simple. We just saw that if you're using a very simple mathematical model then you get underfitting and you have too much regularization. And we have not discussed regularization yet but it's a technique that is used to uh control the overfitting underfitting. Okay. And then you have too little training data. You have too little train training data. The the model will not learn the actual patterns. It will come up with a simple line. Okay. And then you're training for very few or less training cycle. If you want to fix underfitting, you can use more powerful model. See becoming data scientist or AI engineer is I think more art than science because you have so many models to pick from. You have support vector 2:50:25 machine, random forest, decision tree and even in those models you have so many parameters. In support vector machine we saw you can fine-tune it using the two parameters right like gamma and C and also you can specify different kernels. So it's like tuning different knobs which can give you the best performance. So to fix underfetting you have to use more powerful model for a given situation. then you add or engineer better features because underfitting happens when you have less number of features or weak features. So if you are adding more better features okay then you're reducing the regularization more data does not fix underfitting. Okay. So you you have to look at these three factors and to fix overfit you have to get more training data. You can use a simpler model. You can use fewer 2:51:26 features and increase regularization. These are general guidelines folks. For a given situation you might use one or multiple. It's not like you have to use all four. Okay? It's something situational. You have to decide by by looking at your data by experimenting and so on. The idea is you want to give the model less room to memorize. To demonstrate overfitting and underfitting, I will go over this notebook quickly. We'll not write all the code with hand because I don't feel it's necessary. Here we have used make moons function from scikitlearn data sets to create a synthetic data set. So we are creating a data set here. Okay. And then we are using the standard scaler to scale it. Then you are doing train test split. Then you are doing uh some kind of plotting. So make moons function will give you this half moons 2:52:26 you know like like this kind of half moons data. So here you have two kind of classes okay blue and red. And let's say you want to train a model to classify these two categories. Now what we'll do is we will train support vector machine with different values for gamma. See gamma less value means underfitting. Gamma's value if it is middle one lesser one it is good fit. If it is very high it is over fit. So when you train the model see you're running a for loop here and you're passing this gamma 0.01 1,000 in three iterations. Okay. So for loop is running for three iterations. Every iteration you are creating a support vector machine. You're changing only value of gamma. C is same, kernel is same and you are fitting the model and then you are calculating training and taste accuracy and you are just doing some plotting. So here if gamma is 0.01 model will under 2:53:28 fit. You see it is just drawing a straight line. If gamma is one it is a good fit. See it is drawing a proper boundary. You see uh through this boundary you are able to differentiate blue and red dots better. Okay. And overfitting is like there is no clear boundary u and everything is kind of mixed up. Okay. And there is some kind of validation using training score and cross validation score. Okay. So we are going to look at cross validation by the way little later but please take a look at this notebook. It's not too hard. You will get an idea. To summarize gamma in support vector machine is one way to control this overfitting problem. Okay. So by supplying higher gamma you are getting balance fit. lower gamma is under fit and extremely high gamma is overfitting. So this notebook just shows 2:54:31 one way to handle this overfitting underfitting problem which is tuning up your gamma value in support vector machine. Now let's understand one related topic called bias and variance. It is related to the overfitting and underfitting concept that we already discussed. Okay. So let's assume you are training a home price prediction model and these are my data points. Now I split my data points into training and test data points. The blue data points are training data points. Okay. So I call train test split. The red data points are in test. Blue ones are in my X train. Okay. So my blue data points is basically X train and my red or orange I don't know what that color is is X taste. Now when I train a model clearly this is overfitting okay it is just too much 2:55:32 fitting the training data points. When I do that what will happen is my train data set error will be zero but my test data set error might be high. Let's say 100. I'm just making up this number folks. Let's say I get taste data set error as 100. Now I have a different use case. Okay. So let's say a different data scientist rain this and they split the data points differently. So train test split is kind of random. So you'll not get exactly these blue dots in your train all the time. If you pass same random state of course you'll get it. But let's assume that a different data scientist is training this model. They are passing a different value of a random state. And second time their train test split is different. So now I'm getting all these blue points as my training data points. My model is same. My approach modeling approach number of iterations everything is same. So it will still overfit. It will go through all the training data sets. But this 2:56:32 time the test data set error will be different. Okay. So whenever you are doing overfitting your taste data set error will be different from one split to another. Here it was 100, here it is 27. Okay. So both are valid cases. You're using same model, same number of iterations. Your train and test split is different. Let's say you passed different value of random state in both these uh experiments. Now your test errors are different. your train errors are zero in both the cases. You see train errors are zero. So this is called high variance. So variance always looks at the taste error. Based on the split, based on the experiment, the error is very different. So it's called high variance. So this is a case of high variance. Test error varies greatly based on a selection of a training data set. Okay. Let's look at 2:57:32 another use case. Let's say you are underfitting a model. You're drawing a simple straight line here. Your train and test data set error. Again, I'm making up these numbers. Okay, let's say 43 47. And then second data scientist runs a different experiment. They have different train test split. You see, watch the blue and red lines. Okay, different split. Uh once again, you are having a straight line because the modeling approach is same. It is still underfitting. and my taste error. Train error is also kind of similar. So here taste errors are kind of similar. It is called low variance. But then my training error is higher. Okay. So bias is a measurement of how accurately a model can capture pattern in a training data. When your train error is big, it is said to have a high bias. Okay. Whenever training error is big, it is said to 2:58:34 have a high bias. Model is biased. It's not capturing the true pattern. You know, it's not capturing like this kind of a true pattern. But when you have a balanced fit model like this, your train and test data error will be lower in both the cases. It's like whatever your split is train test split however you do that your test error and train error will be low. So you have low variance low bias. Okay. So once again over fit under fit balance fit. Uh the the t-et picture is just to for you to help remember this and this is called bull's diagram. So you have low bias low variance high bias high variance. Okay. So ideally you want you want to target this kind of model. Once you get this kind of model then model performs really well. So it's bias is low variance is low. If high bias then see this is the target right? This is what you want to 2:59:35 be. But then high bias will pull your model in a different direction. But the variance is low, which means these diamonds are not very far apart. When you have high variance and low bias, you're still centered around your goal or aim. But see, high variance means there is there is difference. See here there is no difference. The points are nearby. That's why it's low variance. See points are nearby. And low bias means they are closer to the circle. See closer to circle. both but then variance is high which means the points are scattered and high bias high variance is the worst case scenario you know where points are scattered also they are very far away from the goal all right so I hope you got the idea this is one of those questions which again gets asked a lot during the interview let's move on to the next algorithm which is decision tree let's say you have this kind of 3:00:37 class split and you can easily draw the decision boundary okay this straight line is like a decision boundary between the two classes. But what if you have this kind of data set? See here you can't draw straight line. You have to draw multiple lines step by step. So first I separate out the bottom purple data points using this straight line. Then I create this another vertical boundary to separate out the top left green data points and so on. So this is your decision tree. You are making checks okay [snorts] step by step and you are dividing it into different groups. Let's assume you have this kind of data set. You want to predict based on company job degree if person's salary will be greater than $100k a year or not. Now here of course the salary depends on your job. What kind of job role you have 3:01:38 degree to an extent however nowadays degrees are losing its importance. But let's assume degree play is playing the factor here. Company obviously if you work for Google versus some random pharma company with low budget you will get more salary in Google. Okay. And here the green rose okay this is important convention. Green rose means person's salary is higher than 100k. See 100k purple rows or red rows actually here is purple in the next slide I will have red rose purple rows means salary is less than 100. So on this data set if I ask you to use a decision tree let's say you make decision step by step how would you do it? Let's think about it. So the first thing we can say is which is my company. Okay. So you divide these rows based on the company. So Google has all these rows. Facebook has all these rows. ABC pharma has all these rows. It turns out 3:02:41 all the people in Facebook are getting greater than 100k. Maybe they got lot of funding, a lot of extra money to spend. Facebook is known to uh spend money unnecessarily. So now every single person in Facebook is getting 100K. So then my decision is clear right I can have a decision tree where I say which is the company if it is Facebook yes you are getting 100K at least based on this data set. So tomorrow a new person comes in and they say they are working in Facebook. I can say yeah they are making more than 100k because in my data set training data set every person working in Facebook is making greater than 100k. How about Google? Well in Google we can do further split. ABC pharma also we can do further split. So let's think about it. How should we do the split? Look at this green line means person is making greater than 100k. So if you think about it sales executive see we have two sales executive both of 3:03:44 them are making less than 100k red means less than 100k. So then I can have this kind of a tree. Okay. So I can say that in Google if you are a business manager you see business manager you make 100k. If you're a sales executive you make less than 100k and for computer programmer is kind of mix. So see business manager yes you make greater than 100k. So folks remember this is the problem we are trying to solve. We are trying to classify people into two categories whether they make a greater than 100k or less than. And here we are saying if you are in Google business manager you'll make greater than 100k if you're in Google sales executive you'll make less than 100k and for this there is a question mark. So then you further split you further split using what degree. So if you're bachelor less than 100k, masters greater than 100k and you can do that for rest of the nodes and 3:04:45 you get this kind of tree. Now tomorrow a new person comes in they are working in Google they are a programmer computer programmer and they are having a master's degree. So then you will run through the decision tree. You will say company Google yes programmer yes master degree you'll say yes they will make greater than 100k okay pretty simple now how do you decide the ordering of features because see we had all these columns right job degree we first consider company as a first level split we could have considered job as well degree as well okay so this is where the algorithm comes in play we need to carefully select Otherwise we'll do lot of unnecessary split. Our decision tree will not be a good quality tree. So how do you select ordering of features? Well, there are couple of criterias. The first one is let's let me show you example. So you are here 3:05:47 splitting by company. Okay, Facebook and ABC Pharma. And let's say as an alternative option you split by degree and I'm just taking a subset of the actual data set. Now what happens here is in the first category I have low entropy. Low entropy means all six are making greater than 100k. Zero there are zero people making less than 100k. And in ABC pharma one person you know green is one uh red is three. In this case I have four greens. See 1 2 3 four greens and four red. So high entropy. High entropy means there is high randomness like high mix of both green and red. Okay. So here if you split by company you are doing high information gain. Think about it. It's very intuitive. When you split by company you're getting high information which means you are getting clear splits with low entropy. Here there is some entropy but still see 3:06:49 just look at the colors. All green all red just one green. Here is mix. Here is like kuri you know red green red green kind of like low information gain. So then what you will do is you will take a feature. So here the feature is company here the feature is a degree and then you will do a split and measure the information gain and whatever feature gives you the high information gain you select that initially. Okay. And in the second step you just repeat the process. There is another method that you can use which is called genie impurity. And here also see you do a split and you try to figure out the impurity of the split. So here the impur is less because everything is purple. There is only one green. Okay. Uh and then you do a further split and try to figure out the impurity of each split. And based on 3:07:50 that you u make your decision you know you you make those ages and so on. If you Google for genie impurity formula you will find this 1 minus you know sigma 1 to n pi squar where pi is the probability of data point belonging to a class in the node. Okay. So you can look at the formula here they have explained it in detail. Okay. So see you split and then you look at the five blues, five greens, okay? And then you calculate this genie impurity by using the probability. Okay, so this is the math. We're not going to go into math but just remember that you use genie impurity at each split and then you create your decision tree. All right, let's jump into coding. Now we will now do classification using decision tree. I have loaded the same hard disease data set that we have looked at it before and 3:08:52 I have trained SVM classifier. So if you remember our SVM section, we have already done it. Okay. Now we are going to train the model on the same data set but using decision tree. So let's import decision tree. Okay. You'll say skarntree import decision tree. you will create a model. Okay, this is entry classifier and it has a random state parameter. So if you check the parameters, it has couple of them. Criteria by default is genie. Genie will measure the quality of a split. You can also supply other values entropy, log loss and so on. Okay, we'll we'll just keep default which is genie. based on a situation you can change it to a different one and if you want to look at the mathematics of it you can click on it and check the math formulas okay now let's train the 3:09:53 model I will say model fit x train y train and then we will uh print the classification report so classification report is going to be the y test okay and then y predicted so Y predicted is nothing but model.predict predict on X test right so that is your Y predicted and I will also supply target name so it's not like 0 and one you see some labels here and let's print the report okay so this is a usual practice by data scientists that they run the model on different classifier so I ran it on SVM then I ran it on decision tree you can then run it on logistic IC regression and so on. Now here my accuracy precision recall is lower than what I got from SVM and this will happen because you're running a two 3:10:54 different algorithms. Okay. So here the accuracy was 85%, here it is 70%, even prison recall numbers you can compare and that can happen. So let's try to solve this problem. So I will I mean I'm not sure if I'll be able to solve the problem but at least I will make an attempt. So in decision tree classifier now you have all these dials and knobs that you can modify. So the first one I will change is the max depth. So when you have a default whatever you go with default. Okay what's the default value? Let's see. So when you have max depth none it will just keep on exploding until all the leaves are pure or until all leaves contains less than minimum sample split. Okay. So there is a likelihood that it will overfit because if you're trying to do a decision tree which is too detailed, you're taking decisions for each and every node, you're becoming very very specific and your uh model will overfit. So one thing 3:11:58 we can do is control max depth. Okay. And this you will learn by trial and error, right? There is no like u fixed guideline. So let's say I want to keep the max depth as three. So once the tree depth is three, I want to stop it. So that way I get some generic results. It's not like I just keep on splitting until every leaf is one node and and that becomes too specific. That becomes overfitted. So let's train the model same as the previous one. Okay, same as the previous one. Uh or let me just do it differently here. So let me do it here. Okay. So this is same as previous one but the only thing I changed it max depth is three. And when I run it you notice that my accuracy improves from 70 to 80%. See so that that that's the power. Let me do this. Okay. This is not working. Okay. Okay. I think three 3:12:59 three is good. And also one one other thing you can do is for the previous classifier. Okay. So let me run it again. uh here you can u measure the accuracy on train and test data set. So since this was overfitting I'm assuming that the train accuracy is going to be higher test accuracy will be lower. Okay see train accuracy was 100%. Because we split the tree I I hope you're getting it. We split the 32 all the way down and we are making specific decision but test accuracy is 0.7. Now let's print this for this other case. See here train accuracy this is this is what you should have train accuracy is not that high but then taste accuracy is higher. You want to build a model which can generalize better. Now I would like to actually 3:13:59 print this particular tree. So there is a convenient method. So from skarn tree you can say plot tree and in plot tree you specify your model and okay so it is plotting it but it's not visually appealing. So let me import mattplot lip and I'm going to change this code to this. Okay. So here now see look at this. So this is my tree. This is how it worked. So it had all these different parameters tell genie samples and so on. And using those okay let me print data frame here so that we know see all these parameters uh cholesterol this tell 3:15:00 and based on that it is making all these decisions and you can see that this uh blue nodes are the disease classes see classes disase and the orange nodes are the nodes with no disase So you are making all these checks okay the is less than 4.5 and all this then go here then the second run you calculate the CA okay CA less than.5 less than 3.5 and then you're doing further breakdown see this is beautiful right this I I love this method it can just give you a proper decision tree and now you can use this to make predictions on the new samples. Now I want to see what kind of feature importance it gave to each of these features. Okay. So the features here are all this H sex CP all these are the features. I want to know how did it decide to go with TL first. Right. So TL 3:16:00 T h is clearly the most important feature. Then there is CA then there is old pick etc. For this there is this uh variable in your model in your train model feature importances. Okay. So this is your array numpy array with all the columns. So I have all these columns right as so on. So let me just uh print this df dot columns. So this is AS6 CP whatever call FPS race tell whatever and you see the tell is uh the last one 49 see that goes the most higher value if you want to plot it you can use mattplot lip and plot it like this see so in this tree is having almost 49 see this 49 the second one is CP P which is 3:17:04 less than.3 greater than 2. So this is CP. See 1 2 3. It's pretty straightforward folks. Pause the video, watch it. So after training my model, it decided these features were important based on the genie criteria that we had. All right folks, now comes the most important part of this tutorial which is exercise. So in exercise you have the breast cancer data set. Okay. So if you look at it you have all the features and based on that you are detecting whether the person has cancer or not. Right? Targets one and zero. Right? That is your target. And here is the problem statement. You will take this notebook. I know you all are sincere students. You're not going to look at the solution unless you have sincerely tried in this notebook. And here we have given all the steps. You just need to fill the steps with the code that you're writing. Okay. Now we are going to try different models. So I would like you to try 3:18:05 logistic regression, SVM, decision tree, all of that and find out which model performs the best. Once you have found it out in the comments below, please post it so that I know you're not just watching video, you are working on the exercises as well. The next algorithm is the one that gets used mostly in the industry. In our company Atlick, when we are working on classification problems, most of the time we use random forest or XG Boost. XG Boost is something we'll cover later, but right now we're talking about random forest. And actually, it's a pretty easy algorithm to understand. What we do here is see forest has trees. So we use multiple trees essentially and we do the average voting. So let's say you have this data set and you have 3:19:06 whatever let's say 20 columns okay and you have 100 rows. Now you will train multiple trees because forest has multiple trees, right? Same principle. So let's say this is my tree number one. For this tree, what you will do is you will pick a subset of columns, not all 20 columns. And I'll tell you why we pick subset of columns in a bit. But as far as method is concerned, we pick subset of columns for this tree. So let's say for the first split, okay, so this is a tree. For the first split we pick five columns. Okay. So let's say B, D, E, and K and something else. Let's say L. So for the first split, we pick that column. So tree one will always have five columns randomly selected. In the second split again, you 3:20:06 will randomly pick five new columns. So you will do random. So same column might be repeated. So let's say second time I got B and then C and then F and M. And in the third split again five columns. The idea is tree one will work on five columns. Okay. In terms of rows you will go with 100 rows only. But these are randomly sampled bootstrap rows. What does that mean? So what you'll do is you will randomly pick one row from this 100. Okay? So let's say I pick row number four. So row number four I pick. Then you put four back again. Randomly pick. So it's not like four doesn't get repeated. Four may get repeated again. So in the next one, let's say I got row number 72. So I'm randomly randomly sampling. I I hope you 3:21:06 know random sampling concept from statistics. then I got 68. Then I may get four again, then I may get 72 again. Okay? So you're randomly sampling 100 rows and on those 100 rows you combine these five columns and you train tree number one. Similarly you train number two, tree number three. This can be configured how many trees you want to supply is a configurable parameter in random forest uh class that we will use from skarn. Okay. So you train all these trees. Okay, all these let's say three trees and then when once the training is done when you're making prediction for a new incoming row you will let it go through these three trees. It is as if you have three experts and you are passing your decision through three experts. So let's say you are working on binary classification problem and you're 3:22:08 trying to decide if the uh transaction is fraud or not. The first tree says yes it is fraud. Second one says no it is not fraud. Third one says it is yes it is fraud. Then you will take a majority what okay? So you will take majority vote. So my majority v is yes. So then you will say uh okay this transaction is fraud. So essentially you are creating multiple experts and then for actually making a decision you pass the decision through three experts and then you take the majority vote. Sometime back I wrote a LinkedIn post on this. The random forest is like a third empire in cricket. When they want to make a decision they look at front view side view they have different cameras. So this decision tree individual decision trees that we are creating are like this different views. Okay. So you are 3:23:08 looking at a situation from different angles and then you're using this method of wisdom of crowds to make a final decision. Now see front view can only see certain things. It can't see things from the side. So same way in our decision tree we took only few columns. Correct? So now we are looking at some narrow view of the situation. Same way in the second decision tree we would have used a different set of column. So you're looking at a side view and then then you combine all these views to make a decision. So just remember this cricket analogy. Okay. When you are a third empire you're making a decision you look at things from different angles and then you make an aggregate decision and random forest is exactly doing that. Check which data set we are using to train our random forest model. Titanic data set. Your statistical machine learning journey will not be over if you have not written at least one notebook on Titanic data set. Titanic is a very 3:24:09 popular data set in the data science world. And what it does is it predicts the probability of survival based on different factors. So here's a passenger ID. Here is whether they survived or not. You know Titanic when the ship crashed when the person survived or not based on the ticket class. So first class, second class, third class, name of course doesn't matter, sex, okay, gender, then the all this number of parents, number of siblings, ticket and so on. So I have this data set here. Okay. Passenger ID survived P class. This is the target variable folks. All right. all of this okay fair cabin embark and so on and what I have done is I have loaded this data into my data frame it looks something like this okay once again survived is a target variable based on person's class their fair 3:25:12 whether they embarked or not we want to calculate the probability of their survival right So these are all the columns. You can read through the notebook to get an idea. For cleaning and pre-processing, we are going to drop all these columns. See passenger ID, passenger name, these columns do not contribute to your uh target. Correct? So we we are just dropping all of this. We are also filling the missing values in age and embark. So we have written some pandas code. We have done this before. So I'm not doing again step by step, but you can read through it. Okay, see we are filling NA values. We are getting dummies and handling all these categorical variables. Once the data set is ready, we will do train test split in our usual fashion. Then we'll train a decision tree classifier. I want to compare decision tree performance with random forest. That's why I train decision tree classifier and I got the 3:26:14 test accuracy of 76%. Train accuracy is 98%. See here I am scoring on X train and Y train. Train accuracy will be higher because it's like you are giving students the exact same questions that they prepared while preparing for the exam. In the exam you are giving the exact same test. Of course they are going to perform better. Yeah, they'll get 98%. But on test which is a new set of questions that they have not seen the accuracy is 76%. And this is precision and recall. Now let's now let's use random forest. So you import it from skarn assemble. It is called ensemble because it is doing the voting. It is training multiple trees and doing the voting. And whenever you do that, whenever you're using uh wisdom of crowds and multiple experts and doing the majority voting etc. it is called ensemble. Okay. So this is an ensemble technique. And then we will use random forest classifier. N estimators is actually number of trees. Okay. This is 3:27:16 100 by default. So let's check the documentation. You can say scale on random forest. And if you look at the estimators it is 100 number of trees. Okay. So let's see uh the number of trees in the forest. Okay. So they used to have 10 before. Now they have changed it to 100. Okay. And random state. Now we'll do forest dot fit x train y train train the model and then print the accuracy both on train as well as test and then the classification report the classification report straightforward we have done this like 100 times by now so once again not writing code from scratch here you can see it is performing little better than tree single tree got used 76% % accuracy. These are the precision recall numbers here. Precision recall 3:28:18 numbers improved as well as accuracy improved. Okay, number of estimators were 100. Now you can try it on 10 trees also. Okay, 10. Okay, 10 is pretty much same. Okay, 50. You can try like different numbers or you can try like a for loop. So I want to try with 10 and 100 and I want to calculate the accuracy. 10 trees. Taste accuracy is this 100 is pretty much similar but you can play with it for your data set. Different number of trees will give you different performance. This is something you experiment and find out the best value for an estimators. All right. So that's how it's pretty straightforward. That's how random forest works. You can check the feature importances as well. Same way as we did it with a decision tree. So here what predicted the survival of on Titanic were fair. So let's say if they were paying a higher fair which means they were in a higher class they were rescued earlier. Then uh 3:29:21 sex also played a role. Okay. So I think female they survived better if I'm not wrong. But sex male is a categorical variable right? Sex male sex female sex female we removed to get rid of the dummy variable trap. uh then age, class, all these factors played a role. All right. So one thing we have to remember is when you have decision tree it can potentially overfit because you are making specific decisions all the way until the last node. But in case of random forest since we are using different estimators it can generalize better. So random forest will not overfit. I mean it can still overfit but compared to decision tree probability of overfitting is less in random forest and that is the reason in the industry random forest is one of the popular algorithms. Then comes exercise in the exercise we have wine data set again 3:30:21 this is one of the standard data sets from skarn library and you're trying to find out the wine like target wine right 0 1 or two. So go through this uh and complete the exercise as usual. Tell me what is the accuracy you are getting in the comments below. So far we have been doing train test split and training our model and testing it. So let's say when you have 100 rows and when you do train test split let's say for test let's say we are getting 20% of rows right. So those 20% of rows can be anything and remaining will be the training rows and you're training a model. Let's say this is a model. You train it on this uh train rows and then you test it on the test rows. Okay. So on the test rows you will test it you will get a score. But one issue with this approach is that you are taking just a sample of data to scoring the 3:31:23 model. Let's say on this kind of split the model scores say 92%. But if a different data scientist splits the rows in a different way. All right? So let's say here your random state value is 42 and this is the split but a different person might supply random state as 10 or something else and now you're getting a different kind of splits altogether right so your taste rows are something very different than what you had before okay and in this case when you uh score your model. Okay, when you score your model, it only gets let's say 82%. So the this scoring is kind of random if you think about it. It depends on the kind of split that you will have in data. So to tackle this 3:32:24 problem, we use something called kfold cross validation. Okay. So now we'll talk about kfold cross validation and I will explain how this works. What I would like to do is I want you to pause this video and use your brain, use your original thinking to tell me how we can solve this problem. It is a very common sense approach folks. You can use that approach to solve this problem. Okay? So, pause this video and think about it. All right. I'm assuming you paused this video, you did your thinking and now we are going to discuss what is kful cross validation. So one approach we can use here is let's say you take row number 1 to 20 for example okay and use this as your test data okay so this is your test data and the remaining one is your train data so row number 20 200 is train and this is test 3:33:28 and using this you will train let's say model number one so you're supplying all this for training. Then for evaluation, you're supplying your test rows and you are getting score number one. In the second round, what you do is you take those same 100 rows and then use row number 20 to 40 as your test data and remaining rows as your train data. Okay? So you're supplying all this train data to a new model. You don't use this model by the way. You create a new model. So let's say if you're using support vector machine, okay, you create an instance of support vector machine, you train this and then you create another instance of support vector machine using the same parameters and you train this model number two on this green train data. Once it is trained, now you test it on this test data, you get score number two. So you repeat this process. So this 3:34:30 is first trial, second trial. You repeat this process all the way till the fifth trial where you will have model number five on which see you will train it on row number 1 to 80 and then you test it on this test data and then you score that as well. And now you take the average of score number one to five. So you're going to take the average of all these scores. Okay. So you take averages average of all these scores and whatever final average you get is the ultimate performance of your support vector machine and let's say for the same data set you do this trial for logistic regression also random forest. You want to find out which model should work best for my data. So you can do 3:35:32 this kind of trial and let's say average score for random forest comes out to the to be the best. Then you pick that model as your final model. Okay. So this is called kf fold cross validation. Number of folds here are five. Essentially you are dividing your data set into five folds and using one fold for test remaining fold for train and you're repeating that process iteratively you're getting individual scores getting an average and then measuring the performance. It is like you have a student and you are taking one test then you are using different set of questions you are taking another test you're using different set of question taking third fourth five test and then you are averaging students performance in all those five exams. Now you get a good understanding of how a student is in a given subject. One thing to note here is that in the actual production setting what we do is when we have our entire 3:36:32 data set let's say you have 100 rows right so let's say you have for example 1,000 rows okay so you will do first train and test split so let's say this is your X train Y train okay you do train and test split and this train and test split is kind of random so Now in your train data set you get let's say 80% which is 800 rows and then in your test data set you get 200 rows correct then on this 800 you will run your cross validation. So what you will do is now see whatever 100 rows example I gave is actually this. So on this you will do let's say five folds. So you create five folds 1 2 3 4 5 and you run model one model two model five 3:37:34 all the way and you get score one score two all the way till score five and then you take the average score average score now you're saying okay this model is great okay so this is the average score let's You tried random forest, logistic regression, all of that and you are saying okay my random forest is performing the best. Now to get an idea on how this model is going to perform in production, you have to still test it with this test data set. Okay. So here the train test split you are doing is this is not called test. This is called validation data set. So there is a different term. This is a validation test and this is the actual test on which you want to run this performance evolution. The reason is see these scores that you're getting 3:38:35 are from the train data set and they may be biased because see M1, M2, M3 you're using like different type of splits but the standard practice is that before you deploy your model to production you want to test it on unseen data. the data that it has not seen at all. That's why we split the test data set in the beginning and we kept it uh separate. We did not touch this data. We did our cross validation on this 800 rows. Okay. So, so there is this term right? So, there is this train data set. There is this test data set. But there is this validation data set. You will hear these three terms. So train is this green this red which you are performing during cross validation is actually validation data set that's why it's called cross validation and the test data set is something you are not seen at all before you deploy the model you will run your evolution on that test data set and that 3:39:36 will give you some number. So of course you use average score as well but in addition you use this test code to get the true understanding of how it will perform on unseen data. Let's write our code. Now we are using same hard disase data set. We loaded it here in data frame. We trained SVM and random forest models. SVM got 66% score. Random forest got 85% score. Okay. Now we will use the kffold cross validation. So the way you do that is from skarn model selection import kfold and then in kfold there is this argument. So how many splits you want? We talked about five fold cross validation in the theory. We will use three folds. Okay just just to keep things simple. Usually people use five uh folds but you know we can use uh 3:40:37 three as well. This is just a learning exercise. All right, we just created three-fold um object. Now we will use it to split this particular data. And the way you do that is for train index test index in KF do.split split okay data and then you print train index test index. So what this is doing is in the first trial see the test data set was 012 and 3 to 8 was your train. Second trial 3 4 5 was your test remaining train and then 6 7 8. Okay. So this way it will do the train test split. If you have four let's say something you can't divide equally then it will just assign a few more elements in the first four or 3:41:38 whatever but but it will handle it. Okay it's not going to fail. So just look at this indexes it will become very clear how this works. Now we can use this split on our actual data. So what we'll do is for train index index and KF.split split we have X and Y you know our actual data set see X and Y on the hard this is we are going to use this split and then to get X train X traced etc you can use X doilock soil is a method in pandas where you supply the index and it will return those many elements so now we got X train and X test and let's say I want to now try support vector machine and random forest on this data set. I want to see which model performs better. So as a first attempt I will use support vector machine. I will train it and then I will score it. We are doing multiple tries right. So I 3:42:38 will store the score in a Python list. Okay. So we will have random forest scores as well. And then let's create random forest. Here we have random forest. Number of trees, decision trees are 40. Same thing I'm training the model on same data set. Then I'm scoring it. Now here we are using the score which is accuracy. You can use precision and recall as well. Okay, we are keeping things simple. Just using the score here. Let's run it. So now what it did is it has created three split. So the for loop ran three times. The first time it uh tried it on the first fold then second fold and then third fold and then it stored all these scores. So now when you look at these scores see this is the score of SVM and obviously random forest is 3:43:39 performing better. Now to get average score you can use np dot mean. Okay you can just print it and same thing for here you can just say print np dot mean rf scores and see now you can use this one number what is the accuracy of my sim model 62%. 81% is random forest. So I will obviously go ahead with random forest uh algorithm. Okay. So this is such a beautiful way to get your model trained on diverse set of test and perform uh the average. It's like a student you are taking multiple test here we took three test for a student and we got the performance average score was 62% for SVM 81% for random forest. Now there has to be a better method for doing all of this and there that method is called 3:44:40 cross val score. So whatever we did manually while running the for loop you will do the same exit for loop in one line. Okay. So let's do that. So we are importing cross valore function and in that function you will pass your model which is support vector machine. you pass your X and Y and you pass your cross validation object. So here KF right the same three-fold uh object we are passing here and when you run it see you got the exact score 61 65 60 61 65 60 and same thing for random forest classifier. Okay, XY this and you got 83 see 83 80 and 79. So this is very good in real life you will use this function directly. You're not going to write this for loop. I just wanted to write it so that you get the understanding of internals of it. Now other than um kfold 3:45:45 there is another method for called stratified kfold as well. So let me explain what is stratified k-fold. So let's say you are using co- kffold cross validation and in that you will always take first 20 rows in your array 0 to 20 index right fixed index you will take for the testing. Okay. And for the training you will use 20 to row number 100. But what if for our heart disease prediction all these 0 to 20 rows are the patient with heart disease one and all these patients are zero. See this can happen or let's say there is some kind of imbalance. Let's say out of this 20 18 rows are people with heart disase one and here in 20 to 100 most of the 90% of the people are without heart disease. Now your model is not getting trained properly because there is some kind of class imbalance and the classes are not 3:46:47 represented equally in train and test data set. So this is a problem to solve this problem you use something called stratified kfold in that it will not use the fixed set of rows. Okay. So it's not going to use fixed set of rows like 0 to 20 etc. What it will do instead is it will pick 20 rows at random. Not random but it will use some kind of class presentation and then it will pick 80 rows in such a way that there are equal number of ones and zeros. Okay. So ones and zeros. So let's say here uh people having heart disease is let's say 15% 15% has heart disease and 85% doesn't have heart disease. It will be same kind of distribution here. So here in a train data set also 15% will have heart 3:47:49 disease 85% will not have heart disease. So that way when you train your model so now you train model number one on this. Okay. and then you get the score. Then you do the same process again and you use like different types of different row indexes so that you get the equal same representation like this you train it on model two get score two and so on. So this stratifier kfold is more popular because it can represent the classes in a equal way and you will get a better quality model. So you'll do same thing as kfold instead of kfold you are importing stratified kfold and let's say you are doing uh threefold. Now let's say you have this kind of data set right you have numbers and you have labels also. So let's say this is heart disease right the last four people have heart disease the first four do not. Now if 3:48:51 you had done kfold it would just go fix index but in stratified let's see what it will do when you do this you see in the first fold uh the test data set it got 01 and five. So two patient without heart disease one patient with heart disease. Okay. So 1/3 is hard disase, 2/3 is no no hard disase. And in the training as you can see it is 2 3 4. Okay. And then 6 7 8 6 7 8 2 3 4 and 6 7 8. So it's kind of equal. Okay. 0 1 4. So 0 1 and 4 and 5 7 8. So 5 78. So here is doing the equal. So equal number of people with heart disease without hard dis. Okay. So 3:49:53 it will do something like this. And then uh in this crosswell score uh method by the way. So let me do this. See in the crosswell score method if you pass CVS3 so I want three folds by default. Then by default it is using stratified kfold. You just specify how many folds you want. You don't need to pass this SKF. I mean even if you pass SKF it's going to work. Okay. Okay. So let me just do this S KF. See this is going to work. Okay. But what I want to say here is statified kfold is default. So even if you just simply specify three-fold see you are getting the same result. All right. So just a recap the default train test split is noisy. It can get you bias 3:50:55 results. It is better you take multiple exams for a student and you can do that using k-fold cross validation. You use stratified kfold cross validation whenever you want to represent the classes in a equal manner and this is what it gets used in production by default. And folks as usual we have an exercise folder. Please check it out. Please finish the exercise. Let's say now we are training a model on our standard iris flower data set where there are features and target label. The first question that comes to our mind is which model should I pick from. There are so many to choose from. Now there are certain guidelines on the models. But the best approach you can use is just train the model and see which one is performing better. And we did that using cross validation score function in the last section. Okay. Now let's say cross validation score is giving SVM to 3:51:57 be the best model. But then the next question arises which is which parameters are the best. See here I'm saying kernel RBF C 30 gamma auto. But RBF can be replaced by linear or poly C is an integer. This value can be 30 40 10 5 6 7,000. Which which value is the best? Gamma again it's a float right so hyperparameter tuning is the process of finding the best value of parameters for a given model and that is what we are now going to learn we will use something called grid search CV to find out the best parameters for the model folks here I have loaded the Iris flower data set okay this is something we have seen before so pretty familiar thing Now I want to find out which parameters are going to work the best for support vector machine. Let's say I tried logistic regression, random forest [clears throat] everything and I 3:52:58 feel that SVM is the best model. But within SVM there are parameters kernel C etc. How do I find the best parameters? Well, one approach is you use crosswell score method and then you try it on different parameters. So here is linear here I'm trying it on RBF and then see CV is equal to 5. Okay. So then I can change uh gamma or let's say C to be 20. Okay. So RBF and then same RBF but C value is 20. So I can do that but this approach is very time consuming right? So it's like manually it's tedious process. So one thing we can do is we can run a for loop. That's pretty straightforward, right? So, we create all the possible values of kernel and C that you want to try into a pandas list and then we run this kind of a for loop. See for loop. It's a nested for loop. So, first for loop is on kernel which is 3:53:59 RBF and linear. Second one is on the value of C. And every time you pass kernel value and C value and you put all those scores into the average scores and then you print it. See here by looking at it I can say RBF with C value 1 and 10 is giving you the best performance. Remaining is 96. Okay, this one is also giving you best performance. Okay, we are using a very small uh data set but if you have a real data set with realistic values, huge data volume, then you will see the differences into the performance here. Okay. Now folks, uh this is also tedious. See if we have to do this I mean skarn folks whoever created that library would have thought about uh something better something which is like you call some function and pass values of kernel and see and it should give you the the scoring okay so 3:55:01 that method is called grid search CV okay and the way you call this method is you will say grid search CV you will create your model here and then you are passing your parameter grid. See this is called parameter grid. You want to try these values of C these values of kernel and it will run the permutation combination. So it will take RBF then take value of C as 1 10 20 train models then linear again 1 10 20. So it will do six iterations total and every iteration is it will use u the five-fold validation. Okay. So on CLF you can simply call fit method and then you can check the CV result cross validation results and see this is what you get. Okay. Now this is hard to read. So how about we uh store this CV results into 3:56:04 panda's data frame. So import pandas as pd then pd dot dataf frame clf dot cv results and then df. Now this is better right. So here what you can see is the uh let me just make it little smaller parameter value. See parameter one rbf c is one kernel is this. Okay. And then the splits the the scoring of every split. See total five split, right? 0 to four. And the mean test score. I think this is probably the most important value for you. Mean test score. So here you can see the performance. So I already see 98 is a best performance. And see here you see the value of kernel and see both. If you want to just see a subset of columns which are something more useful then you 3:57:05 get this. See, so I mean if you try it on a bigger grid, you might have too many results and then you can take the maximum value of this and take these parameters. So now you should train your model with C is equal to 1 and kernel is equal to RBF because that is giving you the best performance. So this entire process of finding the best parameters for model is called hyperparameter tuning. It offers lot of convenient methods. See you have best parameters here and you also have the best score and not only that you have something called best estimator. So this is the model that you can take you know this is the model you can take and on this model you can call your predict function and you can save this model to disk and and import it uh to use in your program. Now you might have thought about this problem which is if I have a huge data 3:58:05 set let's say million records and if my parameter grid is pretty big let's say permutation and combination for my parameters is 20 now am I going to run the training on million records 20 times and it is not 20 folks it is doing cross validation as well so in each iteration it is doing five different runs okay so 20 into 5 is 100 you're training 100 models on million records that's too much computation. So to tackle the computation problems scikitlearn comes with another variant of grid search CV which is called randomize search CV. Now this is exactly same as grid search CV. The difference is you can specify uh the iteration. So iteration two means it will randomly pick two permutation and combination of C and kernel value and then it will run only those iterations. So let me just do it here. 3:59:11 So see it ran only two permutation combination one C is equal to 1 RBF and linear. If you run it again uh let's run it again. I think it's just or let me you know because of random state let me do it like five. So when I do five it will see 3 into 2 is total 10 combination 3 into 2 is sorry it's six combination total. Okay out of six it is going to take only five. So let's say if you have something like this now how many combinations are there? 10 right five into 10 so out of 10 it will pick only five see four TRB so it is random so every time it will pick the random I mean here because of random state maybe but see every time it is picking the random if you want to control the reproducibility you give a random state 4:00:12 and then remaining things are same it will still give you the best performance among the all so this lets you save on your compute time. Sometimes you don't want to wait, you don't want to waste too many computation resources on running through all the permutation and combination and this randomized search CV will help you with that objective. Now, what if I want to try different models and for each model I want to try different parameters? Well, that can be done. So, for this you can define this kind of dictionary. Okay, SVM is my model and I want to try these parameters. So random forest is my model. I want to try this parameters. Logistic regression. I want to try these parameters. And then you can run this kind of a for loop. So model name and model parameters. For each of them you run grid search CV. So here is your model. Here is your parameter grid. Okay. CV file whatever. 4:01:14 And then you fit and then you put your best score and best parameters into this kind of a array okay or a list and then you put that into a data frame. Okay, this was not executed. So let me execute that. See now you get this kind of a base score. So SVM for SVM the base score is this and these are the parameters for random forest base score is this and these are the parameters. So for random forest by the way it tried the tree value of 1 5 and 10 and out of that it found 10 to be giving the best performance. So this allows you to try different models and for each model you are trying different parameters. All right so in the production setting in the industry this is what data scientists would do. They would try different models and then different parameters they will come up with the best model best performing model and that is the one they will deploy in production. Let's now take a look at 4:02:16 anse symbol learning or wisdom of crowds. We have already seen one technique which is random forest. Random forest is essentially ensemble learning technique. So in ensemble learning you have multiple models. So you have model one, model two, model 3 like that you have trained them on different data sets. Okay. So in in case of random forest we sample rows and we take subset of columns and we train these models and then we take the majority watt. Okay. So we take majority what majority what and then there is a final decision. So this is what happens in case of classification. If it is regression here you will take average. So let's say you're trying to predict home price. You will have three models. 4:03:18 They will predict the home price. You take an average. It's like if you want people to estimate how many balls are there in a container, you ask 10 people, they will come up with 10 different estimates and you average it. The probability that average will be more closer to reality is higher than individual person guessing. That is the fundamental principle here. You have multiple experts, multiple models. you're taking majority what in case of classification or average in case of regression. Now ensible learning has multiple techniques. Okay. So there is something called bagging. Random forest is a bagging technique. Then there is another thing called boosting. Okay. So random forest comes under bagging. So random forest and in boosting we have something called XG 4:04:18 boost. These two models are very very popular in the industry when we work on client projects at at these are the two popular methods that we use. So in bagging essentially you have a data set. So let's say there is a data set which has 100 rows. 100 rows and you do bootstrapping. Bootstrapping uh is called random sampling with replacement. So this is a concept in statistics where let's say to create 100 rows for our model one. This is random forest essentially. What you do is you sample this into a bucket. So let's say you got row number one. Then you put one back. So next time when you sample it, let's say you get number five. Okay, you put five back. Next time when you sample it, it it is likely you you get one back. So there there might be duplicates. So this way you get 100 4:05:21 samples. You train first model, model one. Same way you do again random sampling, get 100 samples, there will be duplicates, train model two and so on. In case of random forest, we know that we also sample on columns. Now, what is the benefit of uh bootstrapping? The benefit is that each of these models will see the subset of data. So, just like our third empire example, they will have a good view on subset of data. Okay. Uh so let's say if there is an outlier there are like few outliers those outlier may get filtered here because you're doing sampling in the sampling outlier may not get picked okay in that case the quality of M1 will be higher so outlier gets filtered and also each of these models are getting subset of data that way uh 4:06:21 these models will have a view on like different sides or different dimensions of your data and when you average the results out you overall get better results. So that is bagging. The second one is boosting. So see in bagging you are training the models in parallel multiple models in parallel. In boosting it's reverse it is sequential. So in boosting let's say you have a decision tree and decision tree with only one level is called stump. Okay. So you train a weak classifier. This is also called weak classifier or weak model. You train the model. You pass a data and you check the output and on the data points or the records where it made mistakes, you train another model. Let's say another decision tree so that this gets better at the mistakes that the previous guy made. Then you train another one. Okay? So you do 4:07:23 something like this. So you're kind of boosting the you know prediction capability of your model in each stages. So that is your boosting uh method and the popular technique uh that we have is XG boost. Now folks in the interest of time I'm not going to go through coding etc. But if you want to know the math in detail, if you want to know all the techniques such as ADA boost, regression walkth through regression math, okay, all of that, then refer to this machine learning course that we have. Okay, we we have the boot camp as well. So the boot camp machine learning course is part of the boot camp and in this boot camp we have explained all the math everything in super detail fashion. We have all the ROC everything like you just look at the curriculum it is very much uh in detail and uh the projects 4:08:23 that we have are real life projects. So this credit risk modeling project covers uh the complex statistical methods such as weight of evidence, information value and this project uh is something that the um you know some of the banks some of the lending institutes uses actually. So this is based on the real project that one of the big companies in India has uh built their model on. Okay. Okay, so it's a very real uh project related to NBFC domain. Um we are doing model fine-tuning using Optuna KS statistics. K statistics kind of something that gets used often in uh the industry setting. So we have covered all of that. So please check out the curriculum if you have interest. So far we have discussed only supervised learning. Now let's talk about unsupervised learning. Let's just quickly review what we 4:09:25 covered in supervised learning. In supervised learning, you essentially have labels. By label, what I mean is when you are doing fraud transaction detection, you have whether the transaction is fraud or not. So this is the label. If you're doing spam email detection, you know whether email is spam or not. Okay. So in your training data set, you have the labels which is the categories. If it is classification problem, if it is a regression problem, you have your target variable which is the price of the home. This is supervised learning. Now let's say you want to do supervised learning for a kid who is playing in your home. You might have a kid at home and you know what happens when you give them a lot of toys on the floor. So there's like a big pile of toys on the floor. It is evening time. You're going to sleep and you want your kid to learn discipline. So you'll say that, "Hey beta, please classify these toys into 4:10:27 doll, car, and truck." See? So you're giving them these three buckets. One for doll, one for car, one for truck. and you're asking them to put the uh toys in the appropriate bucket and they will do that. So this is once again a supervised machine learning because every toy has a label now a car, doll, truck and they're classifying as per the label. But let's say you don't have these label buckets. You give the kid two empty buckets and you ask the kid to categorize the toys into two different buckets. You're asking uh let's say this girl name is Radha. You're saying hey Radha please put this bucket toys into this buckets and you classify as per your own thinking. So Radha will 4:11:28 classify them as color. So she will put all the red color toys into one bucket, all blue toys into another bucket or she may classify them by objects, you know, all cars in one, all trucks in one. She can just detect the patterns in the given data and she will classify them. Okay, this is unsupervised machine learning. Here you don't have labels. All you have is data and you look into the data and you try to find the patterns in the data. Okay? Red color, blue color, big, small, cars versus trucks and so on. In my company at we have a lot of documents. So we get all these documents, legal documents, client requirement documents, invoices in one drive. So our one drive is terabytes of data and people create folders to categorize it. But you know folders get messy at some point. So in this kind of cases you can use classification. You 4:12:30 can look at the content of the document and using the content you can classify whether it's a legal document, whether it's invoice and so on. Okay. So essentially you have all these data points and you are trying to cluster them. So this is called clustering by the way. you are clustering the data points together based on the characteristic or based on the patterns that you find in those data points. When I was at Bloomberg, we were using this clustering method for actually outlier detection. So we had all this earning estimate data points. It was a consensus and you just cluster them and you will find this one data point which is outside of all these clusters. So we were using clustering technique to detect outliers. So clustering can be used for multiple purposes. Outlier detection was one of the side effects or one of the objective of doing 4:13:33 clustering. Okay. And at that time we used this algorithm called DB scan. Anyway to summarize unsupervised learning what we do is we provide an unlabelled data set to ML program machine learning program and then it learns to identify patterns and structures in data without any explicit guidance like our case of Radha. You did not tell Radha to put red toys into one bucket and blue into another one. All you said is identify patterns in your data and cluster them as per your own thinking. Okay. So in unsupervised there are multiple algorithms. DB scan, K means, hierarchal, clustering. These are different types of algorithms. In supervised, we have already seen random forest, exeost, linear regression, logistic, SVM. These are all supervised learning algorithms. Okay. So in terms of algorithms you can or in terms of problem statement you can classify 4:14:34 machine learning problems into supervised and unsupervised. In supervised you can have further classification regression and classification. Okay. So you can remember this picture. This picture gives you the holistic understanding of the task types and the algorithm landscape in the world of machine learning. Let's now look at the popular unsupervised machine learning technique called Kins. Let's say you have all these data points and you want to form clusters. Just by visually looking at it, you can think about these two clusters, right? Like purple and green. Let's see how we can form these clusters. So you start with these data points and then there is this hyperparameter called K. So K means has this thing called K which means number of clusters. Now see if you think about it number of clusters can be two it can be three it can be four as well right? 4:15:34 So see if you look at it these two are the clusters but then you can have different type of clusters too right you can have one 2 3 4 it's a hyperparameter folks it is how you look at it so you always start with k which is some assumption on how many clusters are there in your given data so let's say we start with some number let's say we start with two and then we will put the centrids. Centr means the center of the cluster. So we will uh put these two stars which are the centers of these two clusters randomly. Okay. And then you kind of classify the data points into these two clusters. So you connect these two uh centrids. Okay. You draw perpendicular line. Anything on the left hand side is purple cluster. right hand side is green cluster. Now you 4:16:36 compute distance of every point from centr and cluster them accordingly. Okay. So now what you will do is you will adjust the cent so that they become center of gravity for given center. So we form these two clusters. Now you will move this centrid so that they form the center of gravity. Okay. So you have this particular cluster purple cluster here and the center of gravity will be somewhere here. Similarly you have this green cluster here. Center of gravity will be somewhere here. Then you draw this line. Okay? You join these two. Draw a perpendicular line. Anything on the left hand side is purple. So these three will now become purple cluster remaining are green. Okay. So just visualize what happens to these three data points. They become green. You again reccluster every point. 4:17:39 You you repeat the process. Now for these data points you again find center of gravity. So it will be this. So you move this center here. And then for these guys you find center of gravity and you move. Okay? And again you join them and you draw a perpendicular line. Now see this grain will now come to purple. Look what happens to this. See? Okay. You keep on repeating this process unless the data points stop changing the cluster. When data points stop changing the cluster that's when you know that you have formed these two clusters. Okay. So this is a very simple algorithm. Now how do you determine the value of K? See we started with two but it can be four, it can be five, it can be anything. So there is this one technique that you use to determine the optimal number of clusters. What you do is you start with K is equal to 2 3 4 etc. You 4:18:42 start with K equal to 2 and you calculate the sum of squared errors. So sum of squared error is essentially this. See step by step what you do is you start with some number of K. So let's say K2. Then you follow the process that we already talked about and you form these two clusters. Okay. And then you calculate sum of squared errors which is basically the distance uh of center and this. Okay. So square distance basically. So you take this distance, you square it and you sum all the squared distances. So this is sum of squared error one. For cluster two, you compute sum of squared error 2. Okay? And then that becomes the total error. So total error for k = 2. Let's say there is some error. Now you uh assume k's value as three and you 4:19:45 form the clusters by following the same process and you compute the total error for k= 3 then k= 4 compute the total error k= 5 compute the total error okay so you keep on computing this total error and then you plot a chart so here for k= 2 I got this much error k= 3 I got this much error. See this is sum of squared error. Total error. K equal to 4. I got this error. Once you have this chart, you use elbow technique. Elbow is like you know when you have a human elbow uh th this shape looks like an elbow. So you look for a point where the line is becoming a flat. So this is the point where the change is not going to be abrupt. See here the change was abrupt but from this point it started becoming a flat. So this is the elbow point. So this is K equal to 3 actually. Okay. So K equal to 3. So then you will say K= 3 is my optimal number. Clustering people based on their income 4:20:46 is a very popular clustering use case. So I have this data set where I have people's age and their incomes. Okay. And I'm going to now use Kins to perform that clustering. So I have loaded the data into panda's data frame and then I will plot a scatter plot. Okay. So scatter plot looks something like this. So clearly uh here it looks like we have this 1 2 three clusters. Okay. So let's run a k means algorithm. The way you do that is you will import kins class from skarn.cluster and in kins you will specify number of clusters. So let's say number of cluster is three and there is a random state of course to start with that you know certain state. So you have kins object here and on k min you will call fit and 4:21:48 predict. Okay. So fit and predict will take your data frame. So in your data frame you don't want to pass all the columns. You want to pass only age and income because see name is not useful right? You want to cluster based on age and income and here you get y predicted. Okay. So let's see what value you get in y predicted. See for all these data points it has given some kind of cluster number. Okay. This is one, this is two and so on. Now visualizing it this way is not that intuitive. So I'm going to create a new column in my data frame. Okay. We'll call it a cluster. So see now for each of these rows, each of these people we have their category basically their cluster number and you can also see the cluster centers. You know the center of gravity that we talked about those stars. So these are the cluster centers. Now once again this 4:22:50 is not very useful visualization. I would like to view it on a scatter plot. So let's use metplot lib. Okay. Here uh we are creating our three data frames based on clusters 0 1 and 2. Okay. See these are the values 0 1 and two are the three clusters. So I'm creating three data frames and I am uh plotting them on a same scatter plot. And see I got one cluster which is green and this is the center of gravity. Another cluster is red and center of gravity and then black center of gravity. It did not do a well job actually if you think about it. I was expecting this to be one cluster this second and this third. Okay. And the reason is uh scaling plays an important role here because see your age is in range 0 to 100 whereas income is in thousand multiple thousands. So these are not scaled properly. So let's scale 4:23:53 them properly using something called minmax scaler. So there is standard scaler, there is minmax scaler. In minmax scalar you take minimum and maximum value and you kind of uh divide it divide every value with a difference between minimum and maximum. So see this is the formula. Basically in your data points you take x max minus x min and divide every data point. every data point is this x minus x mean and then you will get your scaling. Okay. So you will do scalar fit on both the columns on income as well as age and then you look at df dot head. See now my age and income columns are scaled. If you do df do describe see my min value is zero max is one here min value is zero max is one so you are scaling it into a range of 0 4:24:56 to one and now if you do clustering it's probably going to work better so same code I'm using here number of cluster three you are saying fit and predict so fit and predict will do the clustering and it will also pred predict. Okay, what is the cluster? And once you have this, you will put this or let me just put y predicted. See y predicted. If you compare it with the previous code, it looks now different. Okay, so you have this and you will also print cluster centers. But the best idea you will get when you do matt plot lib plot. Okay, so same plot I'm pointing it here. And now look at this. Look at this beautiful cluster folks. This is black cluster. This purple dot is the center of gravity for that cluster. And same thing here you know we got like this cluster defined beautifully. Now now let's do an elbow plot. Okay. So for elbow plot I 4:25:58 will try k from 1 to 10. So let's take value of k 1 to 10. So this is 1 to 10 and calculate the uh sum of squared errors step by step. So you will run a for loop. So you'll say 4 k in k r range and then compute the clusters. Okay. So these are my clusters and number of clusters is like 1 2 3 and so on. And then you will call fit. See previously we were calling fit and predict because we are predicting the cluster as well. Now we are just fitting uh and we are just computing the cluster internally and then the error can be accessed by using this function km inertia. It's it's not a function it's a property and you will collect all the errors into this python list. Once this errors are computed let me just print it. Okay. 4:26:58 Okay. So these are the errors. Then I mean by looking at this itself you can say see it's a sudden drop stopped here. So it is K equal to 3. But we'll just plot it on a chart. And here see you you see this elbow folks. So again K is equal to 3. So K is equal to 3 is my optimal clusters. All right that's it folks. That is what we have for K minutorial. Once again we have an exercise folder. So please look at it. I know you all are sincere students. You will uh work on this exercise and you will post your results into comment box below. You have to do that promise to me. Okay? You're learning this thing for free. This is my fees folks. Work on exercises and post the results of your exercises into the uh video comments below. Maybe you can create a GitHub repository, post your code there and share the link of that repository or 4:27:58 just simply tell me how many clusters you are getting. If you're working on classification problem, what kind of accuracy you are getting and so on. Now is the time to make use of everything that you have learned so far to build a capstone project. Let's say you are working as an AI engineer for an e-commerce company called Amazon and the CEO of this company is Jeff Kurana. There is one thing that is worrying Mr. Jeff Kurana and that is product returns. Whenever customer is returning the products it is costing company twice. First person buys it. So you have to ship all the affords logistic that goes in place packing etc. Then the person or the customer returns it back. So now there is a reverse logistic. Okay. You have to provide a QR code, customer ships it, you have to handle it, repack it etc. So this is a biggest worry for 4:29:00 Mrs. Jeff Kurana and he's wondering we are living in this AI era. Can we predict a return before the order ships using AI? Think about it. We can use machine learning to predict uh if a customer will return that particular product even before you're shipping the product. Okay. So he's kicking off this product return prediction project. He has assembled a team of AI engineers. They are going to work on it. You are that AI engineer. Okay, Jeff Kurana has asked you to build that project. So you are going to build this project. Here is the data set that you'll be working with. So here you have order ID, product category, subcategory, product price. At the end you have whether this product was returned or not. Okay. So now you have total 12,000 past orders. You want to use this to train your model. Now let's pause for a moment and think what 4:30:01 kind of factors can affect the product return. Fragile item is one column. If the item is fragile, is it likely that the customers will return it? Think about it. Pause this video and think about it for a moment. Well, the answer can be yes because when the product gets shipped during the shipping, if fragile item breaks even a little, then customer will likely return it, right? So the likelihood of fragile item breaking is higher and therefore likelihood of that being returned is higher. Warranty available if the product doesn't have warranty available customer can likely return it. If it has has a warranty then customer has that assurance that if there is a problem I can use warranty to tackle that. Okay. Product return rate. So in the past if let's say 17% or let's say 30% of products were returned if the product return rate is higher on past orders than the new order also it's 4:31:01 likely that it will be returned then category for the entire category like health beauty sports the entire category what is the return rate then defect rate okay in the past this kind of products how many products had a defect seller rating seller return rate fulfillment type okay payment method. Say if customer is doing a payment on delivery most likely they may return it but if they have committed if they have paid using credit card then then return requires some process to it right it's there is some barrier then quantity shipping distance some of these factors may not matter by the way we'll see in the notebook then shipping distance so I recently ordered a standing desk for my home in India when I was in India And what happened is my place the the place where I live my parents' home is in village and it took long time for order to come from Mumbai all the way to 4:32:03 village. So shipping distance was higher and guess what I actually initiated return for it because then I had to come back to US my time was over in India and I couldn't make good use of that desk. So the likelihood of that returning becomes higher when shipping distance is higher and delayed delivery. So I had exact same problem. This is a real problem that happened. Uh the order was delivered because it it is in the village. If it was in city, it would have delivered fast. So if there is a delay in delivery, there is likelihood that the customers might have time sensitive needs. Let's say someone is ordering some product for the birthday party and if the product doesn't arrive before the birthday party. Birthday party is over. Then the product arrives what's the point? They will return it. Right? So any kind of delivery delay can cause uh the the return can affect that return. uh wish list then product page 4:33:04 review customer support calls let's say before ordering or during the process when I was you know about to get my standing desk I had to call customer support because there was a delay there was uncertaintity more number of calls means likelihood of return will increase chat interactions so you're going to customer support chatboard having lot of interaction means it might be a return and this is The final column this is the target column whether the product was returned or not returned. So essentially you have four category of signals product attributes right like pricing rating etc are the attributes of product itself. So that is one type of signal. Second one is seller. So product is product but who is selling it? Sometimes sellers are not professional you know they may not pack things properly. They may not deliver them properly. So seller rating also matters. Third category is logistic, shipping distance, uh what kind of couer service you are using and so on. Customer, it depends on customer 4:34:06 too. Okay. So, how many support calls they had, page views and so on. Pause this video and tell me what is your hunch on which signal do you think predicts returns the best. And folks, when I'm giving you this exercise, please make sure you're working on it. Okay? Cuz you can just fast forward this video and not do it. But I know you all are sincere students. So you're going to work on all the exercises everything that I say here. Okay. So pause this video. Uh write it in not and pan. Which signal do you think predict returns the best? See when you think uh and when you gain that domain understanding that is what will help you in your career. Learning some API syntax is not going to help you. Your original thinking will help you. Therefore you must pause right now. Write down with it with with a note and pen which signal do you think predict returns the best. Okay, now you have written it down. We will see when we build the actual notebook. We'll find out which column has a maximum impact on 4:35:09 the return. But the end to end workflow will look something like this. We are going to take the data do some exploratory data analysis. We'll do some data cleaning feature engineering. We'll build new features, train a model, then we'll do model evaluation and then we'll save the model to disk so that we get this artifact. We get this saved model. And once we have the saved model, we are going to then wrap this in a backend server. Okay. So we will create a backend server using a framework called fast API. So fast API allows you to uh create Python servers, backend servers. And then we will have front end. So this is back end, this is front end. Front end is going to be in NodeJS. Don't worry, you don't have to take NodeJS tutorials. We are going to do w 4:36:10 coding and build this. So there will be a UI which will make the call to this backend server. So so this will be the technical architecture. The UI is going to look something like this. Okay. This is built totally using wipe coding. Here you'll be able to enter the item attributes fragile, delayed, how customer paid and so on. And when you click on predict, it will call the backend server. It will tell you the prediction result. See, you will be able to change all these different parameters and predict whether the item will be returned. See, medium risk and so on. Okay. So, we are going to do hyperparameter tuning. Try different model figure out which model works the best. All right. So, your capstone starts here. Now in my ML practice folder I created a subfolder called capstone and I have placed my data set inside the data set subfolder. See data 4:37:12 set subfolder and this is the same CSV file that we looked at it uh just now and I have created a new notebook here. Okay. So this is how the notebook looks. All I had done is loaded the data into my data frame. 12,000 rows, 26 columns. Okay. So this is how it looks. Now the first thing I'm curious in knowing is the class distribution. I want to know how many rows are returned and how many basically products are not return. You can easily get that by saying that df return value count. So the value count is this number of items which are returned is 3645 and 8355 items are not returned. So there is a little bit of a class imbalance here. You can view this in a bar chart format 4:38:13 by writing this code. So same same numbers but you're writing a bar chart. Okay. Name counts is not defined. So counts is this. And then in the Jupyter notebooks you will use this kind of visualization so that you can do explorative data analysis through a visual way. Many times you share these notebooks with your fellow data scientists and AI engineers and this kind of visualization helps uh in terms of getting the idea in a visual format. Okay, this code is not complex folks. You can use AI to write this code and I don't remember the code syntax here. Okay, all I know is that you can use mattplot lib to plot a bar chart like this. Here you can see percentage around 70% are return 30% are not return as I said before there is some kind of class imbalance and when you're working on this kind of data set such as product 4:39:13 returns fraud detection even in case of fraud transaction analysis number of frauds will be very low compared to the transactions which are not fraud. So you'll always see this kind of class imbalance in these type of problem statements. Now let me also do df info that will give you an idea on the data types. So these are string, these are float, integer. Okay, these are the data types that you have to get the numeric summary. You can use describe. And by the way, there is another function called T. So it can transpose your table. So rows becomes columns, column become rows. And in this case, I think this view is better here. Let's do some analysis. So product price if you look at min is 19, max is 64,000. Oh, see, so here the the range is quite big. And one thing I'm noticing is 75% of my 4:40:15 values are having price below,105 which means my distribution is skewed. What I mean by that is if you plot a histogram of product price. So let's do that. So this is the product price. You can do this as well. Okay. And if you plot histogram, it is right skewed. What it means is most of the items see 75% of items total items have price which is 1105 or less. And you have some values on a higher end. You don't even see those bars because those bars are so tiny. Let's analyze other columns. So discount percentage is okay. Product rating minimum is 1.6. Max is five. Okay. So there are no outliers. Sometimes you see negative values as well. Review count 24116 4:41:16 looks good. Fragile item is 0 and one. It's a binary variable. Warranty available is 0 and one. Fine. Product return rate the min is this. Max is this. Okay. Deflect rate. All right. shipping distance things look okay. I'm not um spotting anything obvious. Of course, as we do further analysis, we will find out details. Now, let's look at the NA value. So, to check NA value, you'll simply run this command. And here you can see that product rating has 240 items which are you do have 240 items for which you don't have product rating available. Review count is this. Now if you look at the total number of items in our data frame which is 12,000 compared to that number of NA values are not that big. Even if you discard these rows is 4:42:18 going to work. Okay. So NA value doesn't seem like a big issue here. Now I want to uh check all the categorical columns which are text columns. Right? there. So if you look at it here in DF info, all these string columns are the categorical columns. For those I want to check unique values because many times there are typos etc. And you might want to fix those. So these are the categorical columns of course order ID etc. I'm ignoring because that doesn't matter. And then for now we will print unique values for each of the categorical columns. How do you do it? Well, very simple. You will say df call value counts. So here it will show all the values and the count for those and you can print a empty empty line here by doing this. So see product categories are fashion, electronics, home, beauty, 4:43:19 sports, books etc. Okay, they look good. Subcategories are mobiles, audio. Let's see if we see any issues, nonfiction, fiction. Okay, at least I I don't see the problem in my first glance. For brand, you have these numbers in the production database. There are some kind of ids brand_1 and two and so on. So, this looks good. I don't see any duplicates here. Of course they are unique value but I'm I'm spotting like you know sometimes B is small there is double underscore I'm I'm looking for those kind of issues fulfillment type marketplace fulfilled seller fulfilled ah look at this marketplace fulfill and marketplace fulfilled they are same but it is showing us a different value because small m and small f so this is something we'll fix okay we will consolidate these two into one bucket payment type looks okay. There is no issue with it. Now 4:44:22 what if we plot return rate by certain columns say I want to know how many cash on delivery items were returned, how many UPI items were returned, all of that. So I want to do that analysis using mattplot lip. So let's do some initialization and for these columns these columns payment method product category and fulfillment type I want to do that analysis okay so let's do that here see you are grouping by that particular column so you're grouping by payment method etc and you're checking the returned basically the value of return mean and then you are shorting them so when you do that see return rate for cash on delivery is higher. You see 0.5 for other payment types it is lower. So this is giving some signal that payment method has influence on our target 4:45:23 variable especially cash on delivery value in product category. Pretty much all categories have similar product rate uh return rate. Fulfillment type also also kind of similar return rate. Okay. So now let's plot a histogram. See we previously plotted histogram for product price. I want to do same thing. So we are doing the same thing here but for couple of uh columns. So I want to target all these columns. These are all numeric columns. And I want to look at the distribution. See checking the distribution will give you some kind of signals statistical signals. So product price of course right skewed on the right hand side there is this flat so it's called right skewed other graphs look okay this is uh normal distribution product page views okay quantity defect rate is also right 4:46:26 skewed okay so see as a data scientist you will do all this analysis so that you get understanding on these features and you can use this understanding to build your model further Okay. So we are developing some understanding. We got some clues so far. Now I want to check the correlation with target. So see this correlation thing is very important here. You want to know correlation is basically when you're checking a correlation between A and B. You're checking how B changes as A changes. Okay. Sometimes they have inverse correlation. When a increases, b decreases. Sometimes they have linear correlation. When a increases, b increases. For example, house price prediction. The square of a house and the price has linear relation. So as square of home 4:47:28 increases, price increases. And some other variables. For example, you're working on some kind of agriculture project. So as temperature increases, the water level or the humidity of the soil decreases. So if a is a temperature and b is the humidity uh humidity in soil or let's say water level you know water level or moisture in soil they will have this kind of inverse correlation okay here I want to know how all of these variables that I have like item is fragile, uh defect rate, how do they impact my end target variable which is product return rate and you can do that by uh plotting 4:48:30 this kind of heat map where you are finding a correlation. So you see correlation. So this function helps you find the correlations. Okay. So let's plot just that function. So here I want to select all those types which has number data type because uh for uh string data types you can't you can't plot it. Okay. And this correlation it it's it's a data frame. So here I mean this data frame is hard to visualize that's why we have to use uh some kind of charting. But see the correlation between product price and product price is one because it's the same variable. As product price increases, product price increases. Same variable. But let's look at the return right. So this return so this is returned or let me just go at the last. So last one is return correct. So in the 4:49:32 return you see check this one. uh 0.05. What is this fragile item? You see so they have some good relationship. See one correlation means one thing changes the other thing changes in the same manner. But 0.25 positive correlation means if the item is fragile it will be likely returned. Okay. What is the other column? Okay. This one check this product return rate. So for a given product in the past if customers have returned the product likely for a new order they will return it. So anything that is positive or closer to one is going to be the factor here. See uh let's see one see one is product product return to return. Okay. So that that all the ones in the diagonal. See all these ones in the diagonal you ignore that that is just uh a noise for 4:50:33 you. This one is also important. See 22 what is it delayed delivery. See [laughter] if the delivery is delayed it will be returned. So let's visualize the same table. So same table folks same table but we are visualizing in a visual format. So here what you have to look at is check this return column. So in return column wherever you see some kind of coloring see red means strong correlation. So look at this color band. Blue means very weak relationship. So here I see things like see uh look at this and this diagonal you ignore because it's product rate to product rate. See diagonal you completely ignore. You look at this last column and look at the colors which are closer to red. So this color so fragile item is going to be a strong indicator. Then product return rate strong indicator. Then brand return rate. Okay. 4:51:34 Shipping distance delayed deriv. See this is giving you all the this signals uh on the strong predictors. Okay. So these columns are called strong predictors uh based on this uh correlation that we have plotted. You can use this this uh data frame also but it's kind of hard to visualize that is the reason we we did this otherwise it's the same data. Okay, it just we are using the visualization for our better understanding. All right now we have developed some understanding on our data. We are doing essentially exploratory data analysis EDA. Okay. Now we are done exploring our data. Let's fix the data. Let's do data cleaning. In data cleaning, remember we had this fulfillment type. This extra fulfillment type. I would put I would consolidate both of these into a single category. So from df let me create a new data frame called df clean. And the way you do that 4:52:36 is df.copy. See if you do just this it will be a shallow copy. it will do referencing. If you do this, it's a deep copy. So, it is doing the true copy of values. And here in the fulfillment type, you know, in a clean data frame, you can just say you can convert that to a title case. So, this one I'm converting it to a title case, which means first letter of every word will make it capital. So, see this M will be converted to capital M. F will be converted to capital F and after that I should see only two values. See I'm now seeing two values. Oh no actually okay here df clean. [laughter] Uh see this is a problem I face whenever I'm using multiple data frames. I keep on referring to the old data frame and then I wonder okay why there is a problem but it you have to refer to DF clean here. 4:53:37 Perfect. Values are clean. Instead of three categories, you see two categories now. Okay. What else can we do? Let me check the data frame. See, immediately I see this order ID column which is not useful. Order ID is just random. So, let's drop that. So, here by the way, you can use this B key to create a new cell or you can go to plus and insert the cell as well. So in DF clean I will say drop. Which columns you want to drop? Order ID. Are there any other columns? Okay. Subcategories warranty. See all other columns looks like they are useful. All right. Let's assign this back. and then df clean drop columns equal to and see 4:54:41 that column is gone. I also want to check for duplicates. For cleaning duplicates, we have an easy method called drop duplicates. Okay. And when you look at it, well, we didn't have any duplicates. But it's good to have this kind of code because tomorrow when the data file changes and if it has duplicates when you execute the entire code duplicates will be removed. All right, we are now done with our exploratory data analysis and data cleaning. Let's now do feature engineering. Feature engineering means you can look at the current features. Features are nothing but these columns which are impacting your target variable and then you can apply some transformations to make that better. You can also create new columns if required. Now what kind of feature engineering we should do for our data set? Well, the first idea I have is see we have the product price which is right skewed. So how about we apply some transformation to make it little more 4:55:44 uniform. Okay. So our product price is essentially looking something like this right. So here we have the price okay and the number of data points and we have most of our data points most of our return records they have price less than 1,000 and then we have some heavy items which has price 60,000. So our histogram looks right skewed. Okay. One thing you can do here is you can apply whenever you have right skewed diagram. It is a standard practice in statistics to apply log transform. What log transform will do is see here there is equal distance between 1,000 2,000 3,000 and so on. In lock transform what will happen is there is one here 4:56:45 and then there will be say 1,000 and then there will be 10,000 and so on. So due to this what will happen is you will have some kind of uniform histogram. You're kind of squeezing all these values into like kind of squeezing them. Okay. If you don't know log fundamentals on my channel I have a feature engineering playlist you can watch those later on. So that is one transformation I will apply. The second transformation I want to apply is look at this customer support call and chat interactions. These are essentially same thing. Here they're using chat medium. In the other one they are using a phone call. How about we combine those and we create a single column called total customer support contacts. So that way you know how many times a user has interacted with customer 4:57:46 support for this particular product. Okay. So how about we do this? We take our clean data frame and then we create a function called add features and we store the feature engineered data frame into the new data frame. Okay. So let's define this function called add features and in D you can add this new column. Okay. Total support context is equal to D of um customer support calls and then uh this one chat interaction. Okay, this is a new column I'm creating. And then for price, let's create a new column called log price. And that is nothing but NP log 1 P. Okay, this is a 4:58:47 function you use to create that log price and of course you will drop the product price. Okay, so you're going to drop it and you return D. All right, so this is what a new data frame is going to look like. So let's check it out. Here you have a log price. Here you have total support context which is the just the sum of these two. And when I look at the log price, let me plot the histogram, the histogram is now looking more uniform. You see, because that is how the log transform works. I will save the clean data set to a CSV file so that just in case if I want to refer to it later on, I can do that. And then for the new columns, I will just print some statistics. So these are the statistics. 4:59:48 Total support contracts, log price and so on. When I check my add feature function, I'm realizing I dropped product price. But I did not drop these two because see we created this new column in our feature engineering phase. So then what's the point of keeping these two as well, right? So let's drop those two because we're just combining all the support calls into one column or a feature called total support context. Therefore we can drop this. All right. So now you have only one column total support contacts. Okay. Now I'm feeling good. This thing looks good. Now what is the next step? Well, next step is train test split. We have done train test split so many times. So I don't have to write all the code line by line. We'll follow the same process that we have followed 100 times before which 5:00:51 is create X and Y and then use train test split to split. We are keeping 8020 ratio here. Okay. So control enter. Train has 9,600 records. test has 2,400 records. Now I am going to do outlier removal. Now you'll be like, "Oh, why you are doing outlier removal right now? Why don't you do it before?" Well, because the test set is something that you see in production. I don't want to touch it. I want to keep it as it is. I want to only remove outliers from train. This is a good practice folks because that way the test data is whatever you are seeing in production. Okay. So to detect the outliers you can use something called a box plot and box plot uses this method of IQR. Okay, if you're not aware about IQR 5:01:51 again it's a statistics uh method and we have a separate video on it. See, I don't want this to be like a statistics video. That's why I will refer you to this video. I have explained it really well in a language that even a child can understand on what is IQR. So you finish this uh project first and then you can clear your fundamentals on IQR. Anything else you want to learn in feature engineering, we have an entire playlist. Okay. So we are going to use box plot. So first let's plot a box plot. So box plot we are plotting for the numeric columns and anything that you see outside this box which is the box boundaries are outliers. For example these dots are outliers. These these dots again for product price are outliers. In these two like review count, shipping distance, discount percentage, we don't have uh outliers. Okay. So let's remove 5:02:52 those outliers here. And we are using seabond library by the way. All we are doing is here we are going through all these uh columns and we just plotting the the box plot. Okay. Let me plot it for a single column so you understand. So when you do SNS dot box plot and when you do something like this when you give it a column it will plot this box plot and anything that is outside these boundaries are the outliers. Okay so that's how it works. So now let's remove it. I'm going to remove the outliers only for the log price. See I have only one column lock price because see page views 2025 you look at it it's not an outlier to be honest. See my honest take is even if 5:03:52 you don't do outlier remole is going to work okay but I want to do it so that you can get an idea on how to do outlier remole. I will do it only for a log price. Some of the very high price items I will remove it. Okay. So removed three extreme outliers from our training data set for the log price. Now let's check our X train once again. See in X train. Okay, our numeric columns are okay. But how about all these uh string columns? We need to do one hot encoding. If you remember machine learning model doesn't understand text columns. So what do we do with text columns? Well, let's encode them. So let me create Python list with binary columns, binary flags and categorical features. Okay. And then we'll have another list with numeric features. So if you look at these three list, it's just a list of columns. Okay. So if you say binary flags, these are 5:04:54 columns which have binary value 1 0. Categorical features are the columns which has more than two categories. Okay. And numeric features are the columns in a data frame which has numeric values. And then we have all features which means all columns. Okay. So if you do all features it is all columns. Now one thing I'm recalling is remember we still have NA values. So maybe let let's get rid of these NA values. See. Yeah. So my numeric numeric columns are not good. I I said like numeric columns are good but no they need some post post-processing. So let's remove these values which has NA in them and rather than removing it how about we substitute NA value with median value that is one of the standard practice okay so I will imputee or handle the missing values by using the median. 5:05:57 So here for all the features I will get the medians. So see train medians these are the medians for all the columns. I got medians for all the columns and then for all the columns in train and test board. Let's do that. Okay I will just fill na with that median value. So when I do that if I do NA now see they don't have NA values. Okay. So our NA values are gone. And then let's do the scaling. So for the numeric features I'm going to use standard scaler and scale those columns. And the third step will be to do one hot encoding for the categorical features. See categorical features I'm saying get dummies. get dummies will actually just do one hot encoding. So let me just show you here. See now see my values are scaled first 5:06:57 of all and then if you look at yeah see product category electronics product category fashion product category home. So these are now one hot and coded. So final feature count is 85. Train and test is this. Now we are going to train not one model but multiple models to see which model gives you the best result. So let's create a model dictionary. I would like to try logistic regression, decision tree, random forests and XG boost. Okay, XG boost is the that in our ensemble techniques we had bagging and boosting. So for boosting we have this XG boost model. Okay. And I have imported all these classes at the very beginning of the notebook. See here you see I have imported xy boost classifier. Xy boost comes from a different library. So you have to do pip install xy boost. Folks throw the code whenever you are 5:07:57 facing installation problem. You are capable of handling it. I don't have to spoon feed you. You just do pip install xy boost. You just do pip install job. You should be able to install things on your own using AI tools. Whenever you're stuck getting error, ask AI. Let us go through all the models one by one and fit them okay and and train them basically. So see it has trained four models. And then I will create a function which takes your model. It will take your X test and Y test and it will give you accuracy score, recall score, all of that. See, whatever values you get by calling the classification uh report function, you can individually call all these functions to get accuracy, recall and so on. Okay, so let's do a comparison for all these model options. I'm passing X test Y test into your build function and creating a 5:08:58 data frame. Okay folks I'm going a little bit fast but if you have any questions on any of the code once again ask chat GPT it will give you an idea. We have a Python for AI video as well by the way where we have covered all the Python fundamentals just in case if you don't know all this code syntax. Okay. So logistic regression seems to be giving the highest accuracy, precision and recall. Okay, recall is higher. So I I want to focus on recall because I want to know like uh if the product is return, I want to catch that. So looking at all these metrics looks like logistic regression is probably the best function. See here decision tree is giving a low performance compared to other models. You can do a nice visualization of same numbers. Okay, using mattplot lab you can do nice visualization. And here once again as you can see logistic regression overall looks like a better model. Okay. So our 5:09:59 best model is basically in the comparison whatever is the first model which is our logistic regression. Uh and we we used F1 score. See F1 score is best for logistic regression and the imbalance uh there is some class imbalance. So when you supply class weight is equal to imbalance that is that is how you can handle class imbalance and now let me do this. So here um you are evaluating the model. So let me just show you see final model is equal to fitted best name. Okay. So this is the final model and on that final model you call evaluate method where you give X test Y test and it will give you the tuned matrix. Okay. Okay, so these are the tuned metrics and let's plot confusion matrix for the same anything on diagonal correct prediction. Anything 5:11:01 here other than diagonal are the errors. There were total like if you look at this number 101 return items which we said are not return and there were total 245 not returned items which we said are returned. Okay, so these are the errors and here is our final classification report for our logistic regression model. And let's look at the coefficient. Coefficient is essentially when you have logistic regression, remember that y is equal to mx + b equation. So that is what it is. But here our variables are lot right. So that's why you see all these numbers. Okay? And there is intercept also. So final model intercept if you look at it this is the intercept. Now these coefficients are basically the feature importance. So see it's like this if you are doing home 5:12:01 price prediction. Okay. And uh home price prediction is y is equal to see our equation is m x + b that is the equation. So for home price we have some kind of coefficient right coe f_sub_1 okay or let's say m_sub_1 multiplied by area right the for home price the area might be the best uh indicator by area I mean okay is it in Bangalore with jpag or which area and then m2 will be square foot square foot and then bedroom. Okay, number of bedrooms. Now, square foot has a larger impact on the home price compared to bedroom. If the bedroom changes from 2 to three, there is not much difference. But if the square foot, I mean square foot and bedrooms are also related by the way. But I'm just saying square foot has a major impact. So when you train this model, this m1, m2, m3 are the 5:13:04 coefficient that you see here and they decide the feature importance. So if M1 value is very high, it means the effect of that on a final home price will be higher. That's why this is a feature importance for area. This is a feature importance for square foot and so on. Okay, so that's what these numbers are. Basically, it's a feature importance and we can plot them on a chart here. Okay, it's just a coefficient I'm plotting. Ignore all this code. It's you can generate using AI. But here what it shows is uh look at the red lines basically protective. So as the item is fragile it will likely be returned. And the blue line is a negative coefficient which means if it is a higher value the item will not be returned. If warrant is available item will not be returned because it's a positive value. Okay. So payment method cash on delivery item will be returned. all of that see 5:14:06 payment method UPI it will not be returned so it's like reverse now folks we are going to save the model to disk so first I will create all these variables for saving the model and in the model data I will have my final model name scalar model columns the feature importance column that I got from here all of that okay and to save the model folks now see we have this library called joblib which we imported correct so joblib is a library that we imported we are going to use that to export the model so you'll say joblib dod dump with model data and it will uh dump that file model file into this artifacts folder okay so control enter no such file or directory called artifacts so let's create that directory I thought it would create it on its own but it did not. So let's RT effects and 5:15:10 now it should work. Control enter. It worked. And look at this. See this is your model folks. If you look at it, it will be gibberish. Okay. But this 9 KB model is your saved model. You can load that model in a different notebook or a different person on different computer can load this model by saying job.loaded. load loaded and then on loaded model you can perform predictions. So let's say I want to take sample you know sample records from X test and I want to predict so I will say load loaded and in loaded by the way the actual model is here. So on that model you can say predict probability or you can just call predict function and then you can predict the probability. So we took the this one sample. So this is the sample. Okay. So for all these five rows these are the probabilities. So these two items will 5:16:12 be return. The other items will not be return. So item number second and third right. So second and third these two will be returned. So these two will be return. Okay. This is return. This is return. Let's see. Okay. Okay, you have to check the individual features. All right, and the model evaluation report. So this is the final model evaluation report which we have seen before. All right, but folks uh this is amazing, right? We just trained the model. Just to summarize, we went through the steps of exploitative data analysis, feature engineering, training the model, finding out the best model, and then we saved the model into this particular artifact. This is it. We are done with the notebook part. And now for this project, what is left is creating back end and front end. We are not going to write front end and back end from scratch. I don't want to make this 5:17:13 NodeJS tutorial or or fast API tutorial. I have used AI cloud code to create the back end and front end code. Okay. So the code looks something like this. See we had a data set and we had this particular notebook. Correct. We then created the back end. So this back end is a fast API back end. So this is the main file. See fast API. And what we are doing here is we are loading the model which we had. You see when you call predict function here. So this is the endpoint. So our front end will call this HTTP endpoint that will call predict function and predict function will load the entire bundle. So if you look at the bundle, it is basically the artifact, right? This artifact model data job. See model data.joblip. it will load that into the bundle and once you have that you will call the 5:18:16 uh predict function on it. So this is the bundle this is the model and you do pre-processing as usual and then you will call probability function and you now check some threshold. Okay so if the threshold is less than.3 risk is low otherwise medium otherwise high. So we have decided these three categories. You can talk to your business manager and determine these risk levels. I just created like three categories. Okay. 1 2 three. Low, medium, high. And then it will return that. So that is the main code for back end. In front end, I didn't even look at it. I am not a front- end programmer. I don't know how it works. Okay. So we going to skip through that and we will now look at the readme file on instructions how to run our code. So let's look at the readme and uh we have also made it a UV based project. So you can just talk to your cloud and just say UV. UV is a Python 5:19:17 package manager which is standard nowadays. So we use that. So let's uh run a back end. To run a back end you just cd into backend directory. See cd and then you run this code. So let me run this command. Essentially this is a UV command that you will run. Okay. So back end is now running for front end. I have a different command prompt. So here in VS code you can create a new command prompt. So see back end is by the way meanwhile started. For front end I will just say cd front end already in the front end directory. npm install is something I have run already. So I will just say npm rundev. You have to install node by the way. So just install node from the server. Another server is running. Okay fine. Server is already running and okay let me just open this in a 5:20:18 browser. That will be better. So in a browser just type localhost col 3000. Look at this. Uhhuh. our okay fail to fetch we getting some error okay there was some kind of hung process so I killed it using task kill and now this is running see this is the model performance we are using logistic regression you can for a new product you can select a category let's say in a fashion category I whatever subcategory brand price whatever item is fragile there is no warranty uh item is delayed and when you say predict see it is high risk. Now when item is not delayed we know all the strong indicators right delayed is strong indicator see it reduced if it is not fragile it will reduce further see 22 27% to 2%. But if you have a lot of chat messages and support messages which means item has some problem it will see 5:21:21 high risk. If you have lot of see if customer is calling like so many times it means the product has a problem. Okay. So just play with it. It is an amazing UI folks. You can add this project to your resumeumé. Create a GitHub repository. Make some changes in it and add it to your uh rum. Add it to your project portfolio. All right folks. So that's it. We covered front end back end. We trained the model. We covered so many aspects and we completed this project end to end. We still have a very important thing left which is exercise for you. Without exercise this project will not be complete and these exercises are my fees to you as a student. You're learning this from YouTube for free. But nothing is free in life. So you have to work on this exercise. If you don't, then I'm going to introduce some kind of AI hack into your computer and your 5:22:21 computer will heat up and burn while you're sleeping. Okay? So, let's be safe, folks. You don't want your computer to be burned. We have this PDF. We are going to attach this PDF down in the description below. And these are the exercises. Just read through this. You will work on all these exercises and then you will write a LinkedIn post. Suchin is one of our cohort students. He's very good in sharing his learnings on LinkedIn. So you can write a LinkedIn post just like how Sachin does where you will share the learning from this project. You will include a GitHub repository. Make a nice GitHub repository with nice readme and so on. Address all these exercises in that. Share your learning here. Okay? And maybe you can attach some kind of demo or a video presentation as well. You can present as if you are presenting to your business stakeholders. Let me show you this another LinkedIn from uh our 5:23:23 student Arian. So here see he has attached this uh video presentation as if he's presenting it to business stakeholders. And by the way, Arian got a job because of uh this work that he did because of the LinkedIn. So LinkedIn will help you increase your credibility. And tag me, tag code basics, tag Hammon. That way we know we can engage. We have a huge network by the way. You see like 250K followers. So when I like your post, comment on it. uh that notification will go to people in my network who who are hiring in AI industry who might be AI leaders. It will overall help with your uh online credibility. Okay. So please make sure you are finishing the exercise. All the code everything is given in video description below. I hope you like this entire crash course. If you did, please share it with your friends who wants to learn this. Also, 5:24:23 in the comments, uh please post uh what feedback you have for this course. Did you like it? What parts did you like it? And uh also post the LinkedIn post. You know, you can uh whatever you're posting on LinkedIn, you can post in the comments of this YouTube video and also post the answers of your exercises. Let's do some engagement in the comments so that other can feel inspired, other can verify answers from your answers and so on. All right. Thank you very much for watching.
