The machine learning models works on a constructive feedback principle. You build a model, get feedback from metrics, make Improvements and continue until you achieve a desirable accuracy Evaluation metrics explain the performance of a model. An important aspect of evaluation metrics is their capability to discriminate among model results.
I have seen plenty of analysts and aspiring data scientists not even bothering to check how robust their model is. Once they are finished building a model. they hurriedly map predicted values on unseen data. This is an incorrect approach.
Simply building a predictive model is not your motive. It’s about creating and selecting a model which gives high accuracy on out of sample data. Hence, it is crucial to check the accuracy of your model prior to computing predicted values. In our industry, we consider different kinds of metrics to evaluate our models. The choice of metric completely depends on the type of model and the implementation plan of the model.
After you are finished building your model, these 11 metrics will help you in evaluating your model’s accuracy. Considering the rising popularity and importance of cross-validation, I’ve also mentioned its principles in this article. And if you’re starting out your machine learning journey, you should check out the comprehensive and
popular ‘Applied Machine Learning course which covers this concept in a lot of detail along with the various algorithms and components of machine learning.
When we talk about predictive models, we are talking either about a regression model (continuous output) or a classification model (nominal or binary output). The evaluation metrics used in each of these models are different.
In classification problems, we use two types of algorithms (dependent on the kind of output it creates)
Class output: Algorithms like SVM and KNN create a class output. For instance, in a binary classification problem, the outputs will be either 0 or 1. However, today we have algorithms which can convert these class outputs to probability. But these algorithms are not well accepted by the statistics community
Probability output: Algorithms like Logistic Regression, Random Forest, Gradient Boosting, Adaboost etc. give probability outputs. Converting probability outputs to class output is just a matter of creating a threshold probability, In regression problems, we do not have such inconsistencies in output. The output is always continuous in nature and requires no further treatment
Illustrative Example
For a classification model evaluation metric discussion, I have used my predictions for the problem BCI challenge on Kaggle. The solution of the problem is out of the scope of our discussion here. However the final predictions on the training set have been used for this article. The predictions made for this problem were probability outputs which have been converted to class output assuming a threshold of 0.5.
Acumen💻 