Permutation Importance -Machine Learning Explainability

Introduction

Machine learning models often act as black boxes, meaning that they can make good predictions but it is difficult to fully comprehend the decisions that drive those predictions. Gaining insights from a model is not an easy task, despite the fact that they can help with debugging, feature engineering, directing future data collection, informing human decision-making, and finally, building trust in a model’s predictions.

You can consider ML as a Multi-variate equation, where we can fit in different weights and get the same result.

Colab Notebook

All the codes executed in this article are available in this notebook. Please take a copy and try executing the notebook. Happy Learning !!!

Permutation Importance

One of the most trivial queries regarding a model might be determining which features have the biggest impact on predictions, called feature importance. One way to evaluate this metric is permutation importance.

Permutation importance is computed once a model has been trained on the training set. It inquires: If the data points of a single attribute are randomly shuffled (in the validation set), leaving all remaining data as is, what would be the ramifications on the accuracy, using this new data?

Process of Permutation Importance

Ideally, random reordering of a column ought to result in reduced accuracy, since the new data has little or no correlation with real-world statistics. Model accuracy suffers most when an important feature, that the model was quite dependent on, is shuffled. With this insight, the process is as follows:

1. Get a trained model.
2. Shuffle the values for a single attribute and use this data to get new predictions. Next, evaluate change in loss function using these new values and predictions, to determine the effect of shuffling. The drop in performance quantifies the importance of the feature that has been shuffled.
3. Reverse the shuffling done in the previous step to get the original data back. Redo step 2 using the next attribute, until the importance for every feature is determined.

Python’s ELI5 library provides a convenient way to calculate Permutation Importance.

Explanation

1. We’ll train a Random Forest Regressor using scikitlearn’s Boston Housing Prices dataset, and use that trained model to calculate Permutation Importance.

2. Loading the dataset,

from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor

import eli5
from eli5.sklearn import PermutationImportance

boston = load_boston()
print(boston.DESCR[20:1420])

3. Split into Train and Test Splits,

# separate data into target & independent variables
x = boston.data
y = boston.target

# split data into train and test set
x_train, x_test, y_train, y_test = train_test_split(x, y, train_size=0.8)
print('Size of: ')
print('Training Set x: ', x_train.shape)
print('Training Set y: ', y_train.shape)
print('Test Set x: ', x_test.shape)
print('Test Set y: ', y_test.shape)

4. Train the model,

# train model on training set
rf = RandomForestRegressor()

# fit model on training set
rf.fit(x_train, y_train)

# calculate score on test set
print('R2 score for test set: ')
print(rf.score(x_test, y_test))

5. Evaluate Permutation Importance

# create permutation importance object using model
# and fit on test set
perm = PermutationImportance(rf, random_state=1).fit(x_test, y_test)

# display weights using PermutationImportance object
eli5.show_weights(perm, feature_names = boston.feature_names)

6. Interpretation

• The values at the top of the table are the most important features in our model, while those at the bottom matter least.
• The first number in each row indicates how much model performance decreased with random shuffling, using the same performance metric as the model (in this case, R2 score).
• The number after the ± measures how performance varied from one-reshuffling to the next, i.e., degree of randomness across multiple shuffles.
• Negative values for permutation importance indicate that the predictions on the shuffled (or noisy) data are more accurate than the real data. This means that the feature does not contribute much to predictions (importance close to 0), but random chance caused the predictions on shuffled data to be more accurate. This is more common with small datasets.

7. In our example, the top 3 features are LSTAT, RM and DIS, while the 3 least significant are RAD, CHAS and ZN.

Conclusion

This article is a brief introduction to Machine Learning Explainability using Permutation Importance in Python. Gaining intuition into the impact of features on a model’s performance can help with debugging and provide insights into the dataset, making it a useful tool for data scientists.