# Hyperparameter Tuning using Python

November 22, 2021 2021-11-22 9:07## Hyperparameter Tuning using Python

Hyperparameter Tuning is a technique of choosing the best hyperparameters to get the maximum out of a Machine Learning model. In this article, we will be going over multiple ways to perform hyperparameter tuning using Python.

## What is a Hyperparameter?

Hyperparameters are values that dictate how a Machine Learning model is built.

These values cannot be estimated by the model from the given data but are rather set by the Machine Learning engineer or Data Scientist when building a model architecture. Some examples of hyperparameters are; depth of a Neural Network, number of trees, learning rate, activation function, etc.

Because these hyperparameters are so important to a model’s performance, speed and accuracy, you must optimize them, and there are several methods to optimize the hyperparameters.

## Hyperparameter Tuning using Python

There are two broad categories to divide hyperparameter tuning methods in Python:

**Manual hyperparameter tuning:**In this method, different combinations of hyperparameters are manually chosen and then the loss/accuracy of the model is measured. This is not a very recommended method as it’s a tedious process and is not effective in cases where there are many hyperparameters to estimate.**Automated hyperparameter tuning:**In this method, the parameters are chosen using an algorithm that automates and optimizes the process.

Before beginning to learn how to perform hyperparameter tuning using Python, let’s set up the dataset first. For this exercise, we’ll use the credit card fraud detection dataset and we’ll use the `sklearn`

Python library to create a training and testing dataset split.

# Importing libraries import pandas as pd from sklearn.preprocessing import StandardScaler from sklearn.model_selection import train_test_split # Reading in the dataset df = pd.read_csv('../input/creditcardfraud/creditcard.csv', na_values = '#NAME?') # Defining independent (X) and dependent variables (y) X = df[['V17', 'V9', 'V6', 'V12']] Y = df['Class'] # Splitting the dataset into training and testing dataset X_Train, X_Test, Y_Train, Y_Test = train_test_split(X, Y, test_size = 0.30, random_state = 101)

Now that we have our training and testing dataset ready, let us learn different methods to perform hyperparameter tuning.

## 1. Hyperparameter Tuning using Random Search

As the name implies, Random Search is a method of choosing a random set of hyperparameters that work best for a given Machine Learning problem.

In this tuning method, we set up a grid of values of different hyperparameters and then, randomly pick out combinations to build, train, and evaluate a Machine Learning model on. The combination of hyperparameters that give the best evaluation results is selected as the final set of hyperparameters.

We can now start implementing Random Search by first defining a grid of hyperparameters that will be randomly sampled when calling RandomizedSearchCV() from scikit-learn’s `model_selection`

module.

# Importing necessary libraries import numpy as np from sklearn.model_selection import RandomizedSearchCV from sklearn.model_selection import cross_val_score # Creating a Random Search grid random_search = {'criterion': ['entropy', 'gini'], 'max_depth': [2], 'max_features': ['auto', 'sqrt'], 'min_samples_leaf': [4, 6, 8], 'min_samples_split': [5, 7,10], 'n_estimators': [20]} # Initializing a Machine Learning model clf = RandomForestClassifier() # Finding the best hyperparameters using Random Search model = RandomizedSearchCV(estimator = clf, param_distributions = random_search, n_iter = 10, cv = 4, verbose= 1, random_state= 101, n_jobs = -1) # Training the model model.fit(X_Train,Y_Train)

We can now evaluate how our model performed using Random Search. In this case, using Random Search leads to a consistent increase in accuracy compared to our base model.

from sklearn.metrics import confusion_matrix from sklearn.metrics import classification_report # Predicting using the best estimators predictionforest = model.best_estimator_.predict(X_Test) # Printing the confusion matrix print(confusion_matrix(Y_Test, predictionforest)) # Printing the classification report print(classification_report(Y_Test, predictionforest))

## 2. Hyperparameter Tuning using Grid Search

Grid Search is very similar to Random Search where we form a grid of values for different hyperparameters but the way this method differs is that this method chooses the combination in a specific order and run the model on each and every combination, records the data, and gives out the combination with the best accuracy or minimum loss.

In order to choose the parameters to use in Grid Search, we can now look at which parameters worked best with Random Search and form a grid-based on them to see if we can find a better combination.

# Importing necessary libraries from sklearn.model_selection import GridSearchCV # Creating a Grid Search grid grid_search = {'criterion': ['entropy', 'gini'], 'max_depth': [2], 'max_features': ['auto', 'sqrt'], 'min_samples_leaf': [4, 6, 8], 'min_samples_split': [5, 7,10], 'n_estimators': [20]} # Initializing a Machine Learning model clf = RandomForestClassifier() # Finding the best hyperparameters using Grid Search model = GridSearchCV(estimator = clf, param_grid = grid_search, cv = 4, verbose= 5, n_jobs = -1) # Training the model model.fit(X_Train,Y_Train) # Predicting using the best estimators predictionforest = model.best_estimator_.predict(X_Test) # Printing the confusion matrix print(confusion_matrix(Y_Test,predictionforest)) # Printing the classification report print(classification_report(Y_Test,predictionforest))

## 3. Hyperparameter Tuning using Bayesian Optimization

Tuning and finding the right hyperparameters for your model is an optimization problem. We want to minimize the loss function of our model by changing model parameters. Bayesian optimization helps us find the minimal point in the minimum number of steps. It can be performed in python using `Hyperopt`

library. In Hyperopt, Bayesian Optimization can be implemented giving 3 three main parameters to the function fmin().

**Objective Function :**defines the loss function to minimize.**Domain Space :**defines the range of input values to test (in Bayesian Optimization this space creates a probability distribution for each of the used Hyperparameters).**Optimization Algorithm:**defines the search algorithm to use to select the best input values to use in each new iteration.

A Trials() object is first created to make possible to visualize later what was going on while the **fmin()** function was running (eg. how the loss function was changing and how to used Hyperparameters were changing).

from hyperopt import hp, fmin, tpe, STATUS_OK, Trials space = {'criterion': hp.choice('criterion', ['entropy', 'gini']), 'max_depth': hp.quniform('max_depth', 10, 12, 10), 'max_features': hp.choice('max_features', ['auto', 'sqrt','log2', None]), 'min_samples_leaf': hp.uniform ('min_samples_leaf', 0, 0.5), 'min_samples_split' : hp.uniform ('min_samples_split', 0, 1), 'n_estimators' : hp.choice('n_estimators', [10, 50]) } def objective(space): model = RandomForestClassifier(criterion = space['criterion'], max_depth = space['max_depth'], max_features = space['max_features'], min_samples_leaf = space['min_samples_leaf'], min_samples_split = space['min_samples_split'], n_estimators = space['n_estimators'], ) accuracy = cross_val_score(model, X_Train, Y_Train, cv = 4).mean() # We aim to maximize accuracy, therefore we return it as a negative value return {'loss': -accuracy, 'status': STATUS_OK } trials = Trials() best = fmin(fn= objective, space= space, algo= tpe.suggest, max_evals = 20, trials= trials) best

Here `'criterion' : 1`

means criterion= “gini” (object at 1th index) and similarly for `max_features`

and `n_estimators`

.

crit = {0: 'entropy', 1: 'gini'} feat = {0: 'auto', 1: 'sqrt', 2: 'log2', 3: None} est = {0: 10, 1: 50, 2: 75, 3: 100, 4: 125}

## 4. Hyperparameter tuning using Optuna

Optuna uses a historical record of trails details to determine the promising area to search for optimizing the hyperparameter and hence finds the optimal hyperparameter in a minimum amount of time. It has the pruning feature which automatically stops the unpromising trails in the early stages of training. another key features are:

**Eager search spaces:**Automated search for optimal hyperparameters using Python conditionals, loops, and syntax.**State-of-the-art algorithms:**Efficiently search large spaces and prune unpromising trials for faster results.**Easy parallelization:**Parallelize hyperparameter searches over multiple threads or processes without modifying code

import sklearn import sklearn.datasets import sklearn.ensemble import sklearn.model_selection import sklearn.svm import optuna # 1. Define an objective function to be maximized. def objective(trial): iris = sklearn.datasets.load_iris() x, y = iris.data, iris.target # 2. Suggest values for the hyperparameters using a trial object. classifier_name = trial.suggest_categorical('classifier', ['SVC', 'RandomForest']) if classifier_name == 'SVC': svc_c = trial.suggest_loguniform('svc_c', 1e-10, 1e10) classifier_obj = sklearn.svm.SVC(C=svc_c, gamma='auto') else: rf_max_depth = int(trial.suggest_loguniform('rf_max_depth', 2, 32)) classifier_obj = sklearn.ensemble.RandomForestClassifier(max_depth=rf_max_depth, n_estimators=10) return accuracy # 3. Create a study object and optimize the objective function. study = optuna.create_study(direction='maximize') study.optimize(objective, n_trials=100)

I hope now it’s clear the features and advantages of each method and the best method will be dependent on the scenario; model and the dataset.

If you are interested in more things data, The Click Reader provides you a large collection of data science resources to study from—check it out by clicking here.