Avoid certain parameter combinations in GridSearchCV
I am using scikit-learn GridSearchCV
to iterate over a parameter space to tune a model. Specifically, I use it to test various hyperparameters in a neural network. The grid looks like this:
params = {'num_hidden_layers': [0,1,2],
'hidden_layer_size': [64,128,256],
'activation': ['sigmoid', 'relu', 'tanh']}
The problem is I am running redundant models when hidden is num_hidden_layers
set to 0
. It will run a model with 0 hidden layers and 64 units, another with 128 units and another with 256 units. All of these models are equivalent since there is no hidden layer. This is very inefficient and means I need to write more code to remove redundancy in the results.
Is there a way to prevent such combinations of parameters, perhaps by passing a tuple of parameters?
source to share
GridSearchCV allows you to pass a list of dictionaries to parameters:
param_grid: dict or list of dictionaries
A dictionary with parameter names (string) as keys and lists of parameter parameters to try as values or a list of such dictionaries, in which case the grids covered by each dictionary in the list are explored. This allows you to search through any sequence of settings.
Thus, you can specify these dictionaries as specific subtasks of the original dictionary. This way you can avoid unnecessary combinations.
source to share
sklearn documentation offers two parameter grids.
So, you can do something like this:
param_grid = [
{'num_hidden_layers': [1,2],
'hidden_layer_size': [64,128,256],
'activation': ['sigmoid', 'relu', 'tanh']},
{'num_hidden_layers': [0],
'hidden_layer_size': [64],
'activation': ['sigmoid', 'relu', 'tanh']}
]
source to share