Hyperparameter Tuning of Tensorflow Model
I've used Scikit-learn GridSearchCV before optimizing my models' hyperparameters, but just wondering if there is a similar hyperparameter optimization tool for Tensorflow (e.g. number of epochs, learning rate, sliding window size, etc .. )
And if not, how can I implement a snippet that effectively triggers all the different combinations?
source to share
Another viable (and documented) option for grid search with Tensorflow is Ray Tune . It is a scalable framework for hyperparameter tuning, especially for deep learning / reinforcement learning.
You can try a quick tutorial here .
It also takes care of Tensorboard logging and efficient search algorithms (like HyperOpt
integrations and HyperBand ) in about 10 lines of Python.
import ray
from ray import tune
def train_tf_model(config, tune_reporter): # 1 new arg for reporting results
# ... train here ....
# ... train here ....
# ... train here ....
pass
ray.init()
tune.run(train_tf_model,
stop={ "mean_accuracy": 100 },
config={
"alpha": tune.grid_search([0.2, 0.4, 0.6]),
"beta": tune.grid_search([1, 2]),
})
(Disclaimer: I am actively involved in this project!)
source to share
Even though it is not explicitly documented (in version 1.2), the package tf.contrib.learn
(included in TensorFlow) defines classifiers that must be compatible with scikit-learn ... However, looking at the source , it seems you need to explicitly specify the environment variable TENSORFLOW_SKLEARN
(for example , before "1"
) to get this compatibility. If it works, you can already use it GridSearchCV
( see this test case ).
However, there are several alternatives. I don't know of any specific for TensorFlow, but hyperopt , Scikit-Optimize, or SMAC3 should be valid parameters. MOE and Spearmint look like a good choice, but now don't seem to be supported.
Alternatively you can look at a service like SigOpt (the company behind the original MOE version).
Edit
About running all the possible combinations of parameters, the basic logic, if you want to implement it yourself, isn't really hard. You can simply define lists of possible values for each parameter and then execute all combinations with itertools.product
. Something like:
from itertools import product param1_values = [...] param2_values = [...] param3_values = [...] for param1, param2, param3 in product(param1_values, param2_values param3_values): run_experiment(param1, param2, param3)
Note, however, that grid searches can be prohibitively expensive in many cases, and even performing a random search in the parameter space is likely to be more efficient (more on this in this post ).
source to share