How can I use very large (> 2M) word embedding in tensorflow?
I am running the model with very large word nesting (> 2M words). When I use tf.embedding_lookup it expects a matrix which is large. When I run, I subsequently exit the GPU memory error. If I decrease the size of the attachment, everything will be fine.
Is there a way to deal with a large investment?
source to share
The recommended way is to use a partitioner to overlay this large tensor into multiple parts:
embedding = tf.get_variable("embedding", [1000000000, 20],
partitioner=tf.fixed_size_partitioner(3))
This will split the tensor into 3 shards along the 0 axis, but the rest of the program will see it as a normal tensor. The biggest advantage is to use a parameter server separator and replication, for example:
with tf.device(tf.train.replica_device_setter(ps_tasks=3)):
embedding = tf.get_variable("embedding", [1000000000, 20],
partitioner=tf.fixed_size_partitioner(3))
The key feature is here tf.train.replica_device_setter
. It allows you to run 3 different processes called parameter servers that store all the variables of the model. The large tensor embedding
will be split across these servers like in this image.
source to share