Correct method to combine Attention, Multi, Remaining cell wrappers in 1.1.0-rc2

I am trying to combine the following:

tf.contrib.rnn.AttentionCellWrapper
tf.contrib.rnn.MultiRNNCell
tf.contrib.rnn.ResidualWrapper
tf.contrib.rnn.LSTMCell

      

I am building a cell like this

  cell = tf.contrib.rnn.AttentionCellWrapper(
    tf.contrib.rnn.MultiRNNCell([
    tf.contrib.rnn.ResidualWrapper(
      cell=tf.contrib.rnn.LSTMCell(dec_units))
      for _ in range(dec_layers)]),
      attn_length=attn_len)

      

This works great if I keep attn_len small (1-2), but increasing attn_len to a larger value (5+) makes the script hang indefinitely with one CPU core tied to 100% at the start of training (0 steps completed).

Is this an appropriate way to combine these elements? Should I override the default values ​​for optional parameters?

+3


source to share





All Articles