Clever Geek Handbook

TensorFlow: How to check for bottlenecks in the data entry pipeline?

I am currently using tf-slim to create and read tfrecord files in my models and thanks to this method an automatic tensor rendering is available showing:

  • A visualization of tf.train.batch batch/fraction_of_32_full

    that is consistently close to 0. I believe it should depend on how fast the dequeue operation gives tf.train.batch a FIFO of its tensor queue.

  • Parallel readers parallel_read/filenames/fraction_of_32_full

    and paralell_read/fraction_of_5394_full

    visualizations, which are always 1.0. I believe this op is what extracts tensors from tfrecords and puts them in a queue ready for decoding.

My question is this: is my dequeuing operation too slow and causing a bottleneck in my model evaluation?

Why is "fraction_of_32" coming up even though I'm using a batch size of 256? Also, is a queue share of 1.0 ideal? As this would mean the data is always ready for GPU work.

If my dequeueing operation is too slow, how can I improve my deletion speed? I checked the source code for tf-slim and it seems that the decoder is built into the function I'm using and I'm not sure if there is an external way for it to work.

+3


source to share


1 answer


I had a similar problem. If the / fraction _of_32_full packet approaches zero, it means that you are consuming data faster than you are creating it.

32 is the default queue size, regardless of the batch size. It is advisable to set it at least as a batch size.



This is the relevant doc: https://www.tensorflow.org/api_docs/python/tf/train/batch

Customization num_threads = multiprocessing.cpu_count()

and capacity = batch_size

can help keep the queue full.

0


source






More articles:


All Articles