Step through a very large directory tree with find
Is find
bash lazy?
I am trying to go through a directory tree to get all files *.jpg
that have at least 700k files in a 3 level directory tree using the following script:
for im in $(find $1 -name '*.jpg');
do
echo im;
# Do something with im...
done
but it echo
takes a long time to print anything from and I'm sure the script works (I tested it with 50k files with 3 levels of directory tree and takes less time, but it prints everything in the end).
Maybe there is a lazy version of find or something I could use to show those echo
while the script is running.
source to share
$()
will highlight all output at find
once. Use a while read
process replace loop (recommended with bash) or pipe instead :
while IFS= read -r im; do
...
done < <(exec find "$1" -name '*.jpg')
exec
- my style is optional. Since we only run one command inside the subshell caused by the process swap, the other fork may not be needed.
or
find "$1" -name '*.jpg' | while IFS= read -r im; do
# Unfortunately anything that happens here is already inside
# a subshell so any variable changes would not affect the
# parent shell.
...
done
If you have incorrect filenames, such as filenames that have a newline, use \x00
as delimiter:
while IFS= read -r -d '' im; do
...
done < <(exec find "$1" -name '*.jpg' -print0)
If one of your commands inside the loop reads input, a different file descriptor will need to be used to prevent these commands from reading input from find
:
while IFS= read -ru 4 -d '' im; do
...
done 4< <(exec find "$1" -name '*.jpg' -print0)
source to share