Parse nginx log and extract IP address, locate geodata for each parsed IP address
I have a question about parsing the nginx log,
have a bug in this code <<< grep "pagename" <<< "$line0"
does not work grep "pagename"
while IFS= read -r line0
do
ipList=$( grep -oP '\b(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))\b'
<<< grep "pagename" <<< "$line0")
for oneIP in $ipList
do
curl -s "http://ipinfo.io/$oneIP"
done
done < /var/log/nginx/access.log
Thank.
source to share
The syntax <<< grep "pagename" <<< "$line0"
is close, <<<X
injects string X into stdin.
The syntax you want is process substitution syntax (combined with a redirect arrow):
< <(grep "pagename" <<< "$line0")
There's also a simpler solution here:
#! /bin/bash
while read -r oneIP; do
curl -s "http://ipinfo.io/$oneIP"
done < <(grep "pagename" /var/log/nginx/access.log \
| grep -oP '\b(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))\b')
The result of the first grep
is piped to the second, and then the ips stream is then redirected to while
-loop, which starts a curl for each of the ips.
You can also parallelize this with gnu parallel :
#! /bin/bash
grep "pagename" /var/log/nginx/access.log \
| grep -oP '\b(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))\b' \
| parallel echo curl -s "http://ipinfo.io/"{}
source to share