How to skip first n objects in jq input
I have a VERY large stream of objects that I am trying to import into MongoDB. I keep getting a broken pipe after about 10,000 objects, so I would like to update my import script to skip the already imported objects and start from the first one that was skipped.
It seems to me that the tool for this would be jq. What I need is a way to skip (output empty) all elements up to the nth, and then output the rest as-is.
I tried using foreach to maintain an object count, but I end up all the time with 1 count value for all objects in my little test case (using the bash doc here):
$ jq 'foreach . as $item (0; (.+1); [ . , if . < 2 then empty else $item end ])' <<"end"
> { "item": "first" }
> { "item": "second" }
> { "item": "third" }
> { "item": "fourth" }
> end
Way out of this:
[
1
]
[
1
]
[
1
]
[
1
]
Any suggestions would be most welcome.
source to share
def skip(n; stream):
foreach stream as $s (0; .+1; select(. > n) | $s);
Example:
skip(1000; inputs)
(When using inputs
and / or input
don't forget that you probably want to use the -n command line option.)
Sledgehammer Approach
try (range(0; 1000) | input | empty), inputs
In this case try
, an error must be avoided if the number of requests is less than the requested number of items.
source to share