Why does FOR / F set empty values ​​for repeated numbers in the rest of the tokens?

While I don't know why the team for

behaves the way it is, there are some simple rules that fit the behavior for

. And here we are only talking about the proposal token

. delims

, eol

, skip

And usebackq

on another day

Step 1 . Found a token condition. The paragraph is parsed and for each requested range (only one, initial, *

) it is determined if it is valid. It is discarded if it is not a valid request (not in the range 1-31 or not *

), but if it is a valid request, for each item requested, a "variable" (possibly a table) is allocated to subsequently hold the data obtained for that token. At the same time, a "set" (possibly a bitmap mask) is defined, specifying that the token number x (the number used to identify the token in the sentence tokens

) will be issued . The same token can be requested multiple times, but in "set" (or bitmask ...) the only effect is to re-indicate that token x will be issued.

Now "set" contains the position of the valid (1-31, *

) tokens that were requested.

After the parser finishes processing the configuration for

, the input file is read into memory, or a command is executed to fetch all of its output into memory, or a literal string is declared as an input buffer.

Step 2 . Prepare the parsing. The table for storing token data is initialized with spaces, and the pointer is set to the first position in the table (first token). If the line was not discarded with skip

, eol

or because it is empty, the tokenizer will scan the input buffer for tokens, otherwise it will search at the end of the line and repeat step 2 for the newline found.

Step 3 . Parse the input buffer. Until the end of the line is reached, for each token found in the line, its position, if in the range (1-31 or *

token), is checked for "set" to determine whether it was requested or not (if this token is in set or token is executed *

). If he was requested, is his data included in the "table"? at position indicated by a table pointer, the pointer is incremented, and the tokenizer continues to repeat step 3 until the end of the string is reached.

Step 4 . The end of the line has been reached. If any token was retrieved or if the only token requested is *

(test for /f "tokens=*" %a in (" ") do echo %a

), execute the code in the sentence do

.

Step 5 . If the shutter speed has for

not been canceled and the end of the buffer has not been reached, there are more lines to process, return to step 2.

This set of steps reproduces all of the observed behaviors in the question, but does not prove if this is the way to code the command for

.

Now let's test it against the code in the question

:::this prints - 1:[i] 2:[] 3:[] 4:[] 5:[] 6:[] 7:[]
for /f "tokens=1,1,1,1,1,1,1" %%a in ("i ii iii iv v vi vii") do (
    @echo 1:[%%a] 2:[%%b] 3:[%%c] 4:[%%d] 5:[%%e] 6:[%%f] 7:[%%g]
)

      

        
        
        
      

    

7 tokens requested, so 7 positions in the table to be passed into the code do

, but the only token that matches "set" is number 1

:::this prints - 1:[i] 2:[ii] 3:[iii] 4:[iv] 5:[] 6:[] 7:[%g]
for /f "tokens=2,3,1-4" %%a in ("i ii iii iv v vi vii") do (
    @echo 1:[%%a] 2:[%%b] 3:[%%c] 4:[%%d] 5:[%%e] 6:[%%f] 7:[%%g]
)

      

        
        
        
      

    

6 requested tokens, 6 positions in the token table, and the "set" will only match 1,2,3,4

:::this prints - 1:[i] 2:[ii] 3:[iii] 4:[] 5:[] 6:[] 7:[%g]
for /f "tokens=1-3,1-3," %%a in ("i ii iii iv v vi vii") do (
    @echo 1:[%%a] 2:[%%b] 3:[%%c] 4:[%%d] 5:[%%e] 6:[%%f] 7:[%%g]
)

      

        
        
        
      

    

6 requested tokens, 6 positions in the table of tokens, and the "set" will only match 1,2,3

setlocal disableDelayedExpansion
for /f "tokens=1-31,1-31,1-31" %%! in (
"33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 "
) do (
 echo 1:[%%!-!]  30:[%%?-?] 31:[%%@-@] 32:[%%A-A] 33:[%%B-B] 34:[%%C-C] 35:[%%D-D] 36:[%%E-E] 37:[%%F-F] 38:[%%G-G] 90:[%%{-{] 
)

      

        
        
        
      

    

93 requested tokens, 93 positions allocated in the token table, "set" will only match items 1-31

edited more cases added to question

the maximum of empty tokens is 250

@echo off
for /f "tokens=1-31,1-31,1-31,1-31,1-31,1-31,1-31,1-31,1-31,1-31,1-31,1-31,1-31,1-31,1-31" %% in (
"1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1") do (
    echo 0x02-%%- 0x07-%%- 0xFE-%%-  0xFB-%%- 0xFA-%%- 
)

      

        
        
        
      

    

No, you can request as many tokens as you can. I've tested 1625 1-30

and an additional 31 (to make sure the parser keeps working) and it handles without issue. Probably the limit is the lengh line. You can query up to 50530 (aprox) tokens (repetition of 1-31, ... to reach the limit), but you are limited to getting valid data for the first 31 tokens and empty data for the rest of the items into a storage table that should retrieve items using one character in the pluggable parameter for

. Using %%^A

(0x01, Alt-001) as a swap parameter for

, you can query up to %%ÿ

(0xFF, Alt-255)