Parse a string into an array based on spaces or "double quote strings"

A function strtok

is a terrible way to tokenize in C, except for one (common) case: simple words separated by spaces. (Even then it is still small due to the lack of re-entry and recursion capability, which is why we invented strsep

for BSD back when.)

It is best in this case to build your own simple state apparatus:

char *p;
int c;
enum states { DULL, IN_WORD, IN_STRING } state = DULL;

for (p = buffer; *p != '\0'; p++) {
    c = (unsigned char) *p; /* convert to unsigned char for is* functions */
    switch (state) {
    case DULL: /* not in a word, not in a double quoted string */
        if (isspace(c)) {
            /* still not in a word, so ignore this char */
            continue;
        }
        /* not a space -- if it a double quote we go to IN_STRING, else to IN_WORD */
        if (c == '"') {
            state = IN_STRING;
            start_of_word = p + 1; /* word starts at *next* char, not this one */
            continue;
        }
        state = IN_WORD;
        start_of_word = p; /* word starts here */
        continue;

    case IN_STRING:
        /* we're in a double quoted string, so keep going until we hit a close " */
        if (c == '"') {
            /* word goes from start_of_word to p-1 */
            ... do something with the word ...
            state = DULL; /* back to "not in word, not in string" state */
        }
        continue; /* either still IN_STRING or we handled the end above */

    case IN_WORD:
        /* we're in a word, so keep going until we get to a space */
        if (isspace(c)) {
            /* word goes from start_of_word to p-1 */
            ... do something with the word ...
            state = DULL; /* back to "not in word, not in string" state */
        }
        continue; /* either still IN_WORD or we handled the end above */
    }
}

      

        
        
        
      

    

Note that this does not take into account the possibility of a double quote inside a word, for example:

"some text in quotes" plus four simple words p"lus something strange"

      

        
        
        
      

    

Work on the state machine above and you will see what "some text in quotes"

turns into one token (which ignores double quotes), but is p"lus

also the only token (including the quote), something

one token, and strange"

is a token. Whether you want it or how you want to handle it is up to you. For more complex but thorough lexical tokenization, you can use a code builder tool like flex

.

Also, when the loop for

finishes, if state

not DULL

, you need to process the last word (I left that outside the code above) and decide what to do if state

IN_STRING

(which means there was no double double quote).