How does the strtok function in C work?
c, strtok
Solution
Two things to know about `strtok`. As was mentioned, it "maintains internal state". Also, it messes up the string you feed it. Essentially, it will write a `'\0'` where it finds the token you supplied, and returns a pointer to the start of the string. Internally it maintains the location of the last token; and next time you call it, it starts from there.
The important corollary is that you cannot use `strtok` on a `const char* "hello world";` type of string, since you will get an access violation when you modify contents of a `const char*` string.
The "good" thing about `strtok` is that it doesn't actually copy strings - so you don't need to manage additional memory allocation etc. But unless you understand the above, you will have trouble using it correctly.
Example - if you have "this,is,a,string", successive calls to `strtok` will generate pointers as follows (the `^` is the value returned). Note that the `'\0'` is added where the tokens are found; this means the source string is modified:
t h i s , i s , a , s t r i n g \0 this,is,a,string
t h i s \0 i s , a , s t r i n g \0 this
^
t h i s \0 i s \0 a , s t r i n g \0 is
^
t h i s \0 i s \0 a \0 s t r i n g \0 a
^
t h i s \0 i s \0 a \0 s t r i n g \0 string
^
Hope it makes sense.
Problem
I found this sample program which explains the `strtok` function: ``` #include <stdio.h> #include <string.h> int main () { char str[] ="- This, a sample string."; char * pch; printf ("Splitting string \"%s\" into tokens:\n",str); pch = strtok (str," ,.-"); while (pch != NULL) { printf ("%s\n",pch); pch = strtok (NULL, " ,.-"); } return 0; } ``` However, I don't see how this is possible to work. How is it possible that `pch = strtok (NULL, " ,.-");` returns a new token. I mean, we are calling `strtok`with `NULL` . This doesen't make a lot sense to me.