Common Lisp reading from file and saving as a list
So, do some general lisp exercises and everything went well until I ran into this strange behavior. I have read text from file (brown.txt) in corpus variable and should be saved as a list. However, I suspect it is not, although sometimes it works as one, but other times it fails.
Here is a basic read from file -> append for list -> list of stores in corpus (split / tokenized in space):
(defun tokenize (string)
(loop
for start = 0 then (+ space 1)
for space = (position #\space string :start start)
for token = (subseq string start space)
unless (string= token "") collect token
until (not space)))
(defparameter *corpus*
(with-open-file (stream "./brown.txt" :direction :input)
(loop for line = (read-line stream nil)
while line
append (tokenize line))))
And below are two expressions that should work, but only the last one (body one). The first returns NIL.
(loop for token in *corpus* do
(print token))
*corpus*
I suspect it has something to do with reading from a file as a stream object and that (append ...) does not create a list from that stream, but instead lazy waits until I want to later or sum it up, at which time it just decided not to work anymore? (makes little sense to me).
source to share