Summary.

All these special tokens [

, ]

, <<

, >>

coming out of the scanner as executable names. [

and are <<

defined to create a marktype object (so they are not statements per se, but are executable names defined in systemdict

where all statements live). ]

and are >>

defined as procedures or statements that execute just like any other procedure or statement. They use an operator counttomark

to locate the opening bracket. But all of these tokens are processed specifically by the scanner, which recognizes them without the surrounding spaces as they are part of its delimiter.

Details.

It all depends on when you look at it. Let me see what the interpreter does with these tokens. I'm going to illustrate this with a line, but it works exactly the same with a file.

So if you have an input line

([4 5 6]) cvx exec

      

        
        
        
      

    

cvx

makes an executable literal object. A program stream is a file object, also referred to as an executable file. exec

pushes an object onto the execution stack, where it is encountered by the interpreter at the next iteration of the internal interpreter's processing loop. When a program thread is executing, the executable object is the topmost one on the execution stack.

The interpreter is used token

to call the scanner. The scanner skips leading spaces, then reads all non-space characters up to the next delimiter, then tries to interpret the string as a number and cannot become an executable name. The parentheses are part of the delimiter set and are therefore called "self-delimiting". So the scanner reads one parenthesis character, stops reading because it is a delimiter, detects that it cannot be a number, so it gives the executable name.

Top of Exec Stack | Operand Stack
(4 5 6]) [        | 

      

        
        
        
      

    

Next, the interpreter loop executes all executable files (unless it's an array). Executing a token means loading it from a dictionary and then executing the definition if it is executable. [

is defined as an object -mark-

, just like a name mark

. This is not technically a statement or procedure, it is just a definition. The automatic download happens because the name comes out of the scanner with the flag set.

(4 5 6])  | -mark-

      

        
        
        
      

    

The scanner then prints 4, 5, and 6, which are numbers and go straight onto the operand stack. 6 is limited ]

, which is thrown back into the stream.

(])  | -mark- 4 5 6

      

        
        
        
      

    

The interpreter does not execute the numbers as they are not executable, but it would be exactly the same if it did. The action to execute a number is simply to push it onto the stack.

Then finally the scanner encounters a right parenthesis ]

. And that's where the magic happens. Self-delimited, it doesn't need to follow any spaces. The scanner gives the executable name ]

, and the interpreter executes it on load and finds ...

{ counttomark array astore exch pop }

      

        
        
        
      

    

Or maybe the actual operator doing it. But yes. counttomark

gives the number of items. array

creates an array of that size. astore

fills the array with items from the stack. And exch pop

discard this annoying label once and for all.

For dictionaries, <<

exactly matches [

. He throws a mark. Then you group several key-value pairs, and >>

this is a procedure that does something action ...

{ counttomark dup dict begin 2 idiv { def } repeat pop currentdict end }

      

        
        
        
      

    

Make a dictionary. Identify all pairs. Check the box. Refine the dictionary. This version of the procedure tries to create a fast dictionary by making it double. Move 2 idiv

to dup

to make a small dictionary.

So, to get philosophical, counttomark

this is the operator you use. And this requires a special type of object that is not used for anything else, the object marktype, -mark-

. The rest is just syntactic sugar, allowing you to access this stack-counting ability to create linear data structures.

application

Here's a procedure that simulates reading the interpreter loop from currentfile

.

{currentfile token not {exit} if dup type /arraytype ne {exec} if }loop

      

        
        
        
      

    

exec

is responsible for load

ing (and further executing) any executable names. You can see from this what the token

scanner name really is; and that procedures (arrays) directly encountered in the interpreter's loop are not executed ( type /arraytype ne {exec} if

).

Using token

in strings allows you to do really cool things. For example, you can dynamically construct the bodies of procedures with overridden names. This is very similar to the lisp macro.

/makeadder { % n  .  { n add }
    1 dict begin
    /n exch def
    ({//n add}) token % () {n add} true
    pop exch pop % {n add}
    end
} def

      

        
        
        
      

    

token

reads the entire procedure from a string, substituting the //n

currently evaluated name with its current value. Note that the scanner reads the executable array right away, effectively doing [

... ] cvx

internally before returning (in some interpreters, such as my own xpost

, this allows you to bypass the stack size restrictions before constructing the array, because the array is built in separate memory, but garbage collection level 2 makes this largely irrelevant).

There is also a statement bind

that modifies the procedure by replacing the statement names with the statement objects themselves. These tricks help you play out name queries in critical procedures (like inner loops).