Are Lisp source code files themselves lists?

Question

No matter the Lisp dialect, it looks like every source code file containing Lisp functions isn't itself a list (the first time I was "surprised" by this was when working on my Emacs .el files). I've got a few questions but they're all related to the same "issue" and it's probably just me misunderstanding a few things. Is there a reason why source code files for the various Lisp dialects seems to be a bunch of "disorganized" functions like this: ``` (function1 ...) (function2 ...) (function3 ...) ``` Instead of a "Lisp list" of functions, maybe like this: ``` ( '(function1 ...) '(function2 ...) '(function3 ...) ) ``` I'm a bit surprised in this whole "code is data, data is code" thing to see that source code file themselves apparently aren't neat lists... Or are they!? Are the source code files something you're supposed to "manipulate" or not? What if I wanted to, say, convert one of my .clj (Clojure) source file to some CSS+HTML webpage, isn't it a "problem" that the source code file…

Accepted Answer

In Common Lisp a source file contains `lisp forms` and comments. Lisp forms are either data or Lisp code. Typical operations on a source file are done by the functions `LOAD` and `COMPILE-FILE`.

`LOAD` would read forms from a file and execute them one by one.

`COMPILE-FILE` is much more complex. It typically reads forms and compiles them to some other representation (machine code, byte code, C code, ...). It does not execute the code.

What would it help you if the file contain one list of forms instead of just multiple forms below each other?

- you would have one level of added parentheses

- you would have to read the whole list before you can do anything with it (or alternatively you need a different reader mechanism)

- adding forms to the end of a file by a program would be a pain

- you can't add something into the file which changes the reader interpretation of the rest of the file

- files can't be infinitively long for LOAD

Now for a example a compiler would read lisp forms from a file stream and compile them piece by piece.

If you want all forms you can do

```
CL-USER 170 > (defun read-forms (file)
               (with-open-file (stream file)
                 (loop for form = (read stream nil nil)
                       while form
                       collect form)))
READ-FORMS

CL-USER 171 > (read-forms (capi:prompt-for-file "source file"))
((DEFPARAMETER *UNITS-TO-SHOW* 4.1)
 (DEFPARAMETER *TEXT-WIDTH-IN-PICAS* 28.0)
 (DEFPARAMETER *DEVICE-PIXELS-PER-INCH* 300)
 (DEFPARAMETER *PIXELS-PER-UNIT* (* (/ (/ *TEXT-WIDTH-IN-PICAS* 6)
                                       (* *UNITS-TO-SHOW* 2))
                                    *DEVICE-PIXELS-PER-INCH*))
...
```

If you want to put parentheses around everything use `PROGN`:

```
 (progn
   'form-1
   (defun function-defintion-form () )
   42)
```

`PROGN` preserves also the 'top-level-ness' of its sub forms.

Side note: alternatives to this have been explored in Lisp for decades. The most prominent example is the now defunct Interlisp-D from Xerox. Interlisp-D was developed in parallel to Smalltalk by Xerox PARC. Interlisp-D originally used an structure editor to edit Lisp data and the source code was edited as such Lisp data. The development environment was based on this idea. But in the long run the 'source as text' won. Still you can emulate some of that in many current Lisp environments. For example many Lisp systems allow to write an 'image' of the current execution memory - this image includes all data and all code (also the compiled code). So you can work on this data/code and save an image from time to time.

Are Lisp source code files themselves lists?

Solution

Problem