CLHS: Issue COMPILE-FILE-HANDLING-OF-TOP-LEVEL-FORMS Writeup

#

## Issue COMPILE-FILE-HANDLING-OF-TOP-LEVEL-FORMS Writeup

    Forum:		Compiler
    Issue:		COMPILE-FILE-HANDLING-OF-TOP-LEVEL-FORMS
    References:	CLtL pages 66-70, 143
    Category:	CLARIFICATION
    Edit history:   V1, 07 Oct 1987 Sandra Loosemore
                    V2, 15 Oct 1987 Sandra Loosemore
                    V3, 15 Jan 1988 Sandra Loosemore
    		V4, 06 May 1988 Sandra Loosemore
    		V5, 20 May 1988 Sandra Loosemore
    		V6, 09 Jun 1988 Sandra Loosemore
    		V7, 16 Dec 1988 Sandra Loosemore
    			(Comments from Pitman, change DEFCONSTANT, etc.)
    		V8, 31 Dec 1988 Sandra Loosemore
    			(CLOS additions, etc.)
    		V9, 23 Jan 1989 Sandra Loosemore
    			(remove the CLOS additions again)
    Status: Proposal CLARIFY passed Jan 89

    Problem Description:

    Standard programming practices assume that, when calls to defining
    macros such as DEFMACRO and DEFVAR are processed by COMPILE-FILE,
    certain side-effects occur that affect how subsequent forms in the
    file are compiled.  However, these side-effects are not mentioned in
    CLtL, except for a passing mention that macro definitions must be
    ``seen'' by the compiler before it can compile calls to those macros
    correctly.  In order to write portable programs, users must know
    exactly which defining macros have compile-time side-effects and what
    those side-effects are.

    Inter-file compilation dependencies are distinct from, and not
    addressed by, this issue.

    Proposal: COMPILE-FILE-HANDLING-OF-TOP-LEVEL-FORMS:CLARIFY

    (1) Clarify that defining macros such as DEFMACRO or DEFVAR, appearing
        within a file being processed by COMPILE-FILE, normally have
        compile-time side effects which affect how subsequent forms in the
        same file are compiled.  A convenient model for explaining how these
        side effects happen is that the defining macro expands into one or
        more EVAL-WHEN forms, and that the calls which cause the compile-time
        side effects to happen appear in the body of an (EVAL-WHEN (COMPILE)
        ...) form.

    (2) The affected defining macros and their specific side effects are
        as follows.  In each case, it is identified what users must do to
        ensure that their programs are conforming, and what compilers must do
        in order to correctly process a conforming program.

        DEFTYPE: Users must ensure that the body of a DEFTYPE form is
        evaluable at compile time if the type is referenced in subsequent type
        declarations.  The compiler must ensure that the DEFTYPE'd type
        specifier is recognized in subsequent type declarations.  If the
        expansion of a type specifier is not defined fully at compile time
        (perhaps because it expands into an unknown type specifier or a
        SATISFIES of a named function that isn't defined in the compile-time
        environment), an implementation may ignore any references to this type
        in declarations and/or signal a warning.

        DEFMACRO, DEFINE-MODIFY-MACRO: The compiler must store macro
        definitions at compile time, so that occurrences of the macro later on
        in the file can be expanded correctly.  Users must ensure that the
        body of the macro is evaluable at compile time if it is referenced
        within the file being compiled.

        DEFUN: DEFUN is not required to perform any compile-time side effects.
        In particular, DEFUN does not make the function definition available
        at compile time.  An implementation may choose to store information
        about the function for the purposes of compile-time error-checking
        (such as checking the number of arguments on calls), or to enable the
        function to be expanded inline.

        DEFVAR, DEFPARAMETER: The compiler must recognize that the variables
        named by these forms have been proclaimed special.  However, it must
        not evaluate the initial value form or SETQ the variable at compile
        time.

        DEFCONSTANT: The compiler must recognize that the symbol names a
        constant.  An implementation may choose to evaluate the value-form at
        compile time, load time, or both.  Therefore users must ensure that
        the value-form is evaluable at compile time (regardless of whether or
        not references to the constant appear in the file) and that it always
        evaluates to the same value.

        DEFSETF, DEFINE-SETF-METHOD: The compiler must make SETF methods
        available so that it may be used to expand calls to SETF later on in
        the file.  Users must ensure that the body of DEFINE-SETF-METHOD and
        the complex form of DEFSETF are evaluable at compile time if the
        corresponding place is referred to in a subsequent SETF in the same
        file.  The compiler must make these SETF methods available to
        compile-time calls to GET-SETF-METHOD when its environment argument is
        a value received as the &ENVIRONMENT parameter of a macro.

        DEFSTRUCT: The compiler must make the structure type name recognized
        as a valid type name in subsequent declarations (as for DEFTYPE) and
        make the structure slot accessors known to SETF.  In addition, the
        compiler must save enough information about the structure type so that
        further DEFSTRUCT definitions can :INCLUDE a structure type defined
        earlier in the file being compiled.  The functions which DEFSTRUCT
        generates are not defined in the compile time environment, although
        the compiler may save enough information about the functions to code
        subsequent calls inline.  The #S reader syntax may or may not be
        available at compile time.

        DEFINE-CONDITION: The rules are essentially the same as those for
        DEFSTRUCT; the compiler must make the condition type recognizable as a
        valid type name, and it must be possible to reference the condition
        type as the parent-type of another condition type in a subsequent
        DEFINE-CONDITION in the file being compiled.

        DEFPACKAGE:  All of the actions normally performed by this macro at load
        time must also be performed at compile time.

    (3) The compile-time side effects may cause information about the
        definition to be stored differently than if the defining macro had
        been processed in the "normal" way (either interpretively or by loading
        the compiled file).

        In particular, the information stored by the defining macros at
        compile time may or may not be available to the interpreter (either
        during or after compilation), or during subsequent calls to COMPILE or
        COMPILE-FILE.  For example, the following code is nonportable because
        it assumes that the compiler stores the macro definition of FOO where
        it is available to the interpreter:

            (defmacro foo (x) `(car ,x))
            (eval-when (eval compile load)
                (print (foo '(a b c))))

        A portable way to do the same thing would be to include the macro
        definition inside the EVAL-WHEN:

            (eval-when (eval compile load)
                (defmacro foo (x) `(car ,x))
                (print (foo '(a b c))))

    Rationale:

    The proposal generally reflects standard programming practices.  The
    primary purpose of the proposal is to make an explicit statement that
    CL supports the behavior that most programmers expect and many
    implementations already provide.

    The primary point of controversy on this issue has been the treatment
    of the initial value form by DEFCONSTANT, where there is considerable
    variance between implementations.  The effect of the current wording is
    to legitimize all of the variants.

    Current Practice:

    Many (probably most) Common Lisp implementations, including VaxLisp
    and Lucid Lisp, are already largely in conformance.

    In VaxLisp, macro definitions that occur as a side effect of compiling
    a DEFMACRO form are available to the compiler (even on subsequent
    calls to COMPILE or COMPILE-FILE), but are not available to the
    interpreter (even within the file being compiled).

    By default, Kyoto Common Lisp evaluates *all* top level forms as they
    are compiled, which is clearly in violation of the behavior specified
    on p 69-70 of CLtL.  There is a flag to disable the compile-time
    evaluation, but then macros such as DEFMACRO, DEFVAR, etc. do not make
    their definitions available at compile-time either.

    Cost to implementors:

    The intent of the proposal is specifically not to require the compiler
    to have special knowledge about each of these macros.  In
    implementations whose compilers do not treat these macros as special
    forms, it should be fairly straightforward to use EVAL-WHENs in their
    expansions to obtain the desired compile-time side effects.

    Cost to users:

    Since CLtL does not specify whether and what compile-time side-effects
    happen, any user code which relies on them is, strictly speaking,
    nonportable.  In practice, however, most programmers already expect
    most of the behavior described in this proposal and will not find it
    to be an incompatible change.

    Benefits:

    Adoption of the proposal will provide more definite guidelines on how
    to write programs that will compile correctly under all CL
    implementations.

    Discussion:

    Reaction to a preliminary version of this proposal on the common-lisp
    mailing list was overwhelmingly positive.  More than one person
    responded with comments to the effect of "but doesn't CLtL already
    *say* that somewhere?!?"  Others have since expressed a more lukewarm
    approval.

    It has been suggested that this proposal should also include PROCLAIM.
    However, since PROCLAIM is not a macro, its compile-time side effects
    cannot be handled using the EVAL-WHEN mechanism.  A separate proposal
    seems more appropriate.

    Item (3) allows for significant deviations between implementations.
    While there is some sentiment to the effect that the compiler should
    store definitions in a manner identical to that of the interpreter,
    other people believe strongly that compiler side-effects should be
    completely invisible to the interpreter.  The author is of the opinion
    that since this is a controversial issue, further attempts to restrict
    this behavior should be considered as separate proposals.

    It should be noted that user-written code-analysis programs must
    generally treat these defining macros as special forms and perform
    similar "compile-time" actions in order to correctly process
    conforming programs.

Copyright 1996-2005, LispWorks Ltd. All rights reserved.