LispMacroDiscussion got too big for me to be able to edit it, so I moved the last little bit over here, because I'm too tired to do the right thing. We might be getting closer to a conclusion, though, so I'm hoping that we'll be able to refactor this mess away soon. -- AdamSpitz ---- [Answer to Adam's questions; after this, RefactorMe or DeleteMe -- AlainPicard] Adam -- thanks. Now I think I've gotcha. I was wrong after all; you don't understand ''read time''. Well, you understand it ''technically'', but you don't grok it. No, you cannot step through a macro in the debugger, because the macro is ''gone'', gone forever. The macro is a construct which is explicitly meant to take action at READ time '''only'''. Yes, you don't need macros. You can just use lambda everywhere, and get something which is (as near as I can tell from your description) equivalent to SELF (which looks like a cool language, btw). Lispers use macros ''anyway'' because it is the construct which allows them to build a new language in which to describe and express their problems. Macros are about building new ''syntax'' into the language (which is fully on par with the original language. This is important, because the seamlessness between the user's extensions and the original language allows even more future, regular extensions.) Lisper's aren't worried that they cannot mapcar #'IF; it's not an operation which makes a lot of sense. Similarly, the macros they define don't make sense in a functional sense (otherwise they'd have been defined as functions) so the loss of "uniformity" is not perceived as a loss. Say you have an electrical engineering signals processing application. You might like to write things like: (transform signal -> low-pass-filter -> high-pass-filter -> notch-filter) This is just syntactic sugar, yes, but can you not see how this sugar makes it possible to write excruciatingly ''clear'' applications? The program is short, concise, and ''obviously correct'' because it has been expressed in a language close to the application domain. I dunno if I'm helping, so I'll stop now. :-) --AlainPicard Yes, I agree that that's very clear, and I value that. But are you sure that it's the syntactic sugar that's making your code clear? I think that the thing that makes your code clear is the ability to give names to all the concepts in your mind and represent those concepts by their names. Lisp gives you the ability to give names to the concepts of "transform", "signal", "low pass filter", "high pass filter", and "notch filter", and that's why the code is clear - it says exactly what you want to say, and doesn't say anything else. In Self, I might write: signal transform: lowPassFilter & highPassFilter & notchFilter (That's sending a message named "transform:" to the signal object, passing in a list of the three filter objects. The & message is just Self's idiom for constructing lists.) I might be tempted to define a -> method for signal objects, so that I could write: signal -> lowPassFilter -> highPassFilter -> notchFilter ...which I think is kinda cool, but not really the point; I think the first version is almost as readable. (Isn't it?) Self happens to give me the ability to define this particular kind of syntactic sugar, but the really big gains came from the ability to express the concepts of the problem directly in the language. Notice that there are two requirements here: you need to be able to give names to your concepts, and you need to be able to use those names to represent the concepts. Macros let me give a name to just about anything, but don't let me use that name in some situations. I'm willing to believe that this isn't a problem, but I don't yet understand why. How did it come about that all the concepts that are more succinctly expressed through macros are concepts that don't need to be used as first-class values at runtime? -- AdamSpitz I'm not sure the first version is as readable. This example seems to go to the crux of the matter, as I understand it. This form: (transform signal -> low-pass-filter -> high-pass-filter -> notch-filter) Seems to me to read a lot like a description of what some actual signal processing hardware might look like, in that sense it is close to the domain. The ->s might transform into some explicit list-building code, or they might not, who cares? The point is exactly that this is a new language with new syntax, where -> means connect the output of the signal processing block on the left to the input of the block on the right. Whereas the form: signal transform: lowPassFilter & highPassFilter & notchFilter Definitely isn't a new language, it's still Self, and to use it requires understanding of building explicit lists and sending messages to objects, which is not very much like assembling signal processing hardware at all. I guess you either think that the Lisp macro approach here is a win or you don't. Then the form: signal -> lowPassFilter -> highPassFilter -> notchFilter is a lot more than just cute, it gets closer to the "looking like the hardware might" property, which again you either think is a win or not. But it still isn't a new language, and to understand it means understanding sending the signal object the message -> with an argument, the object you get from sending the message -> to the object... and so on. Again, not much like plugging together the hardware. Now, Lisp supports a bunch of computational styles already, functional, procedural and so on, any one of which can of course be used to model any domain you like, but Lisp macros let you introduce new styles than map most directly onto the problem in hand. Signal processing in hardware is probably closer to functional programming than the others, but that doesn't necessarily mean that a functional style program most clearly expresses a given signal processing problem. But the signal-processing style language that the macros let us build can most clearly express a given signal processing problem. Macros or not isn't much of a formal, computer science type issue, it's a pragmatic, software engineering type issue. -- KeithBraithwaite There are some interesting thought patterns emerging here. Keith, the way you talked about object-oriented programming ("sending the signal object the message -> with an argument, the object you get from sending the message -> to the object... and so on") sounds very weird to me. That's not the way I read it at all. When I read that code, my brain doesn't use the words "object" or "message" or "argument". I don't think of "signal" as "a Self object representing the signal" - I think of it as "the signal". And "->" means "connect it to", and "lowPassFilter" means, "the low pass filter". So I read that code as, "Connect the signal to the low pass filter," which is remarkably similar to what you said (substituting "the signal" and "the low pass filter" for "the block on the left" and "the block on the right"). I was surprised when you said, "not much like plugging together the hardware," because I was thinking that it was ''exactly'' like plugging together the hardware. "signal" and "lowPassFilter" are the hardware, and "->" is how I plug them together. I think you're looking at the language, and I'm looking at the objects. I think that Lisp doesn't make the objects real enough, and you think that Self doesn't make the language direct enough. Does that make any sense? If it does, then I think I understand macros better. If you think of a program as a bunch of sentences in a language, then you want that language to be as powerful as possible (and so macros make perfect sense). But if you think of a program as a world of interacting objects, then you want those objects to be as real as possible (and so a concept that's only real part of the time makes no sense at all). -- AdamSpitz Adam, that's an excellent observation. I don't know what you mean by objects not being ''real enough'', but I certainly do think of a program as a bunch of sentences, and I want to speak to the computer in a language that we're ''both'' comfortable with. FWIW, when I showed the TRANSFORM example, I had in mind a replacement for an expression like: (let ((output (apply-filter #'notch-filter (apply-filter #'low-pass-filter (apply-filter #'high-pass-filter signal))))) (do-stuff output)) and the TRANSFORM lets me "invert" the direction in which I need to read the arguments (in a traditional functional programming style). This way, I get functional application but I can "see" filters being processed in the way I mean to connect them. This is why your observation (of "a bunch of sentences") resonates strongly with me, although I, of course, see it as a profound change. The above looks like LISP. The (transform foo -> bar -> baz ...) doesn't; when I read it, I don't have to do any "mental mapping" between my problem domain and my implementation language; they have become one and the same. -- ap Quick, irrelevant question about that expression there: Couldn't you define a function called "transform" that takes a signal and a &rest parameter (is that the right term?), so that you could write: (transform signal #'high-pass-filter #'low-pass-filter #'notch-filter) That puts the filters in the proper order, and makes the code just as short as the macro version. (And you could still go ahead and write the macro to give you the cool -> notation, if you wanted to. It would just expand trivially into a call to this function.) In any case, though, I think I'm happy. Here are some questions I'd like to think more about, regarding this bunch-of-sentences/world-of-objects thing: 1. Does it make a difference who you are? (Maybe some people think better about bunches of sentences, and some people think better about worlds of objects.) 2. Does it make a difference what the problem is? (Maybe some programs make more sense as a bunch of sentences, and some programs make more sense as a world of objects.) 3. Is there a way to combine the two worldviews? Is there any reason to? See RealObject for my attempt to explain what I mean by "not real enough." -- AdamSpitz Let me suggest a few nice full-scale examples of what you can do with Lisp macros, which I think give the full flavour: * The book ParadigmsOfArtificialIntelligenceProgramming by PeterNorvig, which has a lovely PrologLanguage compiler implemented by macros (and lots of more). * ''The Definition and Implementation of a Programming Language Based on Constraints'' by GuySteele, which introduces constraint-based programming and incrementally builds a constraint-based programming language out of Lisp macros - it's wonderfully readable and does lots of very nice stuff just in passing. You can get this online at http://publications.ai.mit.edu/publications/pubsDB/pubs.html (AITR-595), though it's a 20MB scan and perhaps a challenge to print (but worth the effort! Even if just the first section.) Incidentally, this is also the best case-study in PiecemealGrowth I've ever seen. * OnLisp by PaulGraham which is all about macros and is now available online, though I haven't read much of it. -- LukeGorrie ---- Adam, you may be interested in "First Class Macros have Types" http://citeseer.nj.nec.com/494837.html.