wingologA mostly dorky weblog by Andy Wingo2012-08-15T21:34:25Ztekutihttps://wingolog.org/feed/atomAndy Wingohttps://wingolog.org/quasiconf 2012: lisp @ frosconhttps://wingolog.org/2012/08/15/quasiconf-2012-lisp-froscon2012-08-15T21:34:25Z2012-08-15T21:34:25Z

Are you a Lisper, in the big-tent sense of the term? You within a day's transport of St. Augustin in Germany? Well shucks, have I got a thing for you: Quasiconf, a lisp sub-conference of FrOSCon, held next weekend (25-26 August 2012).

The full program isn't quite out yet, but they invited yours truly to talk there, and talk I will. In fact I'll talk thrice: they gave me 90 minutes, so I'll do three little presentations.

In the first, I'll finally give that talk I've been threatening to do for years, about why delimited continuations are the bee's knees, how they can be implemented, and what kind of lovely code they enable. The example will be a small memcached server and client.

Next I'll talk about things that implementing JavaScript has taught me about Scheme implementations. They were very surprising lessons to me, so I hope they will entertain the attendees. No spoilers, though!

Finally, now that Guile 2.0 has been out for a year and a half or so, it's a good time to look back at what worked well and what didn't. If you are even remotely interested about what it takes to maintain and enhance a mature language implementation, this will be an interesting talk for you. I'll also take the opportunity to summarize the progress on the development branch, which will probably be Guile 2.2.

So that's the thing. It's somewhat of a late announcement, but hey, if you can go, it's pretty much free and should be an interesting get-together. See you there!

Andy Wingohttps://wingolog.org/international lisp conference 2009, day three, part 1https://wingolog.org/2009/03/25/international-lisp-conference-day-three-part-2009-03-25T14:49:34Z2009-03-25T14:49:34Z

Another day, another parenthetical note. It's probably worth repeating that these are just personal impressions, take them with a salt lick.

François-René Rideau spoke about XCVB, a packaging system for Common Lisp. Slides here. From what I gather, XCVB tries to make separate compilation of Lisp libraries more robust via mechanisms that isolate compile-time side-effects.

More interesting was where he headed at the end of his talk, and expanded on in a lightning talk: that the things that we do are intertwined with the stories we tell others and ourselves. The bad stories are about things. The better stories are about people making things.

Next up was a presentation by Gary King from Franz Inc., about AllegroGraph, a massive RDF triple-store capable of storing and searching billions and billions of triples. Specifically, as far as I could tell, it was about adding temporal and geospatial capabilities to queries -- so you could ask "Of the people that Bob called last week, who has a friend within 5 miles of the Boston area?"

It was quite impressive, but it makes me queasy too. Who has billions and billions of facts and needs to query them? Well, the CIA, the NSA, and similar organizations. Others too of course, epidemiologists for example, but that doesn't ease the quease.

Next up, something totally different. John Hastings lives in Nebraska and has a problem with grasshoppers -- they tend to eat up all of the grass that cattle graze on. Ranchers out there would douse their land with chemicals at any word of a grasshopper outbreak. But that's not good for the land, and it's also expensive. So he wrote an expert system to help ranchers say what the conditions are where they live, in Lisp, and help them decide what to do. The program gets deployed into peoples' web browsers via one of the Lisp-on-the-JVM projects. Voilà fewer chemicals!

Next, a talk by an engineer at Cadence Design Systems, on how they use Lisp to lay out certain kinds of electronics called "systems-in-a-package", such as the circuit boards and the amplifiers and conditioners and such. His point appeared to be that giving the user a way to extend and specialize certain designs via generic functions lets them deliver applications without source, but still allow some extensibility. I didn't care for the "freedom subtracted" aspect, but seeing more about chip design processes was interesting.

Jorge Vallejos from the Vrije Universiteit in Brussels took the floor next. His talk was on distributed computing within the Actors paradigm, whereby objects on different machines can reference and send messages to each other. The interesting part was how he used actors as containers for unadorned objects, and used generic functions to "invoke" remote objects. The meta-object protocol allowed him to extend base Lisp to have remote asynchronous procedure calls, while keeping a natural Lisp feel. It was fine, but something about RPC as a distributed programming metaphor doesn't sit right with me. I probably missed something.

Didier Verna followed, with a delightful talk comparing the speed of Common Lisp and C++. He works at a lab in France that does lots of numerical processing on images, and is the only Lisper in a roomful of C++ wizards. The funny thing is that he says he doesn't care about speed, because things are fast enough, but that he was tired of getting shit from his C++-loving coworkers, so he decided to do some measuring.

This guy is serious. This is his second paper in a projected series of 5 or so; his last installment showed that Lisp and C are comparable in speed. This talk compared instantiation time for C++ objects to instantiation of Lisp structures and CLOS objects, showing again that Lisp was comparable in speed with C++. Unfortunately the paper and slides don't appear to be online; I'll ask him about this.

Also, Verna gave a lightning talk in which he explained that Lisp is Jazz and Jazz is Lisp. When asked by the audience about other things, he explained that they were Jazz too. Also Lisp is Aikido, and buy his CD.

* * *

In a later post, I'll get to Sussman's talk on evolvability and robust design. It will take some thinking to summarize.

Andy Wingohttps://wingolog.org/international lisp conference -- day twohttps://wingolog.org/2009/03/24/international-lisp-conference-day-two2009-03-24T14:15:14Z2009-03-24T14:15:14Z

So, picking up where I left off...

David Moon, about whom the internet knows surprisingly little, gave a wild talk on a language he's been working on: the Programming Language for Old-Timers. Actually the talk wasn't about the language, but its macro system. PLOT is a language with syntax, but whose grammar is defined in itself.

PLOT macros take token streams, produced by a fixed tokenizer, and perform LL(1) parsing on that stream, building up abstract syntax tree objects or returning other token streams. Actually, given that macro output can be AST objects, macros take AST fragments as well. The interesting thing is that macros are hygienic, that they track source locations, and that they have access to the full PLOT language themselves. It seems that you can actually implement Lisp macros on top of PLOT macros.

Anyway, a sharp tool, but I look forward to seeing what Moon carves with it.

Next up was a talk by Joe Marshall, a Scheme hacker now working at Google, about implementing continuations on machines with no continuation support, like the JVM or CLR. The basic idea was for functions to be modified to return an OK/Error flag, with the actual return value in an additional output argument. The compiler should do some live-variable analysis, and have a separate flow path that saves variable values all the way down the stack, as functions return "Error", and that can be restored when you reinstate a continuation.

It's nasty. Joe must really want Scheme on the CLR.

Next there was Alexey Radul, a grad student of Sussman's, giving a talk entitled "The Art of the Propagator". The concepts were interesting, although hindered by Radul's poor diction. The basic idea appears to be an approach to calculating constraints-based systems in the presence of backtracking and conflicting information, with electronics as a metaphor -- connecting boxes and wires, signals flowing back and forth. Not like any problem I have ever seen in practice, though.

Following that were a set of lightning talks, the last of which was legendary: one M. Greenberg (who was that guy?) talking about the history of an old macro system, TRAC. Apparently all computation in TRAC was done by macros. Macros could rewrite their definitions out of existence, and thus be recursive and terminate. It was startling, but Greenberg steeled our resolve with the words, "Don't be afraid of macros -- they'll sense your fear, and fuck you up."

Then, a panel discussion on the future of Lisp, the ostensible topic of the conference. From left to right there was Pascal Costanza, an academic from Brussels hacking mostly on CLOS-related things; Edi Weitz, an industrial Common Lisp hacker; Ravi Nanavati, an erstwhile Schemer but current Haskeller; Rich Hickey, Clojure language designer; Alan Bawden, former Scheme Steering Committee member; Duane Rettig, Allegro CL hacker; and Scott McKay, a former Scheme and Dylan hacker now working for ITA software. Kent Pitman moderated.

It started good, with the long view, then things devolved. Eventually the audience started to talk with each other instead of with the panel, and for some reason settled down onto the "why isn't Lisp popular" navel-gaze. Quite disappointing.

After dinner, there was a staged "debate" on "Macros: Are they a menace?" It was obvious that the panel (Steele, Gabriel, Costanza, and the scapegoat, someone whose name I forget) was having a good time. At one point, Gabriel wrote out the "(defmacro .....)" on the chalkboard, noted that the "..." is often large and thus heavy and must sink to the bottom, leaving the parenthesis before the "defmacro" looking like a frowny face: q.e.d.

More seriously, the arguments against macros centered on control: control by bosses on workers, with the idea that macros make custom languages, thus making individual programmers less replaceable. While I don't believe the premise, the conclusion to me comes from another world. I hack Free Software precisely because I like my trade, and want to practice it in an environment free from coercion and domination.

The "debate" had an interlude, in which Costanza asked Sussman why MIT had switched away from Scheme for their introductory programming course, 6.001. This was a gem. He said that the reason that happened was because engineering in 1980 was not what it was in the mid-90s or in 2000. In 1980, good programmers spent a lot of time thinking, and then produced spare code that they thought should work. Code ran close to the metal, even Scheme -- it was understandable all the way down. Like a resistor, where you could read the bands and know the power rating and the tolerance and the resistance and V=IR and that's all there was to know. 6.001 had been conceived to teach engineers how to take small parts that they understood entirely and use simple techniques to compose them into larger things that do what you want.

But programming now isn't so much like that, said Sussman. Nowadays you muck around with incomprehensible or nonexistent man pages for software you don't know who wrote. You have to do basic science on your libraries to see how they work, trying out different inputs and seeing how the code reacts. This is a fundamentally different job, and it needed a different course.

So the good thing about the new 6.001 was that it was robot-centered -- you had to program a little robot to move around. And robots are not like resistors, behaving according to ideal functions. Wheels slip, the environment changes, etc -- you have to build in robustness to the system, in a different way than the one SICP discusses.

And why Python, then? Well, said Sussman, it probably just had a library already implemented for the robotics interface, that was all.

* * *

Now it's Tuesday morning, which started with some guy talking about bombing people with Lisp. Or maybe it was about supplying parts to the military. At one point there was a slide with a little clip-art tank and a clip-art pistol and a clip-art fighter jet. Sheesh.

Finally, via Richard Gabriel: Cobol on Cogs. Enjoy.

Andy Wingohttps://wingolog.org/international lisp conference 2009 -- day onehttps://wingolog.org/2009/03/23/international-lisp-conference-day-one2009-03-23T15:36:03Z2009-03-23T15:36:03Z

Greetings from Cambridge! The American one, that is. I'm at the International Lisp Conference, which is an OK set of talks, but an excellent group of almost 200 Lisp hackers. What follows is not strictly journalism, but rather reflections of a more subjective nature.

Sunday was reserved for "tutorials", four sessions of an hour and a half in length, with an expository and relaxed flavor. There were three sessions running at a time, which made for some hard choices.

The first session that I went to was "Monads for the Working Lisp Programmer", which started off by explaining monads, and in the second session looked at their uses. As a working Lisp programmer, I guess it's good to be able to talk down Haskell weenies, but something about the complexity of the "plumbing" doesn't sit right. People keep talking about separating the "plumbing" from the computation -- as if programming somehow involves clogged pipes or something. The presenters did a good job, but "different strokes, different folks". I jetted out after the first session.

Next I decided to drop in on the second session of Pascal Constanza's tutorial on CLOS, generic functions, and the meta-object protocol. It was very clearly presented, and even though I've been working with GOOPS (a CLOS-a-like) for a long time, I came away with some style tips. Also seeing his LispWorks environment was very interesting. So much of programming is workflow and environment, something that often requires you to look over the shoulder of a master. I was a bit frustrated though, as I came with questions but it seemed Pascal had a curriculum he wanted to stick to. I'll corner him later.

Actually, that's the best thing about this conference, this ability to corner smart people. So far they don't have that harried, ragged look yet.

In the afternoon, I went to Rich Hickey's Clojure tutorials (parts 2 and 3). What a treat. Hickey's instincts and good taste as a language designer are evident. All languages deserve to have his persistent data structures. His discussion of state and identity was particularly lucid, as well. (Hickey say that the talk on state and time was basically the same one he gave at InfoQ.)

Personally I don't care for the JVM, and I feel Clojure's worth as a general-purpose language is actually hampered by the lack of tail recursion, but I really think that his design ideas should percolate out to all the dynamic languages of the world.

Now it's Monday morning, and I'm sitting in on the plenary technical talks and software demonstrations. They're OK, but thus far yesterday had a better feel. I'm very much looking forward to the afternoon sessions.

Andy Wingohttps://wingolog.org/dangling pointershttps://wingolog.org/2008/11/19/dangling-pointers2008-11-19T10:38:57Z2008-11-19T10:38:57Z

I have here three pointers into the ether.

with apologies to rubén darío

A conversation between Dave Moon, Dan Weinreb, and Cliff Click on Azul, a massive multiprocessor Java machine.

The Azul is a Java machine like the Symbolics that Moon and Weinreb worked on was a Lisp machine. Easily the most interesting read of the last week or two, via Michael Weber.

This one will be on the final exam.

house in order

The company I work for, Oblong Industries, just opened up their web site on Friday. Go check it out!

I like how they hook visitors with the videos, then pull a Yegge with all of the text they have there. I approve!

on repeat

Silver Jews' latest album, Lookout Mountain, Lookout Sea, is quite addictive. It's no American Water but I can't get enough of Party Barge.

Send us your coordinates, I'll send a Saint Bernard...

Andy Wingohttps://wingolog.org/dispatch strategies in dynamic languageshttps://wingolog.org/2008/10/17/dispatch-strategies-in-dynamic-languages2008-10-17T16:54:03Z2008-10-17T16:54:03Z

In a past dispatch, I described in passing how accessors in Scheme can be efficiently dispatched into direct vector accesses. In summary, if I define a <point> type, and make an instance:

(define-class <point> ()
  (x #:init-keyword #:x #:accessor x)
  (y #:init-keyword #:y #:accessor y))

(define p (make <circle> #:x 10 #:y 20))

Access to the various bits of what I call the object closure of p occurs via lazy, type-specific compilation, such that:

(y p)

internally dispatches to

(@slot-ref p 1)

So this is all very interesting and such, but how do we actually do the dispatch? It turns out that this is a quite interesting problem, one that the JavaScript people have been taking up recently in other contexts. The basic problem is: given a generic operation and a set of parameters with specific types, how do you determine the exact procedure to apply to the arguments?

dynamic dispatch from a moppy perspective

This problem is known as dynamic dispatch. GOOPS, Guile's object system, approaches it from the perspective of the meta-object protocol, the MOP. A MOP is an an extensible, layered protocol that allows the user to replace and extend parts of a system's behaviour, while still allowing the system to maintain performance. (The latter is what I refer to as the "negative specification" -- the set of optimizations that the person writing the specifications doesn't say, but has in the back of her mind.)

The MOP that specifies GOOPS' behavior states that, on an abstract level, the process of applying a generic function to a particular set of arguments is performed by the apply-generic procedure, which itself is a generic function:

(apply-generic (gf <generic>) args)

The default implementation looks something like this:

(define-method (apply-generic (gf <generic>) args)
  (let ((methods (compute-applicable-methods gf args)))
    (if methods
	(apply-methods
         gf (sort-applicable-methods gf methods args) args)
	(no-applicable-method gf args))))

That is to say, first we figure out which methods actually apply to specific arguments, then sort them, then apply them. Of course, there are many ways that you might want the system to apply the methods. For some methods, you might want to invoke all applicable methods, in order from most to least specific; others, the same but in opposite order; or, in the normal case, simply apply the most specific method, and allow that method to explicitly chain up.

In effect, you want to parameterize the operation of the various components of apply-generic, depending on the type of the operation. With a MOP, you effect this parameterization by specifying that compute-applicable-methods, apply-methods, sort-applicable-methods, and no-applicable-method should themselves be generic functions. They should have default implementations that make sense, but those implementations should be replaceable by the user.

At this point the specification is awesome in its power -- completely general, completely overridable... but what about efficiency? Do you really have to go through this entire process just to invoke a method?

performance in dynamic languages

There are four general techniques for speeding up an algorithm: caching, compiling, delaying computation, and indexing.

Peter Norvig, A Retrospective on PAIP, lesson 21.

Just because the process has been specified in a particular way at a high level does not mean that the actual implementation has to perform all of the steps all of the time. My previous article showed an example of compilation, illustrating how high-level specifications can compile down to pointer math. In this article, what I'm building to is an exegesis of GOOPS' memoization algorithm, which to me is an amazing piece of work.

The first point to realize is that in the standard case, in which the generic function is an instance of <generic> and not a subclass thereof, application of a generic function to a set of arguments will map to the invocation of a single implementing method.

Furthermore, the particular method to invoke depends entirely on the concrete types of the arguments at the call site. For example, let's define a cartesian distance generic:

(define-method (distance (p1 <point>) (p2 <point>))
  (define (square z)
    (* z z))
  (sqrt (+ (square (- (x p1) (x p2)))
           (square (- (y p1) (y p2))))))

Now, if we invoke this generic on some set of arguments:

(distance a b)

The particular method to be applied can be determined entirely from the types of a and b. Specifically, if both types are subtypes of <point>, the above-defined method will apply, and if not, no method that we know about will apply.

Therefore, whatever the result of the dispatch is, we can cache (or memoize) that result, and use it the next time, avoiding invocation of the entire protocol of methods.

The only conditions that might invalidate this memoization would be adding other methods to the distance generic, or redefining methods in the meta-object protocol itself. Both of these can be detected by the runtime object system, and can then trigger cache invalidation.

implementation

So, how to implement this memoization, then? The obvious place to store the cached data is on the generic itself, the same object that has a handle on all of the methods anyway. In the end, dynamic dispatch is always at least one level indirected -- even C++ people can't get around their vmethod table.

Algorithms in computer science are fundmentally about tradeoffs between space and time. If you can improve in one without affecting the other, assuming correctness, then you can say that one algorithm is better than the other. However after the initial optimizations are made, you are left with a tradeoff between the two.

This point is particularly salient in the context of generic functions, some of which might have only one implementing method, while others might have hundreds. How to choose an algorithm that makes the right tradeoff for this wide range in input?

Guile does something really interesting in this regard. First, it recognizes that the real input to the algorithm is the set of types being dispatched, not the set of types for which a generic function is specialized.

This is based on the realization that at runtime, a given piece of code will only see a certain, reduced set of types -- perhaps even one set of types.

linear search

In the degenerate but common case in which dispatch only sees one or two sets of types, you can do a simple linear search of a vector containing typeset-method pairs, comparing argument types. If this search works, and the vector is short enough, you can dispatch with a few cmp instructions.

This observation leads to a simple formulation of the method cache, as a list of a list of classes, with each class list tailed by the particular method implemention.

In Algol-like pseudocode, because this algorithm is indeed implemented in C:

def dispatch(types, cache):
    for (i=0; i<len(cache); i++)
        entry=cache[i]
        x = types
        while 1:
           if null(x) && null(cdr(entry)): return car(entry)
           if null(x): break
           if car(x) != car(entry): break
           x = cdr(x); entry = cdr(entry)
    return false

(The tradeoff between vectors and lists is odd at short lengths; the former requires contiguous blocks of memory and goes through malloc, whereas the latter can take advantage of the efficient uniform allocation and marking infrastructure provided by GC. In this case Guile actually uses a vector of lists, due to access patterns.)

hash cache

In the general case in which we see more than one or two different sets of types, you really want to avoid the linear search, as this code is in the hot path, called every time you dispatch a generic function.

For this reason, Guile offers a second strategy, optimistic hashing. There is still a linear search through a cache vector, but instead of starting at index 0, we start at an index determined from hashing all of the types being dispatched.

With luck, if we hit the cache, the linear search succeeds after the first check; and if not, the search continues, wrapping around at the end of the vector, and stops with a cache miss if we wrap all the way around.

So, the algorithm is about the same, but rotated:

def dispatch(types, cache):
    i = h = hash(types) % len(cache)
    do
        entry=cache[i]
        x = types
        while 1:
           if null(x) && null(cdr(entry)): return car(entry)
           if null(x): break
           if car(x) != car(entry): break
           x = cdr(x); entry = cdr(entry)
        i = (i + 1) % len(cache)
    while i != h
    return false

OK, all good. But: how to compute the hash value?

One straightforward way to do it would be to associate some random-ish variable with each type object, given that they are indeed first-class objects, and simply add together all of those values. This strategy would not distinguish hash values on argument order, but otherwise would be pretty OK.

The standard source for this random-ish number would be the address of the class. There are two problems with this strategy: one, the lowest bits are likely to be the same for all classes, as they are aligned objects, but we can get around that one. The more important problem is that this strategy is likely to produce collisions in the cache vector between different sets of argument types, especially if dispatch sees a dozen or more combinations.

Here is where Guile does some wild trickery that I didn't fully appreciate until this morning when following all of the code paths. It associates a vector of eight random integers with each type object, called the "hashset". These values are the possible hash values of the class.

Then, when adding a method to the dispatch cache, Guile first tries rehashing with the first element of each type's hashset. If there are no collisions, the cache is saved, along with the index into the hashset. If there are collisions, it continues on, eventually selecting the cache and hashset index with the fewest collisions.

This necessitates storing the hashset index on the generic itself, so that dispatch knows how the cache has been hashed.

I think this strategy is really neat! Actually I wrote this whole article just to get to here. So if you missed my point and actually are interested, let me know and I can see about explaining better.

fizzleout

Once you know exactly what types are running through your code, it's not just about dispatch -- you can compile your code given those types, as if you were in a static language but with much less work. That's what Psyco does, and I plan on getting around to the same at some point.

Of course there are many more applications of this in the dynamic languages community, but since the Fortran family of languages has had such a stranglehold on programmer's minds for so many years, the techniques remain obscure to many.

Thankfully, as the years pass, other languages approach Lisp more and more. Here's to 50 years of the Buddhist virus!