wingolog

harvest time on maggie's farm

Finally got a Flumotion release out last week. Hounding us until the end was a terrible bug manifesting itself as random connection loss between the different processes in the server, and 100% CPU usage that couldn't be traced to anything. None of the profilers I tried (or wrote!) gave any clue as to what was up.

The problems were solved when we switched away from forking out job processes to doing fork+exec, which is a more supported model in Twisted. It could be that we weren't correctly cleaning out all of the file descriptors in the child's main loops implementing the reactor in Twisted, causing processes to wake up all the time. It is difficult to analyze exactly what is the state that needs cleaning up in a program like that, instead of analyzing exactly what state to keep for the new process. Also, exarkun in #twisted had an interesting observation, which was that with python's refcounting gc, just about every time you access a variable you modify its refcount, which requires that the child have its own copy of that memory page. Really takes away the copy-on-write advantages of forking processes.

The moral of the story would be that usually you don't want to fork in Python. Changing this to execute separate processes took about 4 hours one afternoon, and took away just about all of the bugs we had been seeing. Thank Jesus!

Also those four hours were krazy 4G1L3. Only thing was it was on my machine, and I have focus-follows-mouse, a different keyboard from Thomas, swapped caps and control, and I use emacs and he uses vi. But somehow we limped along. Agile limping.

very crucially serious notes

A perspicacious analysis of the advantages of being a nice person.

Also this is the crucial phrase of this sentence. I think this convention is so great I'm going find a professional typographer to ask what they think about it. It's about time that some of the popular exponents of this writing style get their well-deserved recognition!

Other things I should write about at some point: more notes on using baz and arch-pqm, my very pleasant impressions of SLIME and SBCL, a recent Aikido seminar with Miyamoto sensei (coming all the way from Hombu dojo), an upcoming trip to the states.