Tuesday, December 23, 2008


Andrew has been pretty enthusiastic about Scala, which he is using for his new factor graph inference library, FACTORIE. I've been reading the new Programming in Scala to learn more. I've not written any Scala code yet, except for a "Hello world" to test the Eclipse plugin. I like the improved conciseness compared with Java, the functional programming facilities, the more uniform object system, the more consistent data structures, the pattern matching facilities. I'm still unsure about the syntax, which feels a bit too clever at times, and skeptical about the use of type-erasure semantics for generics, which has many pitfalls in my experience with teaching and using Java. However, the Scala designers seem to be well aware of the problems with (at least Java's version of) type erasure and are exploring ways of addressing them.

The future is a shoe being thrown at a human face - forever! — Crooked Timber

The future is a shoe being thrown at a human face - forever! — Crooked Timber: [...]Which brings me to the actual topic of the post; a link to a post on my other blog where, in comments, we have been working out job-related mortality rates for a variety of professions.

[...Units: deaths/year-on-the-job...]

President, USA - 2283
Pope – 2126
Hezbollah, active member – 1000-2000
Prime Minister, UK - 937
Monarch, England/UK – 912
Monarch, France – 890
Soldier, Iraqi Army – 800
Soldier, US Army (while in Iraq, March 2003-March 2006) – 392
Fisherman, Pacific Northwest crab fishing – c400
Timber-cutter – 117
Fisherman, overall – 85
Soldier, US Army – c80
(Via Matt Yglesias.)

What about backcountry skiers?

Sunday, December 21, 2008

Eastern Sierra from afar

I flew yesterday over the Eastern Sierra just North of Mono Lake on my way to Philadelphia for Christmas. I looked hungrily at the snowy terrain down there, which I want to visit before this season is over.

Sunday, December 7, 2008

Police launch appeal after grim series of avalanche fatalities

Police launch appeal after grim series of avalanche fatalities: The man, a Nicois, was part of a group of three boarders who crossed under netting at the top of the pistes to ski a ridge into a bowl. [...] was buried under 300cm of snow. [...] was not wearing an avalanche beacon. (Via PisteHors.com.)

The beacon would have likely been a sad redundancy in this case, just helping the rescuers to recover the body more easily. The odds of being dug alive from under three meters of avalanche debris are negligible.

Although the statistics are not fully in, it seems that a reduction in fatalities among practiced backcountry riders is being offset by an increase in reckless behavior by "side country" riders who get into uncontrolled terrain from ski lifts. Moving further into speculation, easier to use equipment and very aggressive publicity for off-piste riding in movies, internet videos, and magazines may also be factors.

Saturday, December 6, 2008

Let it snow

Ed Berry:
Models have now generally caught on to what some would describe as a large amplitude reverse/negative phase of the PNA teleconnection by week-2. I think it is probable this situation may persist the rest of this month per above. As discussed in past postings, particularly weeks 2-4 there should be episodes of significant precipitation along the USA west coast associated with “cold” digging troughs, leading to an active southwest flow storm track across the Plains. Weather ramifications, including high impact blizzard conditions and intense thundersnow in the cold sectors and severe local storms in the warm sectors of synoptic lows, should be understood.

Let’s Talk About Python 3.0

Let’s Talk About Python 3.0: Thoughtful essay by James Bennett in support of Python 3.0. Love the opening.

I agree that the opening is a great cautionary tale. However, the actual Python 3.0 content is even better, because it gives an interesting example of difficult software engineering tradeoffs being worked out through serious debate and attention to conceptual, craft, and social issues in programming. I especially like the discussion about Unicode, because I never understood why previous solutions allowed confusion between a string over a given alphabet and byte encodings of such strings. But then I remember that computer science faculty and students seem to have perversely conspired to make automata and formal languages seem totally irrelevant, so that the simple notion of an injective homomorphism from $\Sigma^*$ to $\Gamma^*$ is no longer obvious to all concerned. (Via Daring Fireball.)

Waiting for snow

High pressure still rules the Sierra, but a door may be cracking open:


Sea of Poppies, thanks to Chris Lydon's interview of Amitav Ghosh.

Tuesday, December 2, 2008

Truthy math

Slightly evil question: is TeX truthiness for math? It's so beautifully typeset, it feels so true.

Sunday, November 16, 2008

fly through snowy mountains

fly through snowy mountains (Via tingilinde.)

Strong reminder of favorite heli rides into the heart of the Monashees, and of the view over the Eastern Sierra from the ridges above Tioga Pass. Where's the snow?

Thursday, November 6, 2008

Freedom and music from Africa: April 25, 1974-November 5, 2008

Revolução dos CravosMr. PresidentToumani DiabatéThe overthrow of the fascist dictatorship in Portugal as a consequence of the failure of the colonial war in Africa and its impact on Portuguese officers. The election of Barack Obama. Twice in my lifetime, in my direct experience, the horrors that Europeans visited on Africa, the moral rot of discrimination and prejudice, were returned to us Europeans and descendants of Europeans as an opportunity to start afresh. Tonight, the great Malian musician Toumani Diabaté dedicated his overwhelming performance in San Francisco to the New America. As Diabaté played, I so wanted to hug the "kids" — Elena, Daniel, and Deva — hoping with them for a bit more peace, a bit more fairness, a bit more tolerance, a bit more reason, and a lot more music and reawakening of our better natures from Africa.

Sunday, October 5, 2008

Goldilocks Optimum for Science Books

There's a delicate balance in science books for the “general” public between being too technical for casual reading and being too wand-waving for readers with a scientific background. I've been lucky with some of my choices over the last few months: The Best of All Possible Worlds, The Poincaré Conjecture, Microcosm, and Ice, Mud and Blood. I'm now two-thirds through D. S. Richeson's Euler's Gem on the history and consequences of Euler's famous V - E + F = 2 formula for spherical polyhedra. I spent quite a bit of time with planar graphs and this formula when working on my first research work, on automated graph layout, but I never knew or I have forgotten some of the tastiest parts of this story, for example some of the formula's most elegant proofs. Richeson's book is just right for me, with enough detail about arguments, analyses and proofs to teach me (or remind me) of important ideas, but light enough to read a chapter or two before sleep.

Wednesday, September 24, 2008

The Real Chile?

The Real Chile?:

We just completed our third, last, and best Fiestas Patrias--those week-plus long September celebrations that commemorate everything from Chile's independence from Spain, to Latin America's biggest and baddest military force, to the US-backed, CIA supported 11 September 1973 coup that ushered in the reign of General Pinochet and the death of Salvador Allende. (Via home is where your skis is.)

Wonderful essay about the subtleties, surprises, beauties, and contradictions of Chile and its peoples. A must read for anyone who cares about the messy heritage of colonial Iberia.

Saturday, September 13, 2008


I've updated the “Recently Read” sidebar. If you have an interest in mountaineering, Fallen Giants is worthwhile, although it has less technical analysis about the progress of high-altitude climbing than I'd like, it gets a bit repetitive, and its view of contemporary climbing is too predictable (all about the evil offspring of commerce and self-actualization). Traffic is a lot of fun, and very helpful to those who seethe about the cluelessness of other drivers (you know who you are!). The Lightness of Being could have done with a better editor — some repetitions, unflagged forward references, missing definitions — but it's really stimulating at the LHC opens up for business.

I'm close to finishing Haruki Murakami's memoir What I Talk About When I Talk About Running, which I'm loving although I'm not a runner. He teaches without preaching about focus, endurance, limitations, and aging.

Sunday, September 7, 2008

Friedman on ‘drill, drill, drill’: It’s like someone chanting ‘IBM Selectric typewriters’ during the IT revolution.

Friedman on ‘drill, drill, drill’: It’s like someone chanting ‘IBM Selectric typewriters’ during the IT revolution.: On NBC’s Meet The Press this morning:

FRIEDMAN: I’m actually not against drilling. What I’m against is making that the center of our focus because we are on the eve of a new revolution, the energy technology revolution. It would be, Tom, as if on the eve of the IT revolution, the revolution of PCs and the internet, someone was up there standing and demanding, “IBM Selectric typewriters, IBM Selectric typewriters.” That’s what “drill, drill, drill” is the equivalent of today.
(Via Think Progress.)

I agree with the sentiment, but Friedman's comment is historically inaccurate. There were quite a few such people back then (and mainframe vs minicomputer, and mini vs PC, and assembler vs C). They just didn't have the lobbying clout of the drillers.

The Singularity Is Far

The Singularity Is Far: In this post, I wish to propose for the reader’s favorable consideration a doctrine that will strike many in the nerd community as strange, bizarre, and paradoxical, but that I hope will at least be given a hearing.  The doctrine in question is this: while it is possible that, a century hence, humans will have built molecular nanobots and superintelligent AIs, uploaded their brains to computers, and achieved eternal life, these possibilities are not quite so likely as commonly supposed, nor do they obviate the need to address mundane matters such as war, poverty, disease, climate change, and helping Democrats win elections. [...] The fifth reason is my (limited) experience of AI research. [...] For whatever it’s worth, my impression was of a field with plenty of exciting progress, but which has (to put it mildly) some ways to go before recapitulating the last billion years of evolution.  The idea that a field must either be (1) failing or (2) on track to reach its ultimate goal within our lifetimes, seems utterly without support in the history of science (if understandable from the standpoint of both critics and enthusiastic supporters).  If I were forced at gunpoint to guess, I’d say that human-level AI seemed to me like a slog of many more centuries or millennia (with the obvious potential for black swans along the way). [...] And I can’t helping thinking that, before we transcend the human condition and upload our brains to computers, a reasonable first step might be to bring the 17th-century Enlightenment to the 98% of the world that still hasn’t gotten the message. (Via Shtetl-Optimized.)

Read the whole thing. I'm not sure I'd give as much credit to the Singularitarians as Scott does, but I'm in complete agreement with his estimation of AI: “a field with plenty of exciting progress, but which has (to put it mildly) some ways to go before recapitulating the last billion years of evolution.”

Saturday, August 30, 2008

The Thousand Best Popular-Science Books

The Thousand Best Popular-Science Books: Over at Cocktail Party Physics, Jennifer has cast a baleful eye on the various lists of the world’s greatest books, and decided that we really need is a list of the world’s greatest popular-science books. I think the goal is to find the top 100, but many nominations are pouring in from around the internets, and I suspect that a cool thousand will be rounded up without much problem. (Via Cosmic Variance.)

Follow the links for some interesting lists. Since my library is scattered among several places in Philadelphia and Palo Alto, I'll rely on my very fallible memory for this unordered list of ten books that stayed:

  • Microcosm, Carl Zimmer
  • The Best of All Possible Worlds, Ivar Ekeland
  • The Fabric of Reality, David Deutsch
  • Annals of the Former World, John McPhee
  • Thin Ice, Mark Bowen
  • The Plausibility of Life, Marc Kirschner and John Gerhardt
  • Battle of Wits, Steve Budiansky
  • Alan Turing: the Enigma, Andrew Hodges
  • The Illusion of Conscious Will, Daniel Wegner
  • Three Roads to Quantum Gravity, Lee Smolin

Saturday, August 23, 2008

Chile trip report

Chile August 08

I just returned from a road trip in Chile with my friend Rick Winfield. Our original plan was to try to summit and ski three volcanos: Lonquimay, Antuco, and Volcán Nevado in the Nevados de Chillán complex. In the end, the stormy weather — heavy winds and snow in half of our days there — kept us to lift-served skiing more than we had planned, but we got the bonus of some nice powder skiing in Nevados de Chillán and La Parva.

We arrived in Santiago early August 15, and we drove all day (770 km) to Corralco, the small resort on the South side of Lonquimay. It blew and snowed moderately all of Saturday, as shown on the first 12 photos in the album. To do something, we skinned twice from the lodge to the ski area (around 300 m vertical) where we saw Chilean soldiers practicing winter skills, and where we practiced crampon travel on a wind-scoured slope.

Sunday (photos 13 to 28) was summit day. We got a ride on a snow cat to the ski lifts, because the road was still partly unplowed, and rode the chair to the top of the small bump (1900 m) on the right of the photos of Lonquimay. We started skinning from there at around 11:30am along the SE ridge until it got too steep at around 2500 m, and then we used our crampons to reach the 2850 m summit at around 2:30 pm. It was very windy and cold most of the way, with a few lulls in the wind that didn't last. Cris, a Chileno snowboarder we had talked with in Corralco, reached the summit just after Rick and before me. You can see the windblown snow and ice in several of the summit photos, including Cris's photo. It was that windy. We descended the East face, which had a good cover of nicely carvable windpacked snow, with no evidence of significant slabs. It was a great ride the around 1300 m down to the lodge, finishing at around 4 pm.

Cris had told us that Antuco's snow level was high and required a trudge off the snow. We were in Chile to ski, not to hike, so we decided to bypass Antuco and drive directly to Las Trancas for three days skiing Nevados de Chillán.

Monday was very windy, and we guessed that the SE shoulder of Volcán Nevado, which is critical for the ascent, could be very icy and exposed. We decided instead to seek good snow in lee sides, and we found some. Tuesday started snowy, windy, and with terrible visibility. The Don Otto, Nevados de Chillán's longest lift, was not running. We poked around trying to find wind-deposited goodness, and it got better and better, with visibility improving a bit by late afternoon. It snowed more overnight, and Wednesday dawned sunny if still windy. We decided to chase powder (sorry, no pictures, I was too busy skiing) and we weren't disappointed. Even though we had some 15 cm of new snow and a lot of wind, we did not find any significant wind slabs on the steep lee sides we skied, mostly in the Pirigallo valley (we avoided suspicious rollover and corniced areas on the notorious Elefante ridge). So, Rick did not summit Volcán Nevado (I had done it back in 2003), but we had a great day of fresh tracks.

We drove back to Santiago after skiing, and got a very late dinner at one of our favorite places there, Etniko.

Thursday, we drove to La Parva through dense morning traffic, and got there at around 10:30 am. We kept skiing the steeps around the Las Aguilas lift, where we found some nice, mostly untracked snow. I quit exhausted at 3 pm after six days of ascent and descent, Rick took one more run, and we drove back to Santiago to change clothes, pack, and drive back to the airport for our 16 hours of travel back to San Francisco.

Thanks: Above all, Rick for driving and for putting up with a slower climbing and skiing companion; Corralco staff for their hospitality and help with transportation to the volcano; Cris for advice on Antuco; Stuart at Cabañas Los Andes for a good stay and good stories; Owen and his family for an entertaining dinner and skiing company; staff at the Santiago Crowne Plaza for help with baggage and changing after skiing; Chilenos for their friendliness and willingness to put up with my fractured Spanportugish; Delta Airlines for getting us and all of our gear there and back without a hitch; our friends at CASA Tours for introducing us to this stunning part of the world; and Frédéric Lena for the advice in his beautiful Chile-Argentina: Handbook of Ski Mountaineering in the Andes.

Sunday, August 17, 2008

Vista do forte de Almádena

Notas esparsas de férias- vista do forte de Almádena: (Via aguarelas de Turner.)

My sister has been on vacation and posted this picture. There are many beautiful rocky coasts in the US, from Maine to Mendocino, but this particular composition of cliff and sea color doesn't match anything I've ever seen in this continent. Maybe it's just how it evokes those intense memories of childhood summers. But this picture, coming as it does as I take my own vacation in Chile exploring skiable volcanos (more pictures to come), makes a striking contrast with today's snow-swept araucarias.

araucarias in the snow

Sunday, August 10, 2008

clueless "targeted advertising" found in today's email

clueless "targeted advertising" found in today's email: Picture_1

(Via tingilinde.)

How's the poor computer to know about the details of meteor shower observation? At least it didn't try to push bath supplies...

Wednesday, August 6, 2008

Sunday, August 3, 2008

The Atomic Cloud

The Atomic Cloud: At our recent Social Media Summit, in a break out session with Marti Hearst, Joanna Robinson and others, the topic of the open social network came up: who owns the network implicit in Facebook, what happens if LinkedIn evaporates, etc. [...] Now, imagine a different architecture, one in which our identities are tied intimately to a mobile device that we carry around, and the device is actually an active node in the network. It not only stores your data (your own social network) but also acts as a server for that information (according to appropriate policies). This would remove the risk of the single point of failure and the problems of social graph ownership. (Via Data Mining.)

Back in the 90s, when I worked for AT&T, we spent a lot of time brainstorming about similar ideas, and so did many others. The big problem is that we still need a centralized naming system so that devices know who they are talking to and which devices are authorized to do what. The fantasy or a totally anarchic social networking system is no less a fantasy than extreme libertarian fantasies about radical human autonomy. All of our human transactions depend on social systems for naming and authentication. Thus, the problems with possible failure and abuse of social networking infrastructure are mainly political and economic, not technical. As we have seen with all the political and economic conflicts and compromises with the current Internet naming system.

Coincidentally, Dave Winer has been writing recently about closely related issues, in a more pragmatic but maybe still too optimistic vein.

Saturday, August 2, 2008

El Viento Viene, El Viento Se Va

Maybe my favorite Manu Chao song, but also a beautiful backcountry skiing trip report from Las Leñas.

Not this year...

My blog logo picture to the right was taken in 2006 from an elevation of about 2300m on the Llaima volcano in Chile. The picture on the left of this post is of Llaima itself early July. Quite a different scene. I'm hoping to climb and ski Lonquimay, on the logo picture's horizon, in a couple of weeks, but surely I won't be getting any closer to Llaima than that. The Pacific Ring of Fire is alive and well.

olympic hill climbing

olympic hill climbing: A NY Times piece on The Incline. It sounds painful. (Via tingilinde.)

Looks like perfect training for backcountry touring. I wonder if there's one around here...

Friday, August 1, 2008

Brain Rules and Computer Science Education

Brain Rules and Computer Science Education: I just finished reading Brain Rules by John Medina [...] The point is that in medical school students are learning the theory of medicine at the same time they are practicing it [...] I think Computer Science education can really benefit from such a model. I obviously don't have all the details worked out here, but imagine that every Computer Science department (or set of departments) had a software company on the side. As students go through the program, they start getting tasks from that company to build software for it, participate in designs, see how product decisions affect engineering processes, and even see some company politics at work. [...] Given that these companies are likely to have challenges competing in the market, they need to address a niche of customers who are willing to put up with lousy service, mediocre products and delays in software release cycles. I.e., customers who have nowhere else to go! [...] These customers are called scientists (Via Alon Halevy's Blog.)

We are on the same page, Alon.

The Knack

Sorry, mom. I know it was tough do deal with the leaking chemicals in my bedroom and my weird sleeping schedule, but it's not been all bad... (Hat tip to Bev).

Saturday, July 26, 2008

Happy watch and a cool Google Maps mashup

declination.pngAfter several months of not finding the parts in stock and being busy with other things, I finally got the replacement wrist band and battery door (thanks to Backcountry.com) for my battered Suunto Observer. After a bit of a struggle to replace the parts — I don't have watchmaker skill or tools, but a Leatherman is incredibly versatile — I have a working watch again. I set the time, elevation (7m), and recalibrated the compass. Then I thought to add the local magnetic declination, and a quick search found this cool tool.

Sunday, July 20, 2008

Over Hudson Bay

Here's another batch of photos from my ICML return flight. The sea ice on Hudson Bay was far into its summer disintegration. Some of the patterns of dirty ice look like fossils or plankton skeletons with a mysterious fractal geometry. What would it feel like to touch it?

A beach funeral

George Brassens lovers on the net have uploaded some amazing performances, including one of what may be my all-time favorite ballad Supplique pour être enterré à la plage de Sète with its eternal exhortation “le vin et le pastis d'abord!”

I don't know why, but this couplet really does it for me:

Trempe, dans l'encre bleue du golfe du Lion,
Trempe, trempe ta plume, ô mon vieux tabellion,

(Dip it in the blue ink of the Gulf of Lion/Dip it, dip your pen, oh my old notary)

Georges Brassens in the park

In her photo series about a public park, my sister Isabel recalled of one of those beautifully wry, timeless Brassens songs:

Quand les mois auront passé
Quand seront apaisés
Leurs beaux rêves flambants
Quand leur ciel se couvrira de gros nuages lourds
Ils s'apercevront émus
Qu' c'est au hasard des rues
Sur un d'ces fameux bancs
(Via aguarelas de Turner.)

Saturday, July 19, 2008

Henry James is My Ambien

Henry James is My Ambien: As I now learn, late-period James is a very different animal than early-period James. I’m reading “The Wings of the Dove”, and find that, in spite of being a native speaker of english and well-read in James’ contemporaries, I simply cannot understand what he writes some fraction of the time [...] dust off your sentence diagramming skills and see what you can make of this, which starts and ends well, but in the middle veers off into I know not where:

The difficulty with Densher was that he looked vague without looking weak — idle without looking empty. It was the accident, possibly, of his long legs, which were apt to stretch themselves, of his straight hair and his well-shaped head, never, the latter, neatly smooth, and apt, into the bargain, at the time of quite other calls upon it, to throw itself suddenly back and, supported behind by his uplifted arms and interlocked hands, place him for unconscionable periods in communion with the ceiling, the treetops, the sky.
(Via Cosmic Variance.)

Maybe being a non-native speaker gives me an advantage, since I've spent a lot more time figuring out English (and French) literary and philosophical writing. That James paragraph is a breeze compared with some Proustian appositive marathons. Here's the core of the longer sentence with brackets marking where various qualifiers (nonrestrictive relatives, adjectival and adverbial phrases) are elided to avoid overflowing the reader's short-term memory:

It was the accident [...] of his long legs [...], of his straight hair and his well-shaped head [...] apt [...] to throw itself suddenly back and [...] place him for unconscionable periods in communion with the ceiling, the treetops, the sky.

Not so hard, right? Of course, it removes much of the wonderful life in the original, which had me see Densher sitting over there, outside in this perfectly cool Palo Alto early afternoon, maybe engaged and maybe not in the question I would have asked him.

Without wanting to be critical of Julianne, whose posts I really enjoy, I was at first surprised at her claimed confusion with that sentence, but then I remembered the total lack of elementary linguistic education in schools here and in Europe. In middle school back in Portugal, I suffered through the torture of an absurd application of Latin grammar to the analysis of Portuguese sentence structures (No, being a Romance language is not enough to make classical Latin grammar a good fit). The whole thing seemed totally illogical to me, and I barely passed the class. It wasn't until later in college, when through a friend I saw the first examples of modern formal syntax, that I had a huge "ahah" moment that I still feel vividly more than 30 years later. It's a shame that the main scientific discoveries of the last 100 years of linguistics, which are so useful in helping us figure out how language works, continue to be ignored or misrepresented in our educational systems.

Friday, July 11, 2008

Flying over Greenland

Returning from ICML on July 9, on United 901 Frankfurt->SFO, I was treated to hours of amazing views of Iceland, Southern Greenland, and Baffin Bay. I had a window seat and I took over 150 photos. I still have to do some pruning — I don't have much experience taking pictures through an airplane window — and anyway Picasa uploading is failing for me at the moment (Unsupported output format: text/xml (status 400)?). Here's a picture of a scene that is as beautiful in its subtle color as it is ominous: pools of meltwater on the ice sheet. More pictures and commentary later.

Update: The Picasa discussions suggested reinstalling the uploader. I did it with Google Updater, and photo uploading is working again. Here are the least fuzzy 64 pictures I took, East to West during 50 minutes of overlying the Southern tip of Greenland. There was always some mist in the air, and thicker clouds as well, so terrain coverage is not continuous. You can see the approach to the deeply glaciated East coast, then a few shots of the crevassed icecap with some melt pools, and then the Western edge, mostly free of ice.

Thursday, July 10, 2008

The Importance of Stupidity

I Like Feeling Stupid: Earlier this summer I read the excellent essay by Martin A. Schwartz in The Journal of Cell Science (May 20, 2008, doi: 10.1242/10.1242/jcs.033340) on "The importance of stupidity in scientific research." He describes how even very smart people might not enjoy research because they don't like feeling "stupid" all the time. (Via FemaleScienceProfessor.)

Very well put, read the whole FSP post. Unfortunately, the original The Journal of Cell Science article is behind a subscription wall, so it's available only to those of you who enjoy the privilege of a well-stocked academic library. There's a big difference between voluntarily suffering feelings of stupidity by working on hard problems and being condemned to stupidity by obsolete barriers to the free circulation of scientific information.

Sunday, June 29, 2008

Exponential Stupidity

An Undertaking of Great Advantage, But Nobody to Know What It Is: Cosma Shalizi summarizes and links to a Wired report about an FBI data mining project. Leaving aside constitutional and political issues, what stuck me most in the report is was the following quote:

[...] the Justice Department claims that with this new data mining center’s access to billions of personnel records the “universe of subjects will expand exponentially.”
(Via Three-Toed Sloth.)

If the “universe of subjects” will expand exponentially, so will the universe of false positives for a fixed acceptance rate. Since the FBI's investigative capabilities cannot grow exponentially for demographic reasons, their only solution would be to decrease the acceptance rate of their data mining procedure. Given their past problems with false negatives, I'm not sure that would be tenable. Unless their data mining methods are getting exponentially better, which in my experience never happens.

Lest I sound too negative about data mining, here's a high-accuracy data mining rule: When high-ranking bureaucrats use the adverb “exponentially,” they are deluded or being economical with the truth.

Thursday, June 26, 2008

"These can be aptly compared with the challenges, problems, and insights of particle physics"

"These can be aptly compared with the challenges, problems, and insights of particle physics": For this survey, I decided to start with some historical background, and so I went back to the famous ALPAC report. [...] This report is mainly remembered as a rather negative assessment of the quality, rate of progress, and economic value of research on Machine Translation. It's widely credited with eliminating, for more than two decades, nearly all U.S. Government support for MT research. [...] However, I had forgotten — if I ever knew — that as the title "Computers in Translation and Linguistics" suggests, the ALPAC report had another and more positive message. Here's a sample:

Today there are linguistic theoreticians who take no interest in empirical studies or in computation. There are also empirical linguists who are not excited by the theoretical advances of the decade–or by computers. But more linguists than ever before are attempting to bring subtler theories into confrontation with richer bodies of data, and virtually all of them, in every country, are eager for computational support.
(Via Language Log.)

Read Mark's whole post. But I can't resist the feeling that Mark is being too charitable towards ALPAC, who seem to have predicted the future of natural language processing exactly wrong. Use of computational methods and large corpora in linguistics is still the exception, while (statistical) machine translation is arguably the area of language processing that is making the most remarkable progress. At this year's ACL meeting, the pervasive impact of the core idea of statistical MT — mining co-occurrences in text to discover language units and unit correspondences - not only in advancing MT but also in many other areas of natural language processing. And statistical MT has generated new demand for advances in robust language analysis methods, as the recent progress of MT methods based on statistical grammars shows.

It's interesting how ALPAC could have gotten it so wrong. The main methods and metaphors of statistical MT are based on ideas from communication theory and (implicitly) from cryptology that should have been very natural to Pierce and his colleagues. Instead, they seemed to be captured by the formal syntax revolution in linguistics, and they expected that the main line of progress would be developing detailed linguistic theories and analyses. Twenty years later, the contrast between steady measurable advances in statistical speech recognition — which drew heavily from those communication theory/crypto ideas — and the relative floundering of symbolic methods in AI and language processing finally undid ALPAC and sent renewed DARPA funding into corpus collection and annotation, statistical NLP, and crucially statistical MT. Today, there's still very limited use of computational methods in core linguistic research, but you can get free machine translation services online that are getting steadily better by mining ever growing data with statistical learning methods.

Monday, June 23, 2008

The End of Theory: The Data Deluge Makes the Scientific Method Obsolete

The End of Theory: The Data Deluge Makes the Scientific Method Obsolete: But faced with massive data, this approach to science — hypothesize, model, test — is becoming obsolete. [...] There is now a better way. Petabytes allow us to say: "Correlation is enough." We can stop looking for models. We can analyze the data without hypotheses about what it might show. We can throw the numbers into the biggest computing clusters the world has ever seen and let statistical algorithms find patterns where science cannot. (Via Wired News.)

I like big data as much as the next guy, but this is deeply confused. Where does Anderson think those statistical algorithms come from? Without constraints in the underlying statistical models, those "patterns" would be mere coincidences. Those computational biology methods Anderson gushes over all depend on statistical models of the genome and of evolutionary relationships.

Those large-scale statistical models are different from more familiar deterministic causal models (or from parametric statistical models) because they do not specify the exact form of observable relationships as functions of a small number of parameters, but instead they set constraints on the set of hypotheses that might account for the observed data. But without well-chosen constraints — from scientific theories — all that number crunching will just memorize the experimental data.

Saturday, June 21, 2008


After Dark

I hadn't read novels for a while. Maybe I was a bit disappointed by my fiction choices, maybe there were just too many interesting non-fiction books, like Microcosm, drawing my attention. But recently I had several trips where I needed something to read on the plane or at the hotel. I enjoyed Chabon's The Yiddish Policemen's Union (although, like Kavalier&Klay, it could have benefitted from a tougher editor), but After Dark is something else. Spare, subtle, deceptively plain. If quantum entanglement can have a fictional embodiment, this may be it. The action seems random, but correlated in unexpected ways. Even in my own memory. When I started reading it on the flight from MSP to SFO (returning from ACL in Columbus), it evoked the long-ago reading of Report on Probability A, by Brian Aldiss. Aldiss did not achieve Murakami's grit and emotional density, but he got the entanglement.

I had never read any novel by Murakami. Then I read a self-portrait in The New Yorker that used his taking up of long-distance running as the driver for the story of becoming a full-time fiction writer. I had to read something by him after that.


In other book news, the beautiful Chile-Argentina: Handbook of Ski Mountaineering in the Andes by Frédéric Lena arrived in a well-wrapped package from Grenoble. Beautiful photos, detailed maps and routes of many Andean ski tours, a few of which I've done. Now I just need to sort out my trip down there this summer...

Friday, June 6, 2008

Dreaming of Andean snow

View N from Llaima.jpgIt's fall in the Southern hemisphere, and those of us who have been smitten by the terrain, landscape, and people down there start dreaming of climbing another volcano to ski it. Frédéric Lena has climbed and skied many of them, and has now created a great site and book for all of us Andes skiing fanatics.

Sunday, June 1, 2008

Sierra Spring

I forgot my camera in the rush to pack and drive up to Strawberry on Friday evening, but even camera-less, I had a wonderful two days of backcountry spring skiing around Tahoe with friends. On Saturday, we hiked and skinned up from Carson Pass to Round Top and skied N-facing corn snow from the shoulder of Round Top down to the bowl between Round Top and The Sisters. Then we just had to slog out over melting snow and open ground bursting with wild flowers. On Sunday, we skinned up to ski three excellent runs off Tamarack Peak above the Mount Rose Highway, and we just had to remove our skis once on the way back to the road. Both days were warm, but some clouds and a cool breeze kept the snow from turning into mush and threatening wet slides on the aspects we traveled. Good skiing on June 1st is yet another of those Sierra gifts.

Sunday, May 25, 2008


Carl Zimmer's Microcosm, p. 149:

The fact that E. coli and a man-made network show some striking similarities does not mean that E. coli was produced by intelligent design. It actually means that human design is a lot less intelligent than we like to think. Instead of some grand, forward-thinking vision, we create some of our greatest inventions through slow, myopic tinkering.

Inappropriate Mathematics for Machine Learning

Inappropriate Mathematics for Machine Learning: Reviewers and students are sometimes greatly concerned by the distinction between:

  1. An open set and a closed set.
  2. ASupremum and a Maximum.
  3. An event which happens with probability 1 and an event that always happens.

I don’t appreciate this distinction in machine learning & learning theory. All machine learning takes place (by definition) on a machine where every parameter has finite precision. Consequently, every set is closed, a maximal element always exists, and probability 1 events always happen.

The fundamental issue here is that substantial parts of mathematics don’t appear well-matched to computation in the physical world, because the mathematics has concerns which are unphysical. (Via Machine Learning (Theory).)

John is deeply confused here. One of the most important jobs of mathematical abstraction is to make it easier to reason about whole classes of problems whose size is data dependent. That's why we use Turing machines in the theory of computation, and the continuum when working with approximations and rates of change whose precision cannot be specified in advance. Constructive mathematics, however clever, is a niche pursuit because it makes simple arguments about rates, approximation, and existence much harder than they are with Weierstrass epsilon-delta arguments and their hugely successful development in analysis, topology, probability, and applied mathematics. It would be wonderful if there was a constructive mathematics that was as easy to work with as those classical areas. I spent quite a bit of time and effort studying intuitionistic and constructivist methods a long while back, and my frustrated conclusion then was that these approaches that are supposedly closer to computation actually make the simplest classical arguments a huge chore, for very uncertain payoff.

Sticking my neck out: constructivism is misguided because it believes in a single fabric for mathematics. It refuses to accept that mathematics is a patchwork of methods that work at different levels of abstraction and are not fully inter-translatable. Which is not surprising if you recognize that mathematics is a big messy workshop of tools for abstract thought, not the incomplete projection of a Platonic ideal.

Thursday, May 22, 2008

On owning books

On owning books:

My father will mail me a copy of Willie Morris' North Toward Home and a few Durrells; I will mail him the hilarious and poignant A Thousand Shall Fall, and an old edition of Grimble's We Chose the Islands. And we argue over whom first heard about Andy Adam's The Log of a Cowboy; then we realize we both have editions in our homes.

Here is the thing: it's hard to say who owns these books. They are ours, collectively; they fling back and forth between Texas and California, and either household is only a temporary resting place. These books are shared, because they are appreciated; loved, because they are enjoyed with others.


Whatever digital (ebook) books look like in the future, if they do not embody the right to share, in an unrestricted and platform independent manner, they will be poorer things.

This is called the first sale doctrine. It's part of why people love books -- a love built from sharing. It's what makes libraries possible. A world where content is licensed, and sold with restrictions on use, is a world less full of enthusiastic readers; less full of love.

(Via The Patry Copyright Blog.)

I have this theory that much of what we consider “higher intelligence” could not operate without external memory — books, notes, letters, drawings, {black|white}boards — much like a Turing machine needs its tape to go beyond finite state. When we exchange books, we build each other's minds.

Sunday, May 18, 2008

They must be having a great Spring skiing season in Colorado

PHL->SFO this morning, I slept in my left-hand-side, over-the-wing window seat, much of the route before the first rise of the Rockies. Looking down, I didn't see the usual landmarks, and at first I thought we were on a very northerly route. Then I saw three obvious ski areas, the last two close together, with the last one very big and vaguely familiar. Lots of snow still down to below tree line. Eventually I figured out that the last two neighboring ski areas were likely to be Aspen and Snowmass, with the distinctive jagged N-facing cirque below the summit at Snowmass. Later we crossed the Sierra SE->NW, with Mammoth and June Mountain clearly visible, and Tioga Pass briefly glanced ahead of the wing, Dana Couloir still white (and probably totally icy).

Language as shaped by the brain

In a recent Language Log comment, Shimon Edelman pointed to "great new paper by Christiansen and Chater" on the "the logical problem of language evolution". During a recent trip to Philadelphia and New York that had nothing to do with language evolution, I printed the paper, and then proceeded to stay up late into the night to finish reading it. It was not just jet lag. The paper is indeed very interesting. Since it is a review that cites many primary sources that I have not read, my support is cautious, but its evolutionary argumentation against the hypothesis of innate, genetically-enconded language-specific brain machinery appears very solid. I am also sympathetic to their hypothesis that "language has evolved to fit prior cognitive and communicative constraints, then it is plausible that historical processes of language change provide a model of language evolution; indeed, historical language change may be language evolution in microcosm." But the section in which they develop this hypothesis is less tightly argued than the rest of the paper, with some just-so story-telling drifting by. Their enthusiasm may have pushed them a bit further than the very fragmentary evidence warrants, but it's not surprising that they are enthusiastic after their convincing demolition of rationalist fantasies that have been holding us back for a long time.

Saturday, May 17, 2008

Reading "Microcosm"

I'm on page 73 of Microcosm, Carl Zimmer's account of E. coli's role in the development of modern biology. So far, it's the most crisply written and informative science book I've read in a long time, and I've been reading quite a few good ones. It does not condescend, it does not wave its hands when things get more technical. No superfluous "local color" digressions (I don't really need irrelevant anecdotes from some scientist's life). Yet, the writing is lively and almost humorous, as life (ours or E. coli's) is pretty funny in its contradictions, kludges, and amazingly successful hacks.

Friday, May 16, 2008

We have violated the prime directive

We have violated the prime directive: Noah Smith and I are co-supervising Tae Yano on a project involving analysis of political blogs, and Tae left a pile of results and code on her CMU web site as a way of communicating with us...world-readable. Surprisingly someone at one of the blogs she spidered, Little Green Footballs, actually noticed, leading to a lot of investigative work in this fascinating thread (Via Cranial Darwinism.)

Best story of the week. A whole post-deconstructionist novel could be written around this. This thread entry takes the prize:

#474: magine a natural language program that could respond to comments with charm and style, sort of a robo-blogger. Now imagine an army of them, all set to monitor a different political blog, run by a campaign manager for a politician. Add to its writing ability an encyclopedic memory, with instant access to famous quotes, historical facts, trivia, statistics, and every word ever uttered by the opposition. You now have an army of ultimate bloggers, all completely under the control of one campaign manager... no more "going off message" by some underpaid/volunteer lackey, just high quality counter-opinion, ready to be inserted into the blogs of anyone who disagrees with your candidate. This research will eventually lead to robo-blogging to kill emerging scandals and alternative opinions on issues... no more Rathergates as they will be smothered in the cradle by the most charming bloggers around -- the poli-bots.

For all I know, this could be going on. Mark V. Chain, the Bell Labs robo-flamer, fooled many on Usenet a couple of decades ago. The Turing test gets much easier pass when the machine is craftily designed to exploit the cognitive and emotional biases of the judges.

Monday, May 12, 2008

Powerset Launches!

Powerset Launches!: Meanwhile, I'm waiting for Fernando to pounce.[...]Secondly (actually, this is more important) pundits are going to write about the wikipedia-only issue. They're not getting it. 90% of search results come from a tiny fraction of web pages due to the huge redundancy on the web and the differences between searcher needs and author/publisher intents. The task isn't to always search that huge set, but to get the answers to the user. (Via Data Mining.)

No pouncing. The demo is interesting, which is what I expected from the smart folks at Powerset. Congratulations to them. But I've seen nothing yet that contradicts my earlier observations in this blog or in the NYT.

As for the huge redundancy of the web, forgive my skepticism. If I search for "Matthew Hurst", I want your blog first, not some more popular page that happens to make some of your blog redundant. The tail of the distribution matters.

Saturday, April 26, 2008

Celebrating April 25th with a Spanish director and musicians from three continents

Yesterday was the 34th anniversary of the peaceful overthrowing of Salazar's fascist dictatorship that crippled Portugal for 48 years. Today, by happy coincidence, the San Francisco International Film Festival was showing Fados by Spanish director Carlos Saura. Growing up during Salazar's decline and his colonial war in Africa, fado was for many of my generation backward, sentimental, reactionary. Except that a few musicians like Zeca Afonso in Portugal and Chico Buarque in Brasil knew better. With Chico Buarque's Fado Tropical superimposed on historic footage of the military and popular movement that overthrew the dictatorship (it was so strange to feel that almost surely some of the faces in those scenes were those of people we knew), Saura brings out simultaneously the hope of that movement 34 years ago, the wounds of Portugal's colonial history, and the reborn hope that fado is not just nostalgia, but also a deep stream from the past into the future of the living music of three continents.

Sunday, April 20, 2008

Sierra wind

My afternoon flight from PHL to SFO flew just south of Mono Lake, Tioga Pass, and Yosemite. The strong winds over the Sierra formed striking cloud streams that poured over the Sierra crest into the Mono basin. The air seemed filled with dust in the sunset light.

Sunday, April 13, 2008

Reading "The Best of All Possible Worlds"

I'm reading Ivar Ekeland's The Best of All Possible Worlds, and I can't resist posting this quotation:

We are a far cry from Plato, for instance, who taught that the objects we observe are but images, or shadows, of originals, the only true and real Objects, which exist in a world above one own, the world of Ideas. [...] Poincaré points out that science does not need that kind of belief: there is no need for objects to exist in any other way than to relate our sensations with common experience. Clearly there is no more room for metaphysics: science can be concerned with relating only facts, not things.

A residual Platonism often confuses natural science, mathematics, and computer science. Time to read Poincaré.

Saturday, April 12, 2008

Franklin Institute Awards

belief network I'll be back to Philadelphia the coming week to participate in Franklin Institute Awards events. I played a small part in the awards process, and I'm delighted that two pioneers of machine intelligence, Takeo Kanade and Judea Pearl, are being recognized.

Sunday, March 30, 2008

Strong AI

Strong AI: While we can recognize an intuitive path to the goals of strong-AI (a self-aware intelligence machine) through the simulation of the human brain in a very literal sense, it is more appropriate to think of the brain itself as an implementational detail. What is more interesting is to capture the fundamental truths of intelligence and self-awareness abstractly and then implement them in an appropriate manner with the tools at hand. The big difference here is that this approach leads to a deeper understanding of intelligence. (Via Data Mining.)

The assumptions in both forks of this alternative totally miss biology. First, there is no reason to believe that there is a meaningful boundary between the brain and the rest of the body that allows a brain to be simulated in isolation. Second, there is a deep essentialism in its use of terms like "intelligence" and "self-awareness" as abstract functions independent of biological embodiment. We know so little about even the most basic cognitive functions that debating these two views of strong AI is like debating space travel one thousand years before Newton.

Saturday, March 29, 2008

Sol Mountain 2008

I've finally got around to sorting through the pictures I took at Sol Mountain early March. The trip started with 60 cm of new snow, and continued with cold and often sunny weather for most of the week. A few of the pictures show the sensitive state of some of the snowpack, but we were still able to ski steep, deep, and sometimes wide open slopes for much of this fantastic trip. I didn't get my camera out much in the first couple of days because I was too busy climbing, but others did and I hope we'll have some good pictures from them at some point.

Tuesday, March 25, 2008

Study: Warming Threatens Lake Tahoe's Pristine Waters

Study: Warming Threatens Lake Tahoe's Pristine Waters: Climate change is altering water circulation in Lake Tahoe to extent that the lake's fabled cobalt blue waters may turn a murky green within a decade, a study finds. (Via Wired News.)

TahoeNahuel Huapi

That would be an environmental and esthetic disaster beyond accounting. Tahoe and Nahuel Huapi are twin jewels of the last glaciation, deep blue shades surrounded by white and dark green that remind us of the stunning world that we evolved into but are now undoing.

Saturday, March 22, 2008


Ever since I started skiing in the backcountry, I've been on Fritschi bindings. They got me into the sport, and I didn't have many complaints. Serious backcountry skiers kept telling me that Fritschis are heavy and have an unnatural walking mode, both of which waste energy on tour. But the fiddly aspects of the alternative, Dynafit bindings, and the fact that I finally had very good boots that worked both inbounds and in the backcountry, made me procrastinate. But seeing again the efficiency of Dynafit gear in my recent week in Canada, and the good reviews of recent Dynafit-compatible boots, broke the logjam. I stopped at the Backcountry in Truckee last weekend and I picked up the new setup today: Dynafit Zzero 4 C-TF boots, Dynafit Vertical ST bindings, and Karhu Jak BC 100 skis. So light! Skinning up to Castle Peak was like no skinning I've ever done: the stride was so natural, and the connection between foot and ski so efficient, that it felt like a walk in the park. Skiing down on corn on top of sun crust was smoother also because of the firm and low connection between boot and ski. The new skis are so light and fat that they are deflected more easily than my other stiff and narrower skis, but the boot and binding make up much of the difference. I love this new setup.

Wednesday, March 19, 2008

Theodore Vail is spinning in his grave

This is not Mr. Vail's AT&T:
In 1907, Vail returned to what was essentially his previous job, though now the company was known as the American Telephone and Telegraph Company, or AT&T. AT&T was in some trouble because its phone patents had expired and other small companies were getting into the business. Suddenly, AT&T had competition. Vail solved this problem in three ways. First, he decided AT&T must have the very best phone system available: he committed the company to building a long-distance system that would cross the entire US. To do this he knew he would have to invest in scientific research, and he encouraged the development of AT&Ts own laboratory, Bell Labs. Second, he cooperated with the competitors, leasing them the use of AT&T's phone lines. Third, he managed to convince the public and the government that the best possible phone system was one that could provide "universal service" around the country -- in essence, the best phone system would come from a monopoly like AT&T.
I discovered today that when I canceled the DSL service that at&t/SBC was unable to provide to my new residence, after two weeks of frustration and confusion, they decided to cancel POTS service too. As of yesterday. Without notice. Does this company actually want me as a customer?

Thursday, March 13, 2008

USENIX to make all its conference proceedings freely available

USENIX to make all its conference proceedings freely available: Free, as in software. [...] Unfortunately, other major technical societies that sponsor conferences and journals still cling to the antiquated notion, rooted in a rapidly-disappearing print-based publishing economy, that they naturally "own" the writings that volunteer authors, editors and reviewers produce. These organizations, which insist on copyright control as a condition of publication, argue that the sale of conference proceedings and journal subscriptions provides an essential revenue stream that subsidizes their other good works. But this income, however well it might be used, has evolved into an ill-gotten windfall. [...] Organizations devoted to computing research should recognize this not-so-new reality better than anyone. It's time for ACM and IEEE to follow USENIX's leadership in making scientific papers freely available to all comers. Please join me in urging them to do so. (Via Matt Blaze's Exhaustive Search.)

Hear, hear! Well done, USENIX.

Fractal Wrongness

Fractal Wrongness: [...]

Funny Pictures

Doesn't that perfectly describe your experience arguing with creationists, intelligent design proponents, crystal healers, holocaust deniers, global warming deniers, and 9/11-truthers? (Via Recursivity.)

I'd have preferred the title to be simply: Wrong at any scale.

Tuesday, March 11, 2008

Real service

While the big service providers like at&t and Amazon repeatedly fail me, the very small service provider at Sol Mountain Touring, with a bit of help from 60 cm of fresh powder, gave me and a dozen others a fantastic week of backcountry ski touring that kept us safe and got us to some of the best powder skiing I ever experienced. More details later when I have enough bandwidth from the monopolies to upload my pictures.

Is service failure the 21st century invariant?

Steve recommended Toumani Diabaté's latest album Mondé Variations. Being a Diabaté fan away from a good independent CD store, I rushed to find it on Amazon MP3. It was there, and I downloaded the small album download seed .amz file. I double-clicked on it, the Amazon downloader started, but then I had to pause the downloader because the net were I was (the Monashee backcountry) was just too slow for the download. I had got no tracks at that point. I resumed the download when I returned to faster net. But the downloader claimed that the tracks could not be downloaded. After several go-rounds with Amazon customer service in which they were unable to fix the problem, I asked for my money back.

The question is, does anybody in consumer services care any longer about reliability? I would be willing to pay CD prices (higher than what Amazon MP3 charges) for a solid, high-quality download service. No such luck.

Broadband? What broadband?

You'd think that in Palo Alto, this most wired of towns, it would be easy to get high-speed internet. Not a chance. After the embarrassing failures of both at&t's and Comcast's online ordering systems, we used the phone to call at&t (née SBC) to get POTS and DSL to our Palo Alto home. What should have been a trivial process turned into a comedy of errors (like giving us a business number and having to backtrack to a residential number, making Ana waste countless hours with halfwitted service representatives). This was all supposedly fixed and DSL service should have been on from 8 pm today. Of course not. After 45 minutes with technical support, they concluded that I have to call provisioning during business hours tomorrow, because there's some mysterious notation in our order record that suggests the provisioning is not finished. I will call them, most likely to cancel their DSL service.

In the meanwhile, I give thanks to T-Mobile for a not too slow EDGE network around here, which I'm using to get internet until the telecom monopolies deign to give me some wired bits.

Sunday, February 17, 2008

Phone company closed on Sundays

Phone company closed on Sundays: After trying the various phone numbers on the AT&T Wireless site, including 1-800-331-0500, 611 from my cellphone, and 800‑288‑2747 from GetHuman.com, it seems that AT&T provides no customer service on Sunday. So, if your phone or their software is broken, you are SOL. (Via Joho the Blog.)

It's not just Wireless. I tried to get service Friday evening for the Palo Alto townhouse I'm moving into. The at&t/SBC site led me through a long set of choices, just to dump me at checkout because of "extended maintenance." Tried again yesterday. Another problem cropped up, an option I had selected on Friday was no longer available. And I was dumped at checkout again. And many other mis/malfuctions. No phone number to buy service. This is what an unregulated monopoly is like.

Sunday, February 10, 2008

East to West

East to West

Early January, we moved to Mountain View where I joined Google Research, on leave from Penn. So much is still provisional (but isn't it always?), but I thought these few pictures got a bit of what we love and miss in Philadelphia, and a bit of what we love in California.

Complexity Illness

Complexity Illness: One of the enduring stereotypes of academia is that people spend a great deal of intelligence, time, and effort finding complexity rather than simplicity. This is at least anecdotally true in my experience.[...]Who submitted a paper to ICML violating the 8 page minimum? Every author fears that the reviewers won’t take their work seriously unless the allowed length is fully used. (Via Machine Learning (Theory).)

These claims come from a highly idealized model of scientific research. Most good ideas start their public careers in a messy, overly complicated state. Scientific ideas, like computer programs, need to be refactored repeatedly by the relevant community before they reach the conciseness and clarity that John hopes for. In his wonderful collection Indiscrete Thoughts, Gian-Carlo Rota wrote:

Every mathematical theorem is eventually proved trivial. The mathematician's ideal of truth is triviality, and the community of mathematicians will not cease its beaver-like work on a newly discovered result until it has shown to everyone's satisfaction that all the difficulties in the early proofs were spurious, and only an analytic triviality is to be found at the end of the road.

My conference submissions often hit the limit because at the time of submission I haven't understood fully what I am proposing, or what experiments are the best to test what I am proposing. Ten years later, if the idea was good enough, it is likely that it and the most revealing experiments could have been written up in half the space. But the only reason that would be possible is that the work was published, and the debate that followed was crucial in distilling out what really mattered from the original rough brew.

As I tell anyone I am advising, the real value of publication is to contribute to the ongoing dialog of science, and learn from the responses more than we could ever learn by thinking alone in our offices. Dialog is messy, ideas come out half-formed, we hesitate, we track back, we speak over each other, we suffer from esprit d'escalier. Waiting to engage until we have the perfect formulation may reduce the chance of embarrassment, but it would also deaden a lively exchange, and take away delicious rejoinder opportunities for our debating opponents.

Saturday, January 19, 2008

Personal control of personal information

Personal control of personal information: [...] there's a very interesting move afoot to set up an open standard to describe user "attention" data - which I gather includes browsing history, mostly, but could also certainly include any other information about what a user is interested in...Netflix reviews, Amazon purchases, search queries, you name it. The hope is to break this data away from the many different sites (each if which controls a piece of it) and put it in the hands of users, who can then go and get purchase recommendations (or what have you) from whoever does the best job. [...] [...] I don't see any particularly horrible technical roadblocks, but I do see a lot of interesting technical problems (e.g. reference resolution over this data!) and of course there might be pushback from the people that control the data now. (Via Cranial Darwinism.)

I see two big technical and political obstacles:

  • Privacy and security controls: how can the user control in a sufficiently fine-grained way who can use what pieces of their attention data for what? There's no solution on the horizon that avoids a trusted third party, and even then it will be difficult for the user and the various parties to reach common understanding about particular uses (see the recent issues with social networking sites, which are not just political).
  • Data collector confidentiality: Attention data related to a particular site, say an online merchant leaks out a bit of information about that site's operations. If enough attention data collected by Acme Corp. is made available by Acme's users to BottomFeeder Inc. for a consideration, BottomFeeder may be able to reverse engineer important aspects of Acme's recommender system, for instance. Preventing such leakages is very hard.

Tuesday, January 1, 2008

Volcano erupts in southern Chile (AP)

Volcano erupts in southern Chile (AP): The Llaima volcano erupts spewing lava, ash and smoke in Conguillio National Park, some 650 kms (403 miles) south of Santiago, Tuesday, Jan. 1st, 2008. Even though 150 people were evacuated from the area, no casualties or damage were reported, officials said. (AP Photo/Hector Andrade)The Llaima volcano erupts spewing lava, ash and smoke in Conguillio National Park, some 650 kms (403 miles) south of Santiago, Tuesday, Jan. 1st, 2008. Even though 150 people were evacuated from the area, no casualties or damage were reported, officials said. (Via Yahoo! News - Science.)

I've part-way to the summit of Llaima in August 2006, but incoming weather prevented a summit attempt. I had hoped to go back to this beautiful, remote volcano in the next year or two, but now it may have to wait, especially if the road and lodge have been damaged.