Monday, June 15, 2009

Rick and friends on Lassen

Mount Lassen 6/12-14/2009: Rick, Erin, Adam, and I decided to brave the rain/thunder showers and headed up to Mt Lassen for the weekend. [...]

(Via Rick's World.)

I so much wish I could have been there... But the ankle still needs work :(.

An economic solution to reviewing load

An economic solution to reviewing load: Hal Daume at the NLP blog bemoans the fact that “there is too much to review and too much garbage among it” and wonders “whether it’s possible to cut down on the sheer volume of reviewing”. [...] There is an economic solution to the problem that bears consideration: Charge for submission. This would induce self-selection; authors would be loathe to submit unless they thought the paper had a fair chance of acceptance. Consider a conference or journal with a 25% acceptance rate that charged, say, $50 per submission. (Via The Occasional Pamphlet.)

I entered the following comment in Stuart's blog:

I don’t think this adds up. Consider a typical academic CS research group with one professor and a few graduate students. As is typical, as a conference deadline approaches, they have several papers in the works, say four, in different states of completion; maybe one is in very good shape, two in fair shape, and the other in poor shape (these are again typical numbers in my experience). If just one paper is accepted, the professor and one student attend the conference, at a typical cost of $4000 for travel, accomodation, and conference registration. If three papers are accepted, maybe the professor and three students attend, at a total cost of $8000. Compared with those costs, the difference between $50 and $200 is utterly trivial; just a couple of slightly better meals, or a cab to the airport instead of the shuttle, would make the difference. The only way this could work would be to have submission charges that are significant relative to the other costs of paper creation and presentation. But if the charges were that high for a given venue, then 1) other venues would undercut it, and 2) rejections would lead to open warfare with authors claiming they were swindled of their fees by inappropriate rejections. The lack of incentive alignment between getting as much from submission fees as possible and doing as little in reviewing as possible would be very destructive of already fragile institutions.

Update: In his comment below, Mark suggests that a two-tier system can quickly get rid of most of the bad submissions, leaving more reviewing capacity for the remaining submissions. Unfortunately, as Darwin noted, “Ignorance more frequently begets confidence than does knowledge.” I've seen way too many highly confident reviewer dismissals of valuable work they were unwilling or unable to understand. Even a relatively low false-positive rate in the initial screening would be enough to create a much bigger hurdle for those submissions that are harder to understand because they are off the beaten path.

Sunday, June 7, 2009

This American Life on the Rating Agencies

This American Life on the Rating Agencies: This weekend's 'This American Life' is about the rating agencies. [...] A few excerpts:

"We hired a specialist firm that used a methodology called maximum entropy to generate this equation," says Frank Raiter, who until 2005 was in charge of rating mortgages at Standard and Poors. "It looked like a lot of Greek letters."

The new bonds were based on pools of thousands of mortgages. If you bought one of these bonds, you were basically loaning money to people for their houses. What the equation tried to predict was how likely the homeowners were to keep making payments.

The system made sense, Raiter says, until loan issuers started offering mortgages to people who didn't have great credit and in some cases didn't have a job.

Raiter says there wasn't a lot of data on these new homebuyers. He says he told his bosses they needed better data and a better model for assessing the riskiness of the loans.
(Via Calculated Risk.)

E. T. Jaynes must be turning in his grave. I'll listen to the podcast soon, but this quote waves a big red flag of overfitting. The last ten years of maxent-related work in machine learning and natural-language processing sow clearly that the maximum entropy principle on its own can be highly misleading when it is applied to data drawn from long-tailed distributions. That's why there's thriving research on ways of regularizing maxent models, for example by replacing equality constraints by box constraints. But even with decent regularization, maxent models are only as good as their choice of event types (features) over which to compute sufficient statistics. If there are correlations in the real world that are not represented by corresponding features in the model, the model may be be overly confident in its predictions.

Maximum entropy, like other statistical-philosophical principles (you know who you are), carries the unfortunate burden of a philosophical foundation that may to some appear to guarantee correct inference without the need for empirical validation. In the case of maximum entropy, the familiar argument is that it produces the least informative hypothesis given the evidence. That seems to imply safety, lack of overreaching. Unfortunately, the principle doesn't say anything about quality of evidence. What if the “evidence” is noisy, incomplete, biased? The principle doesn't say anything about finite-sample effects, as it came from statistical mechanics where the huge number of molecules made (then) those a non-issue. But in biological, social and cultural processes (genomics, language, social relationships, markets) we may as well bet that small-sample effects are never negligible.

Friday, June 5, 2009

Recently read

I have a pile of new fiction on my bedside table, but non-fiction is still winning:

The first two were a very entertaining visits to the watery world, which I mostly love from solid ground given my susceptibility to seasickness. The last one was hard to put down, although part of it was that embarrassing attraction to the scene of a disaster. And even Gillian Tett doesn't do what I think such articles and books should have done from the beginning: borrow Alice, Bob, Carol, Dave, and Eve from cryptography to draw protocol diagrams for CDOs, CDSs, ABSs, and all that insanity of unstable transactions.