Monday, June 15, 2009

An economic solution to reviewing load

An economic solution to reviewing load: Hal Daume at the NLP blog bemoans the fact that “there is too much to review and too much garbage among it” and wonders “whether it’s possible to cut down on the sheer volume of reviewing”. [...] There is an economic solution to the problem that bears consideration: Charge for submission. This would induce self-selection; authors would be loathe to submit unless they thought the paper had a fair chance of acceptance. Consider a conference or journal with a 25% acceptance rate that charged, say, $50 per submission. (Via The Occasional Pamphlet.)

I entered the following comment in Stuart's blog:

I don’t think this adds up. Consider a typical academic CS research group with one professor and a few graduate students. As is typical, as a conference deadline approaches, they have several papers in the works, say four, in different states of completion; maybe one is in very good shape, two in fair shape, and the other in poor shape (these are again typical numbers in my experience). If just one paper is accepted, the professor and one student attend the conference, at a typical cost of $4000 for travel, accomodation, and conference registration. If three papers are accepted, maybe the professor and three students attend, at a total cost of $8000. Compared with those costs, the difference between $50 and $200 is utterly trivial; just a couple of slightly better meals, or a cab to the airport instead of the shuttle, would make the difference. The only way this could work would be to have submission charges that are significant relative to the other costs of paper creation and presentation. But if the charges were that high for a given venue, then 1) other venues would undercut it, and 2) rejections would lead to open warfare with authors claiming they were swindled of their fees by inappropriate rejections. The lack of incentive alignment between getting as much from submission fees as possible and doing as little in reviewing as possible would be very destructive of already fragile institutions.

Update: In his comment below, Mark suggests that a two-tier system can quickly get rid of most of the bad submissions, leaving more reviewing capacity for the remaining submissions. Unfortunately, as Darwin noted, “Ignorance more frequently begets confidence than does knowledge.” I've seen way too many highly confident reviewer dismissals of valuable work they were unwilling or unable to understand. Even a relatively low false-positive rate in the initial screening would be enough to create a much bigger hurdle for those submissions that are harder to understand because they are off the beaten path.

1 comment:

Mark Dredze said...

If the goal is to remove the garbage, such drastic actions aren't really necessary. A two tier model is very effective. I find that it takes a very short amount of time to find a garbage paper. Having only 1-2 reviewers do a simple first pass would reduce the overall load considerably.