Saturday, April 23, 2011

Technology Has Social Consequences

Technology Has Social Consequences | May 2011 | Communications of the ACM: The loss in quality of conference reviewing is just one result of the move to virtual PC meetings. Another outcome is the loss of socialization that took place in PC meetings. It is this lost socialization that contributed to a senior researcher being ignorant of one of the most basic rules of scholarly reviewing.

I quoted the paragraph in Moshe's article that best summarizes his main point. The article is worth reading in full. However, I disagree with both its alleged empirical claims and its main argument. Moshe claims that reviewing quality is down in CS conferences, and reviewers are less aware of good scholarly conduct. However, he provides no empirical evidence for these claims except for one anecdote and some vague impressions, which I could easily counter with old tales of reviewing incompetence and malice. He then proceeds to argue that those failings are the result of less face-to-face interaction among reviewers. Again, he provides no empirical evidence for the claim, and he does not consider alternative explanations. In particular, he does not consider the fact that many areas of CS have grown rapidly. For example, I just did a simple calculation to arrive at an estimate that computational linguistics as a field has grown at an average 6%/year over the last 28 years, making the field five times as big now as when I presented my first ACL paper. This growth forced conferences to adopt more complex structures, with areas, multiple tiers, electronic submission and review discussion, simply to scale up to the much larger population. We can argue about the specifics of reviewing mechanisms, but the old unitary program committee was already collapsing under the strain around 15 years ago for the first-tier conferences I have been involved in.

But there's an even bigger potential problem that Moshe does not discuss. As the field has matured, it takes longer and it is harder for someone to become a good reviewer because there's just more to know. Meanwhile, the number of people coming into the field continues to grow and the number of papers submitted grows in proportion. The result is then that the ratio of submissions to qualified reviewers increases. Less qualified reviewers are enrolled, or qualified reviewers are overloaded. Either way, review quality goes down. For all we know, it is this, and not Web-based program committees, that is the root cause of all the complaining about bad reviewing in the last few years.

Sunday, April 17, 2011

Sugarbowl to Squaw

Drove up to Sugarbowl yesterday (Saturday) to join three other ASI clients for a one-day ski traverse from Sugarbowl to Squaw guided by Logan Talbott. It's just under 8 miles as the crow flies but more like 12 miles given the terrain (once again forgot to set MyTracks on my phone to record the actual track, so this is based on what Logan said). We left the top of Sugarbowl's Lincoln lift at 9:06am and arrived at the Squaw parking lot at 3:59pm, a pretty good pace that included a beautiful lunch break at the top of Tinker Knob (photo). The forecast had been for cloudy skies with 30% chance of showers, but it was dry and fairly warm the whole day. Ski conditions ranged from good corn to very sticky mank, with the good surprise being the drop into Squaw, which had much better snow than we expected for mid-afternoon because of the cooling from the breeze and high clouds. Cold lager at the Cham was a perfect way to finish the day.