Saturday, January 24, 2009

Lessons to learn from the financial crisis

Lessons to learn from the financial crisis: [...] Today's deep thought is: how much of modern computer science is about making the complexities in our world simpler and more understandable - and how much is making the world more complex and more opaque? And how complicated can a system be and still be controlled, maintained, and regulated by human beings? (Via Cranial Darwinism.)

Read William's whole post, very interesting. I was talking with a friend a couple of days ago about the current arguments about different alternative energy sources. He argued that we need for detailed models that take into account physical an market factors in assessing the real costs and benefits from each proposal. I agree, but I added that the models have to be transparent, available for analysis and critique, and describable at different levels of detail for technical analysis, policy study, and general public debate. The opacity of financial models and instruments is not just dangerous in the short term, but it also induces cynicism and facilitates demagoguery, as the current debates on the crisis and rescue packages show.

Featuritis in NLP

Featuritis in NLP: [...]The primary issue here doesn't seem to be the representation that's making things so much slower to train, but the fact that it seems (from experimental results) that you really have to do the multitask learning (with tons of auxiliary problems) to make this work. This suggests that maybe what should be done is just to fix an input representation (eg., the word identities) and then have someone train some giant multitask network on this (perhaps a few of varying sizes) and then just share them in a common format. [...] At the end of the day, you're going to still have to futz with something. You'll either stick with your friendly linear model and futz with features, or you'll switch over to the neural networks side and futz with network structure and/or auxiliary problem representation. (Via natural language processing blog.)

Well, duh. Why do you think NN approaches in NLP never get to a whatever level of performance first, but they come only afterwards with the claim that they can achieve similar performance "without feature engineering"? Could it be because the feature engineering for linear models was what taught the NN practitioner how to choose their non-linear function class?

The situation seems very different in image processing/vision, where NN methods have achieved superior results first for at least some tasks. I don't think it's just a matter of there being more or smarter NN practitioners in those areas (although that might be argued), but also that images have a natural neighborhood structure (hence the success of convolutional nets, for example), unlike the discrete, heavy-tailed language domain.

Monday, January 19, 2009

Wu on Zittrain's Future of the Internet

Wu on Zittrain's Future of the Internet: [...]Tim Wu has a great review of Jonathan Zittrain's book.[...]On the Internet, distributing content is so cheap that economies of scale in distribution just don't matter. On a per-reader basis, my personal blog certainly costs more to operate than CNN. But the cost is so small that it's simply not a significant factor in deciding whether to continue publishing it. Even if the larger sites capture the bulk of the readership and advertising revenue, that doesn't preclude a "long tail" of small, often amateur sites with a wide variety of different content. (Via Freedom to Tinker.)

History of skiing

My lovely daughter Elena gave me Two Planks and a Passion by Roland Huntford, a recently published history of skiing. It's an entertaining book that brings alive many quirky technique and equipment innovators from Norway to California, and the historical foundations of divisions and resentments that are still with us today: nordic vs alpine, tele vs fixed heel, lift-served vs backcountry, competition vs freedom. Contrary to the IHT review linked above, I don't think that the book is swamped by trivia — although the style is a bit breathless at times. On the contrary, if it has a weakness is that it does not give enough detail to allow us to visualize early technique and gear. Yes, the author is a clear nordic partisan and his romantic dismay over the commercialization of alpine skiing is a bit overwrought, especially when he goes rhapsodic over the invention of synthetic ski waxes, which can be hardly be seen as a victory of disinterested amateurism.

Busy busy

I was in Philly for two dissertation defenses (congratulations Drs. Dredze and Liu!) and two thesis proposals, and talk with Penn colleagues. Flew back to SFO yesterday. Huge temperature inversions apparent on the Wasatch front, Western Nevada, and even over the Bay Area. Mono Lake, Lee Vining Canyon, and Mount Dana were visible from my window. Tahoe, Lassen, and Shasta were visible to the North (no photos, I forgot my camera).

We need more snow.