The power of regression analysis by Malcolm Gladwell

As Malcolm Gladwell expressed in a New Yorker article, when it comes to art, most people fall in the Hume camp of reasoning, but a defiant minority always falls in the Kames camp:

“Something has long been argued about art: there is no way of getting beyond one’s own impressions to arrive at some larger, objective truth. There are no rules to art, only the infinite variety of subjective experience. “Beauty is no quality in things themselves,” the eighteenth-century Scottish philosopher David Hume wrote. “It exists merely in the mind which contemplates them; and each mind perceives a different beauty.” Hume might as well have said that nobody knows anything.

But Hume had a Scottish counterpart, Lord Kames, and Lord Kames was equally convinced that traits like beauty, sublimity, and grandeur were indeed reducible to a rational system of rules and precepts. … He genuinely thought that the superiority of Virgil’s hexameters to Horace’s could be demonstrated with Euclidean precision, and for every Hume, it seems, there has always been a Kames—someone arguing that if nobody knows anything it is only because nobody’s looking hard enough.”

I first came across a follower of Kames while studying econometrics at Princeton. My professor, Orley Ashenfelter, wrote a model to predict the quality of wines in Bordeaux based on the temperature and precipitation that year. His model could predict the quality of the wine before it was even on sale and had been tasted by experts like Robert Parker. Needless to say the wine community was outraged. The typical reaction was “How can you reduce our noble art to mere equations? How can you claim to know the quality of the wine without tasting it?” The critics were pointing out “flaws” such as “the model only predicted the exact quality of the wine in one specific year” failing to understand that econometrics is about confidence intervals. The model and the experts only disagreed on one year, 1964, where the experts suggested the wine was not very good, while the model suggested otherwise. Needles to say Professor Ashenfelter’s cellar is full of wine from 1964 🙂

Gladwell’s article covers the travails of a few brave souls as they try to predict hit songs and hit movies. He first describes how a small New York based startup called Platinum Blue analyzed the mathematical relationships among a song’s structural components. The applied their model to thousands of songs and noticed that all the hits came out of a predictable and highly conserved set of mathematical patterns. As McCready, the firm’s founder explains this does not mean that all songs that conform to the pattern will be hits, they still have to “sound right”, but almost certainly songs that fall out of that pattern will not be hits. Interestingly enough, when they first ran their analysis, Platinum Blue was really excited about by the Norah Jones record “Come Away with Me”. It went on to sell twenty million copies and win eight Grammy awards.

Gladwell then describes the story of Dick Copaken, a lawyer and cinephile, who created a company called Epagogix that analyzes screenplay elements predict US box office receipts. Predictably, the industry reception was terrible at first: how could outsiders claim they knew more than the insiders? But after a number of successful tests Epagogix is now working with major studios.

It seems that art is more structured than we thought after all – not that we necessarily want all songs and movies to be blockbusters…

Read the full article at:
http://www.newyorker.com/archive/2006/10/16/061016fa_fact6?currentPage=1