And the winner of the next election is...Big Data

Nate Silver, the statistician behind the New York Times' fivethirtyeight blog correctly predicted the outcome of the US presidential election in all 50 states, convincingly demonstrating the power of big data: but what if that power were applied pro-actively rather than merely predictively?

The Obama camp made good use of data mining and analysis techniques in this year's campaign. Next time around lessons learnt from the experience are likely to be applied even more aggressively, probably by both sides.

In an article in the Sydney Morning Herald, its US correspondent, Nick O'Malley reported how the Obama re-election team, led by former White House deputy chief of staff Jim Messina, built up a massive and sophisticated database of the electorate, which it exploited to good effect.

"The 2008 campaign led by David Axelrod was famous for the way it harnessed a massive network of volunteers across the country and collected information from them - demographics, spending and donation habits, political concerns, voting habits," O'Malley said.

{loadposition stuart}"The problem was that the databases in Chicago could not talk to one another. The donations team kept its data separate from the political team's, which could not access the voter registration campaign team's database.

"Messina's first task was to marry the databases. He raided Silicon Valley for its best and brightest, appointed a chief scientist and moved the new staff into the growing offices in Chicago to write new code."

Naturally the Obama camp has not revealed all its secrets. "Back in January a Newsweek reporter visited the Chicago headquarters to explore the mine, and found a staff committed to protecting its secrets" O'Malley said. Nevertheless, he was able to glean a couple of insights from other published reports.

"On January 22 a young woman in Ohio received an email from Obama's Chicago headquarters telling her the President's healthcare reforms would mean contraception was fully covered by insurance, Slate magazine reported…The email had been crafted specifically for her, a young, liberal, single woman living in a socially conservative part of a crucial state."

And: "Obama's dataminers noticed that women on the west coast between 40 and 49 would hand over donations for the chance to have dinner with Obama and George Clooney, Time has reported. The fund raisers wanted to repeat the success of the Clooney fund raiser on the east coast, so they turned back to the machine to find out which celeb would attract the same demographic. Soon the campaign was offering donors a chance to dine with the President and Sarah Jessica Parker."

The Obama camp's data mining and manipulation also enabled it to make a fair first of predicting the outcome. "Messina…correctly predicted that Obama would win the Latino vote by 71 percent and that the minority turnout would be 28 percent of the vote, and that Obama would win 80 percent of it," O'Malley reported.

This year's US elections - presidential and congressional combined -were famously the most expensive ever. The cost of the presidential race has been put at $US2.6b. Much of this went on scattergun tactics like TV advertising.

By the time the next election comes around in four years time the techniques of big data will have improved significantly, as will the ability to access processing power on tap from the cloud. Candidates' ability to harness these technologies is likely to be a much bigger factor in determining the election outcome.

This article first appeared on iTWire, Australia's leading independent IT&T news and information source.