small data

| No Comments | No TrackBacks

No one talks about big data any more, says Slate's Will Oremus. "Five years ago," he writes, "an article in the New York Times' Sunday Review heralded the arrival of a new epoch in human affairs: 'The Age of Big Data':"

Society was embarking on a revolution, the article informed us, one in which the collection and analysis of enormous quantities of data would transform almost every facet of life. No longer would data analysis be confined to spreadsheets and regressions: The advent of supercomputing, combined with the proliferation of internet-connected sensors that could record data constantly and send it to the cloud, meant that the sort ┬Čof advanced statistical analysis described in Michael Lewis' 2003 baseball book Moneyball could be applied to fields ranging from business to academia to medicine to romance. Not only that, but sophisticated data analysis software could help identify utterly unexpected correlations, such as a relationship between a loan recipient's use of all caps and his likelihood of defaulting. This would surely yield novel insights that would change how we think about, well, just about everything.

"Big data," he continues, "helps to power the algorithms behind our news feeds, Netflix recommendations, automated stock trades, autocorrect features, and health trackers, among countless other tools:"

But we're less likely to use the term big data these days--we just call it data. We've begun to take for granted that data sets can contain billions or even trillions of observations and that sophisticated software can detect trends in them.

Oremus cites Cathy O'Neil's Weapons of Math Destruction and Frank Pasquale's The Black Box Society as illustrations of "the fetishization of data, and its uncritical use, that tends to lead to disaster," and suggests "Another possible response to the problems that arise from biases in big data sets:"

Small data refers to data sets that are simple enough to be analyzed and interpreted directly by humans, without recourse to supercomputers or Hadoop jobs. Like "slow food," the term arose as a conscious reaction to the prevalence of its opposite.

Martin Lindstrom's 2016 book Small Data: The Tiny Clues That Uncover Big Trends looks intriguing, as Oremus concludes:

There is some hope, then, that in moving away from "big data" as a buzzword, we're moving gradually toward a more nuanced understanding of data's power and pitfalls. In retrospect, it makes sense that the sudden proliferation of data-collecting sensors and data-crunching supercomputers would trigger a sort of gold rush, and that fear of missing out would in many cases trump caution and prudence. It was inevitable that thoughtful people would start to call our collective attention to these cases, and that there would be a backlash, and perhaps ultimately a sort of Hegelian synthesis.

No TrackBacks

TrackBack URL:

Leave a comment

About this Entry

This page contains a single entry by cognitivedissident published on October 16, 2017 8:51 AM.

Threepers was the previous entry in this blog.

To Kill a Mockingbird pulled from reading list is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.

Monthly Archives


  • About
  • Contact
OpenID accepted here Learn more about OpenID
Powered by Movable Type 5.031