What is big data?
I’ve worked in digital information startups for the past 15 years. The notion of working with large volumes of advertising and customer data for predictive insights and better decision making is nothing new.
However, the technology, analytics and venture-capital communities have finally found a word that resonates with the idea of ever-expanding data sets — a constant in our world. That word is “big data”.
Being a marketing and word wonk, I’m quick to point out that the term big data is jargon. It’s become so ubiquitous that it risks one of two outcomes: becoming cliche, or becoming nothing more than a clunkier variation of the word “data”. That’s why the term bothers me.
Instead of dwelling on semantics, I thought it would be more worthwhile to unpack the intended meaning of big data. I’m no techie, so I turned to Sanjay Gupta, my colleague at Clickable, who leads our engineering team, including our big-data infrastructure deployments.
What is big data? According to Sanjay:
“Big data” refers to the huge volume of data that cannot be stored and processed using conventional databases in a reasonable time. While the storage problem has been solved by large offline storage systems, analysis tasks need to combine data in many ways; data read from one source may need to be combined with the data from multiple sources. This is where conventional data processing systems fail. Big-data technology tackles this problem by running logic on large data chunks in parallel and then combining them together in a manner that gives global results. We’re now mining data sets that are so large in volume that they become truly representational of the entire problem space. This leads to more accurate predictions and more valuable insights.
What is the promise of big data in marketing?
The ability to leverage big data has opened opportunities for startups to build tools and platforms that can surface insights around demand, supply, consumer behavior, segmentation, positioning and targeting. Big data is a natural fit for marketing because of the huge volumes of sample data that can drive analysis and more meaningful predictions.
Want to learn more about big data?
Go check out my full Q&A with Sanjay over at MediaPost.
In information technology, big data consists of data sets that grow so large that they become awkward to work with using on-hand database management tools. Difficulties include capture, storage, search, sharing, analytics, and visualizing. This trend continues because of the benefits of working with larger and larger data sets allowing analysts to “spot business trends, prevent diseases, combat crime.” Though a moving target, current limits are on the order ofpetabytes, exabytes and zettabytes of data. Scientists regularly encounter this problem inmeteorology, genomics, connectomics, complex physics simulations, biological and environmental research, Internet search, finance and business informatics. Data sets also grow in size because they are increasingly being gathered by ubiquitous information-sensing mobile devices, aerial sensory technologies (remote sensing), software logs, cameras, microphones,Radio-frequency identification readers, and wireless sensor networks. The world’s technological per capita capacity to store information has roughly doubled every 40 months since the 1980s (about every 3 years) and every day 2.5 quintillion bytes of data are created.
Leave a comment