Grids, made up of thousands of meters monitoring energy use throughout their network. Logistics companies are using sensors to track and manage the progress of goods in transit. Retailers are offering customers deals and discounts in exchange for the right to identify when they are in store via their smartphone. IBM is collaborating with medics at
Columbia University Medical Center and the University of Maryland School of Medicine to apply big data analytics tools to medical practice, while California-based start-up Apixio aims to improve information sharing between doctors by analysing everything from CT scans to emails. What links these big data applications is the need to track millions of events per second, and to respond in real time. Utility companies will need to detect an uptick in consumption as soon as possible, so they can bring supplementary energy sources online quickly. If retailers are to capitalise on their customers’ location data, they must be able to respond as soon as they step through the door. In a way, the term ‘big data’ fails to highlight the critical development. Yes, this trend will accelerate the expansion of data volumes, but those have always been growing. The significant change is the way in which the data is produced, and the way it must therefore be collected and analysed.
A NEW APPROACH Most traditional BI vendors would argue, of course, that their high-end tools are well able to process large volumes of data and in real time. But proponents of big data technologies argue that these traditional BI tools were not designed for data created in the manner described above. In the conventional model of business
intelligence and analytics, data is cleaned, cross-checked and processed before it is analysed, and often only a sample of the data is used in the actual analysis. This is possible because the kind of data that is being analysed – sales figures or stock
attempting to make sense of masses of data that has no real structure, no schema”, says Bhambhri. “This is data that can’t just be pushed into a structured repository.” Any organisation having to make sense of these torrents of data using traditional tools will be overwhelmed, says Cloudscale’s McColl. “We need a different approach. The idea of storing it all in a database and then querying it is not going to work.” Just as the web giants were the first to experience the big data phenomenon, they were also the first to build tools with which to handle it.
Google’s landmark
Bill McColl, Cloudscale “What’s happened in the past five years has been an incredible exponential explosion”
counts, for example – can easily be arranged in a pre-ordained database schema, and because BI tools are often used simply to create periodic reports. According to Anjul Bhambhri, vice
president for ‘big data products’ at IBM, this is not true of the kind of data that is now exploding in volume, such as website clicks, continuous meter readings or GPS tracking data. With big data, “organisations are
www.information-age.com
publication of the principles of its GFS file system and MapReduce algorithm in 2004 laid the foundation for a new breed of data capture and management tools based
on massively parallel processing. Both Yahoo and Facebook have added momentum by sharing some of their own expertise, mainly through the Apache Hadoop programme (see box). Big data techniques may have been
developed in the engineering departments of these web powerhouses, but they may well have remained there were it not for a class of companies making the technology accessible to mainstream businesses, says
INFORMATIONAGE APRIL2011 17
Page 1 |
Page 2 |
Page 3 |
Page 4 |
Page 5 |
Page 6 |
Page 7 |
Page 8 |
Page 9 |
Page 10 |
Page 11 |
Page 12 |
Page 13 |
Page 14 |
Page 15 |
Page 16 |
Page 17 |
Page 18 |
Page 19 |
Page 20 |
Page 21 |
Page 22 |
Page 23 |
Page 24 |
Page 25 |
Page 26 |
Page 27 |
Page 28 |
Page 29 |
Page 30 |
Page 31 |
Page 32 |
Page 33 |
Page 34 |
Page 35 |
Page 36 |
Page 37 |
Page 38 |
Page 39 |
Page 40 |
Page 41 |
Page 42 |
Page 43 |
Page 44 |
Page 45 |
Page 46 |
Page 47 |
Page 48 |
Page 49 |
Page 50 |
Page 51 |
Page 52