Sunday, March 14, 2010

The Economist has an interesting article, Data Data Everywhere, describing the 'industrial revolution in data'. The key point (as shown in the chart) is that the ability to collect data is exceeding the capacity to store it.

This has two interesting consequences. First, the ability to store data has exceeded human capacity to capture 'useful' information since the origins of writing. Thus, for the first time in ages we are entering a period where that isn't the case. Second, the capacity to extract knowledge from 'big data' becomes the limiting factor. In some sectors this isn't entirely novel. Thus, for example, what Derek de Solla Price labeled 'big science' -- large scale collaborative research with big budgets, big staffs, big machines and big laboratories like the human genome project or CERN -- has been a major factor in scientific knowledge generation since WWII. What we now have is the spread of this focused ability into the wider society with the resulting ability of mega-corporations (think Wal-Mart and their ability to collect and manage information about store sales, product delivery, etc) as the only ones with the resources necessary to store and extract the relevant information. A similar example is found in Goldman Sachs' use of 'flash trading'.

1 comment:

  1. This reminds me of Luhmann's theory that the speed of transactions in the economy is now so fast that it has exceeded the capacity of economic actors to control those transactions. Transactions are now so fast that they lose the "structure" that contains them. Chaos ensues. Markets crash. Likewise, When we can generate more data than we can analyze, we lose the "container" or structure that makes the information useful to us. As with transactions, due to the sheer volume and speed of generation, data spins out of control. This could result in the misuse of data, causing humans to make very bad decisions based on incomplete or faulty data analysis. Chaos.