Start-Ups Are Mining Hyperlocal Information for Global Insights

The company he co-founded, Premise, created a smartphone application that is now used by 700 people in 25 developing countries. Using guidance from Mr. Soloff and his co-workers, these people, mostly college students and homemakers, photograph food and goods in public markets.

By analyzing the photos of prices and the placement of everyday items like piles of tomatoes and bottles of shampoo and matching that to other data, Premise is building a real-time inflation index to sell to companies and Wall Street traders, who are hungry for insightful data.

“Within five years, I’d like to have 3,000 or 4,000 people doing this,” said Mr. Soloff, who is also Premise’s chief executive. “It’s a useful global inflation monitor, a way of looking at food security, or a way a manufacturer can judge what kind of shelf space he is getting.”

Collecting data from all sorts of odd places and analyzing it much faster than was possible even a couple of years ago has become one of the hottest areas of the technology industry. The idea is simple: With all that processing power and a little creativity, researchers should be able to find novel patterns and relationships among different kinds of information.

For the last few years, insiders have been calling this sort of analysis Big Data. Now Big Data is evolving, becoming more “hyper” and including all sorts of sources. Start-ups like Premise and ClearStory Data, as well as larger companies like General Electric, are getting into the act.

A picture of a pile of tomatoes in Asia may not lead anyone to a great conclusion other than how tasty those tomatoes may or may not look. But connect pictures of food piles around the world to weather forecasts and rainfall totals and you have meaningful information that people like stockbrokers or buyers for grocery chains could use.

And the faster that happens, the better, so people can make smart — and quick — decisions.

“Hyperdata comes to you on the spot, and you can analyze it and act on it on the spot,” said Bernt Wahl, an industry fellow at the Center for Entrepreneurship and Technology at the University of California, Berkeley. “It will be in regular business soon, with everyone predicting and acting the way Amazon instantaneously changes its prices around.”

Standard statistics might project next summer’s ice cream sales. The aim of people working on newer Big Data systems is to collect seemingly unconnected information like today’s heat and cloud cover, and a hometown team’s victory over the weekend, compare that with past weather and sports outcomes, and figure out how much mint chip ice cream mothers would buy today.

At least, that is the hope, and there are early signs it could work. Premise claims to have spotted broad national inflation in India months ahead of the government by looking at onion prices in a couple of markets.

The photographers working for Premise are recruited by country managers, and they receive 8 to 10 cents a picture. Premise also gathers time and location information from the phones, plus a few notes on things like whether the market was crowded. The real insight comes from knowing how to mix it all together, quickly.

Price data from the photos gets blended with prices Premise receives from 30,000 websites. The company then builds national inflation indexes and price maps for markets in places like Kolkata, India; Shanghai; and Rio de Janeiro.

Premise’s subscribers include Wall Street hedge funds and Procter & Gamble, a company known for using lots of data. None of them would comment for this article. Subscriptions to the service range from $1,500 to more than $15,000 a month, though there is also a version that offers free data to schools and nonprofit groups.

The new Big Data connections are also benefiting from the increasing amount of public information that is available. According to research from the McKinsey Global Institute, 40 national governments now offer data on matters like population and land use. The United States government alone has 90,000 sets of open data.

This article has been revised to reflect the following correction:

Correction: November 10, 2013

An earlier version of this article misidentified the month in which the creators of the Spark open-source software received $14 million in funding. It was in October, not November.