Unboxed: Big Data, Trying to Build Better Workers

In telephone call centers, for example, where hourly workers handle a steady stream of calls under demanding conditions, the communication skills and personal warmth of an employee’s supervisor are often crucial in determining the employee’s tenure and performance. In fact, recent research shows that the quality of the supervisor may be more important than the experience and individual attributes of the workers themselves.

New research calls into question other beliefs. Employers often avoid hiring candidates with a history of job-hopping or those who have been unemployed for a while. The past is prologue, companies assume. There’s one problem, though: the data show that it isn’t so. An applicant’s work history is not a good predictor of future results.

These are some of the startling findings of an emerging field called work-force science. It adds a large dose of data analysis, a k a Big Data, to the field of human resource management, which has traditionally relied heavily on gut feel and established practice to guide hiring, promotion and career planning.

Work-force science, in short, is what happens when Big Data meets H.R.

The new discipline has its champions. “This is absolutely the way forward,” says Peter Cappelli, director of the Center for Human Resources at the Wharton School of the University of Pennsylvania. “Most companies have been flying completely blind.”

Today, every e-mail, instant message, phone call, line of written code and mouse-click leaves a digital signal. These patterns can now be inexpensively collected and mined for insights into how people work and communicate, potentially opening doors to more efficiency and innovation within companies.

Digital technology also makes it possible to conduct and aggregate personality-based assessments, often using online quizzes or games, in far greater detail and numbers than ever before.

In the past, studies of worker behavior were typically based on observing a few hundred people at most. Today, studies can include thousands or hundreds of thousands of workers, an exponential leap ahead.

“The heart of science is measurement,” says Erik Brynjolfsson, director of the Center for Digital Business at the Sloan School of Management at M.I.T. “We’re seeing a revolution in measurement, and it will revolutionize organizational economics and personnel economics.”

The data-gathering technology, to be sure, raises questions about the limits of worker surveillance. “The larger problem here is that all these workplace metrics are being collected when you as a worker are essentially behind a one-way mirror,” says Marc Rotenberg, executive director of the Electronic Privacy Information Center, an advocacy group. “You don’t know what data is being collected and how it is used.”

Companies view work-force data mainly as a valuable asset. Last December, for example, I.B.M. completed its $1.3 billion acquisition of Kenexa, a recruiting, hiring and training company. Kenexa’s corps of more than 100 industrial organizational psychologists and researchers was one attraction, but so was its data: Kenexa surveys and assesses 40 million job applicants, workers and managers a year.

Big companies like I.B.M., Oracle and SAP are pursuing the business opportunity. So is eHarmony, the online matchmaking service. It announced in January that it would retool its algorithm for romance so it could examine employee-employer relationships, and enter the talent search business later this year.

THE penchant for digital measurement and monitoring seems most suited to hourly employment, where jobs often involve routine tasks. But will this technology also be useful in identifying and nurturing successful workers in less-regimented jobs? Many companies think so, and can point to some encouraging evidence.

Tim Geisert, chief marketing officer for I.B.M.’s Kenexa unit, observed that an outgoing personality has traditionally been assumed to be the defining trait of successful sales people. But its research, based on millions of worker surveys and tests, as well as manager assessments, has found that the most important characteristic for sales success is a kind of emotional courage, a persistence to keep going even after initially being told no.

The team of behavioral and data scientists at Knack, a Silicon Valley start-up firm, uses computer games and constant measurement to test emotional intelligence, cognitive skills, working memory and propensity for risk-taking. Early pilot testers include the NYU Langone Medical Center, Bain & Company and a unit of Shell, says Guy Halfteck, Knack’s C.E.O.

Unboxed: N.Y.U. Center Develops a ‘Science of Cities’

THE notion of a “science of cities” seems contradictory. Science is a realm of grand theory and precise measurement, while cities are messy agglomerations of people and human foible. But science is precisely the ambition of New York University’s Center for Urban Science and Progress. Founded last year, the center has been getting under way in recent weeks, moving into new office space and firing off its first project proposal to the National Science Foundation.

The center’s director is Steven E. Koonin, a Brooklyn native and graduate of Stuyvesant High School, who came to N.Y.U. after a stint in the Obama administration as the under secretary for science in the Department of Energy. He is both a theoretical physicist and science policy expert. The center shouldn’t lack for intellectual rigor.

The initiative at N.Y.U. is part of a broader trend: the global drive to apply modern sensor, computing and data-sifting technologies to urban environments, in what has become known as “smart city” technology. The goals are big gains in efficiency and quality of life by using digital technology to better manage traffic and curb the consumption of water and electricity, for example. By some estimates, water and electricity use can be cut by 30 to 50 percent over the course of a decade.

Cities from Stockholm to Singapore are deep into smart city projects. The market looms as big, lucrative business for technology companies. “The Smart City movement,” according to a report this month from IDC, a technology research firm, “is emerging and growing as a significant force of innovation and investment at all levels of government.” The N.Y.U. center’s partners include technology companies like I.B.M., Cisco Systems and Xerox, as well as universities and the New York City government.

City governments, like other institutions, have collected data for years to try to become more efficient. There have been some notable achievements, like CompStat, the New York Police Department’s system for identifying crime patterns, introduced in the mid-1990s and later widely adopted elsewhere.

What is different today, says Dr. Koonin, is that digital technologies — sensors, wireless communication, storage and clever software algorithms — are advancing so rapidly that it is becoming possible to see and measure activities in an urban environment as never before.

“We can build an observatory to be able to see the pulse of the city in detail and as a whole,” Dr. Koonin explains.

Dr. Koonin’s digital “observatory” of urban life raises questions about privacy. He is keenly aware of that issue, and vows that the center is engaged in science rather than surveillance. For example, individuals’ names or tax identification numbers would be stripped from personal records.

The collected data, he says, will be the raw material for modeling outcomes — say, the steps required to reduce electricity consumption in a high-rise office building or in an individual apartment. Those modeled predictions, he adds, can guide policy or inform citizens.

“I’d like to create SimCity for real,” Dr. Koonin says, referring to the classic computer simulation game.

To help, Dr. Koonin is forging partnerships with government laboratories to tap their expertise in building complex computer simulations, like climate models for weather prediction.

The path to SimCity will come step by step, through tackling specific projects. The first one is a program to monitor and analyze noise. The largest single cause of complaints to New York’s 311 phone and online service is noise. It is a quality-of-life issue, Dr. Koonin says, and one related to health, especially when noise disrupts sleep.

The 10-member project team includes music professors, computer scientists and graduate students. The group will use the city’s 311 data, but also plans to employ wireless sensors — tiny ones outside windows, noise meters on traffic lights and street corners, perhaps a smartphone app for crowdsourced data gathering. To inform policy choices, data on noise limits for vehicles and muffler costs might be added to the street-level noise readings. Then, computer simulations could predict the likely effect of enforcement steps, charges or incentives to buy properly working mufflers for vehicles without them.

The project, Dr. Koonin says, might also pull in data on traffic flows, garbage pickup times and building classifications. For example, he says, a 2 a.m. garbage pickup could be routed to a neighborhood with little residential housing.

Unboxed: Big Data Is Great, but Don’t Forget Intuition

Andrew McAfee, principal research scientist at the M.I.T. Center for Digital Business, led off the conference by saying that Big Data would be “the next big chapter of our business history.” Next on stage was Erik Brynjolfsson, a professor and director of the M.I.T. center and a co-author of the article with Dr. McAfee. Big Data, said Professor Brynjolfsson, will “replace ideas, paradigms, organizations and ways of thinking about the world.”

These drumroll claims rest on the premise that data like Web-browsing trails, sensor signals, GPS tracking, and social network messages will open the door to measuring and monitoring people and machines as never before. And by setting clever computer algorithms loose on the data troves, you can predict behavior of all kinds: shopping, dating and voting, for example.

The results, according to technologists and business executives, will be a smarter world, with more efficient companies, better-served consumers and superior decisions guided by data and analysis.

I’ve written about what is now being called Big Data a fair bit over the years, and I think it’s a powerful tool and an unstoppable trend. But a year-end column, I thought, might be a time for reflection, questions and qualms about this technology.

The quest to draw useful insights from business measurements is nothing new. Big Data is a descendant of Frederick Winslow Taylor’s “scientific management” of more than a century ago. Taylor’s instrument of measurement was the stopwatch, timing and monitoring a worker’s every movement. Taylor and his acolytes used these time-and-motion studies to redesign work for maximum efficiency. The excesses of this approach would become satirical grist for Charlie Chaplin’s “Modern Times.” The enthusiasm for quantitative methods has waxed and waned ever since.

Big Data proponents point to the Internet for examples of triumphant data businesses, notably Google. But many of the Big Data techniques of math modeling, predictive algorithms and artificial intelligence software were first widely applied on Wall Street.

At the M.I.T. conference, a panel was asked to cite examples of big failures in Big Data. No one could really think of any. Soon after, though, Roberto Rigobon could barely contain himself as he took to the stage. Mr. Rigobon, a professor at M.I.T.’s Sloan School of Management, said that the financial crisis certainly humbled the data hounds. “Hedge funds failed all over the world,” he said.

The problem is that a math model, like a metaphor, is a simplification. This type of modeling came out of the sciences, where the behavior of particles in a fluid, for example, is predictable according to the laws of physics.

In so many Big Data applications, a math model attaches a crisp number to human behavior, interests and preferences. The peril of that approach, as in finance, was the subject of a recent book by Emanuel Derman, a former quant at Goldman Sachs and now a professor at Columbia University. Its title is “Models. Behaving. Badly.”

Claudia Perlich, chief scientist at Media6Degrees, an online ad-targeting start-up in New York, puts the problem this way: “You can fool yourself with data like you can’t with anything else. I fear a Big Data bubble.”

The bubble that concerns Ms. Perlich is not so much a surge of investment, with new companies forming and then failing in large numbers. That’s capitalism, she says. She is worried about a rush of people calling themselves “data scientists,” doing poor work and giving the field a bad name.

Indeed, Big Data does seem to be facing a work-force bottleneck.

“We can’t grow the skills fast enough,” says Ms. Perlich, who formerly worked for I.B.M. Watson Labs and is an adjunct professor at the Stern School of Business at New York University.

A report last year by the McKinsey Global Institute, the research arm of the consulting firm, projected that the United States needed 140,000 to 190,000 more workers with “deep analytical” expertise and 1.5 million more data-literate managers, whether retrained or hired.

Thomas H. Davenport, a visiting professor at the Harvard Business School, is writing a book called “Keeping Up With the Quants” to help managers cope with the Big Data challenge. A major part of managing Big Data projects, he says, is asking the right questions: How do you define the problem? What data do you need? Where does it come from? What are the assumptions behind the model that the data is fed into? How is the model different from reality?

Society might be well served if the model makers pondered the ethical dimensions of their work as well as studying the math, according to Rachel Schutt, a senior statistician at Google Research.

“Models do not just predict, but they can make things happen,” says Ms. Schutt, who taught a data science course this year at Columbia. “That’s not discussed generally in our field.”

Models can create what data scientists call a behavioral loop. A person feeds in data, which is collected by an algorithm that then presents the user with choices, thus steering behavior.

Consider Facebook. You put personal data on your Facebook page, and Facebook’s software tracks your clicks and your searches on the site. Then, algorithms sift through that data to present you with “friend” suggestions.

Understandably, the increasing use of software that microscopically tracks and monitors online behavior has raised privacy worries. Will Big Data usher in a digital surveillance state, mainly serving corporate interests?

Personally, my bigger concern is that the algorithms that are shaping my digital world are too simple-minded, rather than too smart. That was a theme of a book by Eli Pariser, titled “The Filter Bubble: What the Internet Is Hiding From You.”

It’s encouraging that thoughtful data scientists like Ms. Perlich and Ms. Schutt recognize the limits and shortcomings of the Big Data technology that they are building. Listening to the data is important, they say, but so is experience and intuition. After all, what is intuition at its best but large amounts of data of all kinds filtered through a human brain rather than a math model?

At the M.I.T. conference, Ms. Schutt was asked what makes a good data scientist. Obviously, she replied, the requirements include computer science and math skills, but you also want someone who has a deep, wide-ranging curiosity, is innovative and is guided by experience as well as data.

“I don’t worship the machine,” she said.

Unboxed: Stand-Up Desks Gaining Favor in the Workplace

THE health studies that conclude that people should sit less, and get up and move around more, have always struck me as fitting into the “well, duh” category.

But a closer look at the accumulating research on sitting reveals something more intriguing, and disturbing: the health hazards of sitting for long stretches are significant even for people who are quite active when they’re not sitting down. That point was reiterated recently in two studies, published in The British Journal of Sports Medicine and in Diabetologia, a journal of the European Association for the Study of Diabetes.

Suppose you stick to a five-times-a-week gym regimen, as I do, and have put in a lifetime of hard cardio exercise, and have a resting heart rate that’s a significant fraction below the norm. That doesn’t inoculate you, apparently, from the perils of sitting.

The research comes more from observing the health results of people’s behavior than from discovering the biological and genetic triggers that may be associated with extended sitting. Still, scientists have determined that after an hour or more of sitting, the production of enzymes that burn fat in the body declines by as much as 90 percent. Extended sitting, they add, slows the body’s metabolism of glucose and lowers the levels of good (HDL) cholesterol in the blood. Those are risk factors toward developing heart disease and Type 2 diabetes.

“The science is still evolving, but we believe that sitting is harmful in itself,” says Dr. Toni Yancey, a professor of health services at the University of California, Los Angeles.

Yet many of us still spend long hours each day sitting in front of a computer.

The good news is that when creative capitalism is working as it should, problems open the door to opportunity. New knowledge spreads, attitudes shift, consumer demand emerges and companies and entrepreneurs develop new products. That process is under way, addressing what might be called the sitting crisis. The results have been workstations that allow modern information workers to stand, even walk, while toiling at a keyboard.

Dr. Yancey goes further. She has a treadmill desk in the office and works on her recumbent bike at home.

If there is a movement toward ergonomic diversity and upright work in the information age, it will also be a return to the past. Today, the diligent worker tends to be defined as a person who puts in long hours crouched in front of a screen. But in the 19th and early 20th centuries, office workers, like clerks, accountants and managers, mostly stood. Sitting was slacking. And if you stand at work today, you join a distinguished lineage — Leonardo da Vinci, Ben Franklin, Winston Churchill, Vladimir Nabokov and, according to a recent profile in The New York Times, Philip Roth.

DR. JAMES A. LEVINE of the Mayo Clinic is a leading researcher in the field of inactivity studies. When he began his research 15 years ago, he says, it was seen as a novelty.

“But it’s totally mainstream now,” he says. “There’s been an explosion of research in this area, because the health care cost implications are so enormous.”

Steelcase, the big maker of office furniture, has seen a similar trend in the emerging marketplace for adjustable workstations, which allow workers to sit or stand during the day, and for workstations with a treadmill underneath for walking. (Its treadmill model was inspired by Dr. Levine, who built his own and shared his research with Steelcase.)

The company offered its first models of height-adjustable desks in 2004. In the last five years, sales of its lines of adjustable desks and the treadmill desk have surged fivefold, to more than $40 million. Its models for stand-up work range from about $1,600 to more than $4,000 for a desk that includes an actual treadmill. Corporate customers include Chevron, Intel, Allstate, Boeing, Apple and Google.

“It started out very small, but it’s not a niche market anymore,” says Allan Smith, vice president for product marketing at Steelcase.

The Steelcase offerings are the Mercedes-Benzes and Cadillacs of upright workstations, but there are plenty of Chevys as well, especially from small, entrepreneurial companies.

In 2009, Daniel Sharkey was laid off as a plant manager of a tool-and-die factory, after nearly 30 years with the company. A garage tinkerer, Mr. Sharkey had designed his own adjustable desk for standing. On a whim, he called it the kangaroo desk, because “it holds things, and goes up and down.” He says that when he lost his job, his wife, Kathy, told him, “People think that kangaroo thing is pretty neat.”

Today, Mr. Sharkey’s company, Ergo Desktop, employs 16 people at its 8,000-square-foot assembly factory in Celina, Ohio. Sales of its several models, priced from $260 to $600, have quadrupled in the last year, and it now ships tens of thousands of workstations a year.

Steve Bordley of Scottsdale, Ariz., also designed a solution for himself that became a full-time business. After a leg injury left him unable to run, he gained weight. So he fixed up a desktop that could be mounted on a treadmill he already owned. He walked slowly on the treadmill while making phone calls and working on a computer. In six weeks, Mr. Bordley says, he lost 25 pounds and his nagging back pain vanished.

He quit the commercial real estate business and founded TrekDesk in 2007. He began shipping his desk the next year. (The treadmill must be supplied by the user.) Sales have grown tenfold from 2008, with several thousand of the desks, priced at $479, now sold annually.

“It’s gone from being treated as a laughingstock to a product that many people find genuinely interesting,” Mr. Bordley says.

There is also a growing collection of do-it-yourself solutions for stand-up work. Many are posted on Web sites like howtogeek.com, and freely shared like recipes. For example, Colin Nederkoorn, chief executive of an e-mail marketing start-up, Customer.io, has posted one such design on his blog. Such setups can cost as little as $30 or even less, if cobbled together with available materials.

UPRIGHT workstations were hailed recently by no less a trend spotter of modern work habits and gadgetry than Wired magazine. In its October issue, it chose “Get a Standing Desk” as one of its “18 Data-Driven Ways to Be Happier, Healthier and Even a Little Smarter.”

The magazine has kept tabs on the evolving standing-desk research and marketplace, and several staff members have become converts themselves in the last few months.

“And we’re all universally happy about it,” Thomas Goetz, Wired’s executive editor, wrote in an e-mail — sent from his new standing desk.

Unboxed: Making the Case for a Government Hand in Research

Ambitious, sure, but Duolingo recently attracted $15 million of venture capital. The investors are betting on Mr. von Ahn, his idea and his growing team of 18 engineers, language experts and Web designers.

Mr. von Ahn, 33, personifies some of the essential ingredients of America’s innovation culture, when it works well. An immigrant from Guatemala, he has intelligence and entrepreneurial energy to spare. And he has received a helping hand from the federal government. Duolingo began as a university research project financed by the National Science Foundation.

That pattern has been repeated countless times over the years. Government support plays a vital role in incubating new ideas that are harvested by the private sector, sometimes many years later, creating companies and jobs. A report published this year by the National Research Council, a government advisory group, looked at eight computing technologies, including digital communications, databases, computer architectures and artificial intelligence, tracing government-financed research to commercialization. It calculated the portion of revenue at 30 well-known corporations that could be traced back to the seed research backed by government agencies. The total was nearly $500 billion a year.

“If you take any major information technology company today, from Google to Intel to Qualcomm to Apple to Microsoft and beyond, you can trace the core technologies to the rich synergy between federally funded universities and industry research and development,” says Peter Lee, a corporate vice president of Microsoft Research. Dr. Lee headed the National Research Council committee that produced the report, titled “Continuing Innovation in Information Technology.”

The long-term importance of government-supported research may loom small in the current debate over how to reduce the federal deficit. But it is an economic issue worth keeping in mind, and one that points to the kinds of tough choices and trade-offs facing policy makers.

The Budget Control Act, which is scheduled to take effect next January unless Congress shifts course, calls for across-the-board cuts in discretionary spending — for programs other than entitlements like Social Security and Medicare. A new study by the American Association for the Advancement of Science estimates that federal spending on research and development would be trimmed by more than $12 billion in 2013. The National Science Foundation, which finances most government-supported computer science research at universities, would have its budget cut by more than $450 million.

Last week, Senate leaders were trying to negotiate a deficit-reduction deal that would avoid the automatic cuts, but programs like those that finance research are likely to receive rigorous scrutiny from budget-cutters for years into the future. Already, the Advancement of Science report says, government spending on research and development has declined by 10 percent since 2010, when adjusted for inflation.

YET why should government support for scientific research and technology development be spared from the belt-tightening? Unless society benefits inordinately from such spending, there is no case for special treatment. In a new book, “Innovation Economics: The Race for Global Advantage” (Yale University Press), Robert D. Atkinson and Stephen J. Ezell forcefully present the argument for the exceptional role that science and technology play in the economy.

Cutting funding for research and development, Mr. Atkinson said in an interview, is “completely shortsighted.” Spending on science and technology, he said, is an investment that produces a larger economy in the future — generating wealth, jobs and tax revenue. Besides, he said, the bill is not very high in the United States. America ranks 22nd among 30 nations in university funding and research and development funding as a share of the economy’s gross domestic product.

In their book, Mr. Atkinson, president of the Information Technology and Innovation Foundation, a nonprofit policy research group, and Mr. Ezell, a senior analyst at the foundation, define innovation as not only the generation of new ideas but also as their adoption in new products, processes, services and organizational models. In their view, the goal of policy should be to invest in and nurture the development of the innovation pipeline, from basic science to commercialization.

That would call for a more hands-on role for government than is embraced by the mainstream of economic thought, certainly in the United States. The consensus of most economists is that basic science is a “public good,” with the benefits widely shared by society, and thus a worthwhile recipient of government financing. But technology — the application of science to real-world problems — is regarded as a “private good,” with its development best left to the marketplace.

Mr. Atkinson, whose Ph.D. is in city and regional planning, says he is presenting the case for loosening the grip of “neoclassical economists” on policy. He has held economic and technology policy jobs in Rhode Island and for Congress, and has served in advisory groups in the administrations of President Obama, Bill Clinton and George W. Bush. Mr. Atkinson’s nonprofit policy research organization receives financial support from groups including the Alfred P. Sloan Foundation and the Ewing Marion Kauffman Foundation, and from corporations including Intel and I.B.M.

A linchpin of innovation policy, according to Mr. Atkinson, is collaboration between government and industry. As a prime example, he points to Germany and its network of 60 Fraunhofer Institutes, financed 70 percent by business and 30 percent by federal and state government. The institutes, he says, perform applied research intended to translate promising technologies, from polymer research to nanotechnology, into products. These tech-transfer clusters, Mr. Atkinson says, are an important reason for the strength of Germany’s manufacturing sector, even though wages for its factory workers are 40 percent higher than those for American workers.

Taking a page from the German model, the Obama administration announced this year plans for up to 15 manufacturing innovation institutes, public-private collaborations called the National Network of Manufacturing Innovation. The first will be in Youngstown, Ohio, specializing in custom manufacturing using 3-D printing technology. Mr. Atkinson says that while this is a good step, what is needed is a long-term commitment. He noted that Germany’s Fraunhofer initiative began nearly four decades ago, and has grown steadily.

Mr. Atkinson’s focus on research investment, and on the pathways that bring ideas into the marketplace, is gaining increasing attention in economics, according to David B. Audretsch, an economist at Indiana University. “Research and development unequivocally pays off economically,” Mr. Audretsch says. “But the biggest payoff is in ecosystems that take the innovative inputs and make them commercial outputs — products.”

MR. VON AHN’S start-up is a new addition to the growing technology cluster around Pittsburgh, with Carnegie Mellon, a major research university, as its hub. Duolingo combines crowdsourcing and computing, language learning and online translation. People learn a language at no cost, while their lessons feed into Duolingo’s fast-expanding translation database, which is sorted and scored for accuracy with smart software.

The crowdsourcing principle is similar to that used by Mr. von Ahn’s previous company, reCaptcha. Its service employs the widely used Web security feature in which users identify and type words, scanned from old books and newpapers, to gain access to a site or service — in the process helping to accurately digitize those old texts.

But Duolingo is far more ambitious and complex. The exploratory research was supported by a five-year grant from the National Science Foundation, about $120,000 a year, used mostly to pay the tuition and living expenses for a graduate student. Two years later, Mr. von Ahn and his small team had made enough progress to have a working prototype, start the company and attract private investors.

“The grants are really helpful when you’re working on something that is scientifically complex,” Mr. von Ahn says, “when you’re in the early year, or two or more, when you have no idea if it will work.”

Mr. von Ahn, joining other scientists, has made a few trips to Washington to speak to members of Congress and their staffs. His message is this: “Don’t eat your seed corn. It may seem like an easy thing to cut now. But years later, you will regret that you did not invest a tiny portion of your federal budget in research, in the future.”

Unboxed: Data-Driven Discovery Is Tech’s New Wave - Unboxed

Computing may be on the cusp of another such wave. This one, many researchers and entrepreneurs say, will be based on smarter machines and software that will automate more tasks and help people make better decisions in business, science and government. And the technological building blocks, both hardware and software, are falling into place, stirring optimism.

Michael R. Stonebraker, a pioneer in database research, is one of the optimists. Software used by companies and government agencies — in products sold by Oracle, I.B.M., Microsoft and others — descends from research done in the 1970s by Mr. Stonebraker and Eugene Wong, a colleague at the University of California, Berkeley, as well as a team of scientists at I.B.M.

Today, Mr. Stonebraker sees an opportunity for new kinds of ultrafast databases. The new software, he explains, takes advantage of rapid advances in computer hardware to help businesses and researchers find insights in the rising flood of data coming from so many sources, including Web-browsing trails, sensor data, genetic testing and stock trading.

So, at 68, Mr. Stonebraker is a co-founder and chief technology officer of two start-ups in the field of data-driven discovery, VoltDB and Paradigm4.

“Now is the time,” says Mr. Stonebraker, who is an adjunct professor at the Massachusetts Institute of Technology’s computer science and artificial intelligence laboratory. “The economics and the technology are ripe.”

The case for optimism is by no means unqualified. The march of these technologies raises social issues, including privacy concerns, and the timing is uncertain. All of the bold predictions in the 1990s that the Internet would disrupt traditional industries like media, advertising and retailing did come true — a decade later.

But a series of related technologies, scientists and entrepreneurs say, has reached a critical mass — come to a digital boiling point, so to speak — so that new products and capabilities become possible. The technical ingredients, they note, include powerful, low-cost computing and storage spread across thousands of computers. The digital engine rooms of Google and Amazon are prime examples.

Another fast-improving technology involves inexpensive and intelligent sensors, which are crucial to a new breed of automated machines like experimental driverless cars and battlefield drones. Clever software — notably machine-learning algorithms — animates much of the current wave of smarter technology. Two well-known examples are found in Watson, the “Jeopardy”-winning computer from I.B.M., and the movie recommendations on Netflix.

ADVANCES in such underlying technologies are fueling the current excitement in fields like artificial intelligence, robotics and data analysis and prediction. “All parts of the technology pipeline are gearing up at the same time, and that’s how you get this explosion of new applications and uses,” says Jon Kleinberg, a computer scientist at Cornell University.

Behind the seeming explosion, experts say, is a process of technology evolution. Paul Saffo, a technology forecaster, compares the process to the evolutionary biology concept known as “punctuated equilibria” formulated by the paleontologists Stephen Jay Gould and Niles Eldredge. The idea is that species often evolve in periodic spurts.

Yet, they say, there are typically years of progress before a commercial breakthrough in the technological realm.

“Even in Silicon Valley, it takes most technologies 20 years to become overnight successes,” says Mr. Saffo, a consulting professor at Stanford’s school of engineering.

The Internet provides a case study of both technology’s evolutionary progress and its exponential growth. In 1969, there were only four computers connected to the nascent Internet, compared with roughly a billion computing devices today, from laptops to cellphones, says Edward Lazowska, a computer scientist at the University of Washington.

The early increases in connected computers drew scant attention. “But at some point in the late 1990s,” Mr. Lazowska says, “you were going from 4 million to 8 million to 16 million to 32 million to 64 million, and people started to notice that something revolutionary was going on.”

Rocket Fuel is a four-year-old Silicon Valley start-up that uses artificial-intelligence software to place display advertisements for marketers on the Web. The company can not only tailor ads by demographic slices of viewers’ ages, gender and interests, but can also use its predictive algorithms to produce campaigns based on results, says George H. John, the company’s chief executive.

For example, a luxury carmaker might tell Rocket Fuel that it wants to place 100 million ads in the next month, and it will pay the company, say, $80 for generating a sales lead, as evidenced by a potential customer downloading a brochure or filling out an online form.

Rocket Fuel is growing fast, having nearly doubled its work force since the start of the year, to 240. So far in 2012, it has handled campaigns for more than 500 advertisers, including BMW, Duncan Hines, Allstate, Pizza Hut and Ace Hardware. It has raised $76 million in venture funding and debt, and its thousands of computers handle 19 billion bid requests a day on ad exchanges. Each online auction for ad space is typically completed in about 100 milliseconds, a tenth of a second.

Rocket Fuel, Mr. John says, is using some of the ideas he worked on in the 1990s as a doctoral student focusing on artificial intelligence at Stanford — research that was supported with government dollars from the National Science Foundation and other agencies, as is so often the case. In the last few years, building a business around those ideas has become achievable and affordable. “And a lot of it has to do with the underlying technology,” Mr. John says.

FOR Mr. Stonebraker, the hardware advance that opens the door to his start-ups is the striking improvement of solid-state memory, as performance climbs and prices plunge. Solid-state, or flash, memory is most widely known as the lightweight storage technology used in consumer devices like small music players and smartphones.

But increasingly, solid-state memory can be used in big computers, holding a hefty database in memory instead of sending data off to be stored on disk drives. According to Mr. Stonebraker, some data-handling tasks can now be completed 50 times faster than with conventional systems.

“Memory is the new disk,” he says. “The obvious thing to do is to exploit that technology.”

In the yin and yang of computing, it is software that exploits hardware, enabling a computer to do useful things. And machine-learning programs and other data-sifting software are advancing swiftly.

“There is no point in collecting and storing all this data if the algorithms are not able to find useful patterns and insights in the data,” says Mr. Kleinberg at Cornell. “But the software is scaling up to the task.”