The New York Times might be a widely respected chronicler of past events, but can we use it to divine the future? Kira Radinsky, a 27-year-old Israeli computer prodigy dubbed the "web prophet" says yes.
Radinsky, who appeared this year on MIT’s prestigious list of top 35 inventors under the age of 35 (previous winners include the likes of Mark Zuckerberg, Larry Page, and Sergey Brin), and who started university at the age of 15 and received her Ph.D. in computer science at 26, has developed a unique system which she claims has already predicted the first cholera epidemic in Cuba in many decades, many of the riots that started the Arab Spring, and other important world events.
The complex computer algorithms she wrote collect immense volumes of electronic data—most notably several decades of New York Times archives but also anything from Twitter feeds to Wikipedia entries—and processes it to extract little-known cause and effect patterns that can be used to predict future events.
For example, she says, "If a storm comes two years after a drought, a few weeks [after the storm] the probability of a cholera outbreak is huge, especially in countries with low GDP and low concentration of clean water."
This may seem fairly intuitive—people have been making similar predictions for thousands of years—but getting a computer to do it, and to analyze accurately the massive amounts of electronic data present on the web, is another matter. Even simple examples present complex challenges.
As an example, she points to a hypothetical headline that says there is an earthquake in Australia. "You want to predict what’s going to happen [next] so you look up your database and see that there was an earthquake in Turkey," she said. "You need [your program] to understand that Turkey and Australia are [different] countries. So after an earthquake in Turkey, the Red Cross sent help to Ankara. But to predict that an earthquake in Australia would result in the Red Cross sending help to Ankara is incorrect … so [you] have to build a different function from the data to fix that."
It all started as a game, Radinsky says, when she was playing with Google Trends in 2007 (Google Trends a public web facility that analyses the volume of searches for a particular query). She quickly found out that she could predict some of what people would search for—such as hurricanes—based on news reports of recent world events. Then she asked herself if she could adapt this mechanism to predict future developments.
Aided by colleagues, Radinsky started scraping her data and meticulously perfecting the algorithms she uses to analyze it. Six years later, this research has grown into a Ph.D. thesis and a startup company she has founded, alongside a long list of prizes and awards.
But there were many hurdles along the way, some of them offering valuable insights—for example, about media culture.
"One of the problems with the first version was that no matter what happened, it predicted that something would cause death," Radinsky says. "Unfortunately, death makes an interesting headline."
To this date, the predictions don’t always work perfectly—a recent paper Radinsky co-authored with Eric Horvitz, the managing co-director of the Microsoft Research Lab, claims the rate of accuracy is between 70% and 90%. For example, she says, the software successfully predicted the recent fuel riots in Sudan but it also predicted that the Sudanese government would fall, which didn’t happen.
"But again," Radinsky says, "it gives you a probability, not a certainty."
Currently, much of her research is focused on helping companies identify their most promising potential clients and improving the way they do business with each other. This is what her startup company, SalesPredict, does.
"For me, it’s amazing because we finally have a chance to think how we can change the economy and to make the economy much more data-driven," Radinsky says. "It’s like a new era for the economy, I think, and I am so excited to be part of it."
But they also cooperate closely with a stealth-mode startup, SparkBeyond, which she says works partly in the area of medical and humanitarian research, pioneering advanced machine learning algorithms. Recently, her research turned up a new alarming prediction, of an impending deadly cholera epidemic in Zimbabwe, which could break out as early as this month.
So policy makers, beware. The rest of us, stay tuned.
[Image: Flickr user Sarah Gilbert]