When Policy Evaluation, Social Innovation, and Policy Design demand that you Go Look and See
by Martin Krieger
1. What to look for: There is always the idea that if you can’t measure it, you are not doing it right. In much of public policy, you try for various measures, but you know as well that they are proxies, and they may not be reliable or even accurate. So what do you do? What you want to do is to do fieldwork, in effect it is sophisticated theory-informed deep journalism. You interview people, you may use some survey instruments (but often this is only helpful after you have found out what’s up). You find out what is happening. In other words, whether it be in health, education, housing, you have to find out what is happening as a consequence of your policies, and then you might be able to figure out how to measure it. If you don’t know what to look for, you are out of business. Some of the time you are doing exploratory research, and you will see surprises. And it’s not a matter of quantitative stuff, per se. It’s phenomena you had not expected.
Reliability. People worry about objectivity or about bias. But looking at the world is surely more accurate than at some data set whose connection with the world is dubious. The problem is not about measurable, it is to figure out what is going on and then you might want to measure it. Preconceived ideas about what is going on are typically artifacts of fantasies. There may be earlier studies, done by sociologists, anthropologists, economists, historians that will inform what you look for.
A wide range of indicators and methods.
a. Robert Sampson (Chicago, now Harvard) just did a book on neighborhood effects in cities, and surely there were dicey neighborhoods. But they drove up and down the streets and photographed the facades of buildings. Sampson is a sociologist, not a public policy person, but his work is central to public policy.
b. Judith Tendler, an economist at MIT, a former student of Albert Hirschman, thinks like an economist and acts like a social fieldwork researcher. (Michael Piore, of the Economics Dept at MIT, is similar.) What she is looking for are places where things work. The idea is that social scientists are excellent at figuring out what won’t work, they are skeptics at heart. But you can be committed to reliable research, but focus on what works or best practices. Of course, it’s not obvious they can be replicated, but at least you are focusing your efforts. Rather than the so called “gold standard” one hears about in medical research, here you are more like a market researcher trying to find out how to tune the product so that it is more likely you have what people want. One description of some of her recent fieldwork concerning regulation (where it is a commonplace that regulation never works, is inefficient, etc) in Brazil says, These grim expectations in themselves often contribute to regulatory inaction. This research project, therefore, asked how could regulatory actors and others build at the margin on the positive-sum outcomes they were already generating, by observing the patterns running across them.
In sum. You don’t want to act like evaluation researchers, nor like conventional social scientists. You want to act much more inquiringly, trying to see what happens. Once you can tell others what happens then the agency can help you and themselves figure out if this is what they want. Put differently, people are much more inventive than our hypotheses allow. We have to find out what they do with what we put out.
2. Finance theory offers a set of ideas about risk and uncertainty that are also useful in policy studies (design, evaluation): probability, variances, fat tails, value at risk, correlation of actors leading to surprising events. The world is not a random walk, or a geometric random walk, but that might be a good place to begin. In any case, be a Bayesian, do some simulations, and get your decision theory and statistician friends to help you think about potential policies and likely glitches. It’s not about quantitative stuff, it’s about organizing your thinking about a future that is surely uncertain, but you can get a feel for how to bound that uncertainty, and how to insure so that awful things are less likely, consequences are more modest, and losses are compensated. Much of this insurance is self-insurance, about risk management, about not being too stupid. And you can buy flexibility in the future through real options. Still, there will be residual uncertainties and you can be vigilant for surprises and be imaginative about those uncertainties–this is not a solution, but it gives you an active stance. Portfolio notions encourage you to have a range of actions, so that you are less vulnerable to a single line of action not working out.
3. Marketing is having/selling what people want. So you have to do lots of consumer research, test marketing, pilots, so that your new products (those policies) are less likely to flop. You may have limited resources, you may have limited time, but focus groups, competitive analysis (Red Teaming), and backup plans can make it less likely to do not have the capacity to respond to glitches.
4. Creativity You train people, you give them resources, and you hope that some will invent. You can make the resources you give people be like venture capital. I usually run for the fallout shelter when people talk to me about “creativity,” the creative class, and other such. In my experience, maybe 10% (maybe 5%) of faculty or researchers are “creative.” But lots are systematic, interesting, careful, disciplined, and that matters a lot. Maybe they are all “creative” at Google and Goldman Sachs and Pixar and Harvard, and at MIT, Stanford, and …, but I suspect that is nonsense.