Moderator: F. Scott Kieff, US International Trade Commission and George Washington University
Panelists: Stephen Haber, Hoover Institution and Stanford University: Good policy making starts with good research. Literature on patenting lacks the serious empirical work that exists in other fields; ratio of theory/broad claims to empirical evidence is unlike other fields, particularly finance. Academics can broaden and deepen the quantity of research and its quality.
Jay Kesan, University of Illinois: releasing the data is important. Nobody else can kick the tires and look under the hood if you don’t. Be more skeptical/keep an open mind about other inferences from the same data.
Alan Marco, USPTO: Help understand which inferences are justified from the data. Policymakers see a lot of evidence; they agree w/evidence based policy making, but they too frequently see policy based evidence making where reports on one side say something completely different from the other. Need to know how to tell the good from the bad. Better communication from academics: in understanding patents through the lens of patent litigation—self-selected cases aren’t representative of the whole, and we want to be careful about letting the tail wag the dog.
Joshua Wright, US Federal Trade Commission and George Mason University: The principal problem is inadequate quality control leading to overbroad claims. Academy should take quality control seriously, and limit claims. If you compare claims allowed to be taken seriously in antitrust, where norms of scholarship are different—they’re more modest and more tethered to actual inferences an economist would say are valid. That tethering is absent in the IP scholarship world.
[Jessica Silbey, for good reason, will hate the implicit definition of “empirical” at work here as “quantitative.”]
Haber: core problem is that there aren’t enough economists around, many $20 bills on the ground for them to pick up. (I thought the true economist wouldn’t do that …) Law faculty haven’t been trained that way. Economists have upped the game in political science.
Kieff: has the IP/antitrust interface benefited from the antitrust literature’s standards or suffered from the deficiencies in IP?
Wright: both are present. The same economists rationalizing antitrust law are around. Started with big questions—it’s hard to measure innovation! But advance in industrial economics has led to do smaller/narrower studies that can allow strong causal inferences, even if on its own it wouldn’t justify legislation. A drip drip drip of these can fill a bucket. You don’t believe papers, you believe literatures. You get literatures by getting smaller projects. Marginal approach was taken in antitrust, but not enough in patent. Sure, an economist is likely to say the answer is more economists, but I believe that it’s true.
Haber: Causation is a big deal. One can look at data and draw many inferences from it, some of them accurate and some spurious. Data analysis w/o theory: perfect example of this is claims about increased number of lawsuits. People like to draw the inference is that contestability of patents has gone up and therefore the patent system is broken. But another inference is that patents are more valuable and more worth litigating. You can’t figure out which is true (or other explanations) just with numbers. Unless you’re working from a well defined body of theory in which case you have less causation to worry about, you need to worry about your inferences.
Marco: one of the problems of many economists coming in is that policymakers have high expectations. Economists tend to be more reserved in inferences, and so that doesn’t get a lot of fanfare. “How many jobs does IP create?”—those are the questions that economists can’t really answer. Gap-filling: policymakers need to be more patient/understanding: you can have good empirical research or you can have fast.
Wright: another slam on law professors, sigh: law professors can get tenure with big ridiculous claims, but economists can get tenure with a good small contribution. [You’re all just jealous of my jetpack.]
Kieff: So what are some of those $20 bills, tenure-worthy, literature-contributing projects?
Haber: Patent trolls: what we know is miniscule compared to the amount of noise about them. Look at the actual business, specifically: is a patent troll a financial intermediary or a predator on the innovation system? Theory of finance would suggest that intermediaries arise when there’s some market asymmetry creating a return. In patent space, the asymmetry is between individual inventors and large firms implementing patents. Is the behavior of patent trolls consistent w/financial intermediaries, or consistent w/ the mafia? (I’m not sure those are in any way exclusive, actually, given accounts of how the mafia works—and how the big banks work. I don’t see why an economist would start out assuming these were distinct models.)
Kesan: that’s a project of mine: are there differences in the way they settle or take cases to trial?
Marco: Identify the market failure we’re trying to solve very clearly. A lot of times we take the system as a whole, but there are a lot of component parts. One area to refine analysis quickly would be to ID component parts: does patent examination/prosecution affect the way patents are used later on?
Wright: the source for demand for such research has to be the legal academy or the government. The gov’t is more likely to be the source of demand than the law schools. The FTC has played a role in some areas. ID research questions and testable hypotheses: a good role. Do something modest and descriptive and contribute knowledge to the world.
Josh Sarnoff: Rule 11 motions—would be important to know, hard to collect data. We know almost nothing about licensing market, b/c of secrecy issues—but there’s not a systematic requirement to collect all the data we want to know; dramatic change at dramatic cost—only gov’t can do that.
Q: funders of research: what should they keep in mind?
Kesan: as quality of scholarship goes down, it’s seen as more of a political football than normal science. Normal science gets more funding/attention. Convince funders that there is normal science to be done.
Haber: innovation policy is vitally important to the US’s future. Funding of studies of innovation policy by the gov’t relative to the importance of the task is not commensurate, and thus either the gov’t will do it or private entities w/vested interests will do it—or no one will do it. Gov’t is better. Federal Reserve funded most of the finance literature.
Q: we don’t know prices or quantities in the private market that is crucial here. The influential work on contracts has mostly been theory, unless there’s a large literature using private data [Wright disagrees]. What is the policymaker’s role in interpreting academic evidence? If we accept the premise that a lot of this research is bad, if the policymaker knows it, what does the policymaker do with that? What’s the standard of disclosure for the policymaker in relying on evidence?
Haber: ask the question—was this piece of evidence published in a peer refereed journal or not? Simple metric, easily applied. Plus it’s never the case that one study is dispositive. The Q for policymakers is whether there’s a literature whose weight points to a conclusion.
Wright: there are now literature reviews, and literature reviews of literature reviews. The difficulty is that most studies observe either Price or Quantity. Most of the time here we’re less interested in P or Q and more interested in rate of innovation or investments made in innovation, and the problem is that we don’t agree on what a measurement is—harder than measuring prices. But there are some papers that do this. The observability of underlying contracts can be difficult but there are many marginal gains to be had. I can count serious peer-reviewed research designs in last 15 years on both hands.
Kieff: what framing context would you give journalists or staffers as background? Sense of diversity, contestation/testing in the literature.
Marco: also read How To Lie With Statistics—statistical inference wouldn’t hurt.
Kesan: caution on both sides. When I talk to staffers, I take extra trouble to disclose the limitations of my studies. I’ve become sensitized to that after seeing the way my work has been consumed. If you are a staffer/policymaker, it’s worth asking academics: what about the other evidence?
Haber: two things that are low-cost: (1) learn that Google Scholar exists. Look for a review essay about a literature that’s been published in a peer reviewed venue. Read the first five pages, before it’s down in the weeds, and get a sense of the state of a particular literature. (2) Academics need to meet staffers and journalists on their turf, not the other way around. We should be better at making our work more accessible and making our work more available.
Kieff: ideas for organizers of academic work to make their work more impactful? Should they have a governance structure mindful of their source of funds, fiduciary or other duties they might owe that might conflict with independence in writing?
Marco: peer review is still there to correct apparent bias. Economists are notoriously terrible at making their results understood outside their narrow field. NBER is getting good at putting out 2-page summaries. Still need to know how policymakers should be using them. Incumbent on academics to improve their communication.
Wright: There are reputational sanctions in economics that don’t exist or exist with less force in law. Everyone high-fives each other and says the papers are brilliant. But less of a culture of saying a paper is bad and shouldn’t influence policy. (He does not go to the conferences I go to.) This culture developed when law professors didn’t do empirical work. Now, any law professor can run a regression without a license, and that doesn’t serve academia well. He is for more shaming of law professors.
Marco: who do bad research.
Wright: at least those. (Ha. Ha. Though he is a very good public speaker—quite charming.) Need research that can be replicated; that reduces fights over industry-funded or gov’t-funded source. That tones down the criticism. There’s a demand for objective interpretation of results—meta-analysis of fields, similar to literature review.
Kieff: do you think that policymakers should keep anything in mind when an academic is writing as an academic but also has clients to serve and owes duties to a client in other contexts? [Somehow I feel that there is someone in particular being targeted.]
Wright: it happens. Some briefs are really good, and others aren’t. If there’s empirical information, the fundamental question is whether the thing is valid. If I can’t see the data, it doesn’t get any weight.
Kieff: should a reader keep in mind that a lawyer currently representing a client, paid or unpaid, who has taken a position on the topic being written about can’t advance to an academic audience a position inconsistent with his or her client’s?
Wright: I don’t know the professional responsibility component. I have a healthy skepticism for paid-for advocacy, but that’s a rebuttable presumption. I’d be willing to concede that the skepticism meter goes up a notch or two.
Haber: from outside the legal academy, the presumption is that the field is about getting to the truth. This requires both careful scholarship and persuasion of others where you divulge your sources of funding, make data available publicly, share requests for data, and practice open science. He is hearing that legal norms are different. [Because he is hearing a weird subset of claims.] Suggests a host of institutional problems in the legal academy for playing an effective role in evidence based policymaking—seems like conflicts of interest should be divulged. Legal academy was founded on advocacy. That’s not about getting to the truth; it’s about serving the client. That puts it in a difficult position for doing social science, about getting to the truth. How to draw some bright lines about which enterprise people are engaged in.
Kieff: ITC docket generates economic studies. If we were asked to study these issues, we’d enjoy it.