By Anne M. Burton

John Mullahy and Edward Norton are the 2025 recipients of ASHEcon’s Willard G. Manning Memorial Award for the Best Research in Health Econometrics, recognized for their paper “Why Transform Y? The Pitfalls of Transformed Regressions with a Mass at Zero” (Oxford Bulletin of Economics and Statistics 2024). I spoke with them about their award-winning work, collaboration, and lessons for health economists.

The Pitfalls of Transformations

Anne: Congratulations on the award! What is your paper about, and what’s the key message for researchers?

Edward: We looked at what we think is an important set of issues around how best to model outcomes when you have a lot of positives that may be highly skewed and some zeroes, and this is an issue that people have looked at in a variety of ways over many years, including taking the natural log of an outcome (plus a constant) and the inverse hyperbolic sine. In our research we found some very interesting properties of this, that it wasn’t behaving the way people thought and it was causing more problems than people realized. Our main takeaway message is pretty simple: “if your outcome includes zeroes, do NOT take the natural log or the inverse hyperbolic sine or any of these other transformations; do something else instead.”

John: Applied researchers should really think carefully about what is their estimand, or what is the quantity they are trying to understand from a theoretical or conceptual perspective. The answer to that question should dictate the remainder of the analysis, rather than backing out what is the estimand that is implied by the way one might pursue analyses based on transformations. It’s really nothing more than doing good empirical science in the right direction. And ultimately a lot of the work that Will Manning did I think had that flavor to it as well.

Anne: It sounds like the main takeaway is “think before you code”!

John: That would be a great title for the paper! Don’t do things just because a lot of other people have done things.

Risks, Incentives, and Good Science

Anne: Exactly. I think it’s easy for people who especially are on the tenure track these days, who are getting all this advice that’s “you need to publish, you need to publish, you need to publish,” and they’re just trying to get stuff out as quickly as possible without taking the time to slow down and think, “am I putting out my best effort at what the truth is?”.

John: This could be a whole separate conversation but you just hit on an important point so I’ll elaborate on it. It’s pretty easy for Edward and me to say what we’re saying here, because our incentives are very different at our career stage. Good science I think depends on it, but the incentives are very different for those on the tenure track.

A New Collaboration

Anne: You each have a longstanding research agenda in health econometrics, but this is the first paper you two have written together—how did that come about?

Edward: I’ve always been interested in methods and trying to use the most appropriate one for answering the question. We got into this paper because John had already written a paper, and I was about to go on sabbatical and he posted about his paper on Twitter. It was about the inverse hyperbolic sine, and I had started to see some other people using the inverse hyperbolic sine and I didn’t know what it was so I downloaded a copy, made a note to myself to read this paper and learn about it. I read it on sabbatical, downloaded the Stata code, started running things, and thought it dealt with zeroes seemingly very nicely. I was playing around with the code and found something puzzling or odd about John’s example, so at some point I sent him an email. That led to further conversation, we explored this puzzle further and I got curious about this method that other people were using. The way I learn about things is with data, examples, and making graphs. In our conversations back and forth we decided there was enough new material here that we could write a paper explaining that there were real problems with the inverse hyperbolic sine, as well as Box-Cox transformations and log transformations.

John: That’s a good summary, Edward. You also didn’t mention the paper that you wrote, but we can talk about that in a minute. The paper I wrote was close to being accepted in Stata Journal but I pulled it after Edward and I talked about it for awhile because I thought, “well, I’m not sure it’s going to add much”. The paper I wrote was trying to say, “if the [Stata] user community is going to use this transformation method, then what would be the implications for trying to back out something like an estimand that you could possibly care about”. It’s beyond imagination that an estimand of interest in applied health microeconometrics would be this thing that the inverse hyperbolic sine is doing. Normally you would want to try to say something much simpler. The problem is if you’re going to do this, how do you back-transform it to get to where you might want to be, and that turns out to be either impossible or only doable under very stringent assumptions. But then that raises the question, why go through all these complicated gymnastics in the first place if you don’t have to? Give Edward credit for the title of our paper; why transform in the first place if you don’t have to? If you have to fix what you transformed post-hoc to get where you want it to go, why don’t you just go there directly in the first place?

Edward: So we wrote a paper that simply said, “there’s a problem, don’t use these transformations if you have zeroes as part of the distribution”. I shared it with Jon Skinner, who’s at Dartmouth, and he said, “you’ve got to add more to the paper, you have to give people a ‘here’s a good alternative,’ you can’t just stop there and say there’s a problem”.

Anne: Here’s a problem, and here’s a solution, and it’s pretty simple to implement.

John: The last part of that is really important, which is it’s pretty simple. We’re working on another paper right now but it’s top secret so we can’t divulge what it’s about!

On Coauthoring Well

Anne: That’s great! If it’s top secret I won’t ask you what it’s about! That’s a nice segue to my next question: how do you find coauthors for your papers, and what makes a good coauthoring relationship?

John: Shouldn’t we be having this conversation one-on-one rather than together?

Anne: Well if you said you’re writing another paper together then I’m assuming the first paper didn’t go so horribly!

Edward: I’m not sure I have any great insights other than the obvious, which is that everybody on the team should bring something to the table. You want everybody to be invested in the project, willing to work, do their share.

John: I do a little professional development shtick on this topic with our grad students, and I’ll try to share it briefly. Big picture: think like an economist. You have an idea; at the margin, is there value in bringing in another collaborator on this project? There can be, but now you introduce coordination costs, so do a benefit-cost test. If it passes, you have two coauthors; on the margin, do you bring in a third coauthor? If so, you’re going from one dimension of coordination costs to three. Go back to Coase’s (1937) Theory of the Firm, when does it make sense to do things internally vs. externally? It’s not costless to add people to a project, because you potentially have different working styles. Edward and I both try to get early drafts reasonably well written, but other people like to just get it down in some form and worry about tidying it up later. That’s a coordination issue—it can be hard to work with people with different work styles. I think it’s too easy to think about the benefits of coauthors without the recognition that there are costs. Another thing to keep in mind is career stage—working with more senior [in the profession] people can be great but they are on a different timeline, so if you are junior and on a tenure clock, seniors may not always be willing or able to get a paper out the door when you need it. Part of that is because they have more demands on their time (e.g., service responsibilities).

Anne: It sounds like what you’re both saying can be distilled into even simpler economic principles: is a potential coauthor is an economic complement or substitute to you, and what are your respective comparative advantages?

John: Also, once you define your team of collaborators, you should be sure to have the discussion about how you arrange the authors on the paper: alphabetical (econ norm) or first author/last author like in other fields. It’s a much better conversation to have before you write the paper compared to after. It’s also much easier to add authors than to subtract authors: if you bring somebody in but they don’t contribute a lot, that can be hard to deal with down the road.

The Annual Health Econometrics Workshop

Anne: Speaking of working together, this is the first time you’ve written a paper together, but it’s not the first time you two have worked together on something. I’m referring to the Annual Health Econometrics Workshop (AHEW), which you have both served on the Organizing Committee for for many years. Tell me a little bit more about this conference, which seems (to me at least) to fly under the radar a little bit but I really enjoyed attending in 2019 when I was a graduate student.

Edward: The inspiration for AHEW was a conference held in Europe that’s been going on for about 35 years now: the European Workshop on Econometrics and Health Economics, organized by Andrew Jones and Owen O’Donnell. John and I, and Anirban Basu, were all regular attendees of that conference for many years. We were sitting at dinner together during one conference and said, “why aren’t we doing this in North America?”. Anirban took the lead on funding, we held the first one in Chicago, where Anirban was at the time, and we held the second one in Ann Arbor (where I am). It’s a great opportunity to focus on papers that are more methods heavy. We don’t intend it to fly under the radar but…

John: The target audience may be a bit small (people with a substantive interest in health topics and a substantive interest in methods). There are plenty of very good empirical papers in health economics that are not making substantive advancements methodologically. A huge amount of credit goes to Anirban who did a lot of the heavy lifting early on with funding sources. Originally, we were thinking of calling it something Great Lakes related because we were all in Michigan, Wisconsin, Illinois, and a few other places, but the conference got taken to a national level pretty quickly. One unique feature of the conference is the discussant-presentation model, which is challenging because it loads a lot of work on the discussants, but done right it’s really beneficial for authors to see the paper through somebody else’s eyes.

Edward: The authors and the audience both!

What Makes a Good Referee?

Anne: I want to switch gears a little bit now. Edward, you received the American Journal of Health Economics referee of the year award in 2024 (congratulations!). And you both have been involved on the other side of the refereeing process as journal editors for several journals. What makes a referee report good, and what makes a good referee?

Edward: The referee has two different audiences that are important. First of all, the editor in charge of the paper needs to get clear signals about what should happen to the paper. Reject: provide a clear signal about why the paper is not good; for example, the data or methods are not good fit for the journal. Accept: advocate for the paper. In the middle (most papers): give the editor clear directions on what to do, and give the authors clear directions on what to do, e.g., I’m concerned about this particular issue, and here’s a path forward. A canonical [referee] complaint is about instruments, and “I don’t like your instrument, but here’s another one that I think would work” is way more helpful than “I don’t like your instrument”. If you don’t like the theory, suggest specific changes or provide a path forward both for the editor who has to make a decision, and for the authors who are potentially revising their paper for that journal or another journal.

John: That was a great answer. Amplifying a couple points: one is that if a referee report has eight points in it, it can be helpful for an editor to indicate which of those points are the most important ones, rather than assuming that all eight of them bear the same weight. It’s the editor’s job to assess, but editors sometimes rely on referees, and the easier the referee can make it on the editor, the better, and the more the editor can rely on the referee now and in the future. Another one, that I’ve had a lot of discussions about, and one where reasonable people can disagree, is “for whom is the referee working?”. I personally feel the referee is not principally working for the authors. I view the referee as an agent of the readership of the journal, with the editor of the journal as an intermediary. What you’re trying to do is ultimately trying to get your readership best informed. I think that means sometimes not spending a lot of your time helping authors improve papers that won’t end up getting published in the journal. It’s a principal-agent problem times two: 1) editor is the principal, referee is the agent, 2) readers are the principal, referee is the agent.

Edward: The primary goal of the review is to help the editor make a decision. Not just up or down, but if it’s a revise, what is the path toward getting an acceptance? What are the two or three big things that need to get changed, dropped, et cetera?

A Tribute to Will Manning

Anne: That’s really helpful advice. Thank you both for making the time to talk with me today—this conversation has been really great! Is there anything else you would like to add?

Edward: One of the things that made this particular paper special and the award special is that we both knew and worked with Will Manning, and this topic is something that we know he would have been very interested in and would have appreciated. It’s all very special for those related reasons.

John: I endorse that times two. And Anne, let me thank you for doing this great work for ASHEcon that you’re doing with the newsletter.

Anne: Thanks, and you are welcome! I don’t have a way of knowing how many people read the newsletter so it’s nice to hear when people read it and find it useful! I will pass your compliment along to my great team of coeditors.

Citation:

Mullahy, John and Edward C. Norton. 2024. “Why transform Y? The pitfalls of transformed regressions with a mass at zero” Oxford Bulletin of Economics and Statistics 86(2):417-447.