Vjylku

I'm currently an Economics MA student doing a referee report on a paper that employs a dated empirical method.Specifically I'm looking at a paper on an application of the Almost Ideal Demand System^* for gas prices in 2012.

It should be noted that this method was suggested in 1980 but then improved upon in 1998 by other researchers (called the Quadratic Almost Ideal Demand System) after discovering certain issues with it.

This got me thinking about how exactly those on the cutting edge of research seem to lag behind by over a decade (or more) in method and still manage to get published.

This is concerning because it shows that published researchers make mistakes and dont review all relevant research before publishing. I can imagine that in the hard sciences and medicine this would happen also (which is especially concerning knowing that those on the cutting edge could be decades behind in knowledge which has been around for a while).

Is this acceptable? If so, why?

* _{for those interested i made a video teaching the topic here: https://www.youtube.com/watch?v=zsTknpqUzh4}

What you are describing is not uncommon. In my field people still use methods developed 50 years ago. Some of these methods are still valid and have proven to be robust, some of these are flawed with known improvement, and some of these are down right logically inconsistent but people still use them because of inertia.

Whether using an outdated method is a critical flaw in a paper depends on many factors. But it eventually comes down to whether the flaw in the method invalidates the main conclusion. For example, if the main result is qualitative, and the improvement from the new method is incremental, then it's not a big deal. If the result is supported by multiple lines of evidence, then the fact that one of them is flawed is then less severe of a problem. If the method is known to fail in special cases and it is clear that the data do not fall into such cases, then it is also not a big concern.

Overall, for better or worse, people are going to be more forgiving if the newer method is not well known or the improvement is marginal.

What you call "an outdated method" another may call "the well-understood method".

In neuroscience, this is a very common occurrence. There are new techniques for analyzing different types of neural recordings coming out each month in a number of journals, and each one aims to improve on a specific aspect of a predecessor. Unfortunately, the new techniques are exactly that—new—and therefore untested against lots of data with different initial conditions. There are a good number of researchers who will simply ignore all the new techniques until people have developed them to a place of comfort. Even for those that do gain acceptance, they may not be appropriate for every type of analysis¹.

I'm unfamiliar with your specific case, but I have seen similar concepts elsewhere in Econ, where older published techniques remain highly popular because (1) they're well-understood and (2) the new techniques were created to fix problems that not present in all databases, or not relevant for a given analysis. The old fogies sometimes do have something to offer.

_{¹ In one case, a technique called DCM became widely popular in a very short period of time, and consequently was very quickly becoming widely misused. It got so bad that the authors actually published a paper titled "Ten simple rules for dynamic causal modeling" with the goal of educating researchers how to use the technique. (Biomed researchers in general don't have a great track record of performing world-class data analysis, but thats a separate story...)}

I'm currently ... doing a referee report on a paper... [Author did X] Is this acceptable?

You're the referee, so you tell us!

As a referee you have the authority to use your discretion here and decide what kind of recommendation you want to give to the editor. You have identify that the authors use an outdated method of analysis that has some problems highlighted in later literature. You should point this out in your review, and you will then need to decide how big of an issue this is. Is the old method sufficiently poor that the method should be revised to the improved method from 1998? If so then perhaps a revise and resubmit might be appropriate (assuming other aspects of the paper are okay).