by sylvar

It has been about a month since Science published Electronic Publication and the Narrowing of Science and Scholarship by James Evans. I’ve waited some time to comment because the results were somewhat nonintuitive, leading to some deeper thinking.

The results seem to indicate that greater access to online journals results in fewer citations. The reasons for this are causing some discussion. Part of what I wlll maintain is that papers from 15 years ago were loaded with references for two reasons that are no longer relevant today: to demonstrate how hard the author had worked to find relevant information and to help the reader in their searches for information.

Finding information today is too easy for there to be as great a need to include a multitude of similar references.

Many people feel the opposite, that the ease in finding references, via such sites as PubMed, would result in more papers being cited not less. Bench Marks has this to say:

Evans brings up a few possibilities to explain his data. First, that the better search capabilities online have led to a streamlining of the research process, that authors of papers are better able to eliminate unrelated material, that searching online rather than browsing print “facilitates avoidance of older and less relevant literature.” The online environment better enables consensus, “If online researchers can more easily find prevailing opinion, they are more likely to follow it, leading to more citations referencing fewer articles.” The danger here, as Evans points out, is that if consensus is so easily reached and so heavily reinforced, “Findings and ideas that do not become consensus quickly will be forgotten quickly.” And that’s worrisome–we need the outliers, the iconoclasts, those willing to challenge dogma. There’s also a great wealth in the past literature that may end up being ignored, forcing researchers to repeat experiments already done, to reinvent the wheel out of ignorance of papers more than a few years old. I know from experience on the book publishing side of things that getting people to read the classic literature of a field is difficult at best. The keenest scientific minds that I know are all well-versed in the histories of their fields, going back well into the 19th century in some fields. But for most of us, it’s hard to find the time to dig that deeply, and reading a review of a review of a review is easier and more efficient in the moment. But it’s less efficient in the big picture, as not knowing what’s already been proposed and examined can mean years of redundant work.

But this is true of journals stored in library stacks, before online editions. It was such a pain to use Index Medicus or a review article (reading a review article has always been the fastest way to get up to speed. It has nothing to do with being online or not) and find the articles that were really needed. So people would include every damn one they found that was relevant. The time spent finding the reference had to have some payoff.

Also, one would just reuse citations for procedures, adding on to those already used in previous papers. The time spent tracking down those references would be paid out by continuing usage, particularly in the Introduction and Materials & Methods sections. Many times, researchers would have 4 or 5 different articles all saying the similar things or using the same technique just to provide evidence of how hard they had worked to find them (“I had to find these damned articles on PCR generated mutagenesis and I am going to make sure I get maximum usage out of them.”)

There are other possible answers for the data that do not mean that Science and Scholarship are narrowing, at least not in a negative sense. A comment at LISNews leads to one possible reason – an artifact of how the publishing world has changed.
The comment takes us to a commentary of the Evans’ article.While this is behind the subscription wall, there is this relevant paragraph:

One possible explanation for the disparate results in older citations is that Evans’s findings reflect shorter publishing times. “Say I wrote a paper in 2007” that didn’t come out for a year, says Luis Amaral, a physicist working on complex systems at Northwestern University in Evanston, Illinois, whose findings clash with Evans’s. “This paper with a date of 2008 is citing papers from 2005, 2006.” But if the journal publishes the paper the same year it was submitted, 2007, its citations will appear more recent.

[As an aside, when did it become Evans’s rather than Evans’? I’d have gotten points of from my English teacher for that. Yet a premier journal like Science now shows that I can use it that way.]

The commentary also mentions work that appears to lead to different conclusions:

Oddly, “our studies show the opposite,” says Carol Tenopir, an information scientist at the University of Tennessee, Knoxville. She and her statistician colleague Donald King of the University of North Carolina, Chapel Hill, have surveyed thousands of scientists over the years for their scholarly reading habits. They found that scientists are reading older articles and reading more broadly–at least one article a year from 23 different journals, compared with 13 journals in the late 1970s. In legal research, too, “people are going further back,” says Dana Neac u, head of public services at Columbia University’s Law School Library in New York City, who has studied the question.

So scientists are reading more widely and more deeply. They just do not add that reading to their reference lists. Why? Part of it might be human nature. Since it is so much easier to find relevant papers, having a long list no longer demonstrates how hard one worked to find them. Citing 8 articles at a time no longer means much at all.

That is, stating “PCR has been used to create mutations in a gene sequence ^23-32” no longer demonstrates the hard work put into gathering those references. It is so easy to find a reference that adding more than a few looks like overkill. That does not mean that the scientists are not reading all those other ones. They still appear to be, and are even reading more, they just may be including only the relevant ones in their citations.

Two others put the data into a different perspective. Bill Hooker at Open Reading Frame did more than most of us. He actually went exploring in the paper itself and added his own commentary. Let’s look at his response to examining older articles:

The first is that citing more and older references is somehow better — that bit about “anchor[ing] findings deeply intro past and present scholarship”. I don’t buy it. Anyone who wants to read deeply into the past of a field can follow the citation trail back from more recent references, and there’s no point cluttering up every paper with every single reference back to Aristotle. As you go further back there are more errors, mistaken models, lack of information, technical difficulties overcome in later work, and so on — and that’s how it’s supposed to work. I’m not saying that it’s not worth reading way back in the archives, or that you don’t sometimes find overlooked ideas or observations there, but I am saying that it’s not something you want to spend most of your time doing.

It is much harder work to determine how relevant a random 10 year old paper is than one published last month. In the vast majority of cases, particularly in a rapidly advancing field (say neuroscience) papers that old will be chock full of errors based on inadequate knowledge. This would diminish their usefulness as a reference. In general, new papers will be better to use. I would be curious for someone to examine reference patterns in papers published 15 years ago to see how many of the multitude of citations are actually relevant or even correct?

Finally, one reason to include a lot of references is to help your readers find the needed information without having to do the painful work of digging it out themselves. This is the main reason to include lots of citations.

When I started in research, a good review article was extremely valuable. I could use it to dig out the articles I needed. I loved papers with lots of references, since it made my life easier. This benefit is no longer quite as needed because other approaches are now available to find relevant papers in a much more rapid fashion than just a few years ago.

Bill discusses this, demonstrating that since it is so much easier to find relevant article today, this need to help the reader in THEIR searches is greatly diminshed.

OK, suppose you do show that — it’s only a bad thing if you assume that the authors who are citing fewer and more recent articles are somehow ignorant of the earlier work. They’re not: as I said, later work builds on earlier. Evans makes no attempt to demonstrate that there is a break in the citation trail — that these authors who are citing fewer and more recent articles are in any way missing something relevant. Rather, I’d say they’re simply citing what they need to get their point across, and leaving readers who want to cast a wider net to do that for themselves (which, of course, they can do much more rapidly and thoroughly now that they can do it online).

Finally, he really examines the data to see if they actually show what many other reports have encapsulated. What he finds is that the online access is not really equal. Much of it is still commercial and requires payment. He has this to say when examining the difference between commercial online content and Open Access (my emphasis):

What this suggests to me is that the driving force in Evans’ suggested “narrow[ing of] the range of findings and ideas built upon” is not online access per se but in fact commercial access, with its attendant question of who can afford to read what. Evans’ own data indicate that if the online access in question is free of charge, the apparent narrowing effect is significantly reduced or even reversed. Moreover, the commercially available corpus is and has always been much larger than the freely available body of knowledge (for instance, DOAJ currently lists around 3500 journals, approximately 10-15% of the total number of scholarly journals). This indicates that if all of the online access that went into Evans’ model had been free all along, the anti-narrowing effect of Open Access would be considerably amplified.

[See he uses the possessive of Evans the way I was taught. I wish that they would tell me when grammar rules change so I could keep up.]

It will take a lot more work to see if there really is a significant difference in the patterns between Open Access publications and commercial ones. But this give and take that Bill utilizes is exactly how Science progresses. Some data is presented, with a hypothesis. Others critique the hypothesis and do further experiments to determine which is correct. The conclusions from Evans’ paper are still too tentative, in my opinion, and Bill’s criticisms provide ample fodder for further examinations.

Finally, Deepak Singh at BBGM provides an interesting perspective. He gets into one of the main points that I think is rapidly changing much of how we do research. Finding information is so easy today that one can rapidly gather links. This means that even interested amateurs can find information they need, something that was almost impossible before the Web.

The authors fail to realize that for the majority of us, the non-specialists, the web is a treasure trove of knowledge that most either did not have access to before, or had to do too much work to get. Any knowledge that they have is better than what they would have had in the absence of all this information at our fingertips. Could the tools they have to become more efficient and deal with this information glut be improved? Of course, and so will our habits evolve as we learn to deal with information overload.

He further discusses the effects on himself and other researchers:

So what about those who make information their life. Creating it, parsing it, trying to glean additional information to it. As one of those, and having met and known many others, all I can say is that to say that the internet and all this information has made us shallower in our searching is completely off the mark. It’s easy enough to go from A –> B, but the fun part is going from A –> B –> C –> D or even A –> B –> C –> H, which is the fun part of online discovery. I would argue that in looking for citations we can now find citations of increased relevance, rather than rehashing ones that others do, and that’s only part of the story. We have the ability to discovery links through our online networks. It’s up to the user tho bring some diversity into those networks, and I would wager most of us do that.

So, even if there is something ‘bad’ about scientists having a more shallow set of citations in their publications, this is outweighed by the huge positive seen in easy access for non-scientists. They can now find information that used to be so hard to find that only experts ever read them. The citation list may be shorter but the diversity of the readers could be substantially enlarged.

Finally, Philip Davis at The Scholarly Kitchen may provide the best perspective. He also demonstrates how the Web can obliterate previous routes to disseminate information. After all the to-do about not going far enough back into the past for references, Philip provides not only a link (lets call it a citation) from a 1965 paper by Derek Price but also provides a quote:

I am tempted to conclude that a very large fraction of the alleged 35,000 journals now current must be reckoned as merely a distant background noise, and as far from central or strategic in any of the knitted strips from which the cloth of science is woven.

So even forty years ago it was recognized that most publications were just background noise. But, what Philip does next is very subtle, since he does not mention it. Follow his link to Price’s paper (which is available on the Web, entitled Networks of Scientific Papers). You can see the references Price had in his paper. a total of 11. But you can also see what papers have used Price’s paper as a reference. You can see that quite a few recent papers have used this forty year old paper as a reference. Seems like some people maintain quite a bit of depth in their citations!

And now, thanks to Philip, I will read an interesting paper I would never have read before. So perhaps there will be new avenues to find relevant papers that does not rely on following a reference list back in time. The Web provides new routes that short circuits this but are not seen if people only follow databases of article references.

In conclusion, the apparent shallownesss may only be an artifact of publishing changes, it may reflect a change in the needs of the authors and their readers, it may not correctly factor in differences in online publishing methods, it could be irrelevant and/or it could be flat out wrong. But it is certainly an important work because it will drive further investigations to tease out just what is going on.

It already has, just by following online conversations about it. And to think that these conversations would not have been accessible to many just 5 years ago. The openness displayed here is another of the tremendous advances of online publication.

Technorati Tags: Open Access, Science, Social media, Web 2.0