Critique of the Rimm study

I have read a preprint of the Rimm study of pornography and I am so distressed by its lack scientific credibility that I don't even know where to begin critiquing it. Normally when I am sent a publication for review, if I find a flaw in it I can identify it and say "here, in this paragraph, you are making some unwarranted assumptions". In this study I have trouble finding measurement techniques that are *not* flawed. The writer appears to me not to have a glimmer of an understanding even of basic statistical measurement technique, let alone of the application of that technique to something as elusive and ill-defined as USENET.

I have been measuring USENET readership and analyzing USENET content, and publishing studies of what I find since April 1986. I have spent years refining the measurement techniques and the data processing algorithms. Despite those 9 years of working on the problem, I still do not believe that it is possible to get measurements whose accuracy is within a factor of 10 of the truth. In other words, if I measure something that seems to be 79, the truth might be 790 or 7.9 or anywhere in between. Despite this inaccuracy, the measurements are interesting, because whatever unknowns it is that they are measuring, these unknowns are similar from one month to the next, so that the study of trends is meaningful. As long as you are aware of what it is that you are taking the ratio of, it is also meaningful to compare USENET measurements, because whatever the errors might be, they are often similar in two numbers from the same measurement set, and they are multiplicative, so they tend to cancel out in quotient.

In other words, in the results that I publish, the two kinds of measurements that are meaningful enough to pay attention to for serious scholarship are the normalized month-to-month trends in the readership percentages of a given newsgroup, and the within-the-same-month ratio of the readership of one newsgroup to the readership of another. The reason that I publish the numbers is primarily to enable trend analysis; it is not reasonable to take a single-point measurement seriously.

No matter what the level of accuracy you are seeking, it is imperative that you understand what it is that you are measuring. Whenever you cannot measure an entire population, you must find and measure a sample, and the error in your measurement will be magnified if your sample is not a representative sample. A small error in understanding the nature of the sample population will lead to an error like the famous "Dewey defeats Truman" headline in the 1948 US Presidential election. A large error in understanding the nature of the sample population can lead to results that are completely meaningless, such as measuring pregnancy rates in a population whose age and sex are unknown.

Rimm has made three "beginner's errors" that, in my opinion, when taken together, render his numbers completely meaningless:

1. He has selected a very homogeneous population to measure. While he has chosen not to identify his population, he has included enough of his sample data to allow me to correlate his numbers with my own numbers for the same measurement period. His data correlate exactly with my numbers for Pittsburgh newsgroups in that measurement period; only his own university (Carnegie-Mellon) has widespread enough campus networking to make it possible for him to sample that large a population. It is therefore almost certain that he has measured his own university. I received my Ph.D. in Computer Science from Carnegie-Mellon University, and I am very aware that it is dominantly male and dominantly a technology school. The behavior of computer-using students at a high-tech urban engineering school might not be very similar to the behavior of other student populations, let alone non-student populations.



2. He has measured only one time period, January 1995. Having lived at Carnegie-Mellon University for a number of years, I know first-hand that student interests in January are extremely different from student interests in September or April. When measuring human behavior about which very little is known, it is important to take numerous measurements over time and to look for time series. Taking the last few years worth of my data and doing a trend analysis in the newsgroups that he has named as pornographic shows an average 3:1 seasonal trend change between low-readership months (November and April) and high-readership months (September and January). But the trends are different in different newsgroups. A single-point measurement is not nearly as meaningful as a series of measurements.



3. He makes the assumption that by seeing a data reference to an image or a file, it is possible to tell what the individual did with the file. We in the network measurement business are very careful to explain what it is that our measurements mean. Here is the standard explanation that I publish with my monthly measurements to talk about the number that Rimm calls "number of downloads".

   To "read" a newsgroup means to have been presented with the opportunity to look at at least one message in it. Going through a newsgroup with the "n" key counts as reading it. For a news site, "user X reads group Y" means that user X's .newsrc file has marked at least one unexpired message in Y.

   Rimm used my network measurement software tools to take his data, and he did not anywhere in his article state that he had made changes to them, so I must conclude that his numbers and my numbers are derived from the same software. But the number that he is using for "number of downloads" is the same number that I call "number of readers" by the above definition. It has nothing to do with the number of downloads. In fact, it is not possible for this measurement system to tell whether or not a file has been downloaded; it can tell whether or not a person has been presented with the opportunity to download a file but it cannot tell whether the user answered "yes" or "no".

Brian Reid, Ph.D. Director, Network Systems Laboratory Digital Equipment Corporation Palo Alto, California reid@pa.dec.com

http://www.research.digital.com/nsl/people/reid/bio.html