Critical Methods and Modern Tests

Over the last few days there have been a flurry of posts at Language Log that could be related to Biblical criticism, though that is not the intent of their authors. What they are actually discussing is authorship identification and then spin spotting, with an interesting twist at the end.

Here are some key posts to check:

• Authorship identification in the news, which discusses how an expert spots a fake–or in this case, possibly not so much.
• Dumb mag buys grammar goof spin spot fraud, in which a magazine staff gets caught in some basic errors.
• And finally SpinSpotter unspun, where we learn that the software discussed in the previous post might actually not be doing anything at all, though perhaps it is collecting ideas from users.

Why does this remind me of Biblical criticism? It seems to me that it demonstrates how easily one can be misled on things like this, and how important it is to thoroughly check such claims. It’s quite easy for an expert to say that, based on his careful study of certain criteria, a certain document was written (or not written) by a particular author. But when you go and ask for the basis of that claim, you may find that the expert has very little material with which to work.

Noting that an author uses phrases that he doesn’t use elsewhere is significant if you have a large body of that author’s work. If you have only a small amount, you really have little to work with. Statistics work that way. You need significant samples before the numbers mean anything.

I recall an experiment I did while a college student. I wanted to test my ability to decode something in a simple substitution cypher. I explained the process to my sister, and she created her cypher and then encoded some text using it. It was just a short paragraph, but I after an hour or so I gave up and asked her what it was.

My problem? She had chosen a passage from Encyclopedia Britannica discussing the history of one of the Chinese dynasties. I don’t remember which, but the names of the emperors made all the probabilities off kilter. Even though the paragraph was in English, there were so many transliterated Chinese names that my probability charts were all off. I think I should have figured it out anyhow, but the fact is that I didn’t. A larger sample would have made the work quite trivial.

Similarly the effort to automatically spot “spin” relies on sampling. Not every instance of the passive voice is intended to obscure agency, even if one correctly identifies the passive, as this program did not. One would have to find better criteria than simply passive voice.

The ease with which some people are deceived on this is also very interesting. If someone talks authoritatively enough, provides enough technical sounding detail, people will tend to believe them. I think we build this view of expertise in many mystery shows today. How many people get their idea of what expert examination can accomplish from fiction? There the “expert” confidently points out a few items that make the case air tight. In reality, of course, such testimony will be placed in context, and alternative explanations will be provided by the defense. (See! I have indicated agency quite clearly in a passive sentence.)

I like Biblical criticism, and I like to apply critical methodology to various texts. I’m not arguing against that. What I am suggesting is that in Biblical studies we generally have very small samples, and thus we are often drawing substantial conclusions from insubstantial evidence. You work with what you have. At the same time, we need to be very careful to state our conclusions with the appropriate humility.

There are very few “assured results” in textual studies.