Here’s a perspective from someone vaguely acquainted with the relevant way of doing science (surveys, depth interviews) that Hernandez was talking about: it may be a long time before a peer-reviewed academic journal will sign off on the kind of research he is talking about. Either he is a poor spokesperson or the project itself is deeply flawed from a methodological perspective.
In terms of the academic credentials Hernandez cited, there doesn’t seem to be any training in research design for the social sciences, psychology or parapsychology. The FREE board of directors’ background doesn’t seem to make up for that lack of expertise in any substantial way. This left me wondering which, if any, survey or qualitative research consultants FREE drew upon to design their study.
In the spirit of “it’s not bad science because it’s about UFOs or ETs, it’s bad science because it’s bad science” here are some of the red flags I saw:
1. Problems with definitions. On the one hand, the FREE study wants to look at all kinds of different phenomena – ghosts, NDE, OBE, etc. – under the label of ET contact. FREE defines ET as a non-human, intelligent being associated with UFO contact. Danger, danger Will Robinson: risk of circular logic at work. Under pressure from Chris, Hernandez later says the FREE study is not about just any old anomalous contact experience; it’s focused only on UFO-related experiences. (What Would LAM Say?) All of this imposes a fake reality on categories that are themselves fluid and significantly culturally determined. It helps to front load the responses of participants in the research and fools few others.
Further into the interview, Hernandez makes it clear that the FREE team assumes that other forms of anomalous experience will fall into line with the ET experience model quickly enough. They already believe that different types of encounter can be explained by the quantum hologram theory of interdimensional physics. Hernandez himself knows this to be true because he received it in a “download” or telepathic contact message. Telepathic downloads or not, academic journals tend to frown on reasoning backwards from assumed findings.
2) Problems with research design.
A) Survey friendliness. Typically, social science researchers do not design 600+ item surveys that take hours to complete. 600 items suggests the researchers don’t really know what they’re looking for and that they don’t care about the response rate – they don’t really believe their targeted survey respondents have a life, as the saying goes. Hernandez says it takes respondents hours to fill out the survey, which will help guard against hoaxers. He doesn’t comment on how that might bias response rates and the data collected in general – which it definitely will. In real life, survey researchers try to cut down the time it takes to complete a survey in order to maximize response likelihood of responses from the population being sampled.
B) Sampling strategy. Findings that report a percentage of this or a percentage of that are meaningless without knowing the sampling universe, population and response rates. At one point Gene says, “This is a random poll?” Hernandez answers yes, which suggests he is ignorant about a crucial point. A random poll has a very specific meaning in terms of the sampling procedures that have been followed to represent the views of a broader population. The percentages Hernadez reports, e.g. abductions accounting for only 30 percent of UFO/ET contact, are completely meaningless in a statistical sense. Hernandez does not seem to understand that.
C) Ethics. Phase III - a detailed psychological profile aimed at least in part at ruling out psychiatric disorders which Hernandez jocularly comments the board of directors might not like to take themselves, and Phase IV - a days-long, probing, in-depth, formal, one-on-one interview – as described by Hernandez raise a couple of basic red flags. One red flag is that this sounds like the kind of research that has significant potential to harm the subjects involved. Will any conclusions reached by Phase III data analysis be matched against actual medical or treatment records to verify validity? What protocols will be followed for ensuring confidentiality and HIPPA protections? What institutional review board will ensure and be legally liable for ensuring protection of human research subjects?
C - The more fundamental red flag is that the research design is itself backwards. You don’t go from close-ended survey items to open-ended depth interviews. It’s the other way around. If you need your survey respondents to explain in depth why they answered the survey questions the way they did, you just put up a blinking neon sign that tells everyone within a 50 internet mile radius “I have no idea what I’m doing!”
In my experience as a former academic there are two fundamental problems involved in provoking academic or scientific interest UFOs and their cognate phenomena. There’s an unwillingness to accept that there may be a problem that is not solvable per se and there’s an unwillingness to accept that a problem may require an explanation outside of the scientific framework. Those are real obstacles and they should not be minimized. What can be minimized is doing really bad research on the assumption that no one is going to notice or care.
Throw out a subject-specific grant or scholarship to first or second year grad students in sociology, anthropology or psychology with provisos that they have to review certain literature and work with a certain board of advisors to conduct research. Then you’ll get cutting edge research design and methods put to work for a paper that could be published.
