I recently asked a friend of mine at another studio how he liked one of its films. Innocent enough question (note the use “liked”). It had just gone through its first test screening, and I was curious—did it turn out “well,” or “not so well.” What I got back was more like a response about someone’s high school equivalency graduation exam. “It did great in the top two boxes!” he exuberantly exclaimed. “Our total was 77, but we were 88 among males under 21!”

“Cool,” I said, as I tried in vain to remember where 77 in the top two boxes fit into the studio spin line. I pressed on: “Did you like it?” After a few seconds, my friend admitted that he thought it had turned out “okay.” They’d have to find the right release date. For him, though, the good news was that it had scored high enough with test audiences that if the picture didn’t work, he could blame marketing.

Now, I’ve never worked at another studio, but I have friends at almost all of them. I can guarantee you that this completely stupid conversation happens over and over again, everywhere. I’ve been a part of them, and I’ll say it for the record: The test screening process is being abused by people who have no faith in their basic human taste and instinct. Forget about simply saying you “liked” or “didn’t like” something; now it’s, “How did it score?” All anyone wants to know is “How did it score?” This entire industry is research obsessed. What happened to personal opinion?

The moguls of yesteryear didn’t test screen a film for a recruited audience until it was almost complete. Major decisions about structure, story, performance, and music had already been implemented, and the most that might happen to the film would be a few lifts here and there. Over the decades, this process has devolved to the point where a very unfinished product is taken into a consumer environment, questionnaires are immediately handed out after the lights come up, and the audience is invited to recut and sometimes reshoot the film. For too many in Hollywood, the last thing they hear about a film has become the first thing they say about it, and the last thing they heard came from the test audience.

Why? How did this happen? Research can be a valuable tool, but used in the wrong way it can lead to filmmaking by committee of the worst kind. It’s not the fault of the research people or the test audience. The studios invite them to participate. It’s how the studios react to the information that sometimes creates the problem: “Oh no! We’re good with young males, but older females are down. Get those numbers up!” So a male-oriented scene is cut, and maybe the length of the film is shortened, and then: “Hey, older females are up but young males are down. Maybe change the music? Can we get an NSYNC title song?” Instead of finishing the film you read and greenlit, you start a game of “chase the numbers”! Some films cannot be all things to all people, yet the testing process can become an attempt to change a seafood salad into an all-you-can-eat buffet.

We don’t ask the audience to get involved when we decide to spend the tens of millions of dollars in the first place, so why do we empower them after the fact? Fear and anxiety, that’s why—they’ve replaced taste and instinct. When studios spend what they spend to make a commercial film, they want to know that they’re going to be okay as soon as possible. Fine. However, there are two basic problems with the way studios are clinging to this kind of research.

1. Test screenings work for straight down the middle, broad-based, crowd-pleasing films. Period. That’s it. Those are films that can actually be graded by numbers. In our case, films such as The Wedding Singer, Rush Hour, Austin Powers: The Spy Who Shagged Me, The Mask, Dumb and Dumber...no problem. They’re supposed to please everyone. So the top two boxes (by the way, that means the boxes on the test questionnaires marked “excellent” and “very good”) need to add up to a high number (you want mid-80s to high 90s for a CEO to smile). The “definite recommend” box on the questionnaire, which points to potential word of mouth, is the other score you want in the healthy numbers. You want to be north of 65 to celebrate that one.

Try to apply this system to any film that’s dark or controversial or kind of a curve ball, and that’s it. Forget it. You’re scoring 50s or less in the top two boxes, and for the rest of the film’s life you’re trying to convince the world that all is not lost. It’s our job as professionals to distinguish which films are suited to test screenings and which are not. Too often, the testing process is relied upon as a crutch and a substitute for an honest, human opinion. The ugly truth is that some executives find it safer to point to a survey than to put themselves personally on the line. This process must change, because it really harms films that run against traditional form and content. And let’s face it—those can be the most interesting and daring of the lot.

2. People are recruited in malls and the like to attend test screenings. I’m sure some of you have been approached (“Hey! Wanna see a free movie?”) by a nice-enough-looking man or woman holding a clipboard and sporting a smile, as if they’re offering you a handshake and a campaign promise. They show you a paragraph describing the plot of the film and who’s in it. Maybe you go, maybe you don’t. If you go, you could find yourself watching a film that you had no idea you’d be watching. That’s because it’s impossible to capture the tone and integrity of an entire film in a paragraph. You can only supply premise.

We all know that we decide to go to a movie based on TV ads and the trailer. They give you a flavor of what the film is and, in effect, act as a natural filter. They tell you, “Hey, if you wanna see this kind of movie, come on down!” Those test-screening recruiting paragraphs cannot do that, so a studio might find itself with a theater full of people who would never have chosen to see the kind of film they’re seeing—and their questionnaire scores will reflect that. You’ve lost what you had gained with the media campaign. No audience is ever really prepared for what they’re seeing, yet these are the first people who affect the film’s completion. This must be taken into account. I had to stress this point with films such as Seven, Boogie Nights, and even the first Austin Powers (we managed to find the only three test audiences that didn’t get the joke). How people are recruited matters, yet sometimes is all too easily forgotten.

What I find the most valuable about the testing process, and all the filmmakers I’ve worked with agree, is the ability to watch the film with an audience. Feel their vibe. Feel the rhythm. When do they shuffle their feet? When are they hooked? Are people looking at their watches? You can tell simply from audience reaction what might need to be done to the film. If you have to look at numbers, it’s more appropriate to check them on the mainstream, obvious crowd pleasers. You’ve probably heard the famous story of the Fatal Attraction test screening that demanded a new ending. Sometimes the answer is obvious, and the audience can supply it with numerical validation. I argue that, more often than not, it cannot, and shouldn’t be looked to for that purpose. With any adult, serious, or daringly original piece of work, research needs to be tailored, and taken with a grain of salt. There is no substitute for a gut reaction.

I’ve heard about and seen numbers being used to impede and confuse a process for films that require (gasp!) taste and instinct. Hollywood could use more of that. The industry answer for “How did you like the movie?” should not be how it scored in the top two boxes. It gets us further away from what moved us to make the films in the first place, right or wrong, good or bad. There’s an emotional reality involved that the filmmakers and the studio should not forget or abandon when faced with a flood of research information.

The harsh reality is that the real score in the top two boxes is the vote at the box office, and that is almost determined, conceptually, by the time you push the little green button.