Okay...I'm writing this out to see whether it makes sense. Spoilers for the Sixth Sense and Pan's Labyrinth.
In a mystery, there are two plots. The first plot is the one the reader sees. The detective hunts the murderer among a number of suspects, eventually discovering the criminal and ensuring he or she comes to justice. The second plot is the one the reader doesn't see, but guesses at -- why did the murderer do it? What does the murderer do to cover up the act -- more murders, concealing or destroying clues, etc. The second plot pokes up from time to time in the form of clues, but it isn't fully revealed until the end of the story.
In a story with a plot twist, again, two plots. Think Sixth Sense: a psychologist tries to help a little kid who thinks he's seeing ghosts. In and of itself -- and this is important -- the story sells itself. The underlying plot is that the psychologist himself is a ghost, but it isn't obvious. Again, the second plot pokes up from time to time, but isn't fully revealed until the end of the story.
Now, let's posit a third type of story.* Any fantasy story in which a person goes from one view of reality into a second is this type of story, but The Prestige fits in this category, too. The essence of this story is that you think you're getting story A, following story-A-type rules, but really, you're getting story B. Pan's Labyrinth. If you walked into the theater knowing nothing about the story, you'd think you were going to watch a movie about the Spanish resistance, a girl, and her mother. But you'd have hints...the girl likes fantasy stories. There's a fantasy-type narrator. The bug. The stone statue.
However. Until you get to the transformation of the bug, this is not a fantasy movie (except it is). First plot, resistance; second plot, fantasy. Several clues or hints poke through the first plot, but the first plot, on its own, is interesting, and follows the rules of that world. In fact, there's only one point in the entire movie (I think?) where the rules of the first plot are broken at all, when the girl escapes her room via a chalked-in door. Throughout the movie, both the first plot and the second retain their integrity and interest. Here's another important element: in order to resolve the situation in the first plot, the situation in the second plot must also be resolved -- the stakes for both plots are tied together. And, at the end, the second plot is reconnected to the first plot, almost as a plot twist. I'm thinking of this as a "transformative" plot.
I'm working on a mystery script right now, beating my head on the wall, really. Here's the plot. A guy gets salmonella poisoning, ends up in the hospital somewhat delusional, and thinks he sees a vampire suck his roommate's blood. The roommate is dead, but unmarked, in the morning.
First problem. The first plot, the salmonella poisoning, is missing any kind of hook to it; it doesn't stand on its own as a mystery plot. The second plot, the supposed vampire, isn't necessary to be resolved in order to fix the first plot: there's no stakes** in it for the salmonella victim to solve it.
So. The first plot: A guy goes to a romantic restaurant to propose to his girlfriend, only to discover the wedding ring is missing and he has what seems to be food poisoning. Was it the cook, an ex-boyfriend? Was his girlfriend cheating on him? Or was it the girl's brother, who thinks the only guy good enough for his baby sister is...him?
Not a great plot, but I don't have time to write the Great American Novel.
The second plot is going to stay pretty much the same. But the guy hears the victim say, "I hid it in the butter, babe," or something similar that night during the attack. And in the morning, the guy is going to discover the victim is actually the cook. Or the brother. I haven't decided which. So now the guy has to find out a) whether he was poisoned or really did just get food poisoning, b) whether his girlfriend is mixed up in this, and c) just what the hell is going on, anyway.
I still may not be able to pull this off, but at least I don't feel like I'm wasting time.
*There's a fourth, too, that fits the pattern of Alice in Wonderland, in which the switch between plots happens almost immediately. It seems that the first plot doesn't need to be all that interesting, but then, you only spend a couple of pages with the first plot.
**Sorry.