Understanding joke structure

I’ve been thinking rather hard about why some formulations of the same joke are funnier than others.  Then having worked through the steps myself I then realised, “oh yes, that’s what so and so was on about”. (For example Jimmy Carr and Lucy Greeves’ The Naked Jape, and The Serious Guide to Joke Writing by Sally Holloway) I have obviously absorbed these ideas subconsciouly, which I am claiming is a good thing. What I don’t claim is that any of this is original. But some of it is useful. At least to me. (And if it’s not useful to you, um, sorry.)

I don’t want to get too theoretical. Funny is one of those things we know when we see it. A theory is only helpful (to a working writer) to the extent that it helps us fix things when they go wrong or improve things to make them better.

So this is just one theory, it’s not the theory of joke structure. But I think it’s a useful one. And it’s this: Jokes confound expectations in a way that makes sense. (Again not an original claim by me, you should be worried if it was.)
To put it another way, the joke is in the gap between what we expect and what we get. But what we get has to “fit”, otherwise it’s just confusing/disturbing and not funny.
So therefore the set up line create an expectation. And the punchline subverts, twists or otherwise plays with it.
So far, so good. But how is this useful in practice when trying to be funny. Well, say you have an idea about a joke but you’re not sure how to structure it. Then the set up line should be the “thing we expect” (in this context, which may be weird or surreal, so long as it follows directly from the current set up). And the punchline should do two things. First provide the twist or logical leap that gets you the funny. Second, do it in a way that makes sense given the set up.
For example, this structure for a joke doesn’t really work:
How did the chicken achieve his objective of getting to the other side of the road?
By crossing it!
It makes sense. But there is no surprise. You have already said too much in the setup and the punchline just follows directly from the set up with no surprise.
(OK some of you might actually find this funny, but only because it is subverting the nature of a well known joke in the form of a meta-joke. It wouldn’t work if you weren’t familiar with the original. And this is itself an example of surprise. The anti-joke works because we have expectations about jokes.)
Whereas in the original joke the punchline works because it is a surprise (usually when we ask a question like that about motive we expect a different sort of answer, e.g. he wanted to buy some birdseed from Tescos). And it makes sense. Yes, the chicken did cross the road to get to the other side. Not what we meant when we asked, but there you go.
Equally this (sort of)  joke doesn’t work either.
What does David Cameron have on his cornflakes?
The Blue Man Group.
Technically the punchline is a surprise. But it doesn’t make sense. It’s not funny. It’s just confusing.
Of course there are exceptions to this (please point them out, that is always helpful) but I think that thinking hard about:
1. What your joke’s leap should be
2. What information you need in the setup to create the false expectation and
3. Whether it still make sense once you see the punchline
can really help when trying to make stuff funny.
There is still plenty of room to disagree about what leap is the best surprise, whether it is a surprise and whether it makes sense, but it does at least give us more language to talk about how jokes work rather than just funny/not funny.

(And for an interesting read on how a comedy critic analyses jokes see this Guardian article.)