"A joke for three people" refers to Damien's tendency to make obscure references that only a select few people in the listening audience (and often not even James) understand.
A vast majority of the time, they are Mystery Science Theater 3000 or Giant Bomb references.
The clown does what it wants.