One type of unusual dungeon that actually gets used sometimes is the Hedge Maze. Hedge mazes are familiar to anybody – labyrinths grown from, well, hedges. They evolved from know gardens, a type of garden that features a very strict, symmetric, and usually square layout. A hedge maze could even have grown from such a knot garden, after generous application of black magic by an evil faery queen.
The usability as a dungeon is really well illustrated by the picture on the right, a map of a hedge maze that used to exist in the gardens of Versailles.
Due to the amount of work required, they are almost always part of a palace or so, but in a fantasy setting some madman could just set up a hedge maze for the sole purpose of confusing, capturing, or distracting his enemies. Usually, the game will be less about the maze itself, but more about something to be found at the center, or at the other end: The villain’s castle, shreds of a treasure map, statues that are hints to the location of a cache of art looted in the war.
To make a hedge maze useful, you’d have to provide some sort of mechanism to prevent the characters from simply cutting through the hedges. If the maze is used to toy with the characters, this could be a mere threat (“Don’t even think about cheating, or the Mad Jester will kill the mayor’s daughter!”). The maze itself may be a monster; the hedges have sharp thorns and lash out at any attacker. If the hedge is magical, perhaps cutting a hole into one just leads to the point of origin. In a space opera context, the plants are of a strange, alien variety that draw metals from the ground and cannot be cut by the characters’ knives.
Thematically, the obstacles and opponents found in the maze should be plant-related, or park-related. If the antagonist is a faery or evil jester, add fey and joker-related creatures and traps to the list. Clues should be guarded by puzzles, traps should hinder progress – and these tropes work so much better in a hedge maze because they are set up deliberately as puzzles.