People who look through D&D monster manuals often say things like 'how would this evolve?' or 'why would something like this even exist?' or 'why are there, like, a hundred slightly different humanoid creatures all competing for the same tiny ecological niche?'
I propose that all of these questions can be answered with one and the same answer, and that answer is a snake-man with an unlimited research budget and a whole lot of time on his hands.
Postulate 1: There was an Ancient Snake-Man Empire. (This is true in most fantasy settings already.)
Postulate 2: The Snake-Men maintained a large population of human slaves. (This is also usually true.)
Postulate 3: The Snake-Men loved freaky bio-magical archanotech alchemy and had very limited ethical oversight when it came to using it on mammals.
Given only these three postulates, I shall now explain D&D!
- Dwarves: Human slaves genetically modified for mining.
- Elves: Human slaves genetically modified for use as research assistants.
- Halflings: Human slaves genetically modified for doing delicate jobs on the inside of giant machines, in much the same way as small children were employed in Victorian factories.
- Gnomes, Pixies, Quicklings, Leprechauns, and all those others small annoying fairies: Like halflings, but modified even further in order to carry out tasks deep within the machinery of immense archanotech engines, which required nimble fingers, instinctive magical awareness, and the ability to fit into very small spaces. Quicklings were emergency-repair units, able to climb through a machine to the heart of a malfunction in seconds. Gremlins are fairies who went rogue deep inside the gear forests and began sabotaging machines instead of repairing them.
- Orcs: Human slaves genetically modified for warfare. (This is why they're perpetual minions; they're always looking for a commanding officer, trying to fill a big snake-man shaped hole in their little orcish hearts. Sometimes they'll go to work for an evil wizard who really loves hissing at people, and they'll feel all happy inside, and they won't even know why.)
- Goblins: Defective castoffs of the orc creation project. The Snake-Men tried to exterminate them, but the damn things escaped into the Underdark and multiplied. Their massive genetic defects, the result of their malfunctioning genetic programming, explains their very high rate of mutation and insanity.
- Hobgoblins: Goblins whose genomes finally stabilised at something approximating the desired target (i.e. an orc) after several dozen generations of chaotic interbreeding.
- Bugbears: Weirdly stable strain of goblin mutants blessed / cursed with hereditary gigantism.
- Gnolls: Uplifted hyenas developed as a response to crippling difficulties in keeping armies supplied in the field; as scavengers, they were able to live on the corpses and carrion generated by their own campaigns. Flind sub-race bred for use as NCO class.
- Humans: Uplifted apes genetically modified for use as a general purpose slave-race. An early success of Snake-Man bio-thaumaturgy: hardy, adaptable, and much less intelligent than they think they are. Formed the template for a number of subsequent slave-races developed for more specialised duties.
- Lizard Men: Uplifted lizards bred as soldiers and bodyguards for more conservative Snake-Men, who never really felt comfortable with having all these mammals running around.
- Kobolds: Dwarf strain of lizard men bred as household servants and pets.
- Giants: Human slaves genetically modified for use as shock troopers. Multiple sub-types were created for different environments (cold climate operations, high-altitude operations, etc), thus leading to a proliferation of sub-races.
- Ogres: Stunted and defective giant sub-breed. Too stupid to be useful as soldiers. Kept around for further experimentation after giving rise to Hags and Ogre Magi.
- Ogre Magi and Hags: Mutant ogres who lucked out and acquired hereditary magical powers. The Snake-Men weren't sure why this happened, and kept them (and the ogres) around for additional study and experimentation.
- Dragons: Gigantic bio-engineered terror weapons. Genetically programmed to assert dominance over a region through displays of terrifying force, gather up all useful local resources, and then hand over the materiel thus accumulated to the Snake Man armies when they finally turn up. Their hoarding behaviour is a result of this latter instinct; and now, as dragons sit on their hoards, they are occasionally troubled by the thought that they know they should be turning all this loot and magic over to someone, they just can't quite remember who...
- Trolls: Experimental creatures designed for minimum-fuss harvesting of tissues and organs. Go ahead, take its kidneys: they'll grow back in a couple of minutes!
- Demons, Devils, etc: Pure-energy beings created by the Snake-Men in order to explore the fantastically hostile environment of the Lower Planes. They were mostly built out of dreamstuff harvested from human slaves, which means that hell is basically an enormous projection of the collective dream-life of the slaves of the Snake-Men. (This is why it's so horrible.) These days they go around calling themselves things like 'Orcus, Arch-Duke of the Abyss', and don't like to talk about the whole 'being made by snakes' business.
- Dryads: Magical tree symbiotes created as a solution to the problem of slaves constantly escaping into the woods in heavily forested areas. The dryad charms any passing slaves and keeps them with her; then the Snake-Man slave recovery squads just have to go from one dryad's tree to the next, gathering them all up for reprocessing. Nixies and lamia served similar functions for wetlands and deserts, respectively.
- Nymphs: Over-successful spin-off of the dryad creation problem, with supernatural beauty so powerful that it blinded and killed people rather than enthralling them. Kept around for their potential military uses.
- Minotaurs: Genespliced creatures created as slave-overseers and prison-wardens for populations of human slaves. Their bad tempers and inclinations towards violence were seen as desirable traits. Their ability to instinctively navigate mazes is an ancestral memory of their origins in the endless slave-warrens of the Snake-Men.
- Mermen and Aquatic Elves: Elf and human slaves genetically modified for service in and around the domed Snake-Man cities on the ocean floor.
- Sahuagin: Genetically engineered bioweapon introduced into the seas to hunt down and exterminate escaped populations of aquatic elves and mermen, after traditional methods of containing and recapturing them proved ineffectual in submarine oceanic environments.
- Chimeras, Manticores, Harpies, and other apparently pointless 'crossbreed' creatures: The Snake-Man equivalents of PhD theses, these creatures were engineered by up-and-coming Snake-Man biomancers as demonstrations of bio-thaumaturgical skill. Like PhD theses, the best ones were preserved for posterity in menageries which functioned as living libraries. After the empire fell they escaped and established breeding populations in the wild.
- Slimes, oozes, gibbering mouthers, etc: Created by alchemical run-off from Snake-Man laboratories.
- Ilithids: Created to act as living components of a biological super-computer: their 'always on' telepathy was supposed to enable their minds to be connected into a kind of psychic wi-fi network, able to answer any question provided it was fed enough human brains. Unfortunately for the Snake-Men, the Ilithids soon deduced what was going on, mind-blasted their handlers, and fled into the depths of the Underdark.
- Beholders: Mobile weapons platforms designed for use against high-threat magic-using targets. Their eye-rays made use of bio-thaumaturgical breakthroughs for which the catoblepas, basilisk, medusa, and cockatrice had served as prototype proofs of concept.
- Ghouls: A bioweapon intended for use against population centres; each ghoul infects its victims, who rise as more ghouls in turn, until a large chunk of the population have been replaced by low-level undead easily controlled by any competent Chaotic cleric.
- Vampires: Recipients of a prototype 'immortality treatment' which, while sufficiently developed to have reached the 'human testing' stage, was still exhibiting an unacceptably high number of negative side-effects (blood dependency, light vulnerability, etc) by the time the Snake-Man empire fell.
- Yuan-Ti: In the chaos which followed the fall of the Snake-Man empire, a bunch of desperate humans tried splicing themselves with Snake-Man genetic material in the hope that the guardian automata protecting the remaining supply depots would mistake them for genuine Snake-Men. It didn't work, but at least they got funky snake powers.
- All those giant animals all over the place: Early Snake-Man experiments in magically-engineered gigantism, of the same kind that would later be employed in the creation of giants, dragons, and ultimately the Flesh Moon Project. The Flesh Moon itself was last seen shortly before the final collapse of the Snake-Man empire, propelling itself off into space on its stubby little wings. If it's still alive then it must be the size of a planet by now.