Every animal is not a critter. My dog, for example, is not a critter. Nor is my chicken. No domesticated animal is rightly a critter. Nor are insects and spiders critters unless they are unusually large. A ladybug is a bug. A praying mantis is sometimes a critter, and certain beetles, especially the ones with antlers.
Deer are not critters, either. Critters are generally, although not necessarily mammalian, bigger than your pinky tip and smaller than a... well, deer.
Foxes, skunks, squirrels, chipmunks, voles, moles, possums, raccoons, armadillos, bats, very large bugs, and certain turtles are critters.
Birds and fish are not critters, although crows and hawks come very close.
The word critter also implies destructive or nuisance properties. But bears, mountain lions, buffalo, and anacondas are not critters. They are too large. Calling a twelve-foot anaconda a critter would be like calling it a pest.
Pests are smaller, more ubiquitous, and harder to get rid of than critters. Ants are pests as are baby brothers.
Critter and pest both imply fungibility among species or indifference of the speaker to the identity of the animal. Someone calling the exterminator is unlikely to specify, "The place is crawling with formicidae and we need you right away!" Instead they'll just say, "Ants!" The truck that pulls up will say "Pest Control" and bear a composite picture of a general arthropod lying on its back, a tongue sticking out of its mandibles. No entomologist will quibble, "There's no species on earth that looks like that!" We all know that pests have between six and a dozen legs, jointed body parts, sometimes wings, and usually stingers. The lolling tongue we accept as artistic license.
When you discover something has gone through your garbage, and you've determined it's not the neighborhood dogs, you ascribe it to critters. You do not know if a raccoon has been there or a possum, neither do you care. If something is digging up your roses, you're less concerned with whether its a vole, mole, or armadillo than how you can stop it.
Likewise, no one who has ever been trampled by a rogue elephant would say they'd run up against a critter. Calling an elephant a critter is flatly contemptuous of elephants. It suggests, "I don't know the identity of that massive, gray, tusked animal with its prehensile proboscis that trampled me, and I don't care." Such a dismissive attitude is likely to pique the sensitivity of an elephant and make him trample you all over again.
To recap. Pests are small, numerous, and hard to identify because you usually only get a good look at them after they're squashed. Critters are larger, usually four-legged, wild, and undesirable. Everything else is either an elephant or a buffalo.