Form follows function. Organisms evolve to adapt to their lifestyles and surroundings. Natural selection will shape an organism to its environment through time. A very good example is the thylacine, also known as the Tasmanian tiger or marsupial wolf. It looks a lot like the wolf, and in fact even university students who take zoology courses often cannot tell the skull of a thylacine from the skull of a placental wolf during an exam, according to one story told by a scientist. The way to tell is to look for a hole in a bone that is unique to marsupials. Placental mammals do not have a hole in that particular bone.
Yet the thylacine is more closely related to other marsupials such as the kangaroo, koala, and opossum than it is to the placental wolf. The placental wolf in turn is more closely related to rats, bats, humans, elephants, whales, cows, rabbits, horses, and lions than it is to the thylacine. Scientists call the resemblance between thylacine and wolf convergent evolution or convergence for short. Other remarkable examples of convergence are known. Whales are mammals but they look like a fish. Ichthyosaurs are reptiles and they look like fish as well. Old World treefrogs also look a lot like New World treefrots, and yet Old World treefrogs are more closely related to bullfrogs and New World treefrogs are more closely related to toads.
Because of convergent evolution, scientists have to look deeper than superficial resemblance to figure out relationships, because features that are less likely to be adaptive are more likely to be the result of inheritance from a common ancestor, and therefore are more reliable evidence of a close relationship. They look for anatomical details and also at DNA. Anatomical details of 2 similar features that evolved independently are unlikely to be identical. For example, bat wings are similar to bird wings but their details are very different so that we can tell they are convergences instead of he same feature.
Hyenas are found to be more closely related to cats because they share with cats a feature found in the skull of cats, hyenas, mongooses, linsangs and other cat-like mammals but not in the skulls of dogs, bears, foxes, weasels, otters, seals, and other dog-like mammals. DNA of course also proves that hyenas are more closely relate to cats than to dogs and wolves.
As the WIkipedia explains, "All extant feliforms [cat-like mammals] share a common attribute: their auditory bullae (bony capsules enclosing the middle and inner ear). This is a key diagnostic in classifying species as feliform versus caniform [dog-like mammal]. In feliforms, the auditory bullae are double-chambered, composed of two bones joined by a septum. Caniforms have single-chambered or partially divided auditory bullae, composed of a single bone."
In fact, even before DNA evidence became widely available, the comparative anatomist had already figured out that living carnivores can be divided into two groups, dog-like and cat-like, based on the auditory bullae. Early on, scientists knew that hyenas are more closely related to cats than to dogs because of their auditory bullae.