Sometimes, passengers are not that acute and experienced with regards to abnormal sounds and noises from our cars the way we are. They go inside, sit, talk and then go out, unlike the way we seems to be constantly probing, straining our ears for that strange little noise back there; that minute ping from the engine etc.
Misaligned rear wheels won't make the car wander or pull to a side.