what i figure is...if they wanted to kidnap you, when you stopped by the house, wouldn't they already have tried something?

That's why it's also useful to have stickers of villages. Think someone is following you, go into one of the villages and hope they don't have stickers :D