Wow bro! If I were in your place, I'd be more insulted of the fact that I was mistaken for a driver. Just let it go. He is obviously an idiot and not worth your worries.
C4Y, I think what the the *sshole driver meant is, not literally na driver ba siya but it's like marunong ka bang magmaneho.