Slow down... *slowly*... signal that you're going to turn or something... make him go around you. Or leave the bus lane, if you are on it, and go into the middle lane.
Also, if you are driving a small vehicle make sure you sound your horn generously to let the bus driver know that you are near or going to swerve, etc. always use signals.