personally i do not cooldown the turbo's after running the engine on long trips or short ones. we have a pajero with 4d56 engine and never gave us problem with regards to turbo. the pajero is now almost 8 years old an still boosting strong. wel, that's for the 4d56 i do not not about the d4d engine.
btw, mine is a 2.7 vvti and it is a pleasant to drive. you'd be surprise how fuel efficient the engine is.