If you think the pipe is hot, that's because at a stop, the ambient engine bay temp rises before it begins to cool down. But when you're running, the flow of air over and inside the tube lowers the actual temp...everywhere. Think "windchill factor" a.k.a., which is cooler: a hot day, or a hot day (same temp) but the wind is blowing?
Second, metal is a better heat conductor than rubber, meaning to say that although it heats up quickly, it also removes heat faster, especially if there's airflow "wicking" away that heat.
That the tube heats up is actually a good sign, because theoretically, it's absorbing the heat from the air passing inside it, therefore cooling the air going into the system.