The speaker suggests that if buildings were indeed constructed from paper, they would exhibit a sense of impermanence and fragility. The line “I might feel their drift” implies the ephemeral quality of such buildings, as they would be susceptible to the slightest movement or change.