Allotropes are defined as different physical forms of the same element in the same physical state, so the atoms in allotropes are identical, but bonded in different ways, as you will see with the four forms of carbon.
Diamonds are brittle because although they are extremely hard and durable, their atomic structure allows them to break along specific planes when subjected to force in certain directions. (Think of a diamond lattice like a stack of building blocks)