Adultery is specifically extramarital sex. If you universalized the concept of adultery, and everyone had sex with people regardless of whether they were married or not, being married would not mean anything. Whilst it is possible to imagine a society where people have casual sexual relationships, it is logically self-contradictory to imagine one where people constantly have sex with people who are married to someone else. If that were the case, being married wouldn’t mean anything.