Each territory had a US Marshal who was in charge of law enforcement. He could appoint deputies and order a posse (group of men) to help him hunt down criminals.
Communities could also elect a sheriff to keep the peace in their county.
Law enforcement was hard because:
Territories were huge, which made them very hard to control.
Law enforcers were badly paid, so it was hard to recruit them and many took bribes.
Sheriffs had no legal training so did not always act fairly.
Vigilance committees
Many people took the law into their own hands. Groups of vigilantes would capture, try and punish suspected criminals.
Vigilante committees were effective but often unfair. A suspect’s guilt was usually decided before they were captured. Lynching was common.
Settling claim disputes
Mining communities would agree their own set of rules, and set up their own court and judge. A recorder was chosen to write down all the claims made in the district, and who had made them.