The expression is said to have originated from poker in which a marker/counter was used to indicate the person whose turn it was to deal. If the player did not wish to deal, the responsibility could be passed by the passing of the "buck," as the counter came to be called, to the next player.