Gravitational potential energy of a mass at a point can be understood as the work done against the gravity in moving the mass from infinity to that point, where the work is done by an external force acting in the opposite direction to the gravitational attraction.