Why is the work done on a charge calculated from infinity to a point? Why not from one particular point to other?
infinity is the place that is considered to have no charges, and is at 0 potential all the time.
So, potential at a point in an electric field is defined as work done in bringing an unit positive charge from infinite distance to that point. Actually, we are measuring the potential difference between infinity and the required point. But we've named it potential of the point as it is with reference point infinity.
Basically, infinity is considered as a reference place that is fixed. If other points are considered, then one has to define the other point first and the potential at that point to find the potential at a new point.
N.B.: In practical application, we can measure potential difference and not strictly potential at a point.