What happens to potential energy if the 'source' is removed?

When you move the object away from the electromagnet then you obviously do work(in the scientific sense of the word)i.e. converting your body's chemical energy into movement.Magnetic forces also create the illusion of doing work but they actually do not!This can be simply proven from university physics(it really is a simple proof!)Thus,when you move an object in a magnetic field all the work is done by you and so no energy is added to the system by the magnetic field itself.The only conversion of energy then is from chemical to kinetic energy as a direct result of you moving the object.You may argue that the object will still fly towards the magnet when released,which is true.However,all the work done,and thus all the energy conserved or released,in the system will still be as a result of you moving the object and not the magnetic field of the electromagnet.By this argument then,switching off the electromagnet will have no effect on the total work done in the system and thus will have no effect on the energies stored in the system.

Remember that the value of potential energy at a point is not physically meaningful; only the change in potential energy between two points has any physical effect. The object in the electromagnetic field does not possess potential energy itself, rather the potential energy is a function of position that tells us the energy expenditure necessary to get from one point to another in the presence of some force field. It is more properly a property of the field (See the source link) and should not be visualized as being "stored" by the particle in conceptual terms.
When you turn the electromagnet off, all you do is remove the presence of the force field, and thus the particle has only the change in potential energy it has already experienced. By conservation of energy, this will all go to the particle's kinetic energy, since there is no longer a force field.

No potential energy is stored in the object.
This can be proved in a simple way.

Instead of placing the object first near to the magnet and then to pulling it away, let us place the object straight away in the latter distance.

The object will fly towards the magnet. But we have not given any energy to the object. How is it possible?

This shows that the energy is in the place (field) and not in the object.

When we switch off the magnet, the field goes off. There is no energy at the point to pull the object.

As the other responders said, potential energy doesn't necessarily have any locus in an object or in space - but your question is clear:

In situation (1) you turn off the electromagnet. In situation (2) you first let the object move back to the electromagnet and then turn it off. At this point (with no magnetic field) the location of the object is immaterial in calculating the total energy of the system so where is the extra energy which must be present in case (1)?

Of course the object must be at least temporarily magnetized (like a piece of steel would be) while being pulled back to the electromagnet, and since the force is attractive the directions of the two fields are the same. Its (the object's) field penetrates the core of the electromagnet causing the flux density to increase as the object gets closer and closer. That creates a Faraday's law electromotive force in the windings. The EMF is against the direction of the current which forces the battery or generator to do additional work beyond what it would if such an object were not approaching.

By the same token, when the object was originally pulled away from the electromagnet the electromotive force (from the decreasing flux density) was in the same direction as the current, which meant that work was being done on the circuit - that is, the battery or generator expended less energy then it would have if nothing special had been happening near the electromagnet.

So essentially the stored energy in the battery should register a relative surplus/deficit between the two cases. This is in principle only, since the work energy is negligible in comparison with the energy consumed in normal operation, and of course some or most of the energy from the work will be lost in heat or radiation.