When the map has only one layer and that layer is removed/refreshed the current visible extent maintains the ‘picture’ of said layer.
When this layer is Removed and the map Refreshed the layer is actually removed, but the current extent ‘picture’ of the layer is still visible.
If another layer is then added the current extent contains only the new layer. The original layer no longer appears.
This issue appears to have been introduced with V12 as this is not the case in V10.5.