Although I cannot quote chapter and verse, I recall a statement from the late Pontiff about this, which really should have settled the matter. If I remember correctly, it confirmed 4pm.
But even without that, I think the the logic is rather simple: that hour is traditionally considered the earliest hour for Vespers. Sundays and solemn feasts have First Vespers the evening before, which of course means that the commemoration begins then. First Vespers is the start of the liturgical day.
I suppose that the “anticipated Mass” should technically follow First Vespers, but since public Vespers have always been a rarity in parish churches, at least the timing should be respected.