It is the “no one else matters” aspect. It is possible for two people to be self-centered together. The Church is saying that God made no thing and no human relationship to be self-centered in its essence, even temporarily.
Every act, every intention, everything that a human or a couple can do with their one shared life falls short, if not done in reference to the love of God and neighbor:
One of the scribes, when he came forward and heard them disputing and saw how well he had answered them, asked him, “Which is the first of all the commandments?” Jesus replied, “The first is this: ‘Hear, O Israel! The Lord our God is Lord alone! You shall love the Lord your God with all your heart, with all your soul, with all your mind, and with all your strength.’ The second is this: ‘You shall love your neighbor as yourself.’ There is no other commandment greater than these.” The scribe said to him, “Well said, teacher. You are right in saying, ‘He is One and there is no other than he.’ And ‘to love him with all your heart, with all your understanding, with all your strength, and to love your neighbor as yourself’ is worth more than all burnt offerings and sacrifices.” And when Jesus saw that (he) answered with understanding, he said to him, “You are not far from the kingdom of God.” And no one dared to ask him any more questions. Mark 12:28-34
This is why the Church teaches that NFP may legitimately be used to space pregnancies. Sometimes, denying oneself in order to perhaps delay having children that one wants very much is the least selfish thing. It is not the “have children” or “don’t have children” that is the goal. It is having every act of a marriage, including the acts of sexual intimacy, reflect a united love of and service to God and neighbor.
Neither is it enough for Christian parents to merely be open to giving birth to children. Humans are charged by the nature of being human to be saints. Likewise, by the nature of marriage, married people are charged to raise saints, or else to use the strength not devoted to that work to do the other work of saints that mere humanity calls them to.
It doesn’t mean life isn’t supposed to be pleasurable. God made food pleasurable. There are few virtues that aren’t much more pleasurable, in the long run, than are the vices. It means not allowing one’s own pleasure (or in the case of a couple, the couple’s mutual pleasure) to be the single goal of any act. That is a road away from sanctity, and so is a forbidden road.