Good point, and it becomes problematic when you try to generalise “pleasure”. For instance, one person might find pleasure solely or most satisfactorily through physical pursuits, another through mental ones. Which pleasure is more valid? Now, I think Mill argued that mental pleasure is higher but isn’t this just the bias of an intellectual?
So, we find a real problem in assessing what would give more pleasure and less plain.
Furthermore, why should the standard of “more pleasure, less pain” be accepted prima facie? Only if there is an underlying assumption that this is what fulfills humanity. Yet, many people are fulfilled by pursuing painful activities, that do not give direct pleasure, but some other benefit, such as peace or harmony.