Servile work was work done by serfs. Now it means work that people would avoid doing at all, if they could–hence why the serfs got stuck doing it while some people in the upper classes were excused from it every day. It does not mean physical exertion that people might do because they enjoy it.
For me–that means that servile work includes washing floors and doing bathrooms, totally! I love seeing them clean, but if they would clean themselves, I’d let them do it!
To keep from doing servile work on Sundays and holy days is to exclude the unenjoyable things that can wait for another day from those days in favor of restful, prayerful, and enjoyable pasttimes. That seems to be a blessing of a spiritual practice, not an onerous commandment.