No, becasue if you work to help the poor, you are doing charitable work. Oftentimes, there is not enough money around and so you have to ask the government to fund your sincere efforts in doing charitalble social welfare work which will end up helping those in dire need.
I don;t see it as a sin at all to help people who are in dire need. From the speech given by Mother Theresa we have learned that: On the last day, Jesus will say to those on His right hand, “Come, enter the Kingdom. For I was hungry and you gave me food, I was thirsty and you gave me drink, I was sick and you visited me.” Then Jesus will turn to those on His left hand and say, “Depart from me because I was hungry and you did not feed me, I was thirsty and you did not give me to drink, I was sick and you did not visit me.” These will ask Him, “When did we see You hungry, or thirsty or sick and did not come to Your help?” And Jesus will answer them, “Whatever you neglected to do unto one of these least of these, you neglected to do unto Me!”
So according to Mother Theresa and according to the Bible it is not wrong to do charitable work and it is not selfish at all and it is not ignoring God to do charity or to work for charity for the poor and disadvantaged.