Slaves were used to build the White House and the Capital building. That is history. Washington DC was located in the South and slaves were common there at the time and a lot of unskilled labor was performed by slaves. Keep in mind, this wasn’t unusual for the time and place, and many of the presidents were themselves slave owners.
I don’t know why this is coming as a surprise, and Mr. O’Reilly should apologize for what he said.
Here is some history:
The D.C. commissioners, charged by Congress with building the new city under the direction of the president, initially planned to import workers from Europe to meet their labor needs. However, response to recruitment was dismal and soon they turned to African American—enslaved and free—to provide the bulk of labor that built the White House, the United States Capitol, and other early government buildings.
Stonemason Collen Williamson trained enslaved people on the spot at the government’s quarry at Aquia, Virginia. Enslaved people quarried and cut the rough stone that was later dressed and laid by Scottish masons to erect the walls of the President’s House. The slaves joined a work force that included local white laborers and artisans from Maryland and Virginia, as well as immigrants from Ireland, Scotland, and other European nations.
whitehousehistory.org/questions/did-slaves-build-the-white-house
There is a picture there of payment to slave owners for the use of their slaves.