Yes, for in such a case as that there is a high degree of correlation between the skin color and the culture, making skin color a useful description for referring to the culture.
Ultimately, aside from the initial shock that some might have upon first seeing someone that has significantly different physical features, racism has very little to do with skin color. It is about culture.
I saw this in the public highschool I went to where the population was about 50% black. There was a certain amount of segregation which seemed to occur almost naturally, but it was not cleanly divided along the lines of skin color. The highschool was fed by two middle schools, one on the east side of the city and one on the west side. The richer white kids from the west side hung out together along with the few rich black kids from the west side, the poorer white kids from the west side hung out together, the poorer white kids from the east side hung out together, and the same was true for the blacks. In general, nobody was rich coming from the east side.
The rich black kids who hung out with the rich white kids were in some ways seen by the poorer blacks as traitors - traitors to their way of life. They didn’t live the same way the other blacks did. Some poorer white kids hung out with the black kids, but that was only insofar as they at least partially adopted elements of the black culture.
It was easy to see that these different groups had different ways of life. They talked a little different, wore different sorts of clothes, listened to different sorts of music, had different priorities in life, etc. The rich white kids didn’t care about the rich black kids who hung out with them, because they lived in a similar way. And the poor black kids didn’t mind the poor white kids who hung out with them for the same reason. Certainly there was blatant racism purely on the basis of skin color, but that wasn’t the cause of the sorts of divisions seen in the school population.
Skin-color based racism exists because skin color serves as symbol for “Different!”. Racism results from either our fear or inability to relate to a culture different from our own.