One of the reasons why slavery was considered to be bad for the white races was that it was considered detrimental to white interests because it imported a largely alien race into the midst of the white population in substantially large numbers. In a statement given before the Radical Republican Congressional Committee, arguing for freed blacks to be expelled from Virginia, this was largely Robert Lee’s rationale to put forward his case that slavery was bad for the white race (Emory, 1997: 382-3). In a letter that Robert Lee wrote to his cousin in which he advised him not to employ blacks, he stated: “..wherever you find the Negro, everything is going down around him and wherever you find the white man, you see everything around him improving.” (Emory, 1997:371). He believed that the Negros were a race more interested in a social life, comfort, and ease; as a result, they were not as capable as the white man was of absorbing knowledge and improving the condition of their lives. When they were present in large numbers in society, Lee believed that the overall quality of white society deteriorated. Moreover, he believed that slavery, by thrusting large numbers of blacks into the midst of white society, especially when combined with the prolific breeding of the black race, could pose a threat to the survival of the white race and bring about their downfall.
When the abolitionist program commenced, it posed a further threat to the white race, specifically to the poor whites. As compared to the slave-owning white plantation owners, the poor white man had to struggle to earn a living. When the blacks were freed as a result of the abolitionist movement, it only meant that there were more blacks also released into society to seek jobs and a livelihood.