When they first arrived in America as slaves in the 1600s, Africans joined a society that was divided between master and white servants brought from Europe. In most parts of the South, some of these first African slaves became free either through escape or through emancipation by their owners. It is therefore a misconception that all African Americans in the pre-Civil War South were slaves. Many researchers have also assumed that these free African Americans were the offspring of white slave owners who took advantage of their female slaves. However, these cases represent only a small minority of free African Americans in the South. Most free African Americans were actually the descendants of African American men and white servant women.

In fact, despite the efforts of the various colonial legislatures, white servant women continued to bear children by African American fathers through the late seventeenth century and well into the eighteenth century. It appears that such births were the primary source of the increase in the free African American population for this period. Over two hundred African American families in Virginia descended from white women. Forty-six families descended from freed slaves, twenty-nine from Indians, and sixteen from white men who married or had children by free African American women. It is likely that the majority of the remaining families descended from white women since they first appear in court records in the mid-eighteenth century, when slaves could not be freed without legislative approval, and there is no record of legislative approval for their emancipations.

The history of free African Americans families in colonial New York and New Jersey, by contrast, is quite different from that of free African Americans in the South. Most were descended from slaves freed by the Dutch West India Company between 1644 and 1664 or by individual owners. Researchers have studied these families, especially a group of fourteen families that scholars have traced through at least three generations. None of the fourteen families appears to be descended from a white servant woman and an African American man. However, Lutheran church records from the eighteenth century show that a few such couples had children baptized.