One of the enslaved people Dolley Madison brought with her to this house was a young woman named Mary Ellen Stewart. Stewart was among the African Americans, free and enslaved, who attempted to escape from Washington in April 1848 aboard the schooner Pearl. The escape attempt was not successful and as punishment for her participation in it, Ellen Stewart was confined by a slave trader in Baltimore. Ellen's fate was happier than most of the Pearl's captured slaves however, as her freedom was later purchased with funds raised by contributors in New York and Massachusetts.

Another enslaved person who accompanied Dolley Madison on her return to Washington was Paul Jennings. Like Dolley Madison, Paul Jennings had previously lived on Lafayette Square—in the White House—while James Madison served as President. Only 10 years old when Madison took office, Jennings began his tenure at the White House as a footman whose duties included serving, carrying messages, and assistant to the coachman. He later became the personal valet to James Madison, a role he filled for over 16 years until Madison's death. After her return to Washington, Dolley Madison hired Jennings out to work for friends and neighbors, including to President James Polk. In 1847, he was sold to her Lafayette Square neighbor Daniel Webster through an intermediary, Pollard Webb. Webster gave Jennings his freedom in exchange for working for $8 a month until he repaid the debt.

In 1865, now a free man and working in the Department of the Interior, Paul Jennings wrote the first memoir ever written about life in the White House, entitled A Colored Man's Reminiscences of James Madison. In this work he also described what it was like to see the former First Lady living in poverty on Lafayette Square after her husband's death. He wrote: "Mrs. Madison was a remarkably fine woman. She was beloved by every body in Washington, white and colored. … In the last days of her life . . . she was in a state of absolute poverty, and I think sometimes suffered for the necessaries of life. While I was a servant for Mr. Webster, he often sent me to her with a market-basket full of provisions, and told me whenever I saw anything in the house I thought she was in need of, to take it to her. I often did this, and occasionally gave her small sums from my own pocket. . ."