Many Washington residents, fearing the rumored British attack, had packed what they could on wagons or set out on foot into the countryside on August 22, 1814. Although the town was in a state of pandemonium, the first lady remained calm and directed the rescue of valuable documents, silver, and the full-length 1797 "Lansdowne portrait" of George Washington by artist Gilbert Stuart.

The Madisons' enslaved valet Paul Jennings, the steward Jean Pierre (or French John) Sioussat, and the gardener Thomas McGrath had a hand in taking down the portrait by taking the painting from the wall and breaking the frame. When two New Yorkers—Jacob Barker, a financier, and Robert G. L. De Peyster, a merchant—arrived at the White House to see if they might assist the first lady, Mrs. Madison entrusted the portrait, silver, and state papers to them. On their escape route through Montgomery County, Maryland, Barker and De Peyster placed the Washington portrait in a farmer's barn for safekeeping; then, as Mrs. Madison had instructed, they deposited the cabinet documents and silver at the Bank of Maryland in Baltimore. When the Washington portrait was reinstalled in the White House, within the stone walls that President Washington had overseen, the portrait became integral to the history of the executive residence, not only for its subject but also the moment when Dolley Madison saved it in the "Second War of Independence."