In the early days of gardening at the White House, the gardens were fenced away to facilitate care and watering. A large vegetable garden was planted on the west side, with a water pump nearby. There were ornamental plants, but there is no record of an ornamental garden until Andrew Jackson’s was built on the east side about 1833, centered upon an orangery that provided medicinal fruit through the winter. While the written information on Jackson’s garden is extensive, only one illustration exists, an accidental view captured in a construction shot of the Treasury Building during the 1850s. This extension to Treasury led to demolition of the garden in 1856.

President Ulysses S. Grant’s old army aide, the irascible General Orville Babcock, took charge of improving the grounds and built along the south side of the house narrow flower gardens pierced by winding gravel paths. These survived, vine choked, until Theodore Roosevelt’s renovations of 1902 and Edith Roosevelt’s design of a “colonial” garden where the present Rose Garden now stands.