Over the years people have come to associate Holt House with Presidents John Quincy Adams, Jackson, and Van Buren, but no substantial evidence has been uncovered to support these claims. Since 1891 it has served as the administrative headquarters for the National Zoological Park. The Holt House is generally assumed to have been built in 1805; however, the earliest construction date that can be confirmed is 1827.
The first mention of the house in any official records is a reference to the building in the 1831 will of Roger Johnson. The Johnson family owned the property from 1809 to 1835. The tract of land upon which the house stands was originally called "Pretty Prospect", but came to be known as "Jackson Hill" by 1841.
Built prior to 1827; alterations by Glenn Brown, W.R. Emerson, and Hornblower and Marshall (1890‑1901)
DC listing November 8, 1964
National Register listing April 24, 1973