The Retro is on the site of the first home of Sarah Porter Hillhouse, the first woman in Georgia (and most likely the Southeast) to edit and publish a newspaper. That paper, The Monitor, might have been printed at this location. After the Hillhouse family moved in 1801, the Retro site was used for many purposes, a notable one being the boarding house where in 1802 Peter L. Van Allen, a New York lawyer, challenged William H. Crawford, a noted marksman, to a duel over political matters. This event led to the death of Mr. Van Allen and the birth of the two-party political system in Georgia.

The house became a tavern and hotel in 1805 but burned in 1827. Replacement structures also were destroyed by fire. A hotel in the building was advertised in 1876 as the only one south of the Mason-Dixon Line with an attached ice house. Over the years the building was known by many names, including Sneed's Tavern, Washington Hall, Washington Hotel, Foreman House, Riley House, The Southern Comfort, and the Ivey House.

The current Retro building was constructed in 1920. The upper floor initially housed a barber shop and three apartments. The ground floor was used by different businesses throughout the years including a roller skating rink, barbershop, bookstore, cinema, and stationery store.

The Courtyard was used as an auto repair shop until a storm destroyed the roof in the mid-1980s.