The Hermitage is built in a secluded meadow that was chosen by Rachel Jackson, wife of Andrew Jackson. The original mansion was a two-story Federal-style building, built with bricks manufactured on-site with skilled slave labor, and completed between 1819 and 1821. It had four rooms on the ground floor and four rooms on the second level, each having a fireplace and chimney, and large central hallways opened in warm weather from front to back to form a breezeway. A simple portico was added later. In 1831, while Jackson was away in the White House, he had the mansion remodeled under the direction of architect David Morrison, with flanking one-story wings, a one-story entrance portico with 10 columns, and a small rear portico giving the house a Classical appearance.

In 1834, a chimney fire seriously damaged the house with the exception of the dining room wing. This prompted Jackson to have the current 13-room Greek Revival structure built on the same foundation as the former house, which was completed two years later. The architects for the house were Joseph Reiff and William C. Hume, who were building Tulip Grove across the road. The mansion is built in a rectangular layout, approximately 104 feet from east to west and 54 feet from north to south. The south front is the location of the main entrance, and includes a central block with a five bay two-story structure, with a portico supported by six modified Corinthian style, wooden columns with a simple entablature resting on the capitals. Within the portico is a second-story balcony with simple square balusters. One-story wings, with single fenestrations, flank the mansion and extend beyond the mansion to the front of the portico, so that it is enclosed on three sides. While the southern façade gives the appearance of a flat roof, the three other elevations reveal that the tin covered roof is pitched. The front façade was painted a light tan and sand coating was added onto the columns and trim to simulate the appearance of stone. A near replica of the front portico is found on the north end of the house, though featuring Doric style columns and capped with a pediment. The layout of the main block of the house is four large rooms separated by a center hall. The entry hall with plank flooring painted dark is decorated with block-printed wallpaper by Joseph Dufour et Cie of Paris, depicting scenes from Telemachus' visit to the island of Calypso. At the far end of the hall is the elliptical cantilevered staircase, with mahogany handrail, that leads to the second level. To the left of the hall is the front and back parlors with crystal chandeliers and Italian marble mantels. Leading from the front parlor is the dining room in the east wing. Decorated with a high gloss paint to reflect as much light as possible, the fireplace features a rustic mantelpiece called the "Eighth of January". Carved by one of the veterans of the Battle of New Orleans, he only worked on the mantelpiece on the anniversary of each year until he finished on January 8, 1839. Jackson installed the piece on January 8, 1840. Adjacent to the dining room is a pantry and storage room that leads to an open passageway to the kitchen that is separate from the house to prevent risk of fire as well as to eliminate noise, heat and odors of cooking. To the right of the entrance hall and accessed via a side hall are two bedrooms that were occupied by President Jackson and his son, Andrew Jackson, Jr. In the west wing is a spacious library and office that was used to manage the plantation. On the second level are four bedrooms that were used by family members and guests, including Sam Houston, the Marquis de Lafayette, and Presidents James K. Polk and Martin Van Buren.

The site today covers 1,120 acres, which includes the original 1,050-acre tract of Jackson's plantation, that is overseen by The Andrew Jackson Foundation, formerly called the Ladies' Hermitage Association. The mansion is approached by a cedar lined, 10-foot wide, guitar shaped carriage drive, designed by Ralph E. W. Earl, that made it easier to maneuver carriages in the narrow space. To the east of the house was a 1-acre formal garden designed by Philadelphia-based gardener William Frost in 1819. Laid out in the English four-square kitchen garden style, it consists of four quadrants and a circular center bed contained by unusually long, beveled bricks and pebbled pathways. Originally the garden was used to produce food for the mansion and secondarily as an ornamental pleasure garden. The garden is surrounded by a white picket fence that includes on the north perimeter a brick privy that served as a status symbol and garden feature. Rachel Jackson died in 1828 and Jackson had her buried in the garden she loved. When the house was remodeled in 1831, Jackson also had a classicising "temple & monument" for Rachel's grave constructed. Craftsmen completed the domed limestone tomb with a copper roof in 1832.

LARRY PERSONAL NOTE: I VISITED THIS HOME IN THE 1980's BUT NO LONGER HAD THE PHOTO PRINTS SO A SECOND VISIT WAS JUST TO GET THE PHOTOGRAPHS. I UNINTENTIONALLY "SNUCK" IN THROUGH A BACK ENTRANCE WITHOUT PAYING. OOPS!