When Prospect was acquired in 1849 by John Potter, a wealthy merchant from Charleston, South Carolina, he replaced the colonial structure with the present mansion. In 1878 Robert L. and Alexander Stuart of New York bought the house and accompanying 35-acre estate and deeded it to Princeton University, known at that time as the College of New Jersey. Beginning in 1879, the house served as home of Princeton University’s presidents. James McCosh, its first resident, thought the house was the finest in the world for a college president and that its grounds were like Eden.

As the campus enlarged, students began to take shortcuts across the lawns and garden of Prospect, depriving it of some of its "Garden of Eden" qualities. After a particularly flagrant instance of trespassing by a rampaging football crowd, Woodrow Wilson, then University President and Prospect resident, erected an iron fence enclosing five acres of the grounds in 1904.