The Vanderbilts were intent on raising their own plants to landscape Idle Hour. Accordingly, the Toolhouse and Potting Shed, believed to be designed by Richard Howland Hunt and built by William Bason and Sons, was constructed in 1902. It was situated at the southern end of Central Boulevard, adjoining the greenhouses located on Connetquot Drive. A two and a half story Georgian brick structure of four bays, topped with a hip roof containing two gable roof dormers, its rear (west) side was constructed without windows to facilitate the early growth of seedlings. The exterior brickwork features flying buttresses and segmented arches over the east window and door openings.

The ground floor living-dining area is topped by heavy ceiling beams; reportedly, the storage of apples on the upper floor necessitated beams of their size.

The Potting Shed was converted to a private residence at the end of World War II, and has been the residence of just two families since that time.

Adjoining the house is an "antique" swimming pool, converted from the original power house basement by the first resident in the early nineteen fifties when it was discovered by him during landscaping efforts. The diving block was formed from the original power house chimney area.

Visit The Ice House.