The Gate House marking the western entrance, located at the junction of the old Montauk Highway with the Connetquot River, was a particularly picturesque introduction to Idle Hour. It, too, was approached through an elaborate entrance gate attached to the iron Vanderbilt fence. Because the railroad passed within a few feet, it was also used as a special debarkation point for Vanderbilt guests arriving by train. An example of the nineteenth century architecture of the Vanderbilt estate, it was designed by Richard Morris Hunt. The two story English style cottage was built in 1889 at a cost of $30,000. The first floor is smooth brick, while the second floor is half-timber and stucco, with a bracketed overhang. All windows are diamond pane casements, with oriel windows at the second floor.

Of interest to Vanderbilt historians, in 1889 the West Gate House was the refuge of William K. Vanderbilt, Jr., and his new bride, Birdie Fair Vanderbilt, when the first mansion burned during their honeymoon. It was here they stayed until they boarded a train to New York City on the following day.

It is also, according to local rumor, the place where the senior Mr. Vanderbilt came to sulk when he discovered that his wife was seeing Oliver H.P. Belmont during his absences. Thus, it is known as the "Pouting House".

Our next stop is at the Stables and Coach House.