“Idle Hour” was the home of William K. Vanderbilt Jr. and his wife, Virginia Graham Fair. The couple were married in 1899 and an article from the 12 Apr 1899 issue of The San-Fransisco Call newspaper describes how it burned. The home shown below, was located near Oakdale, Long Island. The mansion built to replace this one, is now part of Dowling College. On their website, you can read some more history and take a virtual tour.
You can read the whole article here.