James W. Gaff was the younger brother of Thomas Gaff. The brothers, along with John Gaff, were often business partners. With Thomas, James owned and operated a papermaking business and a distillery in Philadelphia before moving to Aurora, Indiana in 1841. Family legend states that when James was getting off a river boat, presumably in Aurora, he saw Rachel Conwell and declared she was the most beautiful girl he had ever seen and vowed then and
there he would marry her. He did so in 1845. Paintings of James W. and Rachel hang in the East Parlor at Hillforest. Additionally, there is a portrait of Jane Gaff Shillito, one of their eight children, hanging in the East Parlor.
James and Thomas established the T. & J.W. Gaff & Co. Distillery on the banks of Hogan Creek. James preceded Thomas in moving to Aurora and also building a stately mansion in the city. James’ home was called Linden Terrace after the imported German linden trees he had planted there. After the Civil War, James and his family moved to Cincinnati. Linden Terrace was occupied by one of the Gaff sisters and her husband for a time but was eventually torn down. James is believed to have been instrumental in convincing the Fleishmann family to move to Cincinnati and use their yeast recipe in brewing and distilling. Upon James’ death, he was reputed to be the richest man in Cincinnati. He passed most of his business interests on to his wife Rachel, who operated them until her death. At that time she was believed to be worth 8 million dollars.