In 1846 Orlando Smith, a stonemason from Ledyard CT, found excellent granite on the premises and took an option to buy the farm for a price of $8,000. His elderly mother, Sally Raymond Smith, rode horseback from Ledyard to give to Orlando the $2,000 deposit to bind the deal . Three generations of Smiths developed the quarries into one of the largest and finest granite quarries in the United States and helped make Westerly one of the most important granite centers in the world.
Orlando and Emeline (Gallup) Smith moved into the house in 1848. The first floor of the north wing, including the underpinning, was in poor condition and was rebuilt by Orlando’s father-in-law, Isaac Gallup, also a contractor from Ledyard.
When Emeline and Orlando moved into the house, Dr. Noyes and his wife were residents, and part of the deal was that Dr. Noyes was to have life tenancy, have milk and wood delivered to his back door and have unlimited use to the “dung” pile just as the Smiths had. His saddle bag now sits next to the fireplace in the keeping room. The house was divided so that both families had their privacy. It is believed that the Smiths had the north wing plus the present dining room and access to upstairs bedroom (now called Julia’s room) and perhaps the common room on the second floor, while the Noyes had the keeping room, living room, and upstairs guest room. After Dr. Noyes’ death the Smiths had use of the whole house. Raising four children they needed all the room, and they had various stonecutters and other granite workmen living there also.