Before 1876, La Tour de By was called La Roque de By. The vine has been cultivated there for several centuries. Pierre Tizon, Lord of the Fief of By was the owner in 1599.
The Château remained in the family until 1725, when it fell to the Comte de Gramont, then to Monsieur de Lignac and, in 1860, to Monsieur Alfred Rubichon who built the current Château La Tour de By in 1876, which overlooks the Gironde. It was then bought by Mr. Julien Damoy who died there in 1942.