Sidbury Manor, a red brick Victorian manor house, was built in the 1870s by Stephen Cave who at that time was MP for New Shoreham and Paymaster General in Disraeli's government. His father, Daniel, had visited the fashionable Regency resort of Sidmouth in 1850, travelling down from Bristol where the family resided. He liked Sidmouth, buying a house there, and the surrounding countryside and over a period of the next twenty years bought land and property in and around Sidbury amassing several thousand acres to create a large landed estate.
After his father's death in 1872, Stephen looked to Sidbury and decided to set about building a mansion house on the estate worthy of his eminent position; he appointed David Brandon as architect. The house was completed in 1879 but sadly Stephen died abroad in 1880 and never lived in the house that he had masterminded.
His brother Charles succeeded to the property and for some years thereafter it was used for occasional visits by his family, especially during the summer and the shooting season in the winter. Substantial walled gardens were constructed along with the fish ponds below the house and many trees and shrubs were planted to form the basis of what remains today.
In 1939 a girls school from Westcliff on Sea, Alexandra College, was evacuated to the house and remained there for twenty years, since when the house has been lived in by successive members of the Cave family.