The tiny hamlet of Chalbury amongst the trees in the deep narrow roads near Horton should not be confused with Chalbury Hillfort near Weymouth. The church stands so prominently that Hutchins, the county historian recorded.

'A very high Elm tree standing by the church was used as a landmark because it could be seen from the Channel and might be plainly discerned from the hills that lie above Yarmouth in the Isle of Wight'.

The elm tree was broken in a storm in 1703. Remarkable views are however still available from three sides of the church with Horton tower prominent. The little church itself is white-washed with a red tile roof, and although the chancel and parts of the nave are 13th century the overall impression is 18th century.