St Mary’s Church faces the castle gates on the opposite side of Chilham square, it has an ancient and longstanding connection with the castle and it’s owners that continues to this day.The interior of St Mary's is notable for some very fine memorials, many to members of the Digges family of Chilham Castle. The finest of these is the very grand monument to Mary Kemp, Lady Digges, who died in 1631. She was the wife of Sir Dudley Digges, who had the current Chilham Castle built beside the old Norman keep above the Stour.

Sir Dudley Digges completed the building of Chilham Castle in 1616, on the site of a ruined castle of which only the Keep remains. Out riding, he would have been attracted to the prominent bluff overlooking the Stour Valley and appreciated the sweeping views, as we do today. He terraced the land that fell steeply towards the west and it’s likely he planted the sweet chestnut avenue in the park. About 150 years later, owner Thomas Heron enlarged the park and redesigned the grounds with the help of Lancelot ‘Capability’ Brown. The ‘haha’, which kept livestock contained without interrupting the view with fencing, is one of the best of Capability’s Brown’s innovations. Charles Stewart Hardy, the castle’s owner from 1867 to 1914, dug the lake and planted hundreds of trees. Today, a team of gardeners manages the twenty-acre garden. In summer, it takes two full days a week just to mow the lawns and in September two weeks to reshape the topiaries.