One of the most important 13th century castles in Scotland, it was built by the de Vaux. The family were originally from Rouen in Normandy and came over to Britain following the Norman Conquest. The castle changed hands many times during the Wars of Independence with England in the late 13th and early 14th centuries. Over the following three hundred years it was rebuilt and extended several times.

The massive 13th century drum tower still stands, as does the elegant 14th century gatehouse, and a fine 16th century lodging. Beside the castle are the renowned gardens first recorded in the 16th century. The present gardens include an early 20th century 'Arts & Crafts' herbaceous border and a restored Victorian garden.