This site has been used for military installations since 55AD when the first Roman fort was built upon it during the pacification of the Silures Tribe. In 1091 Robert Fitzhamon built a Norman Keep on the site and it would have been this castle that imprisoned Robert of Normandy after he failed to retake England from his younger brother Henry I following the accidental death King William II (Rufus) in 1100.
The 18th century saw the castle become the property of John Stuart, 3rd Earl of Bute and it was his descendants that refashioned the castle into what we see today. John Crichton-Stuart, 2nd Marquess of Bute hired Henry Holland to enlarge the castle and introduce a Gothic Revival style, while John Crichton-Stuart, 3rd Marquess of Bute and William Burges went further with extensive rebuilding and extending the theme to create an architectural triumph.
It is possibly the 2nd Marquess of Bute, John Crichton-Stuart (10 August 1793 – 18 March 1848), Earl of Dumfries, Lord Lieutenant of Glamorgan and Lord Lieutenant of Buteshire, who haunts the castle in a red cloak. He died in a chapel that is behind the library. He has been seen pushing past people at the chapel doorway and even scowling at the Chief Architect of the time (1976). It is said he has also been seen walking through the library fireplace into the chapel where he died.