The Manor House was built in the neo-classical Palladian style, following the strict principles of proportion and symmetry that owed their origins to ancient Greek and Roman architecture. The rich plasterwork decoration of the interior uses many classically inspired motifs and the splendid Corinthian columns in the Dining Room are also typical of the style.

Although we cannot be certain how the house was used, we can make an educated guess based upon surviving evidence and contemporary accounts. From the entrance hall, a visitor to the house on business (generally in the mornings) may have been taken to the Library, and an afternoon dinner guest to the Dining Room. Upstairs, in the Ballroom or 'Salon' evening entertainment would have been provided for family and friends. Next to it, the Withdrawing Room was for more intimate guests, with the only Bedroom adjoining. A network of narrow stairs, little connecting lobbies and corridors gave the servants easy access to these public areas, although the kitchens were in the much older timber-framed property next door.

The Manor House had a succession of owners after the Hervey family. In the nineteenth century, there are several references to its being used as judges' lodgings during the Assizes. This gave rise to the alternative name 'The Court House', by which the Manor House was known for many years. The best known owner of later years is Walter Edward Guinness, the first Baron Moyne (1880-1944), who owned the house from 1908-1933. He was M.P. for Bury St Edmunds.

In 1947, West Suffolk County Council bought the house and used it as offices. In 1985 it was bought by Bristol Estates. Finally, St Edmundsbury Borough Council bought the house in 1988 and, after restoration, it was opened as a museum in January 1993.

(PLEASE NOTE: In 2007 the Manor House was sold once again to take up its original role as a private house. The museum collections were removed to Moyse's Hall Museum.)