As the metal wall cramps that were on the façade showed, the castle was built by the Lord of Piemont in 1665. Afterwards, it belonged successively to the de Fossez, de Briey, de Laittres and de Gerlache families in 1805.
A hundred years later came Marianist brothers who had fled France because of the Combes Law (Separation of the Churches and the State). They took refuge in Saint-Remy and bought the castle. They had a new building erected to accommodate classrooms and a dormitory. When they left in 1946, the parish priest of Mussy-la-Ville set up a shelter for children in distress. This lasted until 1952 when most of the building was razed to the ground. Of the original castle, only the square tower with a tented roof remains. On one of the corners of this tower, one will still see some traces of the junction with the demolished curtain wall.