A castle’s servants were usually men. The medieval household was based on the older idea of a ‘war band’ and so had a military origin. Its members were supposed to be able to fight as well as serve.

True

Castle servants were indeed normally men. Service to a master was part of the structure of medieval male life, right up to the barons who served the king. A lord would expect to be served by well-born men and boys. A king would expect to be served by the sons of great nobles.