Noone really would completly destroy a castle/fort! so you should have to make the person surrender it via a siege. and then your men occupy it, that would be cool.
defences should take a lot longer to erect, even though i use em alot, it wold make it much more realistic.
rushes are not what could happen in real life, i mean 10 knights cant take down a town, thats stupid.
you should have to have supply lines for your troops, they should have to eat, which would kut back on sending peole from one side of the map to the other and just sit there for the entire game. and you should have to put food on boats in order to have them sail around for the entire game. you could accompany your fleet with fishing boats and they could feed the navy
AOM wold be relly cool if the game would be REAL