No it won't make much difference:) I swap between my sets all the time (Higher armour & Lower) throughout casual games i usually use my 'better mogged' set, but when doing more serious games ill go high armour just incase it makes a difference xD
Also you can use Nobleman's Coat, Nobleman's...