Fact: Not every dog afraid of men, people with hats/sunglasses/deep voices has been abused.

Some dogs have sadly experienced traumatic situations.

Many others however simply react to visual and acoustic stimuli" that are naturally perceived as threatening. A tall and square built, deep voice or certain clothes or accessories can trigger a dog's fear without any previous negative experiences. 

Don't automatically write off fearful reactions as a trait that cannot be improved - instead, help your dog to find out it might not be as scary as he thinks.