These are 5 high-estrogen foods men should avoid to build muscle, lose weight, and get lean. Some foods should be limited, and others should be completely avoided. Instead of increasing estrogen, most men want to learn how to increase testosterone naturally and fast. By learning how to decrease estrogen levels, weight loss becomes much easier.