Learn exactly how to get rid of your chest fat, based on science. Gynecomastia is a rare condition in which men grow breast tissue. Often men suffer from pseudo gynecomastia, rather than real gynecomastia. In that case, you would need to lose body fat to lose your "man boobs".