There is a study in young, healthy men, and they calorically restrict them, and their testosterone does decrease. So, if you're young & healthy and you don't have metabolic syndrome, then caloric restriction will likely decrease your testosterone.