The Elders, Koffi Annan, Jimmy Carter, and Graca Machel were denied visas to enter Zimbabwe