OK, here is a well thought out answer.
One thing that I got from the show is that the zombies don't just eat humans, but small deer and game. They have shown them chowing on deer or pigs. Not sure how much that impacts the stock, but Being that Darryl, the big hunter of the group, only manages to catch squirrel, I just assumed certain large game have become scarce.
Farming is hindered because all it takes are a group of raiders or a heard of zombies to destroy everything. Zombies herd by the thousands in this show, and trample everything and eat anything living. You would need a farm that is very well secured and room for livestock that can graze. You can't rely on crops alone, because they yield only a short time of the year. I would think the safest bet would be securing a fruit orchard with canning equipment, since this is Georgia, I assume there would be plenty of peach orchards.
But at the same time, I think most people would not resort to eating people, and if they got hungry enough that they needed to, they probably wouldn't have the strength to fight off walkers. I read about that there were a couple cases of cannibalism during the great potato famine in Ireland. They lost about 1 million people to hunger in a few short years. Leads me to believe there would be a few isolated incidents, especially with the lack of any law or authority.
The big thing that I don't believe is that they are so organized about it. The random fellow knifing any survivor they find and grilling them up seems a lot more believable. Those terminus people were pretty much basting people in sweet baby rays and slow cooking them to perfection over a grill large enough to feed 100 people. Then keeping multiple meat storage rooms, Killing 8 people at a time, and storing them without refrigeration systems to feed what looked like only 20 people.
The TL;DR version. Cannibalism = Yes. Organized Cannibalism towns = No.