Grounds are complex beasts that do not obey any of the normaly obvious rules. Electrical engineers can take classes to study just grounds at the graduate level. That is more education than I have, and I've seen the math, it is harder than I want to deal with.
I just follow simple rules: no ground loops. If this way audio you would hear strange sounds, and never be happy. This is digital, which is a little more immune, but in this case.
Mind you might have some other problem, but start by eliminating a known potential source of problems.