We also had an extremely shy grey who was very afraid of DH at first. I can't offer any thing other than what others had said, but Bailey eventually came around when DH let her do it on her own terms. Neither of us could walk her though, nor get her out of her crate. We literally had to carry her outside after we checked to make sure there weren't any people around or anything blowing in the yard or making scary noises. Love and patience worked and commitment to make her feel as safe as possible.
Brooke is very afraid of men and children, sometimes when she gets overstimulated, we need to find a quieter place to be and I just keep talking to her in a soft, assuring manner, trying not to coddle her too much.
She may always be "your" dog, sometimes you just have to accept that.