I hate that, too! I experience it a lot while touring, especially. Local girls are resentful of travelers coming in sometimes.
I always try to return references asap. I know how it feels to lose a date because a client couldn't be verified... So frustrating.
I recommend guys join preferred411, because their references are stored on their profile. It saves a lot of waiting and bypasses cockblockers.
Sometimes when all else fails, I will have the client put the pressure on a reluctant reference by saying, "Please let (provider) know to expect a PM/email/text from me. I'd love to meet you, but I cannot until I receive her confirmation." That way, the non-cooperative escort risks offending her client by not answering.