I totally understand you getting the brush off vibe. Unfortunately, I've done that. Not because I didn't want to speak to that person, but I was in a meeting or some other distraction.
To be completely honest, never has someone's call, no matter how "annoying" it may have seemed in that moment, ever effected their hiring chances.
I would recommend calling. And don't necessarily read too much into the reaction you get. Last time I got brushed off, I got the job I'm in now!