When you don't respond with any more information it's a red flag that people can read without a problem and you didn't say anything that can land you in hot water.
That being said with a job situation like the OP mentioned... I think the 'call me' and off the record conversation not made about any direct person wouldn't be a legal issue. The second you made disparaging remarks about an individual though you could. As others have mentioned though beware that admins know each other and talk.