At some point.

When you use humor to keep people at arm's length, is it still humor? How about charm? Solicitousness? "How's that baby of yours?"

At some point, it becomes obvious that all your humor, charm, and solitictousness aren't genuine. You aren't to be touched, you aren't to be trusted. You're a good guy. "He's a good guy," they say. And at your funeral, "Yup. He was a good guy."