How can people talk about absolute rubbish and claim I add nothing to the conversation?

I was in work the other day and they were talking about this girl who called the mananger a liar and they were all joking around making a huge deal out of it. I just laughed nervously and try to avoid this kind of conversation and people always talk about me behind my back I've heard them calling me a and that I add nothing to the conversation. Why are people like this?