Tax collectors were notoriously dishonest, as they always extorted far and above what was owed. That's how they ensured their personal profit. They weren't liked by the people. Matthew was also a Jew, who collected taxes from Jews for the government. That was doubly offensive to them. Matthew was a self-interested and greedy tax collector. But then he met Jesus, who chose Matthew as a follower.

The Calling of Matthew

While Jesus was having dinner at Matthew’s house, many tax collectors and sinners came and ate with him and his disciples. When the Pharisees saw this, they asked his disciples, “Why does your teacher eat with tax collectors and sinners?”

On hearing this, Jesus said, “It is not the healthy who need a doctor, but the sick. But go and learn what this means: ‘I desire mercy, not sacrifice.’ For I have not come to call the righteous, but sinners.”

More Info: en.wikipedia.org