You can remove players from your friends list by selecting their name and choosing the "Remove" option in the social menu.