To remove a user who is assigned to a task, simply click the users under 'Assign to’.
This will result in a pop-up menu of who is currently assigned that task:
Deselect the person you wish to remove from the task:
Then click 'Save'
You have now removed that person from the task.