Talk also with the school counselor. Ask her to observe your son in the classroom so that you'll have an objective opinion of what is going on there. If the problem is not the other children bothering your son, but your son becoming distracted over something else, the counselor may be able to give him some individual time or include him in a small group on work completion.
The Right Environment to Succeed
Elementary School Expert Advice from Barbara Potts