Your concerns about retention are justified. It does damage self-esteem. Furthermore, researchers have found that children with high self-esteem are more likely to succeed in school because they believe that they are capable and have the ability to develop competence in school tasks. Years later, you'll hear children who have been retained making remarks like, "If I had only done this, I'd be in fourth grade now."
We rarely support retention and are even less enthused about holding children back when it is suggested for non-academic reasons. Since your son has shown that he can handle the academic activities, there does not seem to be a good reason to retain him. According to the National Association of School Psychologists, retention has not been shown to be successful when it is linked with "developmental immaturity."