Your son may or may not be the youngest in the kindergarten class, but he will definitely be one of the youngest. Just being young certainly doesn't mean he won't do well in school. Plus, the kindergarten teacher is right about the small-class offering an advantage. It will allow the teacher to give the children more individual attention.
Ask the kindergarten teacher about the expected age range of the 16 children. If almost all the other children will be a full-year older, they are likely to be more mature and larger than your son is. Also, the curriculum for these older children could be focused more on seatwork and other activities that are difficult for younger children to handle. While everyone believes that your son can do the work, keep in mind that being one of the youngest may have certain drawbacks as he gets older. He will not be ready as soon as his classmates for dating, driving, and other teen activities.
If you decide to keep your son out of kindergarten, you do need to keep him in a structured routine. Since he has already graduated from the 3-day, 4-year-olds' class, don't repeat it. You always want to keep him moving forward. The 5-day, 4-year-olds' class is a nice option.