It sounds like you and your husband may cause your son considerable anxiety about doing "well" in school and are ready to blame one another if he doesn't measure up to your standards of excellence. Please don't let your son be affected adversely in his attitudes toward school by you and your husband using this as something to continue fighting about.
Unless both of you can be totally supportive and non-judgmental about his work in school, don't engage him in any talk or activities related to schoolwork. Give the little guy a chance and enjoy him as he learns so many new things, WITHOUT ANY PRESSURE.