My suggestion is for you to see a counselor with your boyfriend. Your son needs you to be his advocate to help him get the support, love, and adult male relationship(s) that he needs. Continuing to stay with your boyfriend under these conditions is unhealthy for your son. As he becomes more and more in need of a father-figure to model himself after as he approaches adolescence, your boyfriend's neglecting him will prove more hurtful. I know that I'm asking you to make some difficult decisions, but you are responsible for your boy's welfare. This man isn't honoring what's most precious to you -- your son. If your boyfriend won't go to counseling, then you need to go on your own. Your son has suffered far too much neglect in this "family" relationship. A therapist can help you put other caring men in the life of your boy.
Mom's Boyfriend Won't Get Close to Her Son
Toddler and Teenager Expert Advice from Carleton Kendrick, Ed.M., LCSW