Going to grandma's house or seeing his father is like a fantasy world for him, where everything's fun. I'm concerned that this could have a negative effect on our relationship. My heart breaks when he wants to stay with his grandmother and not come home with me. Am I doing the right thing?
You may want to enlist your in-laws in helping your son prepare for your arrival. They might help you out by talking positively about your coming to pick him up and reminding him about all the great things that he did with them. They may unconsciously like the fact that he makes a big fuss about staying with them. Appeal to their good nature, explaining how you feel about this without putting them on the defensive.
Your four-year-old is too young to appreciate the struggles that you are going through to make a better life for him. But he does know that you are a constant loving presence in his life and as he ages he will appreciate more and more what you are doing for him. Your growing son will be proud of his mom and you are setting a wonderful example for him. Don't be so hard on yourself. I applaud your efforts and your spirit.