Too much seclusion is also a warning sign. What does your nephew think about the new school? Is he making friends? Does he talk favorably about classes and activities? How are his grades? Is he unaccustomed to challenging academic demands? Have his appetite and sleep habits changed? If your nephew truly appears to be unhappy and withdrawing from everything -- then he may be experiencing some depression.
His mother's plan to involve him in more peer events seems like a good idea. If it does not "jumpstart" him to get back on track, then it may be wise to have him talk with a school counselor about how he is feeling about the new changes in his life. Professional counseling is always an alternative, but start with talking to the school to get firsthand information about his adjustment there.