It is so hard to take sides, especially when it comes to family, but if you've talked to your brother and he won't change, then I think you have to side with your son and stop contact with your brother. It will probably be the only way to get a message to your brother that you are serious. If he promises to change, don't take your son around until you've been around him yourself and seen a change. Good luck!