Is there anything a court can do about a coustodial father alienating his children from their mother?

Yes. Apply to court for a change in circumstances order, the court will consider all evidence, but in general the court may either fine the father for contempt of court, if he still refuses to comply then a small time in prison may be found to be the option most suitable. If any of the children are over 12yrs, then the judge will take their thoughts into consideration before judgement. Paramount is the Welfare of the children.
They can give mom the kids.
Yes, depending on what is stated in the Custody Order. Most including mine, state that we cannot discuss matter pertain to the other party with her. We are not to speak in a negative manner about the other parent, and we must allow unimpeded phone calls, which means we have to let the other parent have phone time with her, and not listen in or interrupt their conversation. If you can prove that he is alienating the child, you can file a contempt motion. Then the matter will be set to be heard by the Judge.

In your best interest try to document, and if need be try to record what you can. That is what I had to do. Instead of have standardized visitation, we had our order state moderate visitation with me and my side of the family. I was in the Army and could not be home every other weekend. Then instead of letting my family see her as often as I would have, he would let them visit with her like 1 day every 2 weeks and no overnights. Which led to a huge blow up, I ended up leaving basic training and going home to take him back to court, where I did not ask for custody back, but did seek standardized visitation. Which was granted. The Judge was a little upset with all the circumstances. But I had showed where when we had called to set up visits and he would not comply. It was a huge mistake not to outline everything to detail, but I did not want to tie my family down to a every other weekend, one day a week, now I wish I had. I ended up getting out of the Army, which I was really sad about, I had always dreamed of going into the military.
If by alienating them you mean not allowing court appointed visitation rights, then yes, that's as much a court order violation as a non-custodial parent not making child support payments.

If by alienating them you mean trying to drive a wedge between the mother and children by bad-mouthing the mother around the kids, nope, nothing illegal about it. It's just not nice.


EDIT: Okay, it sounds like you're doing everything right. Since you voluntarily gave the boys to him with what I assume was a verbal agreement about the accessibility you would have to them, and he's violating that agreement, you can take him to court to get something more fair and ordered by a judge to compel him to do the right thing.
you would need to be more specific as to why you would rather your children didnt see there mum !!!! but in general if you have been through court then your ex can have you back in court for breaking your order x x x
I imagine there is if it could be proven that the father is being mentally abusive to the children due to the emotional distress the father is putting on the children.


Answer:
Bottom line is this is going to be your word against his, (you did not get to detailed in you ?). And unfortunately unless one of two things happen it's going to be really hard for you to regain full custody or even primary custody since the boys have been with their father.
1. Either the father no longer wants to be primary care giver.
2. Unless you can prove to a court/judge it would be in the best interest of the children for them to uproot the children and place back into your custody.

It is not far but to be completely honest here the courts will wonder why you gave custody up in the beginging.
If you appear in court simple stating the father is alienating the children (in your opinion) and you want custody the judge is NOT going to grant it. There has to be more then that for the judge to change the original court order.
If there are some issues going on in the custody arrangement I would recommend you document your concerns and get evidence so when/if you do go to court you will be able to bring the documentation/evidence.

Another route could be in family court to ask the judge to appoint what's called a GAL- Guardian Ad Litem, this is a third party non-bias opinion that reports directly to the courts.
This person will do a seperate investigation and look what is in the child/rens best interest and give a recommendation to the judge.
In my experience the courts really listen to the GAL.
Some states do not do family court/custody/divorces they are appointed to in these states to children who are placed in foster care. But it is worth checking into.

Good Luck