Help me my ex is emotionally abusing my child!!?

I fled from my home from domestic abuse/violence. my daughter had witnessed this. he was seeing my children every fortnight for the weekend until i found out he was drinking whilst they were in his care( he has a drink problem). I pulled contact and he then applied to the court for residence and interim contact, we agreed he would phone them 3 times a week and see them every sunday. boxing day was there first visit with him. my daughter came back an emotional mess. he had told her mummy wouldnt let them see him as i was silly, he told her mummy cant have nothing from this house and then went on to tell her she must stay away from black people! my daughter is 5 yrs old and most of her friends at school are black and asian she was terrified and wouldnt stop cryin as she thuoght she was not allowed to talk to them. i now want to stop the phone calls at least until i see my solicitor who returns next week. am i doing the right thing?