first off, he has a very slim chance of getting sole custody, as most courts are not father friendly. So be happy you are a woman.
Next off, judges don't usually change custody from one parent to the other. It happens but not often.
Let Dad you know you *want* him in his daughter's life, as it is a good thing for her to know you both and develop a relationship with her. If you have an atty, let all contact with the other party be made through the atty and get out of the middle of it.
Just keep doing what is best for your child...