My opinion is that a parent is always preferrable over a babysitter. You would want to be able to parent your child if he was in the care of his father and his father had to go to work. Do unto others...
I don't know what the background issues are regarding medical care, etc. But I always suggest that both sides keep in mind that everyone parents differently. Even couples who are together are challenged by this. Barring neglect and abuse, you have to give the other parent the benefit of the doubt. I understand that you don't care for him, and obviously something happened to change your mind since you decided to become pregnant by him. However, unless he has some sort of severe mental disorder it's highly unlikely that he would want any harm to come to his own child, and the fact that he is exercising his visitation says that he loves and cares for his child.
Also, regardless of what anyone says, everyone is concerned with the CS on some level. After all - it is money. Getting it does make it easier to pay the bills, and paying it sure makes it a little more difficult. So anyone, on either side, that says they don't care about CS is full of it. Everyone cares. However, that is not to say that caring about how much money you're left with after you pay all of your bills means that you don't care about your child. Or that all you care about is the CS. I think that's a very shallow interpretation of a persons motivations, and one that should be put aside when negotiating custody and visitation arrangements. Because, if there is anyone who truly doesn't care about how much money is changing hands, it's the kids. All they care about is their relationship with their parents.
Your ex's failure to attend medical appointments says nothing about how much he cares about his child - especially if you're there at the same time. In fact, if things are contentious between you two (and it seems they are) it's probably a good idea not for him to be there.