unmarried separated mom to 2 yr old... ex wants custody

[karmary]

I am an unmarried mom to a 2 yr old. Ex now is threatening to go to court for custody.
i have always agreed to visitation as often as he wants if he just calls with his schedule on Saturdays so we can plan our week around his schedule. he refuses, shows up whenever he wishes and disrupts daughters & my schedule (even though she is happy to see him, of course)
in the past (before the separation but after my daughters birth) We have gotten into some serious fights. He has pulled me to the ground on three occasions saying he was just hugging/ playing. one other time he choked me, he didn't even realize he how he got there. But he argued that i provoked him. I am a strong woman and have been handling this the best i can without playing a victim. the times he was physical i have fought back to show him his behavior was not ok. He has accused me of being abusive for that. I believe he has adult adhd and some other anger issues. i want him to have visitation with our daughter and to be in her life but i do not want him to have custody.
This week was the first week i refused visitation because he refused to contact me on saturday with a schedule mainly and he started the threatening me with court thing.
i believe his threats are just threats but i would like to be prepared just in case. I can not deal with this behavior on my own anymore but i do not know what direction to go. His name is on my daughters birth certificate, we lived together for five years and were separated shortly after our daughters 1st bday.
at a loss...alittle exhausted. please advise. thank you.