Is my ex trying to terminate my rights?

Some background first. I am the father of a six-year-old little boy who resides with his mother in SC; I reside in NY. His mom and I were never married, but I have never denied paternity and I love my son with all my heart. As it sometimes happens, his mom and I dissolved our relationship, but in an amicable way. Circumstances at the time forced me to move back to NY where I met the woman who would become my wife; we have a two-year-old son. My ex and I have never been to court to legally establish a visitation schedule, child support, or custody, although she of course has De facto custody. We did this of course because we wanted to prove that we could could be friendly enough to not need the courts to intervene. I fear that that decision is going to bite me. Recently I was going through a bout of depression and so I didn't call my son for several weeks. This caused a fight between his mother and me. She has asked if our son can use her fianc�'s last name since they have two daughters. She has also asked me to cancel a planned visit my wife and I had to come and see him, claiming that he is impressionable and I should only come down when we are ready to move to SC (which we are). This has made my heart sink since I love my son so much. She is claiming to have his best interests at heart, but I fear that she is going to turn it around on me and claim some form of abandonment so she can terminate my rights. I have always wanted to maintain civility and a friendly atmosphere. However, I fear that my good nature is being taken advantage of at the cost of my relationship with my son. What should I do?