Confusion over cause of Panic Attacks and Childhood Experience

Hi, there.

So, when I was a child, my father would sleep in my bed with me. Nothing ever happened, it was just that my bed was more comfortable than the couch. I'm not sure why he wasn't comfortable sleeping with his wife in their bed. Whatever. I'm pretty sure he's suffered depression for a long time, is probably an alcoholic although I'm not entirely sure how much he does drink (as he tries to hide it) and he also may have ptsd- I know he sometimes has night terrors and wakes up screaming (though I have never heard him myself, I wonder if that's because I'm a heavy sleeper). I think his father may have been fairly abusive, but we don't talk about these things. Anyway, he treats all of his children absolutely wonderfully (he could occasionally get a bit violent on separate occasions, and could perhaps be a bit emotionally abusive, but we were really annoying children and there were quite a few of us to manage- I don't think these are triggers for me) and both of my parents have sacrificed a lot for us. This is beside the point, I just want to establish that my parents are wonderful, wonderful, hardworking and loving people. The only thing is, I have some trouble with physical intimacy, specifically cuddling. I am more often than not uncomfortable cuddling people, for when it happens I just have flashbacks of cuddling with my father when he'd come and sleep in my bed. There is nothing traumatic that happened here, only loving cuddles between a parent and child, nothing was ever inappropriate in my memory. I just can't seem to get over it when I'm with people.. I often have panic attacks (whether this is due to this or something else, i am not quite sure) when in this kind of situation, or I freeze up. I know it really, really hurts my partner when this happens and she thinks she's a trigger for me and I think it really screws with her mind and I just don't want that to be on her. Yet, I can't seem to tell people about this, either. This post is the first time I've ever told anyone that my papa would sleep with me, and I know she's hurt and feels that I don't trust her enough to tell her, I just can't bring myself to say it, especially because nothing happened that would or should be triggering, it just is- it's a stupid situation that I just can't seem to get over. It might be more sympathetic if something had happened, and while I'm grateful nothing had so far as I can remember, I just don't know how to bring that up without it sounding like something did, or being able to reconcile in their minds that nothing happened yet it's still causing panic attacks...... Help?