Question about weekend getaways without the kid... is it okay?

My fiance has an 11-year-old son. We are getting married in June, and we have been living together for a year and a half. We have primary custody of his son. I am the model soon-to-be stepmom, volunteering at his private school when I am not working, making homecooked meals, helping with homework, discliping him yet still being his friend, etc. My question is this. My fiance and I go away for a weekend getaway, just the two of us, about once every 4-6 months. It is usually local, about a 3-4 hour drive away, and his son stays with grandparents while we are away. I thought this was fine, but recently someone suggested to me that we should not be doing this, and that if you have kids you should always take them with you. I cherish my time alone with my fiance, but I would hate to think we are messing the child up. We took him on a 5 day camping trip to Mammoth Cave last summer and allowed him to bring a friend. Are we being selfish to take these romantic weekend getaways?