You shouldn't be listening to your coworkers. They are not in charge of your life, and you don't need to discuss your personal activities with them. It's really none of their business.

Go out with him and have fun. No, the age difference is not significant. Some experts would say that two people within 7 years of each other are of the same generation. Even if the age difference were greater than 7, 10, or more years, it still wouldn't matter. The May-December generation gap would be the same for an older man with a younger woman, and there doesn't seem to be much controversy with that, so the reverse should hold. The only time the age difference for an older woman and younger man could be an issue is if the woman is past the age to have children and the man wants a biological child. Otherwise, adoption is a viable option.