You’re putting far too much thought into what other people mean by the phrase, particularly in the context of a joke.
Most people are not referring to several different anthropological, sociological, and feminist theories/philosophies.

When you disagree with “gender is a social construct” in a casual setting, intentionally or not, you’re conveying the statement “gender is innately tied to biological sex, there are precisely two, and trans people are invalid”.

It’s better to take the phrase as meaning “having a vagina doesn’t mean you’re a hot pink wearing pretty princess, nor does a penis imply you aren’t. Gender is more complicated than a binary, and we’re better off raising children as little people who tell us who they are than spending too much time being concerned that they only play with plastic figurines compatible with their genitals and playacting the right chores”.

It’s a joke about tricking people into attending an event usually focused on baby genitals, and then instead giving them cake that isn’t coded to the babies genitals with a lecture about how they don’t tell you as much about who this little person will be as people think.