I think they’re terms men often apply to whichever figure in the household represents the greatest constraint on their actions (or the person to whom they most defer).

If they’re married, that’s their spouse, but if they’re living with their parents in a traditional male-dominated household, it’s their father.