Port is red wine, and its full name is Vinho do Porto. It was originally a sweet wine made from Portuguese red grapes. It is usually a dessert wine, which is of course much milder than brandy. But there are also stronger varieties.

The meaning of this sentence is very clear. Brandy is a strong drink, and more heroic people think it is suitable for them. Port is relatively mild and suitable for ordinary people. In fact, Port, only products produced in Portugal can be called Port (Regulations of the European Association for the Protection of Trademarks of Origin Products). Products from other places are just knockoffs.