The human immune system uses an arsenal of effector mechanisms to prevent and counteract infections. Yet, some fungal species are extremely successful as human pathogens, which can be attributed to a wide variety of strategies by which these fungi evade, exploit, and modulate the immune system. These fungal pathogens normally are either harmless commensals or environmental fungi. In this review we discuss how commensalism, but also life in an environmental niche without human contact, can drive the evolution of diverse and specialized immune evasion mechanisms. Correspondingly, we discuss the mechanisms contributing to the ability of these fungi to cause superficial to life-threatening infections.