Opportunistic commensal and environmental fungi can cause superficial to systemic diseases in humans. But how did these pathogens adapt to infect us and how does host-pathogen co-evolution shape their virulence potential? During evolution toward pathogenicity, not only do microorganisms gain virulence genes, but they also tend to lose non-adaptive genes in the host niche. Additionally, virulence factors can become detrimental during infection when they trigger host recognition. The loss of non-adaptive genes as well as the loss of the virulence potential of genes by adaptations to the host has been investigated in pathogenic bacteria and phytopathogenic fungi, where they are known as antivirulence and avirulence genes, respectively. However, these concepts are nearly unknown in the field of pathogenic fungi of humans. We think that this unnecessarily limits our view of human-fungal interplay, and that much could be learned if we applied a similar framework to aspects of these interactions. In this review, we, therefore, define and adapt the concepts of antivirulence and avirulence genes for human pathogenic fungi. We provide examples for analogies to antivirulence genes of bacterial pathogens and to avirulence genes of phytopathogenic fungi. Introducing these terms to the field of pathogenic fungi of humans can help to better comprehend the emergence and evolution of fungal virulence and disease.