The COVID-19 pandemic is a defining global health crisis of our time. While the impact of COVID-19, including its mental health impact, is increasingly being documented, there remain important gaps regarding the specific consequences of the pandemic on particular population groups, including refugees and migrants. This study aims to uncover the impact of the COVID-19 pandemic on the mental health of refugees and migrants worldwide, disentangling the possible role of social and daily stressors, i.e., experiences of discrimination and daily living conditions. Descriptive analysis and structural equation modeling were used to analyze the responses of N = 20,742 refugees and migrants on the self-reporting global ApartTogether survey. Survey findings indicated that the mental health of refugees and migrants during the COVID-19 pandemic was significantly impacted, particularly for certain subgroups, (i.e., insecure housing situation and residence status, older respondents, and females) who reported experiencing higher levels of increased discrimination and increases in daily life stressors. There is a need to recognize the detrimental mental health impact of the COVID-19 pandemic on particular refugee and migrant groups and to develop interventions that target their unique needs.