Environmental factors have been suspected to influence the propagation and lethality of COVID-19 in the global population. However, most of the studies have been limited to correlation analyses and did not use specific methods to address the dynamic of the causal relationship between the virus and its external drivers. This work focuses on inferring and understanding the causal effect of critical air pollutants and meteorological parameters on COVID-19 by using an Empirical Dynamic Modeling approach called Convergent Cross Mapping. This technique allowed us to identify the time-delayed causation and the sign of interactions. Considering its remarkable urban environment and mortality rate during the pandemic, Quito, Ecuador, was chosen as a case study. Our results show that both urban air pollution and meteorology have a causal impact on COVID-19. Even if the strength and the sign of the causality vary over time, a general trend can be drawn. NO2, SO2, CO and PM2.5 have a positive causation for COVID-19 infections (ρ > 0.35 and ∂ > 9.1). Contrary to current knowledge, this study shows a rapid effect of pollution on COVID-19 cases (1 < lag days <24) and a negative impact of O3 on COVID-19-related deaths (ρ = 0.53 and ∂ = -0.3). Regarding the meteorology, temperature (ρ = 0.24 and ∂ = -0.4) and wind speed (ρ = 0.34 and ∂ = -3.9) tend to mitigate the epidemiological consequences of SARS-CoV-2, whereas relative humidity seems to increase the excess deaths (ρ = 0.4 and ∂ = 0.05). A causal network is proposed to synthesize the interactions between the studied variables and to provide a simple model to support the management of coronavirus outbreaks.