Chomsky (e.g. 2010) and others regard unlimited Merge as the defining feature of language, that cannot evolve gradually. The neural implementation of Merge is not well understood (Rizzi 2012, Zaccarella et al 2017), but must involve something functionally equivalent to pointers in working memory. Every Merge requires two pointers, and full syntactic trees may require dozens. Other syntactic paradigms also need pointers.
Humans do hierarchies in general better than chimpanzees. Any hierarchical thinking requires nested pointers in working memory, but they are neurologically expensive and degrade with depth (Crawford et al. 2016). Humans have larger working-memory capacity than chimpanzees, which has been proposed as key to human cognitive evolution (Read 2008, Coolidge & Wynn, 2005). Gradual evolutionary growth of pointer capacity will allow gradually increasing syntactic complexity, without saltations in the underlying computational machinery. Both depth degradation and pointer capacity naturally limit Merge even in modern humans, consistent with corpus data (e.g. Karlsson 2010).
Chomsky, Noam. (2010). Some simple evo devo theses: how true might they be for language? In Richard K Larson, Viviane Déprez, & Hiroko Yamakido (Eds.), The Evolution of Human Language. Biolinguistic Perspectives. Cambridge: Cambridge University Press.
Coolidge, Frederick L & Wynn, Thomas (2005) Working memory, its executive functions, and the emergence of modern thinking. Cambridge Archaeological Journal 15:5-26.
Crawford, Eric & Gingerich, Matthew & Eliasmith, Chris (2016) Biologically plausible, human-scale knowledge representation. Cognitive Science 40:782-821.
Karlsson, Fred (2010) Syntactic recursion and iteration. In Harry van der Hulst, ed., Recursion and Human Language. Berlin/New York: Mouton de Gruyter,
Read, Dwight W (2008) Working memory: A cognitive limit to non-human primate recursive thinking prior to hominid evolution. Evolutionary Psychology 6:676-714.
Rizzi, Luigi (2012) Core linguistic computations: How are they expressed in the mind/brain? Journal of Neurolinguistics 25:489-499.
Zaccarella et al (2017) Building by syntax: the neural basis of minimal linguistic structures. Cerebral Cortex 27:411-421.
2019.