Chomsky (e.g. 2010) and others regard unlimited Merge as the defining feature of language, that cannot evolve gradually. The neural implementation of Merge is not well understood (Rizzi 2012, Zaccarella et al 2017), but must involve something functionally equivalent to pointers in working memory. Every Merge requires two pointers, and full syntactic trees may require dozens. Other syntactic paradigms also need pointers.
Humans do hierarchies in general better than chimpanzees. Any hierarchical thinking requires nested pointers in working memory, but they are neurologically expensive and degrade with depth (Crawford et al. 2016). Humans have larger working-memory capacity than chimpanzees, which has been proposed as key to human cognitive evolution (Read 2008, Coolidge & Wynn, 2005). Gradual evolutionary growth of pointer capacity will allow gradually increasing syntactic complexity, without saltations in the underlying computational machinery. Both depth degradation and pointer capacity naturally limit Merge even in modern humans, consistent with corpus data (e.g. Karlsson 2010).