Teachers are responsible for teaching, and when they plan, they are part of a complex non-linear social practice of curriculum making. When planning, teachers draw on curriculum materials, which are often designed to promote reform; however, previous studies show that this is not always the case. A study on planning for mathematics teaching in Sweden showed that teachers are influenced by those who have formal and informal power when they plan. This article focuses on structural influence on teachers' planning as curriculum work and the possible consequences for teacher agency. The data comes from a focus group discussion about planning with mathematics teachers in primary school. Tools for the analysis were inspired by actor analysis and “three pillars of institutions”. Results show that the structural influence on teachers is at the regulative, the normative, and the cultural-cognitive levels. Regulative influence seems to be short-term, while cultural-cognitive influence seems to last over the long-term. The most prominent influence was normative and related to the textbook. Results shed light on teachers' curriculum work in the micro site of activity when they are planning for mathematics teaching. Based on the results, the article suggests that teachers—as key actors when it comes to teaching—must have the opportunity to discuss and problematize assumptions, ideas that are taken-for-granted, and power relations in curriculum making to achieve agency and be the professional agents of change they are expected to be.