Citizens’ trust of authorities is of general importance, as it can affect trust in society as a whole. The Swedish Transport Agency is a state administrative authority responsible for driving license withdrawal. If withdrawals are experienced as unfair by those affected, this might lead to decreased trust in the agency and in other actors involved. This study accordingly examines how the experiences of people who have had their driving license for a private car withdrawn due to visual field loss might affect their distrust of the authorities, as related to outcomes and processes. Follow-up interviews were conducted with nine people from a previous study. Content analysis revealed one overarching theme, struggling for justice within a system perceived as unfair, which comprised four identified categories: preparedness for and understanding of the withdrawn driving license (WDL); perceptions of assessment methods, reactions and actions regarding the decision, and perceptions of the authorities’ performance. In sum, the vision tests on which withdrawals are based were perceived as unfair, as they did not measure individual driving ability. Furthermore, rejections of appeals led to feelings of hopelessness. Distrust regarding outcomes and processes related to the withdrawals was also fueled by experiences of deficiencies regarding, for example, performance and information.