Dalarna University's logo and link to the university's website

du.sePublications
Planned maintenance
A system upgrade is planned for 10/12-2024, at 12:00-13:00. During this time DiVA will be unavailable.
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • chicago-author-date
  • chicago-note-bibliography
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
The Role Weights Play in Catastrophic Forgetting
Dalarna University, School of Information and Engineering, Microdata Analysis.ORCID iD: 0000-0002-4872-1961
2021 (English)In: 2021 8TH INTERNATIONAL CONFERENCE ON SOFT COMPUTING & MACHINE INTELLIGENCE (ISCMI 2021), IEEE, 2021, p. 160-166Conference paper, Published paper (Refereed)
Abstract [en]

Catastrophic forgetting is the sudden loss of performance when a neural network is trained on a new task or when experiencing unbalanced training. It often limits the ability of neural networks to learn new tasks. Previous work focused on the training data by changing the training regime, balancing the data, or replaying previous training episodes. Other methods used selective training to either allocate portions of the network to individual tasks or otherwise preserve prior task expertise. However, those approaches assume that network attractors are finely tuned, and even small changes to the weights cause misclassification. This fine-tuning is also believed to happen during overfitting and can be addressed with regularization. This paper introduces a method that quantifies how individual weights contribute to different tasks independent of weight strengths or previous training gradients. Applying this method reveals that backpropagation recruits all weights to contribute to a new task and that single weights may be somewhat more robust to noise than previously assumed.

Place, publisher, year, edition, pages
IEEE, 2021. p. 160-166
Series
International Conference on Soft Computing & Machine Intelligence ISCMI, ISSN 2640-0154
Keywords [en]
recurrent neural network, catastrophic forgetting, perturbation analysis, weights
National Category
Computer and Information Sciences
Identifiers
URN: urn:nbn:se:du-39582DOI: 10.1109/ISCMI53840.2021.9654815ISI: 000750613100028Scopus ID: 2-s2.0-85124387311ISBN: 978-1-7281-8683-2 (print)OAI: oai:DiVA.org:du-39582DiVA, id: diva2:1638554
Conference
8th International Conference on Soft Computing & Machine Intelligence (ISCMI), NOV 26-27, 2021, Cairo, EGYPT
Available from: 2022-02-17 Created: 2022-02-17 Last updated: 2023-04-14Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textScopus

Authority records

Hintze, Arend

Search in DiVA

By author/editor
Hintze, Arend
By organisation
Microdata Analysis
Computer and Information Sciences

Search outside of DiVA

GoogleGoogle Scholar

doi
isbn
urn-nbn

Altmetric score

doi
isbn
urn-nbn
Total: 120 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • chicago-author-date
  • chicago-note-bibliography
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf