DC Field | Value | Language |
---|---|---|
dc.contributor.author | Jang, Jinhyeok | - |
dc.contributor.author | Yun, Woo-Han | - |
dc.contributor.author | Kim, Won Hwa | - |
dc.contributor.author | Yoon, Youngwoo | - |
dc.contributor.author | Kim, Jaehong | - |
dc.contributor.author | Lee, Jaeyeon | - |
dc.contributor.author | Han, ByungOk | - |
dc.date.accessioned | 2024-03-06T01:06:14Z | - |
dc.date.available | 2024-03-06T01:06:14Z | - |
dc.date.created | 2024-02-20 | - |
dc.date.issued | 2023-07-25 | - |
dc.identifier.uri | https://oasis.postech.ac.kr/handle/2014.oak/121291 | - |
dc.description.abstract | Recent complicated problems require large-scale datasets and complex model architectures, however, it is difficult to train such large networks due to high computational issues. Significant efforts have been made to make the training more efficient such as momentum, learning rate scheduling, weight regularization, and meta-learning. Based on our observations on 1) high correlation between past weights and future weights, 2) conditions for beneficial weight prediction, and 3) feasibility of weight prediction, we propose a more general framework by intermittently skipping a handful of epochs by periodically forecasting near future weights, i.e., a Weight Nowcaster Network (WNN). As an add-on module, WNN predicts the future weights to make the learning process faster regardless of tasks and architectures. Experimental results show that WNN can significantly save actual time cost for training with an additional marginal time to train WNN. We validate the generalization capability of WNN under various tasks, and demonstrate that it works well even for unseen tasks. The code and pre-trained model are available at https://github.com/jjh6297/WNN. | - |
dc.language | English | - |
dc.publisher | ML Research Press | - |
dc.relation.isPartOf | International Conference on Machine Learning (ICML) | - |
dc.relation.isPartOf | Proceedings of Machine Learning Research | - |
dc.title | Learning to Boost Training by Periodic Nowcasting Near Future Weights | - |
dc.type | Conference | - |
dc.type.rims | CONF | - |
dc.identifier.bibliographicCitation | International Conference on Machine Learning (ICML), pp.14730 - 14757 | - |
dc.citation.conferenceDate | 2023-07-23 | - |
dc.citation.conferencePlace | US | - |
dc.citation.endPage | 14757 | - |
dc.citation.startPage | 14730 | - |
dc.citation.title | International Conference on Machine Learning (ICML) | - |
dc.contributor.affiliatedAuthor | Kim, Won Hwa | - |
dc.description.journalClass | 1 | - |
dc.description.journalClass | 1 | - |
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.
library@postech.ac.kr Tel: 054-279-2548
Copyrights © by 2017 Pohang University of Science ad Technology All right reserved.