University-Industry Cooperation Group of Kyung Hee University

Disclosed are a multimodal federated learning method and system in modality-missing environments. The multimodal federated learning system includes a server and a plurality of clients, each provided with a local model, a neural network structure of the local model provided by each of the clients is identical, a client type of each of the clients is categorized based on whether a modality of training data used to train the local model is missing, each of the clients generates a class-specific local prototype from data of the input modality and transmit the class-specific local prototype to the server, and the server aggregates class-specific local prototypes received from a plurality of clients of different client types, generates a class-specific complete prototype based on the aggregated class-specific local prototypes, and transmits the generated class-specific complete prototype to each of the clients.

Disclosed are a federated learning method and system that can reduce domain skew between clients. The federated learning system according to an embodiment of the present disclosure includes a plurality of clients and a server, and each client is configured to input local data into a feature extractor to extract a feature vector, input the feature vector into a classifier to perform classification for a class of the local data, and train a local model, which includes the feature extractor and the classifier, using a first loss function based on a difference between an output value of the classifier and a preset correct value and a second loss function for contrastive learning between the feature vector and the generalized prototype of each class.

Zusammenfassung wird geladen …

Zusammenfassung wird geladen …

Zusammenfassung wird geladen …

Zusammenfassung wird geladen …

Zusammenfassung wird geladen …

Zusammenfassung wird geladen …

Zusammenfassung wird geladen …

Zusammenfassung wird geladen …