Weight space structure and the storage capacity of a fully connected committee machine

Abstract

We study the storage capacity of a fully connected committee machine with a large number K of hidden nodes. The storage capacity is obtained by analyzing the geometrical structure of the weight space related to the internal representation. By examining the asymptotic behavior of order parameters in the limit of large K, the storage capacity alpha(c) is found to be proportional to K root lnK up to the leading order. This result satisfies the mathematical bound given by Mitchison and Durbin [Biol. Cybern. 60, 345 (1989)], whereas the replica-symmetric solution in a conventional Gardner approach [Europhys. Lett. 41, 481 (1987); J. Phys. A 21, 257 (1988)] violates this bound.