A machine learning method with extra-gradient step

Abstract

This paper deals with the minimization of unconstrained objective functions in the form of finite sums. We present an extra-gradient method with line search strategy and algorithm that uses variable sample size and thus makes the process significantly cheaper. The method is non-monotone, and the adaptive step size αk obtained in the linear search, is a random variable dependent on the sample ξk. The inevitable consequence is that the errors do not induce martingales. The algorithm is tested on a couple of examples, including the machine learning problems. [ 1, 2]