Deep learning has led to a new wave of AI with many significant achievements. Yet, neural networks used in deep learning lack the ability to update itself given new information (continual learning) and do not generalize in limited data conditions. This is in contrast to humans, who can integrate new information and generalize with a few examples. In this work, we take inspiration from the way human brains form memories during sleep to develop an algorithm for deep learning. The sleep-like state implemented during learning of deep neural networks improved continual learning, and generalization under low data and noisy conditions.