Men are deluded, because they accept as separate entities the three modes of nature, besides God. If they accept these as Divine manifestation, they cannot be deluded.
Delusion arises, out of either of the assumptions, regard the body as yours, or accept yourself as the body. Acceptance of the body as yours, is 'mineness' while acceptance that the self, is body is egoism. This Egoism and 'mineness', is called, delusion. It is because of delusion, that a man cannot know, the reality about God. It is a rule, that a man can know the world by isolating himself from it, and he can know God by identifying himself with Him, because, he is totally different from the body, and one with God.
Our affinity with God, is natural because we are His eternal fragment, but our affinity with body and the world, is man-made and artificial. If we accept this artificial affinity as real, that is our delusion, and it obstructs in realizing real affinity.
Now, there is a question, whether first man turned away from God, or got deluded by the three modes of nature. Philosophers are of the opinion, that both of them have been there from times immemorial, they are without beginning. Therefore it cannot be said which of these is former and which is latter. Man should make the right use of freedom given to him, by God. By doing so, he can transcend the three modes of Nature, and be free from the cycle of birth and death. This proves, that by misusing his freedom, and being entangled in sensual and ephemeral pleasures, he cannot realize God
By the term 'Paramavyayam', Lord Krsna says, that He is above the threefold modes of nature i.e., He is unconnected with and free from them, there is no modification in Him. Moreover, He is imperishable. But the man, deluded by modes of nature cannot know His real self.