The Belgium international completed a much-anticipated switch to Spain last summer following seasons of speculation linking Hazard with a move to the La Liga giants.
He has struggled to make appreciable impacts in the preseason and the league, where he made his debut against Levante last weekend.
Besides, the Belgian forward produced a below-par performance in the French capital on his European debut for Real Madrid as Real Madrid lost 3-0 to Paris Saint Germain (PSG).
However, Ferdinand insisted that the Belgian star should have remained at Stamford than joined the Galacticos.
“When you go to a new club, especially the size of Real Madrid, you have got to earn the right to be the man that gets the ball, that they look to first as the talisman,” Ferdinand told BT Sport.
“He has got to do something in the training games and in the games itself, to make them want to give him the ball.
“I’m fearful of him really that he has gone to Real Madrid at the wrong time.
“This is an ageing side, they are not on the way up and they have not got much spark about them at the moment, and that is going to reflect on him, badly, in the long term.”