The Croatian midfielder has been linked with a transfer to the Liga champions for most of the transfer window and appears to closing in a move to the Bernabeu.
Defoe feels that it would be difficult for Spurs to lose a player of the 26-year-old's calibre but feels that it would be a hard move to turn down.
|VEYSEY ON RVP TO MAN UTD
|8/1||Newcastle are 8/1 with Paddy Power to beat Tottenham 2-1
"He's been at the club - and he's been good for the club to be honest. If he does end up going, for us it is hard to lose someone like that.
"But, for him, I would say good luck to him because he's a fantastic player."
The 29-year-old scored the winning goal in England's 2-1 friendly victory over Italy on Wednesday night and hopes to replicate that form for his club.
"It was nice to get on and score. It is still early days with the sharpness and my fitness," Defoe added.
"It is fantastic to not only play for your country but, as a forward, to score is the best feeling in the world.
"The new season has come around quickly. I had a quick break in the summer because of the European Championships.
"But I always look forward to the new season and getting off to a good start. I do feel sharp and I'm looking forward to it."
Defoe also hailed the impact of the younger players in Roy Hodgson's squad, particularly that of his Spurs team-mate, Jake Livermore.
He enthused: "I think the younger lads that came in did well. Jake Livermore came on and did fantastic I thought.
"Especially playing that system [4-3-3], it was important for the midfielders to be brave and get on the ball."