Well yes, of course size is a huge difference maker. Silva would outweigh GSP by approximately 20 lbs if they fought at a catch weight of 177 lbs. All things equal that in it self would be the difference maker.
Another equally important factor is that Silva is a better fighter, period. Look what he does to his opponents. GSP's last 5 fights have been Condit, Shields, Koscheck, Hardy and Alves. Whats a common trait?, he couldn't finish any of them. None of these fighters could even be mentioned in the same league as Anderson Silva and GSP went the distance with them all.
GSP would have a hard time taking Silva to the ground, in part because of the weight and strength difference and in part because Silva is very long and fast, he'd keep GSP away with his fists. Even if GSP got Silva to the ground he has no chance of finishing Silva. Don't forget Silva is a Jiu Jitsu black belt. He also has an excellent ground game.
I think this fight would be a mismatch.