I recently swapped out my iPhone 4S for a Galaxy S3 and I was a bit apprehensive given my pretty mixed impressions of Android from a few years ago.
But really I prefer the S3 in almost every way now and wonder why I put up with the limitations of the iPhone for so long. If you do switch I think you'll be happy.
The 4S does feel a little more solid than the S3 too, but I'm happy not to have the extra weight in my pocket.
I've also seen a lot of iPhones with big cracks in the glass in the back and I'm happy I can just swap out the plastic back of the S3 with a cheap replacement if necessary.
I've seen 3GS practically shatter from a drop of 3ft. I've seen S2 flying all across the room on the concrete floor and not a scratch.
I'm not trying to draw statistics here, it's just my experience that iPhones are more resistant to wear and tear, and Samsungs more resistant to somewhat more extreme shocks.
But really I prefer the S3 in almost every way now and wonder why I put up with the limitations of the iPhone for so long. If you do switch I think you'll be happy.