As far as gold panning for the gems-- and I'd have to double check this-- but I don't think you need more then GM fishing.
You should have a high end luck suit on, though. I have gold panned at 80+ fishing skill all the way to GM+ and I have never had an issue getting the large gems. About 20 a pan, minimum. Sure I could do better-- and perhaps Kane has changed the values of gold panning since I panned last-- but I'm pretty sure its luck and not your fishing skill. Besides when you gold pan you're after large gold nuggets... large gems are just a bonus.
Maybe I'm just lucky.
--Sturger--