Do you have the death register for Joseph? There's a column on that page that asks how many years in the colony. I was thinking that this may show he was somewhere other than Victoria.
I had a quick look at the Pioneer Index and couldn't see his marriage. Do you have a copy of that to see if it tells you where he was in the colonies?
There's an 1856 Electoral Roll of Victoria on Ancestry. A Joseph Hunt is there at "The Loddon, Mandurang". He is No 2631 on this image, living at Kangaroo Flat, occupation miner, and he has a miner's right to vote.
If you can access the Victoria Police Gazettes you may find him mentioned there. They are on FindMyPast, I believe.
There is also the Victoria Government Gazette at http://gazette.slv.vic.gov.au/ where you may find such things as miner's leases, land transctions, etc.
I hope this helps.