I wouldn't worry too much about the difference in birthplace, different people probably talked to the census takers, but I would look for christening records in both towns. The records for the Tal-y-llyn church still exist (at the Powys Archives) so you might be able to find David in them if you can get there since they aren't digitized.
My second great-grandparents were married at the Tal-y-llyn chuch in 1847 and they brought a copy of their marriage record with them to America so I know their fathers' names, but it's no help to me because one was an Evans and the other a Jones, the two most common Welsh names, followed, of course, by Williams.
Some people were still using patronymics in the early 19th century, so your David's father might well have been actually named William something. It's hugely frustrating!