Hi Sarah! Welcome to WikiTree!
That is a tantalizing tale, for sure!
Family stories, in my experience, usually contain a kernel of truth, but might have come down to us slightly different from what actually happened - just like any other oral transmission.
You will probably have to do the research -- we can help -- documenting your parents and grandparents and seeing what comes up, and tracing all the known descendants of the Bingham family, and seeing what comes out of that.
Time frame wise, do you think this is the Lord Lucan? [[Bingham-122|Charles George (Bingham) Bingham Earl of Lucan 4th (1830-1914)]] Or the one before, or the one after? edit: earlier, i'd guess, based on the age of James O'Brien.
You might also look for the name Cardiff in census or church documents in proximity to the Lucan family.
Often the 'smoking gun' evidence does turn up, but not usually when going for it directly. Usually we have to assemble the whole picture in the way we would even if there isn't a family legend.