By historical accounts, James (and several brothers) were the product of a rather scandalous relationship between James, 3rd Earl of Ormond, and his sister's daughter Katherine of Desmond, who had left her own childhood home after accusations of incest with her brother and taken refuge in the home of the 3rd Earl of Ormond, her maternal uncle.
James “Gallda” Butler (from Irish gallda, meaning "alien or Englishman") (died 1434) was the son of James Butler, 3rd Earl of Ormond and Catherine FitzGerald of Desmond. From him springs the Cahir branch of the Butler family who were ennobled as Barons Cahir.
He married a daughter of MacWalter[2] and had issue James Óge Butler, who died in 1448[3] leaving Piers, his heir, and Richard.
"Cahir Castle:James Butler, 3rd Earl of Ormond, was enfeoffed of the manor of Cahir in 1375, and at his death in 1405 he was succeeded in that part of this vast possessions by one of his natural sons, James, the issue of his liason with his mistress and niece Lady Katherine Fitzgerald, daughter of the 3rd Earl of Desmond. This son, Seamus Gallda ("James the Foreigner"), ancestor of the Barons of Cahir, made Cahir his seat early in the 15th century. It was he who built the castle, the largest of that date in the county."
James' son Piers was head of the Cahir branch, with descendant Thomas Butler (1510-1558) being ennobled as 1st Baron Cahir and taking up residence in Cahir Castle following a long dispute with the Earls of Ormond over possession of the castle.[5]
The Peerage website doesn't quite get this James' placement and timeline correct, but it does have a citation that might help validate the other sources(?):
Per The Peerage's read of the cited reference, this James Gallda Butler's mother was Katherine (FitzGerald) Butler instead of Anne Welles, which stands to reason since the James 3rd Earl of Ormond had another son, James 4th Earl of Ormond (by Anne Welles): it would be unlikely that Anne would have two sons named James.
Per The Peerage's read of the cited reference, this James Gallda Butler's mother was Katherine (FitzGerald) Butler instead of Anne Welles, which stands to reason since the James 3rd Earl of Ormond had another son, James 4th Earl of Ormond (by Anne Welles): it would be unlikely that Anne would have two sons named James.