Richard (died 1791), served as Dean of Exeter and succeeded as 3rd Baron Altham, married Dorothy, daughter of John Davey, of Ruxford, Devon. Richard's second son, Richard (died 1761), succeeded his cousin as the 6th earl, and left a son Arthur (1744–1816), whose legitimacy was doubted and his father's English titles were declared extinct. 
Dorothy, married Richard Power, 1st Earl of Tyrone (1630–1690) in 1654;
Elizabeth, married Alexander MacDonnell, 3rd Earl of Antrim (1615–1699);
Frances (died 1704/5), married firstly Francis Windham, of Felbrigg, Norfolk, and secondly (in 1668) John Thompson, 1st Baron Haversham;
Philippa (died 1714/5), married firstly Charles Mohun, 3rd Baron Mohun (their son was the duellist Charles, 4th Baron Mohun), and secondly William Coward, of Wells, Somerset;
Anne, married Sir Francis Wingate J.P. of Harlington Grange, Harlington. The latter arrested John Bunyan and committed him to prison. Bunyan was held overnight at Harlington Grange.
James' sons succeeded as the 3rd, 4th and 5th earls.