Ponsonby had been selected as the first recognised leader of the opposition, rather than leader of an opposition, when the two leading Whig peers Lord Grenville and Earl Grey, proposed him to Whig MPs. Ponsonby was described by Foorde as "a little-known mediocrity who was related to Lady Grey". He proved to be a weak leader, but was unwilling to resign and so retained the leadership of the party in the House of Commons until his death. He was succeeded as party leader by George Tierney.
He married Lady Mary Butler, the daughter of Brinsley Butler, 2nd Earl of Lanesborough and his wife Lady Jane Rochfort. He left an only daughter, Elizabeth, when he died in London on 8 July 1817, who went on to marry Francis Aldborough Prittie, MP, by whom she had six children.