Colmer was elected Jackson County attorney in 1921, becoming district attorney in 1928.
In 1932, Colmer was elected to the House of Representatives as a Democrat from Mississippi's 6th District, located on the Gulf Coast. He was reelected 19 times. His district was renumbered the 5th after the 1960 Census, when Mississippi's declining proportion of the US population due to the Great Migration cost it a congressional seat.
Colmer did not run for reelection in 1972. He endorsed his administrative assistant, Trent Lott, as his successor, although Lott ran as a Republican. Colmer served longer in the U.S. House of Representatives than anyone in Mississippi's history except Jamie Whitten, who served 54 years in Congress from 1941 to 1995. (Fellow Mississippian John Stennis would serve over 41 years in the U.S. Senate from late 1947 until early 1989.)