Born in Statesville, North Carolina, Withers was a four-sport star at Greensboro High School in football, wrestling, baseball, and track. He was captain of the wrestling team and a two-time state champion, winning titles in 1939 and 1941 at 155 pounds. Withers excelled as a member of the Greensboro HS  football and baseball teams, both wining state championships. Withers went on to play football at Elon College in 1941 where he earned All-Conference Honors.  Withers joined the Air Force in 1942 where he served as a flight dispatcher during World War II. Returning home in 1946, Withers received a football scholarship to Guilford College where he was an All-Conference player on a conference championship football team. "Harpo," as he was known during his high school days, was inducted into the Guilford College Football Hall of Fame in 1983.

Graduating from Guilford College in 1949, Withers worked and coached at Gillespie Park Junior High in Greensboro before becoming the Recreation Director and head baseball, basketball, and swimming coach until 1954. He then worked at the Baptist Children’s Home in Thomasville until 1959. Withers then went to Thomasville High School where he was an assistant football coach, head track coach and head wrestling coach. To start the wrestling program at Thomasville, Withers reached out to Dr. Sam Barnes, wrestling coach at UNC-Chapel Hill, to borrow a used mat for the new program. While building a championship program, Withers held wrestling clinics for parents, developed a large fan base to support the team and traveled with his team as far away as New York and New Jersey for experience. In 11 seasons, Wither’s teams won three WNCHSAA Championships (1960, 1961, 1962), two District Championships (1961, 1962), and the North Carolina State Championship in 1961. His Thomasville teams went 92-37-2 from 1959 to 1968. Withers coached two NCHSAA state champions and 13 WNCHSAA champions. Stepping down as head coach in 1968, he continued assisting the Thomasville wrestling team until 1983.  He retired from Thomasville City Schools in 1986, having served as a teacher, coach, and principal for almost forty years. Withers was inducted into the Davidson County (NC) Sports Hall of Fame in 2009 and the North Carolina Chapter of the National Wrestling Hall of Fame in 2010.