Fraser was born in Minneapolis, Minnesota, to Everett and Lois Fraser, immigrants from Canada. His father studied law at Harvard, began teaching at George Washington University and became dean of the University of Minnesota Law School in 1920. Fraser graduated from University High School in 1941 and that year, he entered the University of Minnesota. During college, he was a member of the varsity swimming team.

Fraser graduated from University High School in 1941 and that year, he entered the   University of Minnesota. During college, he was a member of the varsity swimming team, and joined the US Navy ROTC,. 

He was placed on active duty in July 1942 and continued his naval studies on campus until February 1944, when he was commissioned an officer and sent to the Pacific Theater during World War II. Fraser worked as a radar officer into the peacetime that followed, ending in 1946. 

In June 1946 Fraser returned to Minneapolis to study law at the University of Minnesota Law School.

In 1954, Fraser was elected to the Minnesota Senate and served for eight years.

In 1962, he was elected to the House of Representatives from Minnesota's Fifth District. He served there in the 88th, 89th, 90th, 91st, 92nd, 93rd, 94th, and 95th congresses, from January 3, 1963 until January 3, 1979. Fraser is now best known for his work as the chair of the International Organizations and Movements subcommittee, a post he used to hold hearings on human rights violations in U.S. allies. As the historian Barbara Keys has shown, from 1973 to 1976, Fraser was a key leader in Congress in drafting legislation to reduce U.S. aid to countries whose governments engaged in a pattern of "gross violations of human rights." His efforts laid the foundations for much of Jimmy Carter's human rights agenda and transformed the way the U.S. Department of State operates, mandating that it write annual country reports on human rights and ensuring that diplomatic posts take note of human rights issues.[1][2] 

In 1978 he gave up his seat to run for the US Senate. He narrowly lost the 1978 Senate primary election to Bob Short, who then lost in the general election to David Durenberger.