Brown was drafted in the 2nd round of the 1961 draft by Chicago. The Vikings obtained him a year later through trade and he remained with the team until his retirement (1962-74). As a Viking, he holds the team record for rushing attempts with 1,627 and ranks 3rd in team history with 5,757 career rushing yards. He has played in more games than any Vikings running back with 180 games (111 games started) and is tied with Chuck Foreman for a team record with 52 career rushing TDs.

Brown played in Super Bowls IV, VII, and IX and was selected to 4 Pro Bowls (1964, '65, '67, and '68) throughout his career. The final game of his career was Super Bowl IX vs. Pittsburgh. He was inducted into the Vikings Ring of Honor on September 26, 2004.