Harper -- one of several African-American players being profiled by MLB.com during Black History Month -- had stolen 73 bases for the Seattle Pilots in 1969, a record that still stands for a franchise now known as the Milwaukee Brewers.
In other words, Harper had the ability to break Speaker's record. And he did, finishing with 54 stolen bases in 1973, a year he was voted the Most Valuable Player of the Red Sox.
Playing against the Brewers at Fenway Park on Sept. 29, 1973, Harper recorded his 53rd steal to pass Speaker.
"In '73, it was an accomplishment," said Harper, who was inducted into the Red Sox's Hall of Fame in 2010. "I ended up winning the [team] MVP, but I didn't start out that great. I can remember at one point I was on the bench. I was on the bench for quite a while, because I wasn't hitting. I got back in there, and all of a sudden, I found the stroke and hit .281. But it was a year in '73, I don't think anybody else on the team hit. I think [Carl Yastrzemski] had one of his worst years."
But Harper was able to create some excitement with his legs for a 89-73 team that finished eight games out of first place.
"They stopped the game and gave me the base. The umpire was the late Bill Kunkel," Harper said. "They stopped the game, gave me the base and everything. It was a big deal. It wasn't a big deal to me at the time."
It became a bigger deal in the decades that followed. Harper's record stood until 2009, when Jacoby Ellsbury shattered it with 70 steals.
"When Jacoby broke the record, I was glad for him," said Harper. "I know I was one of the best basestealers in my time. That doesn't change just because somebody breaks a record. It doesn't change the way I feel about the way I played the game or the type of player I was."
While 54 stolen bases and a team MVP honor might sound like 1973 was the best of times for Harper, that wasn't exactly true.
The Red Sox were the last team to break baseball's color line in 1959, and black players were not made to feel welcome by the Fenway faithful at that time.
When Harper arrived in Boston in 1972, things hadn't improved all that much.
"The racial climate was tough. The three years I was here as a player, it was tough," Harper said. "I've seen clips of [African-American] people interviewed, and they said they feel more comfortable in the ballpark now. People don't stare at them. In the '70s, it wasn't like that. You didn't see any black fans. They made it uncomfortable for African-American fans.
"Sometimes people think it's an exaggeration from the fans who said, 'I didn't feel comfortable, I didn't feel welcome at Fenway Park.' I'm going to tell you the truth. They were right. How could they feel comfortable when I wasn't? And I was on the field. I know what they were talking about."
Make no mistake about it. Harper heard plenty of racial slurs directed at him while playing at Fenway Park.
Unlike teammate Reggie Smith, Harper just didn't say a whole lot about it publicly.
"Reggie and I heard the same things [from fans] throughout the three years I was there," Harper said. "Reggie's a different personality than I was. Reggie would speak out. Reggie, he couldn't keep it within. Everybody would say, 'Tommy didn't say anything.' I didn't say anything, but that doesn't mean it didn't happen.
"With the fans now, you hear some people say, 'Those fans were exaggerating.' It was bad. I'm going to tell you the truth. It was bad. I lived it in 1972 and '73 and '74. I hear fans, people who have been in Boston, African-American fans who have been in Boston long before I got here, and they would say, 'You know what? I wouldn't go to Fenway Park.' That's why you didn't see any African-American fans.
"Then when I got there as a player, I could understand why. I said, 'Damn, they treat you terribly here.' You just played. You played. You do what you have to do. As a professional, I went to Fenway Park, because it was part of the American League and there was a reserve clause. Where else was I going to go?
"You persevere. The three years I played here, I just went to the ballpark. I went to work, and when I got through with work, I got out of there. You know? It's a lot different than now."
Despite some of his hardships as a player, Harper remains in the Boston area, living in the suburb of Sharon. He still does some work for the Red Sox.
"People ask me this a lot: 'Why did you come back to the Red Sox? Why did you stay [in the Boston area]?'" Harper said. "Well, where was I going? I didn't feel like running from every situation you come in contact with. If an African-American person did that in the '60s, he'd be moving all the time. It not only went on in the South, but people were surprised about Boston, the cradle of democracy and all that, a northern city.
"So where were you going to run? People in Texas had [race issues]. Everywhere in the country you were going to face something. People say, 'Why did you stay in Boston?' Where did you want me to go? South Africa?"
When the group led by John Henry, Tom Werner and Larry Lucchino took over control of the Red Sox in 2002, they vowed to improve the club's racial relations and try to make up for what went on before they got there.
Now, Pumpsie Green, who along with the late Earl Wilson broke the Red Sox's color line in 1959, comes back to Fenway for team events.
"How long has Pumpsie Green been coming back? Only since the new ownership took over," Harper said.