After opening the third with a single to right, Kelly moved to second on a sacrifice bunt. With Cincinnati right-hander Matt Latos paying no attention to Kelly on a 3-0 pitch, he bolted for third and reached standing up.
It was quite a moment for the pitcher the Red Sox acquired along with Allen Craig in exchange for John Lackey on July 31.
It was the first time a Boston pitcher has stolen a base since Sept. 8, 1969, when Bill Landis pulled it off in Cleveland against the Indians. The last Red Sox pitcher to steal third base was Tom Brewer on July 30, 1959, against Cleveland.
"That's a pretty cool stat," Kelly said after the Red Sox's 3-2 win. "I figured since I was in the AL, I wasn't going to get many more chances. I always told my team [I could steal], but the Cardinals never let me run because I pulled my hammy. I figure this might be my only opportunity, and I at least wanted to get one in there and I saw the right opportunity and just took it."
The last pitcher in the Majors to steal third was the Cubs' Ted Lilly on July 23, 2008.
"That kind of [set] me off that he actually did that," said Latos. "It just made me want to make sure I hit my spots and really kick it up a notch."
Kelly did not wind up scoring after his stolen base on Tuesday, as David Ortiz popped up and Yoenis Cespedes lined out.
Back when Landis stole his base, the middle of Boston's batting order included Carl Yastrzemski, Reggie Smith, Tony Conigliaro, Rico Petrocelli and George Scott.
The last time the Reds allowed a stolen base to a pitcher was Aug. 30, 2010, when Randy Wolf did it for the Milwaukee Brewers.
Though Kelly is noted for having good speed for a pitcher, it was his first career stolen base.
What is his next goal as a hitter?
"Hit the ball as far as [Yoenis] Cespedes, but that's not going to happen," said Kelly, referring to his teammate's go-ahead moonshot Tuesday night. "Maybe put it in the air when the wind is blowing out and get a home run. I just try to stick to hitting singles and get on and just be a nuisance on the basepaths and act like another leadoff hitter and have pitchers think about me out there."