Victorino, who hit leadoff and played right field, was 0-for-4.
Middlebrooks, who's been on Boston's disabled list since May 17 with a fractured right index finger, was idle for the second straight day.
"It's still a little swollen," Middlebrooks said of his finger, which he barely could straighten out. "Throwing is pretty difficult. Swinging the bat, I can feel it. It's sore, but it's doable. I've been swinging the bat fine.
"We've haven't taken any steps back. I've just been pretty sore the last couple of days. The two days off is more or less to give it a breather. There aren't any setbacks. I just had some swelling and soreness which was to be expected from getting back to playing third base."
Victorino commenced his second rehab stint Monday at Class A Lowell after being sidelined by a lower-back injury which followed a strained hamstring muscle suffered during Spring Training.
"Everything went well, "Victorino said referring to Monday's game. "The body felt good. There weren't any setbacks today. There wasn't any pain or any soreness. We're going to go tonight. I'm scheduled for three or four at-bats. But I told them if my body feels good I'd like to play nine."
As it turned out, Victorino only played six innings and only had to make one play in the field -- an errant throw to third base during a three-run Norfolk fifth.
Victorino hit a check-swing bouncer to pitcher Josh Stinson in the first; flied out deep to left field in the second; looked at a third strike by lefty Randy Wolf in the third; and grounded out to short in the sixth.
"This was a setback," he said of his back injury. "It kind of came out of nowhere. But our goal was to get back on the West Coast. One part of that trip is out of the way and the second one's probably gone, too. But who knows? I'm trying to push the limits a little bit and trying to get there [Wednesday]. But I think from a medical standpoint, we're looking at the big picture. Flying all the way out west and playing in a game and flying all the way back to play in New York probably isn't such a smart thing to do.
"But I'm pushing it. I'm trying to get them to give me the OK to go. But like I said, I've got to get through a game first. I haven't played nine innings. Before I worry about getting on a flight to Seattle and playing nine innings in a big league game, I have to show I can play nine innings here -- even on back-to-back days. I think that's what we're focused on now."
What Victorino isn't focused on is the criticism he's received.
"As people say, you're made of glass or tissue paper," he said. "I take it as a positive criticism and that just motivates me to want to get back faster. I don't take it in a negative way if somebody says that.
"Injuries are going to happen. You only can prepare and keep yourself in the right frame of mind and in the right shape."
When Victorino rejoins Boston, he realizes he must think with his head instead of his heart.
"I think the thing for us is to understand when I get back my ego's going to tell me to go every day," he said. "But there has to be an understanding that we want to make sure that I play for a good part. It's not about coming back and playing every day and the next thing you know I have a setback."
Whether Victorino will require another rehab game will be determined after an evaluation by Boston management and the team's medical staff.