The Red Sox grabbed a 1-0 lead against Mike Pelfrey in the first inning and then scored four more in the second, the biggest blow coming on Middlebrooks' three-run double.
"Luckily the wind was blowing in. Otherwise, that might have been a grand slam," Pelfrey joked.
Dempster, who signed with Boston as a free agent in December, allowed just one run, three hits, recorded three strikeouts and allowed two walks. He has now allowed three runs in four starts covering 12 2/3 innings this spring.
The right-hander pitched his way out of a jam in the second inning, as Minnesota's Chris Parmelee worked a one-out walk and then took third on Joe Benson's double to left field. Dempster battled back and struck out Oswaldo Arcia and Eric Fryer to end the threat.
"When you go away from the season and the season ends, you lose sight of those situations a little bit, so when you get back into 'em, it kind of gives you an understanding of trying to manage an inning and how to maybe get a strikeout when you can," Dempster said.
Parmelee, who robbed the Orioles' Ryan Flaherty of a home run on Thursday night when he returned to the lineup in right field after missing nine games with a tight right groin, replaced Justin Morneau at first base on Thursday and homered in the fourth.
Up next: The Red Sox will be involved in split-squad action on Friday. At 1:05 p.m. ET, Clayton Mortensen will lead one Boston club against the Orioles in Sarasota. Felix Doubront will then take the JetBlue Park mound at 7:05 p.m. in an MLB.TV contest against the Twins.
Jim Hawkins is a contributor to MLB.com. This story was not subject to the approval of Major League Baseball or its clubs.