It was the 1,300th game for Penn (5-2, 4-0 Ivy League), which became the first program to reach that milestone.
Penn, which had 313 yards, held the Ivy League's top offense in check. Brown (4-3, 2-2), which averaged 413.8 yards and 28.5 points per game, had 304 yards against the Quakers.
Olson was 27 of 47 with four interceptions and 313 yards for Penn.
The Quakers opened the scoring on a fumble recovery by Luke DeLuca late in the first quarter.
The Bears answered with a 42-yard interception return for a touchdown by A.J. Cruz.
Brown's Kyle Newhall was 24 of 42 with 241 yards.