The striker was then booked for the second time in three days for celebrating a goal by removing his shirt.
The former Modena star, who also hit the winner at Tranmere in the last round, was unlucky not to open the scoring as early as the second minute.
Kamara danced his way past Leeds midfielder Matt Spring before unleashing a fierce right-foot drive, which was beaten away at the post by Leeds keeper Neil Sullivan.
The former Scotland shot-stopper then produced a fine one-handed save to turn Andy Griffin's long-range effort over the bar five minutes later.
Returning to Fratton Park, where Leeds were thrashed 6-1 last season, United were missing skipper Paul Butler and Matthew Kilgallon, both banned for red cards against Brighton, the defence was once again also without Stephen Crainey.
In midfield, the injured Jermaine Wright was also still absent.
Sullivan was powerless to prevent Kamara opening the scoring, when the striker took the ball past two Leeds defenders before calming rounding the keeper and firing home from close range.
Linvoy Primus was lucky not to earn another yellow card for Portsmouth with a blatant late tackle on Julian Joachim but then they doubled their lead thanks to an Eyal Berkovic penalty 18 minutes later.
Kelly had carelessly tripped debut-making Valery Mezague as the Cameroon midfielder moved into the area and Berkovic stepped up to send Sullivan the wrong way and drive the resulting spot-kick into the bottom left-hand corner.
Striker Brian Deane pulled Leeds back into the match with a trademark header six minutes before the interval.
Kelly atoned for his penalty error by crossing from the right for Deane to plant a bullet header past Portsmouth keeper Jamie Ashdown from 12-yards out.
Aliou Cisse squandered a great chance to make the game safe for Pompey shortly after the break when he fired straight at the keeper following good work from Kamara.
Deane then saw a second effort ruled out for offside after the break and Frazer Richardson fired a 25-yard curler just wide as The Whites searched for an equaliser.
Pompey boss Harry Redknapp sent on Patrik Berger and the Czech Republic international took control of the game in the closing stages.
Berger struck four shots in the 21 minutes he was on field and he should have netted in the final minute only to mis-hit a shot, but it mattered little as the Premiership team held on to win.