Just because he's viscous toward you/others, doesn't mean that he will be toward his offspring. I don't understand why you prevented him from seeing the birth, and just being there. ALL tenured members of a pack, (father/brother/sister/all) support and defend "their" puppies, will even give up their food/water to/for them. IF it's mother is mistrusting of any pack member, she will take care of that problem for you...trust me on that one !!!
I'd let the Sire have access to the pups and mom, but just in case you've created a situation where there is going to be a problem, monitor him/them closely... for a while anyway.