Have you tried doing your own raw - or are you always going with premade? Maybe there is something in the premade stuff that is having an odd effect?
Given your puppers' stool is completely normal (except slipping out at inappropriate times!) doesn't seem to lead to the fish oil. Normally too much fish oil will cause diarrhoea. You could always try cutting back on it and see if it makes a difference I suppose.
Does your vet support raw feeding? If they don't, you may not get much help from them except the 'change back to kibble' talk