Tips for finishing Class Picnic Shorts with an overlocker/ serger

I have made a few pairs of these shorts prior to having a serger, but they always seemed to fall apart during washing. Now that I have a serger, I thought I would try again. I am having trouble in a few spots, though... 1. The point where the faux fly meets the center rise seam. 2. The two inner thigh side seams that are cut on a diagonal and sewn on the wrong side then turned and sewn on the right side. I pulled the tails of the serger thread back through the stitching, but in both places the serger misses some of the fabric and I am worried it may fray over time. I did use some fray check in hopes to minimize. Should I be finishing my fabric edges before sewing to help? Any suggestions are welcome!