There are a few possible causes of this type of pain.
One would be an inguinal hernia
. A hernia occurs when abdominal contents (fat or intestines) bulge through a defect in the abdominal wall, usually in a downward direction toward the scrotum. If the hernia is large, you should notice a bulge in or above the right side of your scrotum when you are lifting heavy weights or bearing down. However, if the hernia is small you might not notice this, but you might nevertheless experience intermittent discomfort just like you describe, especially with activity. Your primary care doctor
should be able to determine whether or not you have a hernia with a simple physical examination.
Another possibility would simply be muscle sprain or strain, which can occur with physical activity. This should be a soreness closely associated with exercise and usually occurring more in the muscles of the thigh and less experienced as pain in the scrotum or pubic area.
There are rare, serious causes of groin pain as well. For example, a testicular torsion, or twisting of the testicle, is very serious, but usually presents as severe, sudden pain and would not come and go in the mild fashion that you describe.
You should see your primary care doctor to evaluate this symptom.