Lipomas are benign growths of fatty tissue that develop under the skin. Sometimes there is a genetic disposition to them, and often people who have one lipoma will develop others as well. Fortunately, there is virtually no danger of a lipoma turning into cancer, so often they can be left alone, as your previous doctors
The main reasons for removing a lipoma is for cosmetic reasons, for example if the appearance of it bothers you. Also, sometimes lipomas continue to grow and can become uncomfortable because they brush against clothing or are easily knocked on furniture and other objects; this would be another reason to consider removal.
Generally, removal of the lipoma is quite straightforward. Usually, it can be done in the doctor's office with just some local numbing medication injected into the skin at the site. Many doctors are capable of taking off lipomas, however if the lipoma is located in a sensitive area, such as close to blood vessels or delicate structures (as you were told by your doctor), then removal may require referral to a surgeon
If you have decided that removal is the right option for you, then you should talk to your primary care doctor
who may perform the removal themselves or may make a referral for you.