Your small or pinky toes do not play a significant role in the body's ability to maintain its balance, so I doubt that that is what is going on. However, if you are having significant problems with balance or coordination of your gait, I would suggest seeing your primary care doctor
to make sure something serious is not going on.
Some deformities of the foot can run in families, so it would be important for your doctor to take a look at your feet and make sure there is no obvious deformity. Furthermore, there are some very common nerve problems of the feet that can be subtle at first but worsen as you get older. Probably the most common of these is Charcot-Marie-Tooth, which is an inherited nerve problem which leads to weakness of the nerves in the feet, causing gradual deformity of the arches, hammer toes, and problems with walking.
Your doctor will be able to assess the anatomy of your feet, the strength of your muscles, and also your coordination and neurological status. If they uncover any abnormalities they may decide that you need to be referred to a neurologist
for further workup and treatment.