You are definitely correct that there are conditions that involve a defect in the immune system, and people who suffer from these conditions tend to get frequent infections. However, in critical point here is that these infections tend to be rather severe, requiring hospitalization and aggressive treatment.
On the other hand, if mostly what you have experienced is just a few cases of the cold or the flu every winter, then this is unlikely to be related to a serious underlying problem with your immune system. In fact, most people get a few attacks of the common cold every year, and this is completely normal.
What I would suggest, however, is that you mention this concern to your primary care doctor
the next time you see them, for example at your next annual physical
examination. Your doctor can review the types of illness
you have gotten over the last few years and, if there is any concern that this might be related to an immune system defect, then they can certainly perform a work up to look for that and refer you to the necessary specialist for treatment.
Make an appointment with your primary care doctor at your earliest convenience!