Unfortunately, we now know that many patients who had ADHD as children continue to have symptoms into adulthood. This seems to be your case, and I am glad that you have been able to continue treatment for your symptoms.
It certainly does sound like you are having something like a 'wear-off' effect of the Vyvanse in the afternoon. However, this is less typical than with some of the other ADHD medicines, as Vyvanse is supposed to be a once a day, long acting medicine (which theoretically shouldn't have this wear-off effect). It is of course possible that the dose of the medicine is just too low and needs to be adjusted.
It is also possible that something else is going on. I say this because when the ADHD medicines wear off, the principle symptom should be difficulty concentrating, not fatigue per se. For example, if you find that you are most fatigue shortly after eating lunch, this may in fact not be a problem of the medicine but rather of the insulin surge that you are having with lunch. A brief amount of exercise after lunch and limiting carbohydrate intake might fix this.
Regardless, you will need to set up a visit with your psychiatrist
to discuss your medication dosing and see if a change needs to be made.