In a small saucepan, heat milk on medium heat until it simmers. Pour into a large mug and add the chamomile tea bags. Let steep for 4-8 minutes. Discard tea bags. Allow the milk to cool to room temperature.
While the milk is cooling, preheat the oven to 350°F. Line a 12-count standard sized muffin pan with cupcake liners.
Using an electric mixer, beat butter and sugar together in a large bowl on medium high for 30-45 seconds, until creamed together. Use a rubber spatula to scrape down the sides of the bowl to make sure it is well mixed.
Add eggs, vanilla extract, and lemon essential oil or lemon juice. Mix on medium-high for about 30 seconds, until the ingredients are combined.
In a separate medium bowl, whisk together the flour, baking powder, baking soda, and salt. Add the dry ingredients into the wet ingredients. Mix on low speed until incorporated, about 15 seconds. Do not overmix.
Add the cooled milk mixture to the batter. Mix on low speed just until incorporated.
Spoon the batter into the 12 cupcake liners. Clean any drips of batter from the pan to prevent burning.
Bake for 17 minutes. The cupcakes are done when a baking thermometer inserted in the center turns bright red, or a toothpick comes out clean. Remove pan from oven and let cool in the pan for 5 minutes. Then move the cupcakes out of the pan and onto a cooling rack.