Heat the oven to 350°F and line a 12-cup muffin pan with paper liners.
Mix the cake mix, oil, water, and eggs until smooth, then divide the batter among the liners.
Bake for 18 to 23 minutes, cool in the pan for 5 minutes, and finish cooling on a rack.
Mix the peanut butter with 1/4 cups powdered sugar, divide into 12 portions, and roll into balls.
Beat the butter, 2 1/4 cups powdered sugar, vanilla, and milk into a smooth frosting.
Cut a small center from each cooled cupcake, add the filling, replace the cake, frost, and top with a Mini Reese's cup.