Preheat your oven to 350°F (175°C). Grease a baking dish with butter or cooking spray.
Spread one side of each hot cross bun slice with butter.
Arrange half of the bun slices, buttered side up, in the prepared baking dish. Sprinkle half of the raisins or dried fruits over the buns. Top with the remaining bun slices and sprinkle the rest of the raisins or dried fruits on top.
In a medium bowl, whisk together the milk, eggs, sugar, and vanilla extract until well combined.
Pour the milk mixture evenly over the bun slices in the baking dish. Press down gently to ensure all the bread is soaked in the liquid. Let it sit for about 10-15 minutes to allow the bread to absorb the liquid.
If desired, sprinkle ground cinnamon over the top of the pudding for extra flavor.
Place the baking dish in the preheated oven and bake for 35-40 minutes, or until the pudding is golden brown and set in the middle.
Once baked, remove the pudding from the oven and let it cool slightly before serving.
Serve warm, optionally with a dollop of whipped cream or custard on top.