Line two 8 x 8-inch pans with parchment, leaving a generous overhang on two sides, then spray with cooking spray.
In a large, deep pan, melt butter over low heat, add marshmallows, and stir until melted.
Stir in puffed rice cereal until evenly incorporated.
Divide the mixture into two pans, pressing into two even layers, then cool completely.
Let the ice cream soften slightly, then spoon on top of the puffed rice in the first baking pan, spreading evenly to edges.
Remove the puffed rice from the second baking pan, using parchment overhangs as handles. Peel off parchment and place the puffed rice layer on top of ice cream, gently pressing down. Place in the freezer until hardened for about 90 minutes or overnight.
Remove the frozen layers from the pan, peel off the remaining parchment, and place on a cutting board. Cut into 9 equal sandwiches, then return to the freezer.
Melt chocolate chips in the microwave in a shallow bowl in 30-second increments until smooth.
Dip one end of the treats in melted chocolate then place them back in the freezer for a few minutes to harden.