It is safe to charge any Mac (including those with Apple Silicon) with the 96W (or lower/higher) USB-C charger.
Downsides:
- Slower overall charging
- No “fast” charging.
Apple: About Mac power adapters:
If your Mac uses USB-C to charge, you can charge your Mac notebook
with any USB-C power adapter or display. For the best charging
experience, you should use a power adapter or display that provides at
least the minimum wattage of the power adapter included with your
MacBook Air, MacBook Pro, or MacBook.
Apple: Fast charge your 14-inch or 16-inch MacBook Pro:
The MacBook Air (M2, 2022), MacBook Pro (14-inch, 2021), and MacBook
Pro (16-inch, 2021) support fast charge. When you connect the
appropriate power source and cable, you can quickly charge your
battery up to 50 percent in 30 minutes.