Pali was originally just a regular language spoken in the towns and villages of some area of northern India at the time of the Buddha. We’re not sure precisely which area that would have been. It was just one of many languages in which the teachings of the Buddha were communicated and memorized.
Originally neither language had a written form and when, after the passage of many centuries, scriptures in both languages were committed to writing there was a large variety of scripts available. India is a huge country, and many forms of script had evolved in the different regions into which these languages had spread. In addition, the Pali oral teachings had been carried well outside of India and were therefore written in Burmese, Thai, and Sinhalese scripts, not to mention Mongolian and of course Devanagari back home in India. There are even carvings of Pali texts in Aramaic, the language of Christ. Then of course Buddhist teachings were translated into Tibetan, Chinese, and Japanese.
So to cut a long story short, there isn’t one form of writing in which mantras are written!
A good resource for seeing the mantras from this site in various scripts, but principally the ancient India Siddham script is Visible Mantra. They even have examples in Klingon and Elvish!
Here’s an example of the Shakyamuni Mantra in Siddham from Visible Mantra.
Please do visit the site for more examples and for information about sacred scripts.