This has happened to me too! It’s really bad when I shave and I still get ingrown hairs when I wax.
Nair sensitive bikini works has worked best for me — the smell isn’t amazing but I’ve gotten pretty used to it. I leave it on for 10-15 minutes and then just wash it and the hair off. When the hair grows back, the ingrowns are minimal and heal pretty quickly.
One of my friends recommended this product called fur – and said that the ingrown cream is amazing. I haven’t tried it yet.