Note: Target audience 100,000 vs budget $5->$100 = 4.5%->0.5% frequency ~1.0
In a small audience of 100K people the Facebook algorithm will be searching for the best and most likely people to do the action that you want (conversion, click, engagement etc). As you increase the ad budget, you are asking Facebook to show it to less qualified people.
The key thing to watch is not so much the CTR, but the cost per action. Your CTR may go down, but if the events are still happening at an acceptable cost, then CTR doesn’t really matter.
An audience of 100k is generally small.
To relieve this pressure try to find similar audiences and put some budget against these. If you can build an audience closer to 1 Million you might find less issues as you increase your ad budget.
Some audiences and budgets have a balance point. This is a point where Facebook can continue to deliver an acceptable cost per action. Above this balance point, costs start to increase.