It’s understandable that he’ll feel this way with the changes you’ve described and he’ll be looking to you to be the constant in his life. I know how tough it is trying to spread yourself everywhere you need to be and eventually something has got to give. I’m not sure what to advise, but in my case, I went from a full time reasonably well paid job to part time and though the difference in pay is considerable, I do manage to make up a good chunk of the difference in tax credits, housing benefit, etc, and though am not comfortable financially like before, the reduction in my stress and the stability and consistency for my kids is totally worth it. I’m not suggesting you do the same and if your ex claims the tax credits etc , it wouldn’t be so easy, but I would try come up with a long term plan and give your son something to look forward to. In the meantime, you perhaps need to allocate some time that you can commit to, keeping the commitment is a must in my opinion. It sounds like you have had some real quality time and shouldn’t be disheartened when it seems to no avail, I’m sure he appreciated it, but when he’s feeling lost, he’ll want more of your time and just as you may feel out of control of your availability, he will feel frustration too.
If you can set some dates, then maybe involve him in planning what you will do, make an agreement and stick to it. He could have his own diary that you put these in for him to look at for reassurance until the times you meet. Being active may be better than watching TV, but sitting down to watch a film together can be a great bonding experience too, appealing to his interests will probably go a long way.
You may have the added obstacle of being talked about negatively, but as long as you keep showing willing, your son will acknowledge it and be able to separate tales from the truth.
I wish you the best of luck whatever you decide.