1. Tools: pci scanning or Nessus or CSF or Login Failure Daemon, or LFD.
2. You can find a variety of tools (such as Nessus) to test your website for security vulnerabilities. Do some web searching and you can find them. These will certainly help, in the sense that they will catch the most common (easiest) errors. They are the first step.

You can also learn to do deeper pen testing, but it takes time, experience, and training. Do a web search for Kali Linux and Metasploit as examples.
If you run such tools, be careful to warn your provider(s) in advance and get acknowledgement from them. Make sure that they know that you own both systems (tester and target) and will not test any other IP addresses. Many providers have anti-hacking policies and will cancel any VPS that violates them. The same goes with ISPs. They will notice!
Once you have already cleaned up "the low hanging fruit", and need additional testing, it is time to hire a professional.
Security is not about absolute protection, it is about reducing risk. You cannot eliminate all vulnerabilities. Consider it like fire insurance. You can do the easy stuff (remove gasoline cans from your garage, for example), but a flaming airplane could still crash from the sky and burn your house.
Each step you take will reduce your vulnerabilities, but the cost for each level of improvement is much higher than the previous level. You have to decide how much risk you are willing to accept at a reasonable cost.
Security is also an ongoing process. New vulnerabilities are discovered each day. It is your responsibility to keep your VPS up-to-date with the latest patches, and periodically re-run tests.