If you change A records this won't affect email at all (since  email uses MX records). So yes, remove the four wrong IP addresses and it should start working OK! There may be a delay of up to an hour while this takes effect - though after that it should work fine and you should be able to log into  your site on the non-www domain.

I'm not familiar with Godaddy's DNS admin panel, but what you want to do is have an A record with subdomain '@' which points to the Voice IP (this handles the case of using the domain with no www subdomain), and have another A record with subdomain 'www' which also points to the Voice IP address, which will handle the www version. (There will need to be two separate entries, you can't have one for both cases.)

To access the admin pages of your site if the domain isn't working then go to e-voice.org.uk and log in as the admin of your site, then go to e-voice.org.uk/<your site>/admin

If you're logged in as an admin of the site you won't be redirected to the custom domain.

Thanks

Joe