To monitor VNC, watch tcp port 5900
To monitor IPSEC, watch ip protocol 50 (not port 50!), and udp port 500
It sounds like you also might want to watch port 8080.
BTW, you're not going to to see much if you're not running wireshark on your edge device - running it on the windows machine you are testing from might help a little, but it likely needs to run on the box that handles your VPN connection.