Yes, you can decline an accepted invitation anytime to stop further communication.
You may also like to see :
What is the meaning of accepting, declining, or canceling interest with a member on Shaadi.com?