First of all, use your own mind for your own spiritual progress. Forget what others believe or say or give an opinion about.
Secondly, if you take any religion including Hinduism, then religions are mere tools and indicating a path for your own spiritual and other morals progress.
Now many rituals and worships are nothing but a kind of mental satisfaction of the individual, performed for a short period.
It is very important that just do not give up worship after finishing ritual or Puja. Idol worship is not an end of your spiritual journey but it is a beginning for your self-purification.
It is a sad situation that we are never told about the importance of Idol worship which is done purely for material benefits only so we spend very little time in worship.
Your worship through idols or prayers or Samadhi etc becomes your own way of adopting path for your own spiritual progress or to know more about God.
It is certain that God or Allah or Param Tatva are not just handy and easily available at stone throwing distance. One has to work for them and then alone you can get them.
Just like earning money, you work, in the same way, to know or meet God, you have to work sincerely and by observing rules and regulations.
So please close your ears for others’ opinions, how so ever he or she is big or scholar. just follow what your own self tells you to do and do that only.