3 Steps Guide to Follow Before Crawling If crawler can’t access, then the best way to find it through the web server logs. This will help you to understand whether search engine bots like Googlebot can access the product page but not your crawler. Otherwise, the reason that cause the problem including JavaScript.