Knowledgebase
Understanding and Troubleshooting Common Problems Using the Real Browser Feature to Gather Data
Posted by Mark Kabana on 15 March 2012 03:20 PM

NOTE: If you are having issues where the browser and/or Places Scout freezes or locks up while gathering data, please follow the instructions in this article to fix the issue: Using Firefox Portable Version 22 to Gather Data with Places Scout

 

NOTE: If you go through this article and still have problems using the real browser feature, you can always revert back to using the data gathering code from the previous version that does not use a real browser to gather data. To do this, goto the Settings tab in Places Scout, under the Data Gather heading, if you uncheck the ‘Use Real Browser to Gather Data’ option then it will use the old method of gathering data which should still work fine for gathering data. However, it is highly suggested that you try to use and get the real Real Browser feature working properly on your machine to gather data as it is the most accurate data gathering method and also greatly reduces the number of captchas you will encounter when gathering data.

About Places Scout’s New Data Gathering Technology Using a Real Web Browser

Places Scout has built in data gathering technology to use a real web browser to gather data. This technology ensures that the data gathered is accurate 100% of the time as the data gathering process does exactly what a human would do using a real browser.

Places Scout currently supports using either Firefox or Chrome as the browser to use for gathering data using a real browser. Please ensure that you have one of these browsers installed on your computer and that you are using the most up to date version of that browser.

 

How it Works

When Places Scout creates an instance of the browser, it uses a new empty Firefox or Chrome profile that is stripped down and does not contain any data or preferences from your current Firefox or Chrome user profile. A new profile is created every time you start a new data gathering operation, and none of your existing cookies, cache files, or settings are used in the profile that Places Scout creates.

This means that if you are currently logged into your Google account in Firefox or Chrome, that you will not be logged in to your account in the browser instance that Places Scout creates, and none of your Google preferences (such as personalized search, ect) will be used within Places Scout’s browser instance. The reason for this is so that every time start a new data gathering operation in Places Scout, a fresh, clean profile will be used with no cookies or cache files present that could possibly interfere with the results that Google provides to Places Scout.

This new technology is still very new to Places Scout and should work fine on most computers, however, there are numerous things that can interfere with and cause issues with this new data gathering technology. The following is a list of things that can cause this new technology to not work properly.

 

List of things that can cause issues using a Real Browser to Gather Data:


  • Not having the most up to date version of the selected browser installed.
    It is highly recommend that you have the latest version of either Firefox or Chrome installed, otherwise you may run into problems using the real browser.

    If you are using certain Firefox plugins that are not compatible with the latest version of Firefox and do not want to update Firefox, then it is suggested that you use Chrome instead (this is the reason I added support for an additional browser). 

    NOTE: Places Scout may incorrectly alert users that they do not have the most up to date version of Firefox installed. If you are certain that you have the most up to date version installed, and you receive this warning message, simply ignore it and everything should work fine. This will be fixed and addressed in the next update which is coming out very soon.

 

  • Certain Anti-virus Software can interfere with Places Scout’s ability to control the browser
    Certain Anti-virus software can cause problems with Places Scout ability to control and command the browser. Specifically, the anti-virus software ‘WebRootKit’ causes problems by greatly slowing down the speed at which Places Scout can type characters into the browser’s search box, making the process of gathering data painfully slow.

    If you are using Webrootkit and see that the letters are being typed very slow into the search box and it seems to take forever to gather data, please try closing Places Scout, disabling Webrootkit temporarily, opening Places Scout again and running your search to see if that fixes the problems.

    If you do not have Webrootkit installed, but are still experiencing other issues using the real browser feature, please try closing Places Scout, disabling your anti-virus software, opening Places Scout again and running your search to see if that fixes the problems.

 

  • Some installed browser plugins can cause the browser to crash or not work properly
    Some users have reported that the browser does not work properly – sometimes the browser will disappear when gathering data (because a plugin caused the browser to crash), or sometimes the data does not get parsed properly (because certain plugins change the html of the SERP results).

    If you notice that the browser sometimes disappears while gathering data, the first step is to try using the alternate browser option (Chrome) to see if it works properly. If you are still having troubles, try disabling some of the Plugins (which are different than add-ons and extensions) that are installed in Firefox. To do this, in Firefox, goto the Tools Menu, and click on the ‘Add-ons’ menu option – this will bring up the Add-ons Manager window. On the left menu bar, click on the ‘Plugins’ tab. Try disabling some of these Plugins and running a search in Places Scout again to see if this fixes the problem.

 

  • Hitting the shift key or interacting with the browser window while the browser window is being created
    When Places Scout creates the browser instance right after you start gathering data, it is very important that you do not hold down the shift key while this operation is taking place. Otherwise if you do this, Firefox will open in safe mode but will show a safe mode dialog box before it opens, which will interfere with Places Scout’s ability to open and control the browser. Also, please do not try to click or interact with the browser window that briefly appears when you first start gathering data – doing so may interfere with Places Scout’s ability to take control of the browser.

 

  • Changing the default installation directory for either Firefox or Chrome
    Places Scout needs to be able to locate the EXE file to either Firefox or Chrome in order to successfully launch and use the browser. Places Scout will look in the default installation directory to locate the EXE file for Firefox or Chrome - When you installed either Firefox or Chrome, if you chose a different directory than the default installation directory for either of these browsers, Places Scout will not be able to locate the EXE file to launch the browser.

    If you get an error message when attempting to gather data that says:

    "There was an error opening the specified browser. Please ensure that you have the selected browser installed (Firefox or Chrome) on your computer and try again."

    The most likely cause of this issue is that you chose to install either Firefox or Chrome to a different directory that the default installation directory. To check if this is the case, you will need to locate the your installation directory for Firefox / Chrome and see if it is different from the default installation directory for the browser you are trying to use. The default installation directories for each browser are as follows:

    Firefox Default Installation Directory: C:\Program Files (x86)\Mozilla Firefox\firefox.exe
    Chrome Default Installation Directory: C:\Users\[insert windows user name here]\AppData\Local\Google\Chrome\Application\chrome.exe

    If you cannot locate the above default installation directories, please uninstall the browser you are trying to use (Firefox or Chrome), and reinstall the browser and ensure that you do not change the default installation directory when running the installer for the browser. This will fix the issue of Places Scout not being able to find the selected browser.

 

  • Having bad entries in the Windows Hosts File
    The hosts file in Windows is used to map hostnames to IP addresses. If you have malformed entries for localhost or 127.0.0.1, this can cause issues with Places Scout not being able to control the browser properly, and you will see an error message when you try to gather data. To check if this is the case, please locate the Windows Hosts file, it will be in the following directory:

    C:\Windows\System32\drivers\etc\

    In the above directory, locate the file named 'hosts', right click on it, and choose to open the file in notepad. Next, locate the list of entries in the hosts file at bottom of the file. If you see any of the following entries in your hosts file:

    127.0.0.1 0.0.0.0
    localhost 0.0.0.0
    0.0.0.0 127.0.0.1
    0.0.0.0 localhost

    Delete these entries from your hosts file, and save it in notepad. Try to gather data again in Places Scout and see if this fixes the issue.

 

  • General Troubleshooting Tips for Firefox
    IF you are still having issues after trying the above suggestions, then you can try some general troubleshooting tips for Firefox as suggested by the developers of Firefox. Here they are:

    • First, try to reset Firefox. This simple step will eliminate many Firefox problems with only one click. Visit this link for instructions on how to do this: http://support.mozilla.org/en-US/kb/reset-preferences-fix-problems

    • A few other basic trouble shooting things you can try is this: Run all Windows Updates, install all needed service packs, etc.

    • Update your graphics driver (Firefox uses your graphics card for some rendering, and an out of date graphics card driver can cause problems. Upgrade your graphics drivers to use hardware acceleration and WebGL

    • Update all your plugins (Flash, Java, etc.). Visit http://www.mozilla.org/plugincheck/ to do this.

    • Download and Install MalwareBytes Anti-Malware, run a full Scan. http://www.malwarebytes.org/. This helps check that there are no viruses on your computer causing issues. You can uninstall this program after you clean off any infections.

 

Final Notes

Please understand that this new technology using a real browser is very new, and that while it works just fine for most people, others that have unique configurations may experience some issues with this new technology. We are currently aware of a few problems that can happen, and are working extremely hard to address and fix these issues as soon as possible.

If you are experiencing any problems using the new browser feature, you can always revert back to using the data gathering code from the previous version that does not use a real browser to gather data. To do this, goto the Settings tab in Places Scout, under the Data Gather heading, if you uncheck the ‘Use Real Browser to Gather Data’ option then it will use the old method of gathering data which should still work fine for gathering data. You can use the old code while we work out the remaining issues with the new technology.

Lastly, if you experience any other problems that were not outlined in this article, please open up a support ticket and explain in detail the problem you experienced so that we can include a fix to your specific issue in our next release.

(28 vote(s))
This article was helpful
This article was not helpful

Comments (0)
Help Desk Software by Kayako Fusion