Printer Friendly Version Print this thread
Email this thread to a friend eMail this thread to a friend
  • Robots crawling (In: Members Lounge)
  • Using Robots.txt on Your Web Site (In: General Search Engine Optimization)
    • Featured Web Site Template

    Hundreds More at Free Site Templates.com!

    Web Site Partners
    Sponsored Links
    Jet City Software
     
    Whos Here ?
    Reflects user activity within the last 5 minutes
    Moderator(s): g1smd, Logan
    Forum Index · Search Engine Forums · Optimizing Your Website for the Search Engines · Google · Question on using robots.txt to Block Googlebot
    Member Message

    m021478
    Joined: Apr 10, 2008
    # Posts: 5

    View the profile for m021478 Send m021478 a private message

    Posted: 2008-May-21 06:11
    Edit Message Delete Message Reply to this message

    Google help documentation about removing/preventing its spider from indexing all, or a portion , of your site, it mentions:

    -------
    For example, if you're manually creating a robots.txt file, to block Googlebot from crawling all pages under a particular directory (for example, lemurs), you'd use the following robots.txt entry:

    User-agent: Googlebot
    Disallow: /lemurs/
    -------

    Does this mean, for example, that if my domain name was johndoe-dot-com, and I wanted to block all items contained within a particular directory on my site (which in this example, let's call "some_directory"wink, which I would normally access misc files contained in that directory by visiting www.johndoe-dot-com/some_directory/somefile.doc, then I would configure a .txt file with the aforementioned robots.txt configured as such:

    User-agent: Googlebot
    Disallow: /some_directory

    Probably a stupid question, but I did want to double check just to be sure...

    Any suggestions would be greatly appreciated... Thanks!



    Prowler
    Staff
    Joined: Aug 14, 2000
    # Posts: 1795

    View the profile for Prowler Send Prowler a private message

    Posted: 2008-May-21 06:28
    Edit Message Delete Message Reply to this message

    Add a slash at the end of the directory so that every file downstream will be out-of-bounds for the Googlebot.



    Code: [copy]
    Forum Index · Search Engine Forums · Optimizing Your Website for the Search Engines · Google · Question on using robots.txt to Block Googlebot
    You are not permitted to post messages in this forum or topic, because of one or more of the following reasons:
    1. You have not yet logged in, or registered properly as a member
    2. You are a member, but no longer have posting rights.
    3. This is a private forum, for which you do not have permissions.

    If you are a recent member, it's possible that you simply have not yet confirmed your account. Please check your email for a message entitled 'JimWorld Forums: Confirm Your Account' and follow the instructions contained within.

    If you cannot find this message, click here to Re-Send it.

    If you are still experiencing problem, please read the Login Assistance Article for some advice on what may be causing your login not to work properly.

    Switch to Advanced Editor and ... Create a New Topic or Reply to this Thread

    New posts Forum is locked
    © 1995  ·  iWeb, Inc  ·  DBA JimWorld Productions