Comments on: What robots.txt to use with TYPO3?

By: henjo

henjo — Wed, 04 Mar 2009 22:34:57 +0000

Hi Michael,

thanks for your response. I wasn’t aware ‘Sitemap:’ is an allowed statement in the robots files. That would be a nice way to add them.

It would be really cool if Google i.e. automatically traces that into the google.com/webmasters area … But that will be dreaming

Disallowing print pages and non-real urls is a really valuable tip to avoid duplicate content. Never really thought of that before – but that’s a very good idea!

Thanks!

By: Michael Cannon

Michael Cannon — Tue, 03 Mar 2009 04:05:00 +0000

At Acqal, besides the typical directory based disallows, we try to tell search engines to ignore non-realurl URLs and print pages. Additionally, we feed search engines blog or news feeds plus a dynamically generated sitemap.

User-Agent: *
Disallow: /cgi-bin/
Disallow: /t3lib/
Disallow: /typo3/
Disallow: /typo3conf/
Disallow: /typo3temp/
Disallow: /*?id=*
Disallow: /*&type=98
Sitemap: http://www.acqal.com/sitemap.xml
Sitemap: http://www.acqal.com/rss.xml

By: henjo

henjo — Mon, 02 Mar 2009 19:50:00 +0000

Hi Dmitry,

thanks for hoppin’ by. You are absolutely right…
I use to this mainly for the templates (since they are full of dummy code)…

So I would make it fileadmin/templates/ instead.

Ciao,
Henjo

By: Dmitry Dulepov

Dmitry Dulepov — Mon, 02 Mar 2009 19:43:55 +0000

One thing to add: disallowing fileadmin/ will disallow some downloadable files that you may want to place there. For example, on my site I have some pdf files for download. They are in fileadmin/.

I would think twice before disallowing fileadmin/.