Basecamp
Get me at Technoratie
Typo3
WordPressWell – this is what I have put in standard:
User-Agent: *
Allow: /
Disallow: /fileadmin/templates/
Disallow: /typo3/
Disallow: /t3lib/
Disallow: /typo3conf/
Disallow: /typo3_src/
Disallow: /typo3temp/
Disallow: /uploads/
So – what do you put in?
4 Comments for What robots.txt to use with TYPO3?
Dmitry Dulepov | March 2, 2009 at 9:43 pm
Author comment by henjo | March 2, 2009 at 9:50 pm
Hi Dmitry,
thanks for hoppin’ by. You are absolutely right…
I use to this mainly for the templates (since they are full of dummy code)…
So I would make it fileadmin/templates/ instead.
Ciao,
Henjo
Michael Cannon | March 3, 2009 at 6:05 am
At Acqal, besides the typical directory based disallows, we try to tell search engines to ignore non-realurl URLs and print pages. Additionally, we feed search engines blog or news feeds plus a dynamically generated sitemap.
User-Agent: *
Disallow: /cgi-bin/
Disallow: /t3lib/
Disallow: /typo3/
Disallow: /typo3conf/
Disallow: /typo3temp/
Disallow: /*?id=*
Disallow: /*&type=98
Sitemap: http://www.acqal.com/sitemap.xml
Sitemap: http://www.acqal.com/rss.xml
Author comment by henjo | March 5, 2009 at 12:34 am
Hi Michael,
thanks for your response. I wasn’t aware ‘Sitemap:’ is an allowed statement in the robots files. That would be a nice way to add them.
It would be really cool if Google i.e. automatically traces that into the google.com/webmasters area … But that will be dreaming 
Disallowing print pages and non-real urls is a really valuable tip to avoid duplicate content. Never really thought of that before – but that’s a very good idea!
Thanks!
One thing to add: disallowing fileadmin/ will disallow some downloadable files that you may want to place there. For example, on my site I have some pdf files for download. They are in fileadmin/.
I would think twice before disallowing fileadmin/.