Wracając od szczegółu do ogółu: gdyby na wspominanym wyżej serwisie istniał poprawnie skonfigurowany plik robots.txt, nie mielibyśmy o czym dzisiaj rozmawiać

Mamy zatem tylko 2 instytucje finansowe (na 24, czyli 8,5%), które mogą być w miarę spokojne, jeżeli chodzi o konfigurację tego pliku

# Robots.txt utworzony przez MR.R
# Dla strony http://twojastrona.pl

# Zablokuj następujące boty i katalogi w disallow na koncu wpisz jakie ma zablokowac katalogi przed botami jesli ma strona byc spokojna nic nie zmieniaj:

User-agent: Google)*(G)*(http://www.google.com/addurl)*(q={URL}&dq=&submit=Add+URL
User-agent: Splat!)*(G)*(http://www.splatsearch.com/cgi-bin/spider.cgi)*(instant=1&url={URL}
User-agent: SearchIt)*(G)*(http://index.searchit.com/add-site.php)*(url={URL}&email={MAIL}&submit=Send
User-agent: MSN)*(G)*(http://submitit.bcentral.com/msnaddpublic1625703506.aspx)*(msnurl={URL}&email={MAIL}&subscribe=0&Submit1=+++++Submit+Site+++++
User-agent: Walhello)*(G)*(http://addurl.walhello.info/urlgl.cgi)*(url={URL}&button=Suggest+a+URL
User-agent: Daypop)*(G)*(http://www.daypop.com/submit_site)*(extra=1&url={URL}&Submit=Submit+Site
User-agent: whatUseek)*(G)*(http://add.whatuseek.com/cgi-bin/addurl_bottom.cgi)*(col=1&url={URL}&email={MAIL}&sub_wus=Submit+URL
User-agent: amfibi)*(G)*(http://www.amfibi.com/tky/)*(url={URL}&email={MAIL}
User-agent: AxxaSearch)*(G)*(http://www.axxasearch.com/thanks.htm)*(email={MAIL}&name={URL}&lists=search-engine-facts
User-agent: Infotiger)*(G)*(http://www.infotiger.com/addurl.html)*(url={URL}&email={MAIL}
User-agent: accoona)*(G)*(http://www.accoona.com/URLServlet)*(url={URL}&comments=
User-agent: SearchEngine)*(G)*(http://www.searchengine.com/List_Your_Site/Basic_Submit/addURL.asp)*(URL={URL}&eMail={MAIL}
User-agent: searchUK)*(G)*(http://searchuk.com/List_Your_Site/Basic_Submit/addURL.asp)*(URL={URL}&eMail={MAIL}
User-agent: Amfibi)*(P)*(http://addurl.amfibi.com/index.pl)*(url={URL}&email={MAIL}
User-agent: HomerWeb)*(P)*(http://www.homerweb.com/submit_site.html)*(url={URL}
User-agent: Cipinet)*(P)*(http://www.cipinet.com/cgi-bin/submit.pl)*(Url={URL}
User-agent: Scrub The Web)*(P)*(http://www.scrubtheweb.com/addurl.html)*(URL={URL}&Email={MAIL}
User-agent: Net What)*(P)*(http://www.netwhat.com:8080/add)*(URL={URL}&EMAIL={MAIL}&NAME=Krzys&terms=Yes
User-agent: Myahint)*(G)*(http://www.myahint.com/addurl?)*(url={URL}
User-agent: JDGO)*(G)*(http://www.jdgo.com/submit.php?)*(link={URL}
User-agent: Szukacz)*(P)*(http://www.szukacz.pl/DodajUrl.aspx)*(url=3D{URL}
User-agent: SearchSight)*(P)*(http://searchsight.com/receive.htm)*(sURL=3D{URL}&sEmail=3D{MAIL}
User-agent: Websquash)*(P)*(http://www.websquash.com/cgi-bin/search/search.pl)*(Mode=3DAnonAdd&b_submit=3D1&Realm=3DInternet&URL=3D{URL}&EMAIL=3D{MAIL}
User-agent: Susy Search)*(G)*(http://www.susysearch.com/addsite/)*(url=3D{URL}&title=3D{TITLE}&description=3D{DESC}
User-agent: SearchWarp)*(P)*(http://searchwarp.com/Submit.asp)*(LocalSubmission=3D681&Site=3D{URL}&Email=3D{MAIL}
User-agent: AnooX)*(P)*(http://www.anoox.com/add_for_indexing_free.jsp)*(url=3D{URL}&email=3D{MAIL}&checkbox=3Dcheckbox
User-agent: Evisum)*(P)*(http://www.evisum.com/cgi-bin/addsite.cgi)*(email=3D{MAIL}&URL=3D{URL}
User-agent: Unasked)*(P)*(http://unasked.com/submit.asp)*(LocalSubmission=3D262&site=3D{URL}&email=3D{MAIL}
User-agent: DinoSearch)*(P)*(http://dinosearch.com/dinosearch/update.asp)*(email=3D{MAIL}&LocalSubmission=3D892&url=3D{URL}
User-agent: Entireweb)*(P)*(http://services.entireweb.com/addurl/main.php)*(URL=3D{URL}&Email=3D{MAIL}
User-agent: Jayde)*(P)*(http://submit2.jayde.com)*(imgc=&SOURCE=&URL={URL}&EMAIL={MAIL}
User-agent: Burf)*(P)*(http://www.burf.com/submit2.php)*(submittext={URL}
User-agent: Search-O-Rama)*(G)*(http://search-o-rama.com/submit.asp)*(site={URL}&email={MAIL}&LocalSubmission=242
User-agent: Mixcat)*(P)*(http://www.mixcat.com/box.php)*(url={URL}&email={MAIL}&name=Stefan
User-agent: Abacho)*(G)*(http://www.de.abacho.com/anmelden.html)*(form_url={URL}
User-agent: Sahfor)*(P)*(http://www.sahfor.com/addurl_website.asp)*(websiteurl={URL}
User-agent: Big Finder)*(P)*(http://www.bigfinder.com/spider.cfm)*(url={URL}&email={MAIL}
User-agent: Fyber)*(G)*(http://www.fybersearch.com/suggest-a-url.php)*(addurl={URL}&email={MAIL}
User-agent: SonicRun)*(P)*(http://www.sonicrun.com:8081/add)*(URL={URL}&EMAIL={MAIL}&NAME=Tadeusz&terms=yes
User-agent: Acoon)*(G)*(http://www3.acoon.com/addurl.html)*(url={URL}
User-agent: Amidalla)*(P)*(http://www.amidalla.de/cgi-bin/amiadd.cgi?add)*(addurl={URL}
User-agent: Suchfit.de)*(P)*(http://suchfit.de/submit.php)*(url={URL}&email={MAIL}
User-agent: AcrossCan)*(P)*(http://www.acrosscan.com/submit.php)*(url={URL}
User-agent: Hypermaze)*(P)*(http://www.myglobalwebsite.com/cgi-bin/myglobal/addurl.pl)*(url={URL}
User-agent: Claymont)*(P)*(http://www.claymont.com/asub.asp)*(sourceid=2&url={URL}&email={MAIL}
User-agent: Alltheweb)*(P)*(https://siteexplorer.search.yahoo.com/updfeed)*(.crumb=y5K1eFLIt2k&txtUrl={URL}&.scrumb=0&btnSubmitUrl=Submit%20URL
User-agent: ISearch)*(P)*(http://www.i1search.com/site-submit.html?submitting=1)*(data={URL}
User-agent: Shoula)*(P)*(http://www.shoula.com/submit.php)*(URL={URL}&Email={MAIL}
User-agent: RateMyHost)*(G)*(http://www.ratemyhost.com/rate_webhost.php)*(url={URL}
User-agent: ooBdoo)*(G)*(http://www.inter-change.com/cgibin/search2/interchange4.pl)*(url={URL}&email={MAIL}&action=addurl&send=
User-agent: Khoj)*(P)*(http://khoj.com/bin/getuserurls.pl)*(url={URL}&hid=
User-agent: iDoFind)*(P)*(http://www.idofind.com/addurl/add.php)*(yourelink={URL}&youremail={MAIL}
User-agent: Bolivian)*(P)*(http://www.bolivian.com/cgi-bin/LD/registro_LD.cgi)*(url={URL}&email={MAIL}
User-agent: Szukacz.pl)*(P)*(http://www.szukacz.pl/DodajUrl.aspx)*(url={URL}
User-agent: Gooru)*(P)*(http://www.gooru.pl/zglosstrone.php)*(link={URL}&add2cat=y
User-agent: Yahoo!)*(G)*(https://siteexplorer.search.yahoo.com/updfeed)*(txtUrl={URL}&.crumb=xqBJQwi1wv4&.scrumb=0&btnSubmitUrl=Submit URL&class=Search
User-agent: SplatSearch.com)*(P)*(http://www.splatsearch.com/cgi-bin/spider.cgi)*(url={URL}&submit=Submit&instant=1
User-agent: ScrubTheWeb)*(P)*(http://www.scrubtheweb.com//cgi-bin/addurl.cgi)*(URL={URL}&Email={MAIL}
User-agent: Subjex)*(P)*(http://www.subjex.net/servlet/submiturlfromform.php)*(email={MAIL}&url={URL}
User-agent: FyberSearch)*(G)*(http://www.fybersearch.com/suggest-a-url.php)*(email={MAIL}&website={URL}
User-agent: TowerSearch)*(P)*(http://www.aweber.com/scripts/addlead.pl)*(meta_web_form_id=1938414548&meta_split_id=&unit=towersearch&redirect=http://towersearch.com/confirm.php&meta_redirect_onlist=&meta_adtracking=&meta_message=1&meta_required=form&meta_forwads_vars=1&name={NAME}&email={MAIL}
User-agent: Abacho.es)*(G)*(http://194.231.30.205/url/newurl.cfm)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Abacho.it)*(G)*(http://194.231.30.216/url/newurl.cfm)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Abacho.tr)*(G)*(http://194.231.30.203/url/newurl.cfm)*(url={URL}&Mode=AnonAdd
User-agent: Abacho.uk)*(G)*(http://search.abacho.co.uk/url/newurl.cfm)*(url={URL}&Mode=AnonAdd
User-agent: Abraham Search)*(P)*(http://www.abrahamsearch.it/inserimentom.php3)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Acoon)*(G)*(http://www.acoon.de/cgi-bin/addurl.exe)*(url={URL}&layout=en
User-agent: Aeiwi)*(G)*(http://www.aeiwi.com/cgi-bin/submit)*(url={URL}&email={EMAIL}&Submit=Submit
User-agent: Alexa)*(P)*(http://www.alexa.com/cgi-bin/request_bot.cgi)*(url={URL}&submit=Archive%20My%20Site&redirect=http://www.alexa.com/help/webmasters/successful.html
User-agent: Alphaz)*(P)*(http://www.alphaz.ru/cgi-bin/search/search.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=\"www.AlphaZ.ru\"
User-agent: All The Web)*(P)*(http://www.alltheweb.com/add_url.php)*(url={URL}&email={EMAIL}&submit=Add%20URL
User-agent: Altavista)*(G)*(http://addurl.altavista.com/sites/addurl/newurl)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Altavista.at)*(G)*(http://addurl.altavista.com/sites/atde/pos/addurl/submit)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Altavista.au)*(G)*(http://addurl.altavista.com/sites/auen/pos/addurl/submit)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Altavista.be)*(G)*(http://addurl.altavista.com/sites/benl/pos/addurl/submit)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Altavista.br)*(G)*(http://addurl.altavista.com/sites/brpt/pos/addurl/submit)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Altavista.ca)*(G)*(http://addurl.altavista.com/sites/caen/pos/addurl/submit)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Altavista.ch)*(G)*(http://addurl.altavista.com/sites/chfr/pos/addurl/submit)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Altavista.de)*(G)*(http://addurl.altavista.com/sites/dede/pos/addurl/submit)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Altavista.dk)*(G)*(http://addurl.altavista.com/sites/dkda/pos/addurl/submit)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Altavista.es)*(G)*(http://addurl.altavista.com/sites/esca/pos/addurl/submit)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Altavista.fi)*(G)*(http://addurl.altavista.com/sites/fifi/pos/addurl/submit)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Altavista.fr)*(G)*(http://addurl.altavista.com/sites/frfr/pos/addurl/submit)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Altavista.gb)*(G)*(http://addurl.altavista.com/sites/gben/pos/addurl/submit)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Altavista.ie)*(G)*(http://addurl.altavista.com/sites/ieen/pos/addurl/submit)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Altavista.in)*(G)*(http://addurl.altavista.com/sites/inen/pos/addurl/submit)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Altavista.it)*(G)*(http://addurl.altavista.com/sites/itit/pos/addurl/submit)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Altavista.kr)*(G)*(http://addurl.altavista.com/sites/krko/pos/addurl/submit)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Altavista.nl)*(G)*(http://addurl.altavista.com/sites/nlnl/pos/addurl/submit)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Altavista.no)*(G)*(http://addurl.altavista.com/sites/nono/pos/addurl/submit)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Altavista.nz)*(G)*(http://addurl.altavista.com/sites/nzen/pos/addurl/submit)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Altavista.pt)*(G)*(http://addurl.altavista.com/sites/ptpt/pos/addurl/submit)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Altavista.se)*(G)*(http://addurl.altavista.com/sites/sesv/pos/addurl/submit)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Amfibi)*(P)*(http://www.amfibi.com/mfb/search.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=\"AMFIBI\"
User-agent: Arianna)*(G)*(http://giotto5.iol.it/abin/addurl.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Asquickas)*(P)*(http://www.asquickas.com/search/search.cgi)*(url={URL}&Mode=AnonAdd&Action=AddURL&Realm=websites
User-agent: Aesop)*(G)*(http://www.aesop.com/cgi-bin/sub/submiturl.cgi)*(url={URL}&email={EMAIL}&fname=Web&lname=Master&ktj=0&step=check&serv=5
User-agent: Bethel)*(P)*(http://www.bethel-fr.com/moteur/search.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=\"sites chrétiens\"
User-agent: Benadir)*(P)*(http://www.benadir.net/search/search.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Blue Win)*(G)*(http://addurl.altavista.com/sites/chde_bw/pos/addurl/submit)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Bmweb)*(P)*(http://www.bmweb.it/cgi-bin/addurl.pl)*(url={URL}&Mode=AnonAdd
User-agent: Bnknet)*(P)*(http://www.bnknet.com/cgi-bin/yave/search.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=Antilles/Aruba
User-agent: Bolivian)*(P)*(http://www.bolivian.com/cgi-bin/LD/registro_LD.cgi)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Browseabit)*(P)*(http://www.browseabit.com/cgi-bin/BB_Search/search/search.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=All_Other
User-agent: Buscopio)*(P)*(http://www.buscopio.com/NASApp/buscopio/sugerirUrl.jsp)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Cannock-Online)*(P)*(http://cannock-online.co.uk/search_engine/search/search.cgi)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=\"Cannock Web Sites\"
User-agent: Cir.Msk)*(P)*(http://cir.msk.ru/cgi-bin/search/search.cgi)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=\"External Sites Index\"
User-agent: Claymont)*(P)*(http://www.claymont.com/cgi-bin/addurlsform.pl)*(url={URL}&email={EMAIL}&action=submit
User-agent: CollectorsRegistry)*(P)*(http://www.collectorsregistry.com/cgi-bin/search/search.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=remote
User-agent: Conexur)*(P)*(http://www.conexur.com/cgi-bin/search/search.cgi)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Coptic)*(G)*(http://coptic.lcs.mit.edu/cgi-bin/addurl.cgi)*(url={URL}&email={EMAIL}&dq=\"\"
User-agent: CrucesOnline)*(P)*(http://www.crucesonline.com/cgi-local/search.pl)*(url={URL}&AddSite=prompt&Realm=CrucesOnline
User-agent: Cupe)*(P)*(http://www.cupe.ca/scripts/search.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=\"CUPE Crawler\"
User-agent: Cyprusrealtor)*(P)*(http://www.cyprusrealtor.com/_vti_bin/shtml.exe/add_url.html)*(url={URL}&email={EMAIL}&Mode=AnonAdd&C1=ON
User-agent: Deals4u)*(P)*(http://www.deals4u.com/sr/sr.cgi)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=\"The Web\"
User-agent: Dino Search)*(P)*(http://www.dinosearch.com/dinosearch/update.asp)*(submitemail={EMAIL}&url={URL}&
User-agent: Dog Pile)*(P)*(http://dpcatalog.dogpile.com/texis/websearch/modify.html)*(url={SITE}&Name=Web Master&Phone=215-555-1212
User-agent: Entireweb.com)*(G)*(http://www.entireweb.com/cgi-bin/submit/submit.cgi)*(url={URL}&email={EMAIL}&
User-agent: Eviet)*(P)*(http://eviet.com/cgi-bin/moses/addurl.pl)*(url={URL}&email={EMAIL}&action=user_addform
User-agent: ExactSeek)*(P)*(http://www.exactseek.com/cgi-bin/autoadd.cgi)*(url={URL}&email={EMAIL}&ID=autosubmittor
User-agent: Find Once)*(P)*(http://www.findonce.com/submit/Done.Asp)*(url={URL}&
User-agent: Free-1)*(P)*(http://www.free-1.com/cgi-bin/search/search.pl)*(url={URL}&email={EMAIL}&action=addurl&action=spider
User-agent: G4you)*(P)*(http://www.g4you.de/cgi-bin/g4you.pl)*(url={URL}&Mode=AnonAdd&Action=AddURL&Realm=Englisch
User-agent: Goo)*(P)*(http://goo502.goo.ne.jp/cgi-bin/feedback/addurl.cgi)*(url={URL}&email={EMAIL}&
User-agent: HotBot)*(G)*(http://hotbot.lycos.com/addurl.asp)*(url={URL}&email={EMAIL}&MM=1,ACTION=subscribe,SOURCE=hotbot
User-agent: HotBot.fr)*(G)*(http://www.hotbot.lycos.fr/submit.html)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Hotbot.it)*(G)*(http://www.hotbot.lycos.it/submit.html)*(url={URL}&email={EMAIL}&Mode=AnonAdd,engine=hotbot
User-agent: Hotbot.nl)*(G)*(http://www.hotbot.lycos.nl/submit.html)*(url={URL}&email={EMAIL}&Mode=AnonAdd,engine=hotbot
User-agent: Hotbot.se)*(G)*(http://hotbot.spray.se/submit.html)*(url={URL}&email={EMAIL}&Mode=AnonAdd,engine=hotbot
User-agent: Hotbot.uk)*(G)*(http://www.hotbot.lycos.co.uk/submit.html)*(url={URL}&email={EMAIL}&Mode=AnonAdd,engine=hotbot
User-agent: Iguana)*(P)*(http://www.iguana.com.mx/cgi-bin/s.reg)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Improve My Search)*(G)*(http://www.improvemysearch.com/addurl.shtml)*(url={URL}&email={EMAIL}&source=improvemysearch&adultsite=1
User-agent: Info Mak)*(G)*(http://url.infomak.com/index.html)*(url={URL}&
User-agent: Info Seek.jp)*(G)*(http://srv1.infoseek.co.jp/addurl/chkurl.cgi)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Info World)*(P)*(http://www.info-world.com/cgi-bin/search/search.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=\"My Realm 8\"
User-agent: Intermont )*(P)*(http://www.intermont.de/cgi-bin/search/search.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=\"Sitesearch2000\"
User-agent: Internet Cruiser)*(P)*(http://www.krstarica.com/eng/url/url.php)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Iprom)*(P)*(http://www.iprom.co.uk/links/addurl.asp)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Italo)*(P)*(http://www.italo.top.pl/search/add_url.php)*(url={URL}&email={SUBMITEMAIL}&Mode=AnonAdd
User-agent: Jadoo)*(P)*(http://www.jadoo.com/cgi-bin/search.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Jewish-Europe)*(P)*(http://search.jewish-europe.net/search/search.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=nosafim
User-agent: Jnternet)*(G)*(http://jnternet.virtualave.net/search.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Action=AddURL&Realm=English
User-agent: Khsearch)*(P)*(http://www.khsearch.com/search/search.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=English&submit=\"Add New URL\"
User-agent: KnifeSeek)*(P)*(http://www.knifeseek.com/cgi-bin/search/search.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm VALUE=open
User-agent: Kuamp)*(P)*(http://kuamp.kuamp.kyoto-u.ac.jp/cgi-bin/uncgi/mondou-registration.cgi)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Kunstbus)*(P)*(http://www.kunstbus.nl/cgi-bin/search/search.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=\"My Realm 14\"
User-agent: Lotse.SpeedFind)*(G)*(http://lotse.speedfind.de/cgi-bin/lotseaddurl)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Lycos)*(G)*(http://www.lycos.com/cgi-bin/spider_now.pl)*(url={QUERY}&email={EMAIL}&
User-agent: Lycos.de)*(G)*(http://www.lycos.de/service/site_added.html)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Lycos.es)*(G)*(http://www.lycos.es/service/site_added.html)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Lycos.fr)*(G)*(http://www.lycos.fr/service/site_added.html)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Lycos.it)*(G)*(http://www.lycos.it/service/site_added.html)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Lycos.nl)*(G)*(http://www.lycos.nl/service/site_added.html)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Lycos.no)*(G)*(http://www.lycos.no/service/site_added.html)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Lycos.ru)*(G)*(http://www.lycos.ru/service/site_added.html)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Lycos.uk)*(G)*(http://www-english.lycos.com/cgi-bin/spider_now.pl)*(url={URL}&email={EMAIL}&
User-agent: Madagascan)*(P)*(http://search.madagascan.com/index.php)*(url={URL}&Mode=AnonAdd&Realm=\"Internet Malagasy\"
User-agent: Metaeureka)*(P)*(http://www.metaeureka.com/cgi-bin/addurl2.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Meta-Noia)*(P)*(http://meta-noia.com/cgi-bin/buscar/search/search.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=\"Ańadidos\"
User-agent: MediaMarkt )*(P)*(http://portal.mediamarkt.de/search/add_url.php)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: MediaMiner)*(P)*(http://www.mediaminer.org/add_site.php)*(url={URL}&email={EMAIL}&Mode=AnonAdd&type=Miscellaneous
User-agent: Mexiko Playadelcarmen)*(P)*(http://www.mexiko-playadelcarmen.de/cgi-local/search/search.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=\"Playa del Carmen\"
User-agent: Mix Cat)*(P)*(http://www.mixcat.com/addurl.html)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Moneydo)*(P)*(http://moneydo.com/cgi-bin/engine/search.cgi)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: MSN (BCentral))*(G)*(http://submitit.bcentral.com/system/msnaddpublic.cfm)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Mtomoff)*(P)*(http://mtomoff.virtualave.net/cgi-bin/search/search.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=\"Web\"
User-agent: My Global Web Site)*(P)*(http://search.myglobalwebsite.com/cgi-bin/myglobal/moses_addurl.pl)*(url={URL}&email={EMAIL}&submit=Add%20URL
User-agent: Nzexplorer)*(P)*(http://nzexplorer.co.nz/add_a_url.cgi?pbourke@swin.edu.au)*(url={URL}&Mode=AnonAdd
User-agent: Omniseek)*(P)*(http://www.omniseek.com/submit/)*(url={URL}&this=start&site_password=passthis
User-agent: Onceiwas)*(P)*(http://www.onceiwas.com/search/search.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=\"All Sites\"
User-agent: Onlythebest)*(P)*(http://www.onlythebest.f2s.com/search/search.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Action=AddURL&Realm=\"CS Sites\"
User-agent: PikerKiwi)*(P)*(http://pikerkiwi.hypermart.net/cgi-bin/search/search.cgi)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Pimptrain)*(P)*(http://www.pimptrain.com/search.cgi)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Pocoweb)*(P)*(http://www.pocoweb.com/search/search.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=\"My Realm 1\"
User-agent: Poland Online)*(P)*(http://www.polandonline.com/search/search.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=\"My Realm 1\"
User-agent: Ravencom)*(P)*(http://search.ravencom.net/addurl.cfm)*(url={URL}&CheckIt
User-agent: Sailory)*(P)*(http://add-url.sailory.com/add-url/add_url.plx)*(url={URL}&fuse=2
User-agent: Sawaal)*(P)*(http://www.sawaal.com/cgi-bin/cgiwrap/sawaal/geturl1.cgi)*(url={URL}&Mode=AnonAdd
User-agent: Scoop.Evansville)*(P)*(http://scoop.evansville.net/cgi-bin/formmaster)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Search It)*(P)*(http://www.searchit.com/add-site.php)*(url={URL}&email={EMAIL}&
User-agent: Search Web)*(P)*(http://users2.cgiforme.com/chrisdill/addurl.cgi)*(url={URL}&description={DESC}&
User-agent: Search Who)*(G)*(http://www.searchwho.com/cgi-bin/addsite.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Spark Search)*(P)*(http://www.sparksearch.co.uk/cgi-bin/sparksubmit.pl)*(url={URL}&email={EMAIL}&
User-agent: Spider)*(P)*(http://www.spider.lu/cgi-bin/addurl)*(url={URL}&
User-agent: Str82u)*(P)*(http://str82u.com/cgi-bin/search/search.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=\"Str82U.Com\"
User-agent: Subjex)*(P)*(http://www.subjex.com/servlet/SubjexSubmitterServlet)*(url={URL}&email={EMAIL}&action=submit
User-agent: Surf London)*(P)*(http://www.surflondon.co.uk/HTML/links.php3?op=Add)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Surfeu.de)*(P)*(http://www.surfeu.ch/de/search/add_url.php)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Surfeu.fi)*(P)*(http://www.surfeu.fi/search/add_url.php)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: SurfSaturn)*(P)*(http://www.surfsaturn.de/search/add_url.php)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Tanglad)*(P)*(http://www.tanglad.com/cgi-bin/search/search.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=\"My Realm 25\"
User-agent: TechnoFind)*(P)*(http://www.technofind.com.sg/FDML/tfaddurl.FDML)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: ThunderSeek)*(P)*(http://www.thunderseek.com/search/mailit.cgi)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: TracksOnline)*(P)*(http://www.tracksonline.com/cgi-bin/search/search.cgi)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=\"Web 1\"
User-agent: VeryGoodSearch)*(P)*(http://www.verygoodsearch.com/cgi-bin/go2/search.pl)*(url={URL}&action=addurl&action=spider
User-agent: Vindex)*(G)*(http://www.vindex.nl/aanmelden/index.html)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Voila)*(G)*(http://www.voila.fr/submit)*(url={URL}&email={EMAIL}&
User-agent: Wal Hello)*(G)*(http://cgi.walhello.com/urlgl.cgi)*(url={URL}&
User-agent: WapTiger)*(G)*(http://www.waptiger.de/addurl/)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Web Search2000)*(P)*(http://www.websearch2000.de/cgi-bin/search/search.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=\"Topadressen\"
User-agent: WebSquash)*(P)*(http://www.websquash.com/cgi-bin/dir/add.cgi)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: WebSurfer)*(P)*(http://www.io.com/websurfer/cgi-bin/maillist.cgi)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Web Wizard )*(P)*(http://www.tripple.net/contator/webwizard/addurl.asp)*(url={URL}&email={EMAIL}&Mode=AnonAdd&at=true
User-agent: Web Wizard 1)*(P)*(http://www.webwizard1.com/cgi-bin/search.pl)*(url={URL}&Mode=AnonAdd&Action=AddURL&Realm=\"Remote Pages\"
User-agent: WestFront)*(P)*(http://www.westfront.de/search/search.pl)*(url={URL}&Mode=AnonAdd&Action=AddURL&Realm=This Site
User-agent: WhatUSeek)*(G)*(http://add.whatuseek.com/cgi-bin/addurl_bottom.cgi)*(url={URL}&email={EMAIL}&chk1=15
User-agent: WWWhere.com:90)*(P)*(http://submit.wwwhere.com:90/FMPro)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Yeehaa)*(P)*(http://www.yeehaa.com/doaddurl.asp)*(url={URL}&
User-agent: YooZee)*(P)*(http://www.yoozee.com/cgi-bin/asearch/submit.pl)*(url={URL}&
User-agent: Zippy Finder)*(G)*(http://www.zippyfinder.com/cgi-bin/indexer.exe)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: AdultInfoCenter)*(P)*(http://www.adultinfocenter.com/cgi-bin/search/search.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=\"WebMasters\"
User-agent: BlackOrchid)*(P)*(http://www.blackorchid.com/Asearch/search.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=\"Adult Sites\"
User-agent: Club18-21)*(P)*(http://www.club18-21.com/search/search.cgi)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=\"Adult Search Engine\"
User-agent: EVisum)*(P)*(http://www.evisum.com/cgi-bin/addsite.cgi)*(url={URL}&submitemail={EMAIL}&
User-agent: Ilsearch)*(P)*(http://www.ilsearch.com/cgi-bin/search.cgi)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=Security
User-agent: Meta-Ukraine)*(P)*(http://meta-ukraine.com/addurl/url2db.asp)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Pantrinbago)*(P)*(http://www.pantrinbago.com/search/search.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=Open
User-agent: Search-4-porn)*(P)*(http://www.search-4-porn.com/cgi-bin/search/search.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=\"Web\"
User-agent: Search)*(P)*(http://www.search.ch/addurl.pl)*(url={URL}&email={EMAIL}&why=\"\",submit=Zur%20Evaluation,l=de
User-agent: Search.Centre)*(P)*(http://search.centre.ru/cgi-bin/test/add.cgi)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Search Engine)*(G)*(http://url.searchengine.com/cgi-bin/url.cgi)*(url={URL}&email={EMAIL}&country=uk&d=2&w=0&y=1&z=0&g=0&related=0&cl=english&cu=uk
User-agent: Sexavista)*(P)*(http://www.sexavista.nl/cgi/search/search.pl)*(url={URL}&Mode=AnonAdd&Action=AddURL&Realm=\"Sexavista\"
User-agent: SharpIndex)*(P)*(http://www.csharpindex.com/profile/addURL.asp)*(url={URL}&Comments=\"\",Mode=AnonAdd
User-agent: Sofcom)*(P)*(http://www2.sofcom.com.au/cgi-bin/Dir/WWWadd.cgi)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Web Wombat)*(G)*(http://www.webwombat.com.au/Wregister)*(url={NEWURL}&email={EMAIL}&
User-agent: Web Yield)*(P)*(http://web-yield.hypermart.net/search/search.pl)*(url={URL}&email={EMAIL}&Mode=AnonAdd&Realm=\"index\"
User-agent: WWW.RU)*(P)*(http://www.ru/eng/index.work.html)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Yandex)*(P)*(http://www.yandex.ru/cgi-bin/publicaddurl)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Zenon)*(P)*(http://www.zenon.net/cgi/yandsearch)*(url={URL}&email={EMAIL}&Mode=AnonAdd
User-agent: Mastersite)*(G)*(http://www.mastersite.com/cgi-shl/addurl04.pl)*(url={ADD}&SUBMIT=Add It
User-agent: Pepe Search)*(P)*(http://search.pepesearch.com/submitprocess.asp)*(url={URL}&
User-agent: SiteLevel)*(P)*(http://www.sitelevel.com/join.cgi)*(join_email={MAIL}&join_url1={URL}
User-agent: Kerplop Search)*(P)*(http://kerplop.com/)*(t={TITLE}&u={URL}
User-agent: SearchRamp)*(P)*(http://SearchRamp.com/Add.asp)*(LocalSubmission=718&site={URL}&CatNo=0&CountryNo=40&email={MAIL}=PostalCode=00000
Disallow: /

# Zablokuj następujące katalogi i pliki:
User-agent: *
Dis