usteki siteadresi.com daki bölume kendi sitemizi yazıyor..
ve robots.txt oalrak kayıtediyoruz..
son olarakta ftp mizde host sayfasına yani puplic html yada http docsun olduğu bölume atıyoruz...böylece google sitenizi daha kolay indexlemiş oluyor
işte size kod
Kod:
## robots.txt generated by http://www.siteadresi.com robot generator# Please, we do NOT allow nonauthorized robots any longer.#User-agent: asteriasDisallow:User-agent: BackDoorBot/1.0Disallow:User-agent: Black HoleDisallow:User-agent: BlowFish/1.0Disallow:User-agent: BotALotDisallow:User-agent: BuiltBotToughDisallow:User-agent: Bullseye/1.0Disallow:User-agent: BunnySlippersDisallow:User-agent: CegbfeiehDisallow:User-agent: CheeseBotDisallow:User-agent: CherryPickerDisallow:User-agent: CherryPickerElite/1.0Disallow:User-agent: CherryPickerSE/1.0Disallow:User-agent: CopyRightCheckDisallow:User-agent: cosmosDisallow:User-agent: CrescentDisallow:User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0Disallow:User-agent: DittoSpyderDisallow:User-agent: EmailCollectorDisallow:User-agent: EmailSiphonDisallow:User-agent: EmailWolfDisallow:User-agent: EroCrawlerDisallow:User-agent: ExtractorProDisallow: User-agent: FoobotDisallow: User-agent: Googlebot Disallow: User-agent: Gezginbot Disallow: User-agent: Harvest/1.5Disallow: User-agent: hloaderDisallow: User-agent: httplibDisallow:User-agent: humanlinksDisallow: User-agent: InfoNaviRobotDisallow: User-agent: JennyBotDisallow: User-agent: Kenjin SpiderDisallow: User-agent: Keyword Density/0.9Disallow: User-agent: LexiBotDisallow: User-agent: libWeb/clsHTTPDisallow: User-agent: LinkextractorProDisallow: User-agent: LinkScan/8.1a UnixDisallow: User-agent: LinkWalkerDisallow: User-agent: LNSpiderguyDisallow: User-agent: lwp-trivialDisallow: User-agent: lwp-trivial/1.34Disallow: User-agent: Mata HariDisallow: User-agent: Microsoft URL Control - 5.01.4511Disallow: User-agent: Microsoft URL Control - 6.00.8169Disallow: User-agent: MIIxpcDisallow: User-agent: MIIxpc/4.2Disallow: User-agent: Mister PiXDisallow: User-agent: mogetDisallow: User-agent: moget/2.1Disallow: User-agent: mozilla/4Disallow: User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)Disallow: User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 95)Disallow: User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 98)Disallow: User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT)Disallow: User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows XP)Disallow: User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000)Disallow: User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows ME)Disallow: User-agent: mozilla/5Disallow: User-agent: NetAntsDisallow: User-agent: NICErsPRODisallow: User-agent: Offline ExplorerDisallow: User-agent: OpenfindDisallow: User-agent: Openfind data gathereDisallow: User-agent: ProPowerBot/2.14Disallow: User-agent: ProWebWalkerDisallow: User-agent: QueryN ****searchDisallow: User-agent: RepoMonkeyDisallow: User-agent: RepoMonkey Bait & Tackle/v1.01Disallow: User-agent: RMADisallow: User-agent: SiteSnaggerDisallow: User-agent: SpankBotDisallow: User-agent: spannerDisallow: User-agent: suzuranDisallow: User-agent: Szukacz/1.4Disallow: User-agent: TeleportDisallow: User-agent: TeleportProDisallow: User-agent: TelesoftDisallow: User-agent: The IntraformantDisallow: User-agent: TheNomadDisallow: User-agent: TightTwatBotDisallow: User-agent: TitanDisallow: User-agent: toCrawl/UrlDispatcherDisallow: User-agent: True_RobotDisallow: User-agent: True_Robot/1.0Disallow: User-agent: turingosDisallow: User-agent: URLy WarningDisallow: User-agent: VCIDisallow: User-agent: VCI WebViewer VCI WebViewer Win32Disallow: User-agent: Web Image CollectorDisallow: User-agent: WebAutoDisallow: User-agent: WebBanditDisallow: User-agent: WebBandit/3.50Disallow: User-agent: WebCopierDisallow: User-agent: WebEnhancerDisallow: User-agent: WebmasterWorldForumBotDisallow: User-agent: WebSaugerDisallow: User-agent: Website QuesterDisallow: User-agent: Webster ProDisallow: User-agent: WebStripperDisallow: User-agent: WebZipDisallow: User-agent: WebZip/4.0Disallow: User-agent: WgetDisallow: User-agent: Wget/1.5.3Disallow: User-agent: Wget/1.6Disallow: User-agent: WWW-Collector-EDisallow: User-agent: Xenu'sDisallow: User-agent: Xenu's Link Sleuth 1.1cDisallow: User-agent: ZeusDisallow: User-agent: Zeus 32297 Webster Pro V2.9 Win32Disallow: