HTTPS surface reachable (robots ✓, sitemap ✓, title ✓)
Why it matters: Public files — robots.txt, sitemap.xml, head meta — are what attackers see first during reconnaissance. Misadvertised paths, stale sitemaps, and verbose generators leak more than intended (ISO 27001 A.8.9).
robots.txt
present
User-Agent: *
Disallow:/airtellive/*
Disallow:/growth-cms-pages/*
Disallow:/app*
Disallow:/mobile*
Disallow:/forme*
Disallow:/aes*
Disallow:/m/*
Disallow:/hfz-hellotunes/*
Disallow:/Y/*
Disallow:/4g_lte/*
Disallow:/ga*
Disallow:/personal/*
Disallow:/wps/*
Disallow:/prepaid-recharge/complete
Disallow:/postpaid-bill-pay/complete
Disallow:/index-all/product-listing.html
Allow:/bank/*
Allow: /digital-store/
Disallow:/blog/author/*/*/*
Disallow:*&*
Disallow:*/digital-store/*/*utm*
Disallow:/business/b2b/airtel-iq/api-docs/**
# ============================
# AI / LLM Crawlers (Allowed)
# ============================
User-agent: GPTBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: bingbot
Allow: /
User-agent: BingPreview
Allow: /
User-agent: bingGPT
Allow: /
User-agent: Googlebot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Bard
Allow: /
User-agent: Gemini
Allow: /
User-agent: ClaudeBot
Allow: /
# ============================
# Restricted Crawlers
# ============================
User-Agent: Slurp
Disallow: /
# ============================
# Sitemap Directives
# ============================
Sitemap: https://www.airtel.in/sitemap.xml
Sitemap: https://www.airtel.in/infinity-sitemap.xml
Sitemap: https://www.airtel.in/isp-sitemap.xml
Sitemap: https://www.airtel.in/broadband-sitemap.xml
Sitemap: https://www.airtel.in/cms-sitemap.xml
Sitemap: https://www.airtel.in/dth-sitemap.xml
Sitemap: https://www.airtel.in/bank/sitemap.xml
Sitemap: https://www.airtel.in/growth-sitemaps/growth-sitemap6.xml
Sitemap: https://www.airtel.in/growth-sitemaps/growth-sitemap5.xml
Sitemap: https://www.airtel.in/growth-sitemaps/growth-sitemap4.xml
Sitemap: https://www.airtel.in/growth-sitemaps/growth-sitemap3.xml
Sitemap: https://www.airtel.in/growth-sitemaps/growth-sitemap2.xml
Sitemap: https://www.airtel.in/growth-sitemaps/growth-sitemap1.xml
Sitemap: https://www.airtel.in/business-sitemap.xml
sitemap.xml
present — 97 url(s)
head
- title
- Airtel: Wi-Fi, Postpaid, Prepaid, DTH, Bank & Business
- description
- We are India
social
no OpenGraph or Twitter meta tags found