Skip to content

Update robots.txt #198

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
github-actions wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Update robots.txt #198

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions public/robots.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ User-agent: Audisto Crawler
User-agent: AutomatonHeadlessChrome
User-agent: Awario
User-agent: Barkrowler/
User-agent: BeansLLM-CorpusBot
User-agent: bedrockbot
User-agent: BestChange
User-agent: Better Stack
Expand Down Expand Up @@ -107,6 +108,7 @@ User-agent: DataForSeoBot
User-agent: Datenbank Crawler
User-agent: deadlinkchecker
User-agent: DeepSeekBot
User-agent: DentalPresenceAuditBot
User-agent: Digicert DCV
User-agent: DigiCert DCV Bot
User-agent: Dlc/
Expand Down Expand Up @@ -150,13 +152,13 @@ User-agent: grafana/synthetic-monitoring
User-agent: GrapeshotCrawler
User-agent: GroovinaAdsbot
User-agent: GTmetrix
User-agent: GuestpostsBot/
User-agent: HelloworkJobPostingBot
User-agent: Honeybadger Uptime Check
User-agent: http://easyurl.net/monitoring
User-agent: http://www.binarycanary.com
User-agent: http://www.ezoic.com/bot.html
User-agent: Hunyuan
User-agent: ICC-Crawler
User-agent: ImageBot
User-agent: imageSpider
User-agent: InteractiveAdvertisingBureau
User-agent: James BOT
Expand All @@ -181,7 +183,8 @@ User-agent: netEstate NE Crawler
User-agent: omgili
User-agent: PanguBot
User-agent: QwenBot
User-agent: Reflectionbot
User-agent: recraft-image-dataset
User-agent: ReflectionBot
User-agent: RSiteAuditor
User-agent: SalamandraVLM
User-agent: SBIntuitionsBot
Expand All @@ -191,4 +194,5 @@ User-agent: Timpibot
User-agent: VelenPublicWebCrawler
User-agent: w4mwnpbXf3MFAbxOkJRw
User-agent: webzio-extended
User-agent: YiBot
Disallow: /