[handecelikkanat]

• Reads the top domains table from a CSV published with each crawl, instead of computing it from the crawl statistics,
• Extends the table from 500 to 1000 domains.

Changes

• get_stats.sh: also downloads the top domains file of the latest crawl,
  s3://commoncrawl/crawl-analysis/<crawl>/stats/domains-top-1000-extended.csv.gz, into
  stats/top-domains/. Only the latest crawl's file is fetched, since only that one is
  plotted. The script now fails if the latest crawl's stats file is missing or not valid
  gzip, e.g. an error page saved by curl. For testing, the source location can be
  overridden with TOP_DOMAINS_S3 or TOP_DOMAINS_URL.
• plot/domain.py: reads the downloaded file (domain, pages, urls, hosts) and checks its
  columns. %pages and %urls are computed from the crawl totals as before.
• plot.sh: domain.py now only needs the crawl size records, so it reads
  stats/excerpt/size.json.gz.
• plots/domains-top-500.* renamed to plots/domains-top-1000.*, regenerated with
  CC-MAIN-2026-39. plots/domains.md and index.md updated accordingly.
• .gitignore: ignore stats/top-domains/, created by get_stats.sh.

Changes in the data

• Pages and URLs match the previous table exactly for all 500 domains (checked on
  CC-MAIN-2026-39).
• Hosts are slightly lower for some domains, 33 of 500 for CC-MAIN-2026-39, by 49 in
  total. The previous counts were approximate: a host whose URLs were split across two
  input shards was counted twice, cf. Host counts can be off by +1 (higher than actual) #14 . The new counts are exact, from the columnar index.

Breaking change

Links to plots/domains-top-500.csv and plots/domains-top-500.html stop working.
The 500 versions are no longer provided; the data page links the new files. The change
will be announced together with the upcoming Croissant metadata release.

Notes

• stats/excerpt/domain.json.gz is still built, although domain.py no longer reads it.
  Kept in case it is used elsewhere.

[handecelikkanat]

@sebastian-nagel I intend to not touch crawlstats.py and run_stats_hadoop.py for now. They will continue calculating the statistics for top 500 domains (which can be updated to 1000?)

I will update plot/domain.py to use data from my own files rather than from crawl.gz files. Feels like less amount of change in one go.

Let me know if you prefer to update everything all at once. I thought if current system fails, its good to have the old one also running in parallel for another month. Plus I think some other stats calculations use these domain numbers, which I dont want to mess with.

Opinions?

[sebastian-nagel]

Thanks, @handecelikkanat. Looks good.

Don't forget: the top-500 domain files (CSV and HTML) need to be removed from Git and replaced by the top-1000 files. This should be part of the PR.

[handecelikkanat]

Don't forget: the top-500 domain files (CSV and HTML) need to be removed from Git and replaced by the top-1000 files. This should be part of the PR.

• Yes this makes sense, thank you. How is better to handle this? Im always confused which was is clearer for git history. Options: (1) rm 500-files + add 1000-files, (2) or mv (i.e., rename) 500-files to 1000-files.

• I will leave this PR open, it is to be merged at the end of October crawl.
  (I dont know if it is better to mark as Draft in the meanwhile.)

[handecelikkanat]

@sebastian-nagel Converted internal (extended) file to compressed.

Ready for your review. Intentionally left as "Draft" so I dont accidentally merge before completing the To-Do items after crawl, noted above with checkboxes to tick. Those will create the new plots/domains-top-1000.csv and plots/domains-top-1000.html files to add to the PR.

[sebastian-nagel]

Thanks, @handecelikkanat.

Looks good. The many errors when running get_stats.sh are somewhat annoying.

[handecelikkanat]

@sebastian-nagel Regarding approximate numbers in pre-change version of cc-crawl-statistics (#14):

Im seeing a bit too big changes in the current version for mit.edu, unam.mx: 4 and 5 respectively.

Is this normal to get so many differences? Do we get so many results from those?

aliyun.com 114 115 -1
amazonaws.com 4521 4523 -2
apple.com 112 113 -1
asu.edu 646 647 -1
bg.ac.rs 546 548 -2
blogspot.com 187899 187901 -2
bme.hu 442 443 -1
ca.gov 1125 1127 -2
cuni.cz 769 770 -1
cvut.cz 428 430 -2
discord.com 3 4 -1
europa.eu 755 756 -1
free.fr 2334 2335 -1
gov.hr 119 120 -1
in.gov 250 251 -1
kit.edu 675 676 -1
krakow.pl 1224 1225 -1
ku.edu 605 606 -1
mit.edu 1255 1259 -4
mpg.de 515 516 -1
msu.ru 519 520 -1
ntnu.edu.tw 235 239 -4
ntu.edu.tw 523 524 -1
org.ru 476 477 -1
spb.ru 3281 3282 -1
ubc.ca 1780 1781 -1
ucsb.edu 810 811 -1
unam.mx 1131 1136 -5
usf.edu 147 148 -1
usp.br 1116 1117 -1
uwaterloo.ca 176 178 -2
vice.com 8 9 -1
wixsite.com 87267 87268 -1
33 rows differ, sum -49

[handecelikkanat]

@lfoppiano TYVM for the excellent suggestions! I applied some of yours and kept some of mine:

#	Finding	Response
1	Latest-only download in get_stats.sh	On Sebastian's review I already did this change, kept mine.
1a	Exit code of the latest crawl's stats download	Excellent catch, applied ty.
2	Sort by pages in plot/domain.py	Opted to not use. Sort order is defined by Athena query in crawl pipeline atm. If we sort also here, it will silently overwrite the decision made there. Better restrict sorting decision to one place.
3	File and column checks in plot/domain.py	Applied the column check (good catch!). Didn't use existence check, since pandas already raises FileNotFoundError on read attempt and the error does list the expected path.
4	Gzip byte check in plot/domain.py	Good catch, ty, this check was redundant, removed as in your commit.
5	File name and N=1000 duplicated	Leaving as is for now.
6	Rollout warnings	(1) The domains-top-1000.* files appear after I run this code with September's data (after I backfill the data to public bucket and before we merge for October). (2) plots/ vs data/ mix-up: Fixed. (3) Links to the old domains-top-500.* files will stop working: I accept this result. It is clearer for the users that we are from now supplying top-1000 files rather than top-500. It would be more confusing to publish both a top-500 and a top-1000 file. I'll make sure to note in the PR description and in the related announcement.
7	Unused domain.json.gz excerpt	I kept this deliberately in case this is still utilized elsewhere, noted in the PR description.
8	No tests for plot/domain.py	Skipping for now as this would require another review round, but good idea, lets come back to this in another PR, ty.

[lfoppiano]

LGTM