link-extractor

How to Extract All Website Links in Python

To run this:

• pip3 install -r requirements.txt

• python link_extractor.py --help
  

  Output:

  usage: link_extractor.py [-h] [-m MAX_URLS] url
  
  Link Extractor Tool with Python
  
  positional arguments:
  url                   The URL to extract links from.
  
  optional arguments:
  -h, --help            show this help message and exit
  -m MAX_URLS, --max-urls MAX_URLS
                          Number of max URLs to crawl, default is 30.
  

• For instance, to extract all links from 2 first URLs appeared in github.com:

  python link_extractor.py https://github.com -m 2
  

  This will result in a large list, here is the last 5 links:

  [!] External link: https://developer.github.com/
  [*] Internal link: https://help.github.com/
  [!] External link: https://github.blog/
  [*] Internal link: https://help.github.com/articles/github-terms-of-service/
  [*] Internal link: https://help.github.com/articles/github-privacy-statement/
  [+] Total Internal links: 85
  [+] Total External links: 21
  [+] Total URLs: 106
  

  This will also save these URLs in github.com_external_links.txt for external links and github.com_internal_links.txt for internal links.