Unable to read list of urls in file generated by subfinder

[ghexo]

Hi, when I run this request, it is unable to read my subdomains.txt file generated by subfinder, the urls are the same as they always have been, I have tested my permissions and I am able to read the urls via command line to grab them, however httpx is unable to do the same. Any suggestions? I'm assuming it's my problem since no one else seems to have this issue. Thanks in advance. :)

httpx -l subdomains.txt -o subdomains3.txt

__    __  __       _  __

/ /_ / // /_____ | |/ /
/ __ / __/ __/ __ | /
/ / / / // // // / |
// //_/_/ .//||
//

            projectdiscovery.io

[INF] Current httpx version v1.4.0 (latest)

(that is where the command ends, it just stops without reading)

this is what running verbose looks like:

': [urlutil:RUNTIME] failed to parse input url <- parse "\x00x\x000\x007\x00g\x00l\x00a\x00i\x00d\x00s\x00f\x00s\x001\x00a\x00.\x00i\x00d\x00n\x00.\x00d\x00b\x00s\x00.\x00c\x00o\x00m\x00\r\x00": net/url: invalid control character in URL
got [urlutil:RUNTIME] failed to parse input url <- parse "\x00x\x001\x001\x00g\x00l\x00a\x00t\x00w\x00s\x00t\x00g\x001\x00a\x00.\x00t\x00w\x00n\x00.\x00d\x00b\x00s\x00.\x00c\x00o\x00m\x00\r\x00": net/url: invalid control character in URL in unsafe:false
': [urlutil:RUNTIME] failed to parse input url <- parse "\x00x\x001\x001\x00g\x00l\x00a\x00t\x00w\x00s\x00t\x00g\x001\x00a\x00.\x00t\x00w\x00n\x00.\x00d\x00b\x00s\x00.\x00c\x00o\x00m\x00\r\x00": net/url: invalid control character in URL
got [urlutil:RUNTIME] failed to parse input url <- parse "\xff\xfeu\x00a\x00t\x00-\x00d\x00e\x00p\x00l\x00o\x00y\x00.\x00t\x00a\x00p\x00t\x00a\x00p\x00b\x00u\x00n\x00n\x00y\x00e\x00n\x00g\x00i\x00n\x00e\x00.\x00l\x00u\x00d\x00o\x00c\x00l\x00o\x00u\x00d\x00.\x00s\x00i\x00n\x00.\x00a\x00u\x00t\x00o\x00.\x00n\x00o\x00n\x00p\x00r\x00o\x00d\x00.\x00p\x00c\x000\x00.\x00d\x00b\x00s\x00.\x00c\x00o\x00m\x00\r\x00": net/url: invalid control character in URL in unsafe:false
': [urlutil:RUNTIME] failed to parse input url <- parse "\xff\xfeu\x00a\x00t\x00-\x00d\x00e\x00p\x00l\x00o\x00y\x00.\x00t\x00a\x00p\x00t\x00a\x00p\x00b\x00u\x00n\x00n\x00y\x00e\x00n\x00g\x00i\x00n\x00e\x00.\x00l\x00u\x00d\x00o\x00c\x00l\x00o\x00u\x00d\x00.\x00s\x00i\x00n\x00.\x00a\x00u\x00t\x00o\x00.\x00n\x00o\x00n\x00p\x00r\x00o\x00d\x00.\x00p\x00c\x000\x00.\x00d\x00b\x00s\x00.\x00c\x00o\x00m\x00\r\x00": net/url: invalid control character in URL

even though there is no strange characters in the url list, this is what it says...

thanks for any help :)

[olearycrew]

Any chance you could share (either here or on DM on our Discord server https://nux.gg/discord) the input file?

[ghexo]

the input file looks like this:

example.com
subdomain.example.com
subdomain.subdomain.example.com
subdomain.example.com
subdomain.subdomain.example.com

Unfortunately I am unable to share the input file as it is the BB program I am currently working on. However, this is the gist of what it is. It's basically the results from the Subfinder scan. So it's the same as any other scan using Subfinder. No results have https:// or http:// on them, just simply the domain without that. Thanks so much in advance. :)

[olearycrew]

My concern is - what is the encoding of that file. I wonder if the encoding is giving it an issue.

What's the output of file -bi subdomains.txt

[dogancanbakir]

The file is UTF-16 (the \xff\xfe at the start), which Windows PowerShell writes with >. #2706 decodes it. Until it is released, save the file as UTF-8: Get-Content subdomains.txt | Set-Content -Encoding utf8 subdomains-utf8.txt.