Does lan-cache compress content downloaded?

[CyberSecGuy]

I'm downloading PUBG, and frequently running sudo du -h -d 1 data/cache The disk space does not reflect the amount downloaded according to Steam.

https://i.imgur.com/e7eDARL.png

The game is 13GB, but only 12GB is used on data/cache/steam. I have downloaded/deleted/redownloaded 4-5 times now just to ensure all the files are present.

[RyanEwen]

Did the cache grow to 13GB after your 4-5 runs?

I'm sad to say that while this is my repo and I am familiar with NGINX as a web server, I do not fully understand how the NGINX caching mechanism works. This project is based https://github.com/OpenSourceLAN/origin-docker with the biggest changes being organizational.

[CyberSecGuy]

Hey there, currently have PUBG and now GTA V installed, and yet the data/cache/steam reports 53G of data. Which is concerning considering GTA V itself is upwards of 72Gb. Blizzard on the other hand does appear to be the appropriate size.

[RyanEwen]

I wonder if there are new steam content servers that aren't listed in the conf files.

I would like to change how steam in particular is cached, so that it's more like the steamcache project. But I don't get much time to play with this :(

[CyberSecGuy]

I'll look into it in a bit when I have some free time. I also am thinking about seeing if there's a way to cache Netflix/Hulu etc.

[Vykyan]

I love Ryan's lan-cache-docker because it gets you up and running with a heap of cacheing for basically a few terminal lines. But i've had trouble with the steam parts.

I had problems because my server is a little slower so with the way lan-cache-docker handles steam I only get about 20Mb/s at most and it misses about 1/3 of the data every download, where all the others (uPlay/origin) get around 50-100Mb/s and usually download 99.99-100% of the data to my server.

This is not Ryan's fault this is because the way the original lan-cache handles Steam's chunks is a lot slower and uses a lot more cpu power. Where as steamcache handles Steam's cache in a unique (more native steam) approach.

So my solution was to disable the nginx steam part of lan-cache-docker and use an alternate steamcache just for steam.

**NOTE: X.X.X.2 and X.X.X.3 = replace with your own network ie./ 192.168.0.2 and 192.168.0.3 or 172.16.0.2 and 172.16.0.3 or 10.0.0.2 and 10.0.0.3 etc...

• My lan-cache-docker runs on X.X.X.2 ip. I used systemd-networkd to give my ethernet card another IP Address. aka X.X.X.3 which I will use for steamcache.

• I've setup lan-cache-docker on X.X.X.2:80 , and kept the steam.dnsmasq.conf but deleted the steam.nginx.conf.

• Then I run in a seperate docker container using "https://hub.docker.com/r/steamcache/steamcache/" and run that on X.X.X.3:80

ie./

docker run --restart unless-stopped --name steamcache -p X.X.X.3:80:80 -v /path/to/cache:/data/cache -v /path/to/logs:/data/logs steamcache/steamcache:latest

/path/to/cache = where to store cache data
/path/to/logs = where to store cache logs

NOTE: then before you start steamcache;

mkdir /path/to/cache/depot && chmod 777 -R /path/to/cache/depot

(a little fix for steamcache)

• I then changed data/conf/steam.dnsmasq.conf and replaced all the instances of "/NGINX_IP" to "/X.X.X.3".

I then start the steamcache after the lan-cache-docker and it all just works.

With this setup using steamcache to handle only steam I consistently max out my gigabit ethernet and get close to 100% download when using steamcache's version.

(And you can see whats installed because steamcache stores all the downloaded data for a single game/app in ;

/path/to/cache/depot/APPID/

)

[RyanEwen]

Curious if we could change the nginx settings for stream specifically, to work the same way steamcache does things. Iirc they use nginx as well.

[Vykyan]

Curious if we could change the nginx settings for stream specifically, to work the same way steamcache does things. Iirc they use nginx as well.

I've tried repeatedly to do this but they have different setup options set for nginx which don't seem to work with the way lan-cache-docker's nginx works :(

[kregadam1]

I love Ryan's lan-cache-docker because it gets you up and running with a heap of cacheing for basically a few terminal lines. But i've had trouble with the steam parts.

I had problems because my server is a little slower so with the way lan-cache-docker handles steam I only get about 20Mb/s at most and it misses about 1/3 of the data every download, where all the others (uPlay/origin) get around 50-100Mb/s and usually download 99.99-100% of the data to my server.

This is not Ryan's fault this is because the way the original lan-cache handles Steam's chunks is a lot slower and uses a lot more cpu power. Where as steamcache handles Steam's cache in a unique (more native steam) approach.

So my solution was to disable the nginx steam part of lan-cache-docker and use an alternate steamcache just for steam.

**NOTE: X.X.X.2 and X.X.X.3 = replace with your own network ie./ 192.168.0.2 and 192.168.0.3 or 172.16.0.2 and 172.16.0.3 or 10.0.0.2 and 10.0.0.3 etc...

• My lan-cache-docker runs on X.X.X.2 ip. I used systemd-networkd to give my ethernet card another IP Address. aka X.X.X.3 which I will use for steamcache.
• I've setup lan-cache-docker on X.X.X.2:80 , and kept the steam.dnsmasq.conf but deleted the steam.nginx.conf.
• Then I run in a seperate docker container using "https://hub.docker.com/r/steamcache/steamcache/" and run that on X.X.X.3:80

ie./

docker run --restart unless-stopped --name steamcache -p X.X.X.3:80:80 -v /path/to/cache:/data/cache -v /path/to/logs:/data/logs steamcache/steamcache:latest

/path/to/cache = where to store cache data
/path/to/logs = where to store cache logs

NOTE: then before you start steamcache;

mkdir /path/to/cache/depot && chmod 777 -R /path/to/cache/depot

(a little fix for steamcache)

• I then changed data/conf/steam.dnsmasq.conf and replaced all the instances of "/NGINX_IP" to "/X.X.X.3".

I then start the steamcache after the lan-cache-docker and it all just works.

With this setup using steamcache to handle only steam I consistently max out my gigabit ethernet and get close to 100% download when using steamcache's version.

(And you can see whats installed because steamcache stores all the downloaded data for a single game/app in ;

/path/to/cache/depot/APPID/

)
how did you fix the port binding problem and set each container to a specific IP

[RyanEwen]

Hey guys I just took at look at https://github.com/OpenSourceLAN/origin-docker to see if there's been any work there and I noticed a few new commits. This commit is especially interesting:

OpenSourceLAN/origin-docker@7f8a63b

I wonder if we're just missing some new servers. This doesn't help with the speed of Steam caching but may help with completion.

Again, I don't use this project for personal use very often so it's hard for me to find the time to debug this Steam issue.. but if someone wants to give it at shot by adding the new servers to the config, I would happily approve a pull-request to merge in those changes if they fix it.

[Vykyan]

I love Ryan's lan-cache-docker because it gets you up and running with a heap of cacheing for basically a few terminal lines. But i've had trouble with the steam parts.
I had problems because my server is a little slower so with the way lan-cache-docker handles steam I only get about 20Mb/s at most and it misses about 1/3 of the data every download, where all the others (uPlay/origin) get around 50-100Mb/s and usually download 99.99-100% of the data to my server.
This is not Ryan's fault this is because the way the original lan-cache handles Steam's chunks is a lot slower and uses a lot more cpu power. Where as steamcache handles Steam's cache in a unique (more native steam) approach.
So my solution was to disable the nginx steam part of lan-cache-docker and use an alternate steamcache just for steam.
**NOTE: X.X.X.2 and X.X.X.3 = replace with your own network ie./ 192.168.0.2 and 192.168.0.3 or 172.16.0.2 and 172.16.0.3 or 10.0.0.2 and 10.0.0.3 etc...

• My lan-cache-docker runs on X.X.X.2 ip. I used systemd-networkd to give my ethernet card another IP Address. aka X.X.X.3 which I will use for steamcache.
• I've setup lan-cache-docker on X.X.X.2:80 , and kept the steam.dnsmasq.conf but deleted the steam.nginx.conf.
• Then I run in a seperate docker container using "https://hub.docker.com/r/steamcache/steamcache/" and run that on X.X.X.3:80

ie./
docker run --restart unless-stopped --name steamcache -p X.X.X.3:80:80 -v /path/to/cache:/data/cache -v /path/to/logs:/data/logs steamcache/steamcache:latest
/path/to/cache = where to store cache data
/path/to/logs = where to store cache logs
NOTE: then before you start steamcache;
mkdir /path/to/cache/depot && chmod 777 -R /path/to/cache/depot
(a little fix for steamcache)

• I then changed data/conf/steam.dnsmasq.conf and replaced all the instances of "/NGINX_IP" to "/X.X.X.3".

I then start the steamcache after the lan-cache-docker and it all just works.
With this setup using steamcache to handle only steam I consistently max out my gigabit ethernet and get close to 100% download when using steamcache's version.
(And you can see whats installed because steamcache stores all the downloaded data for a single game/app in ;
/path/to/cache/depot/APPID/
)
how did you fix the port binding problem and set each container to a specific IP

followed; https://www.shaneburger.net/lancache-step1/

[RyanEwen]

Hey guys I just took at look at https://github.com/OpenSourceLAN/origin-docker to see if there's been any work there and I noticed a few new commits. This commit is especially interesting:

OpenSourceLAN/origin-docker@7f8a63b

I wonder if we're just missing some new servers. This doesn't help with the speed of Steam caching but may help with completion.

Again, I don't use this project for personal use very often so it's hard for me to find the time to debug this Steam issue.. but if someone wants to give it at shot by adding the new servers to the config, I would happily approve a pull-request to merge in those changes if they fix it.

Came across another updated list of hostnames that might help:
23af075

If anyone gets a chance to add these new hostnames to the steam.dnsmasq.conf and steam.nginx.conf files and it helps, I'll happily accept a pull request. If you don't know how to do a pull request then just paste the contents of the updated files in here and I'll update the ones in the repo.