Serialization / deserialization speed optimizations #357
Description
Hello All,
I am wondering how best to control Message Pack to maximize the speed at which large objects can be read / written to disk. In my specific case the files on disk are approaching 10 GB in size. They contain fairly large Sorted Dictionaries. When I load them into memory the process can take upwards of 6-8 minutes to load. This process is running on a very beefy server with dual CPU totaling 32 cores and having 256 GB of memory. When reading the disk I can see periods of up to 750 MB / second transfer speeds, but there are periods where no disk transfers are occurring even though the load process has not completed. I am wonder if their are some options I should select for the MessagePack Serializer that could improve my performance. When accessing the disks (NVMe in a RAID 5 array) I see a very curious "ringdown" pattern on disk transfers as the load process proceeds. Any help you might suggest would be welcome.
Here is a sample of the code used to load the Dictionaries:
public static T ReadMsgPkFile<T> (string filePath) { using (Stream stream = File.Open (filePath, FileMode.Open, FileAccess.Read, FileShare.Read)) { return (T)MessagePackSerializer.Deserialize<T> (stream); } }
