Huge temp data can not be compressed with NTFS LZ77

Give some feedback on openEMS.

Moderators: thorsten, sebastian

Post Reply
Hale_812
Posts: 172
Joined: Fri 13 May 2016, 02:54

Huge temp data can not be compressed with NTFS LZ77

Post by Hale_812 » Tue 02 Aug 2016, 06:24

I found a strange thing. Temporary data text files can take an awful amount of space, flooding all the hard disk, but can be compressed with 35% ZIP ratio.
So, it is natural to use disk compression here.
BUT, when I enable Windows LZ77 compression, the space is not reduced at all! When I work with HFSS, it is usually reduced twice, but not with OpenEMS.
Why? I guess, because of incremental writing method, where the data is not buffered enough and not cached well, before going to compression layer.
Is it possible to improve the situation by modifying dump-data write techniques? Maybe writing incrementally with FILE_ATTRIBUTE_TEMPORARY would help to hold the data in RAM, until there is some free RAM left?

thorsten
Posts: 1411
Joined: Mon 27 Jun 2011, 12:26

Re: Huge temp data can not be compressed with NTFS LZ77

Post by thorsten » Tue 02 Aug 2016, 08:04

Can you be more specific, which data do you write? Because vtk and hdf5 are both already binary data. And coming from a numerical simulation they tent to look like noise and thus are hardly compressible.
Thus I do not expect NTFS to do much about it.
A different story are the voltage/current probes as they are plain ASCII files. But they tend to be small...

regards
Thorsten

Hale_812
Posts: 172
Joined: Fri 13 May 2016, 02:54

Re: Huge temp data can not be compressed with NTFS LZ77

Post by Hale_812 » Tue 02 Aug 2016, 10:49

I am using your sphere scattering example as a basis for my scattering research.
So, there are model, FF box and Et dump in the temp folder. It grows beyond 150Gb, hitting the limit I got from admin at the server. I tried to compress it and it got reduced by at least 30%.
When I tried to run new simulation, nothing got reduced. I guess, it is because the data was written in portions smaller than the compression block, and ntfs.sys has reacted very wrong.

thorsten
Posts: 1411
Joined: Mon 27 Jun 2011, 12:26

Re: Huge temp data can not be compressed with NTFS LZ77

Post by thorsten » Tue 02 Aug 2016, 17:10

I tried to compress it and it got reduced by at least 30%.
I presume something like zip? I think the ntfs build-in compression is not quite as good.
I don't think that is has something to do with the small blocks, rather that the data indeed is not easily compressible. But that is only my feeling. In any case 30% less is still enormous...
You should think about the data you store and how.
Which file is so big? For example is it necessary to record Et in 3D? Consider using a sub-sampling? Consider dumping in frequency domain rather than in time-domain, especially if your simulation runs a long time (many time steps).

regards
Thorsten

Hale_812
Posts: 172
Joined: Fri 13 May 2016, 02:54

Re: Huge temp data can not be compressed with NTFS LZ77

Post by Hale_812 » Wed 03 Aug 2016, 06:39

no, just applied folder compression first.
The next time it did not shrink when written into the compression-enabled folder.

Zip(RAR's zip code) says 30%, when estimated.

Concerning the dump data, I did not change anything. In your example, you estimate the incoming power from that dump. So, I am not sure, it can be removed safely. And yes, I have a combination of 3D objects, which can not be reduced to 2D scattering easily.

thorsten
Posts: 1411
Joined: Mon 27 Jun 2011, 12:26

Re: Huge temp data can not be compressed with NTFS LZ77

Post by thorsten » Wed 03 Aug 2016, 10:46

I still miss the information which files get this big.

vtk files in are each written in one go, no small portions. Same holds for hdf5 files. The hdf5 interface writes in chunks, but usally that means large blocks.
Both are handled by there respective libraries and there is little I can do about it.
The only files that gets bigger in small portions are the voltage/current probe ASCII files, but they tend to be small and thus aren't the problem?

There are many possibilities to fine grain the settings what shall be dumped. I'm pretty sure you need to put more effort in fine-tuning these settings to reduce the overall amount of data stored.

regards
Thorsten

Hale_812
Posts: 172
Joined: Fri 13 May 2016, 02:54

Re: Huge temp data can not be compressed with NTFS LZ77

Post by Hale_812 » Fri 05 Aug 2016, 07:34

Again, I am sorry for early panic.
These files are n2ff_E_##_h5. And I was wrong. Those just were never that big, so I had impression of good compressability earlier.
Now, when I plot RCS against frequency, those files grow enormously. And it looks like they are already compressed, or can' be compressed for some reason.

Is there a way to configure disk free space reserve to avoid system malfunction when these files hit the limit?

Post Reply