Out of memory while doing merge

[jackkirk]

I have some pretty large jobs involving 50 to 100 scans of 60 million coloured points each, for a total of 3 to 6 billion points per job.

I'm exploring various techniques to winnow this data down to something of the order 1.5 billion points.

I attempted to merge all of them together but ultimately failed with an "out of memory" error.

I have a pretty big rig with 128GB RAM and I have set paging space to 800GB (initial) to 1TB (max)

Watching the Task Manager > Performance > Memory panel while this merge is going on and it indicates "Committed" RAM never goes much more than about 200GB.

Any suggestions as to what is going on?

I'm using 2.12(alpha) but it does the same on 2.11.3

Thanks...

[daniel]

Sadly we have a theoretical limit of 2 billion points per cloud... So I guess you would have to do this in multiple steps...

[jackkirk]

OK good to know - maybe a slightly more explanatory message would be appropriate?

[daniel]

Well, the trick is that by default we don't know why the memory allocation failed (it's more generally because the memory is full, but in this very specific case it's probably more because of the number of points). So it would be definitely possible to also test the number of points, but it would be a huge pain... I would rather make a version of CC that handles more than 2 billion points (much easier, but it would also use slightly more memory...).

[jackkirk]

Given my specific job requirements a version that handles more than 2 billion points would get my vote - 6 billion?

[daniel]

Oh in this case it would be a huuuuuge number (as we would use 64bits instead of 32bits for indexing the points).

[jackkirk]

So not going to happen?

[WargodHernandez]

It's a worthwhile request, I would be interested pursuing.
There is a lot to change though to move from unsigned to something like uint64_t so I would expect to switch to a new using type. It will need to be changed first in CCLib before CloudCompare

We also will need to find any use of int as an index in the codebase

Any objections to a path like this Daniel? Adding it into compile options similar to point size float/double selection.

[daniel]

No I think this is definitely of interest. But probably a huge task!

[harveyshideout]

Hey, did anyone work a way to do huge files, i have a giant scan of the Mayflower but will crash CC or run out of mem. Is there a way to pre decimate the clouds as they import? As all I need is the geometry points for retopology.

thank you

Chris