Canupo - validate Classification (no reference data?)

[eduklupau001]

Hi,

I'm using CloudCompare 2.10.
I'm currently working on my Bachelor thesis. I have a RGB point cloud from a vineyard acquired by a multi-camera system (UAV).
I have to evaluate how well I can train Canupo to distinguish soil / foliage / grape bunches / stem.
So far i got quite a satisfying result for the separation of bare-earth points and grapevine points.
Now I have to validate the result. Most of the papers I read have some kind of reference data, for example acquired by TLS.
I have no reference data so I thought to manually select points and then use those points as reference data.
Unfortunately, I'm a bit stuck. I don't know how to proceed in the validation process. I'm struggling to grasp the "validate_classification" step.
If there's anyone who could give me some food for though or give me a hand somehow I would be really thankful. I read quite a lot but somehow I didn't come to any useful conclusion so far.

Thank you all for your help!
Paula

[Ohlaris]

Hi Paula,

I would like to follow your question as well.
Are you using Canupo classifier tool in CC or you are writing your own with the source code?

Thank you
Kind regards

[eduklupau001]

Sorry for the late reply!

I am using Canupo classifier tool in CC. I trained the classifier using my own trainings data set and then applied it to the whole point cloud. Unfortunately the evaluation of accuracy still bothers me a bit.

Thanks for your interest,
Paula

[lightcuber]

Hi all, I've been using CC to do point cloud classification for the past year and a half. I'm not exactly sure what you mean when you talk about "reference data," but I'm guessing you mean obtaining an independent set of points (not used for training) to evaluation the accuracy of the classification.

Here's what I've been doing:

-I'm not using CANUPO, I'm using my own algorithm which I coded in MATLAB. But the principles are the same. I needed to manually assign training labels, then export them in a .txt to be loaded into MATLAB. I use the snipping tool to accomplish this. Basically, I snip out little clouds for each class, just as you would for training in CANUPO. THEN, once I have a single point cloud for each class, I merge them together and click YES when it asks if I want to create a scalar field with the original cloud index. This gives me a point cloud with a number label for each of the different classes. I can then save this as a .txt and access the labels for each point in MATLAB.

What you could do is perform the above procedure to have a point cloud with your own labels applied to it, then run the CANUPO tool, such that you have scalar fields with BOTH your manual labels AND the CANUPO classification as scalar fields. Then you could export the cloud in ASCII format, load it into MATLAB, and use their statistics tools to calculate things like accuracy, precision, and recall. I use this function, very useful. https://www.mathworks.com/matlabcentral ... p-grouphat

I've gotta run but I'd be happy to continue this conversation. Please let me know if I'm totally off the mark on what you're asking! ;)

Luke

[Ohlaris]

Hi all, I've been using CC to do point cloud classification for the past year and a half. I'm not exactly sure what you mean when you talk about "reference data," but I'm guessing you mean obtaining an independent set of points (not used for training) to evaluation the accuracy of the classification.

Here's what I've been doing:

-I'm not using CANUPO, I'm using my own algorithm which I coded in MATLAB. But the principles are the same. I needed to manually assign training labels, then export them in a .txt to be loaded into MATLAB. I use the snipping tool to accomplish this. Basically, I snip out little clouds for each class, just as you would for training in CANUPO. THEN, once I have a single point cloud for each class, I merge them together and click YES when it asks if I want to create a scalar field with the original cloud index. This gives me a point cloud with a number label for each of the different classes. I can then save this as a .txt and access the labels for each point in MATLAB.

What you could do is perform the above procedure to have a point cloud with your own labels applied to it, then run the CANUPO tool, such that you have scalar fields with BOTH your manual labels AND the CANUPO classification as scalar fields. Then you could export the cloud in ASCII format, load it into MATLAB, and use their statistics tools to calculate things like accuracy, precision, and recall. I use this function, very useful. https://www.mathworks.com/matlabcentral ... p-grouphat

I've gotta run but I'd be happy to continue this conversation. Please let me know if I'm totally off the mark on what you're asking! ;)

Luke

Hi Luke,

That is awesome!
Thanks for replying. This is very exciting. If you could give us more details about your research it would be great. What principles have you used to build your algorithm? Canupo uses Support vector machines so you are using it as well?


I am working with point clouds derived from underwater photogrammetry. Canupo works for identifying some features but it has some limitations so I am searching for other techniques to compare the accuracy within them. I was having doubts about how to perform the ground-truthing to get accuracy results indeed so you have shed some light on it. Thanks

Happy to catch up with you more about this and other topics.


Cheers

[lightcuber]

Hi,

Glad to hear that. For the most details I can direct you to my research gate at the end of this post. You can find my papers there. But I can give a quick summary too so you don't have to read all the boring stuff.

The main applications I'm working on are rockfall hazard and runout mapping/monitoring and other geological natural hazards more generally. So I'm focused on classes that are relevant to that, mainly bedrock areas, tall vegetation, talus, soil/low vegetation or grasses, and snow. My research is on classifying those objects in both lidar and photogrammetry data and evaluating how it could be practically implemented for engineers evaluating rock slope hazards.

Canupo is great as we all know, but I've heard time and time again things like you've said (it works for some classes but not others). I wanted to be able to have a lot more control over the various knobs going on behind the scenes, so it just made more sense to recreate something like it myself externally. This allowed me to test different algorithms (I use Random Forest instead of SVM, tends to be faster and more accurate, plus it's not binary, so I can have one classifier for all 4-5 classes I'm working with instead of a bunch of stacked binary classifiers). The most important thing is I can add more feature types to classify more objects. I added slope features, which makes talus distinguishable from bedrock, and I added intensity from my lidar scanner so it can also filter snow at the same time.

If you're having difficulty classifying objects, you might want to consider implementing your own classifier and adding features that are relevant to the objects of interest. What types of things are you trying to classify?

I'm currently experimenting with color features from photogrammetry data. It's becoming clear to me that photogrammetry point clouds don't work quite as well with geometric features as lidar, but they have pretty spatially accurate color characteristics, so those seem promising to improve classification results (see the following good paper).

Becker, C., Rosinskaya, E., Häni, N., D’Angelo, E., Strecha, C., 2018. Classification of Aerial Photogrammetric 3D Point Clouds. Photogrammetric Engineering & Remote Sensing 84, 287–295. https://doi.org/10.14358/PERS.84.5.287

My researchgate:
https://www.researchgate.net/profile/Luke_Weidner

Either of you are free to hit me up on there if you want to have a more detailed discussion. Happy to continue here as well.

Luke

[Ohlaris]

Hi Luke,


Thanks for the post and sorry for the late reply on this.
I am just after reading your paper "Classification methods for point clouds in rock slope monitoring: A novel machine learning approach and comparative analysis". Thanks for sending this on. Huge congrats on your work! Very happy to read it. It totally shed some light on my work and which has similar aims but for seabed mapping of cold-water corals.

I have some doubts about the process and I think you could be of great help with that. Can I send you a message via research gate ?
I have followed you =)



Cheers!!