Suggestion: step0 separate prepare_data into two scripts
It seems to me that the prepare_data script could be separated into two scripts:
- getting the data and formatting them to a "general format"
- preparing the data for disperse (compute pos and distances from cosmological parameters)
I would make the difference because part one is specific to each catalogue: it can not be generalized. But part two seems to be a more general preparation that could be used for other catalogue. Additionally part two seems to be the part where you put some "scientific value", whereas part one is just formatting.
However I am not quite sure where to make the separation, I'd say here: https://gitlab-p4n.aip.de/Nicola.Malavasi/disperse_kufair/-/blob/main/step0_get_data/prepare_data.py#L137
If you are not sure, I could suggest to empirically figure it out. To do so we could create a step0b_get_survey_Y where Y is another survey you are working on. And see how these scripts would differ from each other. In anycase I think we could now try to add another survey to the repo and see how you feel with switching between dependencies.
Unfortunately I currently have little time to improve on SciTrace although, your feed back gave me quite some idea on improving, but I'll do it as soon as possible.
One thing that is useful to know is that creating a package from a new version of your repo still requires that you remove the local copy of the repo (or that you git pull manually from the .kufair/repositories/... repo).
I hope this helps,
Yori