Improvement: step0 possible memory improvement

This seems quite inefficient to me

#Empty list which will host all the temporary tables
all_tab = []

#Cycle all the files and read them
for fn in mgs_filelist_sorted:
    temptab = tbl.Table.read(fn, format = 'fits')
    all_tab.append(temptab)
    temptab = 0. #Free resources 
    
#Vstack the tables in a single table
legacy_mgs_masked = tbl.vstack(all_tab, join_type = 'exact')

#Remove all_tab to save memory
all_tab = 0.

I am pretty sure there is a more elegant way to do so.

for each file name you open the fit file and load it into memory. Then you make a copy of it and a copy of all already read files and join it together. So at max you use already 2 times the memory you actually need. Still this is good that you then free the memory of the read table.

Then to merge the table together you again make a full copy of your table, so again twice what you actually need. That is somehow fine since you already needed it, so it do not get worse, just as bad.

My question is therefore if it won't be possible to create your array beforehand with the right size, and then just read the files into legacy_mgs_masked directly. In such way you would let astropy taking care of the cache/buffer and reserve only the memory you really use/need.

Again maybe that is not an issue, if the files are small enough.