Improvement: step0 possible memory improvement

Here you make a copy of legacy_mgs_masked and as I understood it, the later is quite large.

#Table has masked values: will fill them with NaNs
legacy_mgs_NS = legacy_mgs_masked.filled(np.NaN)

#Free memory
legacy_mgs_masked = 0.

That is why you then free the memory.

I think this step is not necessary (I am not entirely sure). The reason of the copy when you call legacy_mgs_masked.filled is that legacy_mgs_NS is not a masked array anymore but a ndarray.

However I don't think that there are any reason not to continue with a masked array.

So if the only reason of calling legacy_mgs_masked.filled(np.NaN) is to make sure that the masked values are replace with np.NaN the following is the proper approach:

legacy_mgs_masked.set_fill_value(value=np.NaN)

This will set the fill value to NaN inplace, no copy. And you keep a masked array which is much more efficient (stores less data)

I hope it helps.

In case you get issues later in your script, please comment here, I am pretty sure you can achieve the same with masked array. I would personally always recommend to use them.