Any of the various energy modeling programs would give you a starting place. Model the ducts in various places and with different insulation levels. I know you will not forget to seal the ducts.
10-12" liners are not made. Biggest made I find is 6" Flue temp on gravity is above 800' most 80% gas or lp are in the 350-380'. If a low end replacement is put in goes from quite to very very noisy