GHC can't handle large lookup tables
ghc, haskell
Solution
I can think of two possibilities:
- Serialize the vectors to a file, and at program startup deserialize them (perhaps using unsafeInterleaveIO if you would like this to happen when the lookup table is first used rather than when the program starts).
- If the lookup table is really as sparse as your pseudocode suggests, consider using a sparse data structure -- e.g. `Data.Map` or even just a plain function. If necessary, you can use this sparse data structure to generate your vectors (again at runtime).
Problem
I need to use a lookup table in 3 dimensions. The table itself has 73x73x73 (389017) double values. ``` module Hammer.Texture.Table3D where import qualified Data.Vector as V import qualified Data.Vector.Unboxed as U import Data.Vector.Unboxed (Vector) table3D :: V.Vector (V.Vector (Vector Double)) table3D = V.fromList [table0, table1, ... table72] table0 = V.fromList $ map U.fromList [ [1.973921e+01, 0.000000e+00, ... 0.000000e+00], ..... [1.973921e+01, 0.000000e+00, ... 0.000000e+00]] ..... table72 = V.fromList $ map U.fromList [ [1.973921e+01, 0.000000e+00, ... 0.000000e+00], ..... [1.973921e+01, 0.000000e+00, ... 0.000000e+00]] ``` The problem is that GHC can't handle this size of Vector Double or [Double], the GHC compilation takes a lot of time (~ 2 min) until, finally, the memory blows up. It seems there is a memory leak on GHC or some bug because it works fine for very large String ([Char]). What solutions, if any, are available for creating a "large" lookup tables (Double type) using GHC?