Codeigniter Allowed memory size exhausted while processing large files
activerecord, codeigniter, database, memory, php
Solution
I spent hours troubleshooting this and got nothing obvious. I'm using fgets, I have $query->free_result() in the right places. It didn't help. So then I started checking a loop of about 100 lines, and watched the output of memory_get_usage(). I finally narrowed it down to the Codeigniter Active Record class - every call to the class caused the memory usage to increase by a tiny amount.
Then I found this thread on Ellislabs and I got the answer. CI Active Record saves queries so that if you want to, you can build a query in multiple functions. (I am not even going to go into how dumb it is to have that switched on by default.)
Go to /config/database.php and add
`$db['default']['save_queries'] = FALSE;`
to the end of the file. Then make sure you build and execute queries using Active Record in a single function. If you need to switch it off just for one case, use
`$this->db->save_queries = FALSE;`
in the constructor or wherever you need to put it.
Problem
I'm posting this in case someone else is looking for the same solution, seeing as I just wasted two days on this bullshit. I have a cron job that updates the database using a very large file once a day, using the following code: ``` if (($handle = fopen(dirname(__FILE__) . '/uncompressed', "r")) !== FALSE) { while (($data = fgets($handle)) !== FALSE) { $thisline = json_decode($data, true); $this->regen($thisline); } fclose($handle); } ``` This is in a Codeigniter controller that's only used for cron jobs. The $this->regen function runs through a bunch of different checks and stores the right information from the line in the database. The file itself is over 300MB of JSONs separated by newlines. The problem: it would only process about 20,000 lines before the whole thing ran out of memory.