Managing mega Arrays in PHP

Question

I'm doing a data-mine on millions of old log entries for someone and really want to use PHP in this matter to present the materials a link them easily to the existing PHP system. I run this code in the PHP 5.4.4 in the Terminal (OSX 10.8): ``` // Settings ini_set('error_reporting', E_ALL); // Shows all feedback from the parser for debugging ini_set('max_execution_time', 0); // Changes the 30 seconds parser exit to infinite ini_set('memory_limit', '512M'); // Sets the memory that may be used to 512MegaBytes echo 'Start memory usage: '.(memory_get_usage(TRUE) / 1024)."
"; $x = Array(); for ($i = 0; $i < 1e7; $i++) { $x[$i] = 1 * rand(0, 10); //unset($x[$i]); } echo 'End memory usage: '.(memory_get_usage(TRUE) / 1024)."
"; echo 'Peak memory usage: '.(memory_get_peak_usage(TRUE) / 1024)."
"; ``` This is a simple test with ten-million cycles. The leakage is really bad compared to using dictionaries in Python :(. When I unquote the unset() function to test the usage, it's instantly all u…

Accepted Answer

Use `SplFixedArray` because you really need to see How big are PHP arrays (and values) really? (Hint: BIG!)

```
$t = 1e6;
$x = array();
for($i = 0; $i < $t; $i ++) {
    $x[$i] = 1 * rand(0, 10);
}
```

Output

```
Start memory usage: 256
End memory usage: 82688
Peak memory usage: 82688
```

and

```
$t = 1e6;
$x = new SplFixedArray($t);
for($i = 0; $i < $t; $i ++) {
    $x[$i] = 1 * rand(0, 10);
}
```

Output

```
Start memory usage: 256
End memory usage: 35584
Peak memory usage: 35584
```

But better still i think you should consider a memory based database like REDIS

Managing mega Arrays in PHP

Solution

Problem