PHP Fatal error: Allowed memory size of 268435456 bytes exhausted on NormalizerFormatter.php on line 62
doctrine-orm, php
Solution
You may want to read Doctrine ORM Batch Processing. They are using this pattern:
// $em instanceof EntityManager
$batchSize = 20;
for ($i=1; $i<=10000; ++$i) {
$obj = new MyEntity;
$obj->setFoo('...');
// ... set more data
$em->persist($obj);
if (($i % $batchSize) == 0) {
$em->flush();
$em->clear();
}
}
$em->flush();
As I understand it - after persisting, doctrine holds all mapped objects in memory in identity map. So memory requirements grow all the time with every new object. If you do clear(), it will detach all managed entities and if you have no references to them, they will be garbage-collected.
Beware: it will detach all other entities you are using. Or you can detach only those inserted objects `$em->detach($twNode); // detach from Doctrine, so that it can be GC'd immediately`
Edit: added flush() after for loop to ensure saving all entities even when entity count is not divisible by batch size, as suggested in comment
Problem
I am trying to insert around 20000 records in mysql by doctrine2 ``` foreach ($getIds as $id){// 20000 Ids $twNode = new TwNode(); $twNode->setTwId($id); $twNode->addAcFriend($this->twAccount); $this->twAccount->addFollower($twNode); $this->em->persist($twNode); $this->em->persist($this->twAccount); $this->em->flush(); $this->output->write('W'); $followersWrite++; } } ``` However this error happnes, how can I avoid this Allowed memory error? ``` PHP Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 72 bytes) in /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/NormalizerFormatter.php on line 62 PHP Stack trace: PHP 1. {main}() /Users/whitebear/httproot/myapp/app/console:0 PHP 2. Symfony\Component\Console\Application->run() /Users/whitebear/httproot/myapp/app/console:22 PHP 3. Symfony\Bundle\FrameworkBundle\Console\Application->doRun() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:106 PHP 4. Symfony\Component\Console\Application->doRun() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Bundle/FrameworkBundle/Console/Application.php:83 PHP 5. Symfony\Component\Console\Command\Command->run() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:200 PHP 6. Acme\TopBundle\Command\twCrawlerCommand->execute() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Component/Console/Command/Command.php:242 PHP 7. Acme\TopBundle\Command\twCrawlerCommand->initialRun() /Users/whitebear/httproot/myapp/src/Acme/TopBundle/Command/twCrawlerCommand.php:430 PHP 8. EntityManager52ee6dcbdb2bb_546a8d27f194334ee012bfe64f629947b07e4919\__CG__\Doctrine\ORM\EntityManager->flush() /Users/whitebear/httproot/myapp/src/Acme/TopBundle/Command/twCrawlerCommand.php:284 PHP 9. Doctrine\ORM\EntityManager->flush() /Users/whitebear/httproot/myapp/app/cache/dev/jms_diextra/doctrine/EntityManager_52ee6dcbdb2bb.php:402 PHP 10. Doctrine\ORM\UnitOfWork->commit() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php:389 PHP 11. Doctrine\ORM\UnitOfWork->executeInserts() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:335 PHP 12. Doctrine\ORM\Persisters\BasicEntityPersister->executeInserts() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:952 PHP 13. Doctrine\DBAL\Statement->execute() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/BasicEntityPersister.php:279 PHP 14. Doctrine\DBAL\Logging\LoggerChain->startQuery() /Users/whitebear/httproot/myapp/vendor/doctrine/dbal/lib/Doctrine/DBAL/Statement.php:161 PHP 15. Symfony\Bridge\Doctrine\Logger\DbalLogger->startQuery() /Users/whitebear/httproot/myapp/vendor/doctrine/dbal/lib/Doctrine/DBAL/Logging/LoggerChain.php:54 PHP 16. Symfony\Bridge\Doctrine\Logger\DbalLogger->log() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Bridge/Doctrine/Logger/DbalLogger.php:80 PHP 17. Monolog\Logger->debug() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Bridge/Doctrine/Logger/DbalLogger.php:102 PHP 18. Monolog\Logger->addRecord() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Logger.php:423 PHP 19. Monolog\Handler\AbstractProcessingHandler->handle() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Logger.php:244 PHP 20. Monolog\Formatter\LineFormatter->format() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Handler/AbstractProcessingHandler.php:35 PHP 21. Monolog\Formatter\NormalizerFormatter->format() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php:43 PHP 22. Monolog\Formatter\LineFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/NormalizerFormatter.php:40 PHP 23. Monolog\Formatter\NormalizerFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php:88 PHP 24. Monolog\Formatter\LineFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/NormalizerFormatter.php:70 PHP 25. Monolog\Formatter\NormalizerFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php:88 Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 72 bytes) in /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/NormalizerFormatter.php on line 62 Call Stack: 0.0002 229968 1. {main}() /Users/whitebear/httproot/myapp/app/console:0 0.0126 2063376 2. Symfony\Component\Console\Application->run() /Users/whitebear/httproot/myapp/app/console:22 0.0144 2256968 3. Symfony\Bundle\FrameworkBundle\Console\Application->doRun() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:106 0.1503 11751600 4. Symfony\Component\Console\Application->doRun() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Bundle/FrameworkBundle/Console/Application.php:83 0.1518 11753384 5. Symfony\Component\Console\Command\Command->run() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:200 0.1521 11758040 6. Acme\TopBundle\Command\twCrawlerCommand->execute() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Component/Console/Command/Command.php:242 0.2409 20673112 7. Acme\TopBundle\Command\twCrawlerCommand->initialRun() /Users/whitebear/httproot/myapp/src/Acme/TopBundle/Command/twCrawlerCommand.php:430 101.5883 235069104 8. EntityManager52ee6dcbdb2bb_546a8d27f194334ee012bfe64f629947b07e4919\__CG__\Doctrine\ORM\EntityManager->flush() /Users/whitebear/httproot/myapp/src/Acme/TopBundle/Command/twCrawlerCommand.php:284 101.5883 235069200 9. Doctrine\ORM\EntityManager->flush() /Users/whitebear/httproot/myapp/app/cache/dev/jms_diextra/doctrine/EntityManager_52ee6dcbdb2bb.php:402 101.5883 235069280 10. Doctrine\ORM\UnitOfWork->commit() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php:389 107.0744 259211408 11. Doctrine\ORM\UnitOfWork->executeInserts() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:335 107.1128 259277872 12. Doctrine\ORM\Persisters\BasicEntityPersister->executeInserts() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:952 109.1604 267877360 13. Doctrine\DBAL\Statement->execute() /Users/whitebear/httproot/myapp/vendor/doctrine/orm/lib/Doctrine/ORM/Persisters/BasicEntityPersister.php:279 109.1604 267877408 14. Doctrine\DBAL\Logging\LoggerChain->startQuery() /Users/whitebear/httproot/myapp/vendor/doctrine/dbal/lib/Doctrine/DBAL/Statement.php:161 109.1604 267877408 15. Symfony\Bridge\Doctrine\Logger\DbalLogger->startQuery() /Users/whitebear/httproot/myapp/vendor/doctrine/dbal/lib/Doctrine/DBAL/Logging/LoggerChain.php:54 109.1604 267877592 16. Symfony\Bridge\Doctrine\Logger\DbalLogger->log() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Bridge/Doctrine/Logger/DbalLogger.php:80 109.1604 267877592 17. Monolog\Logger->debug() /Users/whitebear/httproot/myapp/vendor/symfony/symfony/src/Symfony/Bridge/Doctrine/Logger/DbalLogger.php:102 109.1604 267877640 18. Monolog\Logger->addRecord() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Logger.php:423 109.1604 267878968 19. Monolog\Handler\AbstractProcessingHandler->handle() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Logger.php:244 109.1604 267878968 20. Monolog\Formatter\LineFormatter->format() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Handler/AbstractProcessingHandler.php:35 109.1604 267878968 21. Monolog\Formatter\NormalizerFormatter->format() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php:43 109.1604 267878968 22. Monolog\Formatter\LineFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/NormalizerFormatter.php:40 109.1604 267878968 23. Monolog\Formatter\NormalizerFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php:88 109.1605 267881872 24. Monolog\Formatter\LineFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/NormalizerFormatter.php:70 109.1605 267881872 25. Monolog\Formatter\NormalizerFormatter->normalize() /Users/whitebear/httproot/myapp/vendor/monolog/monolog/src/Monolog/Formatter/LineFormatter.php:88 ``` Thanks to your comment, I understood it is relavant with the memory comsumed by $this->em so, I put the codes to unset $this->em every 5000 times; ``` if ($k > 1000){ $this->em->flush(); unset($this->em); $this->em = $this->getContainer()->get('doctrine')->getManager(); } ``` but still in vain, I think I should unset the origin of $this->getContainer()->get('doctrine')->getManager(); is it possible? or am I right?