io.Copy cause out of memory in golang
go
Solution
That's because you are 'copying', to `bodyBuf`, which is an in-memory buffer, forcing Go to try an allocate a block of memory as big as the entire file.
Based on your use of `multipart` it looks like you are trying to stream the file over http? In that case, don't pass a `bytes.Buffer` to `multipart.NewWriter`, directly pass your http connection instead.
Problem
I use io.Copy() to copy a file, about 700Mb, but it cause out of memory ``` bodyBuf := &bytes.Buffer{} bodyWriter := multipart.NewWriter(bodyBuf) //key step fileWriter, err := bodyWriter.CreateFormFile(paramName, fileName) if err != nil { return nil, err } file, err := os.Open(fileName) //the file size is about 700Mb if err != nil { return nil, err } defer file.Close() //iocopy copyLen, err := io.Copy(fileWriter, file) // this cause out of memory if err != nil { fmt.Println("io.copy(): ", err) return nil, err } ``` The error message as follow: ``` runtime: memory allocated by OS (0x752cf000) not in usable range [0x18700000,0x98700000) runtime: out of memory: cannot allocate 1080229888-byte block (1081212928 in use) fatal error: out of memory ``` I allocate enough memory for buf, it cause out of memory in bodyWriter.CreateFormFile() ``` buf := make([]byte, 766509056) bodyBuf := bytes.NewBuffer(buf) bodyWriter := multipart.NewWriter(bodyBuf) fileWriter, err := bodyWriter.CreateFormFile(paramName, fileName) // out of memory if err != nil { return nil, err } ```