How do I handle errors in a worker pool using WaitGroup?

channel, coroutine, go, wait

Solution

Just got the answer my self as I wrote the question and as I think it is an interesting case I would like to share it with you.

The trick to use `sync.WaitGroup` and `chan` together is that we wrap:

select {
    case err := <-p.errors:
        return err
    default:
        p.wg.Done()
}

Together in a `for` loop:

for {
    select {
        case err := <-p.errors:
            return err
        default:
            p.wg.Done()
    }
}

In this case select will always check for errors and wait if nothing happens :)

Problem

I got a problem using `sync.WaitGroup` and `select` together. If you take a look at following http request pool you will notice that if an error occurs it will never be reported as `wg.Done()` will block and there is no read from the channel anymore. ``` package pool import ( "fmt" "log" "net/http" "sync" ) var ( MaxPoolQueue = 100 MaxPoolWorker = 10 ) type Pool struct { wg *sync.WaitGroup queue chan *http.Request errors chan error } func NewPool() *Pool { return &Pool{ wg: &sync.WaitGroup{}, queue: make(chan *http.Request, MaxPoolQueue), errors: make(chan error), } } func (p *Pool) Add(r *http.Request) { p.wg.Add(1) p.queue <- r } func (p *Pool) Run() error { for i := 0; i < MaxPoolWorker; i++ { go p.doWork() } select { case err := <-p.errors: return err default: p.wg.Wait() } return nil } func (p *Pool) doWork() { for r := range p.queue { fmt.Printf("Request to %s\n", r.Host) p.wg.Done() _, err := http.DefaultClient.Do(r) if err != nil { log.Fatal(err) p.errors <- err } else { fmt.Printf("no error\n") } } } ``` Source can be found here How can I still use WaitGroup but also get errors from go routines?

Original source