In Perl, how can I use multiple threads to process parts of a large array?

multithreading, perl

Solution

I don't think what I said earlier about having to use threads::shared is correct. I had to go check the documentation, and I am not sure. I am never sure when it comes to threads with Perl.

Update: As it turned out, my incomplete understanding was revealed again. At the very least, you need `threads::shared` to be able to put the results in `@output` from within each thread.

#!/usr/bin/env perl

use strict; use warnings;
use threads;
use threads::shared;

use List::Util qw( sum );
use YAML;

use constant NUM_THREADS => 20;

my @output :shared;
my @data = ( ([1 .. 10]) x 200);

# note that you'll need different logic to handle left over
# chunks if @data is not evenly divisible by NUM_THREADS
my $chunk_size = @data / NUM_THREADS;

my @threads;

for my $chunk ( 1 .. NUM_THREADS ) {
    my $start = ($chunk - 1) * $chunk_size;
    push @threads, threads->create(
        \&doOperation,
        \@data,
        $start,
        ($start + $chunk_size - 1),
        \@output,
    );
}

$_->join for @threads;

print Dump \@output;

sub doOperation{
    my ($data, $start, $end, $output) = @_;

    my $id = threads->tid;

    print "Thread [$id] starting\n";

    for my $i ($start .. $end) {
        print "Thread [$id] processing row $i\n";
        $output->[$i] = sum @{ $data->[$i] };
        sleep 1 if 0.2 > rand;
    }

    print "Thread $id done!\n";

    return;
}

Output:

- 55
- 55
- 55
…
- 55
- 55
- 55
- 55

Problem

I need help with multithreading in Perl. The basic logic is to initiate 20 threads. I have one array `@dataarray` and I want 20 chunks of data to be passed to each thread. Say, `@dataarray` has 200 rows of data in it, so first 10 rows will be going to thread 1, next 10 should be sent to thread 2, so they don't overwrite each others data and ultimately after processing thread should update return result to `@outputarray` at the same index position as of source `@datarray`. For example: row 19(index position 18) from `@dataarray` was sent to thread number 2 so after processing it thread 2 should update `$outputarray[18] = $processed_string`. Just need to figure out how to send from and to positions of array to a particular thread. ``` #!/usr/bin/perl use strict; use threads; my $num_of_threads = 20; my @threads = initThreads(); my @dataarray; foreach(@threads) { $_ = threads->create(\&doOperation); } foreach(@threads) { $_->join(); } sub initThreads { my @initThreads; for(my $i = 1;$i<=$num_of_threads;$i++) { push(@initThreads,$i); } return @initThreads; } sub doOperation { # Get the thread id. Allows each thread to be identified. my $id = threads->tid(); # Process something--- on array chunk print "Thread $id done!\n"; # Exit the thread threads->exit(); } ```

Original source