Can I use in RDMA via Infiniband Load/Store access from GPU2-Cores to GPU1-RAM in the different PCIe-Bus?
cuda, gpgpu, infiniband, multi-gpu, rdma
Solution
GPUDirect RDMA has a single public implementation at this time, which is via Mellanox Infiniband. You would need to use something like one of the CUDA-aware MPI systems to take advantage of it.
You cannot use `cudaMemcpy` to copy from GPU1 to GPU2 in the example you have shown, i.e. you cannot use `cudaMemcpy` to copy directly between GPUs that live in different host systems.
Problem
As said here: https://developer.nvidia.com/gpudirect I can access from GPU0-Core to the GPU1-RAM on the Same PCIe Bus: - Load/Store - `cudaMemcpy()` It named as "NVIDIA GPUDirect v2 Peer-to-Peer (P2P) Communication Between GPUs on the Same PCIe Bus (2011)": And I can use RDMA via Infiniband to copy from GPU2-RAM to GPU1-RAM on the different PCIe Bus, it is named as "GPUDirect™ Support for RDMA, Introduced with CUDA 5 (2012)": But can I use in RDMA on the different PCIe Bus connected via Infiniband both: - Load/Store (access from GPU2-Core to GPU-RAM) - `cudaMemcpy()` (to copy from GPU2-RAM to GPU1-RAM) Or can I use only `cudaMemcpy()` in RDMA?