Sunday, 2 June 2013

OpenCL on Ubuntu 13.04

Unfortunately two years after my post about getting Intel's OpenCL to work on Ubuntu, the out-of-box situation for using OpenCL isn't much better - you can install the OpenCL library/headers/package config files with a nice and quick `apt-get install ocl-icd-opencl-dev`, but doing that will probably just get you to CL_PLATFORM_NOT_FOUND_KHR (-1001) error from clGetPlatformIDs. That's because you do have the dispatcher, but no actual OpenCL drivers. Therefore I decided to play with the three major OpenCL implementations (Intel, AMD, NVidia).

The reason why one can do the simple apt-get install is a free software project called ocl-icd - this provides libOpenCL.so and implements the dispatcher so that every icd-compatible OpenCL implementation can use it and therefore provides nice basis for using OpenCL, but to get to building and running CL kernels you need to have at least one driver, so let's look at them.

Intel's OpenCL


Intel recently released Intel SDK for OpenCL Applications XE 2013 - but Ubuntu/Debain users are out of luck, you won't find an official .deb on Intel's web. Fortunately that is not a deal breaker and with a few commands you can turn the rpm into a deb package (and it is what I recommended in my old post). But since we have ocl-icd these days, I'd go for a different approach this time - install just the bare minimum to have ocl-icd pick up the Intel driver (and add proper dependencies on ocl-icd-libopencl1 and libnuma1), so I created the debs manually this time.

So, what I did was:
  1. Grab the tgz from Intel's web.
  2. Extract it into a temporary location where you'll find license, readme file, a bunch of scripts and five rpms:
    • opencl-1.2-base-[version] - contains the libOpenCL.so, which we don't need cause we have ocl-icd-libopencl1
    • opencl-1.2-devel-[version] - contains OpenCL headers, which also aren't needed, as those are in opencl-headers package which is a dependency of ocl-icd-opencl-dev
    • opencl-1.2-intel-cpu-[version] - bingo, the ICD
    • opencl-1.2-intel-devel-[version] - a few development tools - offline compiler and Qt-based KernelBuilder application
    • opencl-1.2-intel-mic-[version] - additional libraries to support also Xeon Phi coprocessor besides Core processors - I ignored this one, cause I don't have access to such processor.
  3. Now, lets make binary debian packages from the unzipped files (note that only the intel-cpu package is required to use OpenCL apps, the devel tools are optional), this is pretty easy, following a simple debian package building HOWTO, I put the files from the intel-cpu and intel-devel rpms into the following directory structure:
     |-opencl-driver-intel-cpu
     | \-DEBIAN
     | \-etc
     |   \-OpenCL
     |     \-vendors
     | \-usr
     |   \-lib
     |     \-x86_64-linux-gnu
     |       \-OpenCL
     |         \-vendors
     |           \-intel
     |   \-share
     |     \-doc
     |       \-opencl-driver-intel-cpu
     |-opencl-driver-intel-tools
     | \-bin
     | \-DEBIAN
     | \-usr
     |   \-lib
     |     \-x86_64-linux-gnu
     |       \-intel
     |         \-opencl-1.2-3.0.67279
     |   \-share
     |     \-doc
     |       \-opencl-driver-intel-tools

    The DEBIAN directories contain the control files:
    $ opencl-driver-intel-cpu/DEBIAN/control:
    Package: opencl-driver-intel-cpu
    Version: 3.0.67279-1
    Section: libs
    Priority: optional
    Architecture: amd64
    Depends: ocl-icd-libopencl1 (>= 2.0), libnuma1
    Maintainer: Your Name
    Description: Intel OpenCL CPU implementation
     This package provides Intel OpenCL implementation which can utilize Intel Core processors.

    $ opencl-driver-intel-tools/DEBIAN/control:

    Package: opencl-driver-intel-tools
    Version: 3.0.67279-1
    Section: libs
    Priority: optional
    Architecture: amd64
    Depends: ocl-icd-libopencl1 (>= 2.0)
    Maintainer: Your Name
    Description: Intel SDK for OpenCL Applications development tools
     This package contains the following tools:
      - Intel SDK for OpenCL - Kernel Builder, which enables building and analyzing OpenCL kernels and provides full offline OpenCL language compilation.
      - Intel SDK for OpenCL - Offline Compiler, a command-line utility, which enables offline compilation and building of OpenCL kernels.
    The leaf directory opencl-driver-intel-cpu/usr/lib/x86_64-linux-gnu/OpenCL/vendors/intel contains all the object files from the intel-cpu rpm, and opencl-driver-intel-tools/usr/lib/x86_64-linux-gnu/intel/opencl-1.2-3.0.67279/ contains the binaries from intel-devel rpm with tiny changes so the bash scripts point to a correct location.

    Once this is done, the only remaining file to tamper with is the actual .icd in opencl-driver-intel-cpu/etc/OpenCL/vendors/intel64.icd, which contains just one line with path to the library:
    /usr/lib/x86_64-linux-gnu/OpenCL/vendors/intel/libintelocl.so
  4. Now just run `dpkg-deb --build opencl-driver-intel-cpu` and `dpkg-deb --build opencl-driver-intel-tools` and voila your debs with Intel's OpenCL are ready.
You can also download my opencl-driver-intel-cpu.deb and opencl-driver-intel-tools.deb.

Note that these are here just for reference, you should delete them after making sure that your package looks the same - ie I'm not redistributing these.

Once installed, a simple OpenCL app that lists the available platforms should output something like this:
PLATFORM_NAME: Intel(R) OpenCL
  VERSION: OpenCL 1.2 LINUX
  VENDOR: Intel(R) Corporation
  PROFILE: FULL_PROFILE
    DEVICE:       Intel(R) Core(TM) i7-3632QM CPU @ 2.20GHz
    DEVICE VENDOR: Intel(R) Corporation
    MAX COMPUTE UNITS: 8
    DEVICE VERSION: OpenCL 1.2 (Build 67279)
    DRIVER VERSION: 1.2

NVidia's OpenCL


In Ubuntu 13.04 there are now multiple nvidia packages which contain various versions of nvidia's driver, but I couldn't find anywhere a package with the icd file and therefore even if all the nvidia-3* packages do have the driver, the ICD loader isn't able to find it. Nonetheless, the fix for that is easy:

Run `locate libnvidia-opencl.so`, this will probably find a few files, in my case these are found:
/usr/lib/nvidia-304/libnvidia-opencl.so.304.88
/usr/lib/nvidia-current/libnvidia-opencl.so.1
/usr/lib32/nvidia-304/libnvidia-opencl.so.304.88
/usr/lib32/nvidia-current/libnvidia-opencl.so.1
For some reason though, the *.so.1 are broken symlinks in my case (maybe because I have Optimus laptop), therefore I created the .icd in the following manner:

echo /usr/lib/nvidia-304/libnvidia-opencl.so.304.88 > /etc/OpenCL/vendors/nvidia64.icd

The problem with this is that if the driver gets updated, you'll have to modify the .icd again, so if your *.so.1 symlink is not broken, you should use that instead.

Note that the driver also contains the OpenCL loader library (libOpenCL.so) and depending on your LD_LIBRARY_PATH settings, it might get used instead of the one provided by ocl-icd. That is not necessarily terrible, but keep in mind that NVidia's implementation is OpenCL 1.1, so even if other ICDs support 1.2, you'll be stuck with 1.1. Solution? Just remove the extra libOpenCL.so* from /usr/lib/nvidia-*/

Once done, listing CL platforms and devices should also list NVidia:
PLATFORM_NAME: NVIDIA CUDA
  VERSION: OpenCL 1.1 CUDA 4.2.1
  VENDOR: NVIDIA Corporation
  PROFILE: FULL_PROFILE
    DEVICE: GeForce GT 635M
    DEVICE VENDOR: NVIDIA Corporation
    MAX COMPUTE UNITS: 2
    DEVICE VERSION: OpenCL 1.1 CUDA
    DRIVER VERSION: 304.88

OpenCL and Bumblebee


As previously mentioned, I have an Optimus laptop (with integrated Intel GPU as well as NVidia which is used for more demanding applications). If the NVidia GPU is shut down, you won't see the NVidia platform as available (which I find a bit strange, should be a platform with 0 available devices, no?), but once turned on with a `optirun bash`, things should be working properly, although I haven't tried for example CL-GL interop, I can imagine that might not work with Bumblebee.

One issue I noticed though is that if you have both NVidia and Intel's drivers, Intel's driver will crash any OpenCL app run inside the optirun shell, which seems to be caused by the LD_PRELOAD libraries that VirtualGL uses. So either run your apps inside the shell with `LD_PRELOAD= ./myApp`, or just don't run them inside the optirun shell, use a regular one. As long as the GPU is active it is perfectly able to perform calculations even without VirtualGL set up.

AMD's driver


AMD enables to use both CPU and GPU devices with their driver. I don't have a GPU by AMD, so I only got to try the CPU implementation (luckily Intel's and AMD's CPUs are still compatible enough). The installer that they provide installs the whole SDK into /opt/AMDAPP, changes your /etc/profile to include the directories in LD_LIBRARY_PATH, and installs the icd to /etc/OpenCL/vendors.

What I don't like about this is that their SDK also contains libOpenCL.so, so it will be used instead of ocl-icd's. In this case this is less of a problem than in NVidia's case, cause AMD's implementation isn't limited to just OpenCL 1.1, but if you want to use ocl-icd, just remove the libOpenCL.so* from /opt/AMDAPP/lib/x86_64.

It would be nice to have a similar deb package for the AMD's CL driver, but I didn't get to that, maybe someone else wants to? ;) Anyway:
PLATFORM_NAME: AMD Accelerated Parallel Processing
  VERSION: OpenCL 1.2 AMD-APP (1113.2)
  VENDOR: Advanced Micro Devices, Inc.
  PROFILE: FULL_PROFILE
    DEVICE: Intel(R) Core(TM) i7-3632QM CPU @ 2.20GHz
    DEVICE VENDOR: GenuineIntel
    MAX COMPUTE UNITS: 8
    DEVICE VERSION: OpenCL 1.2 AMD-APP (1113.2)
    DRIVER VERSION: 1113.2 (sse2,avx)

Conclusion


As I was testing the various drivers I encountered quite a few issues - Intel's implementation crashes on ratGPU tests, AMD's pretends to work with my OpenCL face detection but doesn't detect anything (Intel's and NVidia's work fine), on top of that the crashes with Intel and Bumblebee/VirtualGL LD_PRELOAD shell. Samples from AMD's SDK crash when used with ocl-icd ICD loader because they call clReleaseContext(NULL), works with AMD's loader though. But in the end there is also a lot more that actually is working - for example a year ago my face detection didn't work at all with Intel's implementation, now it's fine, many of the SDK samples did work with all three drivers. I'd say there was some good progress.

So that's about the current OpenCL state, it's usable, just not out-of-the-box, I do hope that a year from now at least this post will be just saying "To use OpenCL just run `apt-get install opencl-driver-*`".

15 comments:

  1. I made a related how-to that's not distribution specific:
    http://wiki.tiker.net/OpenCLHowTo

    I've also linked to yours from mine.

    Best,
    Andreas

    ReplyDelete
  2. With one command I was able to find this:
    apt-cache search opencl icd
    *beignet - Intel OpenCL library*
    ocl-icd-dbg - Debug symbols for the generic OpenCL ICD Loader
    ocl-icd-dev - Development files to build a ICD Loader
    ocl-icd-libopencl1 - Generic OpenCL ICD Loader
    ocl-icd-opencl-dev - OpenCL development files
    nvidia-libopencl1 - NVIDIA OpenCL library
    nvidia-opencl-common - NVIDIA OpenCL driver
    *nvidia-opencl-icd - NVIDIA OpenCL ICD*
    amd-clinfo - AMD OpenCL info utility
    amd-libopencl1 - AMD OpenCL library
    *amd-opencl-icd - AMD OpenCL ICD*
    *amd-opencl-icd-legacy - AMD OpenCL ICD (legacy)*
    Now I'm confused.

    ReplyDelete
    Replies
    1. That's on Debian right? Ubuntu has just the ocl-icd-* ones afaict.

      Delete
    2. Yes it's on Debian GNU/Linux.
      amd-opencl-icd:
      Installed: (none)
      Candidate: 1:13.4-2
      Version table:
      1:13.4-2 0
      501 http://ftp.pl.debian.org/debian/ unstable/non-free amd64 Packages
      500 http://ftp.pl.debian.org/debian/ testing/non-free amd64 Packages

      Delete
  3. What's the simple app that shows OpenCL details?

    I am not currently developing using OpenCL, but I have the AMD drivers installed and want to check if it actually works.

    ReplyDelete
    Replies
    1. There's clinfo in AMD's SDK, otherwise ratgpu gives you a nice graphical list of available CL devices.

      Delete
  4. man libOpenCL suggests setting environment variable ... for Intel ICD crash?

    OCL_ICD_ASSUME_ICD_EXTENSION
    If set, contrary the the Kronos specification, the loader will not
    check that the loaded ICDs declare the cl_khr_icd extension. You
    may need to define this environment variable if you are using the
    Intel ICD toguether with optirun(1). Else, a bug into the Intel ICD
    will make the application crash.

    ReplyDelete
  5. Hello. I've linked your guide here:
    http://www.blenderartists.org/forum/showthread.php?300561-Testing-Cycles-with-OpenCL-on-GNU-Linux-%28Intel-nVidia-and-maybe-ATI%29

    Thank you!

    ReplyDelete
  6. I can't install this because my system is x86. Please, can someone check if there are some solution here: http://wiki.freedesktop.org/www/Software/Beignet/

    ReplyDelete
  7. I also have an Optimus machine. If I run my programs under optirun, they detect and use both Nvidia GPU and AMD's driver for CPU. If I link the same programs against Beignet's library, I can detect and use Intel HD4000. But I can't make ocl-icd detect Beignet's driver, so I am not able to use all three platforms from a single program. All three files in /etc/openCL/vendors are looking good. How can I debug this situation? Is it expected?

    ReplyDelete
    Replies
    1. Have you tried lately? I think Beignet gained support for ICD only recently.

      Delete
  8. This comment has been removed by the author.

    ReplyDelete
  9. Nice. Hey could you explain the procedure for doing--> opencl-1.2-intel-mic-[version] - additional libraries to support also Xeon Phi coprocessor besides Core processors - I ignored this one, cause I don't have access to such processor. <--I'm trying to use maximum. If it's not to much to ask it would be great it you would update this article with the support for Xeon Phi.

    ReplyDelete
    Replies
    1. Sorry, as I don't have a Xeon Phi I wouldn't be able to make sure it works, but all the libs should be in the rpm, moving them should do the trick.

      Delete
  10. You should really make a PPA! ;D

    ReplyDelete