Tag Archives: Linux

GPU

Creating a conda environment for GPU programming with pytorch and tensorflow

After a few mis-steps, here is how I set up a conda environment to use in Jupyter with tensorflow, pytorch, and using the GPU.

As a note, I do this on the node with the GPU, so that things (hopefully) compile correctly!

1. Create an environment

First, create a new environment for tensorflow and friends, and activate it.

mamba create -n gpu_notebook cudatoolkit tensorflow nvidia::cuda pytorch torchvision torchaudio pytorch-cuda -c pytorch -c nvidia
mamba activate gpu_notebook

2. Install the python libraries

Install the usual suspect python packages that you will probably want to use. For convenience, I usually put these in a file in my
Git repo called requirements.txt.

$ cat requirements.txt 
jupyter
matplotlib
natsort
numpy
pandas
scipy
scikit-learn
seaborn
statsmodels
pip install -r requirements.txt

3. Reame your jupyter kernel

When you open jupyter there is a list of kernels that you can connect to. (If you have a window open that list will be on the top right.) If you rename your jupyter kernel it makes it
much easier to find the kernel associated with this conda environment. The default name is something like Python 3 which is not helpful if you have lots of them!

a. Find where your kernel is installed

This command shows your jupyter kernels

jupyter kernelspec list

You’ll see your kernel(s) and the locations of them. In the location listed there is a file called kernel.json.

b. Edit that file:

vi $HOME/miniconda3/envs/gpu_notebook/share/jupyter/kernels/python3/kernel.json

c. Change the name to be meaningful

Change the value associated with the display_name key. Set it to something meaningful so you can find it in your browser!

4. Set up the XLA_FLAGS environment variable.

This was essential for me to get tensorflow working. There is a directory somewhere in your conda environment with the libdevice library that is needed. For my installation that was in nvvm/libdevice/libdevice.10.bc. Of course you can find yours with:

find ~/miniconda3/ -name libdevice

You want to set the XLA_FLAGS variable to point to the base of the nvvm folder. This command sets it inside the conda environment so it is always set when the conda environment is activated, and unset when it is deactivated.

conda env config vars set XLA_FLAGS=--xla_gpu_cuda_data_dir=$HOME/miniconda3/envs/gpu_notebook

5. Activate the environment

Don’t forget to submit this to a node with GPU capabilities!

bash

Use column to display tsv files in columns

If you have a tab separated file and view it in a pager like less or more, the columns never line up. Here is a simple way to make those columns appear correct.

column -t file.tsv

For example, here is a file with three columns of words, displayed with cat

If we pass that to column with the -t option to detect the columns, we get nicely organised columns:

However, note that this is not exactly correct, notice that “Paradigm shift” has been split into two columns because the -t option uses whitespace by default, so to display the columns using tabs, we need to add a -s option:

column -t -s$'\t'

Reading and writing to the same file

If you try to modify a file (removing all empty lines for example) using a command like:

 cat file.txt | sed '/^$/d' > file.txt 

you will end up with and empty file.txt. The reason is that bash parses the command line looking for “metacharacters” (  “|” , “>” and “space”  in this case) that separate words, then groups and executes those words according to their precedence. This means that “> file.txt”  get executed FIRST. This creates an empty file.txt (overwriting any existing file) and a “process”  to redirect standard output to that file. Then “cat file.txt” get executed, but by now file.txt is empty. So “cat file.txt” outputs 0 lines,  “sed ‘/^$/d’ ” deletes all 0 empty lines, and 0 lines get written to file.txt . This works as “intended” and bash outputs no error.

You can get around this using a temporal file.

 cat file.txt | sed '/^$/d' > tmp_file.txt
mv tmp_file.txt file.txt
 

But, as file.txt is technically a new file you might lose some information, in particular permissions and whether file.txt was originally a symbolic link or not.

Other options is to use sponge, which is part of moreutils and sadly not standard in many systems.

 cat file.txt | sed '/^$/d' | sponge file.txt
 

 

Linux commands for Mac OS X

We all know that Mac OS X is UNIX certified allowing command line users to use Terminal to execute quick Bash or Perl script pipelines. However, for us that are more familiar with the GNU Linux environment, there are some slight variations in the commands we often use, such as the –complement flag in cut, or the -h flag in sort. These options are non-existent in the OS X versions of the commands, and there are probably more instances of these that I haven’t run into yet.
There is a very simple way to install these GNU commands for your Mac using Homebrew:
brew install coreutils
This will allow you to use the GNU versions of the commands, for example, gsort is used in place of sort, and gcut in place of cut.
If you don’t have Homebrew installed (it’s almost like using apt-get to install programs and commands) you can install it by first installing Xcode and making sure command line tools is installed for Xcode. Then run the following command:
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
Some online blogs recommend installing XQuartz afterwards too.
Note: this was performed on OS X Yosemite version 10.10.3

Changing the volume name on an LVM machine

Changing the volume group name on an LVM machine is easy, but doing it so that it doesn’t break everything is hard. Here are some pointers on how to change the volume group name on an Ubuntu LVM machine.

First, use

vgdisplay

to show your current volume groups, choose the one you want to rename and rename it using vgrename.

vgrename oldname newname

Now you need to edit /etc/fstab to replace the old name with the new one everywhere in that file.

vi /etc/fstab

Edit /boot/grub/grub.cfg and do the same thing

perl -i.old -npe 's/oldname/newname/g' /boot/grub/grub.cfg

Update the image ram disk:

update-initramfs -u   (this updates the current version)

And reboot.

It may well crash here, especially if the volume name has changed. If you end up in busybox, look in /dev/mapper and see what the volume is called. Next, reboot the machine (you probably need to power cycle it for this), and then when the grub menu comes up listing your machines, edit the line that says linux /dev/mapper/…. so that the name listed there is the same as the one you saw.

You should be able to reboot now, and then run

update-grub2

Finally, check that the name you provided in /etc/fstab was correct (mine was not, so I re-edited that file).

 

 

 

Become Even More Productive – 2 New Cool Things!

Over the past couple of days I’ve been toying with a couple of neat ways to access / visualize data. Rob showed me Cooliris, which was pretty mind blowing as far as just looking like it came straight out of a sci-fi movie. More seriously, I’ve found that it is actually a pretty efficient way to do image searches. It displays images much more quickly than a simple google image search result.

The other awesome thing that I’ve found is the Compiz-Fusion suite of window management plugins for Linux. (You can download a version for your distro here). It allows you to manage a large number of extended desktops; this is particularly useful on the EEEPC netbooks which have small monitors! Other cool features worth checking out are the Ring-switch plugin ( + tab for application switches), Desktop Cube & Rotate Cube (Alt-Ctrl-Mouse button 1 click and drag). Turning transparency on the cube is actually pretty nice because it allows you to view a number of desktops all at once.

Some cool things to do just for looks are: 
Enable Skydome on the cube (use custom backgrounds while in cube view).
Try out the water plugin (ctrl + to initiate, Shift+f9 and shift+f8 for rain control).
Turn on wobble plugin for your application windows.
Check out the wiki to learn even more awesome stuff you can do!

Now that you’re all informed, install this stuff on your linux machines, go out, and be impressive!