Jupyter environment setup

This is a short note about how I set up my "data scientist" environment. Different people have different tastes, but what I use, and what I set up is:

• conda for environment and package management (equivalent to virtualenv and pip to say)
• Latest Python (yes, Python 3)
• Jupyter (aka IPython notebook)
• Disable all the autocomplete quotes and brackets stuff, that comes by default with Jupyter
• Set the IPython backend for matplotlib

So, we download Anaconda from: https://www.continuum.io/downloads (Linux 64 bits, Python 3, in my case). We install it by:

bash Anaconda3-2.4.1-Linux-x86_64.sh

We can either restart the terminal, or type the next command, so we start using conda environment:

. ~/.bashrc

We can update conda and all packages:

conda update conda && conda update --all

Then we create a new conda environment (this way we can change package versions without affecting the main conda packages). We name it myenv and specify the packages we want (numpy, pandas...).

conda create --name myenv jupyter numpy scipy pandas matplotlib scikit-learn bokeh

We activate the new environment:

source activate myenv

Now we have everything we wanted installed, let's change the configuration.

We start by creating a default ipython profile.

ipython profile create

Then we edit the file ~/.ipython/profile_default/ipython_kernel_config.py and we add the next lines to make matplotlib display the images with the inline backend, and with a decent size:

c.InteractiveShellApp.matplotlib = 'inline' c.InlineBackend.rc = {'font.size': 10, 'figure.figsize': (18., 9.), 'figure.facecolor': 'white', 'savefig.dpi': 72, 'figure.subplot.bottom': 0.125, 'figure.edgecolor': 'white'}

To disable autoclosing brackets, run in a notebook:

from notebook.services.config import ConfigManager
c = ConfigManager()
c.update('notebook', {"CodeCell": {"cm_config": {"autoCloseBrackets": False}}})

Unified Python

After all these days at EuroPython, there is a thought that keep me thinking. It is about how Python have different ways to represent what it could be considered the same thing.

On today's talk, Alex Martelli pointed out that "def" and "lambda" are actually the same concept. This was part of a more complete idea about that both of them have the wrong name ("function" should be the right), and that lambda actually should disappear, but that's another question.

Also, yesterday, Raymond Hettinger reminded that class are actually dictionaries, something that most Pythonistas know, but which also made me thought.

Then, there is something that I never saw very clearly, and it is the subtle difference between an instance and a dictionary, and how trivial it can be in some case, the difference between person['name'] and person.name.

So, I wanted to do an experiment on how it could look Python, if it would try to merge all this entities in ones single format, and even some other things like avoiding assignments that doesn't follow the assignment pattern (I mean class or function definition here, where instead of my_func = [...] it's used def my_func[...]).

Next, there is how the most stupid example I could invent looks like, but first some definitions to make it easier to understand the idea.

map: could be also "class", "dict", "obj", "hash",... and it's the structure for dictionaries, classes and instances.
seq: a list or tuple, any linear sequence of values.
func: a function or callable, that in Python is defined by "def" or "lambda".


foods = seq:
"meat"
"milk"
"bread"

sounds = map:
"bark" = "woof woof"
"mew" = "meow meow"

animal = map:
"step_size" = None
"sound" = None

"move" = func(self, num_steps):
print("I've moved {} units".format(num_steps * self.step_size))

"talk" = func(self):
print(sounds.{self.sound})

"eat" = func(self, food):
print("I'm eating {}".format(food))

cat = map(animal):
"step_size" = 80
"sound" = "mew"

"eat" = func(self, food):
print("I only eat {} if I want to".format(food))


azrael = map(cat):
"owner_name" = "Gargamel"

azrael.move(5)
for food in foods:
azrael.eat(food)


Of course, there are too many things that should be considered before being able to implement this syntax, but can give an idea on how it could look a more unified approach of Python syntax.

See how the syntax for "sounds", which would be a dictionary, "cat", which would be a class, and "azrael", which would be a instance, is exactly the same.

Being used to Python syntax, it's difficult to say if this syntax could be readable, so far I just find it weird. But what looks clear, is that this syntax would make the language simpler, from the implementation point of view, and probably from the programmer point of view, who would probably need to forget some OP concepts first.

Whatever is the conclusion the reader can get from this example, I think it's quite interesting seeing how a class can look exactly the same way as a dictionary, and how an instance can look exactly as a subclass of the base class.

Building RPMs for Python3.1

While it's been a long time since the first stable version Python 3 was released, it's not yet available on several operating systems. Looking for a repository with Python 3 rpms, I found IUS Community, but I had some problems with it, and I thought on building my own rpms.

The process for building an rpm from a source tarball is pretty easy (if you know the steps). The only problem in this case, is that the .spec file delivered with Python is not updated, so the process fails.

I did required changes to the .spec file, and I uploaded it to: http://files.vaig.be/python-3.1.spec (NOTE, that is necessary to edit the exact version of Python you're building in line 37. Version in uploaded file is 3.1.3, but it could be changes to 3.1.3, 3.1.4rc1,...).

Next, you can find the steps for creating a RPM package for Python 3.1 on a CentOS 5 (using my custom .spec file):


# Install required software
yum install rpm-build gcc expat-devel db4-devel gdbm-devel sqlite-devel ncurses-devel readline-devel zlib-devel openssl-devel

# Download Python source
cd /usr/src/redhat/SOURCES/
wget http://www.python.org/ftp/python/3.1.3/Python-3.1.3.tar.bz2

# Download .spec (rpm specifications file)
cd /usr/src/redhat/SPECS/
wget http://files.vaig.be/python-3.1.spec

# Generate RPMs (and SRPMs)
rpmbuild -ba /usr/src/redhat/SPECS/python-3.1.spec


Compiling Python and creating the RPM will take a while, but after this process, you'll have the RPMs at:


/usr/src/redhat/SRPMS/python3.1-3.1.3-1pydotorg.src.rpm
/usr/src/redhat/RPMS//python3.1-3.1.3-1pydotorg.i386.rpm
/usr/src/redhat/RPMS//python3.1-devel-3.1.3-1pydotorg.i386.rpm
/usr/src/redhat/RPMS//python3.1-tools-3.1.3-1pydotorg.i386.rpm


Restrict multiple simultaneos executions of a Python program

Here you've a simple function to avoid a python script to be executed more than once at the same time:


def use_lock(func, lockfile):
if not os.path.exists(lockfile):
with open(lockfile, 'w') as f:
f.write(str(os.getpid()))
func()
os.remove(lockfile)
return True
else:
return None


To execute a function main() using a lock file "/var/run/myprogram.pid" just write:


use_lock(main, '/var/run/myprogram.pid')


Hope you find it useful.