Tuesday, December 30, 2008

Compiling mysqldb for python on Leopard 64 bit architecture

now there's a problem when compiling the mysqldb module for python

fourspaces describes most of what goes wrong, he also gets the emotional tenor just right.

and the guy who wrote the python module tells us what the final solution is. Check in the comments on the bottom of his page.

So, in essence, you must download and compile python for a 64bit architecture, or set up mysql and apache to work as 32-bit processes.

The easier way is to go the 32 bit route. Download mysql community edition 32-bit, then thin out the apache server so it only uses the i386 arch. The instructions to "thin out" are here, scroll down to the section where it says "Thinning the Apache Executable". Note that at the bottom, they give an alternate way to get the same thing done.

Another painful consideration in favor of the 32-bit route is that if you go 64, then you will have to recompile every single thing that interacts with python, mysql and apache to work as a 64 bit arch program. This is a lot tougher than it sounds since everything by default is set to go 32. The latest bit of torment I had to deal with was getting libxml2 to work. But you decide what's best for you. Below I have instructions on going the 64 bit route.

Compiling Python 2.6.1 for x64:

This is actually harder than simply doing a ./compile & make the thing is that there are four different possible architectures for any program on an intel mac running leopard. There are two basic chip sets, power pc and intel. There are also two different modes (?) to use the system bus (?) in - 32 bit and 64 bit. So you have:
  1. ppc - 32 bit power pc
  2. ppc64 - 64 bit power pc
  3. i386 - 32 bit intel
  4. x86_64 - 64 bit intel
So why not just compile to whatever your chipset can do fastest (that is, ppc64 or x86_64)? Dunno. But the result is, you generally want anything you compile to hit everything possible. So generally, what you do before you compile anything is:
export ARCHFLAGS='-arch i386 -arch ppc -arch ppc64 -arch x86_64'
In this case however, you have to do something else. First off, I recommend going to the python.org community lists and browsing through the archives if you want to find information. Just googling is next to useless because the noise to signal ratio is terrible. In any case, I found this gem on the python community lists which I'm going to describe.

While you should read the above link to compiling python 2.6, I'm going to copy some of the instructions, but with my own modifications. I'm also putting in some links to commands which I think deserve explanation:
  1. do an svn checkout or unzip a tar package
  2. $ cd Python2.6 (or whatever the directory it uninstalls to is)
  3. $ mkdir build
  4. $ cd build
  5. $ ../configure --with-framework-name=Python64 --with-universal-archs=all --enable-framework ---enable-universalsdk=/
some notes:

  • if any of the commands make no sense, you can always try piping it through grep, for example: ./configure --help | grep "universal-sdk" gave me
  • The Python frameworks that come with your computer are stored in /System/Library/Frameworks/Python.framework. Because we set --enable-universalsdk to "/", it stores everything in /Library/Frameworks/Python64.framework this is to make it easier to get rid of.
  • if we had set --with-framework-name=Blah then presumably we would have gotten /Library/Frameworks/Blah.framework eheh.

  • MACOSX_DEPLOYMENT_TARGET=10.5 lets the compiler know we're using Leopard. You can find it in issue 1358. If you don't do this you might get the following error
    ./Modules/posixmodule.c:3592: error: too few arguments to function ‘setpgrp’
Now you have to set your path so that it's looking for your new python build. Please make certain you've done this step. And then test it out when it's done.

Why do I insist on this? Because mod_python is not a miniature python interpreter, it checks to make sure that the build it compiled against is what's available to apache. And if apache doesn't know where to look, it's going to stick with the default python interpreter, which means an interpreter built to a 32-bit architecture (i386 arch). And then it will crap out. And then you will go insane trying to figure out went wrong.

So MAKE SURE python2.6 is in your path!

There are two ways to do this
  1. modify bashrc
  2. muck about with path_helper
I used #2, but I'm going to explain the first version because it's easiest.

Path fixing Method #1 - bashrc:

Modify ~/.bashrc to look something like:
If you followed to above steps to compile python 2.6, then /usr/local/bin is where the python executables will be symlinked to.

Now in ~/.bash_profile add the following line:
source ~/.bashrc
This will tell the terminal to take a gander at what you wrote in ~/.bashrc

On the command line also type in
$ source ~/.bashrc
so that the terminal you are working in will know what's going on.

Look further on for the Testing of the Path.

Path fixing Method #2 - path_helper:

I found a great link at softec.st which what the path_helper is and how to use it better than I could. In any case, I'm just going to list what I did.
  • put /usr/local/bin at the top line of the /etc/paths file
  • as per Denis' article, modify the /etc/profile like so:
    if [ -x /usr/libexec/path_helper ]; then
    eval `/usr/libexec/path_helper -s`
I'm not sure if you have to restart the system. I did, just to be safe. But if someone knows better, feel free to drop me a line in the comments. Now go on to the Testing of the Path, and make sure to do what's there.

the Testing of the Path:

note: please read the highlighted link on using backticks - shell expansion in bash. It will help make what I do with the file command a little more clear.

carry out these commands from the prompt:
  1. $ which python - this will spit out:
  2. $ file `which python` - this will spit out:
    /usr/local/bin/python: symbolic link to /Library/Frameworks/Python64.framework/Versions/2.6/bin/python
  3. $ file -L `which python` - this follows the symbolic link to the file we want and gives us:
    /usr/local/bin/python: Mach-O universal binary with 4 architectures
    /usr/local/bin/python (for architecture i386): Mach-O executable i386
    /usr/local/bin/python (for architecture ppc7400): Mach-O executable ppc
    /usr/local/bin/python (for architecture ppc64): Mach-O 64-bit executable ppc64
    /usr/local/bin/python (for architecture x86_64): Mach-O 64-bit executable x86_64

Recompile mod_python:

There have been a ton of posts on problems compiling mod_python in Leopard. Graham Dumpleton seems to be one of the guys working on mod_python, and he's posted a lot of stuff on it. In particular are the following two links which are worth reading:

In an earlier post I mentioned two helpful blogs on compiling mod_python. One was from blog.amber.org and the other from matterkilla. However do not use what matterkilla says. If you go through the steps I've outlined, his stuff not only won't help, it will break it down. I'm only keeping him here in case nothing is working and you're desperate.

I mentioned up above in this post that you generally should use the command
export ARCHFLAGS='-arch i386 -arch ppc -arch ppc64 -arch x86_64'
to compile it for all architectures. Again, I'm not sure why you it should be this way, but it helps explain some of what matterkilla was getting at.
  1. First things first. Get a checkout of mod_python
    $ svn co http://svn.apache.org/repos/asf/quetzalcoatl/mod_python/trunk mod_python-trunk
  2. change directory to mod_python-trunk and configure
    $ ./configure --with-apxs=/usr/sbin/apxs --with-python=/usr/local/bin/python
  3. $ make
  4. $ sudo make install
  5. $ cd /etc/apache2
  6. $ sudo cp httpd.conf httpd.conf.orig (in case you mess up)
  7. $ sudo vi /etc/apache2/httpd.conf
  8. add this line:
    LoadModule python_module /usr/libexec/apache2/mod_python.so
  9. $ sudo apachectrl restart
  10. $ sudo pray-to-God-it-works

Testing it out:

the mod_python site explains how to test it out. But before you do anything, open up a new terminal window and type in
$ tail -f /var/log/apache2/error_log
this will open up apache's error log file and print out the end of it as more errors get sent to the file.

Now after you've made all the changes that the mod_python site tells you to, restart apache with
$ sudo apachectl restart
switch to the terminal you have tail running in, and watch the error messages scrolling in the error log. If you see ANYTHING that looks like
[error] python_init: Python version mismatch
then you have a real problem. Make sure that the python version you compiled mod_python with is on the path. I wrote above how to test that out using the file and which commands. If that fails, leave a comment. I probably won't be able to help, but who knows?

Now recompile mysqldb and you're all set.

HAHA - just kidding. Of course, that's the 800 lb. gorilla in the room.

Here are a few useful links I'm going to use on how I got it set up:

Geert lists some good fixes for _mysql.c
fourspaces has a post with the perfect heading for this problem
and Red Elephants lists a problem that may or may not crop up

But one problem that will crop up, is that setuptools 0.6c9 will not download automagically. setuptools is used by the MySQLdb installer to both build and install, in other words, you need it.

So before you do anything else, head over to http://pypi.python.org/pypi/setuptools and you'll be able to download the latest copy and follow the instructions for a Mac installation.

With that out of the way, now to the meat of the thing:
  1. open up the file _mysql.c (it should be at the top of the directory structure)
  2. (from geert):
    Extracted the tar, then edited _mysql.c. Commented lines 37 - 39:

    //#ifndef uint
    //#define uint unsigned int
  3. (more geert _mysql.c goodness):
    and changed this:
    uint port = MYSQL_PORT;
    uint client_flag = 0;

    to this:
    unsigned int port = MYSQL_PORT;
    unsigned int client_flag = 0;
  4. now in site.cfg, make sure to set Threadsafe=False - I read a post somewhere that it actually ended up being a problem. So be safe.
  5. $ python setup.py build
  6. $ sudo python setup.py install
  7. at the command prompt start up the python shell type in
    import MySQLdb
And now you should see something like the following:
Traceback (most recent call last):
File "", line 1, in
File "build/bdist.macosx-10.5-universal/egg/_mysql.py", line 7, in
File "build/bdist.macosx-10.5-universal/egg/_mysql.py", line 6, in __bootstrap__
ImportError: dynamic module does not define init function (init_mysql)
At this point, I imagine that anyone who has followed all the above steps is thinking to themselves "I'm going to hire someone to track down this bastard and ...". Ah ah ahhhh! Not so fast my friend. Put away your checkbook, and hold off from calling your local hitman.

Remember our good friends file (with the -L flag) and which ? Well, using the two we determine that all our python executables are indeed where they should be, that is,
cd to that and do ls -la and we get something like:
python -> python2.6-32
python-32 -> python2.6-32
python-64 -> python2.6-64
python-all -> python2.6-all
python-config -> python2.6-config
python2.6 -> python2.6-32
aha! python is linking to python2.6-32. So what do we do? Nothing actually. I mean, you can feel free to make a symlink to python-all if you like. But it isn't necessary. Apache will pick up the right version.

Whether you make a symlink or you just choose to run python-all from /Library/Frameworks/whatever ..., either way, run it, and NOW try import MySQLdb

You'll get:
/Library/Frameworks/Python64.framework/Versions/2.6/lib/python2.6/site-packages/MySQL_python-1.2.2-py2.6-macosx-10.5-universal.egg/MySQLdb/__init__.py:34: DeprecationWarning: the sets module is deprecated
Which is just a lot of monkey jabber for YOU WIN IT!

Well, you've been such a good audience, and this is has been such a long and different post to write, I think it's time for some well deserved Rush!

The Music of the Universe


  1. this was great but a few things:

    I tried the compile options for python and I got an error basically to the effect of:
    checking for uid_t in sys/types.h... rm: conftest.dSYM: is a directory
    checking for ssize_t... yes
    checking for int... yes
    checking size of int... configure: error: cannot compute sizeof (int)
    See `config.log' for more details.

    So I compiled without them and everything was basically fine until the standard MySQLdb issue. Your fix talking about running 'python-all'... is that mainly created when you create the /Library/Framework/Python{Whatever} directory with the extra compile options? Any clue why I might be getting the error I am? is it perhaps a wrong version of gcc? what other would I use?

  2. Hi, thanks for writing this summary. It's very helpful to have everything in one place, especially as google did indeed turn out to be pretty rubbish at giving good results for this query.

    I approached the problem more simply (but less permanently). If you want to just get MySQLdb running with one command here it is:

    arch -i386 /System/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python

    This forces Python to run as 32 bit and means you don't have to reinstall anything at all.

    Best, Dan.