Qt – Compiling QGIS 3.x on Ubuntu 16.04 with Python3.6 and Latest Qt 5 and PyQt5 Modules

compilepyqt5python 3qgis-3qt

The problem:

Now that the compilation issue is over (Unable to compile QGIS 3.x (master) on Ubuntu 16.04: PyQt5.pyqtconfig module does not exist), QGIS 3 (master) refuses to load Python at startup.
I am working on Ubuntu 16.04 (4.15.0-30-generic x86_64 GNU/Linux).
– Python3 version is 3.5.2
– PyQt5 is 5.10.1
– SIP is 4.19.12 (manually installed, detected version during the compilation is 4.19.12, but sip.SIP_VERSION_STR when imported directly in a python3.5 console is 4.19.8. I cannot understand why)
– QScintilla is 2.10.4
These packages have been dowgraded as explained here: Unable to compile QGIS 3.x (master) on Ubuntu 16.04: PyQt5.pyqtconfig module does not exist

Here is the error message I am facing in a popup window during QGIS 3 startup:

Couldn't load PyQt.
Python support will be disabled.


Traceback (most recent call last):
  File "", line 1, in 
ImportError: /usr/lib/x86_64-linux-gnu/libQt5Core.so.5: version `Qt_5.10' not found (required by /usr/local/lib/python3.5/dist-packages/PyQt5/QtCore.so)


Python version:
3.5.2 (default, Nov 23 2017, 16:37:01) 
[GCC 5.4.0 20160609]

QGIS version:
3.3.0-Master 'Master', ad4ddb1

Followed by a second popup window:

An error occurred during execution of following code:
qgis.utils.uninstallErrorHook()


Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'qgis' is not defined


Python version:
3.5.2 (default, Nov 23 2017, 16:37:01) 
[GCC 5.4.0 20160609]

QGIS version:
3.3.0-Master 'Master', ad4ddb1

Then QGIS is ready to use, but there is no Python console (which I really need).

Second try; updating Python3 from `3.5.2` to `3.6.5`:

Then I though it was a Python related issue, so I decided to install Python 3.6.5 (from this PPA: http://ppa.launchpad.net/jonathonf/python-3.6/ubuntu) in parallel to the default system Python 3.5.2 in Ubuntu 16.04.
So I changed the PYTHON_EXECUTABLE, PYTHON_INCLUDE_DIR and PYTHON_LIBRARY to version 3.6 instead of 3.5.
I also made a pip3.6 command in order to install packages for this version of Python and I installed, this time the latest version of the PyQt5 serie:
– PyQt5 version: 5.11.2
– SIP version: 4.19.12
– QScintilla2 PyQt module: 2.10.7

At QGIS startup, the same two popup windows appeared:

Couldn't load PyQt.
Python support will be disabled.

Traceback (most recent call last):
  File "", line 1, in 
ImportError: /usr/lib/x86_64-linux-gnu/libQt5Core.so.5: version `Qt_5.11' not found (required by /usr/local/lib/python3.6/dist-packages/PyQt5/QtCore.so)

Python version:
3.6.5 (default, May  3 2018, 10:08:28) 
[GCC 5.4.0 20160609]

QGIS version:
3.3.0-Master 'Master', ad4ddb1

Followed by:

An error occurred during execution of following code:
qgis.utils.uninstallErrorHook()


Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'qgis' is not defined


Python version:
3.6.5 (default, May  3 2018, 10:08:28) 
[GCC 5.4.0 20160609]

QGIS version:
3.3.0-Master 'Master', ad4ddb1

What I realized at this point:

is that I was using the Qt5 I installed from the Ubuntu repositories, which version is Qt 5.5.1.
That's probably why QGIS cannot find the Qt_5.10' string the first try and theQt_5.11' string in the second try. It probably wants the system Qt version to be the same than the PyQt module, in the first case PyQt was 5.10.1 and in the second 5.11.2. So I assume they should be like a family and share the same version numbers (at least up to the minor digit).
I looked after 'how to install Qt 5.10 or Qt 5.11 on Ubuntu 16.04 and foud these PPA:
– http://ppa.launchpad.net/beineri/opt-qt-5.10.1-xenial/ubuntu
– http://ppa.launchpad.net/beineri/opt-qt-5.11.1-xenial/ubuntu

I added them both, installed Qt5.10 and Qt5.11 in /opt as described on the respective web pages from the PPAs and changed all the Qt5 variable path in cmake to the ones in '/opt/qt51x/lib/cmake/' + the QT_LRELEASE_EXECUTABLE variable to '/opt/qt51x/bin/lrelease' with x in [0 when building QGIS with Python3.5 and x=1 when building with Python3.6].

But then…

New issue:

The configuration and generation is all OK, but the build outputs a lot (!) of C++ errors. I don't want to print them all but it looks like that:

https://pastebin.com/vZ20XPxv

So, to summarize:

Either QGIS compile well with a PyQt 5.11.2 and system Qt 5.5.1 but refuses to load the Python console at startup telling that it cannot find `Qt_5.11' in '/usr/lib/x86_64-linux-gnu/libQt5Core.so
or QGIS doesn't compile when PyQt and Qt version are the same (up to the minor digit).

Best Answer

So, based on AndreJ comment, I uninstall all PyQt5 related packages with pip3.5 (namely pyqt5, sip, qscintilla and pyqtchart) and used the official ones from my Ubuntu distribution.

Here is the cmake log which looks fine:

QGIS version: 3.3.0 Master (30300)
Could not find GRASS 7
Found Proj: /usr/local/lib/libproj.so (5.2.0)
Found GEOS: /usr/lib/x86_64-linux-gnu/libgeos_c.so (3.5.0)
Found GDAL: /usr/local/lib/libgdal.so (2.3.1)
Found Expat: /usr/lib/x86_64-linux-gnu/libexpat.so
Found Spatialindex: /usr/lib/x86_64-linux-gnu/libspatialindex.so
Found Qwt: /usr/lib/libqwt-qt5.so (6.1.2)
Found libzip: /usr/lib/x86_64-linux-gnu/libzip.so
Found Sqlite3: /usr/lib/x86_64-linux-gnu/libsqlite3.so
Found PostgreSQL: /usr/lib/x86_64-linux-gnu/libpq.so
Found SpatiaLite: /usr/lib/x86_64-linux-gnu/libspatialite.so
Qt WebKit support DISABLED.
Found Qt version: 5.5.1
Found QScintilla2: /usr/lib/libqt5scintilla2.so (2.9.1)
Found QtKeychain: /usr/lib/x86_64-linux-gnu/libqt5keychain.so
Found QCA: /usr/lib/x86_64-linux-gnu/libqca-qt5.so (2.1.1)
Found QCA OpenSSL plugin
Pedantic compiler settings enabled
Found Python executable: /usr/bin/python3
Found Python version: 3.5.2
Found Python library: /usr/lib/x86_64-linux-gnu/libpython3.5m.so
Found Python site-packages: /usr/lib/python3/dist-packages
Found PyQt5 version: 5.5.1
Found SIP version: 4.17
Found QScintilla2 PyQt module: 2.9.1
Found GSL: -L/usr/lib/x86_64-linux-gnu -lgsl -lgslcblas -lm
Ctest Binary Directory set to: /opt/QGIS/build_master/output/bin
Configuring done
Generating done

(Note: one has to deactivate QtWebKit because it was deprecated since Qt>=5.5)

And here is an error during the compilation:

[ 30%] Generating core/sip_corepart0.cpp, core/sip_corepart1.cpp, core/sip_corepart2.cpp, core/sip_corepart3.cpp
/bin/sh: 1: /usr/bin/sip: not found
python/CMakeFiles/python_module_qgis__core.dir/build.make:515: recipe for target 'python/core/sip_corepart0.cpp' failed
make[2]: *** [python/core/sip_corepart0.cpp] Error 127
make[2]: *** Deleting file 'python/core/sip_corepart0.cpp'
CMakeFiles/Makefile2:4078: recipe for target 'python/CMakeFiles/python_module_qgis__core.dir/all' failed
make[1]: *** [python/CMakeFiles/python_module_qgis__core.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
Makefile:160: recipe for target 'all' failed
make: *** [all] Error 2

It seems that cmake can find sip during the configuration step but the compilation step with make does not. And I do not have any /usr/bin/sip on my system even after reinstalling python3-sip and python3-sip-dev packages with apt.

Parenthesis:

If you run into a sipconfig error or a:

CMake Error at cmake/FindSIP.cmake:54 (MESSAGE):
  Could not find SIP
Call Stack (most recent call first):
  CMakeLists.txt:753 (FIND_PACKAGE)

after having pip3.5 uninstalled a latest sip package and if you already have the system sip (python3-sip python3-sip-dev) just force the reinstallation of the later:

$ apt-get install python3-sip python3-sip-dev
Reading package lists... Done
Building dependency tree       
Reading state information... Done
python3-sip is already the newest version (4.17+dfsg-1build1).
python3-sip-dev is already the newest version (4.17+dfsg-1build1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

$ apt-get install --reinstall python3-sip python3-sip-dev

Then cmake will find it.

Identified problem:

There is two files in the QGIS sources cmake/ folder:
- FindSIP.cmake
- FindSIP.py

the later beeing called by the former.

In the python file, one can read at line:39:

print("sip_bin:%s" % sipcfg.sip_bin)

this line prints:

sip_bin:/usr/bin/sip

But on the system, there is no such sip file:

ll /usr/bin | grep -i sip
-rwxr-xr-x 1 947 oct 24  2015 dh_sip3*
-rwxr-xr-x 1 868 jan 18  2016 dh_sip*
-rwxr-xr-x 1 59K mai 16 17:00 lsipc*

There is a sip folder /usr/share but no executable.

By trying to run sip:

$ sip -V
The program 'sip' is currently not installed. You can install it by typing: 
sudo apt install sip-dev

So.... install it:

$ apt-get install sip-dev

Check it's there:

$ sip -V
4.17
$ ll /usr/bin/ | grep sip
-rwxr-xr-x 1 947 oct 24  2015 dh_sip3*
-rwxr-xr-x 1 868 jan 18  2016 dh_sip*
-rwxr-xr-x 1 357K jan 18  2016 sip*
-rwxr-xr-x 1 59K mai 16 17:00 lsipc*

Yepee, sip is here now!

And retry the QGIS compilation...:

[...]
[ 99%] Building CXX object python/CMakeFiles/python_module_qgis__gui.dir/gui/sip_guipart0.cpp.o
[ 99%] Building CXX object python/CMakeFiles/python_module_qgis__gui.dir/gui/sip_guipart1.cpp.o
[ 99%] Built target translations
[ 99%] Building CXX object python/CMakeFiles/python_module_qgis__gui.dir/gui/sip_guipart2.cpp.o
[ 99%] Building CXX object python/CMakeFiles/python_module_qgis__gui.dir/gui/sip_guipart3.cpp.o
[100%] Built target staged-plugins-on-build

\o/ YES!

The new issues (2x):

Couldn't load PyQGIS.
Python support will be disabled.


Traceback (most recent call last):
  File "", line 1, in 
  File "/opt/QGIS/build_master/output/python/qgis/core/__init__.py", line 27, in 
    from qgis._core import *
ImportError: No module named 'PyQt5.QtSql'


Python version:
3.5.2 (default, Nov 23 2017, 16:37:01) 
[GCC 5.4.0 20160609]

QGIS version:
3.3.0-Master 'Master', a82de83

To fix this one, one has to install pyqt5.qtsql manually (it does not seem to come along python3-pyqt5):

$ apt-get install python3-pyqt5.qtsql

(...) Same thing for 'PyQt5.QtSvg':

$ apt-get install python3-pyqt5.qtsvg

Then, QGIS finally starts normally with a nice Python console:

The two last issues were clearly due to a lack in the dependencies installation from my side.

The `FindPyQt5.py` error:

By looking in more detail the FindPyQt5.py file, one can see at lines 33-41:

try:
    import PyQt5.pyqtconfig
    pyqtcfg = PyQt5.pyqtconfig.Configuration()
except ImportError:
    import PyQt5.QtCore
    import sipconfig # won't work for SIP v5
    import os.path
    import sys
cfg = sipconfig.Configuration()

So, the pyqtconfig error is related to the first part of the 'try' statement.

Then, in the 'except' part of the code, what I didn't mentionned is that I encountered some error as well which I thought was not important. But it actually was! QtCore module cannot load. It says (from a Python3.5 console):

from PyQt5 import QtCore
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: /usr/local/lib/python3.5/dist-packages/PyQt5/QtCore.so: undefined symbol: PySlice_AdjustIndices

The issue:

The problem is that with the Python3.5 version shipped with Ubuntu 16.04 --- which subversion is Python 3.5.2 --- this is a known bug in PyQt5 version 5.11.2 (latest at that time).

For compatibility reasons I did not want to try upgrading the system Python 3.5.2 to 3.5.4 where this issue seems to have been fixed.

The solution:

One must downgrade PyQt5 from 5.11.2 to 5.10.1.

After several tries, I found that this involves to downgrade as well theses two packages:
- PyQtChart from 5.11.2 to 5.10.1
- and qscintilla from 2.10.7 to 2.10.4

At this point, the QtCore module loads correctly!

But then, some other errors appears during the QGIS first during the configuration step with cmake and then during the compilation;

If there is some "SIP not found" during the configuration process (even if it is installed!) or many errors about a sip_corepart3.cpp during the compilation, one can upgrade it (manually, because the version downloaded by pip is an older version (sip 4.19.8)).
To do that, one must download the tarball from here:
https://www.riverbankcomputing.com/software/sip/download (sip version 4.19.12) and extract it to some folder.
From this folder, run python3.5 configure.py, make and make install (Detailed explanations here: http://pyqt.sourceforge.net/Docs/sip4/installation.html ).
Only then the SIP error disappears and the compilation finally goes to the end!
An issue with Qt WebKit:
/opt/QGIS/src/core/qgswebpage.h:26:20: fatal error: QWebPage: No such file or directory
=> one has to disable the WITH_QTWEBKIT flag in cmake.
QtWebKit is deprecated in Qt >= 5.5. It has been replaced by QtWebEngineCore. (https://doc.qt.io/qt-5.11/whatsnew55.html#deprecated-functionality More infos: https://forum.qt.io/topic/55504/anyone-knows-the-future-of-qt-webkit & http://doc.qt.io/qt-5/qtwebenginewidgets-qtwebkitportingguide.html ).

To summarize, this is the output log from cmake with all the changes. It doesn't show erros anymore:

QGIS version: 3.3.0 Master (30300)
Could not find GRASS 7
Found Proj: /usr/local/lib/libproj.so (5.2.0)
Found GEOS: /usr/lib/x86_64-linux-gnu/libgeos_c.so (3.5.0)
Found GDAL: /usr/local/lib/libgdal.so (2.3.1)
Found Expat: /usr/lib/x86_64-linux-gnu/libexpat.so
Found Spatialindex: /usr/lib/x86_64-linux-gnu/libspatialindex.so
Found Qwt: /usr/lib/libqwt-qt5.so (6.1.2)
Found libzip: /usr/lib/x86_64-linux-gnu/libzip.so
Found Sqlite3: /usr/lib/x86_64-linux-gnu/libsqlite3.so
Found PostgreSQL: /usr/lib/x86_64-linux-gnu/libpq.so
Found SpatiaLite: /usr/lib/x86_64-linux-gnu/libspatialite.so
Qt WebKit support DISABLED.
Found Qt version: 5.5.1
Found QScintilla2: /usr/lib/libqt5scintilla2.so (2.9.1)
Found QtKeychain: /usr/lib/x86_64-linux-gnu/libqt5keychain.so
Found QCA: /usr/lib/x86_64-linux-gnu/libqca-qt5.so (2.1.1)
Found QCA OpenSSL plugin
Pedantic compiler settings enabled
Found PythonInterp: /usr/bin/python3.5 (found suitable version "3.5.2", minimum required is "3") 
Found Python executable: /usr/bin/python3.5
Found Python version: 3.5.2
Found Python library: /usr/lib/x86_64-linux-gnu/libpython3.5m.so
Found Python site-packages: /usr/lib/python3/dist-packages
Found PyQt5 version: 5.10.1
Found SIP version: 4.19.12
Found QScintilla2 PyQt module: 2.10.4
Found GSL: -L/usr/lib/x86_64-linux-gnu -lgsl -lgslcblas -lm
Ctest Binary Directory set to: /opt/QGIS/build_master/output/bin
Configuring done
Generating done

And the compilation goes to the end, but QGIS itself refuses to load the Python console at startup*!

*It's probably better to separate this issue into a new thread:
Compiling QGIS 3.x (master) on Ubuntu 16.04 with Python3.6 and the latest Qt 5 and PyQt5 familiy modules (PyQt5, sip, qscintilla, pyqtchart)

[GIS] QGIS Compiling 3.6.0 Troubleshooting about PyQt5, SIP, Qscintilla on Ubuntu 18.04 LTS

Success it Works

Finally I find a way to start correctly QGIS 3.6.1, with console Python. But I must say I don't decipher correctly the release relationship between SIP, PyQt5, Qt, and QGIS. My conclusion is sip 4.19.7 works for me.

How I proceed ?

First I remove any package (through apt-get (Unbuntu) or pip3 (Python) linked to sip, pyqt5, qscintilla
second with apt I have done the following install from the Ubuntu 18.04 LTS

sudo apt install sip-dev python3-sip pyqt5-dev pyqt5.qsci-dev python-sip-dev python3-sip pyqt5-dev-tools python3-pyqt5 python3-pyqt5.qtmultimedia python3-pyqt5.qtopengl python3-pyqt5.qtquick python3-pyqt5.qtsql python3-pyqt5.qtsvg python3-pyqt5.qtwebkit python3-sip python3-sip-dev libqscintilla2-qt5-dev libqgis-customwidgets libqgis-gui3.4.5 libqscintilla2-qt5-13 openscad python-pyqt5.qsci python-qgis python-qgis-common python3-pyqt5.qsci

An then, I prepared the Makefile with CMake for QGIS 3.6.1, it was correct I runned the compile followed by install it succeeded !

On command line : > sip -V => 4.19.7

And when I run the following python script :

import sys
from sip import SIP_VERSION_STR
import sip
import qgis
print("SIP version:", SIP_VERSION_STR)

The result was :

SIP version: 4.19.13 PyQt version: 5.10.1 QScintilla version: 2.10.2

Two different numbers release for sip curious ?

When I start QGIS 3.6.1 , it failed about Python console and extension loading => I have the following messages "I translate from french language" : can'load extension 'geomovetools' call error to classFactory() ValueError: PyCapsule_GetPointer called with incorrect name

After some turn around on stackexchange I found the suggestion that I will have to recompile sip 4.19.7 from code source => https://github.com/eduardosm/sip-4.19.7

With the following configuration command :

python3 configure.py --sip-module PyQt5.sip --no-tools

And make and sudo make install for this sip built

I did that as indicated!

Now Everything is OK ! and QGIS 3.6.1 starts perfectly on Ubuntu 18.04 LTS

Now On my side I have :

sip -V => 4.19.7 (from /usr/bin/sip)

And when I run the following script :

import sys
from PyQt5.QtCore import QT_VERSION_STR
from PyQt5.Qt import PYQT_VERSION_STR
from sip import SIP_VERSION_STR
import sip
import qgis
from PyQt5.Qsci import QSCINTILLA_VERSION_STR
print(sip.__file__)
print("Qt version:", QT_VERSION_STR)
print("SIP version:", SIP_VERSION_STR)
print("PyQt version:", PYQT_VERSION_STR)
print("QScintilla version:", QSCINTILLA_VERSION_STR)

The result is :

usr/lib/python3/dist-packages/sip.cpython-36m-x86_64-linux-gnu.so
Qt version: 5.9.5
SIP version: 4.19.7
PyQt version: 5.10.1
QScintilla version: 2.10.2

I hope this can help others, for me it was like an escape from a jail where I was trapped for one month. Enjoy!