A python client for accessing ChEMBL web services

Motivation
The CheMBL Web Services provide simple reliable programmatic access to the data stored in ChEMBL database. RESTful API approaches are quite easy to master in most languages but still require writing a few lines of code. Additionally, it can be a challenging task to write a nontrivial application using REST without any examples. These factors were the motivation for us to write a small client library for accessing web services from Python.

Why Python?

We choose this language because Python has become extremely popular (and still growing in use) in scientific applications; there are several Open Source chemical toolkits available in this language, and so the wealth of ChEMBL resources and functionality of those toolkits can be easily combined. Moreover, Python is a very web-friendly language and we wanted to show how easy complex resource acquisition can be expressed in Python.

Reinventing the wheel?

There are already some libraries providing access to ChEMBL data via webservices for example bioservices. These are great, but sometimes suffer from breaking as we implement required schema or implementation details. With the use of this client you can be sure that any future changes in REST API will be immediately reflected in the client. But this is not the only thing we tried to achieve.

Features

During development of the Python client we had in mind all the best practices for creating such a library. We tried to make is as easy to use and efficient as possible. Key features of this implementation are:

• Caching results locally in sqliteDB for faster access
• Bulk data retrieval
• Parallel asynchronous requests to API during bulk retrieval
• Ability to use SMILES containing URL-unsafe characters in exactly same way as safe SMILES
• Lots of syntactic sugar
• Extensive configuration so you can toggle caching, asynchronous requests, change timeouts, point the client to your local web services instance and much more.

These features guarantee that your code using the ChEMBL REST API will be as fast as possible (if you know of any faster way,  drop us a line and we will try to include in the code).

Demo code

OK, so let's see the live code (all examples are included in tests.py file). If you are an ipython notebook fan (like most of us) and prefer reading from notebooks instead of github gists you can click here:

	# necessary imports first:
	from chembl_webresource_client import *
	# create resource object:
	compounds = CompoundResource()
	# before you do anythong else, please check webservices status:
	print compounds.status()
	True
	# If you get 'False' instead, this could mean that your internet connection is broken,
	# your settings are pointing to the wrong server or the server itself is down.
	# get single compound by ChEMBL:
	c = compounds.get('CHEMBL1')
	# `c` is now a python dictionary:
	print c
	{u'smiles': u'COc1ccc2[C@@H]3[C@H](COc2c1)C(C)(C)OC4=C3C(=O)C(=O)C5=C4OC(C)(C)[C@@H]6COc7cc(OC)ccc7[C@H]56',
	u'chemblId': u'CHEMBL1', u'passesRuleOfThree': u'No',
	u'molecularWeight': 544.59, u'molecularFormula': u'C32H32O8',
	u'acdLogp': 7.67, u'stdInChiKey': u'GHBOEFUAGSHXPO-XZOTUCIWSA-N',
	u'knownDrug': u'No', u'medChemFriendly': u'Yes', u'rotatableBonds': 2,
	u'alogp': 3.63, u'numRo5Violations': 1, u'acdLogd': 7.67}
	# for those who prefer xml, this will return xml string:
	# question for the reader: why we choose 'frmt' parameter name, instead of 'format'?
	c = compounds.get('CHEMBL1', frmt='xml')
	print c
	"<?xml version='1.0' encoding='utf-8'?>\n<compound><smiles>COc1ccc2...</compound>"
	# we can get compound by the standard inchi key as well:
	c = compounds.get(stdinchikey='QFFGVLORLPOAEC-SNVBAGLBSA-N')
	print c['molecularFormula']
	'C19H21ClFN3O3'
	# ... or by SMILES, in which case, a list of dictionaries will be returned:
	cs = compounds.get(smiles='COc1ccc2[C@@H]3[C@H](COc2c1)C(C)(C)OC4=C3C(=O)C(=O)C5=C4OC(C)(C)[C@@H]6COc7cc(OC)ccc7[C@H]56')
	print cs[0]['stdInChiKey']
	'GHBOEFUAGSHXPO-UWXQAFAOSA-N'
	# it's so easy to perform substructure search:
	cs = compounds.substructure('COcccc')
	# similarity search is super easy as well:
	cs = compounds.similar_to('COc1ccc2[C@@H]3[C@H](COc2c1)C(C)(C)OC4=C3C(=O)C(=O)C5=C4OC(C)(C)[C@@H]6COc7cc(OC)ccc7[C@H]56', 70)
	# OK, some more complex stuff now - let's get all compound of ids from CHEMBL1 to CHEMBL300:
	cs = compounds.get(['CHEMBL%s' % x for x in range(1,301)])
	# do it again and you will notice it's faster now - cache is working:
	cs = compounds.get(['CHEMBL%s' % x for x in range(1,301)])
	# lets see how many compounds we got back:
	len(cs)
	300
	# hmm, this is strange beacuse there is no compound with chembID = CHEMBL300...
	cs[299]
	404
	# OK, so for some compounds the server will return an error, e.g. 404 = compound does not exist
	# This is fine, as we can remove the 'invalid' compounds with a filter:
	valid_compounds = filter(lambda x: not isinstance(x, int), cs)
	len(valid_compounds)
	170
	# new stuff is working as well, so you can get compound forms...:
	print compounds.forms('CHEMBL415863')
	[{u'chemblId': u'CHEMBL1207563', u'parent': True}, {u'chemblId': u'CHEMBL415863', u'parent': False}]
	# ... and drug mechanisms:
	print compounds.drug_mechnisms('CHEMBL1642')
	[{u'chemblId': u'CHEMBL1862', ..., u'mechanismOfAction': u'Stem cell growth factor receptor inhibitor'}]
	# displaying compound image looks like this:
	from StringIO import StringIO
	from PIL import Image
	buf = StringIO()
	buf.write(compounds.image('CHEMBL1'))
	buf.seek(0)
	im = Image.open(buf)
	im.show()

view raw example.py hosted with ❤ by GitHub

So many features, just for compounds! Let's see targets:

	# our standard invocation:
	from chembl_webresource_client import *
	targets = TargetResource()
	print targets.status()
	True
	# status returns True so we can proceed:
	#first let's get a single target:
	t = targets.get('CHEMBL2477')
	print t
	{u'targetType': u'SINGLE PROTEIN', u'chemblId': u'CHEMBL2477', u'geneNames': u'Unspecified',
	u'description': u'Alpha-amylase 2B ', u'compoundCount': 0, u'bioactivityCount': 0,
	u'proteinAccession': u'P19961',
	u'synonyms': u'1,4-alpha-D-glucan glucanohydrolase 2B,Alpha-amylase 2B,Carcinoid alpha-amylase,AMY2B,3.2.1.1',
	u'organism': u'Homo sapiens', u'preferredName': u'Alpha-amylase 2B '}
	# the same but in xml:
	t = targets.get('CHEMBL2477', frmt='xml')
	print t
	'''<?xml version='1.0' encoding='utf-8'?>\n<target>
	<targetType>SINGLE PROTEIN</targetType><chemblId>CHEMBL2477</chemblId>
	<geneNames>Unspecified</geneNames><description>Alpha-amylase 2B </description>
	<compoundCount>0</compoundCount><bioactivityCount>0</bioactivityCount><proteinAccession>P19961</proteinAccession>
	<synonyms>1,4-alpha-D-glucan glucanohydrolase 2B,Alpha-amylase 2B,Carcinoid alpha-amylase,AMY2B,3.2.1.1</synonyms>
	<organism>Homo sapiens</organism><preferredName>Alpha-amylase 2B </preferredName></target>'''
	# we can get a target by it's uniprot id as well:
	t = targets.get(uniprot='Q13936')
	print t
	{u'targetType': u'SINGLE PROTEIN', u'chemblId': u'CHEMBL1940', u'geneNames': u'Unspecified',
	u'description': u'Voltage-gated L-type calcium channel alpha-1C subunit', u'compoundCount': 178,
	u'bioactivityCount': 246, u'proteinAccession': u'Q13936',
	u'synonyms': u'CCHL1A1,CACNL1A1,Calcium channel, L type, alpha-1 polypeptide, isoform 1, cardiac muscle,Voltage-gated calcium channel subunit alpha Cav1.2,CACNA1C,CACN2,CACH2 ,Voltage-dependent L-type calcium channel subunit alpha-1C',
	u'organism': u'Homo sapiens', u'preferredName': u'Voltage-gated L-type calcium channel alpha-1C subunit'}
	# as with compiunds, we can specify a list of ids to get multiple targets in single call:
	ts = targets.get(['CHEMBL240', 'CHEMBL2477'])
	print ts
	ts
	[
	{u'targetType': u'SINGLE PROTEIN', u'chemblId': u'CHEMBL240',
	u'geneNames': u'Unspecified', u'description': u'HERG', u'compoundCount': 10890,
	u'bioactivityCount': 14056, u'proteinAccession': u'Q12809',
	u'synonyms': u'HERG,hERG-1,ERG ,Voltage-gated potassium channel subunit Kv11.1,Eag homolog,Ether-a-go-go-related protein 1,Potassium voltage-gated channel subfamily H member 2,ERG-1,ERG1,Ether-a-go-go-related gene potassium channel 1,H-ERG,KCNH2,hERG1,Eag-related protein 1',
	u'organism': u'Homo sapiens', u'preferredName': u'HERG'},
	{u'targetType': u'SINGLE PROTEIN', u'chemblId': u'CHEMBL2477',
	u'geneNames': u'Unspecified', u'description': u'Alpha-amylase 2B ', u'compoundCount': 0, u'bioactivityCount': 0,
	u'proteinAccession': u'P19961',
	u'synonyms': u'1,4-alpha-D-glucan glucanohydrolase 2B,Alpha-amylase 2B,Carcinoid alpha-amylase,AMY2B,3.2.1.1',
	u'organism': u'Homo sapiens', u'preferredName': u'Alpha-amylase 2B '}
	]
	# retieving all available targets looks like this:
	all_targets = targets.get_all()
	len(all_targets)
	10983
	# almost 11k targets in a few seconds, great!
	# we can get approved drugs for a given target as well:
	drugs = targets.approved_drugs('CHEMBL1824')
	print drugs[0]['name']
	u'TRASTUZUMAB'

view raw targets.py hosted with ❤ by GitHub

So far, so good! What about assays?

	# first things first:
	from chembl_webresource_client import *
	assays = AssayResource()
	print assays.status()
	True
	# assays are simple, we can get them by chembl ID:
	a = assays.get('CHEMBL1217643')
	print a
	{u'assayType': u'B', u'chemblId': u'CHEMBL1217643', u'journal': u'Bioorg. Med. Chem. Lett.',
	u'assayStrain': u'Unspecified', u'assayOrganism': u'Homo sapiens', u'numBioactivities': 1,
	u'assayDescription': u'Inhibition of human hERG'}
	# xml is supported:
	a = assays.get('CHEMBL1217643', frmt='xml')
	print a
	'''
	<?xml version='1.0' encoding='utf-8'?>\n<assay><assayType>B</assayType>
	<chemblId>CHEMBL1217643</chemblId><journal>Bioorg. Med. Chem. Lett.</journal>
	<assayStrain>Unspecified</assayStrain><assayOrganism>Homo sapiens</assayOrganism>
	<numBioactivities>1</numBioactivities>
	<assayDescription>Inhibition of human hERG</assayDescription></assay>'''
	# as well as bulk retrieval:
	al = assays.get(['CHEMBL1217643', 'CHEMBL1217644'])
	print al
	[
	{u'assayType': u'B', u'chemblId': u'CHEMBL1217643', u'journal': u'Bioorg. Med. Chem. Lett.',
	u'assayStrain': u'Unspecified', u'assayOrganism': u'Homo sapiens', u'numBioactivities': 1,
	u'assayDescription': u'Inhibition of human hERG'},
	{u'assayType': u'B', u'chemblId': u'CHEMBL1217644', u'journal': u'Bioorg. Med. Chem. Lett.',
	u'assayStrain': u'Unspecified', u'numBioactivities': 1, u'assayDescription': u'Inhibition of MAOA'
	}
	]

view raw assays.py hosted with ❤ by GitHub

Can I get bioactovities as well?

	from chembl_webresource_client import *
	assays = AssayResource()
	targets = TargetResource()
	compounds = CompoundResource()
	# Once we have our resources, getting activities is easy:
	bs = assays.bioactivities('CHEMBL1217643')
	len(bs)
	1
	print bs
	[
	{u'units': u'nM', u'reference': u'Bioorg. Med. Chem. Lett., (2010) 20:15:4359', u'target_chemblid': u'CHEMBL240',
	u'target_name': u'HERG', u'bioactivity_type': u'IC50', u'ingredient_cmpd_chemblid': u'CHEMBL1214402',
	u'value': u'5900', u'assay_chemblid': u'CHEMBL1217643', u'parent_cmpd_chemblid': u'CHEMBL1214402',
	u'operator': u'=', u'activity_comment': u'Unspecified', u'name_in_reference': u'26',
	u'assay_description': u'Inhibition of human hERG', u'organism': u'Homo sapiens', u'assay_type': u'B',
	u'target_confidence': 9}
	]
	# after retrieving this you will appreciate caching:
	bs = targets.bioactivities('CHEMBL240')
	len(bs)
	14056
	print bs[1400]
	{u'units': u'nM', u'reference': u'Bioorg. Med. Chem. Lett., (2008) 18:3:994', u'target_chemblid': u'CHEMBL240',
	u'target_name': u'HERG', u'bioactivity_type': u'IC50', u'ingredient_cmpd_chemblid': u'CHEMBL429849',
	u'value': u'844', u'assay_chemblid': u'CHEMBL944859', u'parent_cmpd_chemblid': u'CHEMBL429849',
	u'operator': u'=', u'activity_comment': u'Unspecified', u'name_in_reference': u'4j',
	u'assay_description': u'Displacement of [33S]MK499 from human ERG expressed in HEK cells',
	u'organism': u'Homo sapiens', u'assay_type': u'B', u'target_confidence': 9}
	# xml format is supported, of course:
	bs = compounds.bioactivities('CHEMBL1', frmt='xml')
	print bs
	'''
	"<?xml version='1.0' encoding='utf-8'?>\n<list><bioactivity><target__confidence>9</target__confidence><reference>J.
	Med. Chem., (2008) 51:22:7132</reference><assay__type>B</assay__type><target__name>P-glycoprotein 1</target__name>
	<assay__chemblid>CHEMBL952394</assay__chemblid><operator>Unspecified</operator>
	<bioactivity__type>Activity</bioactivity__type><units>Unspecified</units><target__chemblid>CHEMBL4302</target__chemblid>
	<ingredient__cmpd__chemblid>CHEMBL1</ingredient__cmpd__chemblid><parent__cmpd__chemblid>CHEMBL1</parent__cmpd__chemblid>
	<name__in__reference>9</name__in__reference><activity__comment>Active</activity__comment><value>Unspecified</value>
	<assay__description>Inhibition of human MDR1 overexpressed in mouse NIH/3T3 cells assessed as daunorubicin accum...
	'''

view raw bioactivities.py hosted with ❤ by GitHub

It's completely optional to change any settings in the ChEMBL client. We believe that default values we have chosen are optimal for most reasonable applications. However if you would like to have a play with settings to make our client work with your local webservices instance this is possible:

	# let's import settings class first:
	from chembl_webresource_client.settings import Settings
	# we can now decide which protocol to use:
	Settings.Instance().WEBSERVICE_PROTOCOL = 'http'
	# point our client to our local webservices instance
	Settings.Instance().WEBSERVICE_DOMAIN = 'localhost'
	# and change their prefix:
	Settings.Instance().WEBSERVICE_PREFIX = '/ws'
	# so now root url will look like this:
	# 'http://localhost/ws', instead of default:
	# 'https://www.ebi.ac.uk/chemblws'
	# default request timeout can be easily changed as well:
	Settings.Instance().TIMEOUT = 10
	# we can also decide to swhich off cache:
	Settings.Instance().CACHING = False
	# or switch off fast saving in which case we make sure data sored in cache won't be corrupted:
	Settings.Instance().FAST_SAVE = False
	# when retrieving data asynchronously we can decied how many requests will be made at once:
	Settings.Instance().FAST_SAVE = CONCURRENT_SIZE = 15
	# or how large should the bulk retrieval be to use async requests:
	Settings.Instance().ASYNC_TRESHOLD = 5

view raw settings.py hosted with ❤ by GitHub

GET or POST?

	import requests
	from chembl_webresource_client import *
	# For some special SMILES, using GET method against webservices will end up with failure:
	res = requests.get('https://www.ebi.ac.uk/chemblws/compounds/smiles/CN1C\C(=C/c2ccc(C)cc2)\C3=C(C1)C(C(=C(N)O3)C#N)c4ccc(C)cc4')
	print res.ok
	False
	print res.status_code
	400
	# This is because the SMILES contain slash '/' character, which has a special meaning in URL.
	# The solution to this problem is to use POST request, which will look like this:
	res = requests.post('https://www.ebi.ac.uk/chemblws/compounds/smiles', data={'smiles':'CN1C\C(=C/c2ccc(C)cc2)\C3=C(C1)C(C(=C(N)O3)C#N)c4ccc(C)cc4'}, headers={'Accept':'application/xml'})�
	print res.ok
	True
	print res.content
	"<?xml version='1.0' encoding='utf-8'?>\n<list><compound><smiles>CN1C\\C(=C/c2ccc(C)cc2)\\C3=C(C1)C(C(=C(N)O3)C#N)c4ccc(C)cc4</smiles>..."
	# In that case maybe it's better to use POST for all requests to ChEMBL?
	# Unfortunately (currently) there are only a few methods (search by SMILES, substructure and similarity serach)
	# which support POST. Besides, GET requests contain all necessary data in the URL and such an URL can be embedded
	# on a website, send by email or chat, so in some cases they are more useful then POST.
	# But if we have a large number of SMILES, deciding whether GET or POST should be used for each one of them can be
	# problematic. Our client can transparently handle this problem:
	compounds = CompoundResource()
	# Getting compound by SMILES containing only safe characters...
	cs = compounds.get(smiles='COc1ccc2[C@@H]3[C@H](COc2c1)C(C)(C)OC4=C3C(=O)C(=O)C5=C4OC(C)(C)[C@@H]6COc7cc(OC)ccc7[C@H]56')
	print cs[0]['molecularFormula']
	'C32H32O8'
	# ...looks exaclty the same as using SMILES with some unsafe characters such as slash:
	cs = compounds.get(smiles="C\C(=C/C=C/C(=C/C(=O)O)/C)\C=C\C1=C(C)CCCC1(C)C")
	print cs[0]['preferredCompoundName']
	'MMAOIAFUZKMAOY-UHFFFAOYSA-N'

view raw get_or_post.py hosted with ❤ by GitHub

Benchmarks

We've decided to compare our client with existing bioservices implementation. Before we describe method and results, let's say a few words about installation process. Both packages can be installed from PIP, but bioservices are quite large (1.8MB) and require dependencies not directly related to web retrieval (such as pandas or SOAPpy). On the other hand our client is rather small (<0.5 MB) and require requests, request-cache and grequests.

To compare two libraries, we've decided to measure time of retrieval first thousand of compounds with IDs from CHEMBL1 to CHEMBL1000. We've ignored 404 errors. This is how the code looks for our client:

	from chembl_webresource_client import *
	compounds = CompoundResource()
	cs = compounds.get(['CHEMBL%s' % x for x in range(1,1001)])

view raw client_benchmark.py hosted with ❤ by GitHub

And for bioservices:

	from bioservices import *
	s = ChEMBLdb(verbose=False)
	res = []
	for i in range(1,1001):
	try:
	res.append(s.get_compounds_by_chemblId('CHEMBL%s' % i))
	except:
	res.append(404)

view raw bioservices_benchmark.py hosted with ❤ by GitHub

Both snippets were run 5 times and the average time was computed.


Results:
chembl client with cache: 4.5s
chembl client no cache: 6.7s
bioservices: 9m40s


which means, that our client is 86-145 times faster than the bioservices client.

Installation and source code



Our client is already available at Python Package Index (PyPI), so in order to install it on your machine machine, you should type:

sudo pip install chembl_webresource_client

or:

pip install chembl_webresource_client

if you are using virtualenv.

The original source code is open sourced and hosted at github so you can clone it by typing:

git clone https://github.com/chembl/chembl_webresource_client.git

install latest development version, using following pip command:

sudo pip install git+https://github.com/chembl/chembl_webresource_client.git  

 
or just browse it online.

What about other programming languages?

Although we don't have plans to release a similar client library for other programming languages, examples highlighting most important API features using Perl and Java are published on Web Services homepage. And since the Web Services have CORS and JSONP support, we will publish JavaScript examples in the form of interactive API browser, so stay tuned!


Michal