ChEMBL release 23, technical aspects.

In this blog post, we would like to highlight some important technical improvements we've deployed as a part of the ChEMBL 23 release. You may find them useful if you work with ChEMBL data using FTP downloads and API.

1. FPS format support.

Many users download our SDF file containing all ChEMBL structures in order to compute fingerprints as an immediate next step. We decided to help them and publish precomputed fingerprints in a FPS text fingerprint format. The FPS format was developed by Andrew Dalke to "define and promote common file formats for storing and exchanging cheminformatics fingerprint data sets". It is used by chemfp, RDKit, OpenBabel and CACTVS and we believe it deserves promotion. The computed fingerprints are 2048 bit radius 2 morgan FPs, which we think is the most popular and generic type but please let us know in comments if other type can serve better. We are fully aware that fingerprint type can heavily depend on the specific application but it can be helpful for educational purposes and prototyping.

2. SQLite dump improvements.

As of release 21 we publish a SQLite dump which is an embedded file-based database. This proved to be very useful but as Andrew Dalke noticed on his blog, this dump wasn't optimised. We decided to follow Andrew's advice and pre-analyze tables during dump creation. We hope this will save you a few hours of computing. This is also a first ChEMBL release built with the support of our new Luigi-based pipelines. All the FTP files, including the schema image, SQL dumps and RDF data has been generated automatically using our python workflows. We are hoping to automate more and more parts of the release process, which should result in more frequent data releases and increased reproducibility in the long run.

3. API software updates

ChEMBL API is an open source project. This means that combining it with SQL dumps we provide, everyone can use it to create their own API instance. So far the biggest obstacle with integrating ChEMBL API software with other libraries was the fact, that the API was built on top of very old dependencies. For example, we were using Django 1.5, which was released about 5 years ago. We decided to upgrade the software making it compatible with the latest versions of most critical dependencies. After this change, the ChEMBL API software stack is now compatible with Django 1.11(.2) (which is the LTS edition), haystack 2.6.0, tastypie 0.13.3 and others. As a part of the upgrade process we also switched from virtualenv to conda as a default deployment environment which allowed us to easily install latest RDKit (2017.03.2 at the time of writing) and upgrade the Python interpreter itself (2.7.13). Using conda should make it easier to keep up with a future software updates as well, so from now on our software stack should always be using the latest stable dependencies.

All those changes should have a positive impact on the performance (more about that in the next paragraph), increase the security as well as compatibility with modern software stacks so it should be easier to integrate our software with your existing applications. Also, since all the dependencies are Python3-ready we are much closer to making a switch to Python 3.x. So far we migrated our API client library, which is compatible with both Python 2 and 3.

4. API performance improvements

The main reason to upgrade our software stack was improving the performance of our API. We decided to use Django Prefetch object (introduced in Django 1.7) to fine-tune SQL queries containing joins. We carefully analyzed all SQL queries generated by Django ORM when using the API. Introducing miniconda, which comes with precompiled binaries for python interpreter and libraries like numpy also had a positive impact on the performance, especially molecule images generation. We also added a full text search index on Assay description so now you can perform sophisticated full text queries. For example, searching for Activities, that have related assays with description containing 'TG-GATES', would look like this:

https://www.ebi.ac.uk/chembl/api/data/activity.json?assay_description__icontains=TG-GATES&limit=1000

5. Extending Solr-based search

The above query can be rewritten to use Solr:

https://www.ebi.ac.uk/chembl/api/data/activity.json?/search.json?q=%22TG-GATES%22

This query should be much faster than the one from the previous paragraph. We extended Solr indexes so now they cover 6 ChEMBL entities:

• activities (related targets prefered names and related assays descriptions)
• assays (description)
• documents (abstracts)
• molecules (prefered names, synonyms and related compound records names)
• protein family classification (all levels)
• targets (prefered names and synonyms)

In total we have now indexed 17793020 solr documents. Some more example queries are:

• searching for Alzheimer-related targets: https://www.ebi.ac.uk/chembl/api/data/target/search.json?q=alzheimer
• searching for target by gene name: https://www.ebi.ac.uk/chembl/api/data/target/search.json?q=BRD4

A much more sophisticated query would be one that involves Solr-based search combined with DB-based filtering. For example, getting all assays that match 'inhibitor' in description and have assay type equal to 'A':

https://www.ebi.ac.uk/chembl/api/data/assay/search.json?q=inhibitor&assay_type=A

Such a "federated" query is quite heavy but we managed to optimise this use case. Still, please bear in mind that chaining search with too many filters may cause a timeout if the query is extremely complex.

6. Faster substructure search.

Our API provides the functionality to perform molecule substructure and similarity search. We noticed, that substructure search with the query being a small compound like benzene can lead to timeouts. We decided to enumerate all chemically important small structures and precache the results, which should improve substructure search performance. Please note, that this will have no impact on the speed of substructure search on our main web interface. This is because the interface is not using the API at the moment. We are developing a new API-based interface which should address this problem.

6. New API endpoints

Following new endpoints have been added to the API:

• compound_record - records an occurrence of a molecule in a document
• drug - provides information about approved drugs
• organism - simple organism classification
• target_prediction - target prediction results for clinical compounds, currently used on chembl user interface

7. Better API documentation

We updated the main API web page to reflect recent changes and we added a section with examples. GitHub repository has a new readme file as well and our PyPi packages point to the GitHub repo.

We also recently published a review paper, titled "Using ChEMBL web services for building applications and data processing workflows relevant to drug discovery". The document should be open and deposited into PubMedCentral (https://www.ncbi.nlm.nih.gov/pubmed/28602100) after six months.

8. Training

Please don't forget that we are organising a webinar on the 12th July about the API. More details will be announced soon.

9. Future plans

Our immediate future plans regarding the API are:

• providing a Swagger-based documentation that can be used to generate a client code in any language
• developing a better KNIME node
• publishing a collection of reusable web components that consume the API

Post-doctoral positions

Two exciting post-doctoral projects are available via the ESPOD and EBPOD schemes between the European Bioinformatics Institute and respectively the Sanger Institute and the NIHR Cambridge Biomedical Research Centre (BRC). Post-doctoral fellows appointed via these schemes work on projects under the joint supervision of faculty members from EMBL-EBI and the Sanger or BRC as appropriate. Specifically:

(a) In collaboration with Mathew Garnet at the Sanger Institute, a project to exploit the potential of combining large-scale drug sensitivity screening platforms with the chemogenomics resources and expertise at the EBI.

A full description of the project can be found here: http://www.ebi.ac.uk/sites/ebi.ac.uk/files/groups/research_office/ESPOD2017/05%20Leach-Garnett.pdf.

Applications can be made via the relevant link here: http://www.ebi.ac.uk/research/postdocs/espods

(b) In collaboration with Vasilis Kosmoliaptsis in the Department of Surgery at Addenbrooke’s hospital, to capitalize on our greater understanding of the molecular basis of the immunological response and the ever-growing volumes of genetic and clinical outcomes data to develop new and improved methods for organ transplantation.

Full project description: http://www.ebi.ac.uk/research/postdocs/ebpods/2017-project-2-leach-kosmoliaptsis

Applications via this page: http://www.ebi.ac.uk/research/postdocs/ebpods

Potential applicants should note that these fellowships are awarded after a competitive selection process; more details can be found at the above links or via arl@ebi.ac.uk. The closing date is 1st July.

ChEMBL_23 released

We are pleased to announce the release of ChEMBL_23. This release was prepared on 1st May 2017 and contains:

* 2,101,843 compound records

* 1,735,442 compounds (of which 1,727,112 have mol files)

* 14,675,320 activities

* 1,302,147 assays

* 11,538 targets

* 67,722 source documents

Data can be downloaded from the ChEMBL ftp site: ftp://ftp.ebi.ac.uk/pub/databases/chembl/ChEMBLdb/releases/chembl_23

Please see ChEMBL_23 release notes for full details of all changes in this release: ftp://ftp.ebi.ac.uk/pub/databases/chembl/ChEMBLdb/releases/chembl_23/chembl_23_release_notes.txt

DATA CHANGES SINCE THE LAST RELEASE

In addition to the regular updates to the Scientific Literature, FDA Orange Book and USP Dictionary of USAN and INN Investigational Drug Names and Clinical Candidates, this release of ChEMBL also includes the following new data:

Patent Bioactivity Data

With funding from the NIH Illuminating the Druggable Genome project (https://commonfund.nih.gov/idg), we have extracted bioactivity data relating to understudied druggable targets from a number of patent documents and added this data to ChEMBL.  

Curated Drug Pharmacokinetic Data

We have manually extracted pharmacokinetic parameters for approved drugs from DailyMed drug labels. 

Drug information from British National Formulary and ATC classification

We have now included compound records for drugs that are in the WHO ATC classification or the British National Formulary (BNF). Currently only BNF drugs that already exist in ChEMBL have been assigned compound records. In future releases we will add new BNF drugs to ChEMBL.

Deposited Data Sets

CO-ADD, The Community for Open Antimicrobial Drug Discovery (http://www.co-add.org), is a global open-access screening initiative launched in February 2015 to uncover significant and rich chemical diversity held outside of corporate screening collections. CO-ADD provides unencumbered free antimicrobial screening for any interested academic researcher.  CO-ADD has been recognised as a novel approach in the fight against superbugs by the Wellcome Trust, who have provided funding through their Strategic Awards initiative. Open Source Malaria (OSM) is aimed at finding new medicines for malaria using open source drug discovery, where all data and ideas are freely shared, there are no barriers to participation, and no restriction by patents. The initial set of deposited data from the CO-ADD project consists of OSM compounds screened in CO-ADD assays (DOI = 10.6019/CHEMBL3832881).

Modelled on the Malaria Box, the MMV Pathogen Box contains 400 diverse, drug-like molecules active against neglected diseases of interest and is available free of charge (http://www.pathogenbox.org). The Pathogen Box compounds are supplied in 96-well plates, containing 10​uL of a 10mM dimethyl sulfoxide (DMSO) solution of each compound. Upon request, researchers around the world will receive a Pathogen Box of molecules to help catalyse neglected disease drug discovery. In return, researchers are asked to share any data generated in the public domain within 2 years, creating an open and collaborative forum for neglected diseases drug research. The initial set of assay data provided by MMV has now been included in ChEMBL (DOI = 10.6019/CHEMBL3832761).

FORTHCOMING CHANGES

Schema changes will be made in ChEMBL_24 to accommodate more complex data types. Details of these changes will be released soon. Please follow the ChEMBL blog or sign up to the ChEMBL announce mailing list for details (http://listserver.ebi.ac.uk/mailman/listinfo/chembl-announce)

Changes will also be made in ChEMBL_24 to the way some of the physicochemical properties are calculated. Details of these changes will be announced soon.

Funding acknowledgements:

Work contributing to ChEMBL_23 was funded by the Wellcome Trust, EMBL Member States, Open Targets, National Institutes of Health (NIH) Common Fund, EU Innovative Medicines Initiative (IMI) and EU Framework 7 programmes. Please see https://www.ebi.ac.uk/chembl/funding for more details.

The ChEMBL Team

If you require further information about ChEMBL, please contact us: chembl-help@ebi.ac.uk

# To receive updates when new versions of ChEMBL are available, please sign up to our mailing list: http://listserver.ebi.ac.uk/mailman/listinfo/chembl-announce
# For general queries/feedback please email: chembl-help@ebi.ac.uk
# To report any problems with data content please email: chembl-data@ebi.ac.uk
# For details of upcoming webinars, please see: http://chembl.blogspot.com/search/label/Webinar

Technical internships at ChEMBL.

We are looking for skilled Computer Science (and related fields) students with strong programming skills to join our team for 3-6 month internships. This is not necessarily a summer internship program, you can start whenever convenient for you after being accepted. Please take a look at some of the research ideas / candidate profiles below:

1. Java programmer -  we are looking for a person with experience in Java to develop a prototype of new KNIME nodes for interacting with the ChEMBL API. Experience with REST and/or KNIME is a plus but not a requirement - you can learn it during your internship. A very important thing to note that you should be excited about UX and creating user-friendly and pragmatic GUIs.

2. C++ programmer - we would like to invite a person passionate about C++ and pattern recognition / image processing to experiment with optimising the open-source OSRA code. OSRA is like OCR but for molecules. We want to make it faster and more accurate.

3. C++ programmer with a graph theory knowledge. Chemical compounds are represented as graphs in-silico. We want to be able to quickly generate random graphs that would also be valid compounds. Experience with distributed computing, computing grids, network file systems and map-reduce is a plus but not required.

4. JavaScript programmer - "any application that can be written in JavaScript, will eventually be written in JavaScript". This is why we are looking for a person with JS experience to experiment with:

• Creating prototypes of reusable chemical web widgets using polymer.
• Using emscripten to cross compile some core chemical software written in C++ to JS.

5. A person with a data visualisation skills to explore Kibana and Kibi tools to create beautiful and informative datavis widgets from ChEMBL data.

6. Someone with the Natural Language Processing background to:

• Create a dictionary of common spelling mistakes in chemistry patents.
• Create a network of patent relations using textrank algorithm.
• Explore different approaches to the Named Entity Classification problem.

How to apply?

Just send your CV to kholmes @ ebi.ac.uk with 'ChEMBL Tech Internships' subject.

When to apply?

You can apply anytime but we will only contact selected candidates.

Will all those internships start at the same time?

No, in fact we are planning to select max. 2 most interesting candidates at a given time.

Will I get paid?

The internship is paid 800 GBP per month OR funded by your alma mater (whatever is better for you).

Finding Compounds in Databases using UniChem

Have you ever identified an interesting compound and wondered what else is known about it?  For example is there any bioactivity data on it in ChEMBL or PubChem?  Is there any toxicity data on it (CompTox)?  Then having found interesting data on a compound wondered if it can be purchased or whether it has been patented.  All this can be done using UniChem.  Interested?  

Come along to our webinar on 29th March at 2pm BST (3pm CEST, 9am EDT)

You will however need to register by emailing chembl-help. Places are limited so please let us know as soon as possible if you register but are then unable to attend.

If you want to know more about UniChem please read on.

UniChem (https://www.ebi.ac.uk/unichem/  is a simple system we have developed to cross-reference compounds across databases both internal to EMBL-EBI and externally. Currently we have cross-references to 140 million compounds in 30 different databases. Information about the sources indexed in UniChem can be found here. UniChem is updated weekly with new compounds from these source databases.

So, for example, you can input a database identifier or an InChIKey into UniChem and see links to all the other indexed databases that have information about that compound.

If we take the drug paroxetine and search for it in UniChem, it is found in 22 databases and the UniChem webpage gives links to the paroxetine entries in those databases.

You don’t have to do this compound by compound using the web interface though.  UniChem has a comprehensive set of  web services that you can use to retrieve data or alternatively all the database files and source to source mapping files are available for download.

UniChem relies on the InChIKey to do the mapping between databases and this works fine if two databases have exactly the same structure for a compound.  We all know however that this isn’t always the case.  Sometimes a different salt or isotope was tested or a mistake was made in the stereocentre assignment meaning the InChIKeys no longer match.

However don’t despair.  UniChem connectivity searching can help. https://www.ebi.ac.uk/unichem/info/widesearchInfo  It turns out that because of the clever way that the InChI is built up with layers, this can be deconstructed and mapping can be done such that the relationship between compounds that differ by stereochemistry, isotopes, protonation state etc can all be identified and mapped. You can do this on single components or mixtures.

Taking our paroxetine example:

We have paroxetine and a number of related compounds in ChEMBL. For example:

Maybe someone wanted to genuinely test these related compounds or maybe they are errors (or a mixture of both).  Whatever the reason by using the UniChem connectivity searching feature we can identify any compounds that match paroxetine on the InChI connectivity layer.

The matches identified from a connectivity search starting with paroxetine can be found here:

At the webinar on 29^th March we will describe how this is done in more detail and discuss some use cases.  If you are interested don’t forget to register.

If you want to read more here are links to two papers about UniChem:

Chambers, J., Davies, M., Gaulton, A., Hersey, A., Velankar, S., Petryszak, R., Hastings, J., Bellis, L., McGlinchey, S. and Overington, J.P. 
UniChem: A Unified Chemical Structure Cross-Referencing and Identifier Tracking System.
Journal of Cheminformatics2013, 5:3 (January 2013).

Chambers, J., Davies, M., Gaulton, A., Papadatos, G., Hersey and Overington, J.P.
UniChem: extension of InChI-based compound mapping to salt, connectivity and stereochemistry layers.
Journal of Cheminformatics2014, 6:43 (September 2014)