As a scientist, your life’s work is your publication list. I like to be intimate with mine. Sometimes I just stare at it. I’d buy it a glass of wine if I could. Maybe even caress it softly. Sure, she ain’t much to look at, but she’s mine, and I want to show her off. And if you want a job, you’re going to want to show yours off too. So I’m going to show you how to scrape your publication list from Pubmed with Python.<\/p>\n
When I was a youngster, I used to like writing JavaScript to try and scrape content from webpages in whatever weird way my uneducated little brain could manage. Now I’m a big boy, I prefer to do it nicely, with HTTP requests and XML parsing. But as you will see in part II of the post, I’m still not afraid to get a little rough, and go old school on pages that won’t be obedient. But here, we can be nice and romantic like.<\/p>\n
PubMed has an API<\/a>. But the looks of things it’s pretty old, and doesn’t include many of their newer features, but to show your CV, it’ll work perfect.<\/p>\n The real meat of the script is in two functions. The first is The second meaty function What am I doing? Each returned article is hidden in contained between Finally, I iterate of the papers list and print out appropriate HTML, a step you may or may not want. The output looks like this<\/a> and the full script is available here<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":" As a scientist, your life’s work is your publication list. I like to be intimate with mine. Sometimes I just stare at it. I’d buy it a glass of wine if I could. Maybe even caress it softly. Sure, she ain’t much to look at, but she’s mine, and I want to show her off.…Read more Interacting with PubMed. Part I<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[3,6],"tags":[],"_links":{"self":[{"href":"https:\/\/www.billconnelly.net\/index.php?rest_route=\/wp\/v2\/posts\/44"}],"collection":[{"href":"https:\/\/www.billconnelly.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.billconnelly.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.billconnelly.net\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.billconnelly.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=44"}],"version-history":[{"count":11,"href":"https:\/\/www.billconnelly.net\/index.php?rest_route=\/wp\/v2\/posts\/44\/revisions"}],"predecessor-version":[{"id":816,"href":"https:\/\/www.billconnelly.net\/index.php?rest_route=\/wp\/v2\/posts\/44\/revisions\/816"}],"wp:attachment":[{"href":"https:\/\/www.billconnelly.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=44"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.billconnelly.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=44"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.billconnelly.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=44"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}search_pubmed<\/code>. We load up a url with a set of parameters, most importantly the term<\/code> parameter, which is what we actually search. Your name should suffice. This basically returns to us some XML that contains the key to the results, the “WebEnv” string. We attach this to our params, use the efetch interface, and we get lovely XML document containing the results.<\/p>\ndef search_pubmed(term):\r\n params= {\r\n 'db': 'pubmed',\r\n 'tool': 'test',\r\n 'email':'test@test.com',\r\n 'term': term,\r\n 'usehistory':'y',\r\n 'retmax':20\r\n }\r\n url = 'http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?' + urllib.urlencode(params)\r\n\r\n tree = ET.fromstring(urllib.urlopen(url).read())\r\n\r\n params['query_key'] = tree.find("./QueryKey").text\r\n params['WebEnv'] = tree.find("./WebEnv").text \r\n params['retmode'] = 'xml'\r\n\r\n url = 'http://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?' + urllib.urlencode(params)\r\n data = urllib.urlopen(url).read()\r\n\r\n return data\r\n<\/pre>\nxml_to_papers()<\/code> chews up the XML, and puts the appropriate bits into a dictionary, and then appends the dictionary to a list which it then returns. I use the xml.etree.ElementTree<\/code>. In my opinion, it is not half as nice as the BeautifulSoup<\/code> module. However, for some reason I cannot get my server to run custom modules when it serves HTML. I used a dictionary instead of making a custom class. I did this for one simple reason. A dictionary would work. Why write more code than you have to?<\/p>\n<PubmedArticle><MedlineCitation><\/code> tags. I break each of those up with a tree.findall()<\/code> and then iterate over them. Often I pass these sections of XML to helper functions, because PubMed returns different XML tags depending on the data. i.e. <MedlinePgn><\/code> is not always there, so I need to check if it exists, before I try to get the text it contains.<\/p>\ndef xml_to_papers(data):\r\n tree = ET.fromstring(data)\r\n articles = tree.findall("./PubmedArticle/MedlineCitation")\r\n\r\n papers = []\r\n for article in articles:\r\n paper = dict()\r\n paper["journal_name"] = article.find("./Article/Journal/ISOAbbreviation").text\r\n paper["title"] = article.find("./Article/ArticleTitle").text\r\n paper["authors"] = digest_authors(article.findall("./Article/AuthorList/Author"))\r\n paper["issue"] = digest_issue(article.find("./Article/Journal/JournalIssue"))\r\n paper["year"] = digest_year(article)\r\n paper["page_num"] = checkXML(article, "./Article/Pagination/MedlinePgn")\r\n paper["pmid"] = article.find("./PMID").text\r\n paper["doi"] = checkXML(article, "./Article/ELocationID") \r\n \r\n papers.append(paper)\r\n\r\n return papers\r\n<\/pre>\n