1. <a class="reference external" href="/misc/goto?guid=4958871900305728680">pyquery</a> (with lxml)
</h2>
<div class="section" id="pip-install-pyquery">
<h3>
<tt class="docutils literal">pip install pyquery</tt>
</h3>
<p>
For parsing HTML in Python, <a class="reference external" href="/misc/goto?guid=4958187577038044362">Beautiful
Soup</a> is oft
recommended and it does a great job. It sports a good pythonic API and
it's easy to find introductory guides on the web. All is good in
parsing-land .. until you want to parse more than a dozen documents at a
time and immediately run head-first into performance problems. It's -
simply put - very, very slow.
</p>
What immediately stands out is how fast lxml is. Compared to Beautiful
Soup, the lxml docs are pretty sparse and that's what originally kept me
from adopting this mustang of a parsing library. lxml is pretty clunky
to use. Yeah you can learn and use Xpath or cssselect to select
specific elements out of the tree and it becomes kind of tolerable. But
once you've selected the elements that you actually want to get, you
have to navigate the labyrinth of attributes lxml exposes, some
containing the bits you want to get at, but the vast majority just
returning None. This becomes easier after a couple dozen uses but it
remains unintuitive.
So either slow and easy to use or fast and hard to use, right?
Wrong!
Enter PyQuery
Oh PyQuery you beautiful seductress:
from pyquery import PyQuery
page = PyQuery(some_html)
Easy as pie. It's ever-beloved jQuery
but in Python!
There are some gotchas, like for example that PyQuery, like jQuery,
exposes its internals upon iteration, forcing you to re-wrap:
for paragraph in page('#container > p'):
paragraph = PyQuery(paragraph)
text = paragraph.text()
That's a wart the PyQuery creators ported over from jQuery (where they'd
fix it if it didn't break compatability). Understandable but still
unfortunate for such a great library.
</div>
</div>
2. <a class="reference external" href="/misc/goto?guid=4958871900582367766">dateutil</a>
</h2>
<div class="section" id="pip-install-python-dateutil">
<h3>
<tt class="docutils literal">pip install <span class="pre">python-dateutil</span></tt>
</h3>
<p>
Handling dates is a pain. Thank god <tt class="docutils literal">dateutil</tt> exists. I won't even go
near parsing dates without trying dateutil.parser first:
</p>
>>> s = """Today is 25 of September of 2003, exactly
... at 10:49:41 with timezone -03:00."""
>>> parse(s, fuzzy=True)
datetime.datetime(2003, 9, 25, 10, 49, 41,
tzinfo=tzoffset(None, -10800))</pre>
Another thing that dateutil does for you, that would be a total pain
to do manually, is recurrence:
3. <a class="reference external" href="/misc/goto?guid=4958871900689305452">fuzzywuzzy</a>
</h2>
<div class="section" id="pip-install-fuzzywuzzy">
<h3>
<tt class="docutils literal">pip install fuzzywuzzy</tt>
</h3>
<p>
<tt class="docutils literal">fuzzywuzzy</tt> allows you to do fuzzy comparison on wuzzes strings. This
has a whole host of use cases and is especially nice when you have to
deal with human-generated data.
</p>
Consider the following code that uses the Levenshtein
distance comparing some user input to an array of possible choices.
from Levenshtein import distance
countries = ['Canada', 'Antarctica', 'Togo', ...]
def choose_least_distant(element, choices):
'Return the one element of choices that is most similar to element'
return min(choices, key=lambda s: distance(element, s))
This is all nice and dandy but we can do better. The ocean of 3rd party
libs in Python is so vast, that in most cases we can just import something and be on our way:
4. <a class="reference external" href="/misc/goto?guid=4958871900971515832">watchdog</a>
</h2>
<div class="section" id="pip-install-watchdog">
<h3>
<tt class="docutils literal">pip install watchdog</tt>
</h3>
<p>
<tt class="docutils literal">watchdog</tt> is a Python API and shell utilities to monitor file system
events. This means you can watch some directory and define a
"push-based" system. Watchdog supports all kinds of problems. A solid
piece of engineering that does it much better than the 5 or so libraries
I tried before finding out about it.
</p>
</div>
</div>
5. <a class="reference external" href="/misc/goto?guid=4958871901106242014">sh</a>
</h2>
<div class="section" id="pip-install-sh">
<h3>
<tt class="docutils literal">pip install sh</tt>
</h3>
<p>
<tt class="docutils literal">sh</tt> allows you to call any program as if it were a function:
</p>
from sh import git, ls, wc
checkout master branch
git(checkout="master")
print(the contents of this directory
print(ls("-l"))
get the longest line of this file
longest_line = wc(file, "-L")</pre> </div>
</div>
6. <a class="reference external" href="/misc/goto?guid=4958837507204637626">pattern</a>
</h2>
<div class="section" id="pip-install-pattern">
<h3>
<tt class="docutils literal">pip install pattern</tt>
</h3>
<p>
This behemoth of a library advertises itself quite modestly:
</p>
<blockquote>
Pattern is a web mining module for the Python programming language.
</blockquote>
<p>
... that does <strong>Data Mining</strong>, <strong>Natural Language Processing</strong>, <strong>Machine Learning</strong> and <strong>Network Analysis</strong> all in one. I myself yet
have to play with it but a friend's verdict was very positive.
</p>
</div>
</div>
When I first learned Python os.path was my least favorite part of
the stdlib.
Even something as simple as creating a list of files in a directory
turned out to be grating:
import os
some_dir = '/some_dir'
files = []
for f in os.listdir(some_dir):
files.append(os.path.joinpath(some_dir, f))
That listdir is in os and not os.path is unfortunate and
unexpected and one would really hope for more from such a prominent
module. And then all this manual fiddling for what really should be as
simple as possible.
But with the power of path, handling file paths becomes fun again:
from path import path
some_dir = path('/some_dir')
files = some_dir.files()
Best part of it all? path subclasses Python's str so you can use
it completely guilt-free without constantly being forced to cast it to str and worrying about libraries that check isinstance(s, basestring) (or even worse isinstance(s, str)).