Overriding Default Werkzeug Exceptions in Flask

Let’s play a game here. What HTTP code is this exception:

"message": "The browser (or proxy) sent a request that this server could not understand."

No no, you don’t look at the code in response! That’s cheating! This is actually a default Werkzeug description for 400 code. No shit. I thought something is bad with my headers or encryption, but I would never guess simple Bad request from this message. You could use a custom exception of course, the problem is, that the very useful abort(400) object (it’s an Aborter in disguise) would stick with the default exception anyway.

Let’s fix it, shall we?

There may be several possible ways of fixing that, but what I’m gonna do is just update abort.mapping. Create a separate module for your custom HTTP exceptions custom_http_exceptions.py and put a couple of overridden exceptions there (don’t forget to import abort we’ll be needing that in a moment):

from flask import abort
from werkzeug.exceptions import HTTPException

class BadRequest(HTTPException):
    code = 400
    description = 'Bad request.'

class NotFound(HTTPException):
    code = 404
    description = 'Resource not found.'

These are perfectly functional, but we still need to add it to the default abort mapping:

    400: BadRequest,
    404: NotFound

Note that I import abort object from flask, not flask-restful, only the former is an Aborter object with mapping and other bells and whistles, the latter is just a function.

Now just import this module with * to your app Flask module (where you declare and run your Flask app) or someplace it would have a similar effect on runtime.

Note that you also should have the following line in your config because of this issue:

ERROR_404_HELP = False

I’m not sure why this awkward and undocumented constant isn’t False by default. I opened an issue on GitHub, but no one seems to care.

Remote Debugging with PyCharm

I’m working in a project now, that requires a certain (server) environment to run, hence it is developed on my local machine and then gets deployed on remote server. I thought I’m gonna say bye bye to my favorite PyCharm feature, namely the debugger, but to my surprise remote debugging has been supported for years now. It took some time to figure out (tutorials online are a bit ambiguous), so here is a short report on my findings.

For the sake of this tutorial let’s assume the following:

  • Remote host: foo_host.io
  • Remote user: foo_usr (/home/foo_usr/)
  • Local user: bar_usr (/home/bar_usr/)
  • Path to the project on the local machine: /home/bar_usr/proj

Here goes the step-by-step how-to:

  1. First we need to set up remote deploy, if you haven’t done so already. Go to Tools → Deployment → Configuration. And set up access to your remote server via SSH. I’d use:
    • Type: SFTP
    • SFTP host: foo_host.io (don’t forget to test the connection before applying)
    • Port: 22 (obvously)
    • Root path: /home/foo_usr
    • User name: foo_usr
    • Auth type: Key pair (OpenSSH or PyTTY)
    • Private key file: /home/bar_usr/.ssh/id_rsa (you’d need to generate the key and ssh-copy-id it to the remote machine, which is outside of the scope of this tutorial).
  2. Go to Mappings tab and add Deployment path on server (pehaps, the name of your project)
  3. Now under Tools → Deployment you have an option to deploy your code to remote server. These first three steps could be replaced with simple Git repository on the side of the server, however I sometimes prefer this way.
  4. Now when you have the deployment set up you can go Tools → Deployment → Upload to ..., note however, that it deploys only the file you have opened or the directory you selected in the project view, so if you need to sync the whole project just select your project root.
  5. I use virtualenv, so at this step I need to ssh into the remote machine and set up virtualenv in your project directory (/home/foo_usr/test/.env), which is outside of the scope of this tutorial. If you’re planning on using the global Pyhton interpreter, just skip this step.
  6. Now let’s go File → Settings → Project ... → Project Interpreter. Using gear button select Add Remote. The following dialog window would let you set up a remote interpreter over SSH (including remote .env), Vagrant or using deployment configuration you have set up previously. For the sake of this tutorial I’m going to put something like that there (using SSH of course):
    • Host: foo_host.io
    • Port: 22 (which is there by default)
    • User name: foo_usr
    • Auth type: Key pair (OpenSSH or PyTTY)
    • Private key file: /home/bar_usr/.ssh/id_rsa
    • Python interpreter path: /home/foo_usr/proj/.env/bin/python
  7. If you set up everything correctly, it should list all the packages installed in your remote environment (if any) and select this interpreter for your project.
  8. Now let’s do the last, but the most important step: configure debugging. Go to Edit Configurations… menu and set things up accordingly. For our hypothetical project I will use the following:
    • Script: proj/run.py (or something along these lines)
    • Python interpreter: just select the remote interpreter you have set up earlier.
    • Working directory: /home/bar_usr/proj/ (note that this is working directory on local machine)
    • Path mapping: create a mapping along the lines of /home/bar_usr/proj = /home/foo_usr/proj (although this seems pretty easy, it may get tricky sometimes, when you forget about mappings and move the projects around, be careful).
That’s it. Now we should have a more or less working configuration that you could use both for debugging and running your project. Don’t forget to update/redeploy your project before running as the versions may get async and PyCharm would get all whiny about missing files.

Humble Collection of Python Sphinx Gotchas: Part I

Gotcha 1: Getting Started with Sphinx Theming

Update 02.10.2014: It seems absolutely obvious to me, but apparently it is not that obvious to some: you can’t do anything about your Sphinx theme without some basic CSS and HTML skills. You can only change some of the theme parameters, if the developer was kind enough to add some, but it is well covered in the official documentation.

Customizing Sphinx visuals was somewhat upsetting to me at first, since the process is not straightforward and documentation is scarce. It could be a little bit user-friendlier. However, as soon as you get a grip on the basics, it gets pretty smooth and simple. You may have already tried copying over the default theme only to discover, that there is nothing particularly useful in there for an inquisitive scholar. Only one of the standard Sphinx themes is in fact complete, the rest of them simply inherit its properties to add some minor alterations. This theme is called Basic and it’s a minimal sandbox template, the only theme that could be helpful for getting to the very bottom of Sphinx customization. Later you’d be able to inherit it and create a template, consisting only of alterations, but for a start it’s OK to copy the Basic in its entirety.

Hopefully, you’ve already created a folder for your Sphinx project and initiated it by issuing:

$ sphinx-quickstart

Or you may have an already existing Sphinx project, you want to theme — it’s up to you. In your project folder create the _themes directory and then rename and copy the theme folder there. Basic theme, perhaps, should be located in site_packages folder of your active Python install. On my Mac it’s: /Library/Python/2.7/site-packages/Sphinx-1.2b1-py2.7.egg/sphinx/themes/basic/, but Python on Mac OS X is just weird. If you’re using Linux (/usr/share/sphinx/themes/basic/ on Debian) or Windows, you should look it up yourself.

Next step would be to change conf.py of your project, accordingly. First, we should make sure, that the following line is uncommented and correct:

html_theme_path = ['_themes']

Don’t forget to check the value of the html_theme parameter above:

html_theme = 'renamed_basic'
Basic Theme
Basic theme without customization looks like vanilla HTML with no stylesheet whatsoever.

As of now you may start making alterations to the theme. You will find that HTML files in there aren’t really HTML files, but templates (Sphinx uses Jinja template engine) with some staff automatically inserted on build. You can combine these automatic tags with basic HTML tags and as soon as you figure out how it works, you could move some of the interactive tags around, or get rid of some of them altogether. Don’t forget to check whether you’re not breaking anything though. Most visual aspects of Sphinx theme are modified through the main CSS file, which is located in the static folder. For Basic theme it would be basic.css_t. Notice, that t in the extension brings to our attention the fact, that this is a template. Other than that it could be viewed and edited as a simple CSS file. If you’re interested in the values, provided by Sphinx templates and how you cam make use of them, consult the official documentation. If you want the main CSS file to have some other name, you can change that in theme.conf, there are also some other settings, that could be of interest.

Gotcha 2: Spoiler

If you play with Sphinx templates for some time, you may want to start adding something more interactive. We could easily implement these kinds of things trough JavaScript or jQuerry (which is kindly included in the package).

Let’s go trough a little example project, to recognize, what could be achieved with combined power of Sphinx and JavaScript. You may notice, that as for common elements in HTML themes there is usually a way to address them in JavaScript, it’s not that easy to introduce new classes and wrap some parts of the text into them. There is no custom div tag in Sphinx, unless you do that as .. raw directive which is not really a native way, and breaks lots of things. What I needed to do was to wrap some section of text into a div and make it collapsible on button pressed. You could achieve that by creating your own admonition and assigning it to certain HTML class.

For example:

.. admonition:: Request
		:class: splr
 		Request example and parameters.

You can then easily reference this admonition by its class in your JavaScript. For example you could put this jQuerry script to your page.html template:

    <script type="text/javascript">
    $('.splr').css("background-color", '#F0F0F0');
    $('.splr').css('box-shadow', '0px 0px 1px 1px #000000');
    $('.splr > .last').hide();
    $('.splr > .expanded + .last').show('normal');
    $('.splr > p').click(function() {

If implemented correctly, this script should turn the splr admonition into a collapsible drawer. I’ve been actively using this, when documenting HTTP APIs, since it’s very helpful to hide JSON responses by default. Note, that you could also use this method for special CSS effects. Imagine, if aside from usual note, warning and tip you could have yellow, blue and purple boxes for whatever reason you can think of? Well, it couldn’t be any simpler. Admonitions are good default containers for some parts of your text, that should differ in design or function from the rest of the page.

Gotcha 3: Interactive TOC in Sidebar

Not the worst case, but still a valid comparison of TOC trees in Default theme and our JavaScript enhanced Basic. Note, that on the right the subcategories may be collapsed and opened dynamically.
Not the worst case, but still a valid comparison of TOC trees in Default theme and our JavaScript enhanced Basic.

You may have noticed, that Sphinx often adds TOC to sidebar automatically, if it’s not explicitly placed in the page itself. While this is certainly a very useful feature, sometimes things get out of control. I didn’t use the worst case in the picture on the right, but it could get up to innumerable 1st level sections, each of them could have a number of subsections and so on. Sometimes it is a clear sign, that the page should be reorganized into multiple standalone pages, united by a category, but it’s not always possible or needed. It’s perfectly alright to have a long and deep TOC in some cases and Default Sphinx theme is terrible in that regard (as of 2014 it is fixed and children categories are shown only for the currently selected one).

You could use an updated version of the algorithm from the previous example to collapse some parts of the TOC by default. Note, how this script works with ul and li tags of the TOC tree list. Some stuff is applied recursively on the highest level of the list, some — on the subsequent levels. Especially noticeable with different styles applied to different levels of the list, so that you could tell whether it is 1st, 2nd or even 3rd level title. Here is the full script:

    <script type="text/javascript">
    $('.sphinxsidebar').attr('position', 'fixed');
    $('.sphinxsidebar').css('position', 'fixed');
    $("h3:contains('Table Of Contents')").css('border-bottom', '1px');
    $("h3:contains('Table Of Contents')").text('TOC:');

    $('.sphinxsidebarwrapper li').css('background-color', '#B8B8B8');
    $('.sphinxsidebarwrapper li').css('box-shadow', '0px 0px 1px 1px #000000');
    $('.sphinxsidebarwrapper li').css('color', 'white');
    $('.sphinxsidebarwrapper li').css('border-color', '#000000');

    $('.sphinxsidebarwrapper li > ul > li').css('background', '#D0D0D0');
    $('.sphinxsidebarwrapper li > ul > li').css('box-shadow', '0px 0px 1px 1px #000000');
    $('.sphinxsidebarwrapper li > ul > li > ul > li').css('background', '#F0F0F0');

    $('.sphinxsidebarwrapper li > ul').hide();
    $('.sphinxsidebarwrapper li > .expanded + ul').show('normal');
    $('.sphinxsidebarwrapper li > a').click(function() {
    //hide everything
    //toggle next ul

Perhaps, it won’t look very well, but it is a very simplified version to illustrate the concept. If you get the idea, you may alter or add .css methods to achieve more plausible visuals. You could also work on lower title levels, but you will have to figure it all out for yourself. In this version algorithm works best only with three headings levels from <h1> to <h3>, but it could definitely handle more.

That’s it. I’ve compiled these examples as a full-featured theme project on GitHub. I’m going to polish it to some extent and perhaps, implement more interactive stuff over time. Feel free to contribute to this little project. Nope, never happened. Well, it did happened actually, but it was no good, sorry. If you come upon any issue with this little gotchas of mine, let me know. Sure, I’ve tested everything myself, but you never know. Also, you’re free to use any of these examples as a building block in your work with no attribution, since they’re rather generic and simple.

On dType Suspension

I can safely confess, that a couple years ago I didn’t know a single thing about programming. I was aware of some fairly abstract concepts and had a basic understanding of how it all works, but it definitely wasn’t enough. My English teacher had a saying about the active vocabulary: “You may learn all the words from the dictionary by heart, but unless you use them regularly and naturally, you don’t really know them”. My situation with programming was somewhat similar to knowing lots of trivia, but having no grasp on the practical side of things. I was determined on fixing it as soon as possible. I’ve tried reading a book or two, but it never really got me going. Well, it explained a couple things here and there, but it was like learning things by heart — tedious and irrelevant (on an absolutely unrelated note: Learn Python the Hard Way is great). At that point one of my techie friends suggested me throw the book away and learn by immersion: make an objective, stumble upon problems, see docs and StackOverflow for possible solutions. That was the moment I started looking for the first project, fairly simple, yet more challenging than a mindless Hello World routine.

Once, I was typing down a big portion of plain text on my old slow Android phone, using another memory hog Office Suite, with all those controls, sets of buttons on all sides of the screen and I wished there was something like Focus Writer for Ubuntu: basic, but fairly powerful in terms of achieving that special zen state. There weren’t many such projects in Android market back then (yeah, kids, it was called that in days of old) and this is how the idea of dType has stricken me. The concept was fairly simple: a minimalistic tool, that would let you jot down some text and then pass it to some other application (Evernote, Dropbox, Email, etc.) for saving or processing. It was simple enough to get grip at basics, yet quite challenging for a person, who haven’t seen Java code (or any code) before.

It was the moment, when I started coding. Well, let’s say it was more about googling intensively for just about anything. It was hard. Most of the time I didn’t know, what I was doing and asked fairly inept questions on StackOverflow. I still do, but now at least I can tell, what most parts of my code are for. First, the immersion is like trying to play piano blindfolded — my code probably stunk a big time, but at the end of the day it worked and it was encouraging. Interest in Android development helped me to get a job as a technical writer in a bunch of Android-related projects, especially OpenCV for Android. Since, I’ve been working mainly on C++ API references, I’ve started to delve into OOP concepts. I’ve been thoroughly explained, what is a class, a method and how they relate to each other, interfaces, abstract classes and the rest of this stuff. I’m extremely grateful for my mentors at there. Later, working on some other project, I had a chance to look closer at working Java code and see these concepts applied to Java. I immediately started to refactor dType code once again in attempt to implement thorough OOP design and shake off all the redundancy. My code became a little bit more laconic and neat. Not that it couldn’t get any better, but it was still a huge leap forward for me.

As long as I remember, dType was constantly improving. It was first a bunch of undocumented spaghetti code, which was somewhat straightened out at version 0.16 — it became the earliest version I bother to keep in the repository, since everything before that was a complete disaster. Perhaps, it’s still rather bad, but I’ve managed to shorten it almost twice, provide descriptive JavaDoc (for the sake of it, I know no one will probably bother to read) and fix a lot of issues while at it. I do feel a little attached to this code emotionally, since it is my first coding experience, that has grown into a little indie project of mine. Over the course of two years it has provided me with innumerable challenges and priceless practical experience, but it’s finally time for me to move on. I’ve taken great interest in Python lately, and started a couple of projects in it. Coming back to Java code became more and more daunting to me. I was also advised by several programmers, that I’d better concentrate on getting really proficient with one language for now. My growing frustration with Java verbosity ensured, that I would end up with Python as my language of choice.

Still, it was a hard decision for me to drop dType completely. People do use it and clone it on GitHub (yeah, it had a couple of official clones before the project has been moved back and forward, nevermind the actual numbers on GitHub). This project, though certainly quite niche and facile, does work for some. I decided, that this suspension is going to be more of a role shift for me: from active developer of this application to its maintainer. It will stay as an open repository at GitHub for you to clone and alter, it will stay published on Google Play. You can continue to use it in version 0.71. If people provide some relevant pull requests, I would be happy to merge them and even publish the resulting build as the new version of the app. It’s just that me myself don’t have the time or inclination for introducing new features anymore. It is now exactly the way I envisioned it, when I was starting. My big learning project has reached its objective. It’s finished. My priorities have changed, but if you do care, I would be glad to see your contributions. I’m not naive to think that it could become a huge open source project, mind you, but I do hope, that the app could continue living on its own, while I’m gone.

Update 06.10.2014: No one decided to contribute to this project yet, and probably never will as more than a year has passed since this post. Perhaps, this is for the best, as I’ve seen people complaining about a couple of nasty visual bugs on some devices. Me myself wasn’t able to work with it on Galaxy SIII as the screen gets black from time to time. So if you really want to revive the project, I can only wish you good luck with that. Seriously, if you want to try, give me your contacts, so that I could talk you out of this.

Why Static Site Generators aren’t Good for My Blog

I would be speaking about my own experiences and you may have noticed my in the title, which is there to remind you about the subjective nature of this article. I don’t dare to speak about your blog or any other blog out there, but my own. I have no intention to convert you to my side, yet I would be very glad to see some of the like-minded people out there. I know they exist. My own research on the topic has unveiled, that although static site generators have this really substantial and zealous following, there are sober voices in the crowd, appealing to common sense. This is exactly why Kevin Dangoor went “From WordPress to Octopress and Back” or why Michael Rooney is “Migrating from Octopress to WordPress” — in the exact opposite direction from the majority of switchers.

Masses Migrating from WordPress to Octopress
Illustrating the distinctive trend among majority of switchers.

It doesn’t boil down solely to WordPress vs Octopress debate, as the issue in hands is much broader and may be represented as dynamic site engines vs static site generators. If you’re not aware of the difference between the two, here are the basics: with dynamic site your content is generated dynamically by an application running on the server side, static sites, on contrary, are pre-generated or written outright in HTML. Basically, dynamic sites are a web-applications that could change their behaviour instantly, depending on input and other factors, while static sites: well, they’re just HTML files passively sitting there, waiting for you to open and read them. Sure, with introduction of jQuerry, Java Script and HTML 5 to the mix, the difference gets a little less distinctive, but let’s stop at this level.

So, static vs dynamic. It could be virtually anything: Tumblr vs Hyde, Blogger vs Pelican, Movable Type vs Jekyll, etc. Major differences between the two models are more or less the same. It means that we should be really comparing the models themselves, not their instances.

So, what are the lucrative advantages of the static generation model? What makes people switch so quickly and without looking back? I came up with 3 most important reasons:

  1. Almost endless customizability.
  2. It’s mostly plain markup text files, that you can store in a Git repository.
  3. Increased loading speed and security (well, no features – no loopholes, obviously).

All three are valid points and at some moment I went down the static path myself with all three in mind. First I went with Jekyll, then switched to Octopress. At some point I even tried to make Sphinx-generated site (sic!) work as a personal web-page, but lack of blog awareness and increasing complexity made me abandon this idea. When they say that running this kind of site is the easiest things to do, this is complete nonsense. In terms of comfortable workflow, I only can say a couple good things about pure Jekyll paired with GitHub Pages, but the result was so raw and required so much customization, that straightforward workflow was hardly an advantage. It is positioned as a toy for true geeks, tinkerers, but I don’t see how anyone could really benefit from this kind of tinkering (yeah, even geeks and tinkerers). I work in IT and we are here mainly to solve problems, not create tons of complimentary issues. Instead of reinventing the wheel, you could as well invest your time into something, that really needs to be done, perhaps — writing good stuff for your website.

I’m not the only one who noticed it, but for several months I’ve been experimenting with static generators, I’ve hardly written half a post. It was a common problem, as I googled it. People dug so deep into the endless customization and switching routine that eventually they’ve stopped writing. I may sound conservative to some, but I still think that blogging is mainly about writing. Sure, to some extent, this problem is applicable to dynamic platforms too (ping pong between WordPress and Blogger, anyone?), but with static generators it grows to catastrophic proportions. They are the ultimate time drain for nerds and wannabes. Sure, if your time worth basically nothing and constant tweaking your blog is the best part about having it, be my guest. To me it looks pretty much like that little joke from Fry&Laurie:

Oh, yes, my boyfriend’s a real DIY enthusiast, DIY mad.
He’s decorated the whole room and he’s put up all these bookshelves, and now he’s writing all these books to put on them.

Another seeming advantage of static site generators is their reliance on simple text, that could be utilized in distributed version control systems like Git. Most switchers assume, that they already use plain text and Git for code, why not use it for a blog? The problem is, that it also complicates things instead of easing them. Each generator has its own super-easy-workflow™ with different special folders, commands and scripts — quickly it becomes a mess. In this regard Git is basically one more noose on your neck. I’d focus on one especially nasty implementation of Git in such workflow — Octopress. You should fork the original Octopress repo, the _deploy folder will be used for deploy to the pages repository and sources should be committed to the special source branch. Now imagine what would happen if a somewhat major update gets pushed to the upstream of Octopress and your copy has been significantly modified over time. As someone has put it: “Octopress is great, until it breaks”. If you have some experience with Git and seen the Octopress workflow, you may imagine the hell it could possibly be. Actually, Octopress is itself a heavily modified version of Jenkins. No offence, but it all seems like Rupe Goldberg machine to me.

If you google it, you will find lots of people, performing full-featured benchmark tests of WordPress vs Octopress with a complete disregard for the principal differences between the two. People start speaking of security and speed benefits of static sites, forgetting about all the advantages of dynamic sites, that come at the price of increased complexity and bloat (yes, I do think WordPress is a tad bloated). Imagine the situation: you need to jot down a post draft on a public or someone else’s PC? Will you be cloning the repo, installing ruby, Octopress — setting up all the environment to write a short post? What about mobile support? Should you attempt to clone Octopress to your mobile phone? What about preserving drafts in the cloud without publishing them, but having access to them virtually from anywhere and anytime? Can you really put a price on that? People start using Evernote or similar service for drafts and such, but does it really worth introducing another tool to your workflow? Does mobility and availability worth another couple seconds of load time? My own choice is comfort and efficiency. I want my blog to be complimentary to my technical endeavors, not the other way around.

I’ve started thinking that less is actually more a long time ago and it may apply to blogging as to almost any other area of our life. You may notice, that I don’t even use the standalone WordPress install, but the pretty limited hosted WordPress site. I prefer to pay engineers at WordPress $13 for domain mapping and settle for less choice in themes, plugins and other options to focus on writing. We’re all too lost in the world of different platform and workflow options these days. Google it and you will see hundreds of rants about why platform A is deliberately better, than platform B, why static sites are better, than dynamic. You almost never hear that they help you in writing, no. It’s all about SEO, storage space, customization, load speed and other insignificant stuff, not directly related to blogging. We’re too obsessed with form and seem to forget about goddamn content. But, as I said before, it is all entirely subjective. You may still go down the static route, customize the ass out of your blog. You may even spend several years on writing your own static or dynamic blog engine from scratch, that will sure be absolutely unique and different from anything ever done before. Yet, I’m writing this post in a beautiful distraction-free WYSIWYG editor and my draft will be preserved online, when I press the Save button and no rake deploy is needed ever again.