julik live

Bad API design: a whirlwind tour of strong_parameters

Lately I have been doing time upgrading an application from Rails 3 to Rails 4. Obviously, one of the key parts of the operation in that case becomes a move from attribute protection (attr_accessible'/attr_protected') to strong parameters. Doing so, I have actually struggled a great deal. And after a number of hours spent on reflecting on the whole ordeal I came to the conclusion that it was not entirely my fault, and not even a problem with how the previous state of the application was when the migration had to be done.

The problem is strong_parameters itself. See, when you design an API, you usually assume a number of use cases, and then streamline the workflow for those specific use cases to the best of your ability. When situations arise where your API does not perform satisfactorily, you basically have two choices:

  • Press on and state that these situations are not covered by your API
  • Re-engineer the API to accomodate the new requirements

When designing strong_parameters, the Rails gang apparently went for the first approach. Except that since stating that "you are on your own" is not often encouraged as a message to developers-customers, it has been pretty much been swept under the rug. As a result, strong_parameters have been released (and codified as The solution to input validation) without (in my opinion) due thought process.

Since I was finally able to wrestle through it, behind all the frustrations I could actually see why strong_parameters did not work. It did not work because it is a badly designed API. And I would like to take some time to walk through it, in hopes that it can reveal what could have been done better, differently, or maybe even not at all.

So, let's run through it.

It is both a Builder and a Gatekeeper

By far the biggest issue with the API is this. It is both something we Rubyists tend to call a builder, and something we tend to call a gatekeeper - this is more of my personal moniker. Let's explain these two roles:

  • A Builder allows you to construct an arbitrarily-nested expression tree that is going to be used to perform some operation.
  • A Gatekeeper performs checks on values and creates early (and clear) failures when the input does not satisfy a condition.

For example, Arel is a good citizen of the Builders camp. A basic Arel operation is strictly separated into two phases - building the SQL expression out of chainable calls and converting the expression into raw SQL, or into a prepared query. Observe:

where(age: 19).where(gender: 'M').where(paying: true).to_sql

You can immediately see that the buildout part of the expression (the where() calls) and the execution (to_sql) are separated. The to_sql call is the last one in the chain, and we know that it won't get materialized before we have specified all the various conditions that the SQL statement has to contain. We can also let other methods collaborate on the creation of the SQL statement by chaining onto our last call and grabbing the return value of the chain.

XML Builder is another old friend from the Builders camp, probably the oldest of the pack. Here we can see the same pattern

b.payload do
  b.age 18
  b.name 'John Doe'
  b.bio 'Born and raised in Tennessee'
end
b._target # Returns the output

The obtaining of the result (the output of the Builder) is a definitely separate operation from the calls to the Builder proper. Even though calls to the Builder might have side effects when it outputs at-call-time, we know that it is not going to terminate early because the output object it uses is orthogonal to the Builder itself.

Strong parameters violate this convention brutally. The guides specify that you have to do this:

parameters.permit(:operation, :user => [:age, :name])

If you have strict strong parameters enabled - and this is the recommended approach since otherwise parameters you did not specify will simply get silently ditched - if you have even one single key besides :operation and :user at the top level, you will get an exception. If you supply a parameter that is not expected within :user - the same. The raise will occur at the very first call to permit or require. This means that the "Gatekeeper" function of strong_parameters is happening within the Builder function, and you do not really know where the Builder part will be aborted and the Gatekeeper part will take over.

Since we are so dead-set on validating the input outright, at the earliest possible opportunity, this mandates an API where you have to cram all of your permitted parameters into one method call. This produces monstrosities like thos:

params.require(:user).permit(
  :email,
  :password,
  :password_confirm,
  :full_name,
  :remember_me,
  :profile_image_setting,
  {
    paid_subscription_attributes: [
      {company_information_attributes: [:name, :street_and_number, :city, :zipcode, :country_code, :vat_number]},
      :terms_of_service,
      :coupon_code
    ]
  }
)

Those monstrosities are necessary because applications designed with use of nested attributes, and especially applications designed along the Rails Way of pre-rendered forms, will have complicated forms with nested values. And those forms are very hard to change, because they are usually surrounded by a whole mountain of hard-won CSS full of hacks, and have a very complex HTML structure to ensure they layout properly.

In practice, if we were to take the call above and "transform" it so that it becomes digestible, we would like to first specify all the various restrictions on the parameters, and then check for whether the input satisfies our constraint - divorce the "Builder" from the "Gatekeeper". For instance, like this:

user_params = params.require(:user)
user_params.permit(:email, :password, :password_confirm, :full_name, :remember_me, :profile_image_setting)

paid_subscription_params = user_params.within(:paid_subscription_attributes)
paid_subscription_params.permit(:terms_of_service, :coupon_code, :company_information_attributes)

company_params = paid_subscription_attributes.within(:company_information_attributes)
company_params.permit(:name, :street_and_number, :city, :zipcode, :country_code, :vat_number)

user_params.to_permitted_hash # The actual checks will run here

This way, if we do have a complex parameter structure, we can chain the calls to permit various attributes and do not have to cram al of them into one call.

More on this...

Suspects: Веб-стройка

On a small team, not being dicks sometimes trumps efficiency

There is an inherent difficulty to maintaining velocity. We always want things more streamlined, more efficient, more lean - sky is the limit, really, if the technical realm of a product is not micromanaged by the higher echelons of the company management, but is upgraded by grassroots effort. A good team of engineers worth their salt will, as wel know, improve and clean the product of must of technical debt - all a good manager has to do, really, is not to impede that process.

There is an inherent difficulty to maintaining velocity. We always want things more streamlined, more efficient, more lean - sky is the limit, really, if the technical realm of a product is not micromanaged by the higher echelons of the company management, but is upgraded by grassroots effort. A good team of engineers worth their salt will, as wel know, improve and clean the product of must of technical debt - all a good manager has to do, really, is not to impede that process.

There is an inherent concern however, which becomes especially important when the teams are small. Sometimes, velocity and efficiency need to be sacrificed a little if the team wants to preserve equilibrium in human relationships.

More on this...

Suspects: Веб-стройка

The sad state of human arrogance today

Lately an article has been making rounds on HackerNews where eevee laments on his tribulations when installing Discourse.

I've read it cover to cover, and I must admit it did strike a few notes with me, especially because I write Ruby web apps for a good decade now, with Rails and without. Most of the points named in this article are valid to some extent, but I profoundly disagree with the overall tone and the message.

First let's look at the narrative:

  • Ruby is not a primary language eevee is faimilar with (he's primarily a Python/Rust guy from what I could see - because I have been reading his other articles just a few days prior).
  • He does not have a lot of experience deploying a modern Ruby stack in production for web apps
  • Through his experience with Python he probably misses the fact that the Python deployment story is probably just as painful as Ruby's at this point. Moreover, some horrible relics of Ruby packaging (gemsets) still exist at large in the Python world (virtualenv).
  • He picked a project which is explicitly pushing the envelope in it's usage of fancy tools, and thus indeed wants to have it's mother and the kitchen sink for dependencies. This is not because you need fancy tools, but because for a modern web app you do need search, you do need a job queue, you do need push messaging.
  • Exactly because the developers of Discourse (whom I admire greatly) realise that the dependency story in Discourse is effin hard they suggest, loudly and clearly, to deploy it via images or containers. Eevee chose to use neither of these approaches, but facing the consequences of this decision proved to be a world of pain (exactly as predicted).
  • He has a complex Linux configuration, which to me (from my uneducated perspective at least) looks like a Linux build that has been accrued over the years with all sorts of manual tweaks (90% of them probably having to do with X11 and sound of course), and migrated over and over - as a result of which you indeed end up with a 32 bit runtime on top of a 64 bit kernel. This, for tools that assume a default configuration for most of situations, is a recipe for butthurt.
  • He also had to use a PostgreSQL extension, which does not ship as a default part of the package.

Instead of raising my hands in violent agreement with him, or rebutting his post point by point, I would like to look at the actual reasons why this stuff happens.

More on this...

Quitting VFX, one year on

So that might be a surprize to some people reading this – if someone still reads this blog at all. But around Christmas 2013 I have decided that I had enough. Enough of the being pleasant. Enough of the wonderful Aeron chairs and enough of the Flame. I was just sick to my stomach.

Throughout my life I have been one of those blessed few who never had to work at a place they hated. An essential benchmark of the quality of life for me is that I wake up in the morning and feel like going to work. When that feeling is gone, and the fact of going to work becomes dreadful instead -- that is the clearest indicator for me that it is time to move on.

This in fact turns a major page in my life. I have dedicated 9 years of it to the Flame, just like I wanted to. I have seen the demise of the SGI Tezros (I will forever remember what $hinv stands for and what the Vulcan Death Grip is). I have done some great compositing work, from set extensions to beauty to morphing. I have worked with some of the greatest, most wonderful art directors in Amsterdam – it did turn out that most of them have grey hair, a beard, a belly and an impeccable sense of humor - and, first and foremost, artistic vision.

I worked at an amazing company which gave me so much I will be forever grateful. Of course there were ups and downs, but I was incredibly lucky when I stumbled into it by accident in my search for an internship.

I have enjoyed some very interesting projects, mastered Shake and Nuke, became one of the best matchmovers in town. Have beta-tested new versions of Flame and had the pleasure of meeting the wonderful, predominantly-Francophone team of it's makers. I've met a mentor who transformed me from a rookie into a good all-around Flame op. I got pretty much everything I wanted.

For a month I even lived at the office, when the situation with my landlady became unbearable. I worked my year as the night shift Flame operator - so that chevron is earned too.

The logical next step would be London. The problem with that is - I was not prepared to dump my residence status in the EU just for the privilege of working on the next fab rendition of Transformers IX. You see, most people in Western countries are equal, but people with less fortunate passports are somewhat less equal than others. So if I were to move on to the UK it would mean a new alien status, a new residence permit, a new work permit, and all the other pleasures such moves entail.

Also I got tired of client work and all of it's facets that tend to drive you to exhaustion. It was extremely challenging and very rewarding for me - especially me being an overall introvert that I always was - but at a certain point I've started losing the grip. The guy that once was Julik started to become some other person. Some other person that I didn't like seeing in the mirror when I brushed my teeth. Some person that had to develop reactions to the outside stimuli that I did not condone. In short - the beautiful honeypot I was so eager to devour started to consume me.

So after having dedicated 9 years to visual effects and Flame, it was time to turn the page. Since I was developing pretty much all those 10 years (I started about a year prior to this blog being first put online) becoming a Ruby developer seemed par for the course. I love programming, I love Ruby, and I've made some great stuff using it. Tracksperanto has become the de-facto go-to piece of kit for many, and even though I never got code contributions for it – most of the post bunch speaks Python only – I was able to maintain it with excellent longevity and it saved many a matchmove project, both for myself and for many others.

Work/life balance

This has more to do with company culture, but in the past years I've learned that when you process yourself into a skewed work/life balance you have to understand the benefits and the costs. Essentially, when doing all-nighters, participating in sprints, crunch times, enabling your mobile phone at night - think very well what you are giving up, who benefits from it and whether the compensation is adequate. Overtime is not free - you are paying for it with your health, with your family life, with your friendships and love and care and affection. Being on call is not free.

You absolutely should do it when doing the work you love - but consider whether what you are getting for it. When in doubt, re-read Loyalty and Layoffs.

The game here is detecting when the overages and pressures stop being of benefit to you, and only stay beneficial to your employer or client. It is an extremely fine line, and most good managers are good at masking it from view. It took me a full 15 years of working in different roles to get the feeling for when enough is enough, and even at that I had to switch careers to restore the balance.

Don't think that development is foreign to crunchtime (read some jwz if in doubt) but it is highly likely that you will be confronted with it sooner in a VFX career. Remember - how much of it you will take, and what for - is your decision. Nobody - not your employer, not your manager, not your parents - are responsible for the decisions you make about it.

More on this...

Matchmover tip: obtaining the actual field of view for any lens using a survey solve

The best camera is the one that's with you Henri Cartier-Bresson

For a long time we have used a great technique at HecticElectric for computing the lens field of view (FOV) values. When you go to film something, you usually record the millimeter values of the focal length of the lenses you use (or "whereabouts" if using zooms instead of primes). This approach, however, is prone to error - because 3D software thinks in terms of abstract field of view angle, not in terms of the combination of the particular focal length + particular sensor/film size.

So we devised a scheme to reliably compute the field of view from shots of specifically chosen and prepped objects, that we call shoeboxes. This yields very accurate FOV values for any lens (including zooms at specific settings) and can be used with any camera/lens combination, including an iPhone.

More on this...

Building Nuke plugins on the Mac, Christmas 2014 edition

Building Nuke plugins on the Mac, Christmas 2014 edition

As 2014 is folding to a fanfare close, we here at julik live would like to take you, dear reader, to 2009. To recap what happened in 2009:

  • The Icelandic government and banking system collapse
  • Albania and Croatia are admitted to the North Atlantic Treaty Organization (NATO).
  • The Treaty of Lisbon comes into force.
  • Large Hadron Collider gets reactivated.
  • Apple Computer Inc. releases OSX 10.6 Snow Leopard

That last fact is of uttermost importance for us. We are going to time-travel to 2009 to see how life is like in a world where everybody uses systems so old they cannot be obtained legally. To the world of high-end VFX software.

See, I made a Nuke plugin back in the day. Because I really wanted to and I needed it. As faith would have it, I also really wanted to share it because it could be useful for many other VFX artists using SynthEyes -- which I still consider the greatest, fastest and the most affordable 3D tracking package ever.

However that plugin development thing puts me way out of my comfort zone. See, this is C++ we are talking about - but if only. We are also smack in the middle of the DLL hell because, as you might imagine, a plugin for someone else's software is a DLL that you have to build (or a .dylib, or a .so - a shared library for short). Now, 5 years have passed and I somehow managed to maintain a build pipeline for the plugins intact - for quite a long while for example I specifically delayed upgrading from 10.5 just to avoid everything that is related to this dependency hell. Dependency hell in the world of building shared libraries for closed-source hosts is defined as:

  • you have to build against system library versions that have the same ABI and same versions
  • you have to build with exactly the same compiler
  • you have to build against system headers that match the libraries
  • you have to build with non-system libraries that match the system libraries. Fancy some boost perhaps? Or some fancy OpenEXR library? Or some image compression libraries? If you do, we need to talk. Trust me, afte going through this ordeal a few times you will reconsider their use and appreciate the zen of doing without.
  • you (mostly) have to build with non-system libraries that match the versions used by your host application

That leads to an interesting development cycle. At the beginning of that process, if you are lucky and starting early, you will have a machine that carries somewhat-matching dependency versions for all of the above (or the dependency versions are obtainable). At the beginning you will invariably struggle with obtaining and installing all that stuff since it by definition is going to be obsolete already, but that is mostly manageable. I remember that I didn't have to install anything special back when I was building on 10.5 at the beginning of SyLens' lifetime.

More on this...

Suspects: Mac

Why I am still using Jeweler

Rolling Ruby gems is an artistic affair. Fashions change. Sometime ago it was considered fine to edit your gemspec.rb by hand. Somewhat later a trend of using Hoe emerged. A little while later Bundler came into the picture, and it had a command as well to generate gems. Wind a few more years - and the fashion is now to roll with gem new.

The funny thing about it is that multiple people tend to jump on it and migrate their projects from one gem setup/maintenance system to another. It is just like with haute-couture or testing frameworks -- it is cool as long as you are using the latest one in fashion at the moment. Get distracted for just a couple of months - and you are no longer the fashion club, but a retrograde old bureaucrat stuck in his old ways of doing things.

However, not many people focus on what is way more important in the story of making Ruby gems and makine them actually shine - the stability. Let's go to an example. With my modest portfolio I clock over 20 gems in my decade of doing Ruby for fun and profit. Repeatability is way more important for me in this process than any fashions currently being flung around. In my view, for me as a maintainer of a whole salvo of gems, I need to have a few very simple conditions that are met by whatever tool I use for rolling gems:

  • There should be a command I can easily memorise to initialise a blank slate gem, and it should have the right setup in place.
  • All tests/specs for the gem should run with $bundle exec rake, no exceptions.
  • I should be able to do a release, including tagging and pushing it to the gem server, with $bundle exec rake release
  • I should not have to edit any files except of the changelog/version to roll such a release. Simple git history / gitignore is sufficient.
  • The gem set up with that tool of choice has to be runnable by Travis, and for most of my gems I still support Ruby 1.8.7

Having this process helps me by reducing the friction to release gems. When I want to release a library, absolutely the last thing I want to be worried about is how to streamline the workflow of doing those things. Simply because if I do, each new gem that I release or update is going to obtain a release process of it's own - it's just like building plugins for expensive post-production applications all over again. Every new version you roll, for any new version of it's dependencies, becomes a unique snowflake - one has a Gemfile, another one is managed by a manual gemspec.rb, yet another one assembles itself from git... and it goes on and on until you have to actually check what kind of release pipeline du jour has been in effect when you were last twiddling with a certain gem.

The longevity of many of my projects - tracksperanto is no exception with a running history of regular updates over it's 5 year existence - also owes to a stable release pipeline.

I've only went through the changes in the gem release pipeline twice. First switch was from manual gemspec.rb editing and a hodgepodge of Rake tasks to Hoe. The second was from Hoe to Jeweler, because I was unable to make Hoe function on Travis with the vast array of Ruby versions I wanted and because I got fed up with the manual MANIFEST file, where I always forgot to add a file or two.

So far Jeweler, with all of it's possible problems and limitations and an extra dependency, given me the most precious thing in the gem release/maintenance process - and that is that I don't have to think about that process. For any single gem that I maintain, I know that brake followed by brake release is going to do the update I am after - and I can concentrate on the code that my library is offering instead of the fashions of the build pipeline.

That is way more important, and way more precious to me than knowing that I am following the latest trend in a volatile universe of open-source. I am ready to pay the price of being called old-fashioned and having extra 10 lines in my Rakefile, along with a dependency practically none of my users will even download. By corollary, it means that your pull request to my projects proposing to remove Jeweler and go about some more bikeshedding is likely to be rejected. Not because I am a jerk, but because it supports a repeatable process I have developed muscle memory for, and changing that muscle memory is the last item on my priority list.

And I suggest you, dear reader, to do the same - pick a rubygem release/bootstrapping process that works for you, verify it, trust it and stick to it, instead of joining the bikeshedding fest - whatever that process might be. What your actual gem does is way more important than what you are using to roll it.

Suspects: Веб-стройка

On the benefit of laptop stands

When you look at pictures from trendy startup offices you often see laptop riser stands.

One might think that you would do that to make your desk look neater. Or that it is for better posture. Or just to have yet another neat Apple-y looking apparatus in your vicinity for more hipster look. However, there is a benefit to laptop riser stands that some do not quite see upfront.

However, there's more to it than meets the eye.

More on this...

Suspects: Mac

Checking the real HTTP headers with curl

curl, the seminal swiss army knife of HTTP requests, is quite good at many things. Practically everybody knows that you can show headers using the -I flag:

$curl -I http://example.com/req

However, this is plain wrong in a subtle way. See, I sends a HEAD request, which is sometimes used to probe the server for the last modification date and such. However, most of the time you want to check a real GET as opposed to a HEAD. Also, not all web frameworks will automatically implement a HEAD responder for you in addition to a GET responder. It's also downright misleading because with quite a few proxies the headers you are going to be getting from the server will be different for a GET as opposed to a HEAD.

To perform a "real" GET, hit curl with a lowercase i as opposed to upprecase.

$curl -i http://logik-matchbook.org/shader/Colourmatrix.png

However, this will pollute your terminal with horrible binary-encoded strings (which is normal for a PNG after all)... There are ways to do a full GET and only show a header, the easiest being doing this:

$curl -s -D - http://logik-matchbook.org/shader/Colourmatrix.png -o /dev/null

Works a treat but is long and not memorizable. I put it in my .profile as headercheck:

alias headercheck='curl -s -D - $1 -o /dev/null'

So anytime you want to check headers with a real GET request, hit:

$headercheck http://url.that/you-want-to-check

and you are off to the races.

Suspects: Веб-стройка

OWC Data Doubler caveat with 10.9 Mavericks

Shoved a spanky Samsung SSD into my MBP using a Data Doubler from OWC - easy to confuse with the DiskDoubler of yore, by the way.

So, pried open the laptop, put in the SSD into the doubler plate, screwed everything back in. Turns out due to somewhat inferior SATA port on my particular MacBook Pro model the disk would never work properly in the first place. The Mavericks installer just stopped with the "Installation failed" message, leaving the partition for DiskUtility to repair.

The fix is surprisingly head-on. Apparently the SATA port used by the optical drive is not so good after all, you got to have your 6G SSD on your primary SATA, where the old hard drive is. Just swap the old HD with the new SSD, so that you have a SATA 3G device in your doubler plate, on the not-so-good SATA port destined for the optical drive.

So off goes the lid again, the SSD and the old hard drive swap places and the Mavericks install then proceeded as it should, rewarding me with a pristine desktop.

SSD bliss.

Suspects: Mac

Aspirine not included.