julik liveTracksperanto is fully Ruby 2.0 ready
Just added Ruby 2.0.0 to Tracksperanto's Travis config and the results are in. Our compatibility matrix now looks like this:
So now we also happily run 2.0.0 in addition to 1.8.7 and 1.9.x. Feels nice to stay compatible and current at the same time.
This also means that likely most of the Tracksperanto dependencies are also compatible with 2.0.0.
For modern development Javascript indeed is a shit language
I'm sorry, but the Crockford arguments do not cut it.
Javascript is so bad, on so many levels - it's not even funny. This is why I am so surprised everyone jumped on the node bandwagon with such excitement - yes, Node is faster than Ruby, but it's unfathomable to me that someone in his clear mind would want to rewrite his app in Node without being 100% focused on the evented model.
JS has inherent, deep issues. They are not solvable by making a new ECMA spec. They are not solvable by wrapping a better syntax around it like Coffeescript does. They are not solvable by standardizing on a require implementation or by introducing classes. There is an ECMA language with classes - it's called ActionScript and it's just as shitty as JS itself. These warts just are - and as long as the masses accept it as the status quo it's going to be exactly like the PHP framework landscape to this day: everyone and their mother will be spending man-years trying to create an infrastructure of tools around a shit language that will resist those efforts every single second.
Let me explain why I say that JS is awful. Of course, there are nice things in it - but the problem is that their utility is disputable. Prototypal inheritance, for example, is severly limited in utility - because all it offers you are function overrides. The "everything is a function" approach, while also gimmicky (look ma, I can also call this!), is also not particularly useful - because a function is not an object, not a datastructure that can carry data.
And then the real warts begin. Let's simply enumerate:
JS has callable attributes
This is a shitty design decision which most of the languages have made right at the start. In retrospect, it's difficult to blame the designers because they might have had performance issues - and, to boot, if you are not used to message-passing language the whole idea of "some attributes are callable and some are not" seems absolutely legal.
Hobjects are unusable for stable keys
The mixup between objects and hashes is also a very bad idea, because it defies the premise that objects can have metadata on them - which alllow you to establish a rudimentary type system or at least any kind of introspection.
Fobjects are unusable for type systems since an object does not carry any type information.
This one is a biggie. Even in the Ruby world, where everything is happily quacking like a duck, we often use the Object#class to get information about an object. A fairly standard procedure of styling an HTML element to a certain model object for example:
<div class='<%= model.class %>' id='<%= [model.class, model.id].join %>' >…
is impossible in JS because the only types offered are 'Object', 'function' and primitives. It's awful in all the ways Java is awful, and then some.
Null everywhere
Trying to use a constant with a wrong name by mistake?
MyApp.SYNC // should have been MyApp.SYNC_FETCH
Nothing will happen. Since ojects are hashes, and the language provides zero facilities for constants, our constant with the wrong key will be undefined, and will happily bleed into the callee. This makes stack traces huge.
Callback hell
JS lacks a decent facility for deferreds. It's created for evented execution, which is fundamentally not multithreaded. Your calls are interspersed with event callbacks - when your code is idling, callbacks are executed. However, JS lacks a simple facility for doing this:
var res = await AjaxReq.fetch('/long-request')
// because you are waiting for a result, here the runtime would
// schedule event handling, DOM redraws and whatever else it can
// squeeze in while you await
res.name // this will be only executed once res is available
Of course the JS community is doing exactly what the PHP community has been doing all along - they try to fix a bad language with even worser tooling. How? By using more callbacks and, on a good day, callback chains
when(<ERMAGHERD RIDICULOUSLY LONG CALLBACK>
// 48 lines of code down
).then(<HOLYSHIT WHEN WILL THIS BE OVER>
// 23 lines down
).then(<GIVE ME SOME COFFEE ALREADY>)
In a normal situation this would have been fixed simply by adding a wait primitive to the language which would schedule the events when the result is still being fetched.
The proliferation of callbacks leads to the programming style where everything is async, but as a matter of fact 80 percent of the code you write has to be synchronuous. Simply because 80 percent of the programming is about doing one motherfucking thing with another motherfucking thing and you need both of them to complete the action.
Terrible exception handling
Exception handling in JS is terrible. It exists, but in a rudimentary form - you can see the call stack (that's going to consist of a dozen anonymous functions and one function that is named - that on a good day), and you can see an error message. Since I am not bashing on the DOM, I will only mention three most often encountered errors:
undefined is not a function
cannot call property 'xyz' of undefined
Both of these stem from the fact that fu(ck)bjects in JS have no defined methods - they only have properties. The JS runtime will never have a way to know whether your fubject is supposed to have a method that can be called, or a property of a certain name - it will just assume it being a missing hash key. It just doesn't know any better!
I remember people in the Ruby community complaining about Ruby's backtraces and error messages being not good enough - and Rubinius went to address this. You know where error messages are particularly fucked up? In fucking Javascript. Because the two absolutely basic, crucial exceptions that you want to get and see every single time - NameError (when you are adressing a class-ish or constant-ish something which is not defined) and 'NoMethodError' are simply impossible with the sloppy way the language is built.
And yes, functions are nice, and prototypes are nice and all that - but if you want to build a JS app having any kind of reasonable complexity, you will be bound to write code like this:
var cv = Marionette.CollectionView.extend({
itemView: MyApp.Views.WidgetView;
});
What is the error that you will get if MyApp.Views.WidgetView is not defined yet? undefined is not a function of course! Where will you get it? When the CollectionView will try to instantiate your item view. Not when you define the variable cv, no no! It will explode later, and rest assured - you will be tearing your hair out for a couple of minutes until you see where the error came from.
And why? Simply because everything is a hash and the language is incapable of doing any kind of introspection.
It absolutely perplexes me that people who have used Ruby are moving to Node and calling it a good tool. Node might be great. The language running in it is shit though, and until this gets at least marginally better I will do without Node just fine, thank you.
I can understand that some people wanted to escape the MRI infrastructure by going Node, because - you know - learning Japanese is hard. If you don't speak Japanese, your chances of making a noticeable improvement in MRI approach zero - you will not be treated nicely.
JS is shit, and if we care at least a tiny bit we should do everything in our power to either sunset it, or to move it into the 'assembler for the web' realm where it would be a vehicle for decent languages that can actually get the job done without driving you up the wall. Being nice will not help it, and CoffeeScript is not radical enough. Support your local transpiler initiative today.
Update: nice to know that I am not alone in this.
Checklist for custom form controls in your wep app
Recently I had a privilege of reviewing a web app that is directly relevant to my work field. It is in fact an iteration on the system we are actively using at the company. After having a poke here and there, I was surprised to find that the custom controls epidemic was not over for some people.
All popup menus on the system (all select elements) were implemented as custom HTML controls with custom event handling. I hate this kind of thing not because it doesn't look like a native control - this is in fact secondary. Most of the apps I use daily (Flame, Smoke, Syntheyes, Nuke) do not use the native UI controls at all.
What I hate is a substantial reduction in useful behavior compared to a native control. There is a load of stuff humanity has put into menu implementations for the past 40 years. Every custom select implementation is bound to be reinventing the wheel, in a bad way.
Imagine you are all giggly and want to make a custom select element - a menu. Or your boss does not get UX and lives in the art-directorial LSD fuckfest of the late nineties, and absolutely requires a custom control. Ok then, roll up your sleeves.
Your mission with such a control is to make something that satisfies the art direction requirements (I will not open the can of worms questioning how sane an art director wanting a custom select is), but at the same time is discoverable using the standard operating system idioms. And usable. Let's hit it:
Your custom select implementation must:
- Be tabbable and atomatically obtain a tab index
- Open the menu positioned so that it does not end up off-screen unless it has to
- Resize the menu to it's longest item, outside of the parent element
- Allow the opened menu to be semi-collapsed if it's near the edge of the screen and some items do not fit
- Respond to typing the beginning of the text in the menu item.
- Respond to arrows for going up and down the menu item once it's activated, including cycling
- Respond to "Enter" and "Return" keys that activate the popup
- Respond to "Enter" and "Return" keys to activate the selected item and close the popup
- Not close when the mouse is moved outside of the popup area while the popup is open
- All of the above on all supported browsers
- Activate the right selection menu on touch devices (you have a touch-friendly site right?)
On top of it it has to look like a menu that you can use as a menu (should be discoverable), but since this is an artistic requirement we will assume that your clueless art director already took care of that.
This is just a small list of what the OS does for you, for free. Have fun implementing these and see you in a week or two. Or, if you are a sane person, say "fuck it" and go make features that can make your app really shine. And use a friggin' select.
Running .command files on OSX in other interpreters than sh
We all know that on OSX you can create so-called .command files. In a nutshell, they are renamed shell scripts.
When you make them executable and double-click them, they will run by themselves within Terminal.app.
A little less known fact about them is that you can actually script them in any language of your choosing. For the reasons of distribution it's better to stick to the versions of things available on OS X by default, obviously. You do this by modifying the shebang, just like you would for any other shell script.
For example, a .command file that runs via Ruby would look like this:
#!/usr/bin/ruby
puts "Hello from #{RUBY_VERSION}"
Note that I am not using /urs/bin/env here to get at the right version of Ruby since that turned out to play up on 10.6 somehow. I stick to the system ruby instead.
Gracefully handling NFS mounts on OSX laptops
For the last few years, all the work I do has been equally split between Flame systems running on Linux and a couple of Macbook Pro's running OSX. At Hectic we make use of NFS to make the same servers available to all of our client workstations which are an equal mix between Linux and Windows (Macs do not ouse our NFS facilities much). One of the problems I have encountered has been configuring the NFS mounts on my laptops for graceful timeout.
Now, in the ideal world NFS is designed to handle unmounts gracefully of course. That is, the client is supposed to suspend or fail on IO operations when the requisite mount
is not found. Apple, however, in it's infinite wisdom, designed a slightly different system for it's OSX Server infrastructure. The mounts that you were used to designate in the Directory Services application or, more recently, in Disk Utility (explained in tutorials like this one are
managed by automountd, Apple's daemon controlling the directory mounts. This is the daemon that originally was designed to provide automounted user home directories and other
handy things in the /Network folder on the root drive. In Mountain Lion this feature has been removed, but people try to use automountd nevertheless, explained in a post here.
However, automountd has been designed for limited applications - like computer labs at colleges and universities. It is not capable of detecting offline servers or stale mounts, and even with all the settings tweaked it will never timeout on an NFS mount. In practice, this means that if I have some NFS mounts defined on my laptop and I take the laptop
somewhere where the NFS servers cannot be reached, the following will happen:
- All navigation services dialogs in all applications will beachball when trying to access the stale mounts
- All applications having documents open off of these servers will beachball
- Due to Lion and Mountain Lion's automatic document reopening of last-used documents per application apps that support this feature will beachball again on startup.
This is not pretty. What you actually want is a nice dialog like this:
when the mounts are gone and then to be able to proceed with your business.
And this turned out to be remarkably simple to achieve. The easy solution is - do not use automountd at all, but mount manually. I do it with a Ruby script that I need to run in the morning once my laptop is up and running on the company network. When I come home, the OS falls back to the natural behavior
of simply unmounting the stale shares instead of having automountd hammer on them indefinitely.
Messages versus slots - two OOP paradigms
I've had this discussion with Oleg before, but this still keeps coming up again and again. See, alot of people are pissed at Ruby that you cannot do this
something = lambda do
# execute things
end
something() # won't work
and also cannot do this
meth = some_object.method(:length)
meth(with_argument) # call the meth
However this comes from the fact that Ruby is a message-passing language as opposed to a slot language. What I tend to call a slot language is something that assumes objects are nothing more than glorified hash tables. The keys then become either instance variables or, if they store a function, "callables" (the way Python calls it).
So you might have an object Car that has:
[ Car ] -> [weight, price, drive()]
all on the same level of the namespace. Languages that operate on the "slots" paradigm usually have the following properties:
- It is very easy to rebind a method to another object - just copy the value of the slot
- You can iterate over both ivars and methods in the same flow
- Encapsulation is a decoration since everything in fact still is in the same table
- You can call variables directly since local namespace is also a glorified hashmap with keys for variable names and concent that iscallable.
Due to various reasons I dislike this approach. First of all, I tend to look at objects as actors that receive messages. That is, the number of internal variables stored within an object should not be visible from the outside, at least not in a formal way. Imagine a way to figure out the length of a string in a slot language.
str.len # is it a magic variable?
str.len() # is it a method?
# or do we need a shitty workaround which wil call some kind of __length__?
len(str) # WTF does that do??
There is ambiguity whether the value is callable or not, and this ambiguity cannot be resolved automatically because this expression is not specific in a slot language:
# will it be a function ref?
# or will it be the length itself?
m = str.len
and therefore languages like Python will require explicit calling all the time. This might be entering parens, or doing
m.do_stuff.call()
Most OOP (or semi-OOP) systems known to us in the classic sence (even CLOS as far as I know) are slot systems - IO, Python, Javascript and Self are all slot-based. This gives one substantial advantage: not having to specify that you want to move a block of code as a value explicitly. So you can do, for example in JS
// boom! we transplanted a method
someVar.smartMethod = another.otherMethod;
All of the rest are actually inelegant kludges. First of all, since a method can be used like a value for moving code from place to place, you always need explicit calling. Second, your slots in the object do not distinguish between values and methods so you have to additionally question every value in the slot table whether it is a method or a variable. Also, in slot languages you need to specify that an instance variable is private or not (since normally everything is in one big hashtable anyway).
On the other side we have message-passing languages, like Smalltalk and Ruby. In there anything you retrieve from an object passes through a method call whether you want it or not, because there are two namespaces - one for ivars and the other for methods.
You know that ivars from the outside of the objects are off-limits, and you know that everything exposed to the outside world iscallable, by definition. You also get the following benefits:
- everything only ever goes through getters and setters, so you just don't have to think about them all the time
- you tend to avoid using objects as glorified hashmaps
- you get alot of smart syntax shortcuts
- refactoring a property into a getter/setter pair is a non-issue to the consumer code
Message-passing languages also adhere to the UAP.
When implementing your next programming language, first make it a message-passing system, and if you need performance improvements make internal opaque shortcuts to bypass the getter/setter infrastructure when direct properties are accessed. You will spare alot of people alot of useless guessing and typing.
Sequencing AJAX requests on "send-last basis"
So imagine you have this autocomplete thingo. Or a dynamic search field. Anything that updates something via AJAX when it's value changes.
Now, it's tempting to do this (I am speaking prototype.js here, jQuery would be approximately the same.
function updateResults(field) {
new Ajax.Request('/s', {
method:'get',
parameters: {"term": field.value},
onSuccess: function(transport){
var archiveList = transport.responseText.evalJSON();
displayResults(archiveList); // This updated the DOM
}
});
}
new Form.Element.Observer(searchField, 0.3, function(form, value){
updateResults(form);
});
However, we got a problem here. Imagine that the user types something, and the observer fires when the field contains
some. Now, logically some would find more entries than something and probably the search (the request) will take longer. So if you visualise the call timeline here's what you will see (pardon my ASCII):
| | ^ ^
GET /s?term=so | | Update DOM for "so"
| GET /s?term=some Update DOM for "some" |
| |===========================| |
| |
|=====================================================================|
So the more specific request will complete faster and will update the DOM first. However, after some seconds when the previous request completes the previous request will overwrite the DOM again with less specific results.
What you need to do to prevent that is to discard the requests that have been sent before the last one. To do this, use a recorded sequence number for the request per window, and discard the results that come too late. Of course this will not stop the client from pounding on the server (ultimately you will need clever tricks like caching responses, increasing observe intervals and such) but it will at least ensure that the user is looking at the most specific result.
function updateResults(field) {
// First init our counter
if (!window._reqSeq) window._reqSeq = 0;
// Increment request counter
window._reqSeq +=1;
// Store the sequence number of this request
var thisRequestOrderNo = window._reqSeq;
new Ajax.Request('/s', {
method:'get',
parameters: {"term": field.value},
onCreate: showSpinner,
onSuccess: function(transport){
// ...and if this request is not the last one sent
// discard it's payload.
if(thisRequestOrderNo < window._reqSeq) {
console.debug("Request was stale, skipping");
return;
}
var archiveList = transport.responseText.evalJSON();
updateDocument(archiveList);
},
});
}
This works since now the callgraph will look like this:
| | ^
GET /s?term=so | |
| GET /s?term=some Update DOM for "some"
| |=============(seq:3)=======| X
| onSuccess returns early
|==================================(seq:2)=============================|
Hat tip to Orbling for his SO answer on this.
Migrating a typical roll-your-own comment system to Disqus
So, apparently disqus rocks. When I've applied it to this site I could reach a substantial reduction in code size, since most of the code (especially public-facing code) was dealing with
- comment spam handling
- banning of commenters
- OpenID authentication
- comment formatting
and so on. All of these are a burden and disqus does all of these better than I ever could. However, if you have a homebrew blog system (as I do in case of Inkpot) you will not have a simple plugin provided to flush all of the existing comments to Disqus. However, you can use this gist as a template - all it takes is a little Builder and some lap-dancing to output namespaces properly.
Run this to a file and you will get a Wordpress export that you can import into Disqus. If you store emails properly all of the comments will be also wired into their respective accounts there, automatically.
Matchmover tip: making use of EXIF comments to know your field of view
So you got this batch of reference photos and just into doing some image - based modeling. Awesome! Except that you vaguely remember that there was a zoom lens used on the photo camera. So you'd say "not possible" right?
Nothing further from the truth! Canons actually have these smart lenses that report their current focal length to the camera, and this is something you can use to your advantage. Just like these super-expensive films do with their Cooke/i lenses but you never will since your production couldn't afford them ;-) So, the magic.
Step one, load the pics into some software with decent EXIF support (like Lightroom or iView). Note the lens values and the camera model:
Step two, snoop the internets for sensor sizes:
Step three, enter film back data into your matchmoving software so that it knows how to properly compute FOV from the focal length
Step four punch in the lens millimeters.
Step five - profit!
Tracing a Ruby IO object for reads
So Tracksperanto has a progress bar. Actually both the commandline and the web version have one, which helps to improve the user experience a great deal (some files uploaded on the site were converting for more than a few minutes). However, we need to display this progress bar somehow and to do it we need to quantify the progress made.
For exporting this has been simple.
tracker_weight_in_percent = trackers.length / 50.0
trackers.each_with_index do | t, i |
export_tracker(t)
report_progress( (i + 1) * tracker_weight_in_percent )
end
However, for importing stuff the pattern is not so evident. Most parsers in Tracksperanto work like this:
p = Parser.new
trackers = p.parse(file_handle_or_io)
It's completely opaque for the caller how the Parser can quantify it's work and report on it. Parsers do send status messages that we can use for status line of the progress bar, but no percentages (they also mostly consider the IO unquantifiable and just read it until it eof?s. Also, there are many parsers and introducing quantification into each one of them would be kinda sucky.
So I looked for a different approach. And then an idea came to me: we are reading an IO (every parser does). Parsers mostly progress linearly, that is - they make trackers as they go along the IO (with a few exceptions), so the offset at which the IO is can be a good quantifier of where the import is. What if we feed the parsers an IO handle that can report on itself?
Easy! To do that, we make use of one of the most lovely features of the Ruby standard library that is delegate.rb. As it says,
This library provides three different ways to delegate method calls to an object. The easiest to use is SimpleDelegator. Pass an object to the constructor and all methods supported by the object will be delegated. This object can be changed later.
Advantages: it's not method_missing. That is, the object will properly respond_to? everything it's asked about, give proper kind_of? clues and so on. Second, you don't have to worry about method signatures and forwarding since you can just call super. Ruby delegates give us the capability to quickly "aspectize" our IO objects with some traits wrapped around their standard methods. So off we go.
First of all, I went to the Ruby stdlib doco and found all the reading methods that the IO supports. There are not that many of them by the way. Here's how our skeleton delegate will look like:
class ProgressiveIO < DelegateClass(IO)
def initialize(with_io)
__setobj__(with_io)
end
end
This is all we need to have a good wrapper for IO's, without method_mising tricks of any kind. Then we will write a copy of the ActiveSupport returning idiom
private
def returning(result)
yield; result
end
This one will come handy later on. Now we make a method that will report on the pos of the IO since this is the one that tells us how far we are inside it.
def notify_read
pos # This will change later
end
And now we can override the readers. The general pattern goes like this:
def getc
returning(super) { notify_read }
end
Why super? Well, the default behavior for the class that the DelegateClass() function creates is to forward the calls using the __getobj__ - think of it being method_missing, but with a few smarts inside (look at the delegate.rb source for more info on that). In our case super will do
___getobj__.send(method_name, *any_args, &block_if_given)
Now, there was a question on why we do not use alias_method_chain and things like that to implement these. Well, in our case we do not want to change the way a specific object works. What we are after is the after advice in AOP terms. Also, note that for many objects the calls within them may be intertwined - so if you change the read behavior and some other method in the object calls read or reads are done recursively you might be screwed. What we want is a non-destructive wrapper around the behavior of the object as a whole that does not change anything about the object in particular, that is we want a facade around the object that will also execute our tracing functions. Also our goal is to be able to put this facade around any object that conforms to the calling conventions of an IO, and not to change the method behavior on it.
If the method accepts arguments you need to make a splat-capture
def seek(*a)
returning(super) { notify_read }
end
In both of these cases, super will call the custom class delegate.rb made for us, which, in turn, will call the contained object. However, using just super without parentheses and such frees us from managing the arguments to it - a default super handles it all, except for blocks (this is tricky). Here's how we implement the each
def each(sep_string = $/, &blk)
# Report offset at each call of the iterator
result = super(sep_string) do | line |
yield(line)
notify_read
end
end
alias_method :each_line, :each
Here we inject our notify_read call into the block because we want notifications to come every line, not when the method is called.
We also want to override each_byte
def each_byte(&blk)
# Report offset at each call of the iterator
super { |b| yield(b); notify_read }
end
Same here - no alias_method_chain, just plain-ole super.
Now we need to make the callback that the object will use to report on it's reads. The easiest way to do callbacks in Ruby is by using the to_proc method. It will transform a callable object into something that you can save as a Proc object, thus a variable! Moreover, you can do this with a verbatim block passed to the method
def save_block(&blk)
@stashed_block = blk.to_proc # I stashed your block!
end
so that
save_block do | argument_of_the_block |
# everything you request here will be done by the Proc object
end
So the easy way for us would be to implement a constructor for our delegated IO that accepts a block that reports the progress.
def initialize(with_io, &offset_callback)
__setobj__(with_io)
@callback = offset_callback.to_proc
end
and rewrite our notify_read method to do this:
def notify_read
@callback.call(pos)
end
And presto: everytime your IO is read by something in the system, you will get a call on your block.
file = ProgressiveIO.new(File.open("/tmp/blobz", "r")) do | offset |
puts "Read to offset #{offset}"
end
# Now pass the file to anything that expects a File object
Progress bars galore! Here's how the class looks, ready to be snatched