technical and social progress toward ECMAScript 6 at facebook – Next, traverse and modify the syntax tree: – Example: ...rest parameters
technical and social progress toward ECMAScript 6 at facebook – Next, traverse and modify the syntax tree: – Example: ...rest parameters
0 2
pivotal-meetup-talk
Slides for my talk about ECMAScript 6 and Regenerator at Pivotal LabsOn Github benjamn / pivotal-meetup-talk
technical and social progress toward ECMAScript 6 at facebook
Ben Newman (Facebook)NodeJS @ Pivotal Labs19 March 2014
{ github,
twitter,
instagram,
facebook
}.com/benjamn
Wikipedia
titlecapitalization.com
We have the opportunity, as technologists, to make certain kinds of problems disappear forever.
“We should have been doing it this way all along!”
Congratulations! You're not done.
Not even if you're right.
Not even if everyone agrees that you're right.
GitHub is strewn with better ways of doing things that never got properly evangelized.
Code just left there without warranty of any kind, either expressed or implied, in the vague hope that other people face needs that are similar to the ones that motivated the author.
Often “better” just means “better tailored to my particular use case.”
In which case there's no mystery why the evangelical message went nowhere.
There has to be a way forward.
And it can't involve throwing away all of your code and starting over from scratch, as tempting as that might be.
As the evangelist, you have a responsibility to yourself, to your ideas, and to the people you want to help, to devise and execute an incremental migration plan.
Other people won't be willing to buy in unless you justify the effort and short-term increase in complexity.
Programming languages are notoriously difficult to “fix forward.”
Why isn't every Python project using Python 3 yet?
It has that 2to3 script.
The maintainers have realistic expectations.
Biggest barrier: changes in unicode semantics.
Attempting to solve a problem once and for all typically requires adapting many different, somewhat-functional past solutions.
How can ECMAScript 6avoid the Python 3 trap?
Crazy idea: ease into the new language by simulating its most useful features in the current version of JavaScript (ECMAScript 5).
The great virtue of this idea is that it makes sense at every stage.
Even if it never ultimately connects with the next version of the language, or takes a long time getting there, we still benefit.
Example: => function syntax
[3, 1, 10, 28].sort((a, b) => a - b)
[3, 1, 10, 28].sort(function(a, b) {
return a - b;
}.bind(this))
First, import some utilities and parse the code:
var recast = require("recast");
var types = recast.types;
var traverse = types.traverse;
var n = types.namedTypes;
var b = types.builders;
var ast = recast.parse(
"[3, 1, 10, 28].sort((a, b) => a - b)"
);
Next, traverse and modify the syntax tree:
traverse(ast, function(node) {
});
Next, traverse and modify the syntax tree:
traverse(ast, function(node) {
if (n.ArrowFunctionExpression.check(node)) {
}
});
Next, traverse and modify the syntax tree:
traverse(ast, function(node) {
if (n.ArrowFunctionExpression.check(node)) {
var body = node.body;
if (node.expression) {
node.expression = false;
body = b.blockStatement([b.returnStatement(body)]);
}
}
});
Next, traverse and modify the syntax tree:
traverse(ast, function(node) {
if (n.ArrowFunctionExpression.check(node)) {
var body = node.body;
if (node.expression) {
node.expression = false;
body = b.blockStatement([b.returnStatement(body)]);
}
var funExp = b.functionExpression(
node.id, node.params, body,
node.generator, node.expression
);
}
});
Next, traverse and modify the syntax tree:
traverse(ast, function(node) {
if (n.ArrowFunctionExpression.check(node)) {
var body = node.body;
if (node.expression) {
node.expression = false;
body = b.blockStatement([b.returnStatement(body)]);
}
var funExp = b.functionExpression(
node.id, node.params, body,
node.generator, node.expression
);
var bindExp = b.callExpression(
b.memberExpression(funExp, b.identifier("bind"), false),
[b.thisExpression()]
);
}
});
Next, traverse and modify the syntax tree:
traverse(ast, function(node) {
if (n.ArrowFunctionExpression.check(node)) {
var body = node.body;
if (node.expression) {
node.expression = false;
body = b.blockStatement([b.returnStatement(body)]);
}
var funExp = b.functionExpression(
node.id, node.params, body,
node.generator, node.expression
);
var bindExp = b.callExpression(
b.memberExpression(funExp, b.identifier("bind"), false),
[b.thisExpression()]
);
this.replace(bindExp);
}
});
Finally, reprint the code:
console.log(recast.print(ast).code);
// Which prints:
[3, 1, 10, 28].sort(function(a, b) {
return a - b;
}.bind(this))
If you already have a build step for static resources, you can be cooking with arrow functions in a matter of minutes!
recast, v.
to give (a metal object) a different form by melting it down and reshaping it. to form, fashion, or arrange again. to remodel or reconstruct (a literary work, document, sentence, etc.). to supply (a theater or opera work) with a new cast.Recast recap:
var recast = require("recast");
var ast = recast.parse(source);
transform(ast); // Anything goes.
console.log(recast.print(ast).code);
“Non-destructive partial source transformation” – Ariya Hidayat
Instead of simply pretty-printing the whole tree, recast.print tries to recyle the original source code wherever possible.
Example: ...rest parameters
Input (ES6):function max(a, ...rest) {
rest.forEach(x => { if (x > a) a = x; });
return a;
}
function max(a) {
var rest = Array.prototype.slice.call(arguments, 1);
rest.forEach(function(x) { if (x > a) a = x; }.bind(this));
return a;
}
Example: ...rest parameters
traverse(ast, function(node) {
});
Example: ...rest parameters
traverse(ast, function(node) {
if (n.Function.check(node) && node.rest) {
}
});
Example: ...rest parameters
traverse(ast, function(node) {
if (n.Function.check(node) && node.rest) {
var sliceCallee = mx(mx(mx("Array", "prototype"), "slice"), "call");
}
});
Example: ...rest parameters
traverse(ast, function(node) {
if (n.Function.check(node) && node.rest) {
var sliceCallee = mx(mx(mx("Array", "prototype"), "slice"), "call");
var sliceArgs = [
b.identifier("arguments"),
b.literal(node.params.length)
];
}
});
Example: ...rest parameters
traverse(ast, function(node) {
if (n.Function.check(node) && node.rest) {
var sliceCallee = mx(mx(mx("Array", "prototype"), "slice"), "call");
var sliceArgs = [
b.identifier("arguments"),
b.literal(node.params.length)
];
var sliceCall = b.callExpression(sliceCallee, sliceArgs);
}
});
Example: ...rest parameters
traverse(ast, function(node) {
if (n.Function.check(node) && node.rest) {
var sliceCallee = mx(mx(mx("Array", "prototype"), "slice"), "call");
var sliceArgs = [
b.identifier("arguments"),
b.literal(node.params.length)
];
var sliceCall = b.callExpression(sliceCallee, sliceArgs);
var restVarDecl = b.variableDeclaration("var", [
b.variableDeclarator(node.rest, sliceCall)
]);
}
});
Example: ...rest parameters
traverse(ast, function(node) {
if (n.Function.check(node) && node.rest) {
var sliceCallee = mx(mx(mx("Array", "prototype"), "slice"), "call");
var sliceArgs = [
b.identifier("arguments"),
b.literal(node.params.length)
];
var sliceCall = b.callExpression(sliceCallee, sliceArgs);
var restVarDecl = b.variableDeclaration("var", [
b.variableDeclarator(node.rest, sliceCall)
]);
node.body.body.unshift(restVarDecl);
}
});
Example: ...rest parameters
traverse(ast, function(node) {
if (n.Function.check(node) && node.rest) {
var sliceCallee = mx(mx(mx("Array", "prototype"), "slice"), "call");
var sliceArgs = [
b.identifier("arguments"),
b.literal(node.params.length)
];
var sliceCall = b.callExpression(sliceCallee, sliceArgs);
var restVarDecl = b.variableDeclaration("var", [
b.variableDeclarator(node.rest, sliceCall)
]);
node.body.body.unshift(restVarDecl);
node.rest = null;
}
});
Example: class syntax
Input (ES6):class Derived extends Base {
constructor(value) {
super(value + 1);
}
getValue() {
return super.getValue() - 1;
}
static getName() {
return "Derived";
}
}
function Derived(value) {
Base.call(this, value + 1);
}
Derived.prototype = Object.create(Base.prototype);
Derived.prototype.constructor = Derived;
Derived.prototype.getValue = function() {
return Base.prototype.getValue.call(this) - 1;
};
Derived.getName = function() {
return "Derived";
};
Transform code much more involved than previous examples; see es6-class-visitors.js for all the gory details.
Good time to mention that all our transforms are open source (click on the link to open in a new tab).What our classes used to look like:
var Class = require('Class');
var copyProperties = require('copyProperties');
function Derived(value) {
this.parent(value + 1);
}
copyProperties(Derived.prototype, {
getValue: function() {
return this.parent.getValue() - 1;
}
});
copyProperties(Derived, {
getName: function() {
return "Derived";
}
});
Class.extend(Derived, Base);
Original plan:
Add a jslint warning that complains when you try to commit new code that assigns .prototype properties Point to a wiki article in the warning message Post about the new syntax in some internal Facebook groups Wait for the diffs to roll in Declare victory (just in time for ECMAScript 12)Wishful thinking!
- jslint tells you that your code might be bad after you've written it
- Almost no one reads wiki articles unless hopelessly stuck
- I didn't want to make reeducating thousands of Facebook engineers my full-time job
- As long as the old style of defining classes dominated the codebase, that's the style engineers would (reasonably!) mimic
What to do?
We have lots of this (ES5):var Class = require('Class');
var copyProperties = require('copyProperties');
function Derived(value) {
this.parent(value + 1);
}
copyProperties(Derived.prototype, {
getValue: function() {
return this.parent.getValue() - 1;
}
});
copyProperties(Derived, {
getName: function() {
return "Derived";
}
});
Class.extend(Derived, Base);
class Derived extends Base {
constructor(value) {
super(value + 1);
}
getValue() {
return super.getValue() - 1;
}
static getName() {
return "Derived";
}
}
What to do?
We want more of this (ES6):class Derived extends Base {
constructor(value) {
super(value + 1);
}
getValue() {
return super.getValue() - 1;
}
static getName() {
return "Derived";
}
}
function Derived(value) {
Base.call(this, value + 1);
}
Derived.prototype = Object.create(Base.prototype);
Derived.prototype.constructor = Derived;
Derived.prototype.getValue = function() {
return Base.prototype.getValue.call(this) - 1;
};
Derived.getName = function() {
return "Derived";
};
We've solved this problem before.
It's just a source tranformation! What's different?
ES5 to ES6 (the “wrong” direction) Only needs to be performed once! Needs to accommodate multiple existing idioms Needs to generate pretty code, as if (re)written by hand The diffs need to be human-reviewable“Non-destructive partial source transformation”
All told:
Lessons
If you make the output human-readable enough, reviewers may not even realize it was machine-generated.
The transform script should be absolutely littered with fail-stop assertions.
Set yourself up to iterate rapidly, accommodating more and more exotic cases as you encounter them.
Meaning: you should be able to change your mind as often as you like, git reset --hard, and rerun the script.
Feel free to fix rare cases by hand, but stack them in separate commits.
So that you can keep running the script on top of those changes without having to remake the changes every time.
Fixing something unusual by hand can be much faster than teaching the transformer how to fix it, relative to the number of occurrences.
Make the transform script idempotent. No excuses!
You will end up re-running the script on already-transformed files, as your script improves, no matter how confident you are.
Think about files with multiple classes, some of which you knew how to transform before you knew how to handle the others.
Use GNU parallel to run the transform script in many processes simultaneously.
find ~/www/html/js/lib | \
grep "\.js$" | \
time parallel ~/www/scripts/bin/classify --update
228.03s user 12.25s system 1229% cpu 19.548 total
Look at that CPU percentage!
Besides speed, the main advantage here is simplicity: the transform script only needs to take one file name argument!
This assumes that individual files can be handled independently.
Humans have right-of-way.
You can always “rebase” by re-running your script—an incredible luxury, if you think about it.
This happens so frequently that I'm tempted to say it the conversion wouldn't be possible without this property.
Identify stakeholders, and convert whole functional units of code at once.
Even if you think you can get away with the transform without any objections, you can't avoid the complaint that the code wasn't broken before.
I didn't do this at first and it bit me.
A big long-term benefit was that I had more and more successful examples to point to over time. Social proof!
This might be the biggest lesson of all.
Set intermediate milestones, and prioritize them aggressively.
Killing off Class.extend in favor of extends/super only really depended on the conversion of files using Class.extend.
Convert a few high-profile internal projects Convert all internal projects Convert all proper CommonJS modules Convert non-modules that use Class.extend to modules so we can kill Class.js and start using extends/super sooner Convert relatively modern products like /messages Some day—maybe!—finish modularizing everything, but no hurry
New code mimics existing code, and the future is longer than the past.
The best way to get rid of old idioms is to get rid of them yourself.Non-ES6 example: JSX
Input (ES6++):<div>
<h3>TODO</h3>
<TodoList items={this.state.items}/>
<form onSubmit={this.handleSubmit}>
<input onChange={this.handleChange} value={this.state.text}/>
<button>{'Add #' + (this.state.items.length + 1)}</button>
</form>
</div>
React.DOM.div(null,
React.DOM.h3(null, "TODO"),
TodoList( {items:this.state.items}),
React.DOM.form( {onSubmit:this.handleSubmit},
React.DOM.input( {onChange:this.handleChange, value:this.state.text}),
React.DOM.button(null, 'Add #' + (this.state.items.length + 1))
)
)
Why fork the language?
- People keep writing HTML-like templating languages, so there must be some value in that.
- JSX affords a useful layer of indirection, allowing us to change our minds about whether to require new keywords or not, how to handle children, what to do about whitespace, React.DOM namespacing, &c.
- Huge benefits from adding similar syntax to PHP at Facebook, and we believe the similarity is worthwhile.
- Arguably much less aggressive than something like CoffeeScript, or even most other templating languages.
- We're content with JSX never becoming part of the ECMAScript standard. That was never the point.
Can we express JSX in “real” ES6?
<div>
<h3>TODO</h3>
<TodoList items={this.state.items}/>
<form onSubmit={this.handleSubmit}>
<input value={this.state.text}/>
<button>{
'Add #' + (this.state.items.length + 1)
}</button>
</form>
</div>