Meteor app is unstable with cobwebs
Situation:
I want to run some application on my ubuntu vps server for crawl testing. My application is using meteor-router
out of the "atmosphere" with a package manager mrt
. On my local mac os x 10.8 with phantomjs installed with brew
everything goes well. I am getting a good snapshot of my page by adding
http://sample.com/?_escaped_fragment_=
for the url.
Problem:
Let's try to do the same on my ubuntu vps server. 2 ways:
1) do not copy the application to the server and run it with the command mrt run
: it is unstable. Sometimes that's good. But sometimes my dynamic content is empty. Like my db is empty.
2) copy the unconnected app to the server and mrt bundle fname.tgz
that, then unzip the .tgz and run it main.js
with node. Thus, spiderable does not work completely. i am getting empty instead of dynamic data every time i try.
My ideas:
My ubuntu machine has a lot less memory and cpu resources than my local machine. So it takes longer to create dynamic content, but phantom thinks the page is complete and takes a snapshot before rendering the meteor.
Any suggestions?
source to share
I think I solved this problem. This is really a problem in the spiderable.js file. this module starts phantomjs in REPL state and gives it stdin code like this:
var url = '" + url + "';
var page = require('webpage').create();
page.open(url);
setInterval(function() {
var ready = page.evaluate(function () {
if (typeof Meteor !== 'undefined'
&& typeof(Meteor.status) !== 'undefined'
&& Meteor.status().connected) {
Meteor.flush();
return Meteor._LivedataConnection._allSubscriptionsReady();
}
return false;
});
if (ready) {
var out = page.content;
out = out.replace(/<script[^>]+>(.|\\n|\\r)*?<\\/script\\s*>/ig, '');
out = out.replace('<meta name=\"fragment\" content=\"!\">', '');
console.log(out);
phantom.exit();
}
}, 100);
The problem is when all the Meteor conditions are passed, it thinks the page.content is 100% updated. But this is not the case. The solution I found and tested is to wrap the block if
in setTimeout
(500ms worked just fine for me):
if (ready) {
setTimeout(function () {
var out = page.content;
out = out.replace(/<script[^>]+>(.|\\n|\\r)*?<\\/script\\s*>/ig, '');
out = out.replace('<meta name=\"fragment\" content=\"!\">', '');
console.log(out);
phantom.exit();
}, 500);
}
source to share
I believe the correct way to do this is by passing in a callback page.open
like ( see docs ):
page.open(url, function (status) {
...
};
Also, if you want to rely on the timeout for a snapshot, I would decrease the timeout and wrap it in a loop to speed it up and make it more reliable:
page.open(url, function (status) {
if(status !== 'success') {
phantom.exit();
return;
}
function isReady() {
return page.evaluate(function () {
if('undefined' === typeof Meteor
|| 'undefined' === typeof(Meteor.status)
|| !Meteor.status().connected)
return false;
Meteor.flush();
return Meteor._LivedataConnection._allSubscriptionsReady();
}
}
function trySnapshot() {
if(!isReady()) {
setTimeout(trySnapshot, 100);
return;
}
console.log(page.content
.replace(/<script[^>]+>(.|\\n|\\r)*?<\\/script\\s*>/ig, '')
.replace('<meta name=\"fragment\" content=\"!\">', '')
);
phantom.exit();
}
trySnapshot();
};
I also think my last snippet will often execute with no timeout at all because the page.open
callback is called at the appropriate time
source to share