How to upgrade Drupal 6 to Drupal 7

Another detailed article, this time on how to upgrade from D6 to D7: http://www.ostraining.com/blog/drupal/migrate-drupal-6-to-drupal-7/

Setting up multilingual menus in Drupal 7

Reference to another detailed article: http://renaudjoubert.com/en/article/setting-multilingual-menus-d7

Nginx + CDN + GoogleBot or how to avoid many useless Googlebot hits

If you’re like me and you’ve developed a CDN distribution for your website’s content (while waiting for SPDY to be widely adopted and available in mainstream distributions), you might have noted that the Googlebot is frequently scanning your CDNs, and this might have made your website a bit overloaded.

After all, the goal of the CDNs are (several but in my case only) to elegantly distribute contents across subdomains so your browser will load the page resources faster (otherwise it gets blocked by the HTTP limit or any higher limit set by your browser of simultaneous content download).

Hell, in my case, this is the number of page scans per day originating from the Googlebot on only one of my CDN-enabled sites (I think there are like 5 different subdomains). And these are only the IPs that requested the site the most:


3398: 66.249.73.186
1380: 66.249.73.27
1328: 66.249.73.15
1279: 66.249.73.214
1277: 66.249.73.179
1109: 66.249.73.181
1109: 66.249.73.48
1015: 66.249.73.38
822: 66.249.73.112
738: 66.249.73.182


As you can see, it sums up to about 13,000 requests in just 24h. On the main site (the www. prefixed one), I still get 10,000 requests per day from the Googlebot.

So if you want to avoid that, fixing it in Apache is out of the scope here, but you could easily do it with a RewriteCond line.
Doing it in Nginx should be relatively easy if you have different virtual host files for your main site and the CDN (which is recommended as they generally have different caching behaviour, etc). Find the top “location” block in your Nginx configuration. In my case, it looks like this:

        location / {
                index  index.php index.html index.htm;
                try_files $uri $uri/ @rewrite;
        }

Change it to the following (chang yoursite.com by the name of your site):

        location / {
                index  index.php index.html index.htm;
                # Avoid Googlebot in here
                if ($http_user_agent ~ Googlebot) {
                    return 301 http://www.yoursite.com.pe$request_uri;
                }
                try_files $uri $uri/ @rewrite;
        }

Reload your Nginx configuration and… done.

To test it, use the User Agent Switcher extension for Firefox. Beware that your browser generally uses DNS caching, so if you have already loaded the page, you will probably have to restart your browser (or maybe use a new browser instance with firefox –no-remote and install the extension in that one *before* loading the page).

Once the extension is installed, choose one of the Googlebot user agents in Tools -> Default User Agent -> Spider – Search, then load your cdn page: you should get redirected to the www page straight away.

BeezNest fixes new Chamilo security flaws in 48h

Secunia gave the Chamilo Security team full details of three security flaws detected in version 1.9.4 on Monday the 4th of March 2013. These flaws were detected by Fernando Muñoz, a regular contact for Chamilo in terms of security. We are very grateful for his and Secunia’s work.

Within less than 48 hours, the Chamilo Security team, part of BeezNest‘s development team, has reviewed, patched, published and communicated a fix to its community through the Chamilo wiki and Chamilo Security team’s Twitter account, doing its best effort to inform everyone in time.
In fact, a patch was available within 12h, but it took us some time to go through the right communication channels and make the patch accessible to the greatest number.

Our fast turn around time on this security issue is a reflection of the priority and focus we place on security. Security is more than a side item for us, it’s part of our core principles.

At Chamilo, we embrace security research and show our appreciation to all involved. Thank you for helping us make Chamilo better!

BeezNest offers hosting, development and maintenance services for any Chamilo LMS portal. Feel free to contact us for a quote if you need help. Chamilo is free software and you are absolutely not forced to use our services. We can help you out if you need it, that’s all!

The biggest challenges for start-ups in Peru

Today I went to a conference organized by the Lima Valley coordinators team, hosted by PriceWaterHouseCooper Lima, and hosted by ex-entrepreneur-turned-speaker Bowei Gai.

Perú is a nice country: vast, with lots of different geographies, some of the deepest canyons in the world, adventure sports with not too much regulations, jungle, beach, mountains climbing around 6000 meters above see level, subtropical climate and extreme high-altitude climate (and gigantic lakes). There are around 30M people (and apparently more Twitter accounts than that, although that’s probably due to over-use of social marketing) and around 10M just in Lima, the capital. Nothing in comparison to Mexico DF, but still…

Recently, Peru has been considered as a very active hub of entrepreneurship, probably due, between others, to the existence of a very active team of entrepreneur-lovers called Lima Valley.

A little bit about me: I’m co-founder of Lima Valley (I was part of the team invited by Daniel Falcón to organize a few meeting and try to find a way to improve entrepreneurship in Peru back in 2008, but then stopped participating actively in mid-2009, so I couldn’t take much credit, really).

I have a start-up in Belgium and a start-up in Peru. I’ve been around in Perú since 2007 and I’m the leader of a famous free software (>open source) e-learning platform project called Chamilo. I have spent the last 10 years of my life “trying out” different business models around free software.

Although my financial success is nothing close to a miracle, I have been able to manage (obviously not alone) a project that now has 3.5M users around the world and generates (for our company alone) a reasonnable income per year (and almost no benefits given to the high amount we invest in research and development), with a steady growth year over year. That’s considering it’s a free platform, so not only our sales are more difficult to pitch, but we also make it easier for other companies to compete against us, because we think you can make a *honest* living out of services to improve education and that our product improves education and should be available for anybody who wants to put the extra effort of downloading and installing it. This is not our only source of revenue, but we are quite the experts in e-learning platforms in Latin America.

I’ve also been mentoring candidates for entrepreneurship at the Start-up Academy for several years, a project that was launched by Arturo Canez and Álvaro Zarate at the end of 2010, if I remember the date well. This has given me a wider and more profound view of what entrepreneurship is, what works, what doesn’t and which external elements can influence it.

One of the things I didn’t expect from today was Bowei asking, during the round table, for the public’s (about 80 people) thoughts about the biggest advantages and the biggest challenges in launching a start-up in Peru. I would have brought a long list of bullet points, but I’m the kind of very extensive speaker, not really ready for a one-minute speaking time style of his question to the public, so I decided to shut it up and write about it later on this blog.

One thing that kind of surprised me, though, is Bowei telling, as a conclusion address, that we didn’t need to wait for the government or any big organization to act in order to start our own business… I would have thought this was basic stuff for entrepreneurs! As if the only thing entrepreneurs were waiting for was a big funding enveloppe to start developing their ideas. Come on! I wouldn’t have assumed that (they didn’t know) in front of an entrepreneur crowd, but again, it’s better said than kept quiet.

The problem with funding in Peru is that it is practically impossible to get some as an IT start-up, unless you have very good friends in social sector A (B doesn’t even count). Money isn’t the source of the problem. Not having the right contacts is. You can be looking around and talking to people for years (in start-up time your project was born and dead twice) before you find people that combine the “believe in you” factor and the “I got that money to invest” factor. Having success as a start-up is not just about having a great idea *and* having the skills to build it (and that’s already difficult to find in one single person or great team). You also need massive amounts of good karma from a massive amount of people, and that’s where funding projects are useful (they already gather people ready for that).

Yes, you can definitely be successfull with a great idea and great skills, but this requires the 3rd element: good relationships. And that’s where most geeks fail, because it’s hard to be good technically and have a great social life at the same time, and that’s where you need the almost-impossible-match-without-money-being-involved: getting a social-life guy together with a geek in a relationship based on respect and mutual understanding. And this is where these funding projects get (very) useful.

Anyway, here is a list of challenges any start-up will meet in Peru. I’ll start with the ones quoted during the conference, either by panelists or by the public (with a little extra explanation from me):

• Mindset for entrepreneurship: people do not have the entrepreneurship model embedded in their mindset, so it’s difficult to have other people support you (if they don’t understand what you do). It’s even more difficult to find people to ally with you. People having a good idea tend to think their work is done and the rest must be handle by the tech guys.
• Funding by government: there is very little effort from the government to finance start-ups
• Trust: it is difficult to develop trust-based relationships in Peru
• Infrastructure: the current one, both in terms of roads and internet, make a huge number of initiatives impossible to implement. Even in Lima (only 30% of the national population), some areas do not have practical access to internet, nor proper roads network, for that matters
• Skills: it is very difficult to find people able to produce technological products
• Legal bluriness: a lot of legal processes linked to entrepreneurship are not defined or very complicated, which makes formalizing a real challenge. The chamber of commerce of Lima is currently hitting that nail very strongly for other businesses types as well, and I hope they will improve this situation for start-ups, as a side-effect. The government is also starting to involve the population in reducing the number of useless procedures (ver Trámite de más)

Here is my own list of challenges

• Mindset of investment: You can analyze it however you want, but the proportion of successfull worlwide start-up that have started inside the USA is ridiculously high in comparation to those having started outside the USA. This is true for Europe as well. It is incredibly more difficult (but US people probably don’t get how much) to promote an innovative solution if you don’t start in the US market. Why? There are at least 3 reasons, which I’ve tried to test over the last 5 years: (1) racism (or the fact that you are not ready to invest in some people from the third world unless this ensures very big money); (2) propensity or easiness to buy, digitally (US people are much more ready to use digital payment systems than the rest of the world, which makes all kinds of micro-payment services work much better from the start, as you remove a huge problem for adoption); (3) language (it is true that Latin America has a larger audience than the USA, but face it, people in China have a much higher chance of knowing English than knowing Spanish, even more so for early-adopters profiles, making for a much larger user base from the very start – when the start-up needs it the most); (4) history (people around the world are much more likely to put their trust in a start-up from the USA because they’ve already heard so much about others reaching success).
• Trust: It’s already been mentioned before, but trust is an incredible ally if you have it. A huge problem is Peru it that it is plagued by untrustable people who take each and every possibility to put you in a corner and kill you. Obviously, this is not a majority (far from that) but the existence of these people is enough to plague the whole society like cancer and make anyone doubt any alliance, even with their best friends. The lack of public voice against them encourages this behaviour. The “vivo” stereotype is pretty much accepted in society an there is almost nothing done about it. I believe this certainly achieves killing about 30-40% of any entrepreneurship project in Peru.
• Keywords: this goes a little around the mindset point mentioned before. For some reason, people all around do not tie the term “start-up” with innovation yet. They seem to believe that a start-up is a little shop you open at the corner and that it is only meant to serve local people. Once you mention you are doing “innovation” (keyword), then they open their eyes and start listening to you (sometimes)
• Copy-pasting-updating and the lack of legal boundaries: although innovation is present in Peru, and many people call it inventivity, a lot of cases are still copy-pasting what someone else did and quickly building up something similar. This has the very damaging effect of considering any innovation (at least in the computer-based-solutions) as a “cheap” process, which then makes it impossible to properly invest in doing innovation the right way and really “invent” stuff. This is further encouraged by the lack of enforcement of Peruvian laws: infringing a software patent is not even considered when building most web applications. If you find it on the web, then it’s free (as in freedom) to use. This encourages copy and reduces the benefits of innovation. I’m not saying software patents are a good thing (I, for one, am completely against them), but there needs to exist a mechanism by which the innovators are glorified (free software licenses encourage building up reputation and respect of the innovators), and an enforcement of such mechanism. If everybody uses Windows for free and does not question its security, philosophy or usability, then there is no incentive for anyone else to build a better product.
• Payment systems: it is still very difficult to reach a large number of people as non-human-assisted payment methods are not accessible to all. There are efforts being made in this direction, but it seems like there is no real political intention to massively distribute it still (the plan has been around for 3 years already)
• Internal racism: a cultural problem almost invisible to whom doesn’t analyse Peru’s society is internal racism, in particular towards people living in or coming from the Andes (“Serranos” as they are called). It is pretty difficult to get all these people together woking on the same team as they tend not to be able to trust each other. I won’t go into the details, but this is both a potential problem inside the start-up and outside it, as you try to develop the business based on people close to you
• Government understanding: even though funding is not a requirement for start-ups, the problem is that the government representatives in power do not even begin to understand what innovation really is. Of course, there are a few exceptions (maybe 10 people in a relevant position) but that’s not nearly enough to support a culture of innovation. It goes even further than that: some gov staff actually believe that the government money is their money and that they should save it to avoid spending it on the wrong items.
• Government funding: there are, at the moment, programs for funding of innovation, but if you compare the list of proposed projects an the list of projects which have been assigned funding, you find yourself in front of a huge amount of agronomical projects and maybe 1% of technological-services-based projects. It’s just that the judges for these grants do not even understand these projects, I’m sure, so they fall back to stuff they know. If you read about the numbers, you then realize that, any given year, the budget spent on innovation projects is about only 40% of the total budget they had on their hands. The rest goes to the national reserve (I hope).
• Government politics about start-ups: so in reaction to what neighbouring countries do (Chile is a recent example), the Peruvian government decides to also promote a fund for start-ups, and they launch a program to do that… OK, so the program has been advertised in November 2012 by the minister of production, and you listen now (March 2013) that the “plan” is still not finished, and that they “hope” it will be ready in 9 months. Come on! Get serious. If the ministry of production understood start-ups, they would also understand that dozens of good projects will die in the meantime, and that the only right way to proceed is by trial and error (agile), improving the plan every year, but starting now. Ideally, they would also include real entrepreneurs in the team building this project, and not just government staff (which is rather the opposite mindset, by definition). The problem with these programs being politicalized is that, if badly managed, they confuse entrepreneurs and make them loose their time doing proposals oriented at the interests of the project for the government, finally to be discarded because the judges don’t even understand it. Even when you send a project that makes sense 3 months before the project deadline, it gets rejected on the basis of one block not being clear enough, but it gets rejected when there is no possibility for you to fix this. If the government is going to be left with more than 60% of the budget on their hands anyway, what’s the real point of asking for people in areas they don’t understand to participate?
• Corruption: although it is always difficult to prove, corruption is present everywhere. Corruption doesn’t work (at all) with the start-up business models, based on transparency and rewarding efficiency. I put it last, however, because these last year corruption seems not to have touched that many start-ups.

Some of the advantages of Peru mentioned by panelists or the public:

• Inventivity of the people (I really don’t agree with that being an advantage of Peru, as it could be said of any developing country: by essence, they have to do more with less, but that has nothing to do with being more inventive, in my opinion, it’s just a matter of opportunity)
• Biodiversity (true, but that’s also doingthe case for Brazil, Madagascar, Thailand, and a long list of others with anything like “jungle” in their geography)
• Cultural diversity (true, but that can also be a disadvantage)
• Food (true, but it still lacks a large-scale improvement in distribution, conservation and hygiene before it can be massified and treated as an advantage to the outside world)

My opinion on Peru’s advantages for start-ups (work in progress):

• Virgin market: many things can be “tried out” here without much impact in case of failure. Don’t abuse it: most farmaceutical companies still sell drugs here that have been prohibited from sale in the USA or Europe. That’s not the kind of “trying out” that I feel comfortable doing…
• Low fares: the cost of production for many things is very low (more so out of Lima), so if you  are “starting up” with something that involves local goods, production costs will be low. However, don’t rely too much on exportations, as the corresponding processes are really out of proportion and mostly accessible only to large businesses. Watch out for the cost of highly-skilled staff, though, as these can be in complete disproportion to the rest of the workforce and the cost of living (demand is very high, offer very low)
• Delivery costs: Mobility is very low cost (if it doesn’t involve air transport) inside Peru, so hiring a team to deliver products on a motorbike in Lima, for example, can be extremely inexpensive (in comparison to Europe, where delivery was considered, 10 years ago, not to be possible under 30€/hour)
• One language to in the commerce bind them: Latin America, to a few exception (the most notable being Brazil) is a huge continent where everyone speaks Spanish. That’s a huge advantage in terms of commerce over Europe, although not so much over USA+Canada or China. It also has tons of local languages (there are more than 35 languages still alive in Peru alone).

In conclusion, as for any market, if you want to have success as a start-up, you have to know the market and find the best advantage points for your case. Peru is certainly not a paradise for start-ups, and current entrepreneurs here can certainly be hailed for there ability to go against the current. Levelling up the skills of staff in general would really improve the situation exponentially, though. No… really!

Clean Redmine cache

To refresh Redmine cache, do the following from the Redmine directory (/usr/local/share/redmine, maybe ?):

rake tmp:cache:clear
rake tmp:sessions:clear

http://www.redmine.org/projects/redmine/wiki/RedmineUpgrade#Step-5-Clean-up

Database schema for Chamilo 1.9.4

A lot of people have been asking for a database schema of Chamilo 1.9. Here it is. The editable format can be found in the documents for Chamilo LMS: http://support.chamilo.org/documents/57

Chamilo LMS 1.9.* database schema

The image itself weights 3.2MB and has a few colours to try and focus on the important parts. All the tables in semi-circle at the bottom are course-specific data tables

PHP’s call_user_func_array() is slow

I just solved an issue that had me waking up in the middle of the night for weeks (just pushing it a bit) and I have to thank PHP’s community and in particular a guy named Brad Proctor for writing a very short but very important comment in the PHP manual, and I quote:

This function is relatively slow (as of PHP 5.3.3) and if you are calling a method with a known number of parameters it is much faster to call it this way:

$class->{$method}($param1, $param2);

He mentions it for PHP 5.3.3, but this is true for PHP 5.3.10 and probably for 5.4 (haven’t tested that one yet).

And indeed, it might be a bit difficult to catch this, but we developed a migration system from external databases to Chamilo LMS, and we wanted to make it very flexible, so what we did was build a configuration file (a sort of dictionary) for which content of which tables went into which other tables. In doing this, we also indicated a “converter function” for each type of data.

So the script being kind of generic and all, th easiest way to call the right function when desired was to simply use call_user_func_array().

Everything went well for the first tables. That’s when we came to one big table (6 million rows) and the migration started to take a very long time. It still took about 0.3s per row, but multiply this by 6 million and you get about 55 days of migration script running without a pause. That’s really difficult to deal with. More even when you hav to deliver within 28 days.

Now the problem is that finding it is a little bit tricky, because putting some pofiling mechanism in place will obviously slow down the operation, and because there is a considerable amount of “previous data” that needs to be inserted before we get to the real problem (thus making us wait for almost hours before seeing the right profiling results).

It is even more tricky when seeing that the real time loss is occurring between the end of the function’s execution and the handling over of the control to the calling script/function…

So the process would be something like this:

0.5000s  Calling call_user_func_array(‘function_a’,$params); from general context

0.5001s  Inside function_a()

0.5011s  Executed something relatively complex inside function_a()

0.5012s Ending function_a()

0.8315s Next line of call_user_func_array() call in general context

That’s right, 0.33s lost just to recover the general context (in my case). That means you’ll loose that amount of time each time you call call_user_func_array().

A good reason to avoid it, just by creating a switch() {…} on the function name, for example.

By calling the function directly (within a switch), the results have been speeded up to something like this:

0.5000s  Calling function_a($params); from general context

0.5001s  Inside function_a()

0.5011s  Executed something relatively complex inside function_a()

0.5012s Ending function_a()

0.5014s Next line of call_user_func_array() call in general context

So, in short, if you can avoid it, avoid using call_user_func_array() at all cost!

Update: thanks to @marvil07 for pointing me towards the same kind of discussion on call_user_func_array() in Drupal. Apparently, they came to the same conclusion that it should be avoided, if at all possible.

Varnish stops file download after 60 seconds

The default installation of Varnish 3 on Debian-based system (including Ubuntu) will set a default timeout for sending files to users (i.e. for users files download) to 60 seconds.

To change this limit, it is necessary to update your default Varnish configuration, in /etc/default/varnish.

For example, if you have something like this:

DAEMON_OPTS="-a :80 \
             -T localhost:6082 \
             -f /etc/varnish/default.vcl \
             -S /etc/varnish/secret \
             -s malloc,256m"

make sure you add the send_timeout param, like this (see the -p line):

DAEMON_OPTS=”-a :80 \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-S /etc/varnish/secret \
-p send_timeout=900 \
-s malloc,256m”

This will obviously increase the maximum download time to 900 seconds (15 minutes), which will make it much more practical and less stressfull to download large files.

The logic behind this very short limit is that, if you’re optimizing your site with Varnish, you shouldn’t make it busy with handling one long-term connection with a large file like this (meanwhile, thousands of other connections could have been treated) and your large files should reside on a CDN or something like this.

Chamilo 1.9.4 released, clear the way!

Today we release Chamilo 1.9.4, with a number of notable improvements in speed, usability and features, but you know… that’s nothing compared to what we’re preparing for version 1.10. We’ve got a pocket full of kryptonite ready to blow your eyes out this year, but let’s focus on 1.9.4 for now. You can download it from here: http://www.chamilo.org/en/download. The free campus will be upgraded, probably this week-end, to have everybody enjoying the improvements.

For this great release, I’d like to personally thank our AMAZING team. We’ve all been hard at work, and they deserve some praise. In particular, thanks to Julio, Yoselyn, Hubert, Jérémie, José, Laura, Noa and, of course, Michela for their efforts in all directions, helping Chamilo spread like crazy and helping build this brilliant piece of software that you’re probably downloading already. Many more people deserve thanks. They’ve all (I hope I didn’t forget anybody) figuring in the Chamilo 1.9.4 credits. My sincere apologies to the Chamilo translators for having failed once again to distinguish their contributions and thank them properly for that. You make Chamilo a better tool for your communities and this is invaluable!

You can find the full changelog of Chamilo 1.9.4, as always, on http://stable.chamilo.org/documentation/changelog.html, but if you prefer a short version, have it here:

• Teachers can now generate an export in PDF all the certificates of their students at once (and print them as one big file)
• Improvement to the attendances tool, allowing you to mark “late” students and warning you about overlaps
• Stylesheets can now be uploaded *and downloaded* directly from the admin page, so happy editing!
• Exercises now generate score reports by questions categories (nice!)
• You can now group questions by the same media resource (a large text, a video, etc)
• LP image is now shown as a starter icon on the course homepage if you want to
• You can now create student classes through CSV import
• General reports now show IP address to identify where this student was connecting from (you know… his exam seemed so good, for once…!?)
• Considerably improved learning path usability
• Single Sign On class has improved security with blocking per IP (or IP range) the Single Sign On origin
• Improved responsive design: better positionning of most blocks
• HTML5 support improvements
• WCAG improvements (alt/title tags and other stuff)
• SEO improvements: H1 and H2 tags reorganized
• BigBlueButton plugin improvements and fixes
• Fixed issues with IE9 (with and without compatibility mode) in exercises
• Fixed problem with accentuated chars in image zones question types
• Fixed issues with files upload in HTTPS
• Fixed XLS questions import
• Fixed courses copy bugs
• Translation to Tagalog

But 1.9.4 is just one step further from 1.9.0, with 1.10 looming in th background for the big leap. 1.9.6 should be ready in about two months from now (mid March?), with just around 50 issues to review and fix (which will give us plenty of time to look at other important Chamilo stuff), then 1.10 should come out somewhere around May (that’s a very wide guess). I don’t want to sell it too early, so you’ll just have to believe that what is coming is much more than what 1.9 was in comparison to 1.8. With 1.10, we should get a large foot into the large corporate environment scale, with Chamilo being used progressively by hundreds of thousands of users on a few portals, with more features, better efficiency (still) and a more reliable behaviour.

To match our ambitions, we’ve worked even harder so that Chamilo could now be automatically tested, documented, analyzed (and soon packaged) by our brand new Jenkins server at http://testing.chamilo.org. With this one, we’ll be able to optimize our review cycle, spending more time on important stuff, and less time fighting against undetected regressions. Including the new Symfony framework bits that we’ve been adding recently, Chamilo LMS (just that branch) has now about 1 million lines of code, which makes it a huge development project, so it was time for us to get armed with the proper weapons. Two new developers (maybe more) will soon be joining the team as well. Our previous projects have won us a long series of good relationships and we are on our way to grow, both as an association and as a group of companies. Talking about that, the association subscribed almost 50 new personal members last year, so if you’re interested, don’t hesitate to check out the guide on how to become a member.

Our crowdfunding portal is on its way too, so you’ll be able to show your support for a specific feature. More about this soon…

And finally, Chamilo certification exams will soon be updated to Chamilo 1.9.4. You can buy an exam attempt at https://shop.beeznest.com. The process is completely automated. Video courses will be available for sale soon too.

Want to have your own Chamilo 1.9.4 portal? Why not try Chamihost for free for 15 days?

Oh, by the way, today’s Chamilo’s 3rd birthday!

The amazing Chamilo project is warming up for 2013. Are you?