Archive

Posts Tagged ‘migration’

PHP’s call_user_func_array() is slow

February 3, 2013 Leave a comment

I just solved an issue that had me waking up in the middle of the night for weeks (just pushing it a bit) and I have to thank PHP’s community and in particular a guy named Brad Proctor for writing a very short but very important comment in the PHP manual, and I quote:

This function is relatively slow (as of PHP 5.3.3) and if you are calling a method with a known number of parameters it is much faster to call it this way:

$class->{$method}($param1, $param2);

He mentions it for PHP 5.3.3, but this is true for PHP 5.3.10 and probably for 5.4 (haven’t tested that one yet).

And indeed, it might be a bit difficult to catch this, but we developed a migration system from external databases to Chamilo LMS, and we wanted to make it very flexible, so what we did was build a configuration file (a sort of dictionary) for which content of which tables went into which other tables. In doing this, we also indicated a “converter function” for each type of data.

So the script being kind of generic and all, th easiest way to call the right function when desired was to simply use call_user_func_array().

Everything went well for the first tables. That’s when we came to one big table (6 million rows) and the migration started to take a very long time. It still took about 0.3s per row, but multiply this by 6 million and you get about 55 days of migration script running without a pause. That’s really difficult to deal with. More even when you hav to deliver within 28 days.

Now the problem is that finding it is a little bit tricky, because putting some pofiling mechanism in place will obviously slow down the operation, and because there is a considerable amount of “previous data” that needs to be inserted before we get to the real problem (thus making us wait for almost hours before seeing the right profiling results).

It is even more tricky when seeing that the real time loss is occurring between the end of the function’s execution and the handling over of the control to the calling script/function…

So the process would be something like this:

0.5000s  Calling call_user_func_array(‘function_a’,$params); from general context

0.5001s  Inside function_a()

0.5011s  Executed something relatively complex inside function_a()

0.5012s Ending function_a()

0.8315s Next line of call_user_func_array() call in general context

That’s right, 0.33s lost just to recover the general context (in my case). That means you’ll loose that amount of time each time you call call_user_func_array().

A good reason to avoid it, just by creating a switch() {…} on the function name, for example.

By calling the function directly (within a switch), the results have been speeded up to something like this:

0.5000s  Calling function_a($params); from general context

0.5001s  Inside function_a()

0.5011s  Executed something relatively complex inside function_a()

0.5012s Ending function_a()

0.5014s Next line of call_user_func_array() call in general context

So, in short, if you can avoid it, avoid using call_user_func_array() at all cost!

Update: thanks to @marvil07 for pointing me towards the same kind of discussion on call_user_func_array() in Drupal. Apparently, they came to the same conclusion that it should be avoided, if at all possible.

On migrations to production systems

January 12, 2011 Leave a comment

Migrations on production systems is one of the most complex thing there is in the field of software because it requires all-in-one system administration, database administration, software development and quality management, each at a high level of skill. We generally delegate  a certain level of quality management to the customer (because they’re most qualified in telling us what’s wrong) but all the other aspects have to be dealt with, without the smallest mistake, otherwise you don’t get a running system in the end.

Migrating from Dok€os to Chamilo, quick guide

October 5, 2010 7 comments

To respect the wishes of the trademark holder, Dok€os(TM) is not mentioned by its true trademark in the article below, and will simply be replaced by “Dok”. Any resemblance to an existing LMS would be totally fortuitous.

For all of you worried about how to migrate from Dok to Chamilo and until when you’ll be able to do it, here’s a quick introduction to the idea.

Chamilo 1.8.6.2, the first version of Chamilo ever, is in fact what should have been Dok 1.8.6.2 if it had ever come out. As most of the development team moved away from the project to Chamilo, you can consider Chamilo 1.8.6.2 as the real next version of Dok. and all following versions as the real continuation of the project philosophy. This has a series of practical advantages:

  1. you can easily upgrade any Dok. portal up to 1.8.6.1 to Chamilo (any 1.x version, recommending the latest stable version)
  2. you can easily copy a course from Dok. (up to 1.8.6.1) to Chamilo (any 1.x version)
  3. you can easily copy part of a course from Dok. (up to 1.8.6.1) to Chamilo (any 1.x version)
  4. you get all the good things from the first + the good things from Chamilo in one single application

The general procedure for upgrading your portal to Chamilo is just as you would upgrade from one version of Dok. to the next.

The general procedure to copy course content is to go to the course, go to the Course Maintenance tool, select “Create course backup” and pick the elements you want. Download the course backup, create a course in any Chamilo portal, go to the Course Maintenance tool, select Import course backup and insert the backup file. You might have to increase the authorized upload file limit on your Chamilo portal if your backup is weighting more than 20MB.

Considering Dok 2.0 is near and diverges a little from Chamilo, it will probably not be easily possible to migrate from this version to Chamilo, so you have to make a choice *before* you migrate to Dok.2.0. Of course, if you’ve already made the wrong choice and want to migrate, do not hesitate to contact us to see how we can rescue your data with a professional service guarantee.

That’s it! Don’t hesitate to ask if you got more questions.

Migration to Chamilo

September 23, 2010 Leave a comment

We offer software engineering services of

  • migration from Blackboard(TM) to Chamilo 1.8
  • migration from WebCT(TM) to Chamilo 1.8
  • migration from Angel(TM) LMS to Chamilo 1.8
  • migration from Lecando(TM) to Chamilo 1.8
  • migration from Moodle(TM) to Chamilo 1.8
  • migration from ATutor(TM) to Chamilo 1.8
  • migration from Dokeos(TM) to Chamilo 1.8
  • migration from Sakai(TM) to Chamilo 1.8
  • migration from FileMaker(TM) to Chamilo 1.8

We have previous experience in migrating content from WebCT, Lecando and Blackboard to Dok€os.

Migration generally follows the following cycle:

  • Signing of confidentiality and legal responsibility agreements (you have to take the legal responsibility of accessing your LMS data through another system than the contracted LMS)
  • Analyzing the current LMS implementation
  • Exporting sample data
  • Evaluating the percentage of data that can be transferred without loss or damage (generally around 90%)
  • Writing up the software to migrate
  • Exporting the data
  • Importing into Chamilo (running the migration software)
  • Testing on migration server
  • Approving the export
  • Exporting the data again
  • Importing into Chamilo
  • Last checks
  • Chamilo in production

The whole cycle described above generally lasts around a month and a half, and has to be scheduled 2 months in advance. Depending on the amount of data, the easiness to access the data and its specific structure, prices may vary (non-binding prices, only provided as illustration) from US$5,000 to US$100,000 + corresponding taxes.

Why migrate to Chamilo?

Chamilo is probably the easiest LMS system to use. While very complete, its easiness implies:

  • reduced software training costs (1 to 5 i comparison with other open-source LMS, for example)
  • reduced dropping rate from learners and trainers
  • increased visibility of course content. The platform is practically invisible to the student

Contact

BeezNest is the company most active behind the development of Chamilo, an open-source e-learning LMS with more than 1,000,000 users worldwide. Try Chamilo at http://campus.chamilo.org

If you want to get in touch, just send us an e-mail at info at beeznest dot com

Upgrading campus.dokeos.com to 1.8.5

August 5, 2008 1 comment

Tonight I’m upgrading campus.dokeos.com to dokeos 1.8.5. One of the things that is difficult to put in a human mind is the number of users and courses on this portal and the implication it has on upgrade time. Over the last 5 days, 600 new users have registered on this portal, which made it pass the 200K users for the very first time. The portal still works nicely. It’s now on a single cuad-core server, but it used to work on a Pentium 4 for 125K users. That’s the beauty of Dokeos.

However, upgrades to newer versions are always a difficult task. For example, at the moment we are adjusting the permissions on the “courses” directory. Having 29000 courses, we also have 29000 directories. In the past, we tested the limit of the number of files you can get into one single directory on a Pentium 4, by letting those 125000+ users we had at that time upload their picture. Well, the limit is approximately 117000, so we already had to split the user pictures directory into sub-directories about one year ago when we reached that limit. Obviously we can now reach the million before that happens again!

But, with 29000 courses in one directory, we are starting to think that spreading the courses over subdirectories might not be such a bad idea either…

Well anyway, for 29K directories, updating the permissions (with a simple chmod -R command) takes more than an hour on a very good hard disk. Doing it on the users upload directory also takes something like that.

Then comes the real upgrade. When monitored, the main database updates in less than a minute, but the courses databases (considering they are also 29K, so 29K directories in another directory, each of these directories containing 200 tables) take about 2-3 seconds each to update, which means it takes about 60000 (60K) seconds to update the whole portal, which is about 16 hours!

Yes, that’s what we have to do at each new version to keep the free portal running. While the process is running, we can do something else, yes, but we have to keep an eye on the process in case it would just stop, unfinished, so we can fix it and start again. So we organize turns to watch over the process, Arnaud and I. I’m writing these lines as the process started since 2 and a half hour ago, and it’s just migrated 5000 databases. Only 24000 to go. This is why it is such a painful process for you, users.

This is a little update: the campus migration has been processed in less than 6 hours in the end (the automatic process itself), which is considerably less than what the first databases migration rythm showed, so we could get the campus back in place quite quickly.

%d bloggers like this: