Archive for June, 2014

Using Chamilo juju charm to setup a dev environment on Digital Ocean

June 23, 2014 8 comments

If you’re in a hurry/on speed, know this:

  • this procedure is slightly more difficult (so longer) than installing the charm on Amazon
  • you can skip directly to “Installing Juju”
  • if you already have juju installed, you can skip to the last 2 lines of the “Installing juju” section
  • if you already have juju-docean installed and configured, you can skip directly to “Provisioning VMs”
  • otherwise, just continue reading, it’s worth a few minutes…

This tutorial regroups a lot of advanced notions, so if you want to know more about one of the following elements, please follow these links:

Before anything else, please note that the following is highly experimental. There are still a series of issues that should be worked out in order to make this process failproof.

Basic setup

Before we start using commands and stuff, you’ll have to note the following:

  • We are using a Chamilo Charm developed by José Antonio Rey (kudos to him) as a voluntary contribution to the project
  • Charms are configurations to install applications (and stuff) inside the Juju framework
  • The Juju framework is developed by the Ubuntu team, so we’re using an Ubuntu (14.04) desktop (or in this case laptop) to launch all the following
  • Digital Ocean is one cloud hosting provider, which is particularly cheap and good for development purposes. The “default” environment for Juju is Amazon, so we’ll have a few additional steps because of this choice. The Digital Ocean plugin to Juju is developed by geekmush on Github, and as far as I know he is not related to either Ubuntu nor Digital Ocean, so he is also worth praising for his contribution
  • Chamilo requires a web server and a database server. In this Charm, it is assumed that we want both of these on separate virtual machines, so you will need two of them (unless you change the parameters a little)
  • Juju is written in Go but relies on several Python libraries. As such, you’ll have to have python installed on your system and maybe Juju will shout because it is missing a few dependencies. Notably, I installed python3-yaml to avoid a few warnings (it is required for the following, although the installer for Juju says it’s optional)

Installing Juju

On a default Ubuntu desktop installation, you’ll have to install Juju first. Because we are going to use Juju connected to Digital Ocean, we need a recent version of Juju, so let’s add it the unconventional way (with the ppa), launching the following on the command line:

sudo add-apt-repository ppa:juju/devel
sudo apt-get update && apt-get install juju
juju version

For some reason, in my case, this created my home directory’s .juju/ folder with root permissions, which then prevented me to reconfigure my environment (requirement for the Digital Ocean Juju plugin), so I changed permissions (my user is “ywarnier”, so change that to your user):

sudo chown -R ywarnier:ywarnier .juju

Then we need to install the juju-docean plugin:

sudo apt-get install python3-yaml
sudo pip install -U juju-docean

Setting up Digital Ocean access

Now we need to configure our Digital Ocean (D.O.) API so the system will be able to call D.O. in our place and create instances (and stuff).

You first need to grab your API key, client ID and SSH key ID from the Digital Ocean interface. You can do that from the Digital Ocean API page. Obviously, you need a Digital Ocean account to do this and a few bucks of credit (although you can get $10 free credit from several places). If your API key says “Hidden”, that’s because you must have it stored somewhere already (for other services?). If you don’t, you’ll have to re-generate one. Your SSH key ID is the name you gave to the SSH key you use from your computer to connect to your new instances. If you don’t have it, that’s probably because you haven’t configured any. Please do in the “SSH Keys” menu item on the left side of your D.O. panel.

 export DO_CLIENT_ID=aseriesof21alphanumericalcharacters
 export DO_SSH_KEY="user@computer"
 export DO_API_KEY=aseriesof32characters

Setting up the Digital Ocean Juju environment

Now we need a bit of manual config to be able to use Digital Ocean (last bit, promised). Edit the ~/.juju/environments.yaml file and paste the following:

 type: manual
 bootstrap-host: null
 bootstrap-user: root

Just a note: the “type: manual” line implies it is a bit more complicated than on amazon later on, and we will have to launch a few more commands to provision new machines *before* we deploy Chamilo.

Generating the Juju environment

Now we’re going to create our Juju controller. The Juju controller can be an independent Virtual Machine (VM), or it can be the same as the one on which you will deploy Chamilo. It all depends on your budget and your requirements.

juju docean bootstrap --constraints="mem=1g, region=nyc1"
  2014/06/22 11:50.24:INFO Launching bootstrap host
  2014/06/22 11:51.29:INFO Bootstrapping environmen

Note that we took a decision to use a 1GB (RAM) VM here (mem=1g), in a datacenter in New York (region=nyc1). For the record, I tried creating them in nyc2, which is also a valid D.O. datacenter, but it failed miserably (sometimes not creating the VM, sometimes creating it without IP, sometimes creating it fully, but in the end never returning with a proper success response for my environment to be created), so sticking to nyc1 is probably a reasonable time-saver.

Provisioning VMs

To be able to deploy Chamilo, we’ll use two VMs: one for the web server and one for the database

juju docean add-machine -n 2 --constraints="mem=1g, region=nyc1"
2014/06/22 12:44.59:INFO Launching 2 instances
2014/06/22 12:46.42:INFO Registered id:1908893 name:digitalocean-8d14c9bc671555ff872d8d6731f84d68 ip: as juju machine
2014/06/22 12:49.08:INFO Registered id:1908894 name:digitalocean-a9ba29cfe55549f58e5f7e365199c5ed ip: as juju machine

Now, the “-n 2” above allows you to create these 2 instances, but you could also launch 2 different instances of different properties, doing it one by one. In our case, I suggest you use version Trusty of Ubuntu for the MySQL machine, to avoid a little bug in the Precise version of the charm:

juju docean add-machine --constraints="mem=2g, region=nyc1"
juju docean add-machine --series=trusty --constraints="mem=1g, region=nyc1"

The important thing here being that you can later identify the machine itself by a simple ID, using juju status:

juju status
environment: digitalocean
  agent-state: started
  agent-version: 1.19.3
  instance-id: 'manual:'
  series: precise
  hardware: arch=amd64 cpu-cores=1 mem=994M
  state-server-member-status: has-vote
  agent-state: started
  agent-version: 1.19.3
  instance-id: manual:
  series: precise
  hardware: arch=amd64 cpu-cores=1 mem=994M
  agent-state: started
  agent-version: 1.19.3
  instance-id: manual:
  series: trusty
  hardware: arch=amd64 cpu-cores=1 mem=994M

If you made a mistake at some point or just wanna try things out, you can destroy these instances with

juju docean terminate-machine 1

where “1” is the ID of the machine, as shown above before each of them.

Deploying Chamilo

Now we’ve got our machines, we just need to deploy the Chamilo Charm and the MySQL Charm (you need MySQL to run Chamilo):

juju deploy cs:~jose/chamilo --to 1
juju deploy mysql --to 2

Please note that the “–to n” option is to specify on which machine you want to deploy the selected service.

Now, we need to configure Chamilo a little. We’re going to give it a domain name (you’ll have to redirect this domain name to the IP of the first machine – the one with the Chamilo service – in order to use it when ready) and a password for the “admin” user (the user created by default):

juju set chamilo pass=blabla

Now we still need to tell Juju to link the Chamilo service with the MySQL service:

juju add-relation chamilo mysql

And finally, apply all the above and expose the chamilo service to the public:

juju expose chamilo

If something goes wrong with a service, you can always remove it with:

juju destroy-service chamilo

You can replace “chamilo” by the service with which you are having the issue, of course. If that doesn’t work out, you can always remove (terminate) the machine itself (see above).

Useful tricks

You can connect at any time to any of your virtual machines through the command

juju ssh chamilo/0

where “chamilo/0” is the name appearing below “units” in your services.

You can check the status of all your instances with

juju status

Note that, sometimes, you might end up with dozens or hundreds of instances. In this case, it won’t be as practical to show the status of all instances (I have no solution for that now, but I’m sure there is a way to filter the results of a juju status).

You can launch a command on the virtual machines’ command line like this:

juju run --service chamilo "tail /var/log/juju/unit-chamilo-0.log"

This way, you are actually executing the command remotely and getting the results locally.

You can also see the error log locally, connecting in SSH (first) and then launching:

 tail /var/log/juju/unit-chamilo-0.log

Obviously, that gives you a little more flexibility.

Notes about unexpected errors

One of the “silent” things is that Juju considers the default machine to be Ubuntu Precise. In the case of MySQL, the default Charm is configured for Trusty. This means that if you want to install this package, you need to install a virtual machine in Trusty. Otherwise, you might get some other issues. In my case, the Precise Charm didn’t really work (missing yaml), so I decided to go for Trusty.

You can choose the distribution of your machine with –series=trusty, for example:

juju docean add-machine --series=trusty --constraints="mem=2g, region=nyc1"

We tested the chamilo charm relatively extensively.

Unmounting the whole thing

If this was just a test, and you’re happy, maybe you want to remove everything. If so, the quickest way to do that is to launch a destroy-environment command, but you will first need to destroy each machine and, before that, each services that :

juju destroy service chamilo mysql
juju destroy machine 1 2
juju destroy-environment digitalocean

This should reasonnably quickly remove the whole setup.

You should still check your Digital Ocean’s dashboard, though, as apparently it doesn’t always delete the nodes you created with Juju…

Quick commands list for the impatient

Assuming you’re running Ubuntu 14.04 and that you know which values to change in the commands below:

sudo add-apt-repository ppa:juju/devel
sudo apt-get update && apt-get install juju
sudo chmod -R 0700 .juju
sudo apt-get install python3-yaml
sudo pip install -U juju-docean
export DO_CLIENT_ID=aseriesof21alphanumericalcharacters 
export DO_SSH_KEY="user@computer" 
export DO_API_KEY=aseriesof32characters
juju docean bootstrap --constraints="mem=1g, region=nyc1"
juju docean add-machine --constraints="mem=2g, region=nyc1"
juju docean add-machine --series=trusty --constraints="mem=1g, region=nyc1"
juju deploy cs:~jose/chamilo --to 1
juju deploy mysql --to 2
juju set chamilo pass=blabla
juju add-relation chamilo mysql
juju expose chamilo

And connect your browser to (that you must have redirected to the corresponding IP first) and login with admin/blabla.


Does the world’s growth go along with a Moodle sites reduction?

June 20, 2014 5 comments

In a growing world with growing education needs, one would think that systems like Chamilo or Moodle would have a continuous growth ensured, year after year.

Well, this is certainly true for Chamilo LMS (for now), as it is apparently gaining about 100,000 new users per month (average over the last 12 months), but I am a little in shock at how Moodle is (since a year or so) loosing a considerable amount of sites and users over the last 12 months. It’s a bit difficult to track, because Moodle stats site doesn’t show any evolution of amounts over time (anymore), but thanks to our beautiful internet, you can use the Wayback Machine to get the data as it was years ago.

A warning to the reader: I am an active Chamilo evangelist and this is in no way an objective, scientific analysis of the situation. This isn’t to say that I am inventing the existing data. It comes from reliable sources and should be consider as such for all intended purposes.

So, just for the sake of the data, here is how both projects are evolving in number of sites and users over the last 4 years (almost the full life of the Chamilo LMS project). This article is written in June 2014, so the last two lines are pure speculation. The rest, however, comes from real data from Moodle’s site (and the Wayback Machine) and Chamilo’s database.

Year Moodle sites Chamilo sites Moodle users Chamilo users
mid 2011  54K  1K  43M  0.5M
mid 2012  66K  2.3K  58M  1M
mid 2013  84K  6K  71.7M  6.2M
mid 2014  64K  14K  71.1M  7.6M
mid 2015? (linear projection based on last 12m) 44K 22K 70M 9M
mid 2016? (projection based on last 12m) 24K 30K 69M 10.4M
Stats: Moodle usage vs Chamilo usage from 2011 to 2014

Stats: Moodle usage vs Chamilo usage from 2011 to 2014 + projection

The last two lines are an obvious flame. There is no way this could be like this, but if it were (and it’s a linear projection based on the last 12 months from now June 2014, it could be curved projection), then these would converge in number of sites by June 2016 :-)

As always with statistics, there are important things to know about both projects, and while Moodle is a great tool, it is still somehow a friendly rival to Chamilo, but in an ever-growing world with easier installers and cloud computing, I would expect Moodle to continue to grow… pretty much endlessly. Not sot.

A few years back (in June 2012), the Moodle stats page still offered a graphical representation of the number of Moodle installations around the world, over time. You could (at that point) clearly see a Gauss curve reaching the top and starting to go down. However, the numbers here show that even then it managed to grow an additional 13M users (and 18K sites), but then between June 2013 and June 2014, it just stalled in number of users, and literally dropped 22K sites, to lower-than 2012 level in terms of sites. A little bit later than June 2012, the chart was removed (I guess they didn’t like showing the dropping Gauss curve, huh?)

Sure, that also means that most of these sites had a very low number of users (or something like that), as this only went with a loss of 600K users overall (so 30 users per site, on average).

Or maybe Moodle stats just changed their algorithm at some point, dropping thousands of sites that were not really active?

Chamilo and Moodle statistics are bound to be somewhat imprecise, anyway, as they rely on a series of relatively subjective parameters that are interpreted to try to guess if the sites are still active or not.

Plus Chamilo is still clearly not getting to the number of users Moodle has (we’re only at 10%), but the tendency is that we are growing fast enough and that, if it continues as in the last 12 months, we should be crossing

Still, this leaves you thinking… Free software (no barrier to adoption), free or very cheap hardware, growing world population… why did it drop? Will it continue to do so?

Well, that just leaves me with an excellent marketing opportunity, to tell you that, if you want to try Chamilo out, you can test our recently released version 1.9.8 on Now.

Howto: Configuring session expiry time in Chamilo 1.9

We seldom receive a request from users of Chamilo LMS saying their sessions are cut in the middle of their activity. And sure, it might so happen that you are in the middle of the redaction of a very large answer to an open question, or diserting on how the course is going to help you in the forum. And we get that it’s super-frustrating to click “submit” and then get an error page. We do, really.

Now on our side of the fence, we have to cover a series of non-trivial issues…

If you leave a session open for too long, another user might hack your session and get inside the system in your place (unless you use an unflawed SSL certificate to protect your communication).This is generally OK if you are a student, but what if you are an administrator or if you are viewing super-confidential learning content?

Another issue you might have is leaving a public computer without closing your session, and have someone else “follow you” and use your session. This leads to the same problem as above. Finally, not cleaning the sessions from time to time inevitably leads to thousands (or rather hundreds of thousands) of sessions being handled by the server, which inevitably leads to a slow server.

So in the best interest of all, it is important to have a balanced session time. We generally consider that 2 hours is a reasonnable total time. If you’ve been inactive for two hours, then it’s reasonable to get disconnected when you come back. ’cause honestly, you weren’t really studying, were you?

But even with that, we still got complaints from the users, so we decided to put it at 100 hours. OK, so that’s 4 days and 4 hours. Enough, right? That’s the default setting in Chamilo, and you can find it in your main/inc/conf/configuration.php on the line that says:

// Session lifetime
$_configuration['session_lifetime'] = 360000;

That’s the value that comes with a default installation of Chamilo 1.9.*.

Now, even if you have that, and depending on your PHP settings for session handling, you might still need to change two settings, but these are out of Chamilo’s control, directly into PHP’s configuration:

session.gc_maxlifetime = 1440

This is the number of seconds after which the garbage collector (a vacuum cleaner, kind of) of PHP considers that session files, on the server, if left untouched, will be erased, and

session.cookie_lifetime = 0

which defines the time (in seconds) that the cookie will ask to the browser to be stored for. If 0, it means that it will stay there until the browser is closed. If anything more than 0, it will stay there for that number of seconds.

Now… the funny thing here (which makes it even harder to track) is that if you use the default session handler of PHP, called “file”, the time used for the garbage collector to erase the sessions files is *not* the last time the session was accessed, but rather the last time the session was *modified*. This means that, if you have some AJAX block refreshing every 30 seconds or so, this will *not* maintain the session active, unless this AJAX refresh actually modifies something to be stored in the session.

So there are many factors to take into account. Our preferred/recommended setup?

$_configuration['session_lifetime'] = 10800; // 3h

Then, in your PHP config:

session.gc_maxlifetime = 10800
session.cookie_lifetime = 0

But now you know how it works, you can tune it as well.

Kudos to Gumbo on Stackoverflow for the missing bits:

Categories: Chamilo, English, Techie Tags: , , ,

Chamilo v10 – Big changes for a big future

This Sunday 15th of June 2014, we will be releasing version 1.9.8 of Chamilo LMS.

This will be the last big version of the 1.9 branch (version 1.9.0 was released in August of 2012, about 2 years ago). Version 1.9 has served the Chamilo community well, and BeezNest will continue providing support on 1.9.* until 2016 at least.

In the same time (from 1.9.0 to 1.9.8), the Chamilo community has grown 400% times, from less than 1 million users to more than 5 million users today, only in 1.9.* users, which we assume is a sign of appreciation from our community.

But as our community grew, so have most of the installations that use Chamilo, now reaching, for one specific case, the hundreds of thousand users.

All this growth in numbers have required many customizations and optimizations in Chamilo LMS, up to a level where it has been very difficult to integrate them in Chamilo at all, because of its database and files structure hierarchy.

As a reminder, and before we go deeper into versions, remember that Chamilo LMS (versions 1.8, 1.9, etc) and Chamilo LCMS (v2, v3, v4) are completely different software systems, and that they are developed by different teams, under different leadership, but within the context of the Chamilo association (as long as they develop under a certain set of rules protecting the open nature of the software).

So many people ask me what is so special about v10 that will really make Chamilo a better tool? It’s hard to answer that at this point (because not everything is completely certain yet), but this is a shortlist of the most important changes you will see in Chamilo LMS v10 in comparison to v1.9.

First and foremost: the interface will not change in any major way for the final user. Of course, we will have some refreshing that makes it more appealing to the users, but no major change in how elements are presented. This is the top quality of Chamilo LMS in comparison with other systems, and we don’t want to loose that. So for all of you managing a support team on Chamilo: don’t worry, no major change in the way users can use the current tools.

No major tool will be removed, either, to the specific exception of the reservations tool, which has been there for years and that apparently nobody used.

In terms of new tools or major changes to existing tools, this is what v10 will most certainly come with:

  • Super-categorized-random questions selection for exercises
  • Possibility to define new roles (this will be in beta version in v10.0 and improved based on user feedback) and permissions
  • e-mails templating
  • Multi-users, HTML5, mindmapping tool
  • Optional central repository of documents, allowing for many new learning objects view modes and sharing options
  • Integration with common web 2.0 platforms (Google Drive, Dropbox, Youtube, etc)
  • Full HTML5 online editor (moving to CKEditor 4 or superior)
  • New social network “wall” feature, to inform others about the interesting stuff you are learning
  • ODF documents reader
  • OpenBadges support
  • Open Plagiarism detector
  • Video chat (HTML5, not for IE)
  • Audio frequency analysis tool for exercises
  • and some 100 other less important improvements not listed here

And now the category for geeks:

We believe the major change you need to be aware of is that we decided to rely on a series of Symfony 2 (framework) components to manage the core of our application. This involves a series of structural changes (which will be easy to manage with our migration script) but will critically improve the extensibility of Chamilo LMS

  • Use of Doctrine as a database layer manager
  • Unique (single column) row ids in all tables (in 1.9.x, the c_* tables had a combination of course ID and id to form the unique key)
  • Integer course ID (deprecating as much as possible the use of the literal “course code”, making tables faster to search)
  • Use of a central controller, which will allow you to redefine the route to specific scripts, and use friendly URLs (partial implementation in v10)
  • Installation profiles (install with configuration presets)
  • Debian packaging (and new files structure)
  • New centralized authentication mechanism (including OAuth)
  • New web services for listing objects (enabling easier application of mobile apps)
  • Translations managed in Gettext format (and a new translation platform)
  • Full integration of the templating system (Twig), with one JQuery-mobile-enabled template
  • Dropping support for IE8 (most probably) so we can move to SVG icons everywhere
  • Support for IMS/LTI, enabling easier integration of external applications into Chamilo
  • Improvements to enable auto-scaling of Chamilo in the cloud
  • Juju Charm to auto-install Chamilo in your infrastructure
  • Extra fields for all major items (quizzes, questions, learning paths, users, courses, sessions, etc) enabling for easier plugins integration
  • Better plugins framework overall, enabling some functions “hooking” or “redefinition” in specific contexts

This list might change over time. This version is from the 12th of June 2014.

It is important to understand that the structure changes have been kept back for years (literally) to avoid changing Chamilo 1.9 in any major way, and we have now acquired a great experience into large-scale implementations, so the structure changes we are implementing now are good both for the present but also for the foreseeable future (up to 5 years from now), so we should not require any major infrastructure change anytime soon.

We are making big changes now, for what looks like a very big future for Chamilo LMS!

You can find more about our dynamic roadmap here:

Categories: Chamilo, English Tags:

How to not write a condition for code readability

Sometimes you find a perl in some code and it might take you a while to understand.

In the sessions management code of Chamilo, I found this the other day, then decided to leave it for later because I was in a hurry. Then today, I found it again, and again… it took me about 10 minutes to make sure of how the code was wrong and how to make it right.

 if ( $session->has('starttime') && $session->is_valid()) {
 } else {
   $session->write('starttime', time());

To give you the context, we want to know whether we should destroy the session because its time is expired, or whether to extend the period of time for which it is valid (because it is still active now, and we just refreshed a page, which lead us to this piece of code).

OK, so as you can see, we check if the session has a value called “starttime”. If it hasn’t we pass to the “else” statement and define one, starting from now.

If, however, the session has a start time, we check whether it… “is valid” (that’s what the function’s called by, right?) and, if it is… we destroy the session.

Uh… what?

Yeah… if it is valid, we destroy it (DIE SESSION, DIE!)

OK, so something’s wrong here. Let’s check that “is_valid()” method (there was documentation, but it was wrong as well, so let’s avoid more confusion)

  public function is_valid()
    return !$this->is_stalled();
  function is_stalled()
    return $this->end_time() >= time();

OK, so as you can see, is_valid() is the boolean opposite of is_stalled(). “Is stalled”, in my understanding of English (and I checked a dictionnary, just to make sure) means “blocked” (in some way or another). Now it doesn’t really apply to something that is either “valid” or “expired”, either. And this is part of the problem.

The is_stalled() method, here, checks if the end_time for the session is greater than the current time. Obviously, we’ll agree (between you and me) that if the expiration time is greater (further in the future) than the current time, this means that we still have time before it expires, right?

Which means that, if the condition in is_stalled() is matched, we are actually *not* stalled at all. We are good to go.

So I believe the issue here was the developer either confusing the word “stalled” for something else or, as it doesn’t really apply to “expired” or “not expired”, that he simply badly picked the word describing the state.

In any case, now that we’ve reviewed these few lines of code, I’m sure you’ll agree it would be much clearer this way:

 /* caller code */
 if ( $session->has('starttime') && $session->is_expired()) {
 } else {
   $session->write('starttime', time());
 /* called method */
 public function is_expired()
   return $this->end_time() < time();

Conclusion: always choose your functions naming carefully. Others *need* to be able to understand your code in order to extend it.

%d bloggers like this: