My own Firefox Sync – Weave server

Hello,

I am still reluctant to store my personal information online or use free services provided by company that already claims it may stop for any reason (Google, Microsoft). So I do not make an exception for Mozilla for Firefox Sync service.

As I do not have an online server at home and have no other needs to pay for a hosted VM, I want Weave to deploy and run at Free.fr.

The official weave server which consists in User and Sync components requires Alias directive in Apache2 but Free.fr does not allow that directive in htaccess file.

The minimal weave server is now maintained but uses SQLite with PDO interface as a storage. As Free.fr provides a MySQL service, why would I bother with SQLite. It is straightforward to switch to PDO:mysql but Free.fr does not enable this backend. In fact, the only PDO backend supported at the moment is SQLite…

OK, so it is open-source, I have never written PHP yet but it is just another language to learn and the only source I have to change is « weave_storage.php ».

Job done:

  • Prefix tables with weave_ to prevent collision with other services, typically for user table
  • Port SQL create statements and indexes to MySQL
  • Rewrite weave_storage code to replace PDO and exception handling by old-school mysql_* methods and error handling
  • Use class.JSON.php instead of recent included json_(en|de)code methods

In N900 Firefox, open about:config to set your own URL as value of services.sync.serverURL.

After some code cleanup, creation of a sql file, a short documentation in README, here is a package, including a patch based on latest version on date of 2011.03.23. This release should work with Sync plugin versions 1.6.x and 1.7 in Firefox 3.6, 4 or 5:
weave_minimal_server_20110626_mysqlJSON.tar.gz

Additional note for free.fr users, in case of mixed PHP 4 and 5 hosting: you have to create a .htaccess file with content php 5 to enforce PHP 5 usage.

25 commentaires

  1. Hi!
    This is exactly what I was looking for! I already used the JSON class to get around the PHP5.2 restriction. Also I dont’t have sqlit3 compiled into php. My hoster doesn’t support this. I could use a mysql db though.
    I already started with porting PDO to mysql but I am really struggeling… Could you, by any chance, provide me the weave_storage class for mysql? That would be really really great!
    Good job by the way!
    Thanks,
    Boris

  2. Hey Yves!
    This is working great!
    I have some remarks though. I’m seeing some errors in the access log, which show some SQL errors. This is particularly in the storage class line 151 but the statement is not complicated so I guess this was a « one time error ». I guess ne need to worry :)
    The other really minor issue is your readme file, the table name in the SQL statement is missing an « s », it’s « …users » not « …user ».
    Other than that, this is really great! Also I like the prefixing of the tables.
    I already synced a couple of PC’s. You made my day! Thank you very very much for sharing! Like I said I had been struggeling a couple of days with getting this to work :)
    Boris

  3. Its really good work buddy….but its not perfect ;)

    [Mon Mar 28 22:08:33 2011] [error] [client 93.x.119.18] PHP Warning: mysql_result() expects at least 2 parameters, 1 given in /var/www/weave_minimal_server/weave_storage.php on line 515
    [Mon Mar 28 22:08:48 2011] [error] [client 93.x.119.18] PHP Notice: Undefined offset: 4 in /var/www/weave_minimal_server/index.php on line 60

    [Mon Mar 28 22:10:21 2011] [error] [client 93.x.119.18] request failed: URI too long (longer than 8190)
    [Mon Mar 28 22:10:22 2011] [error] [client 93.x.119.18] retrieve_collection: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ‘?)’ at line 1

    hope you can fix it, cause i really don’t like sqlite ;)

  4. Hi,
    Thanks, that is what I was looking for (installing Firefox sync on free.fr), but I got a problem at the very beginning of the index.php: I do not have such a $_SERVER['PATH_INFO'] variable. I have checked with phpinfo. Any idea ?

  5. Hello,
    It does not matter, the value « / » is used when PATH_INFO is not available. Please check you have enabled PHP 5 with .htaccess file.

    I have just uploaded a new version with important fixes included.

    Thanks for reports – it helps improving the code !

  6. Hello Yves,
    I’ve just seen your reply. Thanks
    I had enabled php5, that was not the issue.
    When PATH_INFO or ORIG_PATH_INFO are not found, a 404 error is thrown, it is NOT using « / » by default, as we need the complete path (ex: /1.0/username/info/quota/)
    After reading some posts in forums, I added:
    else if (!empty($_SERVER['SCRIPT_NAME']))
    $path = $_SERVER['SCRIPT_NAME'];
    on index.php, line 56

    When I manually query my server, it seems to behave properly now.

    However, when I try to register it on Firefox 4, I get a message saying that URL is invalid.
    Any idea of what is wrong with my URL ?
    I suspect that the Firefox4 registration form does not handle other servers properly, as for username it requires an email (which is not mandatory on the weave minimal server). I’d like to use the same email (mine), but I get an error saying this email is already used, though I want to use a personnal server …

  7. I am using Firefox4 with Sync 1.7. But, your script is made for 0.5 or 1.0 only.
    I tried to simply allow 1.7 on index.php (line 63 of original script):
    if ($version != ’0.5′ && $version != ’1.0′ && $version != ’1.7′)
    Hoping it would help, and yes it did :)

    To register the server in Firefox4, I had to do (in French, as I have a french firefox version):
    J’ai déjà un compte Firefox Sync. => Se connecter
    Je n’ai pas l’appareil avec moi
    Compte: myUsername
    Mote de passe: myPassword
    Serveur: « Utiliser un serveur personnalisé » et rentrer « http://account.host.com/weave_minimal_server/index.php/ »
    Clé de synchronisation: myPassword

    I am not sure however the service will behave properly, and indeed, though Sync created 3 826 entries, I am now getting a sync unknown error :( I think an updated weave minimal script must be used…

  8. Hello,

    I am not the original author of this code. It comes from http://hg.mozilla.org/services/minimal-server/

    I have just changed persistence on MySQL. There is no chance I can update the code to support a new protocol version. So I will wait for a new release, but job is already in progress: protocol version 1.1, support for email address encoding (but normal username is still OK), new method « collection_usage ».

    At the moment, my Firefox 4 connects properly, I guess its Sync plugin is not in 1.7 version yet.

    I have found no usage of « mcrypt_ » methods in the server code. I guess it is the Sync plugin job to encode/decode data at browser side. Easy to confirm by looking at entries in database.

    Two advices:
    - do not use your username’s password as synchronisation key – it weaks data protection, especially if you do not use https.
    - add some random part in your URL to protect your server for instance /weave_minimal_server_dCHq0my2/index.php/

  9. I have just uploaded a new version which includes recent changes from original code. It may help to avoid errors reported by Sync plugin 1.7.x.

    I have not tested it a lot but it seems per collection quota summary fails. Hum, personally I do not care.

  10. Hi Yves,

    Well I uploaded your script to my web server and setup the required mysql database. I then added one user to the system. However, while setting up my first Firefox browser, I’ve hit a brick wall. When I enter my username, password and the server address (https://sync.domain.com/index.php/), the Sign In screen says that I’ve entered an Incorrect account name or password. I’ve checked the credentials a number of times. How can I check that the server is working correctly?

    Thanks alot for all your efforts,
    Al

  11. Hello Yves,

    Thank you for sharing your code.

    I’m unfortunately getting the same exact issue as Al.
    When I try to setup Sync to use the server I keep getting the error « Nom du compte ou mot de passe incorrect ».

    I’ve tried to connect to the server directly via https://address_to_server/index.php/1.0/username/info/collection . I get a login request which keeps failing when I enter my username and password. I’ve doubled checked my credentials many times.

    Do you have any idea how to fix this ?

    Best,

    semiaddict

  12. As authentication is the very first and mandatory operation, it does mean your username and password are wrong, it may be: incorrect mysql database setup, lacking PHP extension or function if wrong version number…

    If you cannot get PHP error log or Apache access log or MySQL queries, the only way is to guess what happens from the HTTP error code and error message and I catch them with wireshark http://www.wireshark.org/.

    I have just found this page which explains how to read Sync plugin logs:
    http://philikon.wordpress.com/2010/11/12/sync-in-firefox-4-0-beta-7/

  13. Hi Yves,

    I wasn’t able to figure out how to get a clean log from wireshark; the log has way too many lines. I guess I need to filter the logging, but I don’t know how.

    And unfortunately my host doesn’t allow me to access server logs.

    But I was able to get a FF Sync log, which is available here:
    http://pastebin.com/dgfbJZt7

    Any help is highly appreciated.

    Best,

    semiaddict

  14. Hi Yves,

    I’ve managed to install the original version of the minimal server which uses SQLite, but still get the same issue.
    So I’ll be submitting this issue to the original post.

    Thank you for your help.

    semiaddict

  15. Here is a method to test your server without involving Firefox Sync plugin itself:

    Create the URL for your setup and your USERNAME based on the pattern « http://server.domain.com/path/to/weave/index.php/1.0/USERNAME/info/collections »

    Now use a browser to open this URL and provide your username and password in the browser authentication popup.

    Or with a Linux command line, invoke
    wget --http-user USERNAME --http-password "ClearPassword" "http://server.domain.com/path/to/weave/index.php/1.0/USERNAME/info/collections"

    Expected result is [] before any successful sync.
    Then it looks like (numbers are timestamps of last sync):

    Additional information in case of troubles:
    the username must not contain a "@" or else it must be encoded specifically with
    create_user.php script.
    Then a semi-colon ":" in password is known to cause troubles.

  16. If your setup always trigger a 401 code even if username and password are correct,
    maybe your Apache2 configuration required authentication before PHP script is evaluated.

    That is not correct. The index.php script must be accessible without authentication from
    Apache2 point of view. The script asks for credential and validates them on the users table.

    To be sure the index.php is the originator of the authentication request, looks at the realm after « The site says: » – it must be « Weave » or begin with « Weave ».

    If you get a 400 error code, the main cause is a mismatch between the username provided at authentication and the username used in URL after index.php/1.0/

  17. After diagnosing an authentication problem with OVH shared hosting service for a user,
    it appears the Authorization header is available as REMOTE_USER server variable which sounds really odd. This variable is normally used for single-sign-on or proxy authentication and is supposed to contain a username only, not the Authorization answer with Basic xxbase64xx.

    So if you have an authentication issue, it may come from hosting setup. Edit weave_utils.php and add at line 119:

    else if (array_key_exists('REMOTE_USER', $_SERVER))
        /* Found at OVH */
        $auth_str = $_SERVER['REMOTE_USER'];
  18. Hi Yves,

    after FF5, the sync server does not work anymore. This is due to the version change of the protocol. In case you may want to update the archive, these are the changes that need to be done:

    INDEX.PHP:
    …(before)
    if ($version != ’0.5′ && $version != ’1.0′)
    …(after)
    if ($version != ’1.0′ && $version != ’1.1′)

    …(before)
    switch ($collection)
    {
    case ‘quota’:
    exit(local_json_encode(array($db->get_storage_total())));
    case ‘collections’:
    exit(local_json_encode($db->get_collection_list_with_timestamps()));
    case ‘collection_counts’:
    exit(local_json_encode($db->get_collection_list_with_counts()));
    default:
    report_problem(WEAVE_ERROR_INVALID_PROTOCOL, 400);
    }
    …(after)
    switch ($collection)
    {
    case ‘quota’:
    exit(local_json_encode(array($db->get_storage_total())));
    case ‘collections’:
    exit(local_json_encode($db->get_collection_list_with_timestamps()));
    case ‘collection_counts’:
    exit(local_json_encode($db->get_collection_list_with_counts()));
    case ‘collection_usage’:
    $results = $db->get_collection_storage_totals();
    foreach (array_keys($results) as $collection) {
    $results[$collection] = ceil($results[$collection] / 1024); #converting to k from bytes
    }
    exit(local_json_encode($results));
    default:
    report_problem(WEAVE_ERROR_INVALID_PROTOCOL, 400);
    }
    ….

    WEAVE_STORAGE.PHP
    …(insert method)
    function get_collection_storage_totals()
    {
    $select_stmt = sprintf(« select collection, sum(payload_size) as payloadsize from wbo where username = ‘%s’ group by collection »,
    mysql_real_escape_string($this->_username));
    $sth = mysql_query($select_stmt, $this->get_connection());
    if (!$sth) {
    error_log(« get_storage_totals:  » . mysql_error($this->get_connection()));
    throw new Exception(« Database unavailable », 503);
    }

    $collections = array();
    while ($result = mysql_fetch_assoc($sth)) {
    $collections[$result["collection"]] = (int)$result["payloadsize"];
    }
    return $collections;
    }

    Hope this helps.

    Best regards,
    Boris

  19. Your SQL-String in the function « get_collection_storage_totals() » fails! You refer to « wbo » instead of « weave_wbo ».

    The right SQL-String is:

    « SELECT collection, SUM(payload_size) AS payloadsize FROM weave_wbo WHERE username = ‘%s’ GROUP BY collection; »

  20. Hi,
    the weave server produce after awhile a lots of overhang. To clear the overhang i run the optimize statement for the « weave_wbo » table in phpmyadmin: « OPTIMIZE TABLE weave_wbo ».

    Now i am looking for a solution to do that automatically.
    Have somebody a solution for this problem? It must only be further SQL statement to execute this function, but i don’t know where should i put this statement. Any ideas?

    regards
    Hansii

  21. Pingback: Boris Wach

Laisser un commentaire