Menu

Having every bit of information defined at every aspect of application makes system actually finished.
Piotr Masełkowski, Maslosoft Founder

Codeception Acceptance: Visible - Invisible element

200

In one of projects, I have acceptance test for testing if user can be removed. The test was from some time failing. There is a helper method to select grid view action via xpath query. That was a first suspect, so that something in grid view has changed, so that the query was no longer valid. The query was a bit risky, so the new grid view component have added data-id attribute for each row for easier selection.

With new data-id  attribute it could as well be queried with CSS selectors. So I have tried selector with both jQuery and document.querySelectorAll. Both returning proper element. Resulting in selector more obvious then previous xpath one:

document.querySelectorAll('tr[data-id="59c16cd0a3d24b725b6171a8"] a[data-bind*="app.module.uac.user.trash"]')

Executing acceptance tests however yield element not visible error. And there the troubles started. Back to console for testing - works fine. However when executed from test runner - it failed.

Resize Window

After looking at test failed screenshot... Well, the element was on screen, but because the test browser window was narrower than developing one, test failed. The fix for this issue was to resize window before tests. I've made it on AcceptanceTester class, so it was for sure executed:

class MyAcceptanceTester extends AcceptanceTester
{
        public function __construct($scenario)
        {
                parent::__construct($scenario);
                $this->resizeWindow(1900, 1000);
        }
}

Using GPG with PHP on server

Use case presented here is for signing and verifying files, however by tweaking commands it is possible to encrypt and decrypt files too. Generally it's about GPG batch operations.

The idea with GPG (or PGP, which is compatible) is that keys are securely stored in user home dir. PHP comes with PECL extension for providing GPG operations, however it is using and approach where application manages the secret and public key. This could compromise security, as these keys would be probably stored in some PHP variables. The workaround for this could be set up an key pair for web server user.

Transferring key pair

In case You need key pair, issue following command:

gpg --full-gen-key

This will guide You through process of generating key. We will not came into details on generating keys, as it's a side and well known topic.

For batch operations key might be created or exported without password. As it anyway need to be stored somewhere. Password can be provided at command line or possibly by GPG agent.

Keys can be generated directly on server, however when having more than one server, keys need to be distributed. 

Exporting keys

To export public and private keys we'll use following command:

gpg --export key-email@example.com > key-name.pub

By default GPG outputs key to console, so we redirect output to file with > operator.

The similar command is for a private key:

gpg --export-secret-keys key-email@example.com > key-name.asc

Transfer these files with encrypted connection to server. This could be done with GUI client, like FileZilla.

Importing keys

To import keys for web server user - we assume here www-data - we need to create home directory with proper permission first, if that user haven't it already:

sudo mkdir -p /home/www-data/.gnupg
sudo chown -R www-data:www-data /home/www-data/.gnupg sudo chmod 700 /home/www-data/.gnupg

To import keys we'll again use sudo command with -u option to work on behalf of www-data. However even with sudo, the GPG will try to import keys into current user key ring. To overcome this, there we need to pass --homedir parameter to GPG:

sudo -u www-data gpg --homedir /home/www-data/.gnupg --import key-name.pub

As a side note ensure that file key-name.pub is readable by www-data. The command to import the private key is the same, except as a parameter we pass a private key file name:

sudo -u www-data gpg --homedir /home/www-data/.gnupg --import key-name.asc

Verify Transfer

Keys should be now imported. Note that when using with sudo, the --homedir parameter need to be added. To verify that keys have ben imported properly, list them:

sudo -u www-data gpg --homedir /home/www-data/.gnupg --list-keys
sudo -u www-data gpg --homedir /home/www-data/.gnupg --list-secret-keys

Next step is to sign and verify signature as www-data user with imported key. To ensure that exact key is used, we'll use the -u argument, to choose key manually and sign file named test.file:

sudo -u www-data gpg --homedir /home/www-data/.gnupg -u key-email@example.com --output test.sig --detach-sig test.file

The commands started to become lengthy. However parts of the commands are repeatable, so so simple PHP command builder can be used later. For now, let's verify results. First of all, there should be created new file - test.sig - containing binary data with signature. To verify signature call gpg command with --verify option:

sudo -u www-data gpg --homedir '/home/www-data/.gnupg' -u key-email@example.com --verify test.sig test.file
Most likely there will be warning that the key is untrusted, but should verify signature.

Using with PHP

After all those commands, the GPG is ready to be used with PHP. Now is the easy part - just execute proper shell commands. There is no need for external library. As the key ring is owned by web server user commands will use previously imported key. But it is good to point to exact key with -u option. Part of command will be the same for all operations. Set key, add --batch and --quiet options to make gpg command non-interactive and select key.

The idea is to create array with arguments and then join it. Example "class", which is in fact functions container:

class Signatures  
{
    public static function sign($fileName, $sigName)
    {
        $args = self::makeArgs();
        $args[] = '--output ' . escapeshellarg($sigName);
        $args[] = '--detach-sig ';
        $cmd = sprintf('gpg %s %s', implode(' ', $args), escapeshellarg($fileName));
        return self::run($cmd);
    }

    public static function verify($fileName, $sigName)
    {
        $args = self::makeArgs();
        $args[] = '--verify ' . escapeshellarg($sigName);
        $cmd = sprintf('gpg %s %s', implode(' ', $args), escapeshellarg($fileName));
        return self::run($cmd);
    }

    private static run($cmd)
    {
        $result = null;
        $output = [];
        exec($cmd, $output, $result);
        // Return true on successfull command invocation
        return $result === 0;
    }

    private static function makeArgs()
    {
        $args = [
            '--batch',
            '--quiet',
            '--homedir /home/www-data/.gnupg',
            '-u key-email@example.com',
        ];
        return $args;
    }
}

Above example has common parameters hardcoded, just for sake of simplicity. This could be simplified even more, as there are a bit of redundant code. To test is as a web user, either run it through browser or as command line with sudo -u www-data.

Don't release of Friday!

The old story strikes back. Dozen years ago I used to work for local IT company. Even then we had work out a often release cycle, so that usually each month we had a new version of software. This is quite good attitude, as there are less changes between versions, and also team is getting used to release cycles.

Don't release on Friday!
Adam, DevOps admin

The DevOps admin always complained to not release a new version on Friday. But often developers tried to finish tasks before weekend, so usually a new version was ready Thursday afternoon - that is - too late for Thursday release. Being happy with everything tested thoroughly, at Friday morning we insisted admin to roll the release. As we still had whole day for hotfixes. Usually updates were fine, but few times there was some havoc making admin nervous if team will finish hotfix.

Yesterday I had everything almost ready, just needed to finish a new feature - searching by tags in Quick Tips. There were some issues, but at the afternoon everything seemed to work pretty well. Being happy with results I decided to release a new version, even through it was already afternoon.

There comes time to update all dependencies in application (update composer and bower), as on my dev machine these are only linked for easier development. All went smooth. Almost, as some small hotfixes were required. But suddenly catastrophe, cannot edit any HTML field! The worst thing was, that changes were made in uncomitted JavaScript file - this file was merged into balin package, and was meant to be pushed to knockout ES5 plugin. But after doing several updates, the file was reset to state before important fixes were added.

Backup to the rescue

Luckily, I have daily backups set up. Which holds around half milion files and gigs of data. This was the culprit. It turned out, that with so many backed up data, extracting required file would take ages!

Here comes plan B - this file must be on old server, which is still running. It turned out, that the file was not there as it was only merged in balin. But I could not figure out which part is causing havoc - as there was some code which needed to be removed.

In summary, backup would take a half day to extract file, other versions - including those on github were the wrong ones. Tried to figure out what had to be removed by comparing previous versions of built package, but it was not really obvious.

Alternative backup

At some point I realized, that from time to time I do simple backup of my projects using plain old tar command! And that was it! The file was there. It turned out that WeakMap shim was causing troubles. The release was finished at 2 at night!

In summary

Always try to have alternative backup made in a different way!
Oh, and don't release on Friday!

Reviewed an updated documentation for balin

The Maslosoft Balin Project had documentation migrated from old semi-static pages to be able to integrate into main Maslosoft website. Now all live documentation examples have been reviewed, cleaned up, in some cases fixed and also extended with more detailed descriptions.
The project itself is a set of more or less complex Knockout JS binding handlers. It is written in CoffeeScript for better clarity - and in fact writing this project in CoffeeScript made it possible to make it work.

Limiting var_dump output

Sometimes to quickly inspect what's in object, PHP var_dump function can be used. This will show all object properties, including private and protected, also of all nested objects.

Sometimes this is just too much to produce useful output.

Fortunately in recent PHP versions, a new magic method was added: __debugInfo. When implementing this method, it should return array with keys as object names, and it's values as values that should be displayed.

Extra Knockout Binding Handlers Demos

Finally extra binding handlers project - balin have accessible live documentation. Check out this project for inspiration of how binding Knockout JS binding handlers can be made or just use existing ones. There are numerous small helpers as well larger binding handlers, including but not limited to:

Annotations for anonymous classes

A new release of addendum provides annotations for anonymous classes. One of anonymous classes feature is that these can be wrapped in another class. Fortunately PHP has support for reading doc comments from those classes, thus allowing to parse annotations. Addendum will create annotations container just like if it were named class.

Creating Your first PHP annotation with Maslosoft Addendum

254

Creating annotation class read from begining to learn about configuration

Having addendum configured, let's create annotation class. This will allow us to use @ notation on our project classes. As each annotation is in fact class, which has some simple logic to set up metadata container. This container will be used to obtain values configured by annotations. Please not that this values might be different than raw values on @ notations. As value returned by meta container is solely dependent on values set by annotation definition.

This way, we have opportunity to set initial logic in annotation, thus having already prepared and cached values returned by metadata container. To obtain access to container for currently process class, there getEntity is method in annotation. This returns container instance, which can be configured with any property required.

New smarter way to use cache

When dealing with caching, common task is to load cache value, check if it is not empty and depending on it - either set some variable or perform some operation to retrieve this value.
This common tasks might scattered around various places in application. 
New cache component feature comes as remedy, allowing to execute callback if value is not already cached. This makes possible to use one-liners to either get cached value or retrieve it from any source.

Codeception Acceptance: Filling in contenteditable

When working with form fields, Codeception has already prepared tools to fill-in required data. However modern applications often works on editable elements via contenteditable attribute.
This approach has advantage of having visual HTML right away editable. When running in-browser acceptance tests, it is required to fill-in those values too. This can be done with Facebook WebDriver package, included in main codeception package.

jQuery UI jumping sortable items

Sometimes when creating sortable user interface widgets, some various quirks might happen. One of these is that just after starting dragging element, it jumps of from original location. Element will move above cursor for roughly height of it's parent container minus one sortable element.

There are plenty solutions including JavaScript to fix container height. These are clunky as it need to be assigned to each sortable. Even more problematic when used with Knockout JS sortable, as including JavaScript in HTML attributes is not pretty and really not recommended.

I turned out that properly styled container will remedy jumping issue. The point is to make it's overflow auto:

.my-sortable-container{
    overflow: auto;
}

That's it! Now sorted item should not jump anymore. Credits goes to: http://stackoverflow.com/a/24122186/5444623

Selenium bad magic number error

Using selenium for acceptance tests?

It usually run fine, until I got bad magic number error when trying to run any version:

invalid file (bad magic number): Exec format error

It is quite confusing message, basically it means that it cannot be executed. Solution is to execute it using java with -jar option:

java -jar selenium-server-standalone-3.0.1.jar

To make it easier, selenium bootstrap file can be created to launch it:

#!/bin/bash
java -jar /usr/local/bin/selenium-server-standalone-3.0.1.jar
This file can be named, for example /usr/local/bin/selenium with jar file located in same directory.

Using composer with Xdebug on ubuntu

Using composer with Xdebug turned on will result in warning message, saying that Xdebug affects performance - and that's true. Composer will run much slower when used with Xdebug enabled. To overcome this issue, it is possible to disable Xdebug when using composer command.

Note that following steps are meant to be used only on development machine, do not use on production servers!

To turn on and off Xdebug, we will use bash script - which will link or unlink Xdebug configuration - as just disabling Xdebug in config still degrades performance.

Modernized website

As a lot of development effort were put into many Maslosoft's projects, this website was left behind.

Now it's time to keep up, so a highly redesigned system along with content was deployed. This includes documentation for many projects and blog You are reading now. But the most exciting part is a bit of knowledge about quite mature projects developed since... Who knows when.

As an incorrigible optimist, still creating software resulting in comfortable web editing, as it's my point of interest from ages.

  • Projects