Monthly Archives: November 2014

Typo3 scheduler: Reschedule file abstraction layer (FAL) indexing

Leave a reply

While working on a recent Typo3 project, I used an Amazon S3  bucket to store all media files, in my case these were images and videos. As a driver I used the following extension. Until recently, it didn’t support extracting metadata like height and width when uploading images. This problems was fixed in one of the releases, see the information here.

After installing the update, metadata extraction worked for newly uploaded files, but the images, which were uploaded before the fix, still had width and height 0 in the table “sys_file_metadata”. In order to extract the metadata also from those files, I had to add two tasks using the scheduler: “File Abstraction Layer: Extract metadata in storage (scheduler)” and “File Abstraction Layer: Update storage index (scheduler)”.

Two tasks added to the scheduler in order to extract metadata.

Two tasks added to the scheduler in order to extract metadata.

After running the two tasks from the scheduler, I still didn’t have the metadata for any files in the table “sys_file_metadata”. I assume that this happened, because the scheduler ignored all files, probably depending on a previous run of the task. The only thing I had to do is overwriting the column “last_indexed” of the table “sys_file”. Use the following SQL statement on the typo3 database:

UPDATE `sys_file` SET `last_indexed`= 0
WHERE `storage` = 3

Note: I used the where statement, because the storage I used is a Amazon S3 bucket, identified as number “3” of the storages managed by typo3.

After overwriting the value of the column “last_indexed”, I was able to run the two tasks of the scheduler and the images were indexed and the metadata extracted.

Amazon S3 NoSuchKey on folder level – folders don’t exist

2 Replies

While working with Amazon S3 buckets and connecting them to Typo3 I stumbled into a problem: When I copy the contents to the bucket from another bucket or via tools such as BucketExplorer, the XML output (on bucketname.s3.amazonaws.com) showed all files, but not the folders. As an example:

<Contents>
  <Key>folder_one/file_one</Key>
  <LastModified>2014-11-18T14:16:19.000Z</LastModified>
  <ETag>"6e563e7cf3fda1549cb13aeae7a53b24"</ETag>
  <Size>6148</Size>
  <StorageClass>STANDARD</StorageClass>
</Contents>
<Contents>
  <Key>folder_one/folder_two/file_two</Key>
  <LastModified>2014-11-18T14:16:19.000Z</LastModified>
  <ETag>"6e563e7cf3fda1549cb13aeae7a53b24"</ETag>
  <Size>6148</Size>
  <StorageClass>STANDARD</StorageClass>
</Contents>

The problem here is that I supposedly also require entries for the folders themselves, in the example that would be “folder_one/” and “folder_one/folder_two/”. If I don’t have these entries, connected systems like Typo3 won’t recognize, that there are folders. If the entries are missing for the folders on the root level, I even get an exception like this:

HTTP/1.1 404 Not Found x-amz-request-id: XXX x-amz-id-2: XXX Content-Type: application/xml Transfer-Encoding: chunked Date: Tue, 18 Nov 2014 09:10:16 GMT Server: AmazonS3 NoSuchKeyThe specified key does not exist

This is confusing, because when viewing the bucket in the Amazon web interface, it shows the folders while exploring.

After some research I found out that everything in the Amazon S3 buckets is treated as an “object”, files as well es folders. I assume that while uploading or copying to a bucket, the tool being used just copies the objects for the files, but not the objects for the folders. In order to fix this, I wrote a short script in PHP which goes through all objects and adds the missing objects for folders. As an example: Imagine you have an empty bucket and just one file with the object “/folder_one/folder_two/file”. The script will see this path and add the two objects “/folder_one/” and “/folder_one/folder_two/”.

In order to use the script, download the Amazon AWS SDK for PHP here. Prepare a config file, which contains your keys:

<?php
return array(
    'includes' => array('_aws'),
    'services' => array(
        'default_settings' => array(
            'params' => array(
                'key'    => 'YOUR_KEY',
                'secret' => 'YOUR_SECRET_KEY',
                'region' => 'YOUR_REGION',
                'signature'    => 'v4'
            )
        )
    )
);

Note: Fill in your key, secret key and the region. I have to use v4 of the signature, because the bucket I am using is situated in Frankfurt (eu-central-1′). If you use the v2 of the authorization mechanism, just delete the line containing the signature.

The following is the files which contains the logic:

<?php

require 'aws/aws-autoloader.php';
use Aws\Common\Aws;

$aws = Aws::factory('config.php');
$client = $aws->get('s3');

$addObjects = array();

$iterator = $client->getIterator('ListObjects', array('Bucket' => 'YOUR_BUCKET_NAME'));
foreach ($iterator as $object) {
    $path = $object['Key'];

    // only enter the condition, if the path does not end with a slash
    if (substr($path, -1) != "/") {
        // iterate as long as there is a slash in the path
        while (strpos($path, "/") !== FALSE ) {
            $path = substr($path, 0, strrpos($path, "/"));
            if (!in_array($path . "/", $addObjects)) {
                array_push($addObjects, $path . "/");
            }
        }
    }
}

foreach ($addObjects as $object) {
	$params = array(
		'ACL' => 'bucket-owner-full-control',
		'Body' => "",
		'Bucket' => 'YOUR_BUCKET_NAME',
		'Key' => $object
	);
	$client->putObject($params);
}

Be sure to require the autoloader of AWS. We use the factory method of the Aws class and the config file we wrote. The first loop iterates over the object which are available in the bucket. It reads the path of the object and adds all folders, which are a part of the path to the array “$addObjects”. The second loop iterates over the folder paths we just collected and adds objects for them to the bucket. Note: Replace the two occasions of “YOUR_BUCKET_NAME” with the name of the bucket you are using.

 

Ubuntu 14.04 apache webserver installation – hosting Typo3, Joomla or WordPress

1 Reply

This article is a guideline on how to install and configure an apache webserver, which is ready to  host content management systems like Typo3, Joomla, WordPress or other PHP applications. It also give hints for the installation on an Amazon EC2 instance. I used these instructions for a setup on Amazon EC2.

Starting from a fresh ubuntu installation, install LAMP (Linux – Apache – MySql – PHP).

tasksel install lamp-server

The dialog for installation will open, enter the required information (MySql root user password).

Open the hostname file with the following command and enter the public hostname of the server. In my case it is the hostname of the Amazon EC2 instance.

vim /etc/hostname

Installing PHP modules for the webserver

I have a list of PHP modules, which I frequently require in order to host certain applications and CMS systems. Install them with the following command:

apt-get install imagemagick curl libcurl3 libcurl3-dev php5-curl php5-mcrypt php5-gd php5-json php5-dev php5-xsl

After the installation of the modules I usually change some settings of the php.ini, the settings I change are the following:

  • upload_max_filesize = 100M
  • post_max_size = 100M
  • memory_limit = 128M

You can change the php.ini file using the following command:

vim /etc/php5/apache2/php.ini

It’s not necessary to change the limits like that – if you have other requirements, they won’t affect the installation in any way. The last step is to reload Apache:

/etc/init.d/apache2 reload

Installing FTP services for the webserver

Usually I use FTP services for upload (and to connect IDEs via FTP for easier deployment), so we also install a FTP server, in this case the service is called vsftpd. I can be installed using the following command:

apt-get install vsftpd

After that we alter the configuration file of vsftpd in order to allow ubuntu systems users to connect via FTP using their passwords and to upload/download files via FTP. Change the configuration file using the following command:

vim /etc/vsftpd.conf

Change the following three values:

  • local_enable=YES
  • write_enable=YES
  • local_umask=022

After changing and saving the file, restart the service using the following command:

service vsftpd restart

In case you are using an Amazon EC2 instance like me, make sure to add the ports 20 and 21 as inbound to the security group of your EC2 instance. Also, you have to set the value in the vsftpd configuration file “pasv_enable=YES” and add the following lines:

pasv_enable=YES
pasv_min_port=64000
pasv_max_port=64321
port_enable=YES
pasv_address=<your-publicly-resolvable-host-name>
pasv_addr_resolve=YES

Reload the service vsftpd again. Add the port range 64000-64321 to the EC2 inbound rules. The workflow is also explained on stackoverflow here.

Installing sendmail for email sending on the webserver

In order to send emails from the server (which is mainly used by the content management systems) install sendmail using the following command:

apt-get install sendmail

In order to conduct a test, to see if the email sending is working, install the following package:

apt-get install bsd-mailx

Now you can try to send an email from command line using the following:

mail -s "example subject" email@address.eu

Replace your own email address with “email@address.eu”. After writing this command, an empty line will show in the command line – here you have to write the content of your test email. After clicking one more type Return, write a single dot (“.”) to mark the end of the email. You have the option to add a CC email address, after that the email should be send.

Installing webmin and virtualmin for managing the webserver

In order to administrate the server and the virtual hosts on the server via graphic interface in the browser, I am using webmin and virtualmin. Before starting to install webmin, install some required packages using this command:

aptitude -y install perl libnet-ssleay-perl openssl libauthen-pam-perl libpam-runtime libio-pty-perl apt-show-versions libapt-pkg-perl

Visit the homepage of webmin here and check for the latest version of webmin, in my case it was the version 1.710. Download and install webmin using the following two commands:

wget http://downloads.sourceforge.net/project/webadmin/webmin/1.710/webmin_1.710_all.deb
dpkg -i webmin_1.710_all.deb

After, visit the homepage of virtualmin to check for the latest version, in my case it’s 4.12. Download and install like this:

wget http://download.webmin.com/download/virtualmin/webmin-virtual-server_4.12.gpl_all.deb
dpkg -i webmin-virtual-server_4.12.gpl_all.deb

After installation, remove the two .deb files remaining, they are not required anymore.

Installing apache2-mpm-itk for webmin and virtualmin

After a basic installation, the document root of the webserver is usually residing in /var/ww. This option has some drawbacks, for example that users should just have access to their /home directory and that files in /var/www should be owned by www-data. Virtualmin stores the website data in the home directory and adds an user for every domain. This is a good solution, futhermore you can use vsftpd to limit every user to just their home directory. apache2-mpm-itk takes care, that the files in the home directory can still be owned by the user and apache won’t have any problems with this. Install the module and reload apache:

apt-get install apache2-mpm-itk
service apache2 restart

After the installation, you can usually access webmin using the following address: https://YOUR-DOMAIN:10000. Since I am using an Amazon EC2 instance, I also had to take care of the following things:

  • You have to add a rule to allow access on port 10000. In order to add it, go to the Amazon AWS management console and open the EC2 instances. Scroll to the right and click on the Security Group of the EC2 instance, you want to grant access on port 10000 on. Click the “inbound rules” tab, hit “Edit” and click on “Add rule”. Choose “Custom TCP rule”, choose “TCP” protocol, enter the port “10000”, select for source “Anywhere” and save. Now it should be possible to open the webmin link at https://YOUR-DOMAIN:10000.
  • Another thing is that webmin requires an user and a password to login. When using Amazon EC2 instances, the root user should have a key, not a pass. This means, that we have to add an user for webmin. Use the following command to add the user: 
    adduser USERNAME

    Open the following file

    vim /etc/webmin/miniserv.users

    add the following line

    USERNAME:x:0

    and delete the line for the root user. After, open the following file:

    vim /etc/webmin/webmin.acl

    Replace “root” with the username you just added. Restart the webmin service using the following command:

    service webmin restart

    After refreshing the address https://YOUR-DOMAIN:10000, you should now be able to log in.

Configuration Webmin and Virtualmin

After logging in to webmin, go to Webmin – Webmin Configuration and open “Webmin Themes”. Change the Theme to “Blue Framed Theme”. This is the theme working best for me while using virtualmin. Go to “Servers” and click von “Virtualmin Virtual Servers” and start the Post-Installation Wizard. After you made the choices here, click on “Re-check and refresh configuration”. Some problems, depending on your exact configuration might appear, here are the points which I had to change:

  • Deactivate BIND DNS, because I just use the server for hosting the sites, the domains are managed elsewhere and just point to this server.
  • Add virtuser table to sendmail. Open the sendmail menu, click on “Sendmail M4 Configuration” and click “Edit file manually”. Between the lines 
    dnl # Default Mailer setup

    and

    MAILER_DEFINITIONS

    paste the following:

    dnl # Masquerading options
FEATURE(`always_add_domain')dnl
MASQUERADE_AS(`dev-carrenoir.eu')dnl
FEATURE(`allmasquerade')dnl
FEATURE(`masquerade_envelope')dnl
FEATURE(`virtusertable',`hash -o /etc/mail/virtusertable')

    Save the file and click the button “Rebuild Sendmail Configuration” and “Yes, replace it now” after.

  • Virtualmin needs to have the modules “mod_suexec” and “mod_actions” enabled. Go to “Servers” -> “Apache Webserver” -> “Global configuration” -> “Configure Apache Modules”. Check “suexec” and “actions” and click “Enable Selected Modules”.
  • Virtualmin wants us to remove “SetHandler” lines from the php5 configuration file in “/etc/apache2/mods-enabled/php5.conf”. I don’t like this option a lot – but it is not a big problem. We can add the SetHandler instructions in the template for every virtual host which is going to be created. Edit the php5 configuration file using the following command: 
    vim /etc/apache2/mods-enabled/php5.conf

    and comment the two lines starting with “SetHandler”. Reload the webserver after

    service apache2 reload
  • If the suexec command is not found in the system, install the following additional package 
    apt-get install apache2-suexec-custom

    The following file

    vim /etc/apache2/suexec/www-data

    has the first line “/var/www”. Replace this with “/home”, save and close the file. Reload apache

    service apache2 reload
  • If webalizer is not installed yet, install it using the command 
    apt-get install webalizer

After all problems are removed, Virtualmin should load. Now we have to add some configurations to the template, which is used to create new virtual hosts. Open “Servers” -> “Virtualmin Virtual Servers (GPL)” and open “Server Templates”, choose “Default Settings”. From the dropdown, select “Administration user”. For the point “Add domain owners to secondary group” choose “Selected group” and pick “www-data”. Pick “Apache website” from the same dropdown and paste the following to the end of “Directives and settings for new websites”:

<IfModule mpm_itk_module>
AssignUserId ${USER} ${USER}
</IfModule>
<FilesMatch ".+\.ph(p[345]?|t|tml)$">
    SetHandler application/x-httpd-php
</FilesMatch>
<FilesMatch ".+\.phps$">
    SetHandler application/x-httpd-php-source
    # Deny access to raw php sources by default
    # To re-enable it's recommended to enable access to the files
    # only in specific virtual host or directory
    Order Deny,Allow
    Deny from all
</FilesMatch>

Now you should be all set to add the first virtual server.