Amazon S3 NoSuchKey on folder level – folders don’t exist

While working with Amazon S3 buckets and connecting them to Typo3 I stumbled into a problem: When I copy the contents to the bucket from another bucket or via tools such as BucketExplorer, the XML output (on bucketname.s3.amazonaws.com) showed all files, but not the folders. As an example:

<Contents>
  <Key>folder_one/file_one</Key>
  <LastModified>2014-11-18T14:16:19.000Z</LastModified>
  <ETag>"6e563e7cf3fda1549cb13aeae7a53b24"</ETag>
  <Size>6148</Size>
  <StorageClass>STANDARD</StorageClass>
</Contents>
<Contents>
  <Key>folder_one/folder_two/file_two</Key>
  <LastModified>2014-11-18T14:16:19.000Z</LastModified>
  <ETag>"6e563e7cf3fda1549cb13aeae7a53b24"</ETag>
  <Size>6148</Size>
  <StorageClass>STANDARD</StorageClass>
</Contents>

The problem here is that I supposedly also require entries for the folders themselves, in the example that would be “folder_one/” and “folder_one/folder_two/”. If I don’t have these entries, connected systems like Typo3 won’t recognize, that there are folders. If the entries are missing for the folders on the root level, I even get an exception like this:

HTTP/1.1 404 Not Found x-amz-request-id: XXX x-amz-id-2: XXX Content-Type: application/xml Transfer-Encoding: chunked Date: Tue, 18 Nov 2014 09:10:16 GMT Server: AmazonS3 NoSuchKeyThe specified key does not exist

This is confusing, because when viewing the bucket in the Amazon web interface, it shows the folders while exploring.

After some research I found out that everything in the Amazon S3 buckets is treated as an “object”, files as well es folders. I assume that while uploading or copying to a bucket, the tool being used just copies the objects for the files, but not the objects for the folders. In order to fix this, I wrote a short script in PHP which goes through all objects and adds the missing objects for folders. As an example: Imagine you have an empty bucket and just one file with the object “/folder_one/folder_two/file”. The script will see this path and add the two objects “/folder_one/” and “/folder_one/folder_two/”.

In order to use the script, download the Amazon AWS SDK for PHP here. Prepare a config file, which contains your keys:

<?php
return array(
    'includes' => array('_aws'),
    'services' => array(
        'default_settings' => array(
            'params' => array(
                'key'    => 'YOUR_KEY',
                'secret' => 'YOUR_SECRET_KEY',
                'region' => 'YOUR_REGION',
                'signature'    => 'v4'
            )
        )
    )
);

Note: Fill in your key, secret key and the region. I have to use v4 of the signature, because the bucket I am using is situated in Frankfurt (eu-central-1′). If you use the v2 of the authorization mechanism, just delete the line containing the signature.

The following is the files which contains the logic:

<?php

require 'aws/aws-autoloader.php';
use Aws\Common\Aws;

$aws = Aws::factory('config.php');
$client = $aws->get('s3');

$addObjects = array();

$iterator = $client->getIterator('ListObjects', array('Bucket' => 'YOUR_BUCKET_NAME'));
foreach ($iterator as $object) {
    $path = $object['Key'];

    // only enter the condition, if the path does not end with a slash
    if (substr($path, -1) != "/") {
        // iterate as long as there is a slash in the path
        while (strpos($path, "/") !== FALSE ) {
            $path = substr($path, 0, strrpos($path, "/"));
            if (!in_array($path . "/", $addObjects)) {
                array_push($addObjects, $path . "/");
            }
        }
    }
}

foreach ($addObjects as $object) {
	$params = array(
		'ACL' => 'bucket-owner-full-control',
		'Body' => "",
		'Bucket' => 'YOUR_BUCKET_NAME',
		'Key' => $object
	);
	$client->putObject($params);
}

Be sure to require the autoloader of AWS. We use the factory method of the Aws class and the config file we wrote. The first loop iterates over the object which are available in the bucket. It reads the path of the object and adds all folders, which are a part of the path to the array “$addObjects”. The second loop iterates over the folder paths we just collected and adds objects for them to the bucket. Note: Replace the two occasions of “YOUR_BUCKET_NAME” with the name of the bucket you are using.

 

2 thoughts on “Amazon S3 NoSuchKey on folder level – folders don’t exist

  1. Pingback: AWS S3 Bucket MP4 Videos wrong mime-type - meberhard

  2. Pingback: AWS and Typo3: Hosting, Deployment and Cont. Integration

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.