Stream Wrappers (2)

• All wrappers can be used in whole-file operations we looked at earlierreadfile('http://phpne.org.uk/about')
• Custom wrappers can be created

• Whole-file operations are just wrappers for collections of stream operations
• A custom wrapper can be very powerful
• A custom wrapper will contain an entire protocol for an abstration to access remote resources

Custom Stream Wrappers

Register Your Stream Wrapper

// Register the class
stream_wrapper_register('mywrapper', 'MyStreamProtocol');

// Use the wrapper
$fp = fopen('mywrapper://some/path/to/a/data/source', 'r');
readfile(fp);
fclose($fp);
						

• The stream wrapper can be namespaced
• Not sure if autoloaded, but I assume so
• Examples are Google cloud storage, where the "google" wrapper is created

Custom Stream Wrappers

Define Your Stream Wrapper (1)

class MyStreamProtocol {
    // Open a stream
    function stream_open($path, $mode, $options, &$opened_path) {...};

    function stream_close() {...};
    function stream_read() {...};
    function stream_write() {...};
    function stream_eof() {...};
    function stream_tell() {...};
    function stream_seek() {...};
    ...
}
						

• When opening a stream, the path will be a URL, so use parse_url() to parse it.
• See http://www.master.iag.usp.br/manuais/php/br/function.stream-wrapper-register.html
• Documentation here: http://uk3.php.net/manual/en/class.streamwrapper.php

Custom Stream Wrappers

Define Your Stream Wrapper (2)

class MyStreamProtocol {
    ...
    // File-based operations for use with touch(), unlink(), rename(), stat()
    function unlink() {...};
    function rename() {...};
    function rmdir() {...};
    function url_stat() {...};
    ...
}
						

• There is no interface for the wrapper class.
• That is because most of the methods are optional: implement what is relevant.

Custom Stream Wrappers

Define Your Stream Wrapper (3)

class MyStreamProtocol {
    ...
    // Directory-based operations for use with mkdir(), rmdir(), opendir()
    function mkdir() {...};
    function rmdir() {...};
    function dir_opendir() {...};
    function dir_readdir() {...};
    function dir_closedir() {...};
}
						

• Interesting example: http://www.tuxradar.com/practicalphp/15/11/1
• The example lets you write text to the stream (stored in a variable, so a memory-based stream)
• then when you read the stream the text is converted to morese code.
• However, a filter may be a better way to handle that.

Local Files

To access the local filesystem:

// The filesystem class and the adapter.
use League\Flysystem\Filesystem;
use League\Flysystem\Adapter\Local as Adapter;

// Create a local file filesystem object.
$filesystem = new Filesystem(new Adapter('path/to/root'));

// Get the content of a file.
$file_content = $filesystem->read('sub_dir/myfile.txt');

• The filesystem object provides access to files through the adapter
• The local file adapter needs a root directory to build any further directories on

FTP

Other adapters can be dropped into the constructor:

use League\Flysystem\Filesystem;
use League\Flysystem\Adapter\Ftp as Adapter;

$filesystem = new Filesystem(new Adapter(array(
    'host' => 'ftp.example.com',
    'username' => 'username',
    'password' => 'password',
)));

• The FTP adapter uses the standard PHP FTP functions to perform its tasks

Dropbox

Some adapters need external libraries to be included:

use Dropbox\Client;
use League\Flysystem\Filesystem;
use League\Flysystem\Adapter\Dropbox as Adapter;

$client = new Client($token, $appName);
$filesystem = new Filesystem(new Adapter($client, 'optional/path/prefix'));

The Dropbox library in turn needs OAuth and its libraries.

It is all nicely handled by DI.

• The DI can be handled manually in small projects, or a container used to ease use in frameworks.

Operations (1)

Writing complete files:

$fs = new Filesystem($adapter);

// Create a new file (including directories) and set permissions
$fs->write($filename, $content)

// Overwrite the contents of an existing file
$fs->update($filename, $content)

// Write or put, depending if file exists
$fs->put($filename, $content)

// Apend to a file?

• Non-streaming - whole files are written from a string
• write will raise an exception if the file alreadt exists
• update will raise an exception if teh file does not exist
• probably safer to use put() if you are unsure
• Does not appear to be a way to append at this time

Operations (2)

Reading complete files:

$fs = new Filesystem($adapter);

// Read the complete contents of a file as a string
$fs->read($filename)

// Tells us if the file exists
$fs->has($filename)

Operations (3)

File operations:

$fs = new Filesystem($adapter);

// Delete a file
$fs->delete($filename)

// Rename or move a file
$fs->rename($filename)

// Copy a file?

• There is not operation to copy or duplicate a file, but there is an elegant way detailed later

Operations (4)

Directory operations:

$fs = new Filesystem($adapter);

// Create a directory (to any number of levels)
$fs->createDir($path)

// Remove a directory
$fs->deleteDir($path)

// Move a directory? Same as renaming a file
$fs->rename($path)

• Not all adapters support moving of directories, so be careful and test that

File Visibility (1)

The visibility of a file: can it be seen by others or just the current user?

// Two states allowed
AdapterInterface::VISIBILITY_PRIVATE // 'private'
AdapterInterface::VISIBILITY_PUBLIC  // 'public'

• The visibility is very high level - may not be enough for some use-cases

File Visibility (2)

The visibility can be:

• set globally, for all new files to take on
• set for each file creating operation (write)
• applied to existing files

// New files
$fs->write($filename, 'contents', ['visibility' => $visibility]

// Existing files
$fs->getVisibility($filename)
$fs->setVisibility($filename, $visibility)

// Or set globally when instantiating the Filesystem
$fs = new League\Flysystem\Filesystem($adapter, $cache, [
    'visibility' => AdapterInterface::VISIBILITY_PRIVATE
]);

• It would be nice to see some use-cases on how the visibility choices arose.
• Would also be nice to be able to extend the visibility to cater for other needs.

File Visibility (3)

Implementation example, for local files

• public: 0644 (owner=rw, group=r, other=r)
• private: 0600 (owner=rw)

• Other adapters will use other methods
• The broad-brush approach is probably just what is common between the adapters

Listing directories (1)

// List the contents of a directory, recursively if required
$fs->listContents($path [, bool $recurse])

• Returns an array of arrays
• Each item is a "file" or a "dir"
• A basename (e.g. myfile.txt) and a path for each item
• Any other metadata requested and cached for this item
• Any metadata the adapter wants to return

• An array of arrays is items and attributes of each
• Would probably be more useful to return objects
• The path for each item is the full path to an item, which includes filenames (probably a misnomer)

Listing directories (2)

• Custom metadata is very useful
• $fs->listPaths($path [, bool $recurse]) Return just a list of paths
• $fs->listWith($keys, $path [, bool $recurse]) Make sure metadata is requested (fields listed in $keys)

• Custom metadata: e.g. owner, group and full file permissions, public URLs (for Flickr pics)
• Some metadata comes for free when listing a directory, and will always be included by the adapter
• Some metadata needs an additional call to the remote site, and so won't be fetched unless asked for

Flysystem and Streams (1)

Use streams instead of strings to read and write files

// Create a file from a stream
$fs->writeStream($filename, $stream)

// Update a file from a stream
$fs->updateStream($filename, $stream)

// Return a file as a stream
resource $fs->readStream($filename)

• If the remote storage does not support direct streams, then Flysystem will read a whole remote file into a temporary stream and return that

Flysystem and Streams (2)

There is no copy operation in Flysystem; few if any remote storages support it.

Instead, put two streams back-to-back:

// Copy one file to a new file
$fs->writeStream(
    'destination.txt',
    $fs->readStream('source.txt')
);

• This will still stream the source file to the local server, then back out to the remote server.
• The files could be copied between two separate filesystems this way

Flysystem and Streams (3)

So far no support for returning a stream to an open file for writing.

Some remote storage would support this, and some would not.

// This would be useful (but not implemented):
$stream = $fs->openWriteStream('destination.txt', 'a');

fputs($stream, 'foo');

fclose($stream);

• This will still stream the source file to the local server, then back out to the remote server.
• The files could be copied between two separate filesystems this way

Alternative Abstractions (1)

• php-resource offers a comprehensive OPP wrapper for resources.
• shesek offers a simpler OOP wrapper for resources.
• Illuminate\Filesystem\Filesystem used by Laravel. Not adapter-based, so it only supports local filesystem operations.

• For Laravel, it is simple to swap it out through its facade, though that is essentially setting a single, global filestore for the whole application.

Alternative Abstractions (2)

• KnpLabs/Gaufrette takes a similar approach, and has been around for a couple of years.
• discordier/php-filesystem tries to get to some raw file access methods, though seems to have stopped development before many adapters were written. Implements filesystem iterators, which many other libraries do not.
• zikula has abstracted the filesystem and is worth looking at for its coverage of operations.

Enhancements - Adapters

Some interesting adapters

• BBC iPlayer
• Flickr (finish off)
• PDO/MySQL
• IMAP
• Google Cloud Storage
• Test adapter skeleton

• The test adapter could be set with rules on how to respond to requests
• iPlayer includes channels, categories, thumbnails, link to view page
• IMAP - emails are just text files anyway; the adapter could also pull attachments out into virtual folders

Enhancements - Features

• Support for paged files (using streams)
• Writeable streams
• Directory listings (and files) as objects
• Directory listings as streams
• find() operation
• Extend permissions
• copy() operation
• Flyception - Flyserver apdapter that uses a custom stream wrapper that calls the Flyserver adapter for its functionality

• Paged files would include data that can only be retrived in pages, e.g. entries in a blog
• Writeable streams would be supported by only some adapters and storage destinations
• Directories as objects offer better handling of missing metadata
• find() could search for matching files using any criteria - file globbing, metadata match.
• The permissions needs more use-cases, so this is just a hunch at this stage.
• One permissions case: read-only files, generated virtually by the adapter
• The copy() operation could make use of the ability for a remote storage to duplicate files where it can, otherwise would just stream back-to-back.

Summary

• PHP file handling is flexible and powerful
• PHP provides some abstraction already
• We like to abstract files, directories and streams
• Flysystem brings file abstraction into the OOP world
• Flysystem makes it easier to swap remote storage in and out
• Flysystem is still young, and will develop further

Conclusions

• We can no longer assume files will just be written to the local filesystem
• An OO wrapper for file access will makes switching storage destinations easier
• An agreed interface would be good for portability (PSR?)
• Streams and stream wrappers are underused and underappreciated
• Streams are still a core part of file handling, and will be for a long time
• Flysystem is gaining momentum and is worth checking out...
• ...but be aware it is still finding its feet

Demo (1)

• An adapter to access some of Flickr as an abstract filesystem
• Uses OAuth to log into Flickr
• Directory structure is emulated, as are the files
• Metadata includes various Flickr properties of the images
• Demo does not write images at this time, but it would be simple

Demo (2)

The demo architecture

Demo (3)

The demo fetches the contents of a "path" on Flickr

// $app['flickrapi'] is a DI container for the Flickr API
// FlyAdapter is a custom adapter that adstracts Flickr
$filesystem = new Filesystem(
    new Academe\Flickr\FlyAdapter($app['flickrapi'])
);

// Get the path from the user.
$path = $request->get('path', '');

// Get the items on Flickr at that path
$items = $filesystem->listContents($flickr_path);

Demo (4)

If a file is selected, then fetch it for display or download

// Filename selected by user
$file = $request->get('file', '');

if (!empty($selected_file)) {
    // Get metadata for the file.
    $file_metadata = $filesystem->getWithMetadata("$path/$file", array());

    // Get the content of the file.
    $file_content = $filesystem->read("$path/$file", array());

    // The file_content can be streamed, or displayed inline
    // depending on its mimetype
}

• The point if this is to show how simple the code for the application is - it knows nothing about where the files come from.

Demo (5)

The virtual directory structure:

/
|--contacts
|  |--friends
|  |--family
|  |--both
|  |--neither
|     |--metadata.csv
|     |--user1
|     |--user2
|        |--photostream
|           |--image1_small.jpg
|           |--image1_largesquare.jpg
|           |--image2_small.jpg
|           |--image2_largesquare.jpg
|           |--...
|        |--sets
|        |--non-sets
|     |--userX
|--me

Link to the demo (will need a Yahoo/Flickr login)

• Note metadata.csv - it summarises the files in the directory as a CSV file
• Any number of additional sizes could be included.