pshell and Mediaflux

Overview

The pshell client can be used to send native service calls to the underlying mediaflux server.

This permits more powerful features to be exploited, at the cost of more complexity.

The first of these is that you have to give up the familiar idea of files and folders and instead use the concept of assets and namespaces.

An asset is metadata plus content (content is the file) and namespaces are a virtual hierarchy of assets.

Authentication and Delegation

If you are doing a lot of scripting connecting to a Mediaflux remote, it is undesirable to populate lots of plain text files with your login details. This is where delegation becomes useful. Delegate credentials are binary tokens that reside in the home directory of the user that creates them. These credentials automatically perform a login on your behalf when interacting with Pawsey data storage. Here's an example of creating a delegate credential that lasts for approximately one month:

pawsey:/projects>delegate 30
Delegating until: 01-Jun-2016 14:13:15
 
pawsey:/projects>exit

From now on, until the delegate expires, there is no need to login to data storage on the machine where this was run. Any pshell sessions that are run will automatically use the delegate to login. 

Finally, to destroy all delegate credentials, use:

pawsey:/projects>delegate off
 Delegate credentials removed.

Asset metadata

Standard metadata

Mediaflux automatically extracts metadata from files when they are uploaded to the server and become an asset.

The recommended approach is to pick a file format that Mediaflux natively supports in terms of content analysis/metadata extraction.

From http://www.arcitecta.com/Products/DataTypes

To retrieve the default metadata for an asset, the following are equivalent:

asset.get :id "path=/projects/DMF-TEST/sample/10_2015-03-05_111059_DSC_0004.NEF"
asset.get :id 61080987

The structure (indenting) of the document displayed is important and indicates the XML path of the particular metadata item.

Most of the information (name, ctime, version, etc) is standard file metadata. Custom metadata (if any) appears under the meta element.

Custom metadata

Custom metadata can be used if the files to be ingested are not recognised by default, but requires a little more effort.

Suppose a custom metadata document template, called mytype, defines a string called location.

Here is how to set the metadata value of location to a value of "Perth" using pshell:

asset.set :id 25480 :meta < :mytype < :location "Perth" > >

Importing metadata

An automated mechanism of adding metadata to uploaded files may also be utilised.

This is invoked if uploads are performed using the command import instead of the usual put.

Import will look for pairs of files that are of the form filename and filename.meta and attempt to populate filename with metadata from filename.meta

For example, suppose we had two files:

myfile.jpg
myfile.jpg.meta

where myfile.jpg.meta contained the following:

[asset]
geoshape/point/latitude = "10.0"
geoshape/point/longitude = "20.0"
geoshape/point/elevation = "100.0"

[mytype]
name = "the name"
details = "some more information"

We can then import the file using pshell, eg:

import myfile.jpg

Then examine the metadata:

file myfile.jpg

Which will display all the metadata, including:

    geoshape = None    { type=point datum=WGS84 }
        point = None
            latitude = None    { dd=10 }
                deg = 10
                min = 0
                sec = 0
            longitude = None    { dd=20 }
                deg = 20
                min = 0
                sec = 0
            elevation = 100.0
...
    meta = None    { stime=60812 }
      {mytype} = None    { id=2 }
          name = "the name"
          details = "some more information"

which can be queried, as described below.

Simple asset queries

A useful mediaflux command is asset.query which performs a database search on asset metadata items, which are specified via their XML path

For example, to locate all files based on .PNG filename extension anywhere in the entire (visible) namespace hierarchy use:

asset.query :where "name='*.PNG'"

To restrict the search to look only in /projects/Data Team, use:

asset.query :where "namespace='/projects/Data Team' and name='*.PNG'"

To search in /projects/Data Team and all folders underneath it, use:

asset.query :where "namespace>='/projects/Data Team' and name='*.PNG'"

Searching for assets that were created after a certain date would be done as follows:

asset.query :where "namespace>='/projects/Data Team' and ctime>='01-Jan-2017'"

Similar searches could be done using a geometric bounding box if your data has geospatial information attached.

Advanced asset queries

Searching for custom metadata

Suppose your assets had some custom metadata attached to them that was described by the XML document mytype.

If one of the elements in mytype was location (which is of type string), then here is how to search for a particular value:

asset.query :where "xpath(mytype/location) = 'Perth' and namespace>='/projects/myproject'"

Getting more information from the results

By default, asset.query returns only the unique asset identifier (id)

To get the name of the asset as well, use:

asset.query :where "namespace>='/projects/Data Team' and name='*.PNG'" :action get-name

To get arbitrary pieces of metadata, for example the name and checksum, use:

asset.query :where "namespace>='/projects/Data Team' and name='*.PNG'" :action get-value :xpath name :xpath content/csum

By default, mediaflux will label all the metadata items you request as value=. This can be annoying when retrieving multiple items.

To associate a better name with the returned metadata item, use something like:

asset.query :where "namespace>='/projects/Data Team' and name='*.PNG'" :action get-value :xpath -ename checksum content/csum

The presence of the -ename checksum attribute tells mediaflux to return the result as checksum= instead of value=

Simple actions on the results

The query function also has the ability to use the results of a search in certain ways.

For example, to count all PNG files stored in the entire Data Team project or to sum their sizes you would run, respectively:

asset.query :where "namespace>='/projects/Data Team' and name='*.PNG'" :action count
asset.query :where "namespace>='/projects/Data Team' and name='*.PNG'" :action sum :xpath content/size

Here the action value (count or sum) determines what to do with the search result, and content/size is the XML location of the file-size that is stored in the metadata of every asset that matches the query.

Arbitrary actions on the results 

Mediaflux can also send the results of a query to another service call that takes asset identifiers as an input argument.

For example, suppose we wanted to move all PNG files to a new location /projects/Data Team/png-files we would use:

asset.query :where "namespace>='/projects/Data Team' and name='*.PNG'" :action pipe :service -name asset.move < :namespace /projects/Data Team/png-files >

Here our action is to pipe the results to another service call, in this case an asset.move, which is a service call that takes the required destination namespace argument in a shorthand XML document.

Naturally, similar constructions could be used to simply delete these files instead:

asset.query :where "namespace>='/projects/Data Team' and name='*.PNG'" :action pipe :service -name asset.destroy

Namespace manipulation

Recall that assets and namespaces are quite different objects in mediaflux. None of the above queries will help in managing namespaces.

To change the name of the sean namespace to sean2 use:

asset.namespace.rename :namespace /projects/Data Team/sean :name sean2

To move the namespace sean to a new parent location use:

asset.namespace.move :namespace /projects/Data Team/sean :to /projects/Data Team/admins

To delete a namespace entirely, use:

asset.namespace.destroy :namespace /projects/Data Team/sean