pshell and data workflows

Overview

Mediaflux intro
Import data with metadata
Query data based on metadata searches

Material

Mediaflux (ref) is the underlying storage platform that pshell communicates with. It is out of scope to discuss the full capabilities of Mediaflux, but, a few relevant items will be mentioned.

Mediaflux is a database on a filesystem (or filesystems) that can be queried in a somewhat comparable manner as SQL. The Mediaflux database is XML based so the arguments and search terms are couched in the language of XML.

Every file stored in a Mediaflux server gets transformed into an asset and has a unique ID reference - this is the ID reported by pshell in the previous introductory section.

An asset consists of metadata and a reference to where the actual file content is stored ie the file-system path. Both the metadata and file content can be versioned and previous version retrieved. The default behaviour if no particular version is specified is to use the most recent version. Multiple versions of file content impact your usage/quota.

namespaces are the Mediaflux name for the virtual folder structure that was introduced in the previous section.

metadata format - this tells you what queries can be done

The basic format is:

Command :argument1 value1 :argument2 value2 etc

asset.get :id ASSET-ID
asset.get :id "path=FULL-PATH-TO-FILE"

Metadata templates

The doc type shows the metadata that can be queried and specified.

Custom doc types can be made.

Asset is a special "first-order" type attached to everything.

Custom doc types get added under the <meta> element.

asset.doc.type.describe :type asset

Populating first class metadata

This is equivalent to a rename:

asset.set :id ASSET-ID :name "new filename"

A more complex example, where we assign a geospatial location (in this case a point) to an asset:

asset.set :id 1234 :geoshape < :point < :latitude -31.95 :longitude 115.86 :elevation 10.0 > >

This last case is an example of specifying an XML document that details the xpath + value for metadata items.

It is equivalent to an XML metadata document that looks like this:

<geoshape>
	<point>
		<latitude> -31.95 </latitude>
		<longitude> 115.86 </longitude>
		<elevation> 10.0 </elevation>
	</point>
</geoshape>

If we wanted to extract one of those pieces of information we could do:

asset.get :id 1234 :xpath geoshape/point/elevation


value = 10.0

Some arguments to mediaflux require an XML document as input. A common use case might be: populating a metadata template on an asset.

Populating custom metadata

asset.set :id ASSET-ID :meta < :mf-note "This is my note >

asset.query :where "LOGICAL-EXPRESSION"


asset.query :where "namespace='/projects/Demo'"
asset.query :where "namespace>='/projects/Demo' and name='*.jpg'"
asset.query :where "namespace>='/projects/Data Team/sean' and (name='*.JPG' or name='*.PNG')"

Simple actions on the results

asset.query :where "namespace>='/projects/Data Team/sean' and name='*.jpg'" :action get-value :xpath -ename name name

Exercises

Exercise 1 - upload a file and then inspect the metadata in the system.

Solution to exercise 1

put IMG_0222.PNG

asset.get :id "path=/projects/Data Team/sean/IMG_0222.PNG"

asset = None { version=4 id=1377760 vid=74319092 }

type = image/png

namespace = /projects/Data Team/sean

path = /projects/Data Team/sean/IMG_0222.PNG

name = IMG_0222.PNG

meta = None { stime=74319092 }

etc

Exercise - querying metadata - geospatial search? TODO - import some geoshape attached

Exercise - adding metadata - import and/or asset.set?

Exercise - doing things with queries (ie pipes) - eg move all .jpg's to a folder