Versions Compared

Version Old Version 24 New Version 25
Changes made by

Sean Fleming

Sean Fleming

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Overview

  • Mediaflux intro
  • Import data with metadata
  • Query data based on metadata searches 

...

Mediaflux

Mediaflux (ref) is the underlying storage platform that pshell communicates with. It is out of scope to discuss the full capabilities of Mediaflux, but, a few relevant items will be mentioned.
Mediaflux is a database on a filesystem (or filesystems) that can be queried in a somewhat comparable manner as SQL. The Mediaflux database is XML based so the arguments and search terms are couched in the language of XML.
Every file stored in a Mediaflux server gets transformed into an asset and has a unique ID reference - this is the ID reported by pshell in the previous introductory section.
An asset consists of metadata and a reference to where the actual file content is stored ie the file-system path. Both the metadata and file content can be versioned and previous version retrieved. The default behaviour if no particular version is specified is to use the most recent version. Multiple versions of file content impact your usage/quota.
namespaces are the Mediaflux name for the virtual folder structure that was introduced in the previous section.
metadata format - this tells you what queries can be done
Instead of files and folders (pshell) you have:
  • assets - file and associated metadata
  • namespaces - remote folder structure on the mediaflux server
Mediaflux metadata is:
  • XML-based 
  • has it's own query language 
The two main commands, operating on assets and asset metadata are:
  • asset.get - display metadata for an asset
  • asset.query - find assets based on metadata queries
The basic format is:

Command :argument1 value1 :argument2 value2 etc

Code Block
asset.get :id ASSET-ID
asset.get :id "path=FULL-PATH-TO-FILE"


Metadata templates

The doc type shows the metadata that can be queried and specified.

Custom doc types can be made.

Asset is a special "first-order" type attached to everything.

Custom doc types get added under the <meta> element.

Code Block
asset.doc.type.describe :type asset
Populating first class metadata
This is equivalent to a rename:


Code Block
asset.set :id ASSET-ID :name "new filename"


A more complex example, where we assign a geospatial location (in this case a point) to an asset:

Code Block
asset.set :id 1234 :geoshape < :point < :latitude -31.95 :longitude 115.86 :elevation 10.0 > >


This last case is an example of specifying an XML document that details the xpath + value for metadata items.

It is equivalent to an XML metadata document that looks like this:

Code Block
<geoshape>
	<point>
		<latitude> -31.95 </latitude>
		<longitude> 115.86 </longitude>
		<elevation> 10.0 </elevation>
	</point>
</geoshape>


If we wanted to extract one of those pieces of information we could do:

Code Block
asset.get :id 1234 :xpath geoshape/point/elevation

value = 10.0


Some arguments to mediaflux require an XML document as input. A common use case might be: populating a metadata template on an asset.

Populating custom metadata

asset.set :id ASSET-ID :meta < :mf-note "This is my note >

Code Block
asset.query :where "LOGICAL-EXPRESSION"


asset.query :where "namespace='/projects/Demo'"
asset.query :where "namespace>='/projects/Demo' and name='*.jpg'"
asset.query :where "namespace>='/projects/Data Team/sean' and (name='*.JPG' or name='*.PNG')"


Simple actions on the results

Code Block
asset.query :where "namespace>='/projects/Data Team/sean' and name='*.jpg'" :action get-value :xpath -ename name name


Exercises

Exercise 1 - upload a file and then inspect the metadata in the system.

Expand
titleSolution to exercise 1

put IMG_0222.PNG

asset.get :id "path=/projects/Data Team/sean/IMG_0222.PNG"


asset = None    { version=4 id=1377760 vid=74319092 }

    type = image/png

    namespace = /projects/Data Team/sean

    path = /projects/Data Team/sean/IMG_0222.PNG

    name = IMG_0222.PNG

    meta = None    { stime=74319092 }

etc


Exercise - querying metadata - geospatial search? TODO - import some geoshape attached 





...