Now being an ex database programmer ('ex' of anything being of course the very worse and most dangerous type), and of course a storage curmudgeon, I have a passion for the topic of metadata and data. And being somebody having to deal with PBs of object data I naturally have some concerns and views here...
Now normally I agree with Dave on a lot of things - but I have to say I much prefer my scallops to be seared and served on black pudding nice and simply, letting the quality of the flavours shine.
That said I have to agree re his view of being able to segment metadata & object storage models into two areas - but do think there is a place (almost essential IMHO) for both models in the future storage.
We've seen this area tackled by a number of existing technologies re CAS and object stores (Caringo CFS gateway onto Castor object layer is good example) - but are only just starting to see the key new elements test these, namely vast scale (think EBs), geo-dispersal/distribution/replication, low cost.
I do also think it's worth exploring some of the possible types / layers of metadata, for me this breaks into :-
- System / Infrastructure metadata - the metadata mandated by the storage service subsystem for every application using the service and every object held within the storage service. System metadata is under the exclusive control of storage service subsystem, although can be referenced by applications & users. Examples such as object ID, creation data, security, hash/checksum, Storage service SLA attributes (resilience, performance etc) etc.
- Application metadata - This is the metadata associated with each object that is controlled and required by the application service(s) utilising the object. There may be multiple sets of application metadata for a single object, each only accessible by the approved application.
- Object metadata - context & descriptive attributes, object history, related objects, optional user extensible metadata
Some of the metadata relates to the application and infrastructure architecture (eg geo-location information re object distribution & replication) whilst some of the metadata are attribute fields used within the application itself.
Given the above, it should be clear that I certainly agree with an entry Dave made in his blog comments re :-
"interesting note on ownership to which I'd say that there has to be dual ownership, one from the system level (with immutable meta such as creation date, etc.) as well as mutable data (e.g. user generated meta). The meta db then needs to maintain and track 2 different levels. Policy can affect either, fwiw."So some thoughts about where to locate metadata as it relates to the object :-
- As referenced above, I believe 'System metadata' must always reside with the object as it is used by the storage service for mngt, manipulation and control of the object itself, and ensure it's resilience & availability.
- As has been an issue with file-systems for some time, there is always an issue with fragmentation of the underlying persistency layer with vast size differences between objects and metadata when they are tightly coupled
- As a result of needing to traverse the persistency layer to establish the metadata, there are performance issues associated with metadata embedded within the object layer - move the metadata to a record based system and performance & accessibility can increase dramatically
- For certain classes of use (eg web 2 etc) it's often the metadata that is accessed, utilised & manipulated several orders of magitude more often than the objects themselves, thus the above improvements in performance and accessibility of metadata (thin SQL query etc) make major differences
- Clearly if the metadata and objects are held separately the metadata can be delivered to applications without needing to send the objects, similarly the metadata can be distributed separately / in-advance of the object. Thus having major advantages for application scaling and geo-distribution.
- With the split of persistecy location / methods this also allows for security layers to be handled differently for the metadata and the object.
If you like, this could be seen as an argument between IaaS & Paas - and for sure the only sensible answer for a company right now is IaaS, PaaS exposes far too much of the logic, taxonomy, behaviours, trends and metadata layers to the PaaS provider than is healthy.
There is also an additional interest point re metadata - as we move from the System metadata into the Application & Object metadata, should we consider privacy and encryption of the metadata itself? (assuming that the objects will always be protected appropriately) I could see how this will be a requirement in some multi-tenancy environments an for some metadata elements...
Lastly some more questions :-
- How do you cover the topics of backup/recovery of the various metadata elements?
- How do you cope with bulk import / export of the various levels of metadata and their logical / context relationships?
- What standards will emerge for metadata schema definitions and attributes?
- What standards will emerge for policy script language & descriptors that manipulate within the storage systems? (think how to describe an SLA in a programmatic language)
- Can security authorisation & permission tokens exist and be enforced in a separate context and control domain to the identities?
Now I'm off for some lunch, I'm hungry and fancy some seafood for some reason ;)
Cheers
Ian
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_e.png?x-id=0dcfd239-9f3a-437a-9a65-bab5bd894630)
No comments:
Post a Comment