ISCC - Specification v1.x#

Last revised: 2022-10-10 16:43:08

Abstract#

The International Standard Content Code (ISCC), is an open and decentralized digital media identifier. An ISCC can be created from digital content and its basic metadata by anybody who follows the procedures of the ISCC specification or by using open source software that supports ISCC creation conforming to the ISCC specification.

Note to Readers#

For public discussion of issues for this specification, please use the Github issue tracker: https://github.com/iscc/iscc-specs/issues.

If you want to chat with developers, join us on Telegram at https://t.me/iscc_dev.

You can find the latest version of this specification at http://iscc.codes/specification/.

Public review, discussion and contributions are welcome.

About this Document#

This document proposes an open and vendor neutral ISCC standard and describes the technical procedures to create and manage ISCC identifiers. The first version of this document resulted from a prototyping project by the Content Blockchain Project and received funding from the Google Digital News Initiative (DNI). The content of this document results from a voluntary effort of the authors with an open and public consensus process.

Conventions and Terminology#

The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC 2119].

Definitions#

Basic Metadata:

Minimal set of ISCC-specific top-level metadata that applications SHOULD support.

Bound Metadata:

Metadata that is utilized during generation of the ISCC. A change of this bound metadata may therefore impact the derived ISCC..

Extended Metadata:

Industry and application-specific metadata attached to an ISCC.

Character:

Throughout this specification a character is meant to be interpreted as one Unicode code point. This also means that due to the structure of Unicode a character is not necessarily a full glyph but might be a combining accent or similar.

Digital Media Object:

A blob of raw bytes with some media type specific encoding.

Extended Metadata:

Metadata that is not encoded within the ISCC Meta-Code but may be supplied together with the ISCC.

Generic Media Type:

A basic content type such as plain text in a normalized and generic (UTF-8) encoding format.

ISCC:

International Standard Content Code

ISCC Code:

The printable text encoded representation of an ISCC

ISCC Digest:

The raw binary data of an ISCC

Introduction#

An ISCC permanently identifies content at multiple levels of granularity. It is algorithmically generated from basic metadata and the contents of a digital media object. It is designed for being registered and stored on a public and decentralized blockchain. An ISCC for a media object can be created and registered by the content author, a publisher, a service provider or anybody else. By itself the ISCC and its basic registration on a blockchain does not make any statement or claim about authorship or ownership of the identified content.

ISCC Structure#

A Fully Qualified ISCC Digest is a fixed size sequence of 36 bytes (288 bits) assembled from multiple sub-components. The Fully Qualified ISCC Code is a 52 character encoded printable string representation of a complete ISCC Digest. This is a high-level overview of the ISCC creation process:

ISCC Components#

The ISCC Digest is built from multiple self-describing 72-bit components:

ISCC components MAY be used separately or in combination by applications for various purposes. Individual components MUST be presented as 13-character base58-iscc encoded strings to end users and MAY be prefixed with their component name.

Combinations of components MUST include the Meta-Code component and MUST be ordered as Meta-Code, Content-Code, Data-Code, and Instance-Code. Individual components MAY be skipped and SHOULD be separated with hyphens. A combination of components SHOULD be prefixed with "ISCC".

A Fully Qualified ISCC Code is an ordered sequence of Meta-Code, Content-Code, Data-Code, and Instance-Code codes. It SHOULD be prefixed with ISCC and MAY be separated by hyphens.

Component Types#

Each component has the same basic structure of a 1-byte header and a 8-byte body section.

The 1-byte header of each component is subdivided into 2 nibbles (4 bits). The first nibble specifies the component type while the second nibble is component specific.

The header only needs to be carried in the encoded representation. As similarity searches across different components are of little use, the type-information contained in the header of each component can be safely ignored after an ISCC has been decomposed and internally typed by an application.

List of Component Headers#

The body section of each component is specific to the component and always 8-bytes and can thus be fit into a 64-bit integer for efficient data processing. The following sections give an overview of how the different components work and how they are generated.

Meta-Code Component#

The Meta-Code component starts with a 1-byte header 00000000. The first nibble 0000 indicates that this is a Meta-Code component type. The second nibble is reserved for future extended features of the Meta-Code.

The Meta-Code body is built from a 64-bit similarity_hash over 4-character n-grams of the basic metadata of the content to be identified. The basic metadata supplied to the Meta-Code generating function is assumed to be UTF-8 encoded. Errors that occur during the decoding of such a byte string input to a native Unicode MUST terminate the process and must not be silenced. An ISCC generating application MUST provide a meta_id function that accepts minimal and generic metadata and returns a Base58-ISCC encoded Meta-Code component and trimmed metadata.

Inputs to Meta-Code function#

Generate Meta-Code#

An ISCC generating application must follow these steps in the given order to produce a stable Meta-Code:

1. Apply text_normalize separately to the title and extra inputs while keeping white space.
2. Apply text_trim to the results of the previous step. The results of this step MUST be supplied as basic metadata for ISCC registration.
3. Concatenate trimmed title and extra from using a space ( \u0020) as a separator. Remove leading/trailing whitespace.
4. Create a list of 4 character n-grams by sliding character-wise through the result of the previous step.
5. Encode each n-gram from the previous step to an UTF-8 bytestring and calculate its xxHash64 digest.
6. Apply similarity_hash to the list of digests from the previous step.
7. Prepend the 1-byte component header (0x00) to the results of the previous step.
8. Encode the resulting 9 byte sequence with encode
9. Return encoded Meta-Code, trimmed title and trimmed extra data.

See also: Meta-Code reference code

Dealing with Meta-Code collisions#

Ideally we want multiple ISCCs that identify different manifestations of the same intangible creation to be automatically grouped by an identical leading Meta-Code component. We call such a natural grouping an intended component collision. Metadata, captured and edited by humans, is notoriously unreliable. By using normalization and a similarity hash on the metadata, we account for some of this variation while keeping the Meta-Code component somewhat stable.

Auto-generated Meta-Codes components are expected to miss some intended collisions. An application SHOULD check for such missed intended component collisions before registering a new Meta-Code with the canonical registry of ISCCs by conducting a similarity search and asking for user feedback.

But what about unintended component collisions? Such collisions might happen because two different intangible creations have very similar or even identical metadata. But they might also happen by chance. With 2^56 possible Meta-Code components the probability of random collisions rises in an S-curved shape with the number of deployed ISCCs (see: Hash Collision Probabilities). We should keep in mind that the Meta-Code component is only one part of a fully qualified ISCC Code. Unintended collisions of the Meta-Code component are generally deemed as acceptable and expected.

If for any reason an application wants to avoid unintended collisions with pre-existing Meta-Code components, it may use the extra-field. An application MUST first generate a Meta-Code without asking the user for input to the extra-field and then first check for collisions with the canonical registry of ISCCs. After it finds a collision with a pre-existing Meta-Code it may display the metadata of the colliding entry and interact with the user to determine if it is an unintended collision. Only if the user indicates an unintended collision, may the application ask for a disambiguation that is then added as an amendment to the metadata via the extra-field to create a different Meta-Code component. The application may repeat the pre-existence check until it finds no collision or a user intended collision. The application MUST NOT supply auto-generated input to the extra-field.

It is our opinion that the concept of intended collisions of Meta-Code components is a useful concept and a net positive. But one must know that this characteristic also has its pitfalls. It is not an attempt to provide an unambiguous - agreed upon - definition of "identical intangible creations".

Content-Code Component#

The Content-Code component has multiple subtypes. The subtypes correspond with the Generic Media Types (GMT). A fully qualified ISCC can only have one Content-Code component of one specific GMT, but there may be multiple ISCCs with different Content-Code types per digital media object.

A Content-Code is generated in two broad steps. In the first step, we extract and convert content from a rich media type to a normalized GMT. In the second step, we use a GMT-specific process to generate the Content-Code component of an ISCC.

Generic Media Types#

The Content-Code type is signaled by the first 3 bits of the second nibble of the first byte of the Content-Code:

Content-Code-Text#

The Content-Code-Text is built from the extracted plain-text content of an encoded media object. To build a stable Content-Code-Text the plain-text content must first be extracted from the digital media object. It should be extracted in a way that is reproducible. There are many text document formats out in the wild and extracting plain-text from all of them is anything but a trivial task. While text-extraction is out of scope for this specification it is RECOMMENDED, that plain-text content SHOULD be extracted with the open-source Apache Tika v1.23 toolkit, if a generic reproducibility of the Content-Code-Text component is desired.

An ISCC generating application MUST provide a content_id(text, partial=False) function that accepts UTF-8 encoded plain text and a boolean, indicating the partial content flag as input and returns a Content-Code with GMT type text. The procedure to create a Content-Code-Text is:

1. Apply text_normalize to the text input while removing white-space.
2. Create character-wise n-grams of length 13 from the normalized text.
3. Create a list of 32-bit unsigned integer features by applying xxHash32 to the results of the previous step.
4. Apply minimum_hash to the list of features from the previous step with n=64.
5. Collect the least significant bits from the 64 MinHash features from the previous step.
6. Create a 64-bit digest from the collected bits.
7. Prepend the 1-byte component header (0x10 full content or 0x11 partial content).
8. Encode and return the resulting 9-byte sequence with encode.

See also: Content-Code-Text reference code

Content-Code-Image#

For the Content-Code-Image we are opting for a DCT-based perceptual image-hash instead of a more sophisticated key-point detection based method. In view of the generic deployability of the ISCC we chose an algorithm that has moderate computation requirements and is easy to implement while still being robust against common image manipulations.

An ISCC generating application MUST provide a content_id_image(image, partial=False) function that accepts a local file path to an image and returns a Content-Code with GMT type image. The procedure to create a Content-Code-Image is as follows:

1. Apply image_normalize to receive a two-dimensional array of gray-scale pixel data.
2. Apply image_hash to the results of the previous step.
3. Prepend the 1-byte component header (0x12 full content or 0x13 partial content) to results of the previous step.
4. Encode and return the resulting 9-byte sequence with encode

See also: Content-Code-Image reference code

Content-Code-Mixed#

The Content-Code-Mixed aggregates multiple Content-Codes of the same or different types. It may be used for digital media objects that embed multiples types of media or for collections of contents of the same type. First, we have to collect contents from the mixed media object or content collection and generate Content-Codes for each item. An ISCC conforming application must provide a content_id_mixed function that takes a list of Content-Code Codes as input and returns a Content-Code-Mixed. Follow these steps to create a Content-Code-Mixed:

Signature: conent_id_mixed(cids: List[str], partial: bool=False) -> str

1. Decode the list of Content-Codes.
2. Extract the first 8-bytes from each digest (Note: this includes the header part of the Content-Codes).
3. Apply similarity_hash to the list of digests from step 2.
4. Prepend the 1-byte component header(0x18 full content or 0x19 partial content)
5. Apply encode to the result of step 5 and return the result.

See also: Content-Code-Mixed reference code

Partial Content Flag (PCF)#

The last bit of the header byte of the Content-Code is the "Partial Content Flag". It designates if the Content-Code applies to the full content, or just some part of it. The PCF MUST be set as a 0-bit (full GMT-specific content) by default. Setting the PCF to 1 enables applications to create multiple linked ISCCs of partial extracts of a content collection. The exact semantics of partial content are outside of the scope of this specification. Applications that plan to support partial Content-Codes MUST define their semantics.

Data-Code Component#

For the Data-Code that encodes data similarity we use a content defined chunking algorithm that provides some shift resistance and calculate the MinHash from those chunks. To accommodate for small files, the first 100 chunks have a ~140-byte size target while the remaining chunks target ~ 6kb in size.

The Data-Code is built from the raw encoded data of the content to be identified. An ISCC generating application MUST provide a data_id function that accepts the raw encoded data as input.

Generate Data-Code#

1. Apply data_chunks to the raw encoded content data.
2. For each chunk, calculate the xxHash32 integer hash.
3. Apply minimum_hash to the resulting list of 32-bit unsigned integers with n=64.
4. Collect the least significant bits from the 64 MinHash features.
5. Create a 64-bit digest from the collected bits.
6. Prepend the 1-byte component header (eg. 0x20).
7. Apply encode to the result of step 6 and return the result.

See also: Data-Code reference code

Instance-Code Component#

The Instance-Code is built from the raw data of the media object to be identified and serves as checksum for the media object. The raw data of the media object is split into 64-kB data-chunks. Then we build a hash-tree from those chunks and use the truncated tophash (Merkle root) as component-body of the Instance-Code.

To guard against length-extension attacks and second preimage attacks, we use double sha256 for hashing. We also prefix the hash input data with a 0x00-byte for the leaf nodes hashes and with a 0x01-byte for the internal node hashes. While the Instance-Code itself is a non-cryptographic checksum, the full tophash may be supplied in the extended metadata of an ISCC secure integrity verification is required.

An ISCC generating application MUST provide a instance_id function that accepts the raw data file as input and returns an encoded Instance-Code and a full hex-encoded 256-bit tophash.

Generate Instance-Code#

1. Split the raw bytes of the encoded media object into 64-kB chunks.
2. For each chunk, calculate the sha256d of the concatenation of a 0x00-byte and the chunk bytes. We call the resulting values leaf node hashes (LNH).
3. Calculate the next level of the hash tree by applying sha256d to the concatenation of a 0x01-byte and adjacent pairs of LNH values. If the length of the list of LNH values is uneven, concatenate the last LNH value with itself. We call the resulting values internal node hashes (INH).
4. Recursively apply 0x01-prefixed pair-wise hashing to the results of the last step until the process yields only one hash value. We call this value the tophash.
5. Trim the resulting tophash to the first 8 bytes.
6. Prepend the 1-byte component header (e.g. 0x30).
7. Encode resulting 9-byte sequence with encode to an Instance-Code Code
8. Hex-Encode the tophash
9. Return the Instance-Code and the hex-encoded tophash

See also: Instance-Code reference code

Applications may carry, store, and process the leaf node hashes for advanced streaming data identification or partial data integrity verification.

ISCC Metadata#

As a generic content identifier, the ISCC makes minimal assumptions about metadata that must or should be supplied together with an ISCC. The RECOMMENDED data-interchange format for ISCC metadata is JSON. We distinguish between Basic Metadata and Extended Metadata:

Basic Metadata#

Basic metadata for an ISCC is metadata that is explicitly defined by this specification. The following table enumerates basic metadata fields for the top-level of the JSON metadata object:

Extended Metadata#

Extended metadata for an ISCC is metadata that is not explicitly defined by this specification. All such metadata SHOULD be supplied as JSON objects within the top-level meta-array field. This allows for a flexible and extendable way to supply additional industry specific metadata about the identified content.

Extended metadata entries MUST be wrapped in JSON object of the following structure:

ISCC Registration#

The ISCC is a decentralized identifier. ISCCs can be generated for content by anybody who has access to the content. Because of the clustering properties of its components, the ISCC provides utility in data interchange and de-duplication scenarios even without a global registry. There is no central authority for the registration of ISCC codes or certification of content authorship.

As an open system, the ISCC allows any person or organization to offer ISCC registration services as they see fit and without the need to ask anyone for permission. This also presumes that no person or organization may claim exclusive authority about ISCC registration.

Blockchain Registry#

To properly address the questions of identifier uniqueness, ownership and authentication within the ISCC Standard, the assignment of a set of canonical blockchain is a requirement. The distributed nature of blockchains are a perfect fit for long-term persistent identifier registration and resolver services.

The assignment of a set of canonical blockchains is NOT YET part of this specification.

Because this decision is of such vital importance, we suggest waiting for further feedback and additional community involvement before we address these questions either in an updated version of this specification or in a separate specification.

Our recommendation to the community is to agree on a set of decentralized, open, and public blockchains that have specific support for registry-services. This would maximize the value for all participants in the ecosystem. Governance and protocol related questions are being worked on by many projects.

See also: ISCC-Stream specification.

ISCC Embedding#

Embedding ISCC codes into content is only RECOMMENDED if it does not create a side effect. We call it a side effect if embedding an ISCC code changes the content to such an extent, that it yields a different ISCC code.

Side effects will depend on the combination of ISCC components that are to be embedded. A Meta-Code can always be embedded without side effects because it does not depend on the content itself. Content-Code and Data-Code may not change if embedded in larger media objects. Instance-Codes cannot easily be embedded as they will inevitably have a side effect on the post-embedding Instance-Code without special processing.

Applications MAY embed ISCC codes that have side effects if they specify a procedure by which the embedded ISCC codes can be stripped in such a way that the stripped content will yield the original embedded ISCC codes.

ISCC URI Scheme#

The purpose of the ISCC URI scheme based on RFC 3986 is to enable users to discover information like metadata or license offerings from an ISCC marked content by clicking a link on a webpage or by scanning a QR-Code.

The scheme name is iscc. The path component MUST be a fully qualified ISCC Code without hyphens. An optional stream query key MAY indicate the blockchain stream information source. If the stream query key is omitted, applications SHOULD return information from the open ISCC Stream.

The scheme name component ("iscc:") is case-insensitive. Applications MUST accept any combination of uppercase and lowercase letters in the scheme name. All other URI components are case-sensitive.

Applications MAY register themselves as a handler for the "iscc:" URI scheme if no other handler is already registered. If another handler is already registered, an application MAY ask the user to change it on the first run of the application.

URI Syntax#

<foo> means placeholder, [bar] means optional.

iscc:<fq-iscc-code>[?stream=<name>]

URI Example#

iscc:11TcMGvUSzqoM1CqVA3ykFawyh1R1sH4Bz8A1of1d2Ju4VjWt26S?stream=smart-license

Procedures & Algorithms#

Base58-ISCC#

The ISCC uses a custom per-component data encoding similar to the zbase62 encoding by Zooko Wilcox-O'Hearn but with a 58-character symbol table. The encoding does not require padding and will always yield component codes of 13 characters length for 72-bit component digests. The predictable size of the encoding is a property that allows for easy composition and decomposition of components without having to rely on a delimiter (hyphen) in the ISCC code representation. Colliding body segments of the digest are preserved by encoding the header and body separately. The ASCII symbol table also minimizes transcription and OCR errors by omitting the easily confused characters 'O', '0', 'I', 'l' and is shuffled to generate human readable component headers.

encode#

Signature: encode(digest: bytes) -> str

The encode function accepts a 9-byte ISCC Component Digest and returns the Base58-ISCC encoded alphanumeric string of 13 characters, which we call the ISCC-Component Code.

See also: Base-ISCC Encoding reference code

decode#

Signature: decode(code: str) -> bytes

the decode function accepts a 13-character ISCC-Component Code and returns the corresponding 9-byte ISCC-Component Digest.

See also: Base-ISCC Decoding reference code

Content Normalization#

The ISCC standardizes some content normalization procedures to support reproducible and stable identifiers. Following the list of normalization functions that MUST be provided by a conforming implementation.

text_trim#

Signature: text_trim(text: str) -> str

Trim text such that its UTF-8 encoded byte representation does not exceed 128-bytes each. Remove leading and trailing whitespace.

See also: Text trimming reference code

text_normalize#

Signature: text_normalize(text: str, keep_ws: bool = False) -> str

We define a text normalization function that is specific to our application. It takes text and an optional boolean keep_ws parameter as an input and returns normalized Unicode text for further algorithmic processing. The text_normalize function performs the following operations in the given order while each step works with the results of the previous operation:

1. Decode to native Unicode if the text is a byte string
2. Remove leading and trailing whitespace
3. Transform text to lowercase
4. Decompose the lower case text by applying Unicode Normalization Form D (NFD).
5. Filter out all characters that fall into the Unicode categories listed in the constant UNICODE_FILTER. Keep these control characters (Cc) that are commonly considered white-space:
   • \u0009, # Horizontal Tab (TAB)
   • \u000A, # Linefeed (LF)
   • \u000D, # Carriage Return (CR)
6. Keep or remove whitespace depending on the keep_ws parameter
7. Re-Combine the text by applying Unicode Normalization Form KC (NFKC).

See also: Text normalization reference code

image_normalize#

Signature: image_normalize(img) -> List[List[int]]

Accepts a file path, byte-stream or raw binary image data and MUST at least support JPEG, PNG, and GIF image formats. Normalize the image with the following steps:

1. Convert the image to grayscale
2. Resize the image to 32x32 pixels using bicubic interpolation
3. Create a 32x32 two-dimensional array of 8-bit gray-scale values from the image data

See also: Image normalization reference code

Feature Hashing#

The ISCC standardizes various feature hashing algorithms that reduce content features to a binary vector used as the body of the various Content-Code components.

similarity_hash#

Signature: similarity_hash(hash_digests: Sequence[ByteString]) -> bytes

The similarity_hash function takes a sequence of hash digests that represent a set of features. Each of the digests MUST be of equal size. The function returns a new hash digest (raw 8-Bit bytes) of the same size. For each bit in the input-hashes calculate the number of hashes with that bit set and subtract the count of hashes where it is not set. For the output-hash set the same bit position to 0 if the count is negative or 1 if it is zero or positive. The resulting hash digest will retain similarity for similar sets of input hashes. See also [Charikar2002].

See also: Similarity hash reference code

minimum_hash#

Signature: minimum_hash(features: Iterable[int], n: int = 64) -> List[int]

The minimum_hash function takes an arbitrary-sized set of 32-bit integer features and reduces it to a fixed size vector of n features such that it preserves similarity with other sets. It is based on the MinHash implementation of the datasketch library by Eric Zhu.

See also: Minimum hash reference code

image_hash#

Signature: image_hash(pixels: List[List[int]]) -> bytes

1. Perform a discrete cosine transform per row of input pixels.
2. Perform a discrete cosine transform per column on the resulting matrix from step 2.
3. Extract upper left 8x8 corner of the array from step 2 as a flat list.
4. Calculate the median of the results from step 3.
5. Create a 64-bit digest by iterating over the values of step 5 and setting a 1- for values above median and 0 for values below or equal to the median.
6. Return results from step 5.

See also: Image hash reference code

Content Defined Chunking#

For shift resistant data chunking, the ISCC requires a custom chunking algorithm:

data_chunks#

Signature: data_chunks(data: stream) -> Iterator[bytes]

The data_chunks function accepts a byte-stream and returns variable sized chunks. Chunk boundaries are determined by a gear based chunking algorithm based on [WenXia2016].

See also: CDC reference code

Conformance Testing#

An application that claims ISCC conformance MUST pass all required functions from the ISCC conformance test suite. The test suite is available as JSON data in our Github Repository. Test data is structured as follows:

{
    "<function_name>": {
        "required": true,
        "<test_name>": {
            "inputs": ["<value1>", "<value2>"],
            "outputs": ["value1>", "<value2>"]
        }
    }
}

The test suite also contains data for functions that are considered implementation details and MAY be skipped by other implementations. Optional tests are marked as "required": false.

Outputs that are expected to be raw bytes are embedded as HEX encoded strings in JSON and prefixed with hex: to support automated decoding during implementation testing.

{
  "data_chunks": {
    "test_001_cat_jpg": {
      "inputs": ["cat.jpg"],
      "outputs": ["hex:ffd8ffe1001845786966000049492a0008", ...]
    }
  }
}

License#

Copyright © 2016 - 2020 The Authors, Content Blockchain Project

This work is licensed under a Creative Commons (CC BY-NC-SA 4.0).