MARC

From Qontrol.nl Wiki
Jump to: navigation, search

MARC is the Magic Anonet Resource Claims System. This document describes MARC version 2. See an older revision of this document for the previous, now deprecated version of the protocol.

Design requirements

  • Distributes resource claims to all nodes in a network
  • Unattended automatic synchronisation should be possible, but people should not be able to alter each others' data
  • Support for real-time and offline synchronisation
  • Flexible but reliable representation of complex structures (nested arrays, associative arrays, binary strings)
  • Should be simple to implement, audit and use
  • Conflicts should be prevented, and should not affect other resources or updates

Features

  • Authenticated updates: only the first to claim a resource can update it afterwards (ownership based on first claim)
  • Updates are cryptographically signed using the ed25519 public-key signature system
  • Can use any transport that can transfer an arbitrary block of binary data for synchronization (eg HTTP, e-mail, USB stick, UDPMSG3, git, rsync, newsgroups)
  • Can use any storage mechanism that can store blocks of binary data (SQL, flat files)
  • Requires to store only the last known version of each resource (but can store older versions if desired)
  • A node is allowed not to store unwanted data, or to ignore unwanted updates

How?

  • Resources are identified by a label, a 0-255 byte binary string
  • IP addresses, domain names and AS numbers are encoded into a label, all sharing the same namespace
  • Resource details are represented in a complex structure (nested arrays, associative arrays, binary strings), which are encoded (binary format) into a binary string, the resource value
  • Each resource has a serial number, a higher serial number represents a more recent version, and should normally replace an older version in the database
  • Each resource has a status, which tells whether the resource is taken or has been released (and can be taken by someone else)
  • Each time a resource is changed a message is generated, which contains the label, value, status and serial; this message is signed and the cryptographic public key is added to form the update message
  • An update message always contains all resource details

Procedure for importing an update message

  1. recommended: if the update serial is lower than the current unix timestamp - 365*24*60*60 (1 year), ignore the update
  2. recommended: if the update serial is higher than the current unix timestamp + 7*24*60*60 (7 days), ignore the update
  3. if a resource with the given label exists:
    1. if the current serial is higher than or equal to the update serial, ignore the update
    2. ignore the update if:
      1. the key in the update is different from the current key
      2. AND the current serial is not lower than the current unix timestamp - 365*24*60*60 (1 year)
      3. AND (the expiration timestamp is not set OR the expiration timestamp is not lower than the current unix timestamp)
      4. AND (the transfer-to-key extension is not set or the transfer-to-key extension value does not match the key in the update)
  4. optional: if the update is too big (per node configuration), ignore the update
  5. optional: if the label is not acceptable (e.g. does not represent a valid IP network, AS number or domain name; node specific policy), ignore the update
  6. optional: if the resource value is not acceptable (missing fields, unexpected formatting; node specific policy), ignore the update
  7. if the update message is not signed with the public key specified in the update message, ignore the update (this can be done earlier)
  8. optional: import the resource value into existing data structures (e.g. configuration file, resdb, SQL database)
  9. store update message in the local database, identified/indexed by its label
    1. resources having a serial lower than the current unix timestamp - 365*24*60*60 (1 year) or an expiration timestamp lower than the current unix timestamp may be deleted from the local database instead, or may be deleted as part of a periodic database maintenance job

Layout of an update message

  • Version number (1 byte, value 2)
  • Public key (32 bytes)
  • Signature (64 bytes)
  • Resource data (m bytes)
    • Serial/timestamp (4 bytes, 32 bit big-endian integer, contains the current unix timestamp at the time of signing the resource)
    • Label length (1 byte)
    • Label (binary string)
    • Number of extensions (1 byte)
      • List of extensions: (n blocks)
        • Extension identifier (1 byte, value 1=transfer-to-key, 4=expiration-timestamp)
        • Extension data length (2 bytes, 16 bit big-endian integer)
        • Extension data
          • Identifier 1=transfer-to-key: the field contains a 32 byte public key that is allowed to update the resource in addition to the owning key, or no data (Extension data length = 0) to allow the resource to be updated by any key.
          • Identifier 4=expiration-timestamp: the field contains a 32-bit big-endian integer representing timestamp on which the resource is to be deleted. This value should be ignored if it is higher than Serial+365*24*60*60 (more than one year in the future).
    • Encoded value (see Complex structure encoding, this block covers the remainder of the update message)

Complex structure encoding

Complex structures are encoded into a binary string as follows:

  • NULL (no data)
    • Data type, 1 byte, value 0
  • String (binary or text string, may contain a textual representation of a number)
    • Data type, 1 byte, value 1
    • String data
  • List (ordered array with no otherwise relevant indices)
    • Data type, 1 byte, value 2
    • For each item in the collection:
      • Encoded item size in bytes (4 byte, 32 bit big-endian integer)
      • The item is recursively encoded using this scheme
  • Dictionary (array with relevant indices, associative array or list of key-value pairs)
    • Data type, 1 byte, value 3
    • For each item in the dictionary:
      • Key or index string length in bytes, 1 bytes
      • Key or index string
      • Encoded item size in bytes (4 byte, 32 bit big-endian integer)
      • The item is recursively encoded using this scheme

Notes

  • Resources in the deleted status may be removed from the database after some time
  • For efficient client-server request-response synchronisation, the server should store the local update timestamp for each resource. The client can store the highest timestamp it has received from the server, and ask for resources newer than the previous timestamp.
  • An implementation should remove or at least release resources which have not been updates in more than one year (update timestamp < current unix timestamp - 265*24*60*60), so that it can be registered by any key again. An update with a timestamp of more than a year ago should not be imported. An update with a timestamp more than 7 days in the future should not be imported.
  • To prevent expiration of valid resources, an implementation may automatically renew a resource starting 6 months before expiration or may warn the user to take action to renew the resource.

Interactive HTTP synchronisation

This is currently the preferred method. The server runs a CGI script, for which the client makes either a GET, POST or PUT request.

The query string for the request contains at least one parameter, 'version'. This section describes protocol version 3. If the client wishes to receive/pull updates from the server, the query string should include a parameter 'get', with a value of either 0 (to retrieve all records) or the value received in the timestamp extension in the response to the previous request if available (see below).

If the client also wishes to push updates to the server, either the POST or PUT method should be used. For a POST request, the updates to push should be encoded according to the application/x-www-form-urlencoded MIME type, with field names of update[], and set the HTTP Content-Type header accordingly. For a PUT request, all updates are sent as one binary block of data, where each update is prefixed with its length in bytes encoded as a 32 bit big-endian integer.

The server should respond with the HTTP Content-Type header set to application/octet-stream. The response data looks as follows:

  • Protocol version, 1 byte, value 3
  • Number of extensions (1 byte)
  • List of extensions: (n blocks)
    • Extension identifier (1 byte, value 2=counters, 3=timestamp)
    • Extension data length (2 bytes, 16 bit big-endian integer)
    • Extension data
      • Identifier 2=counters: data contains three 32-bit big-endian integers: the number of received update messages, the number of imported update messages and the number of exported update messages
      • Identifier 3=timestamp: data contains a 32-bit big-endian integer, containing the current local unix timestamp minus a few seconds, so that any updates a subsequent synchronization request providing this timestamp as the 'get' parameter will include at least all resources updated after the start of this request which are not part of this response.
  • For each exported update (read until all data has been consumed, ignoring the provided number of exported frames):

Note that the defined extensions are optional. The counters extension is purely for informative purposes, the timestamp extension is only used if the server supports incremental synchronization.

Anonet resource encoding

The different resource types are encoded as follows:

  • Key identity
    • label (33 bytes)
      • Resource type, 1 byte, value 0
      • Public key, 32 bytes, this must be the same key as the key used to sign the resource, otherwise the resource should not be accepted
    • value: dictionary
      • 'owner': optional, string, free text, name of owner
      • 'descr': optional, string, free text, description
      • 'seckey': optional, used to improve deniability by not storing the secret key locally. Contains the first 32 bytes of the secret key in encrypted form.
        • Number of rounds, 32 bit big-endian integer, defining the number of rounds to hash. If rounds is 0, the key is implicitly not encrypted at all.
        • Encrypted key, first 32 bytes of the encrypted secret key as follows: secretkey XOR hash_result, where hash_result is initialized as 64*0x00 and then modified Number of rounds as hash_result = sha512(hash_result + utf-8(passphrase) + publickey), where + means concatenation, passphrase is an UTF-8 encoded string chosen by the user and the public key is the same as used to sign this record.
  • IPv4 network
    • label (6 bytes):
      • Resource type, 1 byte, value 1
      • Network address, 4 bytes, network byte order
      • Prefix length, 1 byte, value 0-32
    • value: dictionary
      • 'owner': optional, string, free text, name of owner
      • 'descr': optional, string, free text, description
      • 'as': optional, numeric string, as number announcing this network
      • 'ns': optional, dictionary, see the Domain resource type, defines the reverse DNS zone for the IP network
  • IPv6 network
    • label (18 bytes):
      • Resource type, 1 byte, value 2
      • Network address, 16 bytes
      • Prefix length, 1 byte, value 0-128
    • value: dictionary
      • 'owner': optional, string, free text, name of owner
      • 'descr': optional, string, free text, description
      • 'as': optional, numeric string, as number announcing this network
      • 'ns': optional, dictionary, see the Domain resource type, defines the reverse DNS zone for the IP network
  • AS number
    • label (5 bytes):
      • Resource type, 1 byte, value 3
      • AS number, 4 bytes, 32 bit big-endian integer
    • value: dictionary
      • 'owner': optional, string, free text, name of owner
      • 'descr': optional, string, free text, description
      • 'speed': optional, numeric string, speed of this node in Mbit/sec
      • 'hasipv6': optional, NULL, existence indicates that the node has IPv6 connectivity
  • Domain
    • label (n+1 bytes):
      • Resource type, 1 byte, value 4
      • Domain name, n bytes, all lower case, FQDN format, no terminating dot
    • value: dictionary
      • 'owner': optional, string, free text, name of owner
      • 'descr': optional, string, free text, description
      • 'ns': recommended, dictionary listing name servers for the zone
        • Key: string, part before the domain name without dot for in-zone nameservers having a glue record or a FQDN with a final dot for out-of-zone nameservers
        • Value:
          • for an in-zone nameserver: list containing strings, each item defines a glue IPv4 or IPv6 address in text representation.
          • for out-of-zone nameserver: NULL. Should be ignored during export of DNS data for tinydns/bind, regardless of type.

Anonet importing rules

An implementation is free to handle resources as it wishes. The following list provides recommendations for handling resources. If resources are to be ignored, an implementation may do so either during import into the MARC database, or when viewing and exporting to external systems like a DNS server.

  • Key identity resources: resources with a label length other than 33 or defining a key other than the signing key (label[1:32] != public key) should be ignored
  • IPv4 network resources: resources with a label length other than 6, with a network not in 1.0.0.0/8 (label[1] != 1) or a prefix length other than 24 (label[4] != 24) should be ignored
  • IPv6 network resources: resources with a label length other than 18, with a network not in fd::/8 (label[1] != 0xfd) or a prefix length other than 48 or 64 (label[17] != 48 && label[17] != 64) should be ignored
  • Domain resources: the domain name should not be empty and should be a valid fully qualified domain name without trailing dot, the domain name should not contain non-ASCII or uppercase characters; it is recommended that an implementation has a configurable list of parent domains under which domain names can be imported, and that only domains directly under a parent domain are imported; this list should contain the .ano tld, so that *.ano can be claimed but not *.*.ano.