Misc¶

ACLs¶

Buffered HTTP¶

Constraints¶

Container Sync Realms¶

class swift.common.container_sync_realms.ContainerSyncRealms(conf_path, logger)¶

Bases: object

Loads and parses the container-sync-realms.conf, occasionally checking the file’s mtime to see if it needs to be reloaded.

clusters(realm)¶: Returns a list of clusters for the realm.

endpoint(realm, cluster)¶: Returns the endpoint for the cluster in the realm.

get_sig(request_method, path, x_timestamp, nonce, realm_key, user_key)¶

Returns the hexdigest string of the HMAC-SHA1 (RFC 2104) for the information given.

Parameters

request_method – HTTP method of the request.
path – The path to the resource (url-encoded).
x_timestamp – The X-Timestamp header value for the request.
nonce – A unique value for the request.
realm_key – Shared secret at the cluster operator level.
user_key – Shared secret at the user’s container level.

Returns

hexdigest str of the HMAC-SHA1 for the request.

key(realm)¶: Returns the key for the realm.

key2(realm)¶: Returns the key2 for the realm.

realms()¶: Returns a list of realms.

reload()¶: Forces a reload of the conf file.

Direct Client¶

Exceptions¶

exception swift.common.exceptions.APIVersionError¶: Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.ChunkReadError¶: Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.ChunkReadTimeout(seconds=None, exception=None)¶: Bases: eventlet.timeout.Timeout

exception swift.common.exceptions.ChunkWriteTimeout(seconds=None, exception=None)¶: Bases: eventlet.timeout.Timeout

exception swift.common.exceptions.ClientException(msg, http_scheme='', http_host='', http_port='', http_path='', http_query='', http_status=None, http_reason='', http_device='', http_response_content='', http_headers=None)¶: Bases: Exception

exception swift.common.exceptions.ConnectionTimeout(seconds=None, exception=None)¶: Bases: eventlet.timeout.Timeout

exception swift.common.exceptions.DeviceUnavailable¶: Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.DiskFileBadMetadataChecksum¶: Bases: swift.common.exceptions.DiskFileError

exception swift.common.exceptions.DiskFileCollision¶: Bases: swift.common.exceptions.DiskFileError

exception swift.common.exceptions.DiskFileDeleted(metadata=None)¶: Bases: swift.common.exceptions.DiskFileNotExist

exception swift.common.exceptions.DiskFileDeviceUnavailable¶: Bases: swift.common.exceptions.DiskFileError

exception swift.common.exceptions.DiskFileError¶: Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.DiskFileExpired(metadata=None)¶: Bases: swift.common.exceptions.DiskFileDeleted

exception swift.common.exceptions.DiskFileNoSpace¶: Bases: swift.common.exceptions.DiskFileError

exception swift.common.exceptions.DiskFileNotExist¶: Bases: swift.common.exceptions.DiskFileError

exception swift.common.exceptions.DiskFileNotOpen¶: Bases: swift.common.exceptions.DiskFileError

exception swift.common.exceptions.DiskFileQuarantined¶: Bases: swift.common.exceptions.DiskFileError

exception swift.common.exceptions.DiskFileXattrNotSupported¶: Bases: swift.common.exceptions.DiskFileError

exception swift.common.exceptions.DriveNotMounted¶: Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.DuplicateDeviceError¶: Bases: swift.common.exceptions.RingBuilderError

exception swift.common.exceptions.EmptyRingError¶: Bases: swift.common.exceptions.RingBuilderError

exception swift.common.exceptions.EncryptionException¶: Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.FileNotFoundError¶: Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.FooterNotSupported¶: Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.InsufficientStorage¶: Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.InvalidAccountInfo¶: Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.InvalidPidFileException¶: Bases: Exception

exception swift.common.exceptions.InvalidTimestamp¶: Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.LinkIterError¶: Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.ListingIterError¶: Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.ListingIterNotAuthorized(aresp)¶: Bases: swift.common.exceptions.ListingIterError

exception swift.common.exceptions.ListingIterNotFound¶: Bases: swift.common.exceptions.ListingIterError

exception swift.common.exceptions.LockTimeout(seconds=None, msg=None)¶: Bases: swift.common.exceptions.MessageTimeout

exception swift.common.exceptions.MessageTimeout(seconds=None, msg=None)¶: Bases: eventlet.timeout.Timeout

exception swift.common.exceptions.MimeInvalid¶: Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.MultiphasePUTNotSupported¶: Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.PartitionLockTimeout(seconds=None, msg=None)¶: Bases: swift.common.exceptions.LockTimeout

exception swift.common.exceptions.PathNotDir¶: Bases: OSError

exception swift.common.exceptions.PermissionError¶: Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.PutterConnectError(status=None)¶: Bases: Exception

exception swift.common.exceptions.QuarantineRequest¶: Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.RangeAlreadyComplete¶: Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.ReplicationException¶: Bases: Exception

exception swift.common.exceptions.ReplicationLockTimeout(seconds=None, msg=None)¶: Bases: swift.common.exceptions.LockTimeout

exception swift.common.exceptions.ResponseTimeout(seconds=None, exception=None)¶: Bases: eventlet.timeout.Timeout

exception swift.common.exceptions.RingBuilderError¶: Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.RingLoadError¶: Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.RingValidationError¶: Bases: swift.common.exceptions.RingBuilderError

exception swift.common.exceptions.SegmentError¶: Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.ShortReadError¶: Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.SuffixSyncError¶: Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.SwiftException¶: Bases: Exception

exception swift.common.exceptions.UnPicklingError¶: Bases: swift.common.exceptions.SwiftException

exception swift.common.exceptions.UnknownSecretIdError¶: Bases: swift.common.exceptions.EncryptionException

Internal Client¶

Manager¶

class swift.common.manager.Manager(servers, run_dir='/var/run/swift')¶

Bases: object

Main class for performing commands on groups of servers.

Parameters: servers – list of server names as strings

force_reload(**kwargs)¶: alias for reload

get_command(cmd)¶

Find and return the decorated method named like cmd

Parameters: cmd – the command to get, a string, if not found raises UnknownCommandError

kill(**kwargs)¶: stop a server (no error if not running)

kill_child_pids(**kwargs)¶: kill child pids, optionally servicing accepted connections

classmethod list_commands()¶

Get all publicly accessible commands

Returns: a list of string tuples (cmd, help), the method names who are decorated as commands

no_daemon(**kwargs)¶: start a server interactively

no_wait(**kwargs)¶: spawn server and return immediately

once(**kwargs)¶: start server and run one pass on supporting daemons

reload(**kwargs)¶: graceful shutdown then restart on supporting servers

reload_seamless(**kwargs)¶: seamlessly re-exec, then shutdown of old listen sockets on supporting servers

restart(**kwargs)¶: stops then restarts server

run_command(cmd, **kwargs)¶

Find the named command and run it

Parameters: cmd – the command name to run

shutdown(**kwargs)¶: allow current requests to finish on supporting servers

start(**kwargs)¶: starts a server

status(**kwargs)¶: display status of tracked pids for server

stop(**kwargs)¶: stops a server

class swift.common.manager.Server(server, run_dir='/var/run/swift')¶

Bases: object

Manage operations on a server or group of servers of similar type

Parameters: server – name of server

conf_files(**kwargs)¶

Get conf files for this server

Parameters: number – if supplied will only lookup the nth server
Returns: list of conf files

get_conf_file_name(pid_file)¶

Translate pid_file to a corresponding conf_file

Parameters: pid_file – a pid_file for this server, a string
Returns: the conf_file for this pid_file

get_pid_file_name(conf_file)¶

Translate conf_file to a corresponding pid_file

Parameters: conf_file – an conf_file for this server, a string
Returns: the pid_file for this conf_file

get_running_pids(**kwargs)¶

Get running pids

Returns: a dict mapping pids (ints) to pid_files (paths)

interact(**kwargs)¶: wait on spawned procs to terminate

iter_pid_files(**kwargs)¶: Generator, yields (pid_file, pids)

kill_child_pids(**kwargs)¶

Kill child pids, leaving server overseer to respawn them

Parameters

graceful – if True, attempt SIGHUP on supporting servers
seamless – if True, attempt SIGUSR1 on supporting servers

Returns

a dict mapping pids (ints) to pid_files (paths)

kill_running_pids(**kwargs)¶

Kill running pids

Parameters

graceful – if True, attempt SIGHUP on supporting servers
seamless – if True, attempt SIGUSR1 on supporting servers

Returns

a dict mapping pids (ints) to pid_files (paths)

launch(**kwargs)¶: Collect conf files and attempt to spawn the processes for this server

pid_files(**kwargs)¶

Get pid files for this server

Parameters: number – if supplied will only lookup the nth server
Returns: list of pid files

signal_children(sig, **kwargs)¶

Send a signal to child pids for this server

Parameters: sig – signal to send
Returns: a dict mapping pids (ints) to pid_files (paths)

signal_pids(sig, **kwargs)¶

Send a signal to pids for this server

Parameters: sig – signal to send
Returns: a dict mapping pids (ints) to pid_files (paths)

spawn(conf_file, once=False, wait=True, daemon=True, additional_args=None, **kwargs)¶

Launch a subprocess for this server.

Parameters

conf_file – path to conf_file to use as first arg
once – boolean, add once argument to command
wait – boolean, if true capture stdout with a pipe
daemon – boolean, if false ask server to log to console
additional_args – list of additional arguments to pass on the command line

Returns

the pid of the spawned process

status(pids=None, **kwargs)¶

Display status of server

Parameters

pids – if not supplied pids will be populated automatically
number – if supplied will only lookup the nth server

Returns

1 if server is not running, 0 otherwise

stop(**kwargs)¶

Send stop signals to pids for this server

Returns: a dict mapping pids (ints) to pid_files (paths)

wait(**kwargs)¶: wait on spawned procs to start

exception swift.common.manager.UnknownCommandError¶: Bases: Exception

swift.common.manager.command(func)¶

Decorator to declare which methods are accessible as commands, commands always return 1 or 0, where 0 should indicate success.

Parameters: func – function to make public

swift.common.manager.format_server_name(servername)¶

Formats server name as swift compatible server names E.g. swift-object-server

Parameters: servername – server name
Returns: swift compatible server name and its binary name

swift.common.manager.kill_group(pid, sig)¶

Send signal to process group

: param pid: process id : param sig: signal to send

swift.common.manager.safe_kill(pid, sig, name)¶

Send signal to process and check process name

: param pid: process id : param sig: signal to send : param name: name to ensure target process

swift.common.manager.setup_env()¶: Try to increase resource limits of the OS. Move PYTHON_EGG_CACHE to /tmp

swift.common.manager.verify_server(server)¶

Check whether the server is among swift servers or not, and also checks whether the server’s binaries are installed or not.

Parameters: server – name of the server
Returns: True, when the server name is valid and its binaries are found. False, otherwise.

swift.common.manager.watch_server_pids(server_pids, interval=1, **kwargs)¶

Monitor a collection of server pids yielding back those pids that aren’t responding to signals.

Parameters: server_pids – a dict, lists of pids [int,…] keyed on Server objects

MemCacheD¶

Why our own memcache client? By Michael Barton

python-memcached doesn’t use consistent hashing, so adding or removing a memcache server from the pool invalidates a huge percentage of cached items.

If you keep a pool of python-memcached client objects, each client object has its own connection to every memcached server, only one of which is ever in use. So you wind up with n * m open sockets and almost all of them idle. This client effectively has a pool for each server, so the number of backend connections is hopefully greatly reduced.

python-memcache uses pickle to store things, and there was already a huge stink about Swift using pickles in memcache (http://osvdb.org/show/osvdb/86581). That seemed sort of unfair, since nova and keystone and everyone else use pickles for memcache too, but it’s hidden behind a “standard” library. But changing would be a security regression at this point.

Also, pylibmc wouldn’t work for us because it needs to use python sockets in order to play nice with eventlet.

Lucid comes with memcached: v1.4.2. Protocol documentation for that version is at:

http://github.com/memcached/memcached/blob/1.4.2/doc/protocol.txt

class swift.common.memcached.MemcacheConnPool(server, size, connect_timeout, tls_context=None)¶

Bases: eventlet.pools.Pool

Connection pool for Memcache Connections

The server parameter can be a hostname, an IPv4 address, or an IPv6 address with an optional port. See swift.common.utils.parse_socket_string() for details.

create()¶

Generate a new pool item. In order for the pool to function, either this method must be overriden in a subclass or the pool must be constructed with the create argument. It accepts no arguments and returns a single instance of whatever thing the pool is supposed to contain.

In general, create() is called whenever the pool exceeds its previous high-water mark of concurrently-checked-out-items. In other words, in a new pool with min_size of 0, the very first call to get() will result in a call to create(). If the first caller calls put() before some other caller calls get(), then the first item will be returned, and create() will not be called a second time.

get()¶: Return an item from the pool, when one is available. This may cause the calling greenthread to block.

exception swift.common.memcached.MemcacheConnectionError¶: Bases: Exception

exception swift.common.memcached.MemcachePoolTimeout(seconds=None, exception=None)¶: Bases: eventlet.timeout.Timeout

class swift.common.memcached.MemcacheRing(servers, connect_timeout=0.3, io_timeout=2.0, pool_timeout=1.0, tries=3, allow_pickle=False, allow_unpickle=False, max_conns=2, tls_context=None, logger=None, error_limit_count=10, error_limit_time=60, error_limit_duration=60)¶

Bases: object

Simple, consistent-hashed memcache client.

decr(key, delta=1, time=0)¶

Decrements a key which has a numeric value by delta. Calls incr with -delta.

Parameters

key – key
delta – amount to subtract to the value of key (or set the value to 0 if the key is not found) will be cast to an int
time – the time to live

Returns

result of decrementing

Raises

MemcacheConnectionError –

delete(key, server_key=None)¶

Deletes a key/value pair from memcache.

Parameters

key – key to be deleted
server_key – key to use in determining which server in the ring is used

get(key)¶

Gets the object specified by key. It will also unserialize the object before returning if it is serialized in memcache with JSON, or if it is pickled and unpickling is allowed.

Parameters: key – key
Returns: value of the key in memcache

get_multi(keys, server_key)¶

Gets multiple values from memcache for the given keys.

Parameters

keys – keys for values to be retrieved from memcache
server_key – key to use in determining which server in the ring is used

Returns

list of values

incr(key, delta=1, time=0)¶

Increments a key which has a numeric value by delta. If the key can’t be found, it’s added as delta or 0 if delta < 0. If passed a negative number, will use memcached’s decr. Returns the int stored in memcached Note: The data memcached stores as the result of incr/decr is an unsigned int. decr’s that result in a number below 0 are stored as 0.

Parameters

key – key
delta – amount to add to the value of key (or set as the value if the key is not found) will be cast to an int
time – the time to live

Returns

result of incrementing

Raises

MemcacheConnectionError –

set(key, value, serialize=True, time=0, min_compress_len=0)¶

Set a key/value pair in memcache

Parameters

key – key
value – value
serialize – if True, value is serialized with JSON before sending to memcache, or with pickle if configured to use pickle instead of JSON (to avoid cache poisoning)
time – the time to live
min_compress_len – minimum compress length, this parameter was added to keep the signature compatible with python-memcached interface. This implementation ignores it.

set_multi(mapping, server_key, serialize=True, time=0, min_compress_len=0)¶

Sets multiple key/value pairs in memcache.

Parameters

mapping – dictionary of keys and values to be set in memcache
server_key – key to use in determining which server in the ring is used
serialize – if True, value is serialized with JSON before sending to memcache, or with pickle if configured to use pickle instead of JSON (to avoid cache poisoning)
time – the time to live

Min_compress_len

minimum compress length, this parameter was added to keep the signature compatible with python-memcached interface. This implementation ignores it

swift.common.memcached.sanitize_timeout(timeout)¶: Sanitize a timeout value to use an absolute expiration time if the delta is greater than 30 days (in seconds). Note that the memcached server translates negative values to mean a delta of 30 days in seconds (and 1 additional second), client beware.

Request Helpers¶

Swob¶

Utils¶

Miscellaneous utility functions for use with Swift.

swift.common.utils.ATTRIBUTES_RE = re.compile('(\\w+)=(".*?"|[^";]+)(; ?|$)')¶: Regular expression to match form attributes.

class swift.common.utils.CloseableChain(*iterables)¶

Bases: object

Like itertools.chain, but with a close method that will attempt to invoke its sub-iterators’ close methods, if any.

class swift.common.utils.ContextPool(size=1000)¶

Bases: eventlet.greenpool.GreenPool

GreenPool subclassed to kill its coros when it gets gc’ed

class swift.common.utils.Everything¶

Bases: object

A container that contains everything. If “e” is an instance of Everything, then “x in e” is true for all x.

class swift.common.utils.GreenAsyncPile(size_or_pool)¶

Bases: object

Runs jobs in a pool of green threads, and the results can be retrieved by using this object as an iterator.

This is very similar in principle to eventlet.GreenPile, except it returns results as they become available rather than in the order they were launched.

Correlating results with jobs (if necessary) is left to the caller.

spawn(func, *args, **kwargs)¶: Spawn a job in a green thread on the pile.

waitall(timeout)¶

Wait timeout seconds for any results to come in.

Parameters: timeout – seconds to wait for results
Returns: list of results accrued in that time

waitfirst(timeout)¶

Wait up to timeout seconds for first result to come in.

Parameters: timeout – seconds to wait for results
Returns: first item to come back, or None

exception swift.common.utils.GreenAsyncPileWaitallTimeout(seconds=None, exception=None)¶: Bases: eventlet.timeout.Timeout

class swift.common.utils.GreenthreadSafeIterator(unsafe_iterable)¶

Bases: object

Wrap an iterator to ensure that only one greenthread is inside its next() method at a time.

This is useful if an iterator’s next() method may perform network IO, as that may trigger a greenthread context switch (aka trampoline), which can give another greenthread a chance to call next(). At that point, you get an error like “ValueError: generator already executing”. By wrapping calls to next() with a mutex, we avoid that error.

class swift.common.utils.InputProxy(wsgi_input)¶

Bases: object

File-like object that counts bytes read. To be swapped in for wsgi.input for accounting purposes.

read(*args, **kwargs)¶: Pass read request to the underlying file-like object and add bytes read to total.

readline(*args, **kwargs)¶: Pass readline request to the underlying file-like object and add bytes read to total.

exception swift.common.utils.InvalidHashPathConfigError¶: Bases: ValueError

class swift.common.utils.LRUCache(maxsize=1000, maxtime=3600)¶

Bases: object

Decorator for size/time bound memoization that evicts the least recently used members.

class swift.common.utils.LogAdapter(logger, server)¶

Bases: logging.LoggerAdapter, object

A Logger like object which performs some reformatting on calls to exception(). Can be used to store a threadlocal transaction id and client ip.

exception(msg, *args, **kwargs)¶: Delegate an exception call to the underlying logger.

getEffectiveLevel()¶: Get the effective level for the underlying logger.

notice(msg, *args, **kwargs)¶: Convenience function for syslog priority LOG_NOTICE. The python logging lvl is set to 25, just above info. SysLogHandler is monkey patched to map this log lvl to the LOG_NOTICE syslog priority.

process(msg, kwargs)¶: Add extra info to message

set_statsd_prefix(prefix)¶: The StatsD client prefix defaults to the “name” of the logger. This method may override that default with a specific value. Currently used in the proxy-server to differentiate the Account, Container, and Object controllers.

statsd_delegate()¶

Factory to create methods which delegate to methods on self.logger.statsd_client (an instance of StatsdClient). The created methods conditionally delegate to a method whose name is given in ‘statsd_func_name’. The created delegate methods are a no-op when StatsD logging is not configured.

Parameters: statsd_func_name – the name of a method on StatsdClient.

class swift.common.utils.LogLevelFilter(level=10)¶

Bases: object

Drop messages for the logger based on level.

This is useful when dependencies log too much information.

Parameters: level – All messages at or below this level are dropped (DEBUG < INFO < WARN < ERROR < CRITICAL|FATAL) Default: DEBUG

swift.common.utils.NR_ioprio_set()¶: Give __NR_ioprio_set value for your system.

class swift.common.utils.NicerInterpolation¶: Bases: configparser.BasicInterpolation

class swift.common.utils.NullLogger¶

Bases: object

A no-op logger for eventlet wsgi.

class swift.common.utils.OverrideOptions(devices, partitions, policies)¶

Bases: tuple

devices¶: Alias for field number 0

partitions¶: Alias for field number 1

policies¶: Alias for field number 2

class swift.common.utils.PipeMutex¶

Bases: object

Mutex using a pipe. Works across both greenlets and real threads, even at the same time.

acquire(blocking=True)¶

Acquire the mutex.

If called with blocking=False, returns True if the mutex was acquired and False if it wasn’t. Otherwise, blocks until the mutex is acquired and returns True.

This lock is recursive; the same greenthread may acquire it as many times as it wants to, though it must then release it that many times too.

close()¶

Close the mutex. This releases its file descriptors.

You can’t use a mutex after it’s been closed.

release()¶: Release the mutex.

class swift.common.utils.PrefixLoggerAdapter(logger, extra=None)¶

Bases: swift.common.utils.SwiftLoggerAdapter

Adds an optional prefix to all its log messages. When the prefix has not been set, messages are unchanged.

exception(msg, *a, **kw)¶: Delegate an exception call to the underlying logger.

process(msg, kwargs)¶

Process the logging message and keyword arguments passed in to a logging call to insert contextual information. You can either manipulate the message itself, the keyword args or both. Return the message and kwargs modified (or not) to suit your needs.

Normally, you’ll only need to override this one method in a LoggerAdapter subclass for your specific needs.

class swift.common.utils.RateLimitedIterator(iterable, elements_per_second, limit_after=0, ratelimit_if=<function RateLimitedIterator.<lambda>>)¶

Bases: object

Wrap an iterator to only yield elements at a rate of N per second.

Parameters

iterable – iterable to wrap
elements_per_second – the rate at which to yield elements
limit_after – rate limiting kicks in only after yielding this many elements; default is 0 (rate limit immediately)

class swift.common.utils.ShardRange(name, timestamp, lower=MinBound, upper=MaxBound, object_count=0, bytes_used=0, meta_timestamp=None, deleted=False, state=None, state_timestamp=None, epoch=None, reported=False)¶

Bases: object

A ShardRange encapsulates sharding state related to a container including lower and upper bounds that define the object namespace for which the container is responsible.

Shard ranges may be persisted in a container database. Timestamps associated with subsets of the shard range attributes are used to resolve conflicts when a shard range needs to be merged with an existing shard range record and the most recent version of an attribute should be persisted.

Parameters

name – the name of the shard range; this should take the form of a path to a container i.e. <account_name>/<container_name>.
timestamp – a timestamp that represents the time at which the shard range’s lower, upper or deleted attributes were last modified.
lower – the lower bound of object names contained in the shard range; the lower bound is not included in the shard range namespace.
upper – the upper bound of object names contained in the shard range; the upper bound is included in the shard range namespace.
object_count – the number of objects in the shard range; defaults to zero.
bytes_used – the number of bytes in the shard range; defaults to zero.
meta_timestamp – a timestamp that represents the time at which the shard range’s object_count and bytes_used were last updated; defaults to the value of timestamp.
deleted – a boolean; if True the shard range is considered to be deleted.
state – the state; must be one of ShardRange.STATES; defaults to CREATED.
state_timestamp – a timestamp that represents the time at which state was forced to its current value; defaults to the value of timestamp. This timestamp is typically not updated with every change of state because in general conflicts in state attributes are resolved by choosing the larger state value. However, when this rule does not apply, for example when changing state from SHARDED to ACTIVE, the state_timestamp may be advanced so that the new state value is preferred over any older state value.
epoch – optional epoch timestamp which represents the time at which sharding was enabled for a container.
reported – optional indicator that this shard and its stats have been reported to the root container.

copy(timestamp=None, **kwargs)¶

Creates a copy of the ShardRange.

Parameters: timestamp – (optional) If given, the returned ShardRange will have all of its timestamps set to this value. Otherwise the returned ShardRange will have the original timestamps.
Returns: an instance of ShardRange

entire_namespace()¶: Returns True if the ShardRange includes the entire namespace, False otherwise.

expand(donors)¶

Expands the bounds as necessary to match the minimum and maximum bounds of the given donors.

Parameters: donors – A list of ShardRange
Returns: True if the bounds have been modified, False otherwise.

classmethod from_dict(params)¶

Return an instance constructed using the given dict of params. This method is deliberately less flexible than the class __init__() method and requires all of the __init__() args to be given in the dict of params.

Parameters: params – a dict of parameters
Returns: an instance of this class

includes(other)¶

Returns True if this namespace includes the whole of the other namespace, False otherwise.

Parameters: other – an instance of ShardRange

increment_meta(object_count, bytes_used)¶

Increment the object stats metadata by the given values and update the meta_timestamp to the current time.

Parameters

object_count – should be an integer
bytes_used – should be an integer

Raises

ValueError – if object_count or bytes_used cannot be cast to an int.

classmethod make_path(shards_account, root_container, parent_container, timestamp, index)¶

Returns a path for a shard container that is valid to use as a name when constructing a ShardRange.

Parameters

shards_account – the hidden internal account to which the shard container belongs.
root_container – the name of the root container for the shard.
parent_container – the name of the parent container for the shard; for initial first generation shards this should be the same as root_container; for shards of shards this should be the name of the sharding shard container.
timestamp – an instance of Timestamp
index – a unique index that will distinguish the path from any other path generated using the same combination of shards_account, root_container, parent_container and timestamp.

Returns

a string of the form <account_name>/<container_name>

overlaps(other)¶

Returns True if the ShardRange namespace overlaps with the other ShardRange’s namespace.

Parameters: other – an instance of ShardRange

classmethod resolve_state(state)¶

Given a value that may be either the name or the number of a state return a tuple of (state number, state name).

Parameters: state – Either a string state name or an integer state number.
Returns: A tuple (state number, state name)
Raises: ValueError – if state is neither a valid state name nor a valid state number.

set_deleted(timestamp=None)¶

Mark the shard range deleted and set timestamp to the current time.

Parameters: timestamp – optional timestamp to set; if not given the current time will be set.
Returns: True if the deleted attribute or timestamp was changed, False otherwise

update_meta(object_count, bytes_used, meta_timestamp=None)¶

Set the object stats metadata to the given values and update the meta_timestamp to the current time.

Parameters

object_count – should be an integer
bytes_used – should be an integer
meta_timestamp – timestamp for metadata; if not given the current time will be set.

Raises

ValueError – if object_count or bytes_used cannot be cast to an int, or if meta_timestamp is neither None nor can be cast to a Timestamp.

update_state(state, state_timestamp=None)¶

Set state to the given value and optionally update the state_timestamp to the given time.

Parameters

state – new state, should be an integer
state_timestamp – timestamp for state; if not given the state_timestamp will not be changed.

Returns

True if the state or state_timestamp was changed, False otherwise

class swift.common.utils.ShardRangeList(initlist=None)¶

Bases: collections.UserList

This class provides some convenience functions for working with lists of ShardRange.

This class does not enforce ordering or continuity of the list items: callers should ensure that items are added in order as appropriate.

property bytes_used¶

Returns the total number of bytes in all items in the list.

Returns: total bytes used

includes(other)¶

Check if another ShardRange namespace is enclosed between the list’s lower and upper properties. Note: the list’s lower and upper properties will only equal the outermost bounds of all items in the list if the list has previously been sorted.

Note: the list does not need to contain an item matching other for this method to return True, although if the list has been sorted and does contain an item matching other then the method will return True.

Parameters: other – an instance of ShardRange
Returns: True if other’s namespace is enclosed, False otherwise.

property lower¶

Returns the lower bound of the first item in the list. Note: this will only be equal to the lowest bound of all items in the list if the list contents has been sorted.

Returns: lower bound of first item in the list, or ShardRange.MIN if the list is empty.

property object_count¶

Returns the total number of objects of all items in the list.

Returns: total object count

property upper¶

Returns the upper bound of the first item in the list. Note: this will only be equal to the uppermost bound of all items in the list if the list has previously been sorted.

Returns: upper bound of first item in the list, or ShardRange.MIN if the list is empty.

class swift.common.utils.Spliterator(source_iterable)¶

Bases: object

Takes an iterator yielding sliceable things (e.g. strings or lists) and yields subiterators, each yielding up to the requested number of items from the source.

>>> si = Spliterator(["abcde", "fg", "hijkl"])
>>> ''.join(si.take(4))
"abcd"
>>> ''.join(si.take(3))
"efg"
>>> ''.join(si.take(1))
"h"
>>> ''.join(si.take(3))
"ijk"
>>> ''.join(si.take(3))
"l"  # shorter than requested; this can happen with the last iterator

class swift.common.utils.StrAnonymizer(data, method, salt)¶

Bases: str

Class that permits to get a string anonymized or simply quoted.

class swift.common.utils.StrFormatTime(ts)¶

Bases: object

Class that permits to get formats or parts of a time.

class swift.common.utils.StreamingPile(size)¶

Bases: swift.common.utils.GreenAsyncPile

Runs jobs in a pool of green threads, spawning more jobs as results are retrieved and worker threads become available.

When used as a context manager, has the same worker-killing properties as ContextPool.

asyncstarmap(func, args_iter)¶: This is the same as itertools.starmap(), except that func is executed in a separate green thread for each item, and results won’t necessarily have the same order as inputs.

class swift.common.utils.SwiftLogFormatter(fmt=None, datefmt=None, max_line_length=0)¶

Bases: logging.Formatter

Custom logging.Formatter will append txn_id to a log message if the record has one and the message does not. Optionally it can shorten overly long log lines.

format(record)¶

Format the specified record as text.

The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.

class swift.common.utils.SwiftLoggerAdapter(logger, extra=None)¶

Bases: logging.LoggerAdapter

A logging.LoggerAdapter subclass that also passes through StatsD method calls.

Like logging.LoggerAdapter, you have to subclass this and override the process() method to accomplish anything useful.

class swift.common.utils.ThreadSafeSysLogHandler(address=('localhost', 514), facility=1, socktype=None)¶

Bases: logging.handlers.SysLogHandler

createLock()¶: Acquire a thread lock for serializing access to the underlying I/O.

class swift.common.utils.Timestamp(timestamp, offset=0, delta=0, check_bounds=True)¶

Bases: object

Internal Representation of Swift Time.

The normalized form of the X-Timestamp header looks like a float with a fixed width to ensure stable string sorting - normalized timestamps look like “1402464677.04188”

To support overwrites of existing data without modifying the original timestamp but still maintain consistency a second internal offset vector is append to the normalized timestamp form which compares and sorts greater than the fixed width float format but less than a newer timestamp. The internalized format of timestamps looks like “1402464677.04188_0000000000000000” - the portion after the underscore is the offset and is a formatted hexadecimal integer.

The internalized form is not exposed to clients in responses from Swift. Normal client operations will not create a timestamp with an offset.

The Timestamp class in common.utils supports internalized and normalized formatting of timestamps and also comparison of timestamp values. When the offset value of a Timestamp is 0 - it’s considered insignificant and need not be represented in the string format; to support backwards compatibility during a Swift upgrade the internalized and normalized form of a Timestamp with an insignificant offset are identical. When a timestamp includes an offset it will always be represented in the internalized form, but is still excluded from the normalized form. Timestamps with an equivalent timestamp portion (the float part) will compare and order by their offset. Timestamps with a greater timestamp portion will always compare and order greater than a Timestamp with a lesser timestamp regardless of it’s offset. String comparison and ordering is guaranteed for the internalized string format, and is backwards compatible for normalized timestamps which do not include an offset.

class swift.common.utils.Watchdog¶

Bases: object

Implements a watchdog to efficiently manage concurrent timeouts.

Compared to eventlet.timeouts.Timeout, it reduces the number of context switching in eventlet by avoiding to schedule actions (throw an Exception), then unschedule them if the timeouts are cancelled.

at T+0, request timeout(10)
=> wathdog greenlet sleeps 10 seconds
at T+1, request timeout(15)

=> the timeout will expire after the current, no need to wake up the
watchdog greenlet
at T+2, request timeout(5)

=> the timeout will expire before the first timeout, wake up the
watchdog greenlet to calculate a new sleep period
at T+7, the 3rd timeout expires

=> the exception is raised, then the greenlet watchdog sleep(3) to
wake up for the 1st timeout expiration

spawn()¶: Start the watchdog greenthread.

start(timeout, exc, timeout_at=None)¶

Schedule a timeout action

Parameters

timeout – duration before the timeout expires
exc – exception to throw when the timeout expire, must inherit from eventlet.timeouts.Timeout
timeout_at – allow to force the expiration timestamp

Returns

id of the scheduled timeout, needed to cancel it

stop(key)¶

Cancel a scheduled timeout

Parameters: key – timeout id, as returned by start()

class swift.common.utils.WatchdogTimeout(watchdog, timeout, exc, timeout_at=None)¶

Bases: object

Context manager to schedule a timeout in a Watchdog instance

swift.common.utils.affinity_key_function(affinity_str)¶

Turns an affinity config value into a function suitable for passing to sort(). After doing so, the array will be sorted with respect to the given ordering.

For example, if affinity_str is “r1=1, r2z7=2, r2z8=2”, then the array will be sorted with all nodes from region 1 (r1=1) first, then all the nodes from region 2 zones 7 and 8 (r2z7=2 and r2z8=2), then everything else.

Note that the order of the pieces of affinity_str is irrelevant; the priority values are what comes after the equals sign.

If affinity_str is empty or all whitespace, then the resulting function will not alter the ordering of the nodes.

Parameters: affinity_str – affinity config value, e.g. “r1z2=3” or “r1=1, r2z1=2, r2z2=2”
Returns: single-argument function
Raises: ValueError – if argument invalid

swift.common.utils.affinity_locality_predicate(write_affinity_str)¶

Turns a write-affinity config value into a predicate function for nodes. The returned value will be a 1-arg function that takes a node dictionary and returns a true value if it is “local” and a false value otherwise. The definition of “local” comes from the affinity_str argument passed in here.

For example, if affinity_str is “r1, r2z2”, then only nodes where region=1 or where (region=2 and zone=2) are considered local.

If affinity_str is empty or all whitespace, then the resulting function will consider everything local

Parameters: write_affinity_str – affinity config value, e.g. “r1z2” or “r1, r2z1, r2z2”
Returns: single-argument function, or None if affinity_str is empty
Raises: ValueError – if argument invalid

swift.common.utils.audit_location_generator(devices, datadir, suffix='', mount_check=True, logger=None, devices_filter=None, partitions_filter=None, suffixes_filter=None, hashes_filter=None, hook_pre_device=None, hook_post_device=None, hook_pre_partition=None, hook_post_partition=None, hook_pre_suffix=None, hook_post_suffix=None, hook_pre_hash=None, hook_post_hash=None, error_counter=None, yield_hash_dirs=False)¶

Given a devices path and a data directory, yield (path, device, partition) for all files in that directory

(devices|partitions|suffixes|hashes)_filter are meant to modify the list of elements that will be iterated. eg: they can be used to exclude some elements based on a custom condition defined by the caller.

Parameters

devices – parent directory of the devices to be audited
datadir – a directory located under self.devices. This should be one of the DATADIR constants defined in the account, container, and object servers.
suffix – path name suffix required for all names returned (ignored if yield_hash_dirs is True)
mount_check – Flag to check if a mount check should be performed on devices
logger – a logger object
devices_filter – a callable taking (devices, [list of devices]) as parameters and returning a [list of devices]
partitions_filter – a callable taking (datadir_path, [list of parts]) as parameters and returning a [list of parts]
suffixes_filter – a callable taking (part_path, [list of suffixes]) as parameters and returning a [list of suffixes]
hashes_filter – a callable taking (suff_path, [list of hashes]) as parameters and returning a [list of hashes]
hook_pre_device – a callable taking device_path as parameter
hook_post_device – a callable taking device_path as parameter
hook_pre_partition – a callable taking part_path as parameter
hook_post_partition – a callable taking part_path as parameter
hook_pre_suffix – a callable taking suff_path as parameter
hook_post_suffix – a callable taking suff_path as parameter
hook_pre_hash – a callable taking hash_path as parameter
hook_post_hash – a callable taking hash_path as parameter
error_counter – a dictionary used to accumulate error counts; may add keys ‘unmounted’ and ‘unlistable_partitions’
yield_hash_dirs – if True, yield hash dirs instead of individual files

swift.common.utils.backward(f, blocksize=4096)¶

A generator returning lines from a file starting with the last line, then the second last line, etc. i.e., it reads lines backwards. Stops when the first line (if any) is read. This is useful when searching for recent activity in very large files.

Parameters

f – file object to read
blocksize – no of characters to go backwards at each block

swift.common.utils.cache_from_env(env, allow_none=False)¶

Get memcache connection pool from the environment (which had been previously set by the memcache middleware

Parameters: env – wsgi environment dict
Returns: swift.common.memcached.MemcacheRing from environment

swift.common.utils.capture_stdio(logger, **kwargs)¶

Log unhandled exceptions, close stdio, capture stdout and stderr.

param logger: Logger object to use

swift.common.utils.closing_if_possible(maybe_closable)¶

Like contextlib.closing(), but doesn’t crash if the object lacks a close() method.

PEP 333 (WSGI) says: “If the iterable returned by the application has a close() method, the server or gateway must call that method upon completion of the current request[.]” This function makes that easier.

swift.common.utils.compute_eta(start_time, current_value, final_value)¶

Compute an ETA. Now only if we could also have a progress bar…

Parameters

start_time – Unix timestamp when the operation began
current_value – Current value
final_value – Final value

Returns

ETA as a tuple of (length of time, unit of time) where unit of time is one of (‘h’, ‘m’, ‘s’)

swift.common.utils.config_auto_int_value(value, default)¶: Returns default if value is None or ‘auto’. Returns value as an int or raises ValueError otherwise.

swift.common.utils.config_fallocate_value(reserve_value)¶: Returns fallocate reserve_value as an int or float. Returns is_percent as a boolean. Returns a ValueError on invalid fallocate value.

swift.common.utils.config_positive_int_value(value)¶: Returns positive int value if it can be cast by int() and it’s an integer > 0. (not including zero) Raises ValueError otherwise.

swift.common.utils.config_read_prefixed_options(conf, prefix_name, defaults)¶

Read prefixed options from configuration

Parameters

conf – the configuration
prefix_name – the prefix (including, if needed, an underscore)
defaults – a dict of default values. The dict supplies the option name and type (string or comma separated string)

Returns

a dict containing the options

swift.common.utils.config_read_reseller_options(conf, defaults)¶

Read reseller_prefix option and associated options from configuration

Reads the reseller_prefix option, then reads options that may be associated with a specific reseller prefix. Reads options such that an option without a prefix applies to all reseller prefixes unless an option has an explicit prefix.

Parameters

conf – the configuration
defaults – a dict of default values. The key is the option name. The value is either an array of strings or a string

Returns

tuple of an array of reseller prefixes and a dict of option values

swift.common.utils.config_true_value(value)¶: Returns True if the value is either True or a string in TRUE_VALUES. Returns False otherwise.

swift.common.utils.csv_append(csv_string, item)¶

Appends an item to a comma-separated string.

If the comma-separated string is empty/None, just returns item.

swift.common.utils.decode_timestamps(encoded, explicit=False)¶: Parses a string of the form generated by encode_timestamps and returns a tuple of the three component timestamps. If explicit is False, component timestamps that are not explicitly encoded will be assumed to have zero delta from the previous component and therefore take the value of the previous component. If explicit is True, component timestamps that are not explicitly encoded will be returned with value None.

swift.common.utils.distribute_evenly(items, num_buckets)¶: Distribute items as evenly as possible into N buckets.

swift.common.utils.document_iters_to_http_response_body(ranges_iter, boundary, multipart, logger)¶

Takes an iterator of range iters and turns it into an appropriate HTTP response body, whether that’s multipart/byteranges or not.

This is almost, but not quite, the inverse of request_helpers.http_response_to_document_iters(). This function only yields chunks of the body, not any headers.

Parameters

ranges_iter –
an iterator of dictionaries, one per range. Each dictionary must contain at least the following key: “part_iter”: iterator yielding the bytes in the range

Additionally, if multipart is True, then the following other keys are required:

”start_byte”: index of the first byte in the range “end_byte”: index of the last byte in the range “content_type”: value for the range’s Content-Type header

Finally, there is one optional key that is used in the
multipart/byteranges case:

”entity_length”: length of the requested entity (not necessarily
equal to the response length). If omitted, “*” will be used.

Each part_iter will be exhausted prior to calling next(ranges_iter).
boundary – MIME boundary to use, sans dashes (e.g. “boundary”, not “–boundary”).
multipart – True if the response should be multipart/byteranges, False otherwise. This should be True if and only if you have 2 or more ranges.
logger – a logger

swift.common.utils.document_iters_to_multipart_byteranges(ranges_iter, boundary)¶

Takes an iterator of range iters and yields a multipart/byteranges MIME document suitable for sending as the body of a multi-range 206 response.

See document_iters_to_http_response_body for parameter descriptions.

swift.common.utils.drain_and_close(response_or_app_iter)¶

Drain and close a swob or WSGI response.

This ensures we don’t log a 499 in the proxy just because we realized we don’t care about the body of an error.

swift.common.utils.drop_buffer_cache(fd, offset, length)¶

Drop ‘buffer’ cache for the given range of the given file.

Parameters

fd – file descriptor
offset – start offset
length – length

swift.common.utils.drop_privileges(user)¶

Sets the userid/groupid of the current process, get session leader, etc.

Parameters: user – User name to change privileges to

swift.common.utils.dump_recon_cache(cache_dict, cache_file, logger, lock_timeout=2, set_owner=None)¶

Update recon cache values

Parameters

cache_dict – Dictionary of cache key/value pairs to write out
cache_file – cache file to update
logger – the logger to use to log an encountered error
lock_timeout – timeout (in seconds)
set_owner – Set owner of recon cache file

swift.common.utils.encode_timestamps(t1, t2=None, t3=None, explicit=False)¶

Encode up to three timestamps into a string. Unlike a Timestamp object, the encoded string does NOT used fixed width fields and consequently no relative chronology of the timestamps can be inferred from lexicographic sorting of encoded timestamp strings.

The format of the encoded string is:: <t1>[<+/-><t2 - t1>[<+/-><t3 - t2>]]

i.e. if t1 = t2 = t3 then just the string representation of t1 is returned, otherwise the time offsets for t2 and t3 are appended. If explicit is True then the offsets for t2 and t3 are always appended even if zero.

Note: any offset value in t1 will be preserved, but offsets on t2 and t3 are not preserved. In the anticipated use cases for this method (and the inverse decode_timestamps method) the timestamps passed as t2 and t3 are not expected to have offsets as they will be timestamps associated with a POST request. In the case where the encoding is used in a container objects table row, t1 could be the PUT or DELETE time but t2 and t3 represent the content type and metadata times (if different from the data file) i.e. correspond to POST timestamps. In the case where the encoded form is used in a .meta file name, t1 and t2 both correspond to POST timestamps.

swift.common.utils.eventlet_monkey_patch()¶: Install the appropriate Eventlet monkey patches.

swift.common.utils.expand_ipv6(address)¶: Expand ipv6 address. :param address: a string indicating valid ipv6 address :returns: a string indicating fully expanded ipv6 address

swift.common.utils.extract_swift_bytes(content_type)¶

Parse a content-type and return a tuple containing:

the content_type string minus any swift_bytes param,
the swift_bytes value or None if the param was not found

Parameters: content_type – a content-type string
Returns: a tuple of (content-type, swift_bytes or None)

swift.common.utils.fallocate(fd, size, offset=0)¶

Pre-allocate disk space for a file.

This function can be disabled by calling disable_fallocate(). If no suitable C function is available in libc, this function is a no-op.

Parameters

fd – file descriptor
size – size to allocate (in bytes)

swift.common.utils.fdatasync(fd)¶

Sync modified file data to disk.

Parameters: fd – file descriptor

swift.common.utils.find_shard_range(item, ranges)¶

Find a ShardRange in given list of shard_ranges whose namespace contains item.

Parameters

item – The item for a which a ShardRange is to be found.
ranges – a sorted list of ShardRanges.

Returns

the ShardRange whose namespace contains item, or None if no suitable range is found.

swift.common.utils.fs_has_free_space(fs_path, space_needed, is_percent)¶

Check to see whether or not a filesystem has the given amount of space free. Unlike fallocate(), this does not reserve any space.

Parameters

fs_path – path to a file or directory on the filesystem; typically the path to the filesystem’s mount point
space_needed – minimum bytes or percentage of free space
is_percent – if True, then space_needed is treated as a percentage of the filesystem’s capacity; if False, space_needed is a number of free bytes.

Returns

True if the filesystem has at least that much free space, False otherwise

Raises

OSError – if fs_path does not exist

swift.common.utils.fsync(fd)¶

Sync modified file data and metadata to disk.

Parameters: fd – file descriptor

swift.common.utils.fsync_dir(dirpath)¶

Sync directory entries to disk.

Parameters: dirpath – Path to the directory to be synced.

swift.common.utils.get_db_files(db_path)¶

Given the path to a db file, return a sorted list of all valid db files that actually exist in that path’s dir. A valid db filename has the form:

<hash>[_<epoch>].db

where <hash> matches the <hash> part of the given db_path as would be parsed by parse_db_filename().

Parameters: db_path – Path to a db file that does not necessarily exist.
Returns: List of valid db files that do exist in the dir of the db_path. This list may be empty.

swift.common.utils.get_expirer_container(x_delete_at, expirer_divisor, acc, cont, obj)¶: Returns an expiring object container name for given X-Delete-At and (native string) a/c/o.

swift.common.utils.get_hmac(request_method, path, expires, key, digest=<built-in function openssl_sha1>, ip_range=None)¶

Returns the hexdigest string of the HMAC (see RFC 2104) for the request.

Parameters

request_method – Request method to allow.
path – The path to the resource to allow access to.
expires – Unix timestamp as an int for when the URL expires.
key – HMAC shared secret.
digest – constructor for the digest to use in calculating the HMAC Defaults to SHA1
ip_range – The ip range from which the resource is allowed to be accessed. We need to put the ip_range as the first argument to hmac to avoid manipulation of the path due to newlines being valid in paths e.g. /v1/a/c/on127.0.0.1

Returns

hexdigest str of the HMAC for the request using the specified digest algorithm.

swift.common.utils.get_hub()¶

Checks whether poll is available and falls back on select if it isn’t.

Note about epoll:

Review: https://review.opendev.org/#/c/18806/

There was a problem where once out of every 30 quadrillion connections, a coroutine wouldn’t wake up when the client closed its end. Epoll was not reporting the event or it was getting swallowed somewhere. Then when that file descriptor was re-used, eventlet would freak right out because it still thought it was waiting for activity from it in some other coro.

Another note about epoll: it’s hard to use when forking. epoll works like so:

create an epoll instance: efd = epoll_create(…)

register file descriptors of interest with epoll_ctl(efd,
EPOLL_CTL_ADD, fd, …)

wait for events with epoll_wait(efd, …)

If you fork, you and all your child processes end up using the same epoll instance, and everyone becomes confused. It is possible to use epoll and fork and still have a correct program as long as you do the right things, but eventlet doesn’t do those things. Really, it can’t even try to do those things since it doesn’t get notified of forks.

In contrast, both poll() and select() specify the set of interesting file descriptors with each call, so there’s no problem with forking.

As eventlet monkey patching is now done before call get_hub() in wsgi.py if we use ‘import select’ we get the eventlet version, but since version 0.20.0 eventlet removed select.poll() function in patched select (see: http://eventlet.net/doc/changelog.html and https://github.com/eventlet/eventlet/commit/614a20462).

We use eventlet.patcher.original function to get python select module to test if poll() is available on platform.

swift.common.utils.get_log_line(req, res, trans_time, additional_info, fmt, anonymization_method, anonymization_salt)¶

Make a line for logging that matches the documented log line format for backend servers.

Parameters

req – the request.
res – the response.
trans_time – the time the request took to complete, a float.
additional_info – a string to log at the end of the line

Returns

a properly formatted line for logging.

swift.common.utils.get_logger(conf, name=None, log_to_console=False, log_route=None, fmt='%(server)s: %(message)s')¶

Get the current system logger using config settings.

Log config and defaults:

log_facility = LOG_LOCAL0
log_level = INFO
log_name = swift
log_max_line_length = 0
log_udp_host = (disabled)
log_udp_port = logging.handlers.SYSLOG_UDP_PORT
log_address = /dev/log
log_statsd_host = (disabled)
log_statsd_port = 8125
log_statsd_default_sample_rate = 1.0
log_statsd_sample_rate_factor = 1.0
log_statsd_metric_prefix = (empty-string)

Parameters

conf – Configuration dict to read settings from
name – Name of the logger
log_to_console – Add handler which writes to console on stderr
log_route – Route for the logging, not emitted to the log, just used to separate logging configurations
fmt – Override log format

swift.common.utils.get_md5_socket()¶

Get an MD5 socket file descriptor. One can MD5 data with it by writing it to the socket with os.write, then os.read the 16 bytes of the checksum out later.

NOTE: It is the caller’s responsibility to ensure that os.close() is called on the returned file descriptor. This is a bare file descriptor, not a Python object. It doesn’t close itself.

swift.common.utils.get_partition_for_hash(hex_hash, part_power)¶: Return partition number for given hex hash and partition power. :param hex_hash: A hash string :param part_power: partition power :returns: partition number

swift.common.utils.get_policy_index(req_headers, res_headers)¶

Returns the appropriate index of the storage policy for the request from a proxy server

Parameters

req_headers – dict of the request headers.
res_headers – dict of the response headers.

Returns

string index of storage policy, or None

swift.common.utils.get_redirect_data(response)¶

Extract a redirect location from a response’s headers.

Parameters: response – a response
Returns: a tuple of (path, Timestamp) if a Location header is found, otherwise None
Raises: ValueError – if the Location header is found but a X-Backend-Redirect-Timestamp is not found, or if there is a problem with the format of etiher header

swift.common.utils.get_swift_info(admin=False, disallowed_sections=None)¶

Returns information about the swift cluster that has been previously registered with the register_swift_info call.

Parameters

admin – boolean value, if True will additionally return an ‘admin’ section with information previously registered as admin info.
disallowed_sections – list of section names to be withheld from the information returned.

Returns

dictionary of information about the swift cluster.

swift.common.utils.get_time_units(time_amount)¶

Get a nomralized length of time in the largest unit of time (hours, minutes, or seconds.)

Parameters: time_amount – length of time in seconds
Returns: A touple of (length of time, unit of time) where unit of time is one of (‘h’, ‘m’, ‘s’)

swift.common.utils.get_valid_utf8_str(str_or_unicode)¶

Get valid parts of utf-8 str from str, unicode and even invalid utf-8 str

Parameters: str_or_unicode – a string or an unicode which can be invalid utf-8

swift.common.utils.get_zero_indexed_base_string(base, index)¶

This allows the caller to make a list of things with indexes, where the first item (zero indexed) is just the bare base string, and subsequent indexes are appended ‘-1’, ‘-2’, etc.

e.g.:

'lock', None => 'lock'
'lock', 0    => 'lock'
'lock', 1    => 'lock-1'
'object', 2  => 'object-2'

Parameters

base – a string, the base string; when index is 0 (or None) this is the identity function.
index – a digit, typically an integer (or None); for values other than 0 or None this digit is appended to the base string separated by a hyphen.

swift.common.utils.hash_path(account, container=None, object=None, raw_digest=False)¶

Get the canonical hash for an account/container/object

Parameters

account – Account
container – Container
object – Object
raw_digest – If True, return the raw version rather than a hex digest

Returns

hash string

swift.common.utils.human_readable(value)¶: Returns the number in a human readable format; for example 1048576 = “1Mi”.

swift.common.utils.is_valid_ip(ip)¶: Return True if the provided ip is a valid IP-address

swift.common.utils.is_valid_ipv4(ip)¶: Return True if the provided ip is a valid IPv4-address

swift.common.utils.is_valid_ipv6(ip)¶: Returns True if the provided ip is a valid IPv6-address

swift.common.utils.ismount(path)¶: Test whether a path is a mount point. This will catch any exceptions and translate them into a False return value Use ismount_raw to have the exceptions raised instead.

swift.common.utils.ismount_raw(path)¶

Test whether a path is a mount point. Whereas ismount will catch any exceptions and just return False, this raw version will not catch exceptions.

This is code hijacked from C Python 2.6.8, adapted to remove the extra lstat() system call.

swift.common.utils.item_from_env(env, item_name, allow_none=False)¶

Get a value from the wsgi environment

Parameters

env – wsgi environment dict
item_name – name of item to get

Returns

the value from the environment

swift.common.utils.iter_multipart_mime_documents(wsgi_input, boundary, read_chunk_size=4096)¶

Given a multi-part-mime-encoded input file object and boundary, yield file-like objects for each part. Note that this does not split each part into headers and body; the caller is responsible for doing that if necessary.

Parameters

wsgi_input – The file-like object to read from.
boundary – The mime boundary to separate new file-like objects on.

Returns

A generator of file-like objects for each part.

Raises

MimeInvalid – if the document is malformed

swift.common.utils.last_modified_date_to_timestamp(last_modified_date_str)¶: Convert a last modified date (like you’d get from a container listing, e.g. 2014-02-28T23:22:36.698390) to a float.

swift.common.utils.link_fd_to_path(fd, target_path, dirs_created=0, retries=2, fsync=True)¶

Creates a link to file descriptor at target_path specified. This method does not close the fd for you. Unlike rename, as linkat() cannot overwrite target_path if it exists, we unlink and try again.

Attempts to fix / hide race conditions like empty object directories being removed by backend processes during uploads, by retrying.

Parameters

fd – File descriptor to be linked
target_path – Path in filesystem where fd is to be linked
dirs_created – Number of newly created directories that needs to be fsync’d.
retries – number of retries to make
fsync – fsync on containing directory of target_path and also all the newly created directories.

swift.common.utils.list_from_csv(comma_separated_str)¶: Splits the str given and returns a properly stripped list of the comma separated values.

swift.common.utils.load_libc_function(func_name, log_error=True, fail_if_missing=False, errcheck=False)¶

Attempt to find the function in libc, otherwise return a no-op func.

Parameters

func_name – name of the function to pull from libc.
log_error – log an error when a function can’t be found
fail_if_missing – raise an exception when a function can’t be found. Default behavior is to return a no-op function.
errcheck – boolean, if true install a wrapper on the function to check for a return values of -1 and call ctype.get_errno and raise an OSError

swift.common.utils.load_recon_cache(cache_file)¶: Load a recon cache file. Treats missing file as empty.

swift.common.utils.lock_file(filename, timeout=10, append=False, unlink=True)¶

Context manager that acquires a lock on a file. This will block until the lock can be acquired, or the timeout time has expired (whichever occurs first).

Parameters

filename – file to be locked
timeout – timeout (in seconds)
append – True if file should be opened in append mode
unlink – True if the file should be unlinked at the end

swift.common.utils.lock_parent_directory(filename, timeout=10)¶

Context manager that acquires a lock on the parent directory of the given file path. This will block until the lock can be acquired, or the timeout time has expired (whichever occurs first).

Parameters

filename – file path of the parent directory to be locked
timeout – timeout (in seconds)

swift.common.utils.lock_path(directory, timeout=10, timeout_class=None, limit=1, name=None)¶

Context manager that acquires a lock on a directory. This will block until the lock can be acquired, or the timeout time has expired (whichever occurs first).

For locking exclusively, file or directory has to be opened in Write mode. Python doesn’t allow directories to be opened in Write Mode. So we workaround by locking a hidden file in the directory.

Parameters

directory – directory to be locked
timeout – timeout (in seconds)
timeout_class – The class of the exception to raise if the lock cannot be granted within the timeout. Will be constructed as timeout_class(timeout, lockpath). Default: LockTimeout
limit – The maximum number of locks that may be held concurrently on the same directory at the time this method is called. Note that this limit is only applied during the current call to this method and does not prevent subsequent calls giving a larger limit. Defaults to 1.
name – A string to distinguishes different type of locks in a directory

Raises

TypeError – if limit is not an int.
ValueError – if limit is less than 1.

swift.common.utils.make_db_file_path(db_path, epoch)¶

Given a path to a db file, return a modified path whose filename part has the given epoch.

A db filename takes the form <hash>[_<epoch>].db; this method replaces the <epoch> part of the given db_path with the given epoch value, or drops the epoch part if the given epoch is None.

Parameters

db_path – Path to a db file that does not necessarily exist.
epoch – A string (or None) that will be used as the epoch in the new path’s filename; non-None values will be normalized to the normal string representation of a Timestamp.

Returns

A modified path to a db file.

Raises

ValueError – if the epoch is not valid for constructing a Timestamp.

swift.common.utils.makedirs_count(path, count=0)¶

Same as os.makedirs() except that this method returns the number of new directories that had to be created.

Also, this does not raise an error if target directory already exists. This behaviour is similar to Python 3.x’s os.makedirs() called with exist_ok=True. Also similar to swift.common.utils.mkdirs()

https://hg.python.org/cpython/file/v3.4.2/Lib/os.py#l212

swift.common.utils.maybe_multipart_byteranges_to_document_iters(app_iter, content_type)¶

Takes an iterator that may or may not contain a multipart MIME document as well as content type and returns an iterator of body iterators.

Parameters

app_iter – iterator that may contain a multipart MIME document
content_type – content type of the app_iter, used to determine whether it conains a multipart document and, if so, what the boundary is between documents

swift.common.utils.md5(string=b'', usedforsecurity=True)¶

Return an md5 hashlib object using usedforsecurity parameter

For python distributions that support the usedforsecurity keyword parameter, this passes the parameter through as expected. See https://bugs.python.org/issue9216

swift.common.utils.md5_hash_for_file(fname)¶

Get the MD5 checksum of a file.

Parameters: fname – path to file
Returns: MD5 checksum, hex encoded

swift.common.utils.mime_to_document_iters(input_file, boundary, read_chunk_size=4096)¶

Takes a file-like object containing a multipart MIME document and returns an iterator of (headers, body-file) tuples.

Parameters

input_file – file-like object with the MIME doc in it
boundary – MIME boundary, sans dashes (e.g. “divider”, not “–divider”)
read_chunk_size – size of strings read via input_file.read()

swift.common.utils.mkdirs(path)¶

Ensures the path is a directory or makes it if not. Errors if the path exists but is a file or on permissions failure.

Parameters: path – path to create

swift.common.utils.modify_priority(conf, logger)¶: Modify priority by nice and ionice.

swift.common.utils.multipart_byteranges_to_document_iters(input_file, boundary, read_chunk_size=4096)¶

Takes a file-like object containing a multipart/byteranges MIME document (see RFC 7233, Appendix A) and returns an iterator of (first-byte, last-byte, length, document-headers, body-file) 5-tuples.

Parameters

input_file – file-like object with the MIME doc in it
boundary – MIME boundary, sans dashes (e.g. “divider”, not “–divider”)
read_chunk_size – size of strings read via input_file.read()

swift.common.utils.normalize_delete_at_timestamp(timestamp, high_precision=False)¶

Format a timestamp (string or numeric) into a standardized xxxxxxxxxx (10) or xxxxxxxxxx.xxxxx (10.5) format.

Note that timestamps less than 0000000000 are raised to 0000000000 and values greater than November 20th, 2286 at 17:46:39 UTC will be capped at that date and time, resulting in no return value exceeding 9999999999.99999 (or 9999999999 if using low-precision).

This cap is because the expirer is already working through a sorted list of strings that were all a length of 10. Adding another digit would mess up the sort and cause the expirer to break from processing early. By 2286, this problem will need to be fixed, probably by creating an additional .expiring_objects account to work from with 11 (or more) digit container names.

Parameters: timestamp – unix timestamp
Returns: normalized timestamp as a string

swift.common.utils.normalize_timestamp(timestamp)¶

Format a timestamp (string or numeric) into a standardized xxxxxxxxxx.xxxxx (10.5) format.

Note that timestamps using values greater than or equal to November 20th, 2286 at 17:46 UTC will use 11 digits to represent the number of seconds.

Parameters: timestamp – unix timestamp
Returns: normalized timestamp as a string

swift.common.utils.override_bytes_from_content_type(listing_dict, logger=None)¶: Takes a dict from a container listing and overrides the content_type, bytes fields if swift_bytes is set.

swift.common.utils.pairs(item_list)¶

Returns an iterator of all pairs of elements from item_list.

Parameters: item_list – items (no duplicates allowed)

swift.common.utils.parse_content_disposition(header)¶

Given the value of a header like: Content-Disposition: form-data; name=”somefile”; filename=”test.html”

Return data like (“form-data”, {“name”: “somefile”, “filename”: “test.html”})

Parameters: header – Value of a header (the part after the ‘: ‘).
Returns: (value name, dict) of the attribute data parsed (see above).

swift.common.utils.parse_content_range(content_range)¶

Parse a content-range header into (first_byte, last_byte, total_size).

See RFC 7233 section 4.2 for details on the header format, but it’s basically “Content-Range: bytes ${start}-${end}/${total}”.

Parameters: content_range – Content-Range header value to parse, e.g. “bytes 100-1249/49004”
Returns: 3-tuple (start, end, total)
Raises: ValueError – if malformed

swift.common.utils.parse_content_type(content_type)¶

Parse a content-type and its parameters into values. RFC 2616 sec 14.17 and 3.7 are pertinent.

Examples:

'text/plain; charset=UTF-8' -> ('text/plain', [('charset, 'UTF-8')])
'text/plain; charset=UTF-8; level=1' ->
    ('text/plain', [('charset, 'UTF-8'), ('level', '1')])

Parameters: content_type – content_type to parse
Returns: a tuple containing (content type, list of k, v parameter tuples)

swift.common.utils.parse_db_filename(filename)¶

Splits a db filename into three parts: the hash, the epoch, and the extension.

>>> parse_db_filename("ab2134.db")
('ab2134', None, '.db')
>>> parse_db_filename("ab2134_1234567890.12345.db")
('ab2134', '1234567890.12345', '.db')

Parameters: filename – A db file basename or path to a db file.
Returns: A tuple of (hash , epoch, extension). epoch may be None.
Raises: ValueError – if filename is not a path to a file.

swift.common.utils.parse_mime_headers(doc_file)¶

Takes a file-like object containing a MIME document and returns a HeaderKeyDict containing the headers. The body of the message is not consumed: the position in doc_file is left at the beginning of the body.

This function was inspired by the Python standard library’s http.client.parse_headers.

Parameters: doc_file – binary file-like object containing a MIME document
Returns: a swift.common.swob.HeaderKeyDict containing the headers

swift.common.utils.parse_options(parser=None, once=False, test_args=None)¶

Parse standard swift server/daemon options with optparse.OptionParser.

Parameters

parser – OptionParser to use. If not sent one will be created.
once – Boolean indicating the “once” option is available
test_args – Override sys.argv; used in testing

Returns

Tuple of (config, options); config is an absolute path to the config file, options is the parser options as a dictionary.

Raises

SystemExit – First arg (CONFIG) is required, file must exist

swift.common.utils.parse_override_options(**kwargs)¶

Figure out which policies, devices, and partitions we should operate on, based on kwargs.

If ‘override_policies’ is already present in kwargs, then return that value. This happens when using multiple worker processes; the parent process supplies override_policies=X to each child process.

Otherwise, in run-once mode, look at the ‘policies’ keyword argument. This is the value of the “–policies” command-line option. In run-forever mode or if no –policies option was provided, an empty list will be returned.

The procedures for devices and partitions are similar.

Returns: a named tuple with fields “devices”, “partitions”, and “policies”.

swift.common.utils.parse_prefixed_conf(conf_file, prefix)¶

Search the config file for any common-prefix sections and load those sections to a dict mapping the after-prefix reference to options.

Parameters

conf_file – the file name of the config to parse
prefix – the common prefix of the sections

Returns

a dict mapping policy reference -> dict of policy options

Raises

ValueError – if a policy config section has an invalid name

swift.common.utils.parse_socket_string(socket_string, default_port)¶

Given a string representing a socket, returns a tuple of (host, port). Valid strings are DNS names, IPv4 addresses, or IPv6 addresses, with an optional port. If an IPv6 address is specified it must be enclosed in [], like [::1] or [::1]:11211. This follows the accepted prescription for IPv6 host literals.

Examples:

server.org
server.org:1337
127.0.0.1:1337
[::1]:1337
[::1]

swift.common.utils.private(func)¶

Decorator to declare which methods are privately accessible as HTTP requests with an X-Backend-Allow-Private-Methods: True override

Parameters: func – function to make private

swift.common.utils.public(func)¶

Decorator to declare which methods are publicly accessible as HTTP requests

Parameters: func – function to make public

swift.common.utils.punch_hole(fd, offset, length)¶

De-allocate disk space in the middle of a file.

Parameters

fd – file descriptor
offset – index of first byte to de-allocate
length – number of bytes to de-allocate

swift.common.utils.put_recon_cache_entry(cache_entry, key, item)¶

Update a recon cache entry item.

If item is an empty dict then any existing key in cache_entry will be deleted. Similarly if item is a dict and any of its values are empty dicts then the corrsponsing key will be deleted from the nested dict in cache_entry.

We use nested recon cache entries when the object auditor runs in parallel or else in ‘once’ mode with a specified subset of devices.

Parameters

cache_entry – a dict of existing cache entries
key – key for item to update
item – value for item to update

swift.common.utils.quorum_size(n)¶

quorum size as it applies to services that use ‘replication’ for data integrity (Account/Container services). Object quorum_size is defined on a storage policy basis.

Number of successful backend requests needed for the proxy to consider the client request successful.

swift.common.utils.quote(value, safe='/')¶: Patched version of urllib.quote that encodes utf-8 strings before quoting

swift.common.utils.random() → x in the interval [0, 1).¶

swift.common.utils.ratelimit_sleep(running_time, max_rate, incr_by=1, rate_buffer=5)¶

Will eventlet.sleep() for the appropriate time so that the max_rate is never exceeded. If max_rate is 0, will not ratelimit. The maximum recommended rate should not exceed (1000 * incr_by) a second as eventlet.sleep() does involve some overhead. Returns running_time that should be used for subsequent calls.

Parameters

running_time – the running time in milliseconds of the next allowable request. Best to start at zero.
max_rate – The maximum rate per second allowed for the process.
incr_by – How much to increment the counter. Useful if you want to ratelimit 1024 bytes/sec and have differing sizes of requests. Must be > 0 to engage rate-limiting behavior.
rate_buffer – Number of seconds the rate counter can drop and be allowed to catch up (at a faster than listed rate). A larger number will result in larger spikes in rate but better average accuracy. Must be > 0 to engage rate-limiting behavior.

swift.common.utils.readconf(conf_path, section_name=None, log_name=None, defaults=None, raw=False)¶

Read config file(s) and return config items as a dict

Parameters

conf_path – path to config file/directory, or a file-like object (hasattr readline)
section_name – config section to read (will return all sections if not defined)
log_name – name to be used with logging (will use section_name if not defined)
defaults – dict of default values to pre-populate the config with

Returns

dict of config items

Raises

ValueError – if section_name does not exist
IOError – if reading the file failed

swift.common.utils.register_swift_info(name='swift', admin=False, **kwargs)¶

Registers information about the swift cluster to be retrieved with calls to get_swift_info.

NOTE: Do not use “.” in the param: name or any keys in kwargs. “.” is used: in the disallowed_sections to remove unwanted keys from /info.

Parameters

name – string, the section name to place the information under.
admin – boolean, if True, information will be registered to an admin section which can optionally be withheld when requesting the information.
kwargs – key value arguments representing the information to be added.

Raises

ValueError – if name or any of the keys in kwargs has “.” in it

swift.common.utils.reiterate(iterable)¶: Consume the first truthy item from an iterator, then re-chain it to the rest of the iterator. This is useful when you want to make sure the prologue to downstream generators have been executed before continuing. :param iterable: an iterable object

swift.common.utils.remove_directory(path)¶

Wrapper for os.rmdir, ENOENT and ENOTEMPTY are ignored

Parameters: path – first and only argument passed to os.rmdir

swift.common.utils.remove_file(path)¶

Quiet wrapper for os.unlink, OSErrors are suppressed

Parameters: path – first and only argument passed to os.unlink

swift.common.utils.renamer(old, new, fsync=True)¶

Attempt to fix / hide race conditions like empty object directories being removed by backend processes during uploads, by retrying.

The containing directory of ‘new’ and of all newly created directories are fsync’d by default. This _will_ come at a performance penalty. In cases where these additional fsyncs are not necessary, it is expected that the caller of renamer() turn it off explicitly.

Parameters

old – old path to be renamed
new – new path to be renamed to
fsync – fsync on containing directory of new and also all the newly created directories.

swift.common.utils.replace_partition_in_path(path, part_power, is_hash_dir=False)¶

Takes a path and a partition power and returns the same path, but with the correct partition number. Most useful when increasing the partition power.

Parameters

path – full path to a file, for example object .data file
part_power – partition power to compute correct partition number
is_hash_dir – if True then path is the path to a hash dir, otherwise path is the path to a file in a hash dir.

Returns

Path with re-computed partition power

swift.common.utils.replication(func)¶

Decorator to declare which methods are accessible for different type of servers:

If option replication_server is None then this decorator doesn’t matter.
If option replication_server is True then ONLY decorated with this decorator methods will be started.
If option replication_server is False then decorated with this decorator methods will NOT be started.

Parameters: func – function to mark accessible for replication

swift.common.utils.round_robin_iter(its)¶: Takes a list of iterators, yield an element from each in a round-robin fashion until all of them are exhausted. :param its: list of iterators

swift.common.utils.rsync_ip(ip)¶

Transform ip string to an rsync-compatible form

Will return ipv4 addresses unchanged, but will nest ipv6 addresses inside square brackets.

Parameters: ip – an ip string (ipv4 or ipv6)
Returns: a string ip address

swift.common.utils.rsync_module_interpolation(template, device)¶

Interpolate devices variables inside a rsync module template

Parameters

template – rsync module template as a string
device – a device from a ring

Returns

a string with all variables replaced by device attributes

swift.common.utils.search_tree(root, glob_match, ext='', exts=None, dir_ext=None)¶

Look in root, for any files/dirs matching glob, recursively traversing any found directories looking for files ending with ext

Parameters

root – start of search path
glob_match – glob to match in root, matching dirs are traversed with os.walk
ext – only files that end in ext will be returned
exts – a list of file extensions; only files that end in one of these extensions will be returned; if set this list overrides any extension specified using the ‘ext’ param.
dir_ext – if present directories that end with dir_ext will not be traversed and instead will be returned as a matched path

Returns

list of full paths to matching files, sorted

swift.common.utils.set_swift_dir(swift_dir)¶

Sets the directory from which swift config files will be read. If the given directory differs from that already set then the swift.conf file in the new directory will be validated and storage policies will be reloaded from the new swift.conf file.

Parameters: swift_dir – non-default directory to read swift.conf from

class swift.common.utils.sockaddr_alg¶: Bases: _ctypes.Structure

swift.common.utils.split_path(path, minsegs=1, maxsegs=None, rest_with_last=False)¶

Validate and split the given HTTP request path.

Examples:

['a'] = split_path('/a')
['a', None] = split_path('/a', 1, 2)
['a', 'c'] = split_path('/a/c', 1, 2)
['a', 'c', 'o/r'] = split_path('/a/c/o/r', 1, 3, True)

Parameters

path – HTTP Request path to be split
minsegs – Minimum number of segments to be extracted
maxsegs – Maximum number of segments to be extracted
rest_with_last – If True, trailing data will be returned as part of last segment. If False, and there is trailing data, raises ValueError.

Returns

list of segments with a length of maxsegs (non-existent segments will return as None)

Raises

ValueError – if given an invalid path

swift.common.utils.storage_directory(datadir, partition, name_hash)¶

Get the storage directory

Parameters

datadir – Base data directory
partition – Partition
name_hash – Account, container or object name hash

Returns

Storage directory

swift.common.utils.streq_const_time(s1, s2)¶

Constant-time string comparison.

Params s1: the first string
Params s2: the second string
Returns: True if the strings are equal.

This function takes two strings and compares them. It is intended to be used when doing a comparison for authentication purposes to help guard against timing attacks.

swift.common.utils.strict_b64decode(value, allow_line_breaks=False)¶

Validate and decode Base64-encoded data.

The stdlib base64 module silently discards bad characters, but we often want to treat them as an error.

Parameters

value – some base64-encoded data
allow_line_breaks – if True, ignore carriage returns and newlines

Returns

the decoded data

Raises

ValueError – if value is not a string, contains invalid characters, or has insufficient padding

swift.common.utils.systemd_notify(logger=None)¶

Notify the service manager that started this process, if it is systemd-compatible, that this process correctly started. To do so, it communicates through a Unix socket stored in environment variable NOTIFY_SOCKET. More information can be found in systemd documentation: https://www.freedesktop.org/software/systemd/man/sd_notify.html

Parameters: logger – a logger object

swift.common.utils.timing_stats(**dec_kwargs)¶: Returns a decorator that logs timing events or errors for public methods in swift’s wsgi server controllers, based on response code.

swift.common.utils.unlink_older_than(path, mtime)¶

Remove any file in a given path that was last modified before mtime.

Parameters

path – path to remove file from
mtime – timestamp of oldest file to keep

swift.common.utils.unlink_paths_older_than(filepaths, mtime)¶

Remove any files from the given list that were last modified before mtime.

Parameters

filepaths – a list of strings, the full paths of files to check
mtime – timestamp of oldest file to keep

swift.common.utils.validate_device_partition(device, partition)¶

Validate that a device and a partition are valid and won’t lead to directory traversal when used.

Parameters

device – device to validate
partition – partition to validate

Raises

ValueError – if given an invalid device or partition

swift.common.utils.validate_sync_to(value, allowed_sync_hosts, realms_conf)¶

Validates an X-Container-Sync-To header value, returning the validated endpoint, realm, and realm_key, or an error string.

Parameters

value – The X-Container-Sync-To header value to validate.
allowed_sync_hosts – A list of allowed hosts in endpoints, if realms_conf does not apply.
realms_conf – An instance of swift.common.container_sync_realms.ContainerSyncRealms to validate against.

Returns

A tuple of (error_string, validated_endpoint, realm, realm_key). The error_string will None if the rest of the values have been validated. The validated_endpoint will be the validated endpoint to sync to. The realm and realm_key will be set if validation was done through realms_conf.

swift.common.utils.whataremyips(bind_ip=None)¶

Get “our” IP addresses (“us” being the set of services configured by one *.conf file). If our REST listens on a specific address, return it. Otherwise, if listen on ‘0.0.0.0’ or ‘::’ return all addresses, including the loopback.

Parameters: bind_ip (str) – Optional bind_ip from a config file; may be IP address or hostname.
Returns: list of Strings of ip addresses

swift.common.utils.write_file(path, contents)¶

Write contents to file at path

Parameters

path – any path, subdirs will be created as needed
contents – data to write to file, will be converted to string

swift.common.utils.write_pickle(obj, dest, tmp=None, pickle_protocol=0)¶

Ensure that a pickle file gets written to disk. The file is first written to a tmp location, ensure it is synced to disk, then perform a move to its final location

Parameters

obj – python object to be pickled
dest – path of final destination file
tmp – path to tmp to use, defaults to None
pickle_protocol – protocol to pickle the obj with, defaults to 0

WSGI¶

Storage Policy¶

class swift.common.storage_policy.BaseStoragePolicy(idx, name='', is_default=False, is_deprecated=False, object_ring=None, aliases='', diskfile_module='egg:swift#replication.fs')¶

Bases: object

Represents a storage policy. Not meant to be instantiated directly; implement a derived subclasses (e.g. StoragePolicy, ECStoragePolicy, etc) or use reload_storage_policies() to load POLICIES from swift.conf.

The object_ring property is lazy loaded once the service’s swift_dir is known via get_object_ring(), but it may be over-ridden via object_ring kwarg at create time for testing or actively loaded with load_ring().

add_name(name)¶

Adds an alias name to the storage policy. Shouldn’t be called directly from the storage policy but instead through the storage policy collection class, so lookups by name resolve correctly.

Parameters: name – a new alias for the storage policy

change_primary_name(name)¶

Changes the primary/default name of the policy to a specified name.

Parameters: name – a string name to replace the current primary name.

get_diskfile_manager(*args, **kwargs)¶

Return an instance of the diskfile manager class configured for this storage policy.

Parameters

args – positional args to pass to the diskfile manager constructor.
kwargs – keyword args to pass to the diskfile manager constructor.

Returns

A disk file manager instance.

get_info(config=False)¶

Return the info dict and conf file options for this policy.

Parameters: config – boolean, if True all config options are returned

load_ring(swift_dir)¶

Load the ring for this policy immediately.

Parameters: swift_dir – path to rings

property quorum¶: Number of successful backend requests needed for the proxy to consider the client request successful.

classmethod register(policy_type)¶: Decorator for Storage Policy implementations to register their StoragePolicy class. This will also set the policy_type attribute on the registered implementation.

remove_name(name)¶

Removes an alias name from the storage policy. Shouldn’t be called directly from the storage policy but instead through the storage policy collection class, so lookups by name resolve correctly. If the name removed is the primary name then the next available alias will be adopted as the new primary name.

Parameters: name – a name assigned to the storage policy

class swift.common.storage_policy.ECStoragePolicy(idx, name='', aliases='', is_default=False, is_deprecated=False, object_ring=None, diskfile_module='egg:swift#erasure_coding.fs', ec_segment_size=1048576, ec_type=None, ec_ndata=None, ec_nparity=None, ec_duplication_factor=1)¶

Bases: swift.common.storage_policy.BaseStoragePolicy

Represents a storage policy of type ‘erasure_coding’.

Not meant to be instantiated directly; use reload_storage_policies() to load POLICIES from swift.conf.

property ec_scheme_description¶: This short hand form of the important parts of the ec schema is stored in Object System Metadata on the EC Fragment Archives for debugging.

property fragment_size¶

Maximum length of a fragment, including header.

NB: a fragment archive is a sequence of 0 or more max-length fragments followed by one possibly-shorter fragment.

get_backend_index(node_index)¶

Backend index for PyECLib

Parameters: node_index – integer of node index
Returns: integer of actual fragment index. if param is not an integer, return None instead

get_info(config=False)¶

Return the info dict and conf file options for this policy.

Parameters: config – boolean, if True all config options are returned

load_ring(swift_dir)¶

Load the ring for this policy immediately.

Parameters: swift_dir – path to rings

property quorum¶

Number of successful backend requests needed for the proxy to consider the client PUT request successful.

The quorum size for EC policies defines the minimum number of data + parity elements required to be able to guarantee the desired fault tolerance, which is the number of data elements supplemented by the minimum number of parity elements required by the chosen erasure coding scheme.

For example, for Reed-Solomon, the minimum number parity elements required is 1, and thus the quorum_size requirement is ec_ndata + 1.

Given the number of parity elements required is not the same for every erasure coding scheme, consult PyECLib for min_parity_fragments_needed()

exception swift.common.storage_policy.PolicyError(msg, index=None)¶: Bases: ValueError

class swift.common.storage_policy.StoragePolicy(idx, name='', is_default=False, is_deprecated=False, object_ring=None, aliases='', diskfile_module='egg:swift#replication.fs')¶

Bases: swift.common.storage_policy.BaseStoragePolicy

Represents a storage policy of type ‘replication’. Default storage policy class unless otherwise overridden from swift.conf.

Not meant to be instantiated directly; use reload_storage_policies() to load POLICIES from swift.conf.

property quorum¶

Quorum concept in the replication case:: floor(number of replica / 2) + 1

class swift.common.storage_policy.StoragePolicyCollection(pols)¶

Bases: object

This class represents the collection of valid storage policies for the cluster and is instantiated as StoragePolicy objects are added to the collection when swift.conf is parsed by parse_storage_policies().

When a StoragePolicyCollection is created, the following validation is enforced:

If a policy with index 0 is not declared and no other policies defined, Swift will create one
The policy index must be a non-negative integer
If no policy is declared as the default and no other policies are defined, the policy with index 0 is set as the default
Policy indexes must be unique
Policy names are required
Policy names are case insensitive
Policy names must contain only letters, digits or a dash
Policy names must be unique
The policy name ‘Policy-0’ can only be used for the policy with index 0
If any policies are defined, exactly one policy must be declared default
Deprecated policies can not be declared the default

add_policy_alias(policy_index, *aliases)¶

Adds a new name or names to a policy

Parameters

policy_index – index of a policy in this policy collection.
aliases – arbitrary number of string policy names to add.

change_policy_primary_name(policy_index, new_name)¶

Changes the primary or default name of a policy. The new primary name can be an alias that already belongs to the policy or a completely new name.

Parameters

policy_index – index of a policy in this policy collection.
new_name – a string name to set as the new default name.

get_by_index(index)¶

Find a storage policy by its index.

An index of None will be treated as 0.

Parameters: index – numeric index of the storage policy
Returns: storage policy, or None if no such policy

get_by_name(name)¶

Find a storage policy by its name.

Parameters: name – name of the policy
Returns: storage policy, or None

get_object_ring(policy_idx, swift_dir)¶

Get the ring object to use to handle a request based on its policy.

An index of None will be treated as 0.

Parameters

policy_idx – policy index as defined in swift.conf
swift_dir – swift_dir used by the caller

Returns

appropriate ring object

get_policy_info()¶

Build info about policies for the /info endpoint

Returns: list of dicts containing relevant policy information

remove_policy_alias(*aliases)¶

Removes a name or names from a policy. If the name removed is the primary name then the next available alias will be adopted as the new primary name.

Parameters: aliases – arbitrary number of existing policy names to remove.

class swift.common.storage_policy.StoragePolicySingleton¶

Bases: object

An instance of this class is the primary interface to storage policies exposed as a module level global named POLICIES. This global reference wraps _POLICIES which is normally instantiated by parsing swift.conf and will result in an instance of StoragePolicyCollection.

You should never patch this instance directly, instead patch the module level _POLICIES instance so that swift code which imported POLICIES directly will reference the patched StoragePolicyCollection.

swift.common.storage_policy.get_policy_string(base, policy_or_index)¶

Helper function to construct a string from a base and the policy. Used to encode the policy index into either a file name or a directory name by various modules.

Parameters

base – the base string
policy_or_index – StoragePolicy instance, or an index (string or int), if None the legacy storage Policy-0 is assumed.

Returns

base name with policy index added

Raises

PolicyError – if no policy exists with the given policy_index

swift.common.storage_policy.parse_storage_policies(conf)¶

Parse storage policies in swift.conf - note that validation is done when the StoragePolicyCollection is instantiated.

Parameters: conf – ConfigParser parser object for swift.conf

swift.common.storage_policy.reload_storage_policies()¶: Reload POLICIES from swift.conf.

swift.common.storage_policy.split_policy_string(policy_string)¶

Helper function to convert a string representing a base and a policy. Used to decode the policy from either a file name or a directory name by various modules.

Parameters: policy_string – base name with policy index added
Raises: PolicyError – if given index does not map to a valid policy
Returns: a tuple, in the form (base, policy) where base is the base string and policy is the StoragePolicy instance for the index encoded in the policy_string.

Misc

Misc¶

ACLs¶

Buffered HTTP¶

Constraints¶

Container Sync Realms¶

Direct Client¶

Exceptions¶

Internal Client¶

Manager¶

MemCacheD¶

Request Helpers¶

Swob¶

Utils¶

WSGI¶

Storage Policy¶

Swift 2.27.0

Page Contents