Merge pull request #338 from joke2k/develop

Release v0.8.0

CHANGELOG.rst

@@ -5,6 +5,33 @@ All notable changes to this project will be documented in this file.
 The format is inspired by `Keep a Changelog <https://keepachangelog.com/en/1.0.0/>`_
 and this project adheres to `Semantic Versioning <https://semver.org/spec/v2.0.0.html>`_.
 
+`v0.8.0`_ - 17-October-2021
+------------------------------
+Added
++++++
+- Log invalid lines when parse .env file
+  `#283 <https://github.com/joke2k/django-environ/pull/283>`_.
+- Added docker-style file variable support
+  `#189 <https://github.com/joke2k/django-environ/issues/189>`_.
+- Added option to override existing variables with ``read_env``
+  `#103 <https://github.com/joke2k/django-environ/issues/103>`_,
+  `#249 <https://github.com/joke2k/django-environ/issues/249>`_.
+- Added support for empty var with None default value
+  `#209 <https://github.com/joke2k/django-environ/issues/209>`_.
+- Added ``pymemcache`` cache backend for Django 3.2+
+  `#335 <https://github.com/joke2k/django-environ/pull/335>`_.
+
+
+Fixed
++++++
+- Keep newline/tab escapes in quoted strings
+  `#296 <https://github.com/joke2k/django-environ/pull/296>`_.
+- Handle escaped dollar sign in values
+  `#271 <https://github.com/joke2k/django-environ/issues/271>`_.
+- Fixed incorrect parsing of ``DATABASES_URL`` for Google Cloud MySQL
+  `#294 <https://github.com/joke2k/django-environ/issues/294>`_.
+
+
 `v0.7.0`_ - 11-September-2021
 ------------------------------
 Added
@@ -219,6 +246,7 @@ Added
 - Initial release.
 
 
+.. _v0.8.0: https://github.com/joke2k/django-environ/compare/v0.7.0...v0.8.0
 .. _v0.7.0: https://github.com/joke2k/django-environ/compare/v0.6.0...v0.7.0
 .. _v0.6.0: https://github.com/joke2k/django-environ/compare/v0.5.0...v0.6.0
 .. _v0.5.0: https://github.com/joke2k/django-environ/compare/v0.4.5...v0.5.0

docs/tips.rst

@@ -128,15 +128,54 @@ You can use something like this to handle similar cases.
 Multiline value
 ===============
 
-You can set a multiline variable value:
+To get multiline value pass ``multiline=True`` to ```str()```.
+
+.. note::
+
+   You shouldn't escape newline/tab characters yourself if you want to preserve
+   the formatting.
+
+The following example demonstrates the above:
+
+**.env file**:
+
+.. code-block:: shell
+
+   # .env file contents
+   UNQUOTED_CERT=---BEGIN---\r\n---END---
+   QUOTED_CERT="---BEGIN---\r\n---END---"
+   ESCAPED_CERT=---BEGIN---\\n---END---
+
+**settings.py file**:
 
 .. code-block:: python
 
-   # MULTILINE_TEXT=Hello\\nWorld
-   >>> print env.str('MULTILINE_TEXT', multiline=True)
-   Hello
-   World
+   # settings.py file contents
+   import environ
+
+
+   env = environ.Env()
+
+   print(env.str('UNQUOTED_CERT', multiline=True))
+   # ---BEGIN---
+   # ---END---
+
+   print(env.str('UNQUOTED_CERT', multiline=False))
+   # ---BEGIN---\r\n---END---
+
+   print(env.str('QUOTED_CERT', multiline=True))
+   # ---BEGIN---
+   # ---END---
+
+   print(env.str('QUOTED_CERT', multiline=False))
+   # ---BEGIN---\r\n---END---
 
+   print(env.str('ESCAPED_CERT', multiline=True))
+   # ---BEGIN---\
+   # ---END---
+
+   print(env.str('ESCAPED_CERT', multiline=False))
+   # ---BEGIN---\\n---END---
 
 Proxy value
 ===========
@@ -156,10 +195,30 @@ Values that being with a ``$`` may be interpolated. Pass ``interpolate=True`` to
    FOO
 
 
+Escape Proxy
+============
+
+If you're having trouble with values starting with dollar sign ($) without the intention of proxying the value to
+another, You should enbale the ``escape_proxy`` and prepend a backslash to it.
+
+.. code-block:: python
+
+    import environ
+
+    env = environ.Env()
+    env.escape_proxy = True
+
+    # ESCAPED_VAR=\$baz
+    env.str('ESCAPED_VAR')  # $baz
+
+
+Reading env files
+=================
+
 .. _multiple-env-files-label:
 
 Multiple env files
-==================
+------------------
 
 There is an ability point to the .env file ___location using an environment
 variable. This feature may be convenient in a production systems with a
@@ -188,7 +247,7 @@ while ``./manage.py runserver`` uses ``.env``.
 
 
 Using Path objects when reading env
-===================================
+-----------------------------------
 
 It is possible to use of ``pathlib.Path`` objects when reading environment file from the filesystem:
 
@@ -210,3 +269,16 @@ It is possible to use of ``pathlib.Path`` objects when reading environment file
    env.read_env(os.path.join(BASE_DIR, '.env'))
    env.read_env(pathlib.Path(str(BASE_DIR)).joinpath('.env'))
    env.read_env(pathlib.Path(str(BASE_DIR)) / '.env')
+
+
+Overwriting existing environment values from env files
+------------------------------------------------------
+
+If you want variables set within your env files to take higher precidence than
+an existing set environment variable, use the ``overwrite=True`` argument of
+``read_env``. For example:
+
+.. code-block:: python
+
+   env = environ.Env()
+   env.read_env(BASE_DIR('.env'), overwrite=True)

docs/types.rst

@@ -34,8 +34,10 @@ Supported types
   * Dummy: ``dummycache://``
   * File: ``filecache://``
   * Memory: ``locmemcache://``
-  * Memcached: ``memcache://``
-  * Python memory: ``pymemcache://``
+  * Memcached:
+    * ``memcache://`` (uses ``python-memcached`` backend, deprecated in Django 3.2)
+    * ``pymemcache://`` (uses ``pymemcache`` backend if Django >=3.2 and package is installed, otherwise will use ``pylibmc`` backend to keep config backwards compatibility)
+    * ``pylibmc://``
   * Redis: ``rediscache://``, ``redis://``, or ``rediss://``
 
 * ``search_url``

environ/__init__.py

@@ -31,7 +31,7 @@
 
 
 __copyright__ = 'Copyright (C) 2021 Daniele Faraglia'
-__version__ = '0.7.0'
+__version__ = '0.8.0'
 __license__ = 'MIT'
 __author__ = 'Daniele Faraglia'
 __author_email__ = '[email protected]'

environ/compat.py

@@ -8,15 +8,15 @@
 
 """This module handles import compatibility issues."""
 
-import pkgutil
+from pkgutil import find_loader
 
 
-if pkgutil.find_loader('simplejson'):
+if find_loader('simplejson'):
     import simplejson as json
 else:
     import json
 
-if pkgutil.find_loader('django'):
+if find_loader('django'):
     from django import VERSION as DJANGO_VERSION
     from django.core.exceptions import ImproperlyConfigured
 else:
@@ -33,7 +33,20 @@ class ImproperlyConfigured(Exception):
     DJANGO_POSTGRES = 'django.db.backends.postgresql'
 
 # back compatibility with redis_cache package
-if pkgutil.find_loader('redis_cache'):
+if find_loader('redis_cache'):
     REDIS_DRIVER = 'redis_cache.RedisCache'
 else:
     REDIS_DRIVER = 'django_redis.cache.RedisCache'
+
+
+# back compatibility for pymemcache
+def choose_pymemcache_driver():
+    old_django = DJANGO_VERSION is not None and DJANGO_VERSION < (3, 2)
+    if old_django or not find_loader('pymemcache'):
+        # The original backend choice for the 'pymemcache' scheme is
+        # unfortunately 'pylibmc'.
+        return 'django.core.cache.backends.memcached.PyLibMCCache'
+    return 'django.core.cache.backends.memcached.PyMemcacheCache'
+
+
+PYMEMCACHE_DRIVER = choose_pymemcache_driver()

environ/environ.py

@@ -26,15 +26,21 @@
     urlunparse,
 )
 
-from .compat import DJANGO_POSTGRES, ImproperlyConfigured, json, REDIS_DRIVER
+from .compat import (
+    DJANGO_POSTGRES,
+    ImproperlyConfigured,
+    json,
+    PYMEMCACHE_DRIVER,
+    REDIS_DRIVER,
+)
 from .fileaware_mapping import FileAwareMapping
 
 try:
     from os import PathLike
+except ImportError:  # Python 3.5 support
+    from pathlib import PurePath as PathLike
 
-    Openable = (str, PathLike)
-except ImportError:
-    Openable = (str,)
+Openable = (str, PathLike)
 
 logger = logging.getLogger(__name__)
 
@@ -116,7 +122,8 @@ class Env:
         'filecache': 'django.core.cache.backends.filebased.FileBasedCache',
         'locmemcache': 'django.core.cache.backends.locmem.LocMemCache',
         'memcache': 'django.core.cache.backends.memcached.MemcachedCache',
-        'pymemcache': 'django.core.cache.backends.memcached.PyLibMCCache',
+        'pymemcache': PYMEMCACHE_DRIVER,
+        'pylibmc': 'django.core.cache.backends.memcached.PyLibMCCache',
         'rediscache': REDIS_DRIVER,
         'redis': REDIS_DRIVER,
         'rediss': REDIS_DRIVER,
@@ -157,9 +164,11 @@ class Env:
         "xapian": "haystack.backends.xapian_backend.XapianEngine",
         "simple": "haystack.backends.simple_backend.SimpleEngine",
     }
+    CLOUDSQL = 'cloudsql'
 
     def __init__(self, **scheme):
         self.smart_cast = True
+        self.escape_proxy = False
         self.scheme = scheme
 
     def __call__(self, var, cast=None, default=NOTSET, parse_default=False):
@@ -181,7 +190,7 @@ def str(self, var, default=NOTSET, multiline=False):
         """
         value = self.get_value(var, cast=str, default=default)
         if multiline:
-            return value.replace('\\n', '\n')
+            return re.sub(r'(\\r)?\\n', r'\n', value)
         return value
 
     def unicode(self, var, default=NOTSET):
@@ -365,16 +374,22 @@ def get_value(self, var, cast=None, default=NOTSET, parse_default=False):
 
         # Resolve any proxied values
         prefix = b'$' if isinstance(value, bytes) else '$'
+        escape = rb'\$' if isinstance(value, bytes) else r'\$'
         if hasattr(value, 'startswith') and value.startswith(prefix):
             value = value.lstrip(prefix)
             value = self.get_value(value, cast=cast, default=default)
 
+        if self.escape_proxy and hasattr(value, 'replace'):
+            value = value.replace(escape, prefix)
+
         # Smart casting
         if self.smart_cast:
             if cast is None and default is not None and \
                     not isinstance(default, NoValue):
                 cast = type(default)
 
+        value = None if default is None and value == '' else value
+
         if value != default or (parse_default and value):
             value = self.parse_value(value, cast)
 
@@ -495,7 +510,10 @@ def db_url_config(cls, url, engine=None):
             'PORT': _cast_int(url.port) or '',
         })
 
-        if url.scheme in cls.POSTGRES_FAMILY and path.startswith('/'):
+        if (
+                url.scheme in cls.POSTGRES_FAMILY and path.startswith('/')
+                or cls.CLOUDSQL in path and path.startswith('/')
+        ):
             config['HOST'], config['NAME'] = path.rsplit('/', 1)
 
         if url.scheme == 'oracle' and path == '':
@@ -732,16 +750,29 @@ def search_url_config(cls, url, engine=None):
         return config
 
     @classmethod
-    def read_env(cls, env_file=None, **overrides):
+    def read_env(cls, env_file=None, overwrite=False, **overrides):
         """Read a .env file into os.environ.
 
         If not given a path to a dotenv path, does filthy magic stack
         backtracking to find the dotenv in the same directory as the file that
         called read_env.
 
+        Existing environment variables take precedent and are NOT overwritten
+        by the file content. ``overwrite=True`` will force an overwrite of
+        existing environment variables.
+
         Refs:
         - https://wellfire.co/learn/easier-12-factor-django
         - https://gist.github.com/bennylope/2999704
+
+        :param env_file: The path to the `.env` file your application should
+            use. If a path is not provided, `read_env` will attempt to import
+            the Django settings module from the Django project root.
+        :param overwrite: ``overwrite=True`` will force an overwrite of
+            existing environment variables.
+        :param **overrides: Any additional keyword arguments provided directly
+            to read_env will be added to the environment. If the key matches an
+            existing environment variable, the value will be overridden.
         """
         if env_file is None:
             frame = sys._getframe()
@@ -757,7 +788,8 @@ def read_env(cls, env_file=None, **overrides):
 
         try:
             if isinstance(env_file, Openable):
-                with open(env_file) as f:
+                # Python 3.5 support (wrap path with str).
+                with open(str(env_file)) as f:
                     content = f.read()
             else:
                 with env_file as f:
@@ -770,6 +802,13 @@ def read_env(cls, env_file=None, **overrides):
 
         logger.debug('Read environment variables from: {}'.format(env_file))
 
+        def _keep_escaped_format_characters(match):
+            """Keep escaped newline/tabs in quoted strings"""
+            escaped_char = match.group(1)
+            if escaped_char in 'rnt':
+                return '\\' + escaped_char
+            return escaped_char
+
         for line in content.splitlines():
             m1 = re.match(r'\A(?:export )?([A-Za-z_0-9]+)=(.*)\Z', line)
             if m1:
@@ -779,12 +818,25 @@ def read_env(cls, env_file=None, **overrides):
                     val = m2.group(1)
                 m3 = re.match(r'\A"(.*)"\Z', val)
                 if m3:
-                    val = re.sub(r'\\(.)', r'\1', m3.group(1))
-                cls.ENVIRON.setdefault(key, str(val))
+                    val = re.sub(r'\\(.)', _keep_escaped_format_characters,
+                                 m3.group(1))
+                overrides[key] = str(val)
+            else:
+                logger.warning('Invalid line: %s', line)
+
+        def set_environ(envval):
+            """Return lambda to set environ.
+
+             Use setdefault unless overwrite is specified.
+             """
+            if overwrite:
+                return lambda k, v: envval.update({k: str(v)})
+            return lambda k, v: envval.setdefault(k, str(v))
+
+        setenv = set_environ(cls.ENVIRON)
 
-        # set defaults
         for key, value in overrides.items():
-            cls.ENVIRON.setdefault(key, value)
+            setenv(key, value)
 
 
 class FileAwareEnv(Env):