Python 3.6 is not available on AWS CodeBuild, Python 3.5 is not available on AWS Lambda

I have a Python 3 project that I am trying to deploy to AWS Lambda via AWS Codestar → Codepipeline → Codebuild → Cloudformation.

My project (which actually just consists of a simple API handler method) is importing a Python 3 project (3 required) (newspaper). I am using Virtualenv 15.1.0 on my home machine and if I install newspaper with Python 3.5 and then boot into Lambda (Python 3.6 runtime) it throws PIL / Pillow related errors.

At first he says he cannot find _image, which appears to be resolved by deleting the PIL directory in the site packages, however that just causes him to throw an error that he cannot find PIL.

If, however, I build with Python 3.6 and then boot into Lambda, it works fine (whether it removes the PIL or not).

So, it seems to me that I cannot install newspaper since 3.5 and try to execute it at runtime 3.6.

So now I'm trying to deploy via Codestar, however Codestar defaults to aws / codebuild / eb-nodejs-4.4.6-amazonlinux-64: 2.1.3, even for Python projects and it all seems to be available in the Yum repository, this is Python 3.5, and of course Lambda only has 3.6 runtimes.

Even if I switch the image inside the Codebuild itself, there seem to be no images generated with the Python3.6 runtime (as per the documentation). Even Docker images don't seem to have Python 3.6.

So, I am trying to install Python 3.6 in Codebuild in the INSTALL step in my buildspec.yml file, however after installation I cannot find the python3 * executable.

The only thing I can think of is to create a Codestar project, edit the code to use Ubuntu and then install everything (just like on a local machine), but there is no way to do this from Codestar and I feel like that might lead me to rabbit hole, and it's hardly automated. Is there a way to make this config as code from my project?

EDIT Trying to build and install Python 3.6 from original works, but then when trying to install Pip, I get errors saying SSL was not installed. And looking back at the build logs, it seems that the other "bits" weren't set as well.

So my questions are here:

• How do I get Python 3.6 into the Codebuild framework provided from the Codestar project?
• Should I keep trying to build it from source or switch to Ubuntu environment?
• How can I automatically configure the image / environment in my code / project?

EDIT 1 For everyone else, my complete buildspec.yml for installing and using Python3.6 is below. Note that it allows you to do everything as quietly as possible to reduce log messages, lower Cloudwatch costs, and speed up the process. I ended up shaving for about 90 seconds from the whole process by doing this (installing Python and building my app). Since CodeBuild's board is based on elapsed time, this is critical.

version: 0.2

phases:
  install:
    commands:
      - yum -qye 0 update
      - yum -qye 0 groupinstall development
      - yum -y install python-devel
      - yum -qye 0 install libxml2-devel libxslt-devel libjpeg-devel zlib-devel libpng-devel openssl-devel sqlite-devel
      - export HOME_DIR=`pwd`
      # I would recommend hosting the tarball in an uncompressed format on S3 in order to speed up the download and decompression
      - wget --no-verbose https://www.python.org/ftp/python/3.6.1/Python-3.6.1.tgz
      - tar -xzf Python-3.6.1.tgz
      - cd Python-3.6.1
      - ./configure -q --enable-loadable-sqlite-extensions
      - make --silent -j2
      - make altinstall --silent
      - cd $HOME_DIR
      - rm Python-3.6.1.tgz
      - rm -rf Python-3.6.1/
      - ln -s /usr/local/bin/python3.6 /usr/bin/python3
      - python3 -m pip install virtualenv
      - pip3 install -U nltk
  pre_build:
    commands:
      - cd $HOME_DIR
      # Start a virtualenv and activate
      - virtualenv -p /usr/bin/python3 $VIRTUAL_ENV_DIR_NAME
      - source $VIRTUAL_ENV_DIR_NAME/bin/activate
      - $VIRTUAL_ENV_DIR_NAME/bin/pip3.6 install nltk
      # If you plan to use any separate resources on Codecommit, you need to configure git
      - git config --global credential.helper '!aws codecommit credential-helper $@'
      - git config --global credential.UseHttpPath true
      # git clone whatever you need
  build:
    commands:
      - cd $HOME_DIR
      - mv $VIRTUAL_ENV/lib/python3.6/site-packages/* .
      - aws cloudformation package --template template.yml --s3-bucket $S3_BUCKET --output-template template-export.json
artifacts:
  type: zip
  files:
    - template-export.json