Squashed commits for fixing CI.

try adding circle config (#1)

* adding automated tests to run on Circle with selenium, still need to add screenshots
Signed-off-by: Vanessa Sochat <vsochat@stanford.edu>

Try taking screenshot (#2)

Adding screenshots (working) back in to tests, ready to PR to upstream
Signed-off-by: Vanessa Sochat <vsochat@stanford.edu>

Don't include driver binaries.

Take screenshot of only viewer element.

Don't remove Travis config, at least for now.

Run Black on tests script.

Fix automated tests for Firefox and some formatting changes.

Update test documentation.

.circleci/config.yml

@@ -0,0 +1,27 @@
+# Python CircleCI 2.0 configuration file
+# Check https://circleci.com/docs/2.0/language-python/ for more details
+#
+version: 2
+jobs:
+  build:
+    docker:
+      - image: selenium/standalone-chrome:3.141
+
+    working_directory: ~/repo
+
+    steps:
+      - checkout
+
+      - run:
+          name: install dependencies
+          command: |
+            sudo apt-get update && \
+            sudo apt-get install -y python3 python3-dev default-jdk python3-pillow python3-numpy python3-pip && \
+            sudo apt-get install -y libjpeg-dev libfreetype6 libfreetype6-dev zlib1g-dev optipng
+            sudo pip3 install selenium Pillow
+
+      - run:
+          name: run tests
+          command: |
+            python3 utils/build/build.py
+            python3 tests/run_tests.py

readme.md

@@ -59,7 +59,29 @@ All user-facing strings can be changed using the `strings` configuration paramet
 The `utils` folder contains the required build tools, with the exception of Python 3.2+ and Java installations. To build a minified version of Pannellum, run either `build.sh` or `build.bat` depending on your platform.
 
 ## Tests
-A limited [Selenium](https://www.seleniumhq.org/)-based test suite is located in the `tests` directory. The tests can be executed by running `python3 run_tests.py`. Running the tests requires Python 3, the Selenium Python bindings, Firefox, [geckodriver](https://github.com/mozilla/geckodriver), [Pillow](https://pillow.readthedocs.io/), and [NumPy](https://www.numpy.org/).
+
+A minimal [Selenium](https://www.seleniumhq.org/)-based test suite is located in the `tests` directory. The tests can be executed by running:
+
+```bash
+python3 run_tests.py
+```
+
+A Selenium-driven web browser (with a Chrome driver, by default) is created, and screenshots are generated
+and compared against previously generated ones in [tests](tests). For example, to regenerate the screenshots
+one can run:
+
+```bash
+$ python3 tests/run_tests.py --create-ref
+```
+
+And to simply run the tests to compare to, eliminate that argument. By default, a random
+port is selected, along with other arguments. One can see usage via:
+
+```bash
+$ python tests/run_tests.py --help
+```
+
+Continuous integration tests are run via CircleCI. Running the tests locally requires Python 3, the Selenium Python bindings, [Pillow](https://pillow.readthedocs.io/), [NumPy](https://www.numpy.org/), and either Firefox & [geckodriver](https://github.com/mozilla/geckodriver) or Chrome & [ChromeDriver](https://chromedriver.chromium.org/).
 
 ## Seeking support
 If you wish to ask a question or report a bug, please open an issue at [github.com/mpetroff/pannellum](https://github.com/mpetroff/pannellum). See the _Contributing_ section below for more details.

tests/run_tests.py

@@ -1,141 +1,290 @@
+#!/usr/bin/env python3
+
 """
 Selenium-based test suite for Pannellum
 
 Dependencies:
-Python 3, Selenium Python bindings, Firefox, geckodriver, Pillow, NumPy
+Python 3, Selenium Python bindings, Pillow, NumPy
+Either: Firefox & geckodriver or Chrome & chromedriver
+
+Run tests for Pannellum, set up with Continuous Integration.
+Contributed by Vanessa Sochat, JOSS Review 2019.
+See the project repository for licensing information.
 """
 
-import http.server
-import time
-import threading
+from random import choice
+from threading import Thread
+from http.server import SimpleHTTPRequestHandler
+from socketserver import TCPServer
+import argparse
 import io
-import subprocess
 import os
+import re
+import subprocess
+import sys
+import time
 import numpy as np
 from PIL import Image, ImageChops
+from selenium.common.exceptions import TimeoutException
 from selenium import webdriver
 
 
-# Set to true to create a new set of reference images
-CREATE_REF = False
-
-
-# Run web server
-print("Starting web server...")
-os.chdir(os.path.dirname(os.path.abspath(__file__)))  # cd to script dir
-os.chdir("..")
-httpd = http.server.HTTPServer(
-    ("localhost", 8000), http.server.SimpleHTTPRequestHandler
-)
-thread = threading.Thread(None, httpd.serve_forever)
-thread.start()
-
-
-# Create a new instance of the Firefox driver
-print("Starting web driver...")
-if os.environ.get("TRAVIS_JOB_NUMBER"):
-    # Configuration for Travis CI / Sauce Labs testing
-    driver = webdriver.Remote(
-        command_executor="https://ondemand.saucelabs.com:443/wd/hub",
-        desired_capabilities={
-            "username": os.environ["SAUCE_USERNAME"],
-            "accessKey": os.environ["SAUCE_ACCESS_KEY"],
-            "tunnel-identifier": os.environ["TRAVIS_JOB_NUMBER"],
-            "build": os.environ["TRAVIS_JOB_NUMBER"],
-            "browserName": "firefox",
-            "seleniumVersion": "3.141.0",
-        },
+class PannellumServer(SimpleHTTPRequestHandler):
+    """Here we subclass SimpleHTTPServer to capture error messages.
+    """
+
+    def log_message(self, format, *args):
+        """
+        Log to standard error with a date time string,
+        and then call any subclass specific logging functions.
+        """
+        sys.stderr.write(
+            "%s - - [%s] %s\n"
+            % (self.address_string(), self.log_date_time_string(), format % args)
+        )
+
+        # Workaround for error trying to GET html
+        if not re.search("div", format % args) and not re.search(
+            "function", format % args
+        ):
+            if re.search("404", format % args):
+                raise IOError(format % args)
+
+    def log_error(self, format, *args):
+        """Catch errors in the log_messages instead.
+        """
+        pass
+
+
+class PannellumTester(object):
+    """Bring up a server with a testing robot.
+    """
+
+    def __init__(self, port=None, browser="Chrome", headless=False):
+        self.handler = PannellumServer
+        if port:
+            self.port = port
+        else:
+            self.port = choice(range(8000, 9999))
+        print("Selected port is %s" % self.port)
+        self.httpd = TCPServer(("", self.port), self.handler)
+        self.server = Thread(target=self.httpd.serve_forever)
+        self.server.setDaemon(True)
+        self.server.start()
+        self.started = True
+        self.pause_time = 100
+        self.browser = None
+        self.headless = headless
+        self.display = None
+        self.driver = browser
+
+    def take_screenshot(self, element_id, filename=None):
+        """Take a screenshot of an element with a given ID.
+        """
+        element = self.browser.find_element_by_id(element_id)
+        img = Image.open(io.BytesIO(element.screenshot_as_png)).convert("RGB")
+        if filename is not None:
+            img.save(filename)
+        return img
+
+    def equal_images(self, reference, comparator, name, threshold=5):
+        """Compare two images, both loaded with PIL, based on pixel differences."""
+        diff = np.mean(np.array(ImageChops.difference(reference, comparator)))
+        print("%s difference: %s" % (name, diff))
+        if diff >= threshold:
+            comparator.save("tests/" + name + "-comparison.png")
+            raise ValueError("Screenshot difference is above threshold!")
+
+    def run_tests(self, create_ref=False):
+        """Run tests for Pannellum."""
+
+        print("Loading page...")
+        self.get_page("http://localhost:%s/tests/tests.html" % self.port)
+
+        print("Running tests...")
+        time.sleep(5)
+
+        assert self.browser.execute_script("return viewer.isLoaded()") is True
+
+        # Check equirectangular
+        assert self.browser.execute_script(
+            "return viewer.getScene() == 'equirectangular'"
+        )
+        if create_ref:
+            self.take_screenshot("panorama", "tests/equirectangular.png")
+            subprocess.call(
+                ["optipng", "-o7", "-strip", "all", "tests/equirectangular.png"]
+            )
+        else:
+            reference = Image.open("tests/equirectangular.png")
+            comparator = self.take_screenshot("panorama")
+            self.equal_images(reference, comparator, "equirectangular")
+        print("PASS: equirectangular")
+
+        # Check movement
+        self.browser.execute_script("viewer.setPitch(30).setYaw(-20).setHfov(90)")
+        time.sleep(2)
+        assert self.browser.execute_script(
+            "return viewer.getPitch() == 30 && viewer.getYaw() == -20 && viewer.getHfov() == 90"
+        )
+        self.browser.find_element_by_class_name("pnlm-zoom-in").click()
+        time.sleep(1)
+        assert self.browser.execute_script("return viewer.getHfov() == 85")
+        self.browser.find_element_by_class_name("pnlm-zoom-out").click()
+        time.sleep(1)
+        assert self.browser.execute_script("return viewer.getHfov() == 90")
+        print("PASS: movement")
+
+        # Check look at
+        self.browser.execute_script("viewer.lookAt(-10, 90, 100)")
+        time.sleep(2)
+        assert self.browser.execute_script(
+            "return viewer.getPitch() == -10 && viewer.getYaw() == 90 && viewer.getHfov() == 100"
+        )
+        print("PASS: look at")
+
+        # Check cube
+        self.browser.execute_script("viewer.loadScene('cube')")
+        time.sleep(5)
+        assert self.browser.execute_script("return viewer.getScene() == 'cube'")
+        if create_ref:
+            self.take_screenshot("panorama", "tests/cube.png")
+            subprocess.call(["optipng", "-o7", "-strip", "all", "tests/cube.png"])
+        else:
+            reference = Image.open("tests/cube.png")
+            comparator = self.take_screenshot("panorama")
+            self.equal_images(reference, comparator, "cube")
+
+        # Check hot spot
+        self.browser.find_element_by_class_name("pnlm-scene").click()
+        time.sleep(5)
+        assert self.browser.execute_script("return viewer.getScene() == 'multires'")
+        print("PASS: hot spot")
+
+        # Check multires
+        if create_ref:
+            self.take_screenshot("panorama", "tests/multires.png")
+            subprocess.call(["optipng", "-o7", "-strip", "all", "tests/multires.png"])
+        else:
+            reference = Image.open("tests/multires.png")
+            comparator = self.take_screenshot("panorama")
+            self.equal_images(reference, comparator, "multires")
+
+        self.httpd.server_close()
+
+    def get_browser(self, name=None):
+        """Return a browser if it hasn't been initialized yet.
+        """
+        if name is None:
+            name = self.driver
+
+        log_path = "tests/%s-driver.log" % name.lower()
+
+        if self.browser is None:
+            if name.lower() == "firefox":
+                fp = webdriver.FirefoxProfile()
+                fp.set_preference("layout.css.devPixelsPerPx", "1.0")
+                self.browser = webdriver.Firefox(
+                    service_log_path=log_path, firefox_profile=fp
+                )
+                self.browser.set_window_size(800, 600)
+            else:
+                options = webdriver.ChromeOptions()
+                options.add_argument("headless")
+                options.add_argument("no-sandbox")
+                options.add_argument("window-size=800x600")
+                self.browser = webdriver.Chrome(
+                    service_log_path=log_path, options=options
+                )
+        return self.browser
+
+    def get_page(self, url):
+        """Open a particular URL, checking for timeout.
+        """
+        if self.browser is None:
+            self.browser = self.get_browser()
+
+        try:
+            return self.browser.get(url)
+        except TimeoutException:
+            print("Browser request timeout. Are you connected to the internet?")
+            self.browser.close()
+            sys.exit(1)
+
+    def stop(self):
+        """Close any running browser or server and shut down the robot.
+        """
+        if self.browser is not None:
+            self.browser.close()
+        self.httpd.server_close()
+
+        if self.display is not None:
+            self.display.close()
+
+
+def get_parser():
+    parser = argparse.ArgumentParser(description="Run tests for Pannellum")
+
+    parser.add_argument(
+        "--port",
+        "-p",
+        dest="port",
+        help="Port to run web server",
+        type=int,
+        default=None,
     )
-else:
-    fp = webdriver.FirefoxProfile()
-    fp.set_preference("layout.css.devPixelsPerPx", "1.0")
-    driver = webdriver.Firefox(firefox_profile=fp)
-    driver.set_window_size(800, 600)
-
-
-def run_tests():
-    # Load page
-    print("Loading page...")
-    driver.get("http://localhost:8000/tests/tests.html")
-
-    # Make sure viewer loaded
-    print("Running tests...")
-    time.sleep(5)
-    viewer = driver.find_element_by_id("panorama")
-    assert driver.execute_script("return viewer.isLoaded()") == True
-
-    # Check equirectangular
-    assert driver.execute_script("return viewer.getScene() == 'equirectangular'")
-    if CREATE_REF:
-        viewer.screenshot("tests/equirectangular.png")
-        subprocess.call(["optipng", "-o7", "-strip", "all", "equirectangular.png"])
-    else:
-        reference = Image.open("tests/equirectangular.png")
-        screenshot = Image.open(io.BytesIO(viewer.screenshot_as_png)).convert("RGB")
-        diff = np.mean(np.array(ImageChops.difference(screenshot, reference)))
-        print("equirectangular difference:", diff)
-        assert diff < 3
-    print("PASS: equirectangular")
-
-    # Check movement
-    driver.execute_script("viewer.setPitch(30).setYaw(-20).setHfov(90)")
-    time.sleep(2)
-    assert driver.execute_script(
-        "return viewer.getPitch() == 30 && viewer.getYaw() == -20 && viewer.getHfov() == 90"
+
+    parser.add_argument(
+        "--headless",
+        dest="headless",
+        help="Start a display before browser",
+        action="store_true",
+        default=False,
     )
-    driver.find_element_by_class_name("pnlm-zoom-in").click()
-    time.sleep(1)
-    assert driver.execute_script("return viewer.getHfov() == 85")
-    driver.find_element_by_class_name("pnlm-zoom-out").click()
-    time.sleep(1)
-    assert driver.execute_script("return viewer.getHfov() == 90")
-    print("PASS: movement")
-
-    # Check look at
-    driver.execute_script("viewer.lookAt(-10, 90, 100)")
-    time.sleep(2)
-    assert driver.execute_script(
-        "return viewer.getPitch() == -10 && viewer.getYaw() == 90 && viewer.getHfov() == 100"
+
+    parser.add_argument(
+        "--create-ref", dest="create_ref", action="store_true", default=False
     )
-    print("PASS: look at")
-
-    # Check cube
-    driver.execute_script("viewer.loadScene('cube')")
-    time.sleep(5)
-    assert driver.execute_script("return viewer.getScene() == 'cube'")
-    if CREATE_REF:
-        viewer.screenshot("tests/cube.png")
-        subprocess.call(["optipng", "-o7", "-strip", "all", "cube.png"])
-    else:
-        reference = Image.open("tests/cube.png")
-        screenshot = Image.open(io.BytesIO(viewer.screenshot_as_png)).convert("RGB")
-        diff = np.mean(np.array(ImageChops.difference(screenshot, reference)))
-        print("cube difference:", diff)
-        assert diff < 3
-    print("PASS: cube")
-
-    # Check hot spot
-    driver.find_element_by_class_name("pnlm-scene").click()
-    time.sleep(5)
-    assert driver.execute_script("return viewer.getScene() == 'multires'")
-    print("PASS: hot spot")
-
-    # Check multires
-    if CREATE_REF:
-        viewer.screenshot("tests/multires.png")
-        subprocess.call(["optipng", "-o7", "-strip", "all", "multires.png"])
-    else:
-        reference = Image.open("tests/multires.png")
-        screenshot = Image.open(io.BytesIO(viewer.screenshot_as_png)).convert("RGB")
-        diff = np.mean(np.array(ImageChops.difference(screenshot, reference)))
-        print("multires difference:", diff)
-        assert diff < 3
-    print("PASS: multires")
-
-
-try:
-    run_tests()
-finally:
-    driver.quit()
-    httpd.shutdown()
-    thread.join()
+
+    parser.add_argument(
+        "--browser",
+        "-b",
+        dest="browser",
+        choices=["Firefox", "Chrome"],
+        help="Browser driver to use for the robot",
+        type=str,
+        default="Chrome",
+    )
+    return parser
+
+
+def main():
+    parser = get_parser()
+
+    try:
+        args = parser.parse_args()
+    except:
+        sys.exit(0)
+
+    # Add this script's directory, in case it contains driver binaries
+    here = os.path.abspath(os.path.dirname(__file__))
+    os.environ["PATH"] = here + ":" + os.environ["PATH"]
+    os.chdir(here)
+
+    # We must be in root directory
+    os.chdir("..")
+
+    # Initialize the tester
+    tester = PannellumTester(
+        browser=args.browser, port=args.port, headless=args.headless
+    )
+
+    # Run tests
+    tester.run_tests(create_ref=args.create_ref)
+
+    # Clean up shop!
+    tester.stop()
+
+
+if __name__ == "__main__":
+    main()

tests/tests.html

@@ -69,6 +69,7 @@ viewer = pannellum.viewer('panorama', {
         "multires": {
             "title": "multires title",
             "type": "multires",
+            "hfov": 85,
             "multiRes": {
                 "basePath": "../examples/multires",
                 "path": "/%l/%s%x%y",

utils/build/build.py

@@ -99,7 +99,11 @@ def build(files, css, html, filename, release=False):
     if release:
         version = read('../VERSION').strip()
     else:
-        version = subprocess.check_output(['git', 'rev-parse', '--short', 'HEAD']).decode('utf-8').strip()
+        if os.path.exists('.git'):
+            version = subprocess.check_output(['git', 'rev-parse', '--short', 'HEAD']).decode('utf-8').strip()
+        else:
+            print('No .git folder detected, setting version to testing')
+            version = "testing"
     js = js.replace('"_blank">Pannellum</a>','"_blank">Pannellum</a> ' + version)
     with open('../../src/standalone/standalone.js', 'r') as f:
         standalone_js = f.read()