Merge branch '9.2RC'

Add module test helper expectRedirectTo + api docs

See merge request hubzilla/core!2138

.debianinstall/README.md

@@ -0,0 +1,158 @@
+
+# How to use
+
+## Disclaimers
+
+- **This script does work with a fresh install of Debian 12 only**.
+- Do not use if you have already installed and configured a webserver or sql server that was not installed by this script.
+
+### Keep it Simple and Stupid
+
+The script keeps everything as simple as possible (KISS):
+
+- Apache as webserver (there is no choice to use another webserver like nginx)
+- default PHP version of Debian
+- one single Hubzilla intance only
+- re-running the script does no harm
+
+### When to use other Scripts
+
+Use the scripts under [homeinstall](https://framagit.org/hubzilla/core/-/tree/master/.homeinstall)
+if you look for more choices. The main differences are:
+
+- Apache or nginx as webserver
+- install multiple instances (domains) that run side by side on the server
+- adds apache vhosts (instead of using the standard doc root /var/www/html)
+- install PHP from https://packages.sury.org/php/ (instead of using the Debian repository)
+- graphical installer whiptail
+- The script stops (fails) if it finds results of a previous installation. (The [debian-setup.sh](https://framagit.org/ojrandom/core/-/tree/dev/.debianinstall) will just jump over it.)
+- If something fails the script tries to clean up everything that was installed up to the point of failure. (That might cause trouble if certbot registered a certificate already.)
+- The script under [homeinstall](https://framagit.org/hubzilla/core/-/tree/master/.homeinstall) seems to be an older version of the scripts used for Streams
+  + [autoinstall](https://codeberg.org/streams/streams/src/branch/dev/contrib/autoinstall)
+  + [easyinstall](https://codeberg.org/streams/streams/src/branch/dev/contrib/easyinstall)
+
+## Preconditions
+
+Hardware
+
++ internet connection and router at home
++ computer connected to your router (a Raspberry 3 will do for very small Hubs)
+
+Software
+
++ fresh installation of Debian 12 (bookworm)
++ router with open ports 80 and 443 for your web server
+
+You can of course run the script on a VPS or any distant server as long as the above sotfware requirements are satisfied.
+
+## How to run the script
+
++ Register your own domain (for example at selfHOST) or a free subdomain (for example at freeDNS)
++ Log on to your fresh Debian
+  - apt-get install git
+  - mkdir -p /var/www
+  - cd /var/www
+  - git clone https://framagit.org/hubzilla/core.git html
+  - cd html/.debianinstall
+  - cp config.txt.template config.txt
+  - nano config.txt
+    - read the comments carefully
+    - enter your values: db pass, domain
+    - (optionally) Enter your values for dyn DNS
+  - ./debian-setup.sh as root
+    - ... wait, wait, wait until the script is finished
++ Open your domain with a browser and step throught the initial configuration of your hubzilla instance.
+  - default database name = hubzilla
+  - default dababase user = hubzilla
+
+## Optional - Switch verification of email on/off
+
+Do this just before you register the first user without email verification.
+
+In a terminal
+
+    su -
+    cd /var/www/html
+
+Check the current setting 
+
+    util/config system verify_email
+
+Switch the verification off
+
+    util/config system verify_email 0
+
+## What the script will do for you...
+
++ install everything required by your hubzilla instance, basically a web server (Apache), PHP, a database (MySQL), certbot,...
++ create a database
++ run certbot to have everything for a secure connection (httpS)
++ create a script for daily maintenance
+  - renew certfificate (letsencrypt)
+  - update of your hubzilla instance for core and addons (git)
+  - update of Debian
+  - restart
++ create cron jobs for
+  - DynDNS (selfHOST.de or freedns.afraid.org) every 5 minutes
+  - Master.php for your hubzilla instance every 10 minutes
+  - daily maintenance script every day at 05:30
+
+The script is known to work without adjustments with
+
++ Hardware
+  - standard PC with Debian 12 (bookworm)
+  - Raspberry 4 with Raspbian, Debian 12 (TODO: needs confirmation after swich to Debian12)
+  - for tesing purposes: under localhost inside a virtual machine, [KVM](https://wiki.debian.org/KVM)
++ DynDNS
+  - selfHOST.de
+  - freedns.afraid.org
+
+# Step-by-Step - some Details
+
+## Preparations
+
+## Configure your Router
+
+Your webserver has to be visible in the internet.  
+
+Open the ports 80 and 443 on your router for your Debian. Make sure your web server is marked as "exposed host".
+
+## Preparations Dynamic IP Address
+
+Follow the instructions in .debianinstall/config.txt.  
+
+In short...  
+
+Your Hubzilla server must be reachable by a domain that you can type in your browser
+
+    cooldomain.org
+
+You can use subdomains as well
+
+    my.cooldomain.org
+
+There are two ways to get a domain...
+
+### Method 1: Buy a Domain 
+
+...for example buy at selfHOST.de  
+
+The cost is 1,50 € per month (2019).
+
+### Method 2: Register a free subdomain
+
+...for example register at freedns.afraid.org
+
+## Note on Rasperry 
+
+It is recommended to run the Raspi without graphical frontend (X-Server). Use...
+
+    sudo raspi-config
+
+to boot the Rapsi to the client console.
+
+DO NOT FORGET TO CHANGE THE DEFAULT PASSWORD FOR USER PI!
+
+## Reminder for Different Web Wervers
+
+For those of you who feel adventurous enough to use a different web server (i.e. Lighttpd...), don't forget that this script will install Apache or Nginx and that you can only have one web server listening to ports 80 & 443. Also, don't forget to tweak your daily shell script in /var/www/ accordingly.

.debianinstall/config.txt.template

@@ -0,0 +1,111 @@
+
+###############################################
+### MANDATORY - database password #############
+#
+# Please give your database password
+# It is better to not use blanks inside the password.
+#   Example: db_pass=pass_word_with_no_blanks_in_it
+db_pass=
+
+###############################################
+### MANDATORY - let's encrypt #################
+#
+# Zot requires encrypted communication via secure HTTP (HTTPS).
+# This script automates installation of an SSL certificate from
+# Let's Encrypt (https://letsencrypt.org)
+#
+# Please give the domain name of your hub/instance
+#
+#     Example: my.cooldomain.org
+#     Example: cooldomain.org
+#
+# You might use "localhost" for a LOCAL TEST installation.
+# This is usefull if you want to debug the server inside a VM.
+#
+#     Example: localhost
+#
+# Email is optional if you use "localhost".
+#
+#
+le_domain=
+le_email=
+
+
+###############################################
+### OPTIONAL - selfHOST - dynamic IP address ##
+#
+# 1. Register a domain at selfhost.de
+#    - choose offer "DOMAIN dynamisch" 1,50€/mon at 04/2019
+# 2. Get your configuration for dynamic IP update
+#    - Log in at selfhost.de
+#    - go to "DynDNS Accounte"
+#    - klick "Details" of your (freshly) registered domain
+#    - You will find the configuration there
+#      - Benutzername (user name) > use this for "selfhost_user="
+#      - Passwort (pass word) > use this for "selfhost_pass="
+#
+#
+selfhost_user=
+selfhost_pass=
+
+###############################################
+### OPTIONAL - FreeDNS - dynamic IP address ###
+#
+# Please give the alpha-numeric-key of freedns
+#
+# Get a free subdomain from freedns and use it for your dynamic ip address
+# Documentation under http://www.techjawab.com/2013/06/setup-dynamic-dns-dyndns-for-free-on.html
+#
+# - Register for a Free domain at http://freedns.afraid.org/signup/
+# - WATCH THIS: Make sure you choose a domain with as less subdomains as
+#   possible. Why? Let's encrpyt issues a limited count of certificates each
+#   day. Possible other users of this domain will try to issue a certificate
+#   at the same day.
+# - Logon to FreeDNS (where you just registered)
+# - Goto http://freedns.afraid.org/dynamic/
+# - Right click on "Direct Link" and copy the URL and paste it somewhere.
+# - You should notice a large and unique alpha-numeric key in the URL
+#
+#       http://freedns.afraid.org/dynamic/update.php?alpha-numeric-key
+#
+#   Provided your url from freedns is
+#
+#	http://freedns.afraid.org/dynamic/update.php?U1Z6aGt2R0NzMFNPNWRjbWxxZGpsd093OjE1Mzg5NDE5
+#
+#   Then you have to provide
+#
+#       freedns_key=U1Z6aGt2R0NzMFNPNWRjbWxxZGpsd093OjE1Mzg5NDE5
+#
+#
+freedns_key=
+
+
+###############################################
+### OPTIONAL - do not mess with things below ##
+#              (...if you are not certain)
+#
+# Usually you are done here
+# Everything below is OPTIONAL
+#
+###############################################
+#
+# Database for your hub/instance
+# If left empty, both your database and user will be named after your zot instance (hubzilla, zap or misty)
+# Use custom name, at least fo the database, if you plan to run more than one hub/instance on the same server
+#
+zotserver_db_name=
+zotserver_db_user=
+zotserver_db_pass=$db_pass
+# 
+#
+# Password for package mysql-server
+#   Example: mysqlpass=aberhallo
+#   Example: mysqlpass="aber hallo has blanks in it"
+#
+mysqlpass=$db_pass
+
+# Password for package phpmyadmin
+#   Example: phpmyadminpass=aberhallo
+#   Example: phpmyadminpass="aber hallo has blanks in it"
+phpmyadminpass=$db_pass
+

.debianinstall/debian-setup.sh

@@ -0,0 +1,527 @@
+#!/bin/bash
+#
+# How to use
+# ----------
+#
+# This file automates the installation of hubzilla: https://framagit.org/hubzilla/core
+# under Debian Linux "bookworm"
+#
+# 1) Copy the file "config.txt.template" to "config.txt"
+#       Follow the instuctions there
+#
+# 2) Switch to user "root" by typing "su -"
+#
+# 3) Run with "./debian-setup.sh"
+#       If this fails check if you can execute the script.
+#       - To make it executable type "chmod +x debian-setup.sh"
+#       - or run "bash debian-setup.sh"
+#
+#
+# What does this script do basically?
+# -----------------------------------
+#
+# This file automates the installation of a Hubzilla instance under Debian Linux
+# - install
+#        * apache webserver,
+#        * php,
+#        * mariadb,
+#        * adminer,
+#        * addons
+# - configure cron
+#        * "Master.php" for regular background processes of your hubzilla instance
+#        * "apt-get update" and "apt-get dist-upgrade" and "apt-get autoremove" to keep linux up-to-date
+#        * run command to keep the IP up-to-date > DynDNS provided by selfHOST.de or freedns.afraid.org
+# - run letsencrypt to create, register and use a certifacte for https
+#
+#
+# Credits
+# -------
+#
+# The script is derived from the easyinstall script of the Streams repository, which is based on
+# - Tom Wiedenhöfts (OJ Random) script homeinstall (for Hubzilla, ZAP,...) that was based on
+# - Thomas Willinghams script "debian-setup.sh" which he used to install the red#matrix.
+
+function check_sanity {
+    # Do some sanity checking.
+    print_info "Sanity check..."
+    if [ $(/usr/bin/id -u) != "0" ]
+    then
+        die 'Must be run by root user'
+    fi
+
+    if [ -f /etc/lsb-release ]
+    then
+        die "Distribution is not supported"
+    fi
+    if [ ! -f /etc/debian_version ]
+    then
+        die "Debian is supported only"
+    fi
+    if ! grep -q 'Linux 12' /etc/issue
+    then
+        die "Linux 12 (bookworm) is supported only"x
+    fi
+}
+
+function check_config {
+    print_info "config check..."
+    # Check for required parameters
+    if [ -z "$db_pass" ]
+    then
+        die "db_pass not set in $configfile"
+    fi
+    if [ -z "$le_domain" ]
+    then
+        die "le_domain not set in $configfile"
+    fi
+}
+
+function die {
+    echo "ERROR: $1" > /dev/null 1>&2
+    exit 1
+}
+
+
+function update_upgrade {
+    print_info "updated and upgrade..."
+    # Run through the apt-get update/upgrade first. This should be done before
+    # we try to install any package
+    apt-get -q -y update && apt-get -q -y dist-upgrade && apt-get -q -y autoremove
+    print_info "updated and upgraded linux"
+}
+
+function nocheck_install {
+    # export DEBIAN_FRONTEND=noninteractive ... answers from the package configuration database
+    # - q ... without progress information
+    # - y ... answer interactive questions with "yes"
+    # DEBIAN_FRONTEND=noninteractive apt-get --no-install-recommends -q -y install $2
+    # DEBIAN_FRONTEND=noninteractive apt-get --install-suggests -q -y install $1
+    DEBIAN_FRONTEND=noninteractive apt-get -q -y install $1
+    print_info "installed $1"
+}
+
+
+function print_info {
+    echo -n -e '\e[1;34m'
+    echo -n $1
+    echo -e '\e[0m'
+}
+
+function print_warn {
+    echo -n -e '\e[1;31m'
+    echo -n $1
+    echo -e '\e[0m'
+}
+
+function stop_zotserver {
+    print_info "stopping apache..."
+    systemctl stop apache2
+    print_info "stopping mysql db..."
+    systemctl stop mariadb
+}
+
+function install_apache {
+    print_info "installing apache..."
+    nocheck_install "apache2 apache2-utils"
+    a2enmod rewrite
+    systemctl restart apache2
+}
+
+function install_imagemagick {
+    print_info "installing imagemagick..."
+    nocheck_install "imagemagick"
+}
+
+function install_curl {
+    print_info "installing curl..."
+    nocheck_install "curl"
+}
+
+function install_wget {
+    print_info "installing wget..."
+    nocheck_install "wget"
+}
+
+function install_sendmail {
+    print_info "installing sendmail..."
+    nocheck_install "sendmail sendmail-bin"
+}
+
+function install_php {
+    # openssl and mbstring are included in libapache2-mod-php
+    print_info "installing php..."
+    nocheck_install "libapache2-mod-php php php-pear php-curl php-gd php-mbstring php-xml php-zip"
+    phpversion=$(php -v|grep --only-matching --perl-regexp "(PHP )\d+\.\\d+\.\\d+"|cut -c 5-7)
+    sed -i "s/^upload_max_filesize =.*/upload_max_filesize = 100M/g" /etc/php/$phpversion/apache2/php.ini
+    sed -i "s/^post_max_size =.*/post_max_size = 100M/g" /etc/php/$phpversion/apache2/php.ini
+}
+
+function install_composer {
+    print_info "We check if Composer is already downloaded"
+    if [ ! -f /usr/local/bin/composer ]
+    then
+        EXPECTED_CHECKSUM="$(php -r 'copy("https://composer.github.io/installer.sig", "php://stdout");')"
+        php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"
+        ACTUAL_CHECKSUM="$(php -r "echo hash_file('sha384', 'composer-setup.php');")"
+        if [ "$EXPECTED_CHECKSUM" != "$ACTUAL_CHECKSUM" ]
+        then
+            >&2 echo 'ERROR: Invalid installer checksum'
+            rm composer-setup.php
+            die 'ERROR: Invalid installer checksum'
+        fi
+        php composer-setup.php --quiet
+        RESULT=$?
+        rm composer-setup.php
+        # exit $RESULT
+        # We install Composer globally
+        mv composer.phar /usr/local/bin/composer
+        print_info "Composer was successfully downloaded."
+    else
+        print_info "Composer is already downloaded on this system."
+    fi
+    cd $install_path
+    export COMPOSER_ALLOW_SUPERUSER=1;
+    /usr/local/bin/composer install --no-dev
+    /usr/local/bin/composer show
+    export COMPOSER_ALLOW_SUPERUSER=0;
+}
+
+
+function install_mysql {
+    print_info "installing mysql..."
+    if [ -z "$mysqlpass" ]
+    then
+        die "mysqlpass not set in $configfile"
+    fi
+	if type mysql ; then
+        echo "Yes, mysql is installed"
+	else
+        echo "mariadb-server"
+        nocheck_install "mariadb-server"
+        systemctl status mariadb
+        systemctl start mariadb
+        mysql --user=root <<_EOF_
+UPDATE mysql.user SET Password=PASSWORD('${mysqlpass}') WHERE User='root';
+DELETE FROM mysql.user WHERE User='';
+DROP DATABASE IF EXISTS test;
+DELETE FROM mysql.db WHERE Db='test' OR Db='test\\_%';
+FLUSH PRIVILEGES;
+_EOF_
+    fi
+}
+
+function install_adminer {
+    print_info "installing adminer..."
+    nocheck_install "adminer"
+    if [ ! -f /etc/adminer/adminer.conf ]
+    then
+        echo "Alias /adminer /usr/share/adminer/adminer" > /etc/adminer/adminer.conf
+        ln -s /etc/adminer/adminer.conf /etc/apache2/conf-available/adminer.conf
+    else
+        print_info "file /etc/adminer/adminer.conf exists already"
+    fi
+
+    a2enmod rewrite
+
+    if [ ! -f /etc/apache2/apache2.conf ]
+    then
+        die "could not find file /etc/apache2/apache2.conf"
+    fi
+    sed -i \
+        "s/AllowOverride None/AllowOverride all/" \
+        /etc/apache2/apache2.conf
+
+    a2enconf adminer
+    systemctl restart mariadb
+    systemctl reload apache2
+}
+
+function create_zotserver_db {
+    print_info "creating zotserver database..."
+    if [ -z "$db_name" ]
+    then
+        die "db_name not set in $configfile"
+    fi
+    if [ -z "$db_user" ]
+    then
+        die "db_user not set in $configfile"
+    fi
+    if [ -z "$db_pass" ]
+    then
+        die "db_pass not set in $configfile"
+    fi
+    systemctl restart mariadb
+    # Make sure we don't write over an already existing database if we install more than one Zot hub/instance
+    if [ -z $(mysql -h localhost -u root -p$mysqlpass -e "SHOW DATABASES;" | grep $db_name) ]
+    then
+        Q1="CREATE DATABASE IF NOT EXISTS $db_name;"
+        Q2="GRANT USAGE ON *.* TO $db_user@localhost IDENTIFIED BY '$db_pass';"
+        Q3="GRANT ALL PRIVILEGES ON $name.* to $db_user@localhost identified by '$db_pass';"
+        Q4="FLUSH PRIVILEGES;"
+        SQL="${Q1}${Q2}${Q3}${Q4}"
+        mysql -uroot -p$mysqlpass -e "$SQL"
+    else
+        echo "database $db_name does exist already"
+    fi
+}
+
+function run_freedns {
+    print_info "run freedns (dynamic IP)..."
+    if [ -z "$freedns_key" ]
+    then
+        print_info "freedns was not started because 'freedns_key' is empty in $configfile"
+    else
+        if [ -n "$selfhost_user" ]
+        then
+            die "You can not use freeDNS AND selfHOST for dynamic IP updates ('freedns_key' AND 'selfhost_user' set in $configfile)"
+        fi
+        wget --no-check-certificate -O - http://freedns.afraid.org/dynamic/update.php?$freedns_key
+    fi
+}
+
+function install_run_selfhost {
+    print_info "install and start selfhost (dynamic IP)..."
+    if [ -z "$selfhost_user" ]
+    then
+        print_info "selfHOST was not started because 'selfhost_user' is empty in $configfile"
+    else
+        if [ -n "$freedns_key" ]
+        then
+            die "You can not use freeDNS AND selfHOST for dynamic IP updates ('freedns_key' AND 'selfhost_user' set in $configfile)"
+        fi
+        if [ -z "$selfhost_pass" ]
+        then
+            die "selfHOST was not started because 'selfhost_pass' is empty in $configfile"
+        fi
+        if [ ! -d $selfhostdir ]
+        then
+            mkdir $selfhostdir
+        fi
+        # the old way
+        # https://carol.selfhost.de/update?username=123456&password=supersafe
+        #
+        # the prefered way
+        wget --output-document=$selfhostdir/$selfhostscript http://jonaspasche.de/selfhost-updater
+        echo "router" > $selfhostdir/device
+        echo "$selfhost_user" > $selfhostdir/user
+        echo "$selfhost_pass" > $selfhostdir/pass
+        bash $selfhostdir/$selfhostscript update
+    fi
+}
+
+function ping_domain {
+    print_info "ping domain $domain..."
+    # Is the domain resolved? Try to ping 6 times à 10 seconds
+    COUNTER=0
+    for i in {1..6}
+    do
+        print_info "loop $i for ping -c 1 $domain ..."
+        if ping -c 4 -W 1 $le_domain
+        then
+            print_info "$le_domain resolved"
+            break
+        else
+            if [ $i -gt 5 ]
+            then
+                die "Failed to: ping -c 1 $domain not resolved"
+            fi
+        fi
+        sleep 10
+    done
+    sleep 5
+}
+
+function configure_cron_freedns {
+    print_info "configure cron for freedns..."
+    if [ -z "$freedns_key" ]
+    then
+        print_info "freedns is not configured because freedns_key is empty in $configfile"
+    else
+        # Use cron for dynamich ip update
+        #   - at reboot
+        #   - every 30 minutes
+        if [ -z "`grep 'freedns.afraid.org' /etc/crontab`" ]
+        then
+            echo "@reboot root http://freedns.afraid.org/dynamic/update.php?$freedns_key > /dev/null 2>&1" >> /etc/crontab
+            echo "*/30 * * * * root wget --no-check-certificate -O - http://freedns.afraid.org/dynamic/update.php?$freedns_key > /dev/null 2>&1" >> /etc/crontab
+        else
+            print_info "cron for freedns was configured already"
+        fi
+    fi
+}
+
+function configure_cron_selfhost {
+    print_info "configure cron for selfhost..."
+    if [ -z "$selfhost_user" ]
+    then
+        print_info "selfhost is not configured because selfhost_key is empty in $configfile"
+    else
+        # Use cron for dynamich ip update
+        #   - at reboot
+        #   - every 5 minutes
+        if [ -z "`grep 'selfhost-updater.sh' /etc/crontab`" ]
+        then
+            echo "@reboot root bash /etc/selfhost/selfhost-updater.sh update > /dev/null 2>&1" >> /etc/crontab
+            echo "*/5 * * * * root /bin/bash /etc/selfhost/selfhost-updater.sh update > /dev/null 2>&1" >> /etc/crontab
+        else
+            print_info "cron for selfhost was configured already"
+        fi
+    fi
+}
+
+function install_letsencrypt {
+    print_info "installing let's encrypt ..."
+    # check if user gave domain
+    if [ -z "$le_domain" ]
+    then
+        die "Failed to install let's encrypt: 'le_domain' is empty in $configfile"
+    fi
+    if [ -z "$le_email" ]
+    then
+        die "Failed to install let's encrypt: 'le_email' is empty in $configfile"
+    fi
+    nocheck_install "certbot python-certbot-apache"
+    print_info "run certbot ..."
+    certbot --apache -w $install_path -d $le_domain -m $le_email --agree-tos --non-interactive --redirect --hsts --uir
+    service apache2 restart
+}
+
+function check_https {
+    print_info "checking httpS > testing ..."
+    url_https=https://$le_domain
+    wget_output=$(wget -nv --spider --max-redirect 0 $url_https)
+    if [ $? -ne 0 ]
+    then
+        print_warn "check not ok"
+    else
+        print_info "check ok"
+    fi
+}
+
+function install_zotserver {
+    print_info "installing addons..."
+    cd $install_path
+    util/add_addon_repo https://framagit.org/hubzilla/addons hzaddons
+    mkdir -p "store/[data]/smarty3"
+    # chmod -R 777 store
+    touch .htconfig.php
+    # The next run of $cron_job (daily-update script) will correct the permissions of the next line
+    chmod ou+w .htconfig.php  
+    cd /var/www/
+    chown -R www-data:www-data $install_path
+	chown root:www-data $install_path/
+	chown root:www-data $install_path/.htaccess
+	chmod 0644 $install_path/.htaccess
+    print_info "installed addons"
+}
+
+function configure_cron_daily {
+    print_info "configuring cron..."
+    # every 10 min for poller.php
+    if [ -z "`grep 'php Zotlabs/Daemon/Master.php' /etc/crontab`" ]
+    then
+        echo "*/10 * * * * www-data cd $install_path; php Zotlabs/Daemon/Master.php Cron >> /dev/null 2>&1" >> /etc/crontab
+    fi
+    # Run external script daily at 05:30
+    # - stop apache/nginx and mysql-server
+    # - renew the certificate of letsencrypt
+    # - update repository core and addon
+    # - update and upgrade linux
+    # - reboot is done by "shutdown -h now" because "reboot" hangs sometimes depending on the system
+    echo "#!/bin/sh" > /var/www/$cron_job
+    echo "#" >> /var/www/$cron_job
+    echo "echo \" \"" >> /var/www/$cron_job
+    echo "echo \"+++ \$(date) +++\"" >> /var/www/$cron_job
+    echo "echo \" \"" >> /var/www/$cron_job
+    echo "echo \"\$(date) - stopping apache and mysql...\"" >> /var/www/$cron_job
+    echo "service apache2 stop" >> /var/www/$cron_job
+    echo "/etc/init.d/mysql stop # to avoid inconsistencies" >> /var/www/$cron_job
+    echo "#" >> /var/www/$cron_job
+    echo "echo \"\$(date) - renew certificate...\"" >> /var/www/$cron_job
+    echo "certbot renew --noninteractive" >> /var/www/$cron_job
+    echo "#" >> /var/www/$cron_job
+    echo "echo \"\$(date) - db size...\"" >> /var/www/$cron_job
+    echo "du -h /var/lib/mysql/ | grep mysql/" >> /var/www/$cron_job
+    echo "#" >> /var/www/$cron_job    
+    echo "# update of $le_domain Zot hub/instance" >> /var/www/$cron_job
+    echo "echo \"\$(date) - updating core and addons...\"" >> /var/www/$cron_job
+    echo "echo \"reaching git repository for $le_domain $zotserver hub/instance...\"" >> /var/www/$cron_job
+    echo "(cd $install_path ; util/udall)" >> /var/www/$cron_job
+    echo "chown -R www-data:www-data $install_path # make all accessible for the webserver" >> /var/www/$cron_job
+    echo "chown root:www-data $install_path/.htaccess" >> /var/www/$cron_job
+    echo "chmod 0644 $install_path/.htaccess # www-data can read but not write it" >> /var/www/$cron_job    
+    echo "echo \"\$(date) - updating linux...\"" >> /var/www/$cron_job
+    echo "apt-get -q -y update && apt-get -q -y dist-upgrade && apt-get -q -y autoremove # update linux and upgrade" >> /var/www/$cron_job
+    echo "echo \"\$(date) - Update finished. Rebooting...\"" >> /var/www/$cron_job
+    echo "#" >> /var/www/$cron_job
+    echo "shutdown -r now" >> /var/www/$cron_job
+
+    chmod a+x /var/www/$cron_job
+
+    # If global cron job does not exist we add it to /etc/crontab
+    if grep -q $cron_job /etc/crontab
+    then
+        echo "cron job already in /etc/crontab"
+    else
+        echo "30 05 * * * root /bin/bash /var/www/$cron_job >> /var/www/daily-updates.log 2>&1" >> /etc/crontab
+        echo "0 0 1 * * root rm /var/www/daily-updates.log" >> /etc/crontab
+    fi
+
+    # This is active after either "reboot" or cron reload"
+    systemctl restart cron
+    print_info "configured cron for updates/upgrades"
+}
+
+########################################################################
+# START OF PROGRAM
+########################################################################
+export PATH=/bin:/usr/bin:/sbin:/usr/sbin
+check_sanity
+
+print_info "We're installing a $zotserver instance"
+install_path="$(dirname "$(pwd)")"
+
+# Read config file edited by user
+configfile=config.txt
+source $configfile
+
+selfhostdir=/etc/selfhost
+selfhostscript=selfhost-updater.sh
+cron_job="cron_job.sh"
+
+#set -x    # activate debugging from here
+
+zotserver=hubzilla
+check_config
+stop_zotserver
+update_upgrade
+install_curl
+install_wget
+install_sendmail
+install_apache
+install_imagemagick
+install_php
+install_composer
+install_mysql
+install_adminer
+create_zotserver_db
+run_freedns
+install_run_selfhost
+ping_domain
+configure_cron_freedns
+configure_cron_selfhost
+
+if [ "$le_domain" != "localhost" ]
+then
+    install_letsencrypt
+    check_https
+else
+    print_info "is localhost - skipped installation of letsencrypt and configuration of apache for https"
+fi
+
+install_zotserver
+
+configure_cron_daily
+
+
+#set +x    # stop debugging from here

.gitignore

@@ -33,8 +33,10 @@ apps/
 home.html
 # page header plugin
 pageheader.html
-# Ignore site TOS
+# Ignore site TOS & gddpr
 doc/SiteTOS.md
+doc/*/SiteTOS.md
+doc/*/gdpr.md
 # themes except for redbasic
 view/theme/*
 !view/theme/redbasic
@@ -47,6 +49,8 @@ doc/html/
 # external repositories for themes/addons
 extend/
 # files generated by phpunit
+tests/.cache
+tests/.phpunit.result.cache
 tests/results/
 
 ## exclude IDE files
@@ -55,6 +59,7 @@ tests/results/
 .buildpath
 .externalToolBuilders
 .settings/
+.pydevproject
 # NetBeans project folder
 nbproject/
 # Kdevelop project files
@@ -75,3 +80,36 @@ composer.phar
 vendor/**/tests/
 vendor/**/Test/
 vendor/sabre/*/examples/
+
+# Exclude dev dependencies
+vendor/bin/pdepend
+vendor/bin/php-parse
+vendor/bin/phpcbf
+vendor/bin/phpcs
+vendor/bin/phpmd
+vendor/bin/phpunit
+vendor/composer/pcre/
+vendor/composer/xdebug-handler/
+vendor/dms/
+vendor/doctrine/
+vendor/myclabs/
+vendor/nikic/
+vendor/pdepend/
+vendor/phar-io/
+vendor/php-mock/
+vendor/phpmd/
+vendor/phpunit/
+vendor/psr/container/
+vendor/sebastian/
+vendor/squizlabs/
+vendor/symfony/config/
+vendor/symfony/dependency-injection/
+vendor/symfony/deprecation-contracts/
+vendor/symfony/filesystem/
+vendor/symfony/polyfill-ctype/
+vendor/symfony/polyfill-mbstring/
+vendor/symfony/polyfill-php80/
+vendor/symfony/service-contracts/
+vendor/theseer/
+# /info is a directory containing site-specific HTML documents
+/info/

.gitlab-ci.yml

@@ -1,8 +1,3 @@
-# Select image from https://hub.docker.com/_/php/
-#image: php:7.3
-# Use a prepared Hubzilla image to optimise pipeline duration
-image: registry.gitlab.com/dawnbreak/hubzilla/core:php7.3
-
 stages:
   - test
   - deploy
@@ -22,6 +17,7 @@ variables:
   # Ignore a Composer warning
   COMPOSER_ALLOW_SUPERUSER: 1
   # Configure MySQL/MariaDB service (https://hub.docker.com/_/mysql/, https://hub.docker.com/_/mariadb/)
+  DB_HOST: mysql
   MYSQL_DATABASE: hello_world_test
   MYSQL_ROOT_PASSWORD: mysql
   # Configure PostgreSQL service (https://hub.docker.com/_/postgres/)
@@ -31,47 +27,69 @@ variables:
 
 
 before_script:
-# pecl and composer do not work with PHP production restrictions (from Hubzilla Docker image)
-- if [ -f /usr/local/etc/php/conf.d/z_prod.ini ]; then mv /usr/local/etc/php/conf.d/z_prod.ini /usr/local/etc/php/conf.d/z_prod.ini.off; fi
-# Install & enable Xdebug for code coverage reports
-- pecl install xdebug
-- docker-php-ext-enable xdebug
-# Install composer
-- curl -sS https://getcomposer.org/installer | php
-# Install dev libraries from composer
-- php ./composer.phar install --no-progress
-# php.ini settings
-- echo 'xdebug.mode=coverage' >> /usr/local/etc/php/php.ini
+  # Install & enable Xdebug for code coverage reports
+  - apt-get update
+  - apt-get install -yqq libicu-dev libjpeg-dev libpng-dev libpq-dev libyaml-dev libzip-dev mariadb-client postgresql-client unzip zip
+  - pecl install xdebug yaml
+  - docker-php-ext-enable xdebug yaml
+  - docker-php-ext-install gd bcmath intl pdo_mysql pdo_pgsql zip
+
+  # Install composer
+  - curl -sS https://getcomposer.org/installer | php
+  # Install dev libraries from composer
+  - php ./composer.phar install --no-progress
+  # php.ini settings
+  - echo 'xdebug.mode=coverage' >> /usr/local/etc/php/php.ini
 
 # hidden job definition with template for MySQL/MariaDB
 .job_template_mysql: &job_definition_mysql
   stage: test
+  variables:
+    HZ_TEST_DB_HOST: $DB_HOST
+    HZ_TEST_DB_TYPE: mysql
+    HZ_TEST_DB_USER: root
+    HZ_TEST_DB_PASS: $MYSQL_ROOT_PASSWORD
+    HZ_TEST_DB_DATABASE: $MYSQL_DATABASE
   script:
-  - echo "USE $MYSQL_DATABASE; $(cat ./install/schema_mysql.sql)" | mysql --user=root --password="$MYSQL_ROOT_PASSWORD" --host=mysql "$MYSQL_DATABASE"
-  - echo "SHOW DATABASES;" | mysql --user=root --password="$MYSQL_ROOT_PASSWORD" --host=mysql "$MYSQL_DATABASE"
-  - echo "USE $MYSQL_DATABASE; SHOW TABLES;" | mysql --user=root --password="$MYSQL_ROOT_PASSWORD" --host=mysql "$MYSQL_DATABASE"
-  - vendor/bin/phpunit --configuration tests/phpunit.xml --coverage-text
+    # Import hubzilla's DB schema
+    - echo "USE $MYSQL_DATABASE; $(cat ./install/schema_mysql.sql)" | mysql --user=root --password="$MYSQL_ROOT_PASSWORD" --host="$DB_HOST" "$MYSQL_DATABASE"
+    # Show databases and relations/tables of hubzilla's database
+    - echo "SHOW DATABASES;" | mysql --user=root --password="$MYSQL_ROOT_PASSWORD" --host="$DB_HOST" "$MYSQL_DATABASE"
+    - echo "USE $MYSQL_DATABASE; SHOW TABLES;" | mysql --user=root --password="$MYSQL_ROOT_PASSWORD" --host="$DB_HOST" "$MYSQL_DATABASE"
+    # Run the actual tests
+    - touch dbfail.out
+    - vendor/bin/phpunit --configuration tests/phpunit.xml --no-progress --stop-on-error --coverage-text --colors=never --log-junit tests/results/junit.xml || exit_code=$?
+    - if [ $exit_code -ne 0 ]; then echo "Test barfed!"; cat dbfail.out; exit $exit_code; fi
+  coverage: '/^\s*Lines:\s*\d+.\d+\%/'
+
 
 # hidden job definition with template for PostgreSQL
 .job_template_postgres: &job_definition_postgres
   stage: test
-  services:
-  - postgres:latest
+  variables:
+    HZ_TEST_DB_HOST: postgres
+    HZ_TEST_DB_TYPE: postgres
+    HZ_TEST_DB_USER: $POSTGRES_USER
+    HZ_TEST_DB_PASS: $POSTGRES_PASSWORD
+    HZ_TEST_DB_DATABASE: $POSTGRES_DB
   script:
-  - export PGPASSWORD=$POSTGRES_PASSWORD
-  - psql --version
-  - psql -h "postgres" -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "SELECT VERSION();"
-  # Import hubzilla's DB schema
-  - psql -h "postgres" -U "$POSTGRES_USER" -v ON_ERROR_STOP=1 --quiet "$POSTGRES_DB" < ./install/schema_postgres.sql
-  # Show databases and relations/tables of hubzilla's database
-  #- psql -h "postgres" -U "$POSTGRES_USER" -l
-  #- psql -h "postgres" -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "\dt;"
-  # Run the actual tests
-  - vendor/bin/phpunit --configuration tests/phpunit-pgsql.xml --testdox
+    - export PGPASSWORD=$POSTGRES_PASSWORD
+    - psql --version
+    - psql -h "postgres" -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "SELECT VERSION();"
+    # Import hubzilla's DB schema
+    - psql -h "postgres" -U "$POSTGRES_USER" -v ON_ERROR_STOP=1 --quiet "$POSTGRES_DB" < ./install/schema_postgres.sql
+    # Show databases and relations/tables of hubzilla's database
+    - psql -h "postgres" -U "$POSTGRES_USER" -l
+    - psql -h "postgres" -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "\dt;"
+    # Run the actual tests
+    - touch dbfail.out
+    - vendor/bin/phpunit --configuration tests/phpunit.xml --no-progress --stop-on-error --coverage-text --colors=never --log-junit tests/results/junit.xml || exit_code=$?
+    - if [ $exit_code -ne 0 ]; then echo "Test barfed!"; cat dbfail.out; exit $exit_code; fi
+  coverage: '/^\s*Lines:\s*\d+.\d+\%/'
 
 # hidden job definition with artifacts config template
-.artifacts_template:
-  artifacts: &artifacts_template
+.artifacts_template: &artifacts_template
+  artifacts:
     expire_in: 1 week
     # Gitlab should show the results, but has problems parsing PHPUnit's junit file.
     reports:
@@ -82,56 +100,37 @@ before_script:
       - tests/results/
 
 
-# PHP7.3 with MySQL 5.7
-php7.3_mysql5.7:
-  <<: *job_definition_mysql
+# PHP8.1 with MySQL 8.0
+php8.1_mysql8.0.22:
+  image: php:8.1
   services:
-  - mysql:5.7
-
-
-# PHP7.3 with MySQL 8 (latest)
-php7.3_mysql8:
+    - mysql:8.0
   <<: *job_definition_mysql
+  <<: *artifacts_template
+
+# PHP8.1 with MariaDB 10.6
+php8.1_mariadb10.6:
+  image: php:8.1
   services:
-  - name: mysql:8
-    command: ["--default-authentication-plugin=mysql_native_password"]
-
-
-# PHP7.3 with MariaDB 10.2
-php7.3_mariadb10.2:
+    - name: mariadb:10.6
+      alias: mysql
   <<: *job_definition_mysql
+  <<: *artifacts_template
+
+# PHP8.1 with PostgreSQL 12
+php8.1_postgres12:
+  image: php:8.1
   services:
-  - name: mariadb:10.2
-    alias: mysql
-
-
-# PHP7.3 with MariaDB 10.3 (latest)
-php7.3_mariadb10.3:
-  <<: *job_definition_mysql
-  image: registry.gitlab.com/dawnbreak/hubzilla/core:php7.3
-  services:
-  - name: mariadb:10.3
-    alias: mysql
-
-
-# PHP7.3 with PostgreSQL latest (11)
-php7.3_postgres11:
+    - postgres:12-alpine
   <<: *job_definition_postgres
-  artifacts: *artifacts_template
-
-
-# PHP7.3 with PostgreSQL latest (11)
-php7.3_postgres11:
-  <<: *job_definition_postgres
-  image: registry.gitlab.com/dawnbreak/hubzilla/core:php7.3
-  artifacts: *artifacts_template
+  <<: *artifacts_template
 
 
 # Generate Doxygen API Documentation and deploy it as GitLab pages
 pages:
   stage: deploy
   cache: {}
-  image: php:7-cli-alpine
+  image: php:8-cli-alpine
   before_script:
     - apk update
     - apk add doxygen ttf-freefont graphviz

.htaccess

@@ -25,10 +25,10 @@ AddType audio/ogg .oga
   # in CGI mode.
 
   RewriteCond %{REQUEST_URI} ^/\.well\-known/.*
-  RewriteRule ^(.*)$ index.php?q=$1 [E=REMOTE_USER:%{HTTP:Authorization},L,QSA]
+  RewriteRule ^(.*)$ index.php?q=$1 "[E=REMOTE_USER:%{HTTP:Authorization},L,QSA,B= ?]"
 
   RewriteCond %{REQUEST_FILENAME} !-f
   RewriteCond %{REQUEST_FILENAME} !-d
-  RewriteRule ^(.*)$ index.php?q=$1 [E=REMOTE_USER:%{HTTP:Authorization},L,QSA]
+  RewriteRule ^(.*)$ index.php?q=$1 "[E=REMOTE_USER:%{HTTP:Authorization},L,QSA,B= ?]"
 </IfModule>
 

.phpcs.xml

@@ -0,0 +1,78 @@
+<?xml version="1.0"?>
+<ruleset
+    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+    name="PHP_CodeSniffer"
+    xsi:noNamespaceSchemaLocation="phpcs.xsd"
+    >
+
+	<description>PHP CodeSniffer config for Hubzilla</description>
+
+    <file>app</file>
+    <file>boot.php</file>
+	<file>include</file>
+	<file>index.php</file>
+	<file>install</file>
+	<file>library</file>
+	<file>tests</file>
+	<file>util</file>
+	<file>view</file>
+	<file>Zotlabs</file>
+
+    <rule ref="Generic">
+        <exclude name="Generic.Arrays.ArrayIndent"/>
+        <exclude name="Generic.Arrays.DisallowLongArraySyntax"/>
+        <exclude name="Generic.Arrays.DisallowShortArraySyntax"/>
+        <exclude name="Generic.Files.EndFileNoNewline"/>
+        <exclude name="Generic.Files.LowercasedFilename"/>
+        <exclude name="Generic.Formatting.MultipleStatementAlignment"/>
+        <exclude name="Generic.Formatting.SpaceAfterNot"/>
+        <exclude name="Generic.Functions.FunctionCallArgumentSpacing"/>
+        <exclude name="Generic.Functions.OpeningFunctionBraceBsdAllman"/>
+        <exclude name="Generic.NamingConventions.CamelCapsFunctionName"/>
+        <exclude name="Generic.PHP.ClosingPHPTag"/>
+        <exclude name="Generic.PHP.RequireStrictTypes"/>
+        <exclude name="Generic.PHP.UpperCaseConstant"/>
+        <exclude name="Generic.WhiteSpace.DisallowTabIndent"/>
+        <exclude name="Generic.WhiteSpace.ScopeIndent"/>
+        <exclude name="Generic.Commenting.DocComment.ContentAfterOpen"/>
+        <exclude name="Generic.Commenting.DocComment.ContentBeforeClose"/>
+        <exclude name="Generic.Commenting.DocComment.LongNotCapital"/>
+        <exclude name="Generic.Commenting.DocComment.MissingShort"/>
+        <exclude name="Generic.Commenting.DocComment.NonParamGroup"/>
+        <exclude name="Generic.Commenting.DocComment.ParamNotFirst"/>
+        <exclude name="Generic.Commenting.DocComment.ShortNotCapital"/>
+        <exclude name="Generic.Commenting.DocComment.SpacingAfter"/>
+        <exclude name="Generic.Commenting.DocComment.SpacingBeforeShort"/>
+        <exclude name="Generic.Commenting.DocComment.TagValueIndent"/>
+        <exclude name="Generic.ControlStructures.InlineControlStructure.NotAllowed"/>
+        <exclude name="Generic.Files.OneClassPerFile.MultipleFound"/>
+        <exclude name="Generic.Files.OneObjectStructurePerFile.MultipleFound"/>
+        <exclude name="Generic.Formatting.SpaceAfterCast.NoSpace"/>
+    </rule>
+
+	<!--
+		Warn about lines longer than 100 columns, lines longer than 150
+		columns will flag an error.
+	-->
+	<rule ref="Generic.Files.LineLength">
+		<properties>
+			<property name="lineLimit" value="100" />
+			<property name="absoluteLineLimit" value="150" />
+		</properties>
+	</rule>
+
+    <!--
+        Mark deprecated functions.
+    -->
+    <rule ref="Generic.PHP.DeprecatedFunctions">
+        <properties>
+            <property name="forbiddenFunctions" type="array" extend="true">
+                <element key="load_config" value="Zotlabs\Lib\Config::Load" />
+                <element key="get_config" value="Zotlabs\Lib\Config::Get" />
+                <element key="set_config" value="Zotlabs\Lib\Config::Set" />
+                <element key="del_config" value="Zotlabs\Lib\Config::Delete" />
+            </property>
+        </properties>
+    </rule>
+
+</ruleset>

CHANGELOG.air

@@ -0,0 +1,48 @@
+"air" is a branch name for revision of Account-Invite-Register at the Hubzilla project
+
+Invite:
+* Rewritten and now language template driven
+* Selectable templates for the invite mails
+* Invitor may add personal notes in the mailtext
++ Invite codes are bound to the recipients email address
+* Invite mod never more creates accounts
+* new db scheme for table register
+* existing register table will be migrated to the new schema even when detected at runtime
+* Bugfix: creating invite codes when admin only calls Invite w/o any further action
+* account library revision also together with invite mod
+* Depending on config: Users may send invitations also
+* Invitations expires, controlled by the invitor
+* Changed and new configs:
+* * invitation_only       As usual before
+* * invitation_also       Beside other registration policies, invitations may be used also
+* * invitation_max_per_day	defaults 50, may be changed in adminUI admin>site
+* * invitation_max_per_user defaults 4
+* Requirements:
+* * Addon language has to be installed
+
+Register:
+* Register panel (form) and js interaction changed
+* Unused registrations expire
+* Depending on config, anonymous registrations (w/o email) are supported
+* :... dont't panic, that may let grow security
+* Even anonymous users have to confirm their registration
+* Registrations may be enabled / disabled time driven for each day in the week (dudy)
+* Unsoliced registration floods may be blocked
+* Limited registrations from one single source ip
+* Using one additional log file, to easy interfare with f2b
+
+Account:
+* An user account always becomes created only if all depending conditions are satisfied
+* AdminUI for site configuration, accounts and registrations enhancements
+* Still untouched, but accountUI needs enhanced async control in case for mass delete
+  with deep level of recursion cascade of the dependencies (channels etc). An open TODO
+  since years for instances with many much users and channels.
+
+History:
+2020.03 Hubzilla Prod version 4.6 (master branch) of hubzilla/core was the base for AIR 
+        that was assigned Version 4.6.2 at sn/core
+2021.02 Hubzilla Prod version 5.2.1 (master branch) of hubzilla/core was new base for AIR
+        that was assigned version 5.2.2 at sn/core (air.5)
+        plus adjustment of hubzilla 5.2.2 (master) to sn/core (air.5) version 5.2.9
+
+

SBOM.md

@@ -2,32 +2,37 @@
 
 |Name|Version|License|Source|
 |----|-------|-------|------|
-|blueimp/jquery-file-upload|10.31.0.0|MIT|https://github.com/vkhramtsov/jQuery-File-Upload.git|
-|brick/math|0.9.1.0|MIT|https://github.com/brick/math.git|
+|blueimp/jquery-file-upload|10.32.0.0|MIT|https://github.com/vkhramtsov/jQuery-File-Upload.git|
+|brick/math|0.9.3.0|MIT|https://github.com/brick/math.git|
 |bshaffer/oauth2-server-php|1.11.1.0|MIT|https://github.com/bshaffer/oauth2-server-php.git|
-|commerceguys/intl|1.0.7.0|MIT|https://github.com/commerceguys/intl.git|
+|commerceguys/intl|1.1.0.0|MIT|https://github.com/commerceguys/intl.git|
 |desandro/imagesloaded|4.1.4.0|MIT|https://github.com/desandro/imagesloaded.git|
 |ezyang/htmlpurifier|4.13.0.0|LGPL-2.1-or-later|https://github.com/ezyang/htmlpurifier.git|
-|league/html-to-markdown|4.10.0.0|MIT|https://github.com/thephpleague/html-to-markdown.git|
+|jbroadway/urlify|1.2.2.0|BSD-3-Clause-Clear|https://github.com/jbroadway/urlify.git|
+|league/html-to-markdown|5.0.1.0|MIT|https://github.com/thephpleague/html-to-markdown.git|
 |lukasreschke/id3parser|0.0.3.0|GPL|https://github.com/LukasReschke/ID3Parser.git|
 |michelf/php-markdown|1.9.0.0|BSD-3-Clause|https://github.com/michelf/php-markdown.git|
 |pear/text_languagedetect|1.0.1.0|BSD-2-Clause|https://github.com/pear/Text_LanguageDetect.git|
-|psr/log|1.1.3.0|MIT|https://github.com/php-fig/log.git|
-|ramsey/collection|1.1.1.0|MIT|https://github.com/ramsey/collection.git|
-|ramsey/uuid|4.1.1.0|MIT|https://github.com/ramsey/uuid.git|
-|sabre/dav|4.1.3.0|BSD-3-Clause|https://github.com/sabre-io/dav.git|
+|phpseclib/phpseclib|2.0.33.0|MIT|https://github.com/phpseclib/phpseclib.git|
+|psr/log|1.1.4.0|MIT|https://github.com/php-fig/log.git|
+|ramsey/collection|1.2.2.0|MIT|https://github.com/ramsey/collection.git|
+|ramsey/uuid|4.2.3.0|MIT|https://github.com/ramsey/uuid.git|
+|sabre/dav|4.1.5.0|BSD-3-Clause|https://github.com/sabre-io/dav.git|
 |sabre/event|5.1.2.0|BSD-3-Clause|https://github.com/sabre-io/event.git|
 |sabre/http|5.1.1.0|BSD-3-Clause|https://github.com/sabre-io/http.git|
 |sabre/uri|2.2.1.0|BSD-3-Clause|https://github.com/sabre-io/uri.git|
-|sabre/vobject|4.3.3.0|BSD-3-Clause|https://github.com/sabre-io/vobject.git|
+|sabre/vobject|4.3.5.0|BSD-3-Clause|https://github.com/sabre-io/vobject.git|
 |sabre/xml|2.2.3.0|BSD-3-Clause|https://github.com/sabre-io/xml.git|
 |simplepie/simplepie|1.5.6.0|BSD-3-Clause|https://github.com/simplepie/simplepie.git|
-|smarty/smarty|3.1.36.0|LGPL-3.0|https://github.com/smarty-php/smarty.git|
-|symfony/polyfill-ctype|1.20.0.0|MIT|https://github.com/symfony/polyfill-ctype.git|
-|twbs/bootstrap|4.5.3.0|MIT|https://github.com/twbs/bootstrap.git|
+|smarty/smarty|3.1.39.0|LGPL-3.0|https://github.com/smarty-php/smarty.git|
+|symfony/polyfill-ctype|1.23.0.0|MIT|https://github.com/symfony/polyfill-ctype.git|
+|symfony/polyfill-php80|1.23.1.0|MIT|https://github.com/symfony/polyfill-php80.git|
+|symfony/polyfill-php81|1.23.0.0|MIT|https://github.com/symfony/polyfill-php81.git|
+|twbs/bootstrap|5.1.3.0|MIT|https://github.com/twbs/bootstrap.git|
+|voku/portable-ascii|1.5.6.0|MIT|https://github.com/voku/portable-ascii.git|
+|voku/stop-words|2.0.1.0|MIT|https://github.com/voku/stop-words.git|
 |fullcalendar/fullcalendar|4.4.2.0|MIT|https://github.com/fullcalendar/fullcalendar.git|
 |miromannino/Justified-Gallery|3.8.1.0|MIT|https://github.com/miromannino/Justified-Gallery.git|
 |fengyuanchen/cropperjs|1.5.7.0|MIT|https://github.com/fengyuanchen/cropperjs.git|
 |ForkAwesome/Fork-Awesome|1.1.7.0|MIT,SIL OFL,CC BY 3.0|https://github.com/ForkAwesome/Fork-Awesome.git|
-|leafo/sticky-kit|1.1.2.0|MIT|https://github.com/leafo/sticky-kit.git|
 |jquery/jquery|3.5.1.0|MIT|https://github.com/jquery/jquery.git|

ServiceWorker.js

@@ -0,0 +1,10 @@
+// This file should be served from the web root to avoid scope and cookie related issues with some browsers
+self.addEventListener('install', function(e) {
+	console.log('install event');
+});
+
+self.addEventListener('fetch', function(e) {
+	// nothing here yet
+	return;
+});
+

Zotlabs/Access/AccessList.php

@@ -2,6 +2,7 @@
 
 namespace Zotlabs\Access;
 
+
 /**
  * @brief AccessList class which represents individual content ACLs.
  *
@@ -17,29 +18,48 @@ class AccessList {
 	 * @brief Allow contacts
 	 * @var string
 	 */
-	private $allow_cid;
+	private ?string $allow_cid;
 	/**
 	 * @brief Allow groups
 	 * @var string
 	 */
-	private $allow_gid;
+	private ?string $allow_gid;
 	/**
 	 * @brief Deny contacts
 	 * @var string
 	 */
-	private $deny_cid;
+	private ?string $deny_cid;
 	/**
 	 * @brief Deny groups
 	 * @var string
 	 */
-	private $deny_gid;
+	private ?string $deny_gid;
 	/**
 	 * @brief Indicates if we are using the default constructor values or
 	 * values that have been set explicitly.
 	 * @var boolean
 	 */
-	private $explicit;
+	private bool $explicit;
 
+	/**
+	 * @brief Keys required by the constructor if the channel array is given.
+	 */
+	private const REQUIRED_KEYS_CONSTRUCTOR = [
+		'channel_allow_cid',
+		'channel_allow_gid',
+		'channel_deny_cid',
+		'channel_deny_gid'
+	];
+
+	/**
+	 * @brief Keys required by the set method.
+	 */
+	private const REQUIRED_KEYS_SET = [
+		'allow_cid',
+		'allow_gid',
+		'deny_cid',
+		'deny_gid'
+	];
 
 	/**
 	 * @brief Constructor for AccessList class.
@@ -53,8 +73,9 @@ class AccessList {
 	 *   * \e string \b channel_deny_cid => string of denied cids
 	 *   * \e string \b channel_deny_gid => string of denied gids
 	 */
-	function __construct($channel) {
-		if($channel) {
+	function __construct(array $channel) {
+		if ($channel) {
+			$this->validate_input_array($channel, self::REQUIRED_KEYS_CONSTRUCTOR);
 			$this->allow_cid = $channel['channel_allow_cid'];
 			$this->allow_gid = $channel['channel_allow_gid'];
 			$this->deny_cid  = $channel['channel_deny_cid'];
@@ -70,13 +91,24 @@ class AccessList {
 		$this->explicit = false;
 	}
 
+	private function validate_input_array(array $arr, array $required_keys) : void {
+		$missing_keys = array_diff($required_keys, array_keys($arr));
+
+		if (!empty($missing_keys)) {
+			throw new \Exception(
+				'Invalid AccessList object: Expected array with keys: '
+				. implode(', ', $missing_keys)
+			);
+		}
+	}
+
 	/**
 	 * @brief Get if we are using the default constructor values
 	 * or values that have been set explicitly.
 	 *
 	 * @return boolean
 	 */
-	function get_explicit() {
+	function get_explicit() : bool {
 		return $this->explicit;
 	}
 
@@ -94,12 +126,13 @@ class AccessList {
 	 *   * \e string \b deny_gid  => string of denied gids
 	 * @param boolean $explicit (optional) default true
 	 */
-	function set($arr, $explicit = true) {
+	function set(array $arr, bool $explicit = true) : void {
+		$this->validate_input_array($arr, self::REQUIRED_KEYS_SET);
+
 		$this->allow_cid = $arr['allow_cid'];
 		$this->allow_gid = $arr['allow_gid'];
 		$this->deny_cid  = $arr['deny_cid'];
 		$this->deny_gid  = $arr['deny_gid'];
-
 		$this->explicit  = $explicit;
 	}
 
@@ -113,7 +146,7 @@ class AccessList {
 	 *   * \e string \b deny_cid  => string of denied cids
 	 *   * \e string \b deny_gid  => string of denied gids
 	 */
-	function get() {
+	function get() : array {
 		return [
 			'allow_cid' => $this->allow_cid,
 			'allow_gid' => $this->allow_gid,
@@ -139,7 +172,12 @@ class AccessList {
 	 *   * \e array|string \b group_deny    => array with gids or comma-seperated string
 	 * @param boolean $explicit (optional) default true
 	 */
-	function set_from_array($arr, $explicit = true) {
+	function set_from_array(array $arr, bool $explicit = true) : void {
+		$arr['contact_allow'] = $arr['contact_allow'] ?? [];
+		$arr['group_allow'] = $arr['group_allow'] ?? [];
+		$arr['contact_deny'] = $arr['contact_deny'] ?? [];
+		$arr['group_deny'] = $arr['group_deny'] ?? [];
+
 		$this->allow_cid = perms2str((is_array($arr['contact_allow']))
 			? $arr['contact_allow'] : explode(',', $arr['contact_allow']));
 		$this->allow_gid = perms2str((is_array($arr['group_allow']))
@@ -157,7 +195,7 @@ class AccessList {
 	 *
 	 * @return boolean Return true if any of allow_* deny_* values is set.
 	 */
-	function is_private() {
+	function is_private() : bool {
 		return (($this->allow_cid || $this->allow_gid || $this->deny_cid || $this->deny_gid) ? true : false);
 	}
 

Zotlabs/Access/PermissionLimits.php

@@ -2,6 +2,7 @@
 
 namespace Zotlabs\Access;
 
+use App;
 use Zotlabs\Lib\PConfig;
 
 /**
@@ -39,10 +40,10 @@ class PermissionLimits {
 	 */
 	static public function Std_Limits() {
 		$limits = [];
-		$perms = Permissions::Perms();
+		$perms  = Permissions::Perms();
 
-		foreach($perms as $k => $v) {
-			if(strstr($k, 'view'))
+		foreach ($perms as $k => $v) {
+			if (strstr($k, 'view'))
 				$limits[$k] = PERMS_PUBLIC;
 			else
 				$limits[$k] = PERMS_SPECIFIC;
@@ -77,15 +78,15 @@ class PermissionLimits {
 	 *   * \b array with all permission limits, if $perm is not set
 	 */
 	static public function Get($channel_id, $perm = '') {
-		if($perm) {
+		if ($perm) {
 			return intval(PConfig::Get($channel_id, 'perm_limits', $perm));
 		}
 
 		PConfig::Load($channel_id);
-		if(array_key_exists($channel_id, \App::$config)
-				&& array_key_exists('perm_limits', \App::$config[$channel_id]))
-			return \App::$config[$channel_id]['perm_limits'];
+		if (array_key_exists($channel_id, App::$config)
+			&& array_key_exists('perm_limits', App::$config[$channel_id]))
+			return App::$config[$channel_id]['perm_limits'];
 
 		return false;
 	}
-}
+}

Zotlabs/Access/PermissionRoles.php

@@ -2,6 +2,8 @@
 
 namespace Zotlabs\Access;
 
+use Zotlabs\Lib\Config;
+
 /**
  * @brief PermissionRoles class.
  *
@@ -17,7 +19,7 @@ class PermissionRoles {
 	 * @return number
 	 */
 	static public function version() {
-		return 2;
+		return 3;
 	}
 
 	static function role_perms($role) {
@@ -27,6 +29,54 @@ class PermissionRoles {
 		$ret['role'] = $role;
 
 		switch($role) {
+
+			case 'public':
+				$ret['default_collection'] = false;
+				$ret['perms_connect'] = [
+					'view_stream', 'view_profile', 'view_contacts', 'view_storage', 'view_pages', 'view_wiki',
+					'send_stream', 'post_comments', 'post_mail', 'post_wall', 'chat', 'post_like', 'republish'
+				];
+				$ret['limits'] = PermissionLimits::Std_Limits();
+				$ret['limits']['post_comments'] = PERMS_AUTHED;
+				$ret['limits']['post_mail'] = PERMS_AUTHED;
+				$ret['limits']['post_like'] = PERMS_AUTHED;
+				$ret['limits']['chat'] = PERMS_AUTHED;
+				break;
+
+			// Hubzilla default role
+			case 'personal':
+				$ret['default_collection'] = true;
+				$ret['perms_connect'] = [
+					'view_stream', 'view_profile', 'view_contacts', 'view_storage', 'view_pages', 'view_wiki',
+					'send_stream', 'post_comments', 'post_mail', 'chat', 'post_like'
+				];
+				$ret['limits'] = PermissionLimits::Std_Limits();
+				$ret['limits']['view_contacts'] = PERMS_SPECIFIC;
+				break;
+
+			case 'group':
+				$ret['default_collection'] = false;
+				$ret['perms_connect'] = [
+					'view_stream', 'view_profile', 'view_contacts', 'view_storage',
+					'view_pages', 'view_wiki', 'post_wall', 'post_comments',
+					'post_mail', 'post_like', 'chat'
+				];
+				$ret['limits'] = PermissionLimits::Std_Limits();
+				$ret['channel_type'] = 'group';
+				break;
+
+			// Provide some defaults for the custom role so that we do not start
+			// with no permissions at all if we create a new channel with this role
+			case 'custom':
+				$ret['default_collection'] = true;
+				$ret['perms_connect'] = [
+					'view_stream', 'view_profile', 'view_contacts', 'view_storage', 'view_pages', 'view_wiki',
+					'send_stream', 'post_comments', 'post_mail', 'chat', 'post_like'
+				];
+				$ret['limits'] = PermissionLimits::Std_Limits();
+				break;
+
+/*
 			case 'social':
 				$ret['perms_auto'] = false;
 				$ret['default_collection'] = false;
@@ -193,13 +243,14 @@ class PermissionRoles {
 				$ret['channel_type'] = 'group';
 
 				break;
+*/
 
-			case 'custom':
 			default:
 				break;
 		}
 
-		$x = get_config('system','role_perms');
+		$x = Config::Get('system','role_perms');
+
 		// let system settings over-ride any or all
 		if($x && is_array($x) && array_key_exists($role,$x))
 			$ret = array_merge($ret,$x[$role]);
@@ -218,13 +269,13 @@ class PermissionRoles {
 		// set permissionlimits for this permission here, for example:
 
 		// if($perm === 'mynewperm')
-		//     \Zotlabs\Access\PermissionLimits::Set($uid,$perm,1);
+		// 		PermissionLimits::Set($uid,$perm,1);
 
 		if($perm === 'view_wiki')
-			\Zotlabs\Access\PermissionLimits::Set($uid, $perm, PERMS_PUBLIC);
+			PermissionLimits::Set($uid, $perm, PERMS_PUBLIC);
 
 		if($perm === 'write_wiki')
-			\Zotlabs\Access\PermissionLimits::Set($uid, $perm, PERMS_SPECIFIC);
+			PermissionLimits::Set($uid, $perm, PERMS_SPECIFIC);
 
 
 		// set autoperms here if applicable
@@ -262,11 +313,11 @@ class PermissionRoles {
 					case 'view_wiki':
 						set_abconfig($uid,$ab['abook_xchan'],'my_perms',$perm,
 							intval(get_abconfig($uid,$ab['abook_xchan'],'my_perms','view_pages')));
-
+						break;
 					case 'write_wiki':
 						set_abconfig($uid,$ab['abook_xchan'],'my_perms',$perm,
 							intval(get_abconfig($uid,$ab['abook_xchan'],'my_perms','write_pages')));
-
+						break;
 					default:
 						break;
 				}
@@ -284,6 +335,7 @@ class PermissionRoles {
 	 */
 	static public function roles() {
 		$roles = [
+
 			t('Social Networking') => [
 				'social_federation' => t('Social - Federation'),
 				'social' => t('Social - Mostly Public'),
@@ -317,4 +369,29 @@ class PermissionRoles {
 		return $roles;
 	}
 
-}
+	/**
+	 * @brief Array with translated role names and grouping.
+	 *
+	 * Return an associative array with role names that can be used
+	 * to create select groups like in \e field_select_grouped.tpl.
+	 *
+	 * @return array
+	 */
+	static public function channel_roles() {
+		$channel_roles = [
+			//'public' => [t('Public'), t('A very permissive role suited for participation in the fediverse')],
+			//'personal' => [t('Personal'), t('The $Projectname default role suited for a personal channel')],
+			//'forum' => [t('Community forum'), t('This role configures your channel to act as an community forum')],
+			//'custom' => [t('Custom'), t('This role comes with the presets of the personal role but allows you to configure it to your needs')]
+			'public' => t('Public'),
+			'personal' => t('Personal'),
+			'group' => t('Community forum'),
+			'custom' => t('Custom')
+		];
+
+		call_hooks('list_channel_roles', $channel_roles);
+
+		return $channel_roles;
+	}
+
+}

Zotlabs/Access/Permissions.php

@@ -41,7 +41,7 @@ class Permissions {
 	 * @return number
 	 */
 	static public function version() {
-		return 2;
+		return 3;
 	}
 
 	/**
@@ -65,17 +65,17 @@ class Permissions {
 			'write_wiki'    => t('Can write to my wiki pages'),
 			'post_wall'     => t('Can post on my channel (wall) page'),
 			'post_comments' => t('Can comment on or like my posts'),
-			'post_mail'     => t('Can send me private mail messages'),
+			'post_mail'     => t('Can send me direct messages'),
 			'post_like'     => t('Can like/dislike profiles and profile things'),
-			'tag_deliver'   => t('Can forward to all my channel connections via ! mentions in posts'),
 			'chat'          => t('Can chat with me'),
-			'republish'     => t('Can source my public posts in derived channels'),
+			'republish'     => t('Can source/mirror my public posts in derived channels'),
+			//'tag_deliver'   => t('Can forward to my contacts via direct messages (forum)'),
 			'delegate'      => t('Can administer my channel')
 		];
 
 		$x = [
 			'permissions' => $perms,
-			'filter' => $filter
+			'filter'      => $filter
 		];
 		/**
 		 * @hooks permissions_list
@@ -84,7 +84,7 @@ class Permissions {
 		 */
 		call_hooks('permissions_list', $x);
 
-		return($x['permissions']);
+		return ($x['permissions']);
 	}
 
 	/**
@@ -96,10 +96,10 @@ class Permissions {
 	 */
 	static public function BlockedAnonPerms() {
 
-		$res = [];
+		$res   = [];
 		$perms = PermissionLimits::Std_limits();
-		foreach($perms as $perm => $limit) {
-			if($limit != PERMS_PUBLIC) {
+		foreach ($perms as $perm => $limit) {
+			if ($limit != PERMS_PUBLIC) {
 				$res[] = $perm;
 			}
 		}
@@ -111,7 +111,7 @@ class Permissions {
 		 */
 		call_hooks('write_perms', $x);
 
-		return($x['permissions']);
+		return ($x['permissions']);
 	}
 
 	/**
@@ -120,20 +120,20 @@ class Permissions {
 	 * Converts [ 0 => 'view_stream', ... ]
 	 * to [ 'view_stream' => 1 ] for any permissions in $arr;
 	 * Undeclared permissions which exist in Perms() are added and set to 0.
- 	 *
+	 *
 	 * @param array $arr
 	 * @return array
 	 */
 	static public function FilledPerms($arr) {
-		if(is_null($arr)) {
+		if (is_null($arr)) {
 			btlogger('FilledPerms: null');
 			$arr = [];
 		}
 
 		$everything = self::Perms();
-		$ret = [];
-		foreach($everything as $k => $v) {
-			if(in_array($k, $arr))
+		$ret        = [];
+		foreach ($everything as $k => $v) {
+			if (in_array($k, $arr))
 				$ret[$k] = 1;
 			else
 				$ret[$k] = 0;
@@ -155,9 +155,9 @@ class Permissions {
 	 */
 	static public function OPerms($arr) {
 		$ret = [];
-		if($arr) {
-			foreach($arr as $k => $v) {
-				$ret[] = [ 'name' => $k, 'value' => $v ];
+		if ($arr) {
+			foreach ($arr as $k => $v) {
+				$ret[] = ['name' => $k, 'value' => $v];
 			}
 		}
 		return $ret;
@@ -170,15 +170,16 @@ class Permissions {
 	 * @return boolean|array
 	 */
 	static public function FilledAutoperms($channel_id) {
-		if(! intval(get_pconfig($channel_id,'system','autoperms')))
+		if (!intval(get_pconfig($channel_id, 'system', 'autoperms')))
 			return false;
 
 		$arr = [];
+
 		$r = q("select * from pconfig where uid = %d and cat = 'autoperms'",
 			intval($channel_id)
 		);
-		if($r) {
-			foreach($r as $rr) {
+		if ($r) {
+			foreach ($r as $rr) {
 				$arr[$rr['k']] = intval($rr['v']);
 			}
 		}
@@ -193,11 +194,11 @@ class Permissions {
 	 * @return boolean true if all perms from $p1 exist also in $p2
 	 */
 	static public function PermsCompare($p1, $p2) {
-		foreach($p1 as $k => $v) {
-			if(! array_key_exists($k, $p2))
+		foreach ($p1 as $k => $v) {
+			if (!array_key_exists($k, $p2))
 				return false;
 
-			if($p1[$k] != $p2[$k])
+			if ($p1[$k] != $p2[$k])
 				return false;
 		}
 
@@ -214,84 +215,85 @@ class Permissions {
 	 */
 	static public function connect_perms($channel_id) {
 
-		$my_perms = [];
-		$permcat = null;
-		$automatic = 0;
+		$my_perms  = [];
+		$permcat   = null;
+		$automatic = get_pconfig($channel_id, 'system', 'autoperms');
 
 		// If a default permcat exists, use that
 
-		$pc = ((feature_enabled($channel_id,'permcats')) ? get_pconfig($channel_id,'system','default_permcat') : 'default');
-		if(! in_array($pc, [ '','default' ])) {
-			$pcp = new Zlib\Permcat($channel_id);
-			$permcat = $pcp->fetch($pc);
-			if($permcat && $permcat['perms']) {
-				foreach($permcat['perms'] as $p) {
-					$my_perms[$p['name']] = $p['value'];
-				}
+		$pc = get_pconfig($channel_id, 'system', 'default_permcat', 'default');
+		$pcp     = new Zlib\Permcat($channel_id);
+		$permcat = $pcp->fetch($pc);
+		if ($permcat && $permcat['perms']) {
+			foreach ($permcat['perms'] as $p) {
+				$my_perms[$p['name']] = $p['value'];
 			}
 		}
 
 		// look up the permission role to see if it specified auto-connect
 		// and if there was no permcat or a default permcat, set the perms
 		// from the role
-
-		$role = get_pconfig($channel_id,'system','permissions_role');
-		if($role) {
+/*
+		$role = get_pconfig($channel_id, 'system', 'permissions_role');
+		if ($role) {
 			$xx = PermissionRoles::role_perms($role);
-			if($xx['perms_auto'])
+			if ($xx['perms_auto'])
 				$automatic = 1;
 
-			if((! $my_perms) && ($xx['perms_connect'])) {
+			if ((!$my_perms) && ($xx['perms_connect'])) {
 				$default_perms = $xx['perms_connect'];
-				$my_perms = Permissions::FilledPerms($default_perms);
+				$my_perms      = Permissions::FilledPerms($default_perms);
 			}
 		}
+*/
 
 		// If we reached this point without having any permission information,
 		// it is likely a custom permissions role. First see if there are any
 		// automatic permissions.
-
-		if(! $my_perms) {
+/*
+		if (!$my_perms) {
 			$m = Permissions::FilledAutoperms($channel_id);
-			if($m) {
+			if ($m) {
 				$automatic = 1;
-				$my_perms = $m;
+				$my_perms  = $m;
 			}
 		}
-
+*/
 		// If we reached this point with no permissions, the channel is using
 		// custom perms but they are not automatic. They will be stored in abconfig with
 		// the channel's channel_hash (the 'self' connection).
 
-		if(! $my_perms) {
+/*
+		if (!$my_perms) {
 			$r = q("select channel_hash from channel where channel_id = %d",
 				intval($channel_id)
 			);
-			if($r) {
+			if ($r) {
 				$x = q("select * from abconfig where chan = %d and xchan = '%s' and cat = 'my_perms'",
 					intval($channel_id),
 					dbesc($r[0]['channel_hash'])
 				);
-				if($x) {
-					foreach($x as $xv) {
+				if ($x) {
+					foreach ($x as $xv) {
 						$my_perms[$xv['k']] = intval($xv['v']);
 					}
 				}
 			}
 		}
-
-		return ( [ 'perms' => $my_perms, 'automatic' => $automatic ] );
+*/
+		return (['perms' => $my_perms, 'automatic' => $automatic, 'role' => $pc]);
 	}
-
+/*
 	static public function serialise($p) {
 		$n = [];
-		if($p) {
-			foreach($p as $k => $v) {
-				if(intval($v)) {
+		if ($p) {
+			foreach ($p as $k => $v) {
+				if (intval($v)) {
 					$n[] = $k;
 				}
 			}
 		}
-		return implode(',',$n);
+		return implode(',', $n);
 	}
+*/
 }

Zotlabs/Daemon/Addon.php

@@ -2,13 +2,13 @@
 
 namespace Zotlabs\Daemon;
 
-require_once('include/zot.php');
-
 class Addon {
 
-	static public function run($argc,$argv) {
+	static public function run($argc, $argv) {
 
-		call_hooks('daemon_addon',$argv);
+		call_hooks('daemon_addon', $argv);
+		return;
 
 	}
+
 }

Zotlabs/Daemon/Cache_embeds.php

@@ -2,7 +2,6 @@
 
 namespace Zotlabs\Daemon;
 
-
 class Cache_embeds {
 
 	static public function run($argc,$argv) {
@@ -20,8 +19,10 @@ class Cache_embeds {
 		$item = $c[0];
 
 		// bbcode conversion by default processes embeds that aren't already cached.
-		// Ignore the returned html output. 
+		// Ignore the returned html output.
 
 		bbcode($item['body']);
+
+		return;
 	}
 }

Zotlabs/Daemon/Cache_query.php

@@ -0,0 +1,42 @@
+<?php
+
+namespace Zotlabs\Daemon;
+
+use Zotlabs\Lib\Cache;
+use Zotlabs\Lib\Config;
+
+class Cache_query {
+
+	static public function run($argc, $argv) {
+
+		if(! $argc == 3)
+			return;
+
+		$r = null;
+		$key = $argv[1];
+
+		$pid = Config::Get('procid', $key, false);
+		if ($pid && (function_exists('posix_kill') ? posix_kill($pid, 0) : true)) {
+			logger($key . ': procedure already run with pid ' . $pid, LOGGER_DEBUG);
+			return;
+		}
+
+		$pid = getmypid();
+		Config::Set('procid', $key, $pid);
+
+		array_shift($argv);
+		array_shift($argv);
+
+		$arr = json_decode(base64_decode($argv[0]), true);
+
+		$r = call_user_func_array('q', $arr);
+
+		if(is_array($r)) {
+			Cache::set($key, serialize($r));
+		}
+
+		Config::Delete('procid', $key);
+
+		return;
+	}
+}

Zotlabs/Daemon/Channel_purge.php

@@ -0,0 +1,34 @@
+<?php
+
+namespace Zotlabs\Daemon;
+
+class Channel_purge {
+
+	static public function run($argc,$argv) {
+
+		cli_startup();
+
+		$channel_id = intval($argv[1]);
+
+		$channel = q("select * from channel where channel_id = %d and channel_removed = 1",
+			intval($channel_id)
+		);
+
+		if (! $channel) {
+			return;
+		}
+
+		do {
+			$r = q("select id from item where uid = %d and item_deleted = 0 limit 1000",
+				intval($channel_id)
+			);
+			if ($r) {
+				foreach ($r as $rv) {
+					drop_item($rv['id'], false);
+				}
+			}
+		} while ($r);
+
+		return;
+	}
+}

Zotlabs/Daemon/Checksites.php

@@ -2,38 +2,44 @@
 
 namespace Zotlabs\Daemon;
 
+use Zotlabs\Lib\Config;
+
 require_once('include/hubloc.php');
 
 class Checksites {
 
-	static public function run($argc,$argv) {
+	static public function run($argc, $argv) {
 
 		logger('checksites: start');
-	
-		if(($argc > 1) && ($argv[1]))
+
+		$site_id = '';
+		$sql_options = '';
+
+		if (($argc > 1) && ($argv[1]))
 			$site_id = $argv[1];
 
-		if($site_id)
+		if ($site_id)
 			$sql_options = " and site_url = '" . dbesc($argv[1]) . "' ";
 
-		$days = intval(get_config('system','sitecheckdays'));
-		if($days < 1)
+		$days = intval(Config::Get('system', 'sitecheckdays'));
+		if ($days < 1)
 			$days = 30;
 
 		$r = q("select * from site where site_dead = 0 and site_update < %s - INTERVAL %s and site_type = %d $sql_options ",
-			db_utcnow(), db_quoteinterval($days . ' DAY'),
+			db_utcnow(),
+			db_quoteinterval($days . ' DAY'),
 			intval(SITE_TYPE_ZOT)
 		);
 
-		if(! $r)
+		if (!$r)
 			return;
 
-		foreach($r as $rr) {
-			if(! strcasecmp($rr['site_url'],z_root()))
+		foreach ($r as $rr) {
+			if (!strcasecmp($rr['site_url'], z_root()))
 				continue;
 
 			$x = ping_site($rr['site_url']);
-			if($x['success']) {
+			if ($x['success']) {
 				logger('checksites: ' . $rr['site_url']);
 				q("update site set site_update = '%s' where site_url = '%s' ",
 					dbesc(datetime_convert()),

Zotlabs/Daemon/Cli_suggest.php

@@ -9,6 +9,7 @@ class Cli_suggest {
 	static public function run($argc,$argv) {
 
 		update_suggestions();
+		return;
 
 	}
 }

Zotlabs/Daemon/Content_importer.php

@@ -0,0 +1,77 @@
+<?php
+
+namespace Zotlabs\Daemon;
+
+use Zotlabs\Web\HTTPSig;
+use Zotlabs\Lib\PConfig;
+
+
+require_once('include/cli_startup.php');
+require_once('include/attach.php');
+require_once('include/import.php');
+
+class Content_importer {
+
+	static public function run($argc,$argv) {
+		cli_startup();
+
+		$page = $argv[1];
+		$since = $argv[2];
+		$until = $argv[3];
+		$channel_address = $argv[4];
+		$hz_server = urldecode($argv[5]);
+
+		$m = parse_url($hz_server);
+
+		$channel = channelx_by_nick($channel_address);
+		if(! $channel) {
+			logger('channel not found');
+			return;
+		}
+
+		$headers = [
+			'X-API-Token'      => random_string(),
+			'X-API-Request'    => $hz_server . '/api/z/1.0/item/export_page?f=&since=' . urlencode($since) . '&until=' . urlencode($until) . '&page=' . $page ,
+			'Host'             => $m['host'],
+			'(request-target)' => 'get /api/z/1.0/item/export_page?f=&since=' . urlencode($since) . '&until=' . urlencode($until) . '&page=' . $page ,
+		];
+
+		$headers = HTTPSig::create_sig($headers,$channel['channel_prvkey'], channel_url($channel),true,'sha512');
+
+		$x = z_fetch_url($hz_server . '/api/z/1.0/item/export_page?f=&since=' . urlencode($since) . '&until=' . urlencode($until) . '&page=' . $page,false,$redirects,[ 'headers' => $headers ]);
+
+		// logger('item fetch: ' . print_r($x,true));
+
+		if(! $x['success']) {
+			logger('no API response',LOGGER_DEBUG);
+			killme();
+		}
+
+		$j = json_decode($x['body'],true);
+
+		if(! is_array($j['item']) || ! count($j['item'])) {
+			PConfig::Set($channel['channel_id'], 'import', 'content_completed', 1);
+			return;
+		}
+
+		$saved_notification_flags = notifications_off($channel['channel_id']);
+
+		import_items($channel,$j['item'],false,((array_key_exists('relocate',$j)) ? $j['relocate'] : null));
+
+		notifications_on($channel['channel_id'], $saved_notification_flags);
+
+		PConfig::Set($channel['channel_id'], 'import', 'content_progress', [
+			'items_total' => $j['items_total'],
+			'items_page' => $j['items_page'],
+			'items_current_page' => count($j['item']),
+			'last_page' => $page,
+			'next_cmd' => ['Content_importer', sprintf('%d',$page + 1), $since, $until, $channel['channel_address'], urlencode($hz_server)]
+		]);
+
+		$page++;
+
+		Master::Summon([ 'Content_importer', sprintf('%d',$page), $since, $until, $channel['channel_address'], urlencode($hz_server) ]);
+
+		return;
+	}
+}

Zotlabs/Daemon/Convo.php

@@ -0,0 +1,66 @@
+<?php
+
+namespace Zotlabs\Daemon;
+
+use Zotlabs\Lib\Activity;
+use Zotlabs\Lib\ActivityStreams;
+use Zotlabs\Lib\ASCollection;
+
+class Convo {
+
+	static public function run($argc, $argv) {
+
+		logger('convo invoked: ' . print_r($argv, true));
+
+		if ($argc != 4) {
+			return;
+		}
+
+		$id           = $argv[1];
+		$channel_id   = intval($argv[2]);
+		$contact_hash = $argv[3];
+
+		$channel = channelx_by_n($channel_id);
+		if (!$channel) {
+			return;
+		}
+
+		$r = q("SELECT abook.*, xchan.* FROM abook left join xchan on abook_xchan = xchan_hash
+			WHERE abook_channel = %d and abook_xchan = '%s' LIMIT 1",
+			intval($channel_id),
+			dbesc($contact_hash)
+		);
+
+		if (!$r) {
+			return;
+		}
+
+		$contact = array_shift($r);
+
+		$obj = new ASCollection($id, $channel);
+
+		$messages = $obj->get();
+
+		if (!$messages) {
+			return;
+		}
+
+		foreach ($messages as $message) {
+			if (is_string($message)) {
+				$message = Activity::fetch($message, $channel);
+			}
+
+			// set client flag because comments will probably just be objects and not full blown activities
+			// and that lets us use implied_create
+			$AS = new ActivityStreams($message);
+			if ($AS->is_valid() && is_array($AS->obj)) {
+				$item = Activity::decode_note($AS);
+				$item['item_fetched'] = true;
+				Activity::store($channel, $contact['abook_xchan'], $AS, $item);
+			}
+		}
+
+		return;
+
+	}
+}

Zotlabs/Daemon/Cron.php

@@ -2,40 +2,55 @@
 
 namespace Zotlabs\Daemon;
 
+use Zotlabs\Lib\Config;
 use Zotlabs\Lib\Libsync;
+use Zotlabs\Lib\Libzotdir;
 
 class Cron {
 
-	static public function run($argc,$argv) {
+	static public function run($argc, $argv) {
 
-		$maxsysload = intval(get_config('system','maxloadavg'));
-		if($maxsysload < 1)
+		$maxsysload = intval(Config::Get('system', 'maxloadavg'));
+		if ($maxsysload < 1)
 			$maxsysload = 50;
-		if(function_exists('sys_getloadavg')) {
+		if (function_exists('sys_getloadavg')) {
 			$load = sys_getloadavg();
-			if(intval($load[0]) > $maxsysload) {
+			if (intval($load[0]) > $maxsysload) {
 				logger('system: load ' . $load . ' too high. Cron deferred to next scheduled run.');
 				return;
 			}
 		}
 
+/*
 		// Check for a lockfile.  If it exists, but is over an hour old, it's stale.  Ignore it.
 		$lockfile = 'store/[data]/cron';
-		if((file_exists($lockfile)) && (filemtime($lockfile) > (time() - 3600)) 
-			&& (! get_config('system','override_cron_lockfile'))) {
+		if ((file_exists($lockfile)) && (filemtime($lockfile) > (time() - 3600))
+			&& (!Config::Get('system', 'override_cron_lockfile'))) {
 			logger("cron: Already running");
 			return;
 		}
-	
+
 		// Create a lockfile.  Needs two vars, but $x doesn't need to contain anything.
+		$x = '';
 		file_put_contents($lockfile, $x);
+*/
 
 		logger('cron: start');
-	
+
+		// If this is a directory server, request a sync with an upstream
+		// directory at least once a day, up to once every poll interval.
+		// Pull remote changes and push local changes.
+		// potential issue: how do we keep from creating an endless update loop?
+
+		$dirmode = Config::Get('system', 'directory_mode');
+
+		if ($dirmode == DIRECTORY_MODE_SECONDARY || $dirmode == DIRECTORY_MODE_PRIMARY) {
+			Libzotdir::sync_directories($dirmode);
+		}
+
 		// run queue delivery process in the background
 
 		Master::Summon(array('Queue'));
-
 		Master::Summon(array('Poller'));
 
 		/**
@@ -46,27 +61,29 @@ class Cron {
 			db_utcnow(),
 			db_quoteinterval('3 MINUTE')
 		);
-	
-		// expire any expired mail
 
-		q("delete from mail where expires > '%s' and expires < %s ",
-			dbesc(NULL_DATE),
-			db_utcnow()
-		);
+		require_once('include/account.php');
+		remove_expired_registrations();
+
+		$interval = Config::Get('queueworker', 'queue_interval', 500000);
 
 		// expire any expired items
 
-		$r = q("select id,item_wall from item where expires > '2001-01-01 00:00:00' and expires < %s 
+		$r = q("select id,item_wall from item where expires > '2001-01-01 00:00:00' and expires < %s
 			and item_deleted = 0 ",
 			db_utcnow()
 		);
-		if($r) {
+		if ($r) {
 			require_once('include/items.php');
-			foreach($r as $rr) {
-				drop_item($rr['id'],false,(($rr['item_wall']) ? DROPITEM_PHASE1 : DROPITEM_NORMAL));
-				if($rr['item_wall']) {
+			foreach ($r as $rr) {
+				drop_item($rr['id'], false, (($rr['item_wall']) ? DROPITEM_PHASE1 : DROPITEM_NORMAL));
+				if ($rr['item_wall']) {
 					// The notifier isn't normally invoked unless item_drop is interactive.
-					Master::Summon( [ 'Notifier', 'drop', $rr['id'] ] );
+					Master::Summon(['Notifier', 'drop', $rr['id']]);
+
+					if ($interval) {
+						usleep($interval);
+					}
 				}
 			}
 		}
@@ -78,9 +95,9 @@ class Cron {
 			dbesc(NULL_DATE),
 			db_utcnow()
 		);
-		if($r) {
+		if ($r) {
 			require_once('include/security.php');
-			foreach($r as $rr) {
+			foreach ($r as $rr) {
 				atoken_delete($rr['atoken_id']);
 			}
 		}
@@ -90,33 +107,35 @@ class Cron {
 		// or dead entries.
 
 		$r = q("select channel_id from channel where channel_dirdate < %s - INTERVAL %s and channel_removed = 0",
-			db_utcnow(), 
+			db_utcnow(),
 			db_quoteinterval('30 DAY')
 		);
-		if($r) {
-			foreach($r as $rr) {
-				Master::Summon(array('Directory',$rr['channel_id'],'force'));
-				if($interval)
-					@time_sleep_until(microtime(true) + (float) $interval);
+		if ($r) {
+			foreach ($r as $rr) {
+				Master::Summon(array('Directory', $rr['channel_id'], 'force'));
+
+				if ($interval) {
+					usleep($interval);
+				}
 			}
 		}
-		
+
 		// Clean expired photos from cache
-		
+
 		$r = q("SELECT DISTINCT xchan, content FROM photo WHERE photo_usage = %d AND expires < %s - INTERVAL %s",
 			intval(PHOTO_CACHE),
 			db_utcnow(),
-			db_quoteinterval(get_config('system','active_expire_days', '30') . ' DAY')
+			db_quoteinterval(Config::Get('system', 'cache_expire_days', 7) . ' DAY')
 		);
-		if($r) {
+		if ($r) {
 			q("DELETE FROM photo WHERE photo_usage = %d AND expires < %s - INTERVAL %s",
 				intval(PHOTO_CACHE),
 				db_utcnow(),
-				db_quoteinterval(get_config('system','active_expire_days', '30') . ' DAY')
+				db_quoteinterval(Config::Get('system', 'cache_expire_days', 7) . ' DAY')
 			);
-			foreach($r as $rr) {
+			foreach ($r as $rr) {
 				$file = dbunescbin($rr['content']);
-				if(is_file($file)) {
+				if (is_file($file)) {
 					@unlink($file);
 					@rmdir(dirname($file));
 					logger('info: deleted cached photo file ' . $file, LOGGER_DEBUG);
@@ -126,80 +145,74 @@ class Cron {
 
 		// publish any applicable items that were set to be published in the future
 		// (time travel posts). Restrict to items that have come of age in the last
-		// couple of days to limit the query to something reasonable. 
+		// couple of days to limit the query to something reasonable.
 
 		$r = q("select id from item where item_delayed = 1 and created <= %s  and created > '%s' ",
 			db_utcnow(),
-			dbesc(datetime_convert('UTC','UTC','now - 2 days'))
+			dbesc(datetime_convert('UTC', 'UTC', 'now - 2 days'))
 		);
-		if($r) {
-			foreach($r as $rr) {
+		if ($r) {
+			foreach ($r as $rr) {
 				$x = q("update item set item_delayed = 0 where id = %d",
 					intval($rr['id'])
 				);
-				if($x) {
+				if ($x) {
 					$z = q("select * from item where id = %d",
-						intval($message_id)
+						intval($rr['id'])
 					);
-					if($z) {
+					if ($z) {
 						xchan_query($z);
 						$sync_item = fetch_post_tags($z);
 						Libsync::build_sync_packet($sync_item[0]['uid'],
-							[ 
-								'item' => [ encode_item($sync_item[0],true) ]
+							[
+								'item' => [encode_item($sync_item[0], true)]
 							]
 						);
 					}
-					Master::Summon(array('Notifier','wall-new',$rr['id']));
+					Master::Summon(array('Notifier', 'wall-new', $rr['id']));
+
+					if ($interval) {
+						usleep($interval);
+					}
 				}
 			}
 		}
 
-
-		// check if any connections transitioned to zot6 and upgrade the connections to zot6 at this hub if so.
-		require_once('include/connections.php');
-		z6trans_connections();
-
 		require_once('include/attach.php');
 		attach_upgrade();
 
-		$abandon_days = intval(get_config('system','account_abandon_days'));
-		if($abandon_days < 1)
-			$abandon_days = 0;
-
-	
 		// once daily run birthday_updates and then expire in background
 
 		// FIXME: add birthday updates, both locally and for xprof for use
 		// by directory servers
 
-		$d1 = intval(get_config('system','last_expire_day'));
-		$d2 = intval(datetime_convert('UTC','UTC','now','d'));
+		$d1 = intval(Config::Get('system', 'last_expire_day'));
+		$d2 = intval(datetime_convert('UTC', 'UTC', 'now', 'd'));
 
 		// Allow somebody to staggger daily activities if they have more than one site on their server,
 		// or if it happens at an inconvenient (busy) hour.
 
-		$h1 = intval(get_config('system','cron_hour'));
-		$h2 = intval(datetime_convert('UTC','UTC','now','G'));
+		$h1 = intval(Config::Get('system', 'cron_hour'));
+		$h2 = intval(datetime_convert('UTC', 'UTC', 'now', 'G'));
 
 
-		if(($d2 != $d1) && ($h1 == $h2)) {
+		if (($d2 != $d1) && ($h1 == $h2)) {
 			Master::Summon(array('Cron_daily'));
 		}
 
 		// update any photos which didn't get imported properly
 		// This should be rare
 
-		$r = q("select xchan_photo_l, xchan_hash from xchan where xchan_photo_l != '' and xchan_photo_m = '' 
+		$r = q("select xchan_photo_l, xchan_hash from xchan where xchan_photo_l != '' and xchan_photo_m = ''
 			and xchan_photo_date < %s - INTERVAL %s",
-			db_utcnow(), 
+			db_utcnow(),
 			db_quoteinterval('1 DAY')
 		);
-		if($r) {
+		if ($r) {
 			require_once('include/photo/photo_driver.php');
-			foreach($r as $rr) {
+			foreach ($r as $rr) {
 				$photos = import_xchan_photo($rr['xchan_photo_l'], $rr['xchan_hash'], false, true);
-				$x = q("update xchan set xchan_photo_l = '%s', xchan_photo_m = '%s', xchan_photo_s = '%s', xchan_photo_mimetype = '%s'
+				q("update xchan set xchan_photo_l = '%s', xchan_photo_m = '%s', xchan_photo_s = '%s', xchan_photo_mimetype = '%s'
 					where xchan_hash = '%s'",
 					dbesc($photos[0]),
 					dbesc($photos[1]),
@@ -211,36 +224,32 @@ class Cron {
 		}
 
 
-		// pull in some public posts
+		// pull in some public posts if allowed
 
-		$disable_discover_tab = get_config('system','disable_discover_tab') || get_config('system','disable_discover_tab') === false;
-		if(! $disable_discover_tab)
-			Master::Summon(array('Externals'));
+		$disable_externals = Config::Get('system', 'disable_discover_tab') || Config::Get('system', 'disable_discover_tab') === false || Config::Get('system', 'site_firehose');
+		if (!$disable_externals)
+			Master::Summon(['Externals']);
 
-		$generation = 0;
+		$restart = false;
 
-		$restart    = false;
-
-		if(($argc > 1) && ($argv[1] == 'restart')) {
-			$restart = true;
+		if (($argc > 1) && ($argv[1] == 'restart')) {
+			$restart    = true;
 			$generation = intval($argv[2]);
-			if(! $generation)
+			if (!$generation)
 				return;
 		}
 
 		reload_plugins();
 
-		$d = datetime_convert();
-
 		// TODO check to see if there are any cronhooks before wasting a process
 
-		if(! $restart)
+		if (!$restart)
 			Master::Summon(array('Cronhooks'));
 
-		set_config('system','lastcron',datetime_convert());
+		Config::Set('system', 'lastcron', datetime_convert());
 
-		//All done - clear the lockfile	
-		@unlink($lockfile);
+		//All done - clear the lockfile
+		//@unlink($lockfile);
 
 		return;
 	}

Zotlabs/Daemon/Cron_daily.php

@@ -2,11 +2,12 @@
 
 namespace Zotlabs\Daemon;
 
+use Zotlabs\Lib\Config;
 use Zotlabs\Lib\Libzotdir;
 
 class Cron_daily {
 
-	static public function run($argc,$argv) {
+	static public function run($argc, $argv) {
 
 		logger('cron_daily: start');
 
@@ -15,14 +16,12 @@ class Cron_daily {
 		 *
 		 */
 
-
 		Libzotdir::check_upstream_directory();
 
-
 		// Fire off the Cron_weekly process if it's the correct day.
 
-		$d3 = intval(datetime_convert('UTC','UTC','now','N'));
-		if($d3 == 7) {
+		$d3 = intval(datetime_convert('UTC', 'UTC', 'now', 'N'));
+		if ($d3 == 7) {
 			Master::Summon(array('Cron_weekly'));
 		}
 
@@ -51,10 +50,26 @@ class Cron_daily {
 			dbesc('sse_id.%')
 		);
 
-		// Clean up emdedded content cache
+		// Mark items seen after X days (default 90)
+
+		$r = dbq("select channel_id from channel where channel_removed = 0");
+		if ($r) {
+			foreach ($r as $rr) {
+				$mark_seen_days = get_pconfig($rr['channel_id'], 'system', 'mark_seen_days', 90);
+				q("UPDATE item SET item_unseen = 0 WHERE
+					uid = %d AND item_unseen = 1
+					AND created < %s - INTERVAL %s",
+					intval($rr['channel_id']),
+					db_utcnow(),
+					db_quoteinterval($mark_seen_days . ' DAY')
+				);
+			}
+		}
+
+		// Clean up cache
 		q("DELETE FROM cache WHERE updated < %s - INTERVAL %s",
-		    db_utcnow(),
-		    db_quoteinterval(get_config('system','active_expire_days', '30') . ' DAY')
+			db_utcnow(),
+			db_quoteinterval(Config::Get('system', 'cache_expire_days', 7) . ' DAY')
 		);
 
 		//update statistics in config
@@ -68,8 +83,8 @@ class Cron_daily {
 
 		// expire old delivery reports
 
-		$keep_reports = intval(get_config('system','expire_delivery_reports'));
-		if($keep_reports === 0)
+		$keep_reports = intval(Config::Get('system', 'expire_delivery_reports'));
+		if ($keep_reports === 0)
 			$keep_reports = 10;
 
 		q("delete from dreport where dreport_time < %s - INTERVAL %s",
@@ -80,31 +95,21 @@ class Cron_daily {
 		// expire any expired accounts
 		downgrade_accounts();
 
-		// If this is a directory server, request a sync with an upstream
-		// directory at least once a day, up to once every poll interval.
-		// Pull remote changes and push local changes.
-		// potential issue: how do we keep from creating an endless update loop?
-
-		$dirmode = get_config('system','directory_mode');
-
-		if($dirmode == DIRECTORY_MODE_SECONDARY || $dirmode == DIRECTORY_MODE_PRIMARY) {
-			Libzotdir::sync_directories($dirmode);
-		}
-
-
 		Master::Summon(array('Expire'));
 		Master::Summon(array('Cli_suggest'));
 
 		remove_obsolete_hublocs();
+		remove_duplicate_singleton_hublocs();
 
-		z6_discover();
+		$date = datetime_convert();
+		call_hooks('cron_daily', $date);
 
-		call_hooks('cron_daily',datetime_convert());
-
-		set_config('system','last_expire_day',intval(datetime_convert('UTC','UTC','now','d')));
+		Config::Set('system', 'last_expire_day', intval(datetime_convert('UTC', 'UTC', 'now', 'd')));
 
 		/**
 		 * End Cron Daily
 		 */
+
+		 return;
 	}
 }

Zotlabs/Daemon/Cron_weekly.php

@@ -2,37 +2,40 @@
 
 namespace Zotlabs\Daemon;
 
+use Zotlabs\Lib\Config;
+
 class Cron_weekly {
 
-	static public function run($argc,$argv) {		
+	static public function run($argc, $argv) {
 
 		/**
 		 * Cron Weekly
-		 * 
+		 *
 		 * Actions in the following block are executed once per day only on Sunday (once per week).
 		 *
 		 */
 
-		call_hooks('cron_weekly',datetime_convert());
+		$date = datetime_convert();
+		call_hooks('cron_weekly', $date);
 
 		z_check_cert();
 
 		prune_hub_reinstalls();
-	
+
 		mark_orphan_hubsxchans();
 
-		// Find channels that were removed in the last three weeks, but 
+		// Find channels that were removed in the last three weeks, but
 		// haven't been finally cleaned up. These should be older than 10
-		// days to ensure that "purgeall" messages have gone out or bounced 
-		// or timed out. 
+		// days to ensure that "purgeall" messages have gone out or bounced
+		// or timed out.
 
-		$r = q("select channel_id from channel where channel_removed = 1 and 
+		$r = q("select channel_id from channel where channel_removed = 1 and
 			channel_deleted >  %s - INTERVAL %s and channel_deleted < %s - INTERVAL %s",
 			db_utcnow(), db_quoteinterval('21 DAY'),
 			db_utcnow(), db_quoteinterval('10 DAY')
 		);
-		if($r) {
-			foreach($r as $rv) {
+		if ($r) {
+			foreach ($r as $rv) {
 				channel_remove_final($rv['channel_id']);
 			}
 		}
@@ -43,14 +46,14 @@ class Cron_weekly {
 			db_utcnow(), db_quoteinterval('14 DAY')
 		);
 
-		$dirmode = intval(get_config('system','directory_mode'));
-		if($dirmode === DIRECTORY_MODE_SECONDARY || $dirmode === DIRECTORY_MODE_PRIMARY) {
-			logger('regdir: ' . print_r(z_fetch_url(get_directory_primary() . '/regdir?f=&url=' . urlencode(z_root()) . '&realm=' . urlencode(get_directory_realm())),true));
+		$dirmode = intval(Config::Get('system', 'directory_mode'));
+		if ($dirmode === DIRECTORY_MODE_SECONDARY || $dirmode === DIRECTORY_MODE_PRIMARY) {
+			logger('regdir: ' . print_r(z_fetch_url(get_directory_primary() . '/regdir?f=&url=' . urlencode(z_root()) . '&realm=' . urlencode(get_directory_realm())), true));
 		}
 
 		// Check for dead sites
 		Master::Summon(array('Checksites'));
-			
+
 		// update searchable doc indexes
 		Master::Summon(array('Importdoc'));
 
@@ -58,5 +61,6 @@ class Cron_weekly {
 		 * End Cron Weekly
 		 */
 
+		return;
 	}
-}
+}

Zotlabs/Daemon/CurlAuth.php

@@ -2,6 +2,8 @@
 
 namespace Zotlabs\Daemon;
 
+use App;
+
 // generate a curl compatible cookie file with an authenticated session for the given channel_id. 
 // If this file is then used with curl and the destination url is sent through zid() or manually 
 // manipulated to add a zid, it should allow curl to provide zot magic-auth across domains.
@@ -10,15 +12,15 @@ namespace Zotlabs\Daemon;
 
 class CurlAuth {
 
-	static public function run($argc,$argv) {
+	static public function run($argc, $argv) {
 
-		if($argc != 2)
+		if ($argc != 2)
 			return;
 
-		\App::$session->start();
+		App::$session->start();
 
 		$_SESSION['authenticated'] = 1;
-		$_SESSION['uid'] = $argv[1];
+		$_SESSION['uid']           = $argv[1];
 
 		$x = session_id();
 
@@ -29,14 +31,14 @@ class CurlAuth {
 
 		$output = '';
 
-		if($e) {
+		if ($e) {
 			$lines = file($f);
-			if($lines) {
-				foreach($lines as $line) {
-					if(strlen($line) > 0 && $line[0] != '#' && substr_count($line, "\t") == 6) {
+			if ($lines) {
+				foreach ($lines as $line) {
+					if (strlen($line) > 0 && $line[0] != '#' && substr_count($line, "\t") == 6) {
 						$tokens = explode("\t", $line);
 						$tokens = array_map('trim', $tokens);
-						if($tokens[4] > time()) {
+						if ($tokens[4] > time()) {
 							$output .= $line . "\n";
 						}
 					}
@@ -46,9 +48,9 @@ class CurlAuth {
 			}
 		}
 		$t = time() + (24 * 3600);
-		file_put_contents($f, $output . 'HttpOnly_' . \App::get_hostname() . "\tFALSE\t/\tTRUE\t$t\tPHPSESSID\t" . $x, (($e) ? FILE_APPEND : 0));
+		file_put_contents($f, $output . 'HttpOnly_' . App::get_hostname() . "\tFALSE\t/\tTRUE\t$t\tPHPSESSID\t" . $x, (($e) ? FILE_APPEND : 0));
 
-		file_put_contents($c,$x);
+		file_put_contents($c, $x);
 
 		return;
 	}

Zotlabs/Daemon/Deliver.php

@@ -2,32 +2,34 @@
 
 namespace Zotlabs\Daemon;
 
-require_once('include/queue_fn.php');
+use Zotlabs\Lib\Queue;
 
 class Deliver {
-	
-	static public function run($argc,$argv) {
 
-		if($argc < 2)
+	static public function run($argc, $argv) {
+
+		if ($argc < 2)
 			return;
 
-		logger('deliver: invoked: ' . print_r($argv,true), LOGGER_DATA);
+		logger('deliver: invoked: ' . print_r($argv, true), LOGGER_DATA);
 
-		for($x = 1; $x < $argc; $x ++) {
+		for ($x = 1; $x < $argc; $x++) {
 
-			if(! $argv[$x])
+			if (!$argv[$x])
 				continue;
 
 			$r = q("select * from outq where outq_hash = '%s'",
 				dbesc($argv[$x])
 			);
 
-			if($r) {
-				queue_deliver($r[0],true);
+			if ($r) {
+				Queue::deliver($r[0], true);
 			}
 
 		}
 
+		return;
+
 	}
 
 }

Zotlabs/Daemon/Deliver_hooks.php

@@ -2,21 +2,22 @@
 
 namespace Zotlabs\Daemon;
 
-require_once('include/zot.php');
-
 class Deliver_hooks {
 
-	static public function run($argc,$argv) {
+	static public function run($argc, $argv) {
 
-		if($argc < 2)
+		if ($argc < 2)
 			return;
 
-
 		$r = q("select * from item where id = '%d'",
 			intval($argv[1])
 		);
-		if($r)
-			call_hooks('notifier_normal',$r[0]);
+
+		if ($r) {
+			call_hooks('notifier_normal', $r[0]);
+		}
+
+		return;
 
 	}
 }

Zotlabs/Daemon/Delxitems.php

@@ -0,0 +1,25 @@
+<?php
+
+namespace Zotlabs\Daemon;
+
+require_once('include/connections.php');
+
+/*
+ * Daemon to remove 'item' resources in the background from a removed connection
+ */
+
+class Delxitems {
+
+	static public function run($argc, $argv) {
+
+		cli_startup();
+
+		if($argc != 3) {
+			return;
+		}
+
+		remove_abook_items($argv[1], $argv[2]);
+
+		return;
+	}
+}

Zotlabs/Daemon/Directory.php

@@ -2,46 +2,47 @@
 
 namespace Zotlabs\Daemon;
 
+use Zotlabs\Lib\Config;
 use Zotlabs\Lib\Libzot;
 use Zotlabs\Lib\Libzotdir;
 use Zotlabs\Lib\Queue;
 
 class Directory {
 
-	static public function run($argc,$argv){
+	static public function run($argc, $argv) {
 
-		if($argc < 2)
+		if ($argc < 2)
 			return;
 
-		$force = false;
+		$force   = false;
 		$pushall = true;
 
-		if($argc > 2) {
-			if($argv[2] === 'force')
+		if ($argc > 2) {
+			if ($argv[2] === 'force')
 				$force = true;
-			if($argv[2] === 'nopush')
+			if ($argv[2] === 'nopush')
 				$pushall = false;
-		}	
+		}
 
 		logger('directory update', LOGGER_DEBUG);
 
-		$dirmode = get_config('system','directory_mode');
-		if($dirmode === false)
+		$dirmode = Config::Get('system', 'directory_mode');
+		if ($dirmode === false)
 			$dirmode = DIRECTORY_MODE_NORMAL;
 
 		$x = q("select * from channel where channel_id = %d limit 1",
 			intval($argv[1])
 		);
-		if(! $x)
+		if (!$x)
 			return;
 
 		$channel = $x[0];
 
-		if($dirmode != DIRECTORY_MODE_NORMAL) {
+		if ($dirmode != DIRECTORY_MODE_NORMAL) {
 
 			// this is an in-memory update and we don't need to send a network packet.
 
-			Libzotdir::local_dir_update($argv[1],$force);
+			Libzotdir::local_dir_update($argv[1], $force);
 
 			q("update channel set channel_dirdate = '%s' where channel_id = %d",
 				dbesc(datetime_convert()),
@@ -49,8 +50,9 @@ class Directory {
 			);
 
 			// Now update all the connections
-			if($pushall) 
-				Master::Summon(array('Notifier','refresh_all',$channel['channel_id']));
+			if ($pushall) {
+				Master::Summon(array('Notifier', 'refresh_all', $channel['channel_id']));
+			}
 
 			return;
 		}
@@ -63,20 +65,18 @@ class Directory {
 
 		// ensure the upstream directory is updated
 
-
-		$packet = Libzot::build_packet($channel,(($force) ? 'force_refresh' : 'refresh'));
-		$z = Libzot::zot($url,$packet,$channel);
-
+		$packet = Libzot::build_packet($channel, (($force) ? 'force_refresh' : 'refresh'));
+		$z      = Libzot::zot($url, $packet, $channel);
 
 		// re-queue if unsuccessful
 
-		if(! $z['success']) {
+		if (!$z['success']) {
 
 			/** @FIXME we aren't updating channel_dirdate if we have to queue
 			 * the directory packet. That means we'll try again on the next poll run.
 			 */
 
-			$hash = random_string();
+			$hash = new_uuid();
 
 			Queue::insert(array(
 				'hash'       => $hash,
@@ -95,8 +95,10 @@ class Directory {
 		}
 
 		// Now update all the connections
-		if($pushall)
-			Master::Summon(array('Notifier','refresh_all',$channel['channel_id']));
+		if ($pushall) {
+			Master::Summon(array('Notifier', 'refresh_all', $channel['channel_id']));
+		}
 
+		return;
 	}
 }

Zotlabs/Daemon/Expire.php

@@ -2,26 +2,30 @@
 
 namespace Zotlabs\Daemon;
 
+use Zotlabs\Lib\Config;
+
+require_once('include/items.php');
 
 class Expire {
 
-	static public function run($argc,$argv){
+	static public function run($argc, $argv) {
 
 		cli_startup();
-		
-		$pid = get_config('expire', 'procid', false);
+
+		$pid = Config::Get('procid', 'expire', false);
 		if ($pid && (function_exists('posix_kill') ? posix_kill($pid, 0) : true)) {
-		    logger('Expire: procedure already run with pid ' . $pid, LOGGER_DEBUG);
-		    return;
+			logger('procedure already run with pid ' . $pid, LOGGER_DEBUG);
+			return;
 		}
-		
+
 		$pid = getmypid();
-		set_config('expire', 'procid', $pid);
+		Config::Set('procid', 'expire', $pid);
 
 		// perform final cleanup on previously delete items
 
 		$r = q("select id from item where item_deleted = 1 and item_pending_remove = 0 and changed < %s - INTERVAL %s",
-			db_utcnow(), db_quoteinterval('10 DAY')
+			db_utcnow(),
+			db_quoteinterval('10 DAY')
 		);
 		if ($r) {
 			foreach ($r as $rr) {
@@ -30,25 +34,23 @@ class Expire {
 		}
 
 		// physically remove anything that has been deleted for more than two months
-		/** @FIXME - this is a wretchedly inefficient query */
 
-		$r = q("delete from item where item_pending_remove = 1 and changed < %s - INTERVAL %s",
-			db_utcnow(), db_quoteinterval('36 DAY')
+		q("delete from item where item_pending_remove = 1 and changed < %s - INTERVAL %s",
+			db_utcnow(),
+			db_quoteinterval('36 DAY')
 		);
 
-		/** @FIXME make this optional as it could have a performance impact on large sites */
-
-		if (intval(get_config('system', 'optimize_items')))
+		if (intval(Config::Get('system', 'optimize_items')))
 			q("optimize table item");
 
 		logger('expire: start with pid ' . $pid, LOGGER_DEBUG);
 
-		$site_expire = intval(get_config('system', 'default_expire_days'));
-		$commented_days = intval(get_config('system','active_expire_days'));
+		$site_expire    = intval(Config::Get('system', 'default_expire_days', 30));
+		$commented_days = intval(Config::Get('system', 'active_expire_days', 7));
 
 		logger('site_expire: ' . $site_expire);
 
-		$r = q("SELECT channel_id, channel_system, channel_address, channel_expire_days from channel where true");
+		$r = dbq("SELECT channel_id, channel_system, channel_address, channel_expire_days from channel where true");
 
 		if ($r) {
 			foreach ($r as $rr) {
@@ -58,17 +60,18 @@ class Expire {
 					continue;
 
 				// service class default (if non-zero) over-rides the site default
-
 				$service_class_expire = service_class_fetch($rr['channel_id'], 'expire_days');
+
 				if (intval($service_class_expire))
 					$channel_expire = $service_class_expire;
 				else
 					$channel_expire = $site_expire;
-	
+
 				if (intval($channel_expire) && (intval($channel_expire) < intval($rr['channel_expire_days'])) ||
 					intval($rr['channel_expire_days'] == 0)) {
 					$expire_days = $channel_expire;
-				} else {
+				}
+				else {
 					$expire_days = $rr['channel_expire_days'];
 				}
 
@@ -83,8 +86,7 @@ class Expire {
 
 			// this should probably just fetch the channel_expire_days from the sys channel,
 			// but there's no convenient way to set it.
-
-			$expire_days = get_config('system', 'sys_expire_days');
+			$expire_days = Config::Get('system', 'sys_expire_days');
 			if ($expire_days === false)
 				$expire_days = 30;
 
@@ -93,13 +95,16 @@ class Expire {
 			}
 
 			logger('Expire: sys interval: ' . $expire_days, LOGGER_DEBUG);
-	
-			if ($expire_days)
+
+			if ($expire_days) {
 				item_expire($x['channel_id'], $expire_days, $commented_days);
+			}
 
 			logger('Expire: sys: done', LOGGER_DEBUG);
 		}
-		
-		del_config('expire', 'procid');
+
+		Config::Delete('procid', 'expire');
+
+		return;
 	}
 }

Zotlabs/Daemon/Externals.php

@@ -2,97 +2,156 @@
 
 namespace Zotlabs\Daemon;
 
-require_once('include/zot.php');
+use Zotlabs\Lib\Activity;
+use Zotlabs\Lib\Libzot;
+use Zotlabs\Lib\ActivityStreams;
+use Zotlabs\Lib\ASCollection;
+
 require_once('include/channel.php');
 
 
 class Externals {
 
-	static public function run($argc,$argv){
+	static public function run($argc, $argv) {
 
-		$total = 0;
+		logger('externals: start');
+
+		$importer = get_sys_channel();
+		$total    = 0;
 		$attempts = 0;
+		$url = '';
 
 		logger('externals: startup', LOGGER_DEBUG);
 
 		// pull in some public posts
 
+		while ($total == 0 && $attempts < 3) {
+			$arr = ['url' => ''];
+			call_hooks('externals_url_select', $arr);
 
-		while($total == 0 && $attempts < 3) {
-			$arr = array('url' => '');
-			call_hooks('externals_url_select',$arr);
-
-			if($arr['url']) {
+			if ($arr['url']) {
 				$url = $arr['url'];
-			} 
+			}
 			else {
+				$networks = ['zot6'];
+
+				if (plugin_is_installed('pubcrawl')) {
+					$networks[] = 'activitypub';
+				}
+
+				stringify_array_elms($networks);
+				$networks_str = implode(',', $networks);
+
 				$randfunc = db_getfunc('RAND');
 
-				// fixme this query does not deal with directory realms. 
+				// fixme this query does not deal with directory realms.
+				//$r = q("select site_url, site_pull from site where site_url != '%s'
+						//and site_flags != %d and site_type = %d
+						//and site_dead = 0 and site_project like '%s' and site_version > '5.3.1' order by $randfunc limit 1",
+					//dbesc(z_root()),
+					//intval(DIRECTORY_MODE_STANDALONE),
+					//intval(SITE_TYPE_ZOT),
+					//dbesc('hubzilla%')
+				//);
 
-				$r = q("select site_url, site_pull from site where site_url != '%s' and site_flags != %d and site_type = %d and site_dead = 0 order by $randfunc limit 1",
+				$r = q("SELECT * FROM hubloc
+					LEFT JOIN abook ON abook_xchan = hubloc_hash
+					LEFT JOIN site ON site_url = hubloc_url WHERE
+					hubloc_network IN ( $networks_str ) AND
+					abook_xchan IS NULL AND
+					hubloc_url != '%s' AND
+					hubloc_updated > '%s' AND
+					hubloc_primary = 1 AND hubloc_deleted = 0 AND
+					site_dead = 0
+					ORDER BY $randfunc LIMIT 1",
 					dbesc(z_root()),
-					intval(DIRECTORY_MODE_STANDALONE),
-					intval(SITE_TYPE_ZOT)
+					datetime_convert('UTC', 'UTC', 'now - 30 days')
 				);
-				if($r)
-					$url = $r[0]['site_url'];
+
+				if ($r) {
+					$contact = $r[0];
+					$url = $contact['hubloc_id_url'];
+				}
+			}
+
+			if (!$url) {
+				continue;
 			}
 
 			$blacklisted = false;
 
-			if(! check_siteallowed($url)) {
+			if (!check_siteallowed($contact['hubloc_url'])) {
 				logger('blacklisted site: ' . $url);
 				$blacklisted = true;
 			}
 
-			$attempts ++;
+			$attempts++;
 
 			// make sure we can eventually break out if somebody blacklists all known sites
 
-			if($blacklisted) {
-				if($attempts > 20)
+			if ($blacklisted) {
+				if ($attempts > 5)
 					break;
-				$attempts --;
+				$attempts--;
 				continue;
 			}
 
-			if($url) {
-				if($r[0]['site_pull'] > NULL_DATE)
-					$mindate = urlencode(datetime_convert('','',$r[0]['site_pull'] . ' - 1 day'));
-				else {
-					$days = get_config('externals','since_days');
-					if($days === false)
-						$days = 15;
-					$mindate = urlencode(datetime_convert('','','now - ' . intval($days) . ' days'));
-				}
+			$cl = Activity::get_actor_collections($contact['hubloc_hash']);
+			if(empty($cl)) {
+				$cl = get_xconfig($contact['hubloc_hash'], 'activitypub', 'collections');
+			}
 
-				$feedurl = $url . '/zotfeed?f=&mindate=' . $mindate;
-
-				logger('externals: pulling public content from ' . $feedurl, LOGGER_DEBUG);
-
-				$x = z_fetch_url($feedurl);
-				if(($x) && ($x['success'])) {
-
-					q("update site set site_pull = '%s' where site_url = '%s'",
-						dbesc(datetime_convert()),
-						dbesc($url)
-					);
-
-					$j = json_decode($x['body'],true);
-					if($j['success'] && $j['messages']) {
-						$sys = get_sys_channel();
-						foreach($j['messages'] as $message) {
-							// on these posts, clear any route info. 
-							$message['route'] = '';
-							$results = process_delivery(array('hash' => 'undefined'), get_item_elements($message),
-								array(array('hash' => $sys['xchan_hash'])), false, true);
-							$total ++;
-						}
-						logger('externals: import_public_posts: ' . $total . ' messages imported', LOGGER_DEBUG);
-					}
+			if (is_array($cl) && array_key_exists('outbox', $cl)) {
+				$url = $cl['outbox'];
+			}
+			else {
+				$url = str_replace('/channel/', '/outbox/', $contact['hubloc_id_url']);
+				if ($url) {
+					$url .= '?top=1';
 				}
 			}
+
+			if ($url) {
+				logger('fetching outbox: ' . $url);
+
+				$obj      = new ASCollection($url, $importer, 0, 10);
+				$messages = $obj->get();
+
+				if ($messages) {
+					foreach ($messages as $message) {
+						if (is_string($message)) {
+							$message = Activity::fetch($message, $importer);
+						}
+
+						if ($message['type'] !== 'Create') {
+							continue;
+						}
+
+						if ($contact['hubloc_network'] === 'zot6') {
+							// make sure we only fetch top level items
+							if (isset($message['object']['inReplyTo'])) {
+								continue;
+							}
+
+							$obj_id = $message['object']['id'] ?? $message['object'];
+
+							Libzot::fetch_conversation($importer, $obj_id);
+							$total++;
+							continue;
+						}
+
+						$AS = new ActivityStreams($message);
+						if ($AS->is_valid() && is_array($AS->obj)) {
+							$item = Activity::decode_note($AS);
+							$item['item_fetched'] = true;
+							Activity::store($importer, $contact['hubloc_hash'], $AS, $item);
+							$total++;
+						}
+					}
+				}
+				logger('fetched messages count: ' . $total);
+			}
 		}
+		return;
 	}
 }

Zotlabs/Daemon/Fetchparents.php

@@ -0,0 +1,42 @@
+<?php
+
+namespace Zotlabs\Daemon;
+
+use Zotlabs\Lib\Activity;
+
+class Fetchparents {
+
+	static public function run($argc, $argv) {
+
+		logger('Fetchparents invoked: ' . print_r($argv, true));
+
+		if ($argc < 4) {
+			return;
+		}
+
+		$channels = explode(',', $argv[1]);
+		if (!$channels) {
+			return;
+		}
+
+		$observer_hash = $argv[2];
+		if (!$observer_hash) {
+			return;
+		}
+
+		$mid = $argv[3];
+		if (!$mid) {
+			return;
+		}
+
+		$force = $argv[4] ?? false;
+
+		foreach ($channels as $channel_id) {
+			$channel = channelx_by_n($channel_id);
+			Activity::fetch_and_store_parents($channel, $observer_hash, $mid, null, $force);
+		}
+
+		return;
+
+	}
+}

Zotlabs/Daemon/File_importer.php

@@ -0,0 +1,71 @@
+<?php
+
+namespace Zotlabs\Daemon;
+
+use Zotlabs\Web\HTTPSig;
+use Zotlabs\Lib\PConfig;
+
+
+require_once('include/cli_startup.php');
+require_once('include/attach.php');
+require_once('include/import.php');
+
+class File_importer {
+
+	static public function run($argc,$argv) {
+
+		cli_startup();
+
+		$page = $argv[1];
+		$channel_address = $argv[2];
+		$hz_server = urldecode($argv[3]);
+
+		$m = parse_url($hz_server);
+
+		$channel = channelx_by_nick($channel_address);
+		if(! $channel) {
+			logger('channel not found');
+			return;
+		}
+
+		$headers = [
+			'X-API-Token'      => random_string(),
+			'X-API-Request'    => $hz_server . '/api/z/1.0/file/export_page?f=records=1&page=' . $page,
+			'Host'             => $m['host'],
+			'(request-target)' => 'get /api/z/1.0/file/export_page?f=records=1&page=' . $page,
+		];
+
+		$headers = HTTPSig::create_sig($headers,$channel['channel_prvkey'],channel_url($channel),true,'sha512');
+
+		// TODO: implement total count
+		$x = z_fetch_url($hz_server . '/api/z/1.0/file/export_page?f=records=1&page=' . $page, false, $redirects, [ 'headers' => $headers ]);
+		// logger('file fetch: ' . print_r($x,true));
+
+		if(! $x['success']) {
+			logger('no API response',LOGGER_DEBUG);
+			killme();
+		}
+
+		$j = json_decode($x['body'],true);
+
+		if(! is_array($j['results'][0]['attach']) || ! count($j['results'][0]['attach'])) {
+			PConfig::Set($channel['channel_id'], 'import', 'files_completed', 1);
+			return;
+		}
+
+		$r = sync_files($channel, $j['results']);
+
+		PConfig::Set($channel['channel_id'], 'import', 'files_progress', [
+			'files_total' => $j['total'],
+			'files_page' => 1, // export page atm returns just one file
+			'last_page' => $page,
+			'next_cmd' => ['File_importer',sprintf('%d',$page + 1), $channel['channel_address'], urlencode($hz_server)]
+		]);
+
+		$page++;
+
+		Master::Summon([ 'File_importer',sprintf('%d',$page), $channel['channel_address'], urlencode($hz_server) ]);
+
+		return;
+	}
+}

Zotlabs/Daemon/Gprobe.php

@@ -9,27 +9,38 @@ use Zotlabs\Lib\Zotfinger;
 // performs zot_finger on $argv[1], which is a hex_encoded webbie/reddress
 
 class Gprobe {
-	static public function run($argc,$argv) {
+	static public function run($argc, $argv) {
 
-		if($argc != 2)
+		if ($argc != 2)
 			return;
 
 		$url = hex2bin($argv[1]);
+		$is_webbie = false;
+		$r = null;
 
-		if(! strpos($url,'@'))
-			return;
+		if (filter_var($url, FILTER_VALIDATE_EMAIL)) {
+			$is_webbie = true;
 
-		$r = q("select * from hubloc where hubloc_addr = '%s' and hubloc_network = 'zot6' limit 1",
-			dbesc($url)
-		);
+			$r = q("select * from hubloc where hubloc_addr = '%s' and hubloc_network = 'zot6' limit 1",
+				dbesc($url)
+			);
+		}
+		elseif (filter_var($url, FILTER_VALIDATE_URL)) {
+			$r = q("select * from hubloc where hubloc_id_url = '%s' and hubloc_network = 'zot6' limit 1",
+				dbesc($url)
+			);
+		}
 
-		if(! $r) {
-			$href = Webfinger::zot_url(punify($url));
-			if($href) {
-				$zf = Zotfinger::exec($href, null);
+		if (!$r) {
+			if ($is_webbie) {
+				$url = Webfinger::zot_url(punify($url));
 			}
-			if(is_array($zf) && array_path_exists('signature/signer',$zf) && $zf['signature']['signer'] === $href && intval($zf['signature']['header_valid'])) {
-				$xc = Libzot::import_xchan($zf['data']);
+
+			if ($url) {
+				$zf = Zotfinger::exec($url, null);
+				if (is_array($zf) && array_path_exists('signature/signer', $zf) && $zf['signature']['signer'] === $url && intval($zf['signature']['header_valid'])) {
+					Libzot::import_xchan($zf['data']);
+				}
 			}
 		}
 

Zotlabs/Daemon/Importdoc.php

@@ -5,31 +5,33 @@ namespace Zotlabs\Daemon;
 
 class Importdoc {
 
-	static public function run($argc,$argv) {
+	static public function run($argc, $argv) {
 
 		require_once('include/help.php');
 
 		self::update_docs_dir('doc/*');
 
+		return;
+
 	}
 
 	static public function update_docs_dir($s) {
 		$f = basename($s);
 		$d = dirname($s);
-		if($s === 'doc/html')
+		if ($s === 'doc/html')
 			return;
 		$files = glob("$d/$f");
-		if($files) {
-			foreach($files as $fi) {
-				if($fi === 'doc/html') {
+		if ($files) {
+			foreach ($files as $fi) {
+				if ($fi === 'doc/html') {
 					continue;
 				}
-				if(is_dir($fi)) {
+				if (is_dir($fi)) {
 					self::update_docs_dir("$fi/*");
 				}
 				else {
 					// don't update media content
-					if(strpos(z_mime_content_type($fi),'text') === 0) {
+					if (strpos(z_mime_content_type($fi), 'text') === 0) {
 						store_doc_file($fi);
 					}
 				}

Zotlabs/Daemon/Importfile.php

@@ -6,22 +6,21 @@ use Zotlabs\Lib\Libsync;
 
 class Importfile {
 
-	static public function run($argc,$argv){
+	static public function run($argc, $argv) {
 
-		logger('Importfile: ' . print_r($argv,true));
+		logger('Importfile: ' . print_r($argv, true));
 
-		if($argc < 3)
+		if ($argc < 3)
 			return;
 
 		$channel = channelx_by_n($argv[1]);
-		if(! $channel)
+		if (!$channel)
 			return;
 
 		$srcfile = $argv[2];
 		$folder  = (($argc > 3) ? $argv[3] : '');
 		$dstname = (($argc > 4) ? $argv[4] : '');
-
-		$hash = random_string();
+		$hash    = random_string();
 
 		$arr = [
 			'src'               => $srcfile,
@@ -35,15 +34,15 @@ class Importfile {
 			'replace'           => true
 		];
 
-		if($folder)
+		if ($folder)
 			$arr['folder'] = $folder;
 
-		attach_store($channel,$channel['channel_hash'],'import',$arr);
+		attach_store($channel, $channel['channel_hash'], 'import', $arr);
+
+		$sync = attach_export_data($channel, $hash);
+		if ($sync)
+			Libsync::build_sync_packet($channel['channel_id'], ['file' => [$sync]]);
 
-		$sync = attach_export_data($channel,$hash);
-		if($sync)
-			Libsync::build_sync_packet($channel['channel_id'],array('file' => array($sync)));
-	
 		return;
 	}
 }

Zotlabs/Daemon/Master.php

@@ -2,58 +2,69 @@
 
 namespace Zotlabs\Daemon;
 
-if(array_search( __file__ , get_included_files()) === 0) {
+use Zotlabs\Lib\QueueWorker;
+
+if (array_search(__file__, get_included_files()) === 0) {
 	require_once('include/cli_startup.php');
 	array_shift($argv);
 	$argc = count($argv);
 
-	if($argc)
-		Master::Release($argc,$argv);
+	if ($argc)
+		Master::Release($argc, $argv);
+
 	return;
 }
 
 
-
 class Master {
 
 	static public function Summon($arr) {
+
+		QueueWorker::Summon($arr);
+		return;
+/*
 		$hookinfo = [
-			'argv'=>$arr
+			'argv' => $arr
 		];
 
-		call_hooks ('daemon_master_summon',$hookinfo);
+		call_hooks('daemon_master_summon', $hookinfo);
 
-		$arr = $hookinfo['argv'];
+		$arr  = $hookinfo['argv'];
 		$argc = count($arr);
 
-		if ((!is_array($arr) || (count($arr) < 1))) {
-			logger("Summon handled by hook.",LOGGER_DEBUG);
+		if ((!is_array($arr) || ($argc < 1))) {
+			logger("Summon handled by hook.", LOGGER_DEBUG);
 			return;
 		}
 
-		$phpbin = get_config('system','phpbin','php');
-		proc_run($phpbin,'Zotlabs/Daemon/Master.php',$arr);
+		$phpbin = Config::Get('system', 'phpbin', 'php');
+		proc_run($phpbin, 'Zotlabs/Daemon/Master.php', $arr);
+*/
 	}
 
-	static public function Release($argc,$argv) {
+	static public function Release($argc, $argv) {
 		cli_startup();
 
+		QueueWorker::Release($argv);
+		return;
+/*
 		$hookinfo = [
-			'argv'=>$argv
+			'argv' => $argv
 		];
 
-		call_hooks ('daemon_master_release',$hookinfo);
+		call_hooks('daemon_master_release', $hookinfo);
 
 		$argv = $hookinfo['argv'];
 		$argc = count($argv);
 
-		if ((!is_array($argv) || (count($argv) < 1))) {
-			logger("Release handled by hook.",LOGGER_DEBUG);
+		if ((!is_array($argv) || ($argc < 1))) {
+			logger("Release handled by hook.", LOGGER_DEBUG);
 			return;
 		}
 
-		logger('Master: release: ' . json_encode($argv), LOGGER_ALL,LOG_DEBUG);
-                $cls = '\\Zotlabs\\Daemon\\' . $argv[0];
-                $cls::run($argc,$argv);
+		logger('Master: release: ' . json_encode($argv), LOGGER_ALL, LOG_DEBUG);
+		$cls = '\\Zotlabs\\Daemon\\' . $argv[0];
+		$cls::run($argc, $argv);
+*/
 	}
 }

Zotlabs/Daemon/Onedirsync.php

@@ -5,75 +5,55 @@ namespace Zotlabs\Daemon;
 use Zotlabs\Lib\Libzot;
 use Zotlabs\Lib\Libzotdir;
 
-require_once('include/zot.php');
-require_once('include/dir_fns.php');
-
-
 class Onedirsync {
 
-	static public function run($argc,$argv) {
+	static public function run($argc, $argv) {
 
 		logger('onedirsync: start ' . intval($argv[1]));
 
-		if(($argc > 1) && (intval($argv[1])))
+		if (($argc > 1) && (intval($argv[1])))
 			$update_id = intval($argv[1]);
 
-		if(! $update_id) {
-			logger('onedirsync: no update');
+		if (!$update_id) {
+			logger('onedirsync: no update id');
 			return;
 		}
 
-		$r = q("select * from updates where ud_id = %d limit 1",
+		$r = q("select * from updates where ud_id = %d",
 			intval($update_id)
 		);
 
-		if(! $r)
-			return;
-		if(($r[0]['ud_flags'] & UPDATE_FLAGS_UPDATED) || (! $r[0]['ud_addr']))
-			return;
-
-		// Have we probed this channel more recently than the other directory server
-		// (where we received this update from) ?
-		// If we have, we don't need to do anything except mark any older entries updated
-
-		$x = q("select * from updates where ud_addr = '%s' and ud_date > '%s' and ( ud_flags & %d )>0 order by ud_date desc limit 1",
-			dbesc($r[0]['ud_addr']),
-			dbesc($r[0]['ud_date']),
-			intval(UPDATE_FLAGS_UPDATED)
-		);
-		if($x) {
-			$y = q("update updates set ud_flags = ( ud_flags | %d ) where ud_addr = '%s' and ( ud_flags & %d ) = 0 and ud_date != '%s'",
-				intval(UPDATE_FLAGS_UPDATED),
-				dbesc($r[0]['ud_addr']),
-				intval(UPDATE_FLAGS_UPDATED),
-				dbesc($x[0]['ud_date'])
-			);
+		if (!$r) {
+			logger('onedirsync: update id not found');
 			return;
 		}
 
 		// ignore doing an update if this ud_addr refers to a known dead hubloc
 
-		$h = q("select * from hubloc where hubloc_addr = '%s'",
-			dbesc($r[0]['ud_addr'])
+		$h = q("select * from hubloc where hubloc_id_url = '%s' order by hubloc_id desc",
+			dbesc($r[0]['ud_addr']),
 		);
 
 		$h = Libzot::zot_record_preferred($h);
 
-		if(($h) && ($h['hubloc_status'] & HUBLOC_OFFLINE)) {
-			$y = q("update updates set ud_flags = ( ud_flags | %d ) where ud_addr = '%s' and ( ud_flags & %d ) = 0 ",
-				intval(UPDATE_FLAGS_UPDATED),
-				dbesc($r[0]['ud_addr']),
-				intval(UPDATE_FLAGS_UPDATED)
-			);
+		if (($h) && (($h['hubloc_status'] & HUBLOC_OFFLINE) || $h['hubloc_deleted'] || $h['hubloc_error'])) {
 
-			return;
+			// 2023-04-12: Try to update anyway since the info is not always correct
+			// This might change after all directory servers run the new code.
+
+			// q("update updates set ud_flags = 9 where ud_hash = '%s' and ud_flags != 9",
+			//	dbesc($r[0]['ud_hash'])
+			//);
+
+			// return;
 		}
 
 		// we might have to pull this out some day, but for now update_directory_entry()
 		// runs zot_finger() and is kind of zot specific
 
-		if($h && ! in_array($h['hubloc_network'], ['zot6', 'zot']))
+		if ($h && $h['hubloc_network'] !== 'zot6') {
 			return;
+		}
 
 		Libzotdir::update_directory_entry($r[0]);
 

Zotlabs/Daemon/Onepoll.php

@@ -2,92 +2,94 @@
 
 namespace Zotlabs\Daemon;
 
+use Zotlabs\Lib\Activity;
+use Zotlabs\Lib\ActivityStreams;
+use Zotlabs\Lib\ASCollection;
+use Zotlabs\Lib\Config;
 use Zotlabs\Lib\Libzot;
 
-require_once('include/zot.php');
 require_once('include/socgraph.php');
-
+require_once('include/feedutils.php');
 
 class Onepoll {
 
-	static public function run($argc,$argv) {
+	static public function run($argc, $argv) {
 
 		logger('onepoll: start');
-	
-		if(($argc > 1) && (intval($argv[1])))
+
+		if (($argc > 1) && (intval($argv[1])))
 			$contact_id = intval($argv[1]);
 
-		if(! $contact_id) {
+		if (!$contact_id) {
 			logger('onepoll: no contact');
 			return;
 		}
 
-		$d = datetime_convert();
+		$sql_extra = '';
+		$allow_feeds = Config::Get('system', 'feed_contacts');
+		if(!$allow_feeds) {
+			$sql_extra = ' and abook_feed = 0 ';
+		}
 
-		$contacts = q("SELECT abook.*, xchan.*, account.*
-			FROM abook LEFT JOIN account on abook_account = account_id left join xchan on xchan_hash = abook_xchan 
-			where abook_id = %d
-			and abook_pending = 0 and abook_archived = 0 and abook_blocked = 0 and abook_ignored = 0
-			AND (( account_flags = %d ) OR ( account_flags = %d )) limit 1",
-			intval($contact_id),
-			intval(ACCOUNT_OK),
-			intval(ACCOUNT_UNVERIFIED)
-		);	
+		$contacts = q("SELECT abook.*, xchan.* FROM abook
+			LEFT JOIN xchan ON xchan_hash = abook_xchan
+			WHERE abook_id = %d",
+			intval($contact_id)
+		);
 
-		if(! $contacts) {
+		if (!$contacts) {
 			logger('onepoll: abook_id not found: ' . $contact_id);
 			return;
 		}
 
-		$contact = $contacts[0];
-
-		$t = $contact['abook_updated'];
+		$contact      = $contacts[0];
 
 		$importer_uid = $contact['abook_channel'];
-		
-		$r = q("SELECT * from channel left join xchan on channel_hash = xchan_hash where channel_id = %d limit 1",
-			intval($importer_uid)
-		);
 
-		if(! $r)
+		$importer = channelx_by_n($importer_uid);
+
+		if (!$importer)
 			return;
 
-		$importer = $r[0];
+		logger("onepoll: poll: ($contact_id) IMPORTER: {$importer['xchan_name']}, CONTACT: {$contact['xchan_name']}");
 
-		logger("onepoll: poll: ({$contact['id']}) IMPORTER: {$importer['xchan_name']}, CONTACT: {$contact['xchan_name']}");
-
-		$last_update = ((($contact['abook_updated'] === $contact['abook_created']) || ($contact['abook_updated'] <= NULL_DATE))	
-			? datetime_convert('UTC','UTC','now - 7 days')
-			: datetime_convert('UTC','UTC',$contact['abook_updated'] . ' - 2 days')
+		$last_update = ((($contact['abook_updated'] === $contact['abook_created']) || ($contact['abook_updated'] <= NULL_DATE))
+			? datetime_convert('UTC', 'UTC', 'now - 7 days')
+			: datetime_convert('UTC', 'UTC', $contact['abook_updated'] . ' - 2 days')
 		);
 
-		if($contact['xchan_network'] === 'rss') {
+		if ($contact['xchan_network'] === 'rss') {
 			logger('onepoll: processing feed ' . $contact['xchan_name'], LOGGER_DEBUG);
-			$alive = handle_feed($importer['channel_id'],$contact_id,$contact['xchan_hash']);
-			if ($alive) {
-				q("update abook set abook_connected = '%s' where abook_id = %d",
+			$alive = handle_feed($importer['channel_id'], $contact_id, $contact['xchan_hash']);
+
+			if (!$alive) {
+				q("update abook set abook_updated = '%s' where abook_id = %d",
 					dbesc(datetime_convert()),
 					intval($contact['abook_id'])
 				);
+				return;
 			}
+
+			q("update abook set abook_updated = '%s', abook_connected = '%s' where abook_id = %d",
+				dbesc(datetime_convert()),
+				dbesc(datetime_convert()),
+				intval($contact['abook_id'])
+			);
+
 			return;
 		}
-	
-		if(! in_array($contact['xchan_network'],['zot','zot6']))
+
+		if ($contact['xchan_network'] !== 'zot6')
 			return;
 
 		// update permissions
 
-		if($contact['xchan_network'] === 'zot6')
-			$x = Libzot::refresh($contact,$importer);
-
-		if($contact['xchan_network'] === 'zot')
-			$x = zot_refresh($contact,$importer);
+		$x = Libzot::refresh($contact, $importer);
 
 		$responded = false;
 		$updated   = datetime_convert();
 		$connected = datetime_convert();
-		if(! $x) {
+		if (!$x) {
 			// mark for death by not updating abook_connected, this is caught in include/poller.php
 			q("update abook set abook_updated = '%s' where abook_id = %d",
 				dbesc($updated),
@@ -103,83 +105,88 @@ class Onepoll {
 			$responded = true;
 		}
 
-		if(! $responded)
+		if (!$responded)
 			return;
 
-		if($contact['xchan_connurl']) {
-			$fetch_feed = true;
-			$x = null;
+		$fetch_feed = true;
 
-			// They haven't given us permission to see their stream
+		// They haven't given us permission to see their stream
+		$can_view_stream = intval(get_abconfig($importer_uid, $contact['abook_xchan'], 'their_perms', 'view_stream'));
 
-			$can_view_stream = intval(get_abconfig($importer_uid,$contact['abook_xchan'],'their_perms','view_stream'));
+		if (!$can_view_stream) {
+			$fetch_feed = false;
+		}
 
-			if(! $can_view_stream)
-				$fetch_feed = false;
+		// we haven't given them permission to send us their stream
+		$can_send_stream = intval(get_abconfig($importer_uid, $contact['abook_xchan'], 'my_perms', 'send_stream'));
 
-			// we haven't given them permission to send us their stream
+		if (!$can_send_stream) {
+			$fetch_feed = false;
+		}
 
-			$can_send_stream = intval(get_abconfig($importer_uid,$contact['abook_xchan'],'my_perms','send_stream'));
-			
-			if(! $can_send_stream)
-				$fetch_feed = false;
+		if ($fetch_feed) {
 
-			if($fetch_feed) {
+			$max = intval(Config::Get('system', 'max_imported_posts', 30));
 
-				if(strpos($contact['xchan_connurl'],z_root()) === 0) {
-					// local channel - save a network fetch
-					$c = channelx_by_hash($contact['xchan_hash']);
-					if($c) {
-						$x = [ 
-							'success' => true, 
-							'body' => json_encode( [ 
-								'success' => true,
-								'messages' => zot_feed($c['channel_id'], $importer['xchan_hash'], [ 'mindate' => $last_update ])
-							])
-						];
-					}
+			if (intval($max)) {
+				$cl = Activity::get_actor_collections($contact['abook_xchan']);
+				if(empty($cl)) {
+					$cl = get_xconfig($contact['abook_xchan'], 'activitypub', 'collections');
+				}
+
+				if (is_array($cl) && array_key_exists('outbox', $cl)) {
+					$url = $cl['outbox'];
 				}
 				else {
-					// remote fetch	
-
-					$feedurl = str_replace('/poco/','/zotfeed/',$contact['xchan_connurl']);		
-					$feedurl .= '?f=&mindate=' . urlencode($last_update) . '&zid=' . $importer['channel_address'] . '@' . \App::get_hostname();
-					$recurse = 0;
-					$x = z_fetch_url($feedurl, false, $recurse, [ 'session' => true ]);
+					$url = str_replace('/poco/', '/outbox/', $contact['xchan_connurl']);
 				}
 
-				logger('feed_update: ' . print_r($x,true), LOGGER_DATA);
-			}
+				if ($url) {
+					logger('fetching outbox');
+					$url = $url . '?date_begin=' .  urlencode($last_update);
 
-			if(($x) && ($x['success'])) {
-				$total = 0;
-				logger('onepoll: feed update ' . $contact['xchan_name'] . ' ' . $feedurl);
-
-				$j = json_decode($x['body'],true);
-				if($j['success'] && $j['messages']) {
-					foreach($j['messages'] as $message) {
-						$results = process_delivery(array('hash' => $contact['xchan_hash']), get_item_elements($message),
-							array(array('hash' => $importer['xchan_hash'])), false);
-						logger('onepoll: feed_update: process_delivery: ' . print_r($results,true), LOGGER_DATA);
-						$total ++;
+					if($contact['xchan_network'] === 'zot6') {
+						$url = $url . '&top=1';
+					}
+
+					$obj      = new ASCollection($url, $importer, 0, $max);
+					$messages = $obj->get();
+
+					if ($messages) {
+						foreach ($messages as $message) {
+							if (is_string($message)) {
+								$message = Activity::fetch($message, $importer);
+							}
+
+							if ($contact['xchan_network'] === 'zot6') {
+								// make sure we only fetch top level items
+								if ($message['type'] === 'Create' && !isset($message['object']['inReplyTo'])) {
+									Libzot::fetch_conversation($importer, $message['object']['id']);
+								}
+								continue;
+							}
+
+							$AS = new ActivityStreams($message);
+							if ($AS->is_valid() && is_array($AS->obj)) {
+								$item = Activity::decode_note($AS);
+								$item['item_fetched'] = true;
+								Activity::store($importer, $contact['abook_xchan'], $AS, $item);
+							}
+						}
 					}
-					logger("onepoll: $total messages processed");
 				}
 			}
 		}
-			
 
 		// update the poco details for this connection
+		$r = q("SELECT xlink_id from xlink where xlink_xchan = '%s' and xlink_updated > %s - INTERVAL %s and xlink_static = 0 limit 1",
+			intval($contact['xchan_hash']),
+			db_utcnow(),
+			db_quoteinterval('1 DAY')
+		);
 
-		if($contact['xchan_connurl']) {	
-			$r = q("SELECT xlink_id from xlink 
-				where xlink_xchan = '%s' and xlink_updated > %s - INTERVAL %s and xlink_static = 0 limit 1",
-				intval($contact['xchan_hash']),
-				db_utcnow(), db_quoteinterval('1 DAY')
-			);
-			if(! $r) {
-				poco_load($contact['xchan_hash'],$contact['xchan_connurl']);
-			}
+		if (!$r) {
+			poco_load($contact['xchan_hash'], $contact['xchan_connurl']);
 		}
 
 		return;

Zotlabs/Daemon/Poller.php

@@ -2,55 +2,40 @@
 
 namespace Zotlabs\Daemon;
 
+use Zotlabs\Lib\Config;
+
 class Poller {
 
-	static public function run($argc,$argv) {
+	static public function run($argc, $argv) {
 
-		$maxsysload = intval(get_config('system','maxloadavg'));
-		if($maxsysload < 1)
+		$maxsysload = intval(Config::Get('system', 'maxloadavg'));
+		if ($maxsysload < 1)
 			$maxsysload = 50;
-		if(function_exists('sys_getloadavg')) {
+		if (function_exists('sys_getloadavg')) {
 			$load = sys_getloadavg();
-			if(intval($load[0]) > $maxsysload) {
+			if (intval($load[0]) > $maxsysload) {
 				logger('system: load ' . $load . ' too high. Poller deferred to next scheduled run.');
 				return;
 			}
 		}
 
-		$interval = intval(get_config('system','poll_interval'));
-		if(! $interval) 
-			$interval = ((get_config('system','delivery_interval') === false) ? 3 : intval(get_config('system','delivery_interval')));
-
-		// Check for a lockfile.  If it exists, but is over an hour old, it's stale.  Ignore it.
-		$lockfile = 'store/[data]/poller';
-		if((file_exists($lockfile)) && (filemtime($lockfile) > (time() - 3600)) 
-			&& (! get_config('system','override_poll_lockfile'))) {
-			logger("poller: Already running");
-			return;
-		}
-	
-		// Create a lockfile.  Needs two vars, but $x doesn't need to contain anything.
-		file_put_contents($lockfile, $x);
+		$interval = Config::Get('queueworker', 'queue_interval', 500000);
 
 		logger('poller: start');
-	
+
 		$manual_id  = 0;
-		$generation = 0;
+		$force   = false;
 
-		$force      = false;
-		$restart    = false;
-
-		if(($argc > 1) && ($argv[1] == 'force'))
+		if (($argc > 1) && ($argv[1] == 'force'))
 			$force = true;
 
-		if(($argc > 1) && ($argv[1] == 'restart')) {
-			$restart = true;
+		if (($argc > 1) && ($argv[1] == 'restart')) {
 			$generation = intval($argv[2]);
-			if(! $generation)
+			if (!$generation)
 				return;
 		}
 
-		if(($argc > 1) && intval($argv[1])) {
+		if (($argc > 1) && intval($argv[1])) {
 			$manual_id = intval($argv[1]);
 			$force     = true;
 		}
@@ -59,148 +44,160 @@ class Poller {
 
 		reload_plugins();
 
-		$d = datetime_convert();
-
 		// Only poll from those with suitable relationships
-
-		$abandon_sql = (($abandon_days) 
-			? sprintf(" AND account_lastlog > %s - INTERVAL %s ", db_utcnow(), db_quoteinterval(intval($abandon_days).' DAY')) 
-			: '' 
+		$abandon_days = intval(Config::Get('system', 'account_abandon_days', 0));
+		$abandon_sql = (($abandon_days)
+			? sprintf(" AND account_lastlog > %s - INTERVAL %s ", db_utcnow(), db_quoteinterval(intval($abandon_days) . ' DAY'))
+			: ''
 		);
 
+		$allow_feeds = Config::Get('system', 'feed_contacts');
+		if(!$allow_feeds) {
+			$sql_extra .= ' and abook_feed = 0 ';
+		}
+
 		$randfunc = db_getfunc('RAND');
-	
+
 		$contacts = q("SELECT abook.abook_updated, abook.abook_connected, abook.abook_feed,
 			abook.abook_channel, abook.abook_id, abook.abook_archived, abook.abook_pending,
 			abook.abook_ignored, abook.abook_blocked,
 			xchan.xchan_network,
-			account.account_lastlog, account.account_flags 
-			FROM abook LEFT JOIN xchan on abook_xchan = xchan_hash 
+			account.account_lastlog, account.account_flags
+			FROM abook LEFT JOIN xchan on abook_xchan = xchan_hash
 			LEFT JOIN account on abook_account = account_id
-			where abook_self = 0
-			$sql_extra 
+			where abook_self = 0 and abook_pending = 0 and abook_archived = 0 and abook_blocked = 0 and abook_ignored = 0
+			$sql_extra
 			AND (( account_flags = %d ) OR ( account_flags = %d )) $abandon_sql ORDER BY $randfunc",
 			intval(ACCOUNT_OK),
 			intval(ACCOUNT_UNVERIFIED)     // FIXME
 		);
 
-		if($contacts) {
-			foreach($contacts as $contact) {
+		if ($contacts) {
+			foreach ($contacts as $contact) {
 
-				$update  = false;
+				$update = false;
 
 				$t = $contact['abook_updated'];
 				$c = $contact['abook_connected'];
 
-				if(intval($contact['abook_feed'])) {
-					$min = service_class_fetch($contact['abook_channel'],'minimum_feedcheck_minutes');
-					if(! $min)
-						$min = intval(get_config('system','minimum_feedcheck_minutes'));
-					if(! $min)
+				if (intval($contact['abook_feed'])) {
+					$min = service_class_fetch($contact['abook_channel'], 'minimum_feedcheck_minutes');
+					if (!$min)
+						$min = intval(Config::Get('system', 'minimum_feedcheck_minutes'));
+					if (!$min)
 						$min = 60;
-					$x = datetime_convert('UTC','UTC',"now - $min minutes");
-					if($c < $x) {
-						Master::Summon(array('Onepoll',$contact['abook_id']));
-						if($interval)
-							@time_sleep_until(microtime(true) + (float) $interval);
+
+					if ($t !== $c) {
+						// if the last fetch failed only attempt fetch once a day
+						$min = 60 * 24;
 					}
+
+					$x = datetime_convert('UTC', 'UTC', "now - $min minutes");
+
+					if ($t < $x) {
+						Master::Summon(['Onepoll', $contact['abook_id']]);
+
+						if ($interval) {
+							usleep($interval);
+						}
+					}
+
 					continue;
+
 				}
 
-
-				if(! in_array($contact['xchan_network'],['zot','zot6']))
+				if ($contact['xchan_network'] !== 'zot6')
 					continue;
 
-				if($c == $t) {
-					if(datetime_convert('UTC','UTC', 'now') > datetime_convert('UTC','UTC', $t . " + 1 day"))
+				if ($c == $t) {
+					if (datetime_convert('UTC', 'UTC', 'now') > datetime_convert('UTC', 'UTC', $t . " + 1 day"))
 						$update = true;
 				}
 				else {
-	
+
 					// if we've never connected with them, start the mark for death countdown from now
-	
-					if($c <= NULL_DATE) {
-						$r = q("update abook set abook_connected = '%s'  where abook_id = %d",
+
+					if ($c <= NULL_DATE) {
+						q("update abook set abook_connected = '%s'  where abook_id = %d",
 							dbesc(datetime_convert()),
 							intval($contact['abook_id'])
 						);
-						$c = datetime_convert();
+						$c      = datetime_convert();
 						$update = true;
 					}
 
 					// He's dead, Jim
 
-					if(strcmp(datetime_convert('UTC','UTC', 'now'),datetime_convert('UTC','UTC', $c . " + 30 day")) > 0) {	
-						$r = q("update abook set abook_archived = 1 where abook_id = %d",
+					if (strcmp(datetime_convert('UTC', 'UTC', 'now'), datetime_convert('UTC', 'UTC', $c . " + 30 day")) > 0) {
+						q("update abook set abook_archived = 1 where abook_id = %d",
 							intval($contact['abook_id'])
 						);
-						$update = false;
 						continue;
 					}
 
-					if(intval($contact['abook_archived'])) {
-						$update = false;
+					if (intval($contact['abook_archived'])) {
 						continue;
 					}
 
 					// might be dead, so maybe don't poll quite so often
-	
+
 					// recently deceased, so keep up the regular schedule for 3 days
- 
-					if((strcmp(datetime_convert('UTC','UTC', 'now'),datetime_convert('UTC','UTC', $c . " + 3 day")) > 0)
-					 && (strcmp(datetime_convert('UTC','UTC', 'now'),datetime_convert('UTC','UTC', $t . " + 1 day")) > 0))
+
+					if ((strcmp(datetime_convert('UTC', 'UTC', 'now'), datetime_convert('UTC', 'UTC', $c . " + 3 day")) > 0)
+						&& (strcmp(datetime_convert('UTC', 'UTC', 'now'), datetime_convert('UTC', 'UTC', $t . " + 1 day")) > 0))
 						$update = true;
 
 					// After that back off and put them on a morphine drip
 
-					if(strcmp(datetime_convert('UTC','UTC', 'now'),datetime_convert('UTC','UTC', $t . " + 2 day")) > 0) {	
+					if (strcmp(datetime_convert('UTC', 'UTC', 'now'), datetime_convert('UTC', 'UTC', $t . " + 2 day")) > 0) {
 						$update = true;
 					}
 
 				}
 
-				if(intval($contact['abook_pending']) || intval($contact['abook_archived']) || intval($contact['abook_ignored']) || intval($contact['abook_blocked']))
+				if (intval($contact['abook_pending']) || intval($contact['abook_archived']) || intval($contact['abook_ignored']) || intval($contact['abook_blocked']))
 					continue;
 
-				if((! $update) && (! $force))
-						continue;
+				if ((!$update) && (!$force))
+					continue;
 
-				Master::Summon(array('Onepoll',$contact['abook_id']));
-				if($interval)
-					@time_sleep_until(microtime(true) + (float) $interval);
+				Master::Summon(['Onepoll', $contact['abook_id']]);
 
+				if ($interval) {
+					usleep($interval);
+				}
 			}
 		}
 
-		$dirmode = intval(get_config('system', 'directory_mode'));
+		$dirmode = intval(Config::Get('system', 'directory_mode'));
 
-		if($dirmode == DIRECTORY_MODE_SECONDARY || $dirmode == DIRECTORY_MODE_PRIMARY) {
-			$r = q("SELECT u.ud_addr, u.ud_id, u.ud_last FROM updates AS u INNER JOIN (SELECT ud_addr, max(ud_id) AS ud_id FROM updates WHERE ( ud_flags & %d ) = 0 AND ud_addr != '' AND ( ud_last <= '%s' OR ud_last > %s - INTERVAL %s ) GROUP BY ud_addr) AS s ON s.ud_id = u.ud_id ",
-				intval(UPDATE_FLAGS_UPDATED),
+		if ($dirmode == DIRECTORY_MODE_SECONDARY || $dirmode == DIRECTORY_MODE_PRIMARY) {
+			$r = q("SELECT * FROM updates WHERE ud_update = 1 AND (ud_last = '%s' OR ud_last > %s - INTERVAL %s)",
 				dbesc(NULL_DATE),
-				db_utcnow(), db_quoteinterval('7 DAY')
+				db_utcnow(),
+				db_quoteinterval('7 DAY')
 			);
-			if($r) {
-				foreach($r as $rr) {
+
+			if ($r) {
+				foreach ($r as $rr) {
 
 					// If they didn't respond when we attempted before, back off to once a day
 					// After 7 days we won't bother anymore
 
-					if($rr['ud_last'] > NULL_DATE)
-						if($rr['ud_last'] > datetime_convert('UTC','UTC', 'now - 1 day'))
+					if ($rr['ud_last'] > NULL_DATE)
+						if ($rr['ud_last'] > datetime_convert('UTC', 'UTC', 'now - 1 day'))
 							continue;
-					Master::Summon(array('Onedirsync',$rr['ud_id']));
-					if($interval)
-						@time_sleep_until(microtime(true) + (float) $interval);
+
+					Master::Summon(['Onedirsync', $rr['ud_id']]);
+
+					if ($interval) {
+						usleep($interval);
+					}
 				}
 			}
-		}	
+		}
 
-		set_config('system','lastpoll',datetime_convert());
-
-		//All done - clear the lockfile	
-
-		@unlink($lockfile);
+		Config::Set('system', 'lastpoll', datetime_convert());
 
 		return;
 	}

Zotlabs/Daemon/Queue.php

@@ -2,84 +2,64 @@
 
 namespace Zotlabs\Daemon;
 
-require_once('include/queue_fn.php');
-require_once('include/zot.php');
+use Zotlabs\Lib\Queue as LibQueue;
 
 class Queue {
 
-	static public function run($argc,$argv) {
-
-		require_once('include/items.php');
-		require_once('include/bbcode.php');
-
-		if($argc > 1)
-			$queue_id = $argv[1];
-		else
-			$queue_id = EMPTY_STR;
+	static public function run($argc, $argv) {
+		$queue_id = ($argc > 1) ? $argv[1] : '';
 
 		logger('queue: start');
 
 		// delete all queue items more than 3 days old
 		// but first mark these sites dead if we haven't heard from them in a month
 
-		$r = q("select outq_posturl from outq where outq_created < %s - INTERVAL %s",
-			db_utcnow(), db_quoteinterval('3 DAY')
+		$oldqItems = q("select outq_posturl, outq_hash from outq where outq_created < %s - INTERVAL %s",
+			db_utcnow(),
+			db_quoteinterval('3 DAY')
 		);
-		if($r) {
-			foreach($r as $rr) {
-				$site_url = '';
-				$h = parse_url($rr['outq_posturl']);
-				$desturl = $h['scheme'] . '://' . $h['host'] . (($h['port']) ? ':' . $h['port'] : '');
+
+		if ($oldqItems) {
+			foreach ($oldqItems as $qItem) {
+				$h = parse_url($qItem['outq_posturl']);
+				$site_url = $h['scheme'] . '://' . $h['host'] . ((!empty($h['port'])) ? ':' . $h['port'] : '');
 				q("update site set site_dead = 1 where site_dead = 0 and site_url = '%s' and site_update < %s - INTERVAL %s",
-					dbesc($desturl),
-					db_utcnow(), db_quoteinterval('1 MONTH')
+					dbesc($site_url),
+					db_utcnow(),
+					db_quoteinterval('1 MONTH')
 				);
 			}
+
+			$old_hashes = ids_to_querystr($oldqItems, 'outq_hash', true);
+
+			logger('Removing ' . count($oldqItems) . ' old queue entries');
+			dbq("DELETE FROM outq WHERE outq_hash IN ($old_hashes)");
+
 		}
 
-		$r = q("DELETE FROM outq WHERE outq_created < %s - INTERVAL %s",
-			db_utcnow(), db_quoteinterval('3 DAY')
-		);
+		$deliveries = [];
 
-		if($queue_id) {
-			$r = q("SELECT * FROM outq WHERE outq_hash = '%s' LIMIT 1",
+		if ($queue_id) {
+			$qItems = q("SELECT * FROM outq WHERE outq_hash = '%s' LIMIT 1",
 				dbesc($queue_id)
 			);
+			logger('queue deliver: ' . $qItems[0]['outq_hash'] . ' to ' . $qItems[0]['outq_posturl'], LOGGER_DEBUG);
+			LibQueue::deliver($qItems[0]);
 		}
 		else {
-
-			// For the first 12 hours we'll try to deliver every 15 minutes
-			// After that, we'll only attempt delivery once per hour. 
-			// This currently only handles the default queue drivers ('zot' or '') which we will group by posturl 
-			// so that we don't start off a thousand deliveries for a couple of dead hubs.
-			// The zot driver will deliver everything destined for a single hub once contact is made (*if* contact is made).
-			// Other drivers will have to do something different here and may need their own query.
-	
-			// Note: this requires some tweaking as new posts to long dead hubs once a day will keep them in the 
-			// "every 15 minutes" category. We probably need to prioritise them when inserted into the queue
-			// or just prior to this query based on recent and long-term delivery history. If we have good reason to believe
-			// the site is permanently down, there's no reason to attempt delivery at all, or at most not more than once 
-			// or twice a day. 
-
-			$sqlrandfunc = db_getfunc('rand');
-
-			$r = q("SELECT *,$sqlrandfunc as rn FROM outq WHERE outq_delivered = 0 and outq_scheduled < %s order by rn limit 1",
+			$qItems = q("SELECT outq_hash FROM outq WHERE outq_scheduled < %s ",
 				db_utcnow()
 			);
-			while ($r) {
-				foreach($r as $rv) {
-					queue_deliver($rv);
-				}
-				$r = q("SELECT *,$sqlrandfunc as rn FROM outq WHERE outq_delivered = 0 and outq_scheduled < %s order by rn limit 1",
-					db_utcnow()
-				);
-			}
-		}
-		if(! $r)
-			return;
 
-		foreach($r as $rv) {
-			queue_deliver($rv);
-		}
+			if ($qItems) {
+				foreach ($qItems as $qItem) {
+					$deliveries[] = $qItem['outq_hash'];
+				}
+
+				shuffle($deliveries);
+				do_delivery($deliveries, true);
+			}
+		 }
 	}
+
 }

Zotlabs/Daemon/Ratenotif.php

@@ -1,126 +0,0 @@
-<?php
-
-namespace Zotlabs\Daemon;
-
-require_once('include/zot.php');
-require_once('include/queue_fn.php');
-
-
-class Ratenotif {
-
-	static public function run($argc,$argv) {
-
-
-		// Deprecated
-		return;
-
-
-		require_once("datetime.php");
-		require_once('include/items.php');
-
-		if($argc < 3)
-			return;
-
-
-		logger('ratenotif: invoked: ' . print_r($argv,true), LOGGER_DEBUG);
-
-		$cmd = $argv[1];
-
-		$item_id = $argv[2];
-
-
-		if($cmd === 'rating') {
-			$r = q("select * from xlink where xlink_id = %d and xlink_static = 1 limit 1",
-				intval($item_id)
-			);
-			if(! $r) {
-				logger('rating not found');
-				return;
-			}
-
-			$encoded_item = array(
-				'type' => 'rating', 
-				'encoding' => 'zot',
-				'target' => $r[0]['xlink_link'],
-				'rating' => intval($r[0]['xlink_rating']),
-				'rating_text' => $r[0]['xlink_rating_text'],
-				'signature' => $r[0]['xlink_sig'],
-				'edited' => $r[0]['xlink_updated']
-			);
-		}
-
-		$channel = channelx_by_hash($r[0]['xlink_xchan']);
-		if(! $channel) {
-			logger('no channel');
-			return;
-		}
-
-
-		$primary = get_directory_primary();
-	
-		if(! $primary)
-			return;
-
-
-		$interval = ((get_config('system','delivery_interval') !== false) 
-			? intval(get_config('system','delivery_interval')) : 2 );
-
-		$deliveries_per_process = intval(get_config('system','delivery_batch_count'));
-
-		if($deliveries_per_process <= 0)
-			$deliveries_per_process = 1;
-
-		$deliver = array();
-
-		$x = z_fetch_url($primary . '/regdir');
-		if($x['success']) {
-			$j = json_decode($x['body'],true);
-			if($j && $j['success'] && is_array($j['directories'])) {
-
-				foreach($j['directories'] as $h) {
-					if($h == z_root())
-						continue;
-
-					$hash = random_string();
-					$n = zot_build_packet($channel,'notify',null,null,'',$hash);
-
-					queue_insert(array(
-						'hash'       => $hash,
-						'account_id' => $channel['channel_account_id'],
-						'channel_id' => $channel['channel_id'],
-						'posturl'    => $h . '/post',
-						'notify'     => $n,
-						'msg'        => json_encode($encoded_item)
-					));
-
-
-					$x = q("select count(outq_hash) as total from outq where outq_delivered = 0");
-					if(intval($x[0]['total']) > intval(get_config('system','force_queue_threshold',300))) {
-						logger('immediate delivery deferred.', LOGGER_DEBUG, LOG_INFO);
-						update_queue_item($hash);
-						continue;
-					}
-
-					$deliver[] = $hash;
-	
-					if(count($deliver) >= $deliveries_per_process) {
-						Master::Summon(array('Deliver',$deliver));
-						$deliver = array();
-						if($interval)
-							@time_sleep_until(microtime(true) + (float) $interval);
-					}
-				}
-	
-				// catch any stragglers
-	
-				if(count($deliver)) {
-					Master::Summon(array('Deliver',$deliver));
-				}
-			}
-		}
-		
-		logger('ratenotif: complete.');
-		return;
-
-	}
-}

Zotlabs/Daemon/Thumbnail.php

@@ -2,33 +2,34 @@
 
 namespace Zotlabs\Daemon;
 
+use Zotlabs\Lib\Config;
 
 class Thumbnail {
 
-	static public function run($argc,$argv) {
+	static public function run($argc, $argv) {
 
-		if(! $argc == 2)
+		if (!$argc == 2)
 			return;
 
 		$c = q("select * from attach where hash = '%s' ",
 			dbesc($argv[1])
 		);
 
-		if(! $c)
+		if (!$c)
 			return;
 
 		$attach = $c[0];
 
-		$preview_style   = intval(get_config('system','thumbnail_security',0));
-		$preview_width   = intval(get_config('system','thumbnail_width',300));
-		$preview_height  = intval(get_config('system','thumbnail_height',300));
+		$preview_style  = intval(Config::Get('system', 'thumbnail_security', 0));
+		$preview_width  = intval(Config::Get('system', 'thumbnail_width', 300));
+		$preview_height = intval(Config::Get('system', 'thumbnail_height', 300));
 
 		$p = [
 			'attach'         => $attach,
 			'preview_style'  => $preview_style,
 			'preview_width'  => $preview_width,
 			'preview_height' => $preview_height,
-			'thumbnail' => null
+			'thumbnail'      => null
 		];
 
 		/**
@@ -40,39 +41,40 @@ class Thumbnail {
 		 *  * \e string \b thumbnail
 		 */
 
-		call_hooks('thumbnail',$p);
-		if($p['thumbnail']) {
+		call_hooks('thumbnail', $p);
+		if ($p['thumbnail']) {
 			return;
 		}
 
-
 		$default_controller = null;
-		
+
 		$files = glob('Zotlabs/Thumbs/*.php');
-		if($files) {
-			foreach($files as $f) {
-				$clsname = '\\Zotlabs\\Thumbs\\' . ucfirst(basename($f,'.php'));
-				if(class_exists($clsname)) {
+		if ($files) {
+			foreach ($files as $f) {
+				$clsname = '\\Zotlabs\\Thumbs\\' . ucfirst(basename($f, '.php'));
+				if (class_exists($clsname)) {
 					$x = new $clsname();
-					if(method_exists($x,'Match')) {
+					if (method_exists($x, 'Match')) {
 						$matched = $x->Match($attach['filetype']);
-						if($matched) {
-							$x->Thumb($attach,$preview_style,$preview_width,$preview_height);
+						if ($matched) {
+							$x->Thumb($attach, $preview_style, $preview_width, $preview_height);
 						}
 					}
-					if(method_exists($x,'MatchDefault')) {
-						$default_matched = $x->MatchDefault(substr($attach['filetype'],0,strpos($attach['filetype'],'/')));
-						if($default_matched) {
+					if (method_exists($x, 'MatchDefault')) {
+						$default_matched = $x->MatchDefault(substr($attach['filetype'], 0, strpos($attach['filetype'], '/')));
+						if ($default_matched) {
 							$default_controller = $x;
 						}
 					}
 				}
 			}
 		}
-		if(($default_controller) 
-			&& ((! file_exists(dbunescbin($attach['content']) . '.thumb')) 
+		if (($default_controller)
+			&& ((!file_exists(dbunescbin($attach['content']) . '.thumb'))
 				|| (filectime(dbunescbin($attach['content']) . 'thumb') < (time() - 60)))) {
-			$default_controller->Thumb($attach,$preview_style,$preview_width,$preview_height);
+			$default_controller->Thumb($attach, $preview_style, $preview_width, $preview_height);
 		}
+
+		return;
 	}
 }

Zotlabs/Daemon/Xchan_photo.php

@@ -0,0 +1,37 @@
+<?php
+
+/** @file */
+
+namespace Zotlabs\Daemon;
+
+class Xchan_photo {
+
+	static public function run($argc, $argv) {
+
+		if ($argc < 3) {
+			return;
+		}
+
+		$url   = hex2bin($argv[1]);
+		$xchan = hex2bin($argv[2]);
+		$force = $argv[3];
+
+		$photos = import_xchan_photo($url, $xchan, false, $force);
+		if ($photos) {
+			$result = q("update xchan set xchan_photo_date = '%s', xchan_photo_l = '%s', xchan_photo_m = '%s', xchan_photo_s = '%s', xchan_photo_mimetype = '%s' where xchan_hash = '%s'",
+				dbescdate(datetime_convert()),
+				dbesc($photos[0]),
+				dbesc($photos[1]),
+				dbesc($photos[2]),
+				dbesc($photos[3]),
+				dbesc($xchan)
+			);
+
+			if (! $result) {
+				logger("xchan photo update failed for $url");
+			}
+		}
+
+		return;
+	}
+}

Zotlabs/Daemon/Zotconvo.php

@@ -0,0 +1,37 @@
+<?php
+
+namespace Zotlabs\Daemon;
+
+use Zotlabs\Lib\Libzot;
+
+class Zotconvo {
+
+	static public function run($argc, $argv) {
+
+		logger('Zotconvo invoked: ' . print_r($argv, true));
+
+		if ($argc < 3) {
+			return;
+		}
+
+		$channels = explode(',', $argv[1]);
+		if (!$channels) {
+			return;
+		}
+
+		$mid = $argv[2];
+		if (!$mid) {
+			return;
+		}
+
+		$force = $argv[3] ?? false;
+
+		foreach ($channels as $channel_id) {
+			$channel = channelx_by_n($channel_id);
+			Libzot::fetch_conversation($channel, $mid, $force);
+		}
+
+		return;
+
+	}
+}

Zotlabs/Extend/Route.php

@@ -2,6 +2,7 @@
 
 namespace Zotlabs\Extend;
 
+use Zotlabs\Lib\Config;
 
 class Route {
 
@@ -38,11 +39,11 @@ class Route {
 	}
 
 	static function get() {
-		return get_config('system','routes',[]);
+		return Config::Get('system','routes',[]);
 	}
 
 	static function set($r) {
-		return set_config('system','routes',$r);
+		return Config::Set('system','routes',$r);
 	}
 }
 

Zotlabs/Extend/Widget.php

@@ -2,6 +2,7 @@
 
 namespace Zotlabs\Extend;
 
+use Zotlabs\Lib\Config;
 
 class Widget {
 
@@ -38,10 +39,10 @@ class Widget {
 	}
 
 	static function get() {
-		return get_config('system','widgets',[]);
+		return Config::Get('system','widgets',[]);
 	}
 
 	static function set($r) {
-		return set_config('system','widgets',$r);
+		return Config::Set('system','widgets',$r);
 	}
 }

Zotlabs/Identity/OAuth2Server.php

@@ -2,6 +2,8 @@
 
 namespace Zotlabs\Identity;
 
+use Zotlabs\Lib\Config;
+
 class OAuth2Server extends \OAuth2\Server {
 
 	public function __construct(OAuth2Storage $storage, $config = null) {
@@ -24,8 +26,8 @@ class OAuth2Server extends \OAuth2\Server {
 
 		$keyStorage = new \OAuth2\Storage\Memory( [
 			'keys' => [
-				'public_key'  => get_config('system', 'pubkey'),
-				'private_key' => get_config('system', 'prvkey')
+				'public_key'  => Config::Get('system', 'pubkey'),
+				'private_key' => Config::Get('system', 'prvkey')
 			]
 		]);
 

Zotlabs/Lib/ASCache.php

@@ -0,0 +1,33 @@
+<?php /** @file */
+
+namespace Zotlabs\Lib;
+
+	/**
+	 *  A wrapper for the cache api
+	 */
+
+class ASCache {
+	public static function isEnabled() {
+		return Config::Get('system', 'as_object_cache_enabled', true);
+	}
+
+	public static function getAge() {
+		return Config::Get('system', 'as_object_cache_time', '10 MINUTE');
+	}
+
+	public static function Get($key) {
+		if (!self::isEnabled()) {
+			return;
+		}
+
+		return Cache::get($key, self::getAge());
+	}
+
+	public static function Set($key, $value) {
+		if (!self::isEnabled()) {
+			return;
+		}
+
+		Cache::set($key, $value);
+	}
+}

Zotlabs/Lib/ASCollection.php

@@ -0,0 +1,152 @@
+<?php
+
+namespace Zotlabs\Lib;
+
+/**
+ * Class for dealing with fetching ActivityStreams collections (ordered or unordered, normal or paged).
+ * Construct with either an existing object or url and an optional channel to sign requests.
+ * $direction is 0 (default) to fetch from the beginning, and 1 to fetch from the end and reverse order the resultant array.
+ * An optional limit to the number of records returned may also be specified.
+ * Use $class->get() to return an array of collection members.
+ */
+class ASCollection {
+
+	private $channel = null;
+	private $nextpage = null;
+	private $limit = 0;
+	private $direction = 0;  // 0 = forward, 1 = reverse
+	private $data = [];
+	private $history = [];
+
+	function __construct($obj, $channel = null, $direction = 0, $limit = 0) {
+
+		$this->channel   = $channel;
+		$this->direction = $direction;
+		$this->limit     = $limit;
+
+		$data = null;
+
+		if (is_array($obj)) {
+			$data = $obj;
+		}
+
+		if (is_string($obj)) {
+			$data            = Activity::fetch($obj, $channel);
+			$this->history[] = $obj;
+		}
+
+		if (!is_array($data)) {
+			return;
+		}
+
+		if (!in_array($data['type'], ['Collection', 'OrderedCollection', 'OrderedCollectionPage'])) {
+			return false;
+		}
+
+		if ($this->direction) {
+			if (array_key_exists('last', $data) && $data['last']) {
+				$this->nextpage = $data['last'];
+			}
+		}
+		else {
+			if (array_key_exists('first', $data) && $data['first']) {
+				$this->nextpage = $data['first'];
+			}
+		}
+
+		if (isset($data['items']) && is_array($data['items'])) {
+			$this->data = (($this->direction) ? array_reverse($data['items']) : $data['items']);
+		}
+		elseif (isset($data['orderedItems']) && is_array($data['orderedItems'])) {
+			$this->data = (($this->direction) ? array_reverse($data['orderedItems']) : $data['orderedItems']);
+		}
+
+		if ($this->limit) {
+			if (count($this->data) > $limit) {
+				$this->data = array_slice($this->data, 0, $limit);
+				return;
+			}
+		}
+
+		do {
+			$x = $this->next();
+		} while ($x);
+	}
+
+	function get() {
+		return $this->data;
+	}
+
+	function next() {
+
+		if (!$this->nextpage) {
+			return false;
+		}
+
+		if (is_array($this->nextpage)) {
+			$data = $this->nextpage;
+		}
+
+		if (is_string($this->nextpage)) {
+			if (in_array($this->nextpage, $this->history)) {
+				// recursion detected
+				return false;
+			}
+			$data            = Activity::fetch($this->nextpage, $this->channel);
+			$this->history[] = $this->nextpage;
+		}
+
+		if (!is_array($data)) {
+			return false;
+		}
+
+		if (!in_array($data['type'], ['CollectionPage', 'OrderedCollectionPage'])) {
+			return false;
+		}
+
+		$this->setnext($data);
+
+		if (isset($data['items']) && is_array($data['items'])) {
+			$this->data = array_merge($this->data, (($this->direction) ? array_reverse($data['items']) : $data['items']));
+		}
+		elseif (isset($data['orderedItems']) && is_array($data['orderedItems'])) {
+			$this->data = array_merge($this->data, (($this->direction) ? array_reverse($data['orderedItems']) : $data['orderedItems']));
+		}
+
+		if ($this->limit) {
+			if (count($this->data) > $this->limit) {
+				$this->data     = array_slice($this->data, 0, $this->limit);
+				$this->nextpage = false;
+				return true;
+			}
+		}
+
+		return true;
+	}
+
+	function setnext($data) {
+		if ($this->direction) {
+			if (array_key_exists('prev', $data) && $data['prev']) {
+				$this->nextpage = $data['prev'];
+			}
+			elseif (array_key_exists('first', $data) && $data['first']) {
+				$this->nextpage = $data['first'];
+			}
+			else {
+				$this->nextpage = false;
+			}
+		}
+		else {
+			if (array_key_exists('next', $data) && $data['next']) {
+				$this->nextpage = $data['next'];
+			}
+			elseif (array_key_exists('last', $data) && $data['last']) {
+				$this->nextpage = $data['last'];
+			}
+			else {
+				$this->nextpage = false;
+			}
+		}
+		logger('nextpage: ' . $this->nextpage, LOGGER_DEBUG);
+	}
+}

Zotlabs/Lib/AbConfig.php

@@ -6,12 +6,17 @@ namespace Zotlabs\Lib;
 class AbConfig {
 
 	static public function Load($chan,$xhash,$family = '') {
-		if($family)
+		$where = '';
+
+		if($family) {
 			$where = sprintf(" and cat = '%s' ",dbesc($family));
+		}
+
 		$r = q("select * from abconfig where chan = %d and xchan = '%s' $where",
 			intval($chan),
 			dbesc($xhash)
 		);
+
 		return $r;
 	}
 
@@ -21,7 +26,7 @@ class AbConfig {
 			intval($chan),
 			dbesc($xhash),
 			dbesc($family),
-			dbesc($key)		
+			dbesc($key)
 		);
 		if($r) {
 			return ((preg_match('|^a:[0-9]+:{.*}$|s', $r[0]['v'])) ? unserialize($r[0]['v']) : $r[0]['v']);
@@ -41,19 +46,19 @@ class AbConfig {
 				dbesc($xhash),
 				dbesc($family),
 				dbesc($key),
-				dbesc($dbvalue)		
+				dbesc($dbvalue)
 			);
 		}
 		else {
 			$r = q("update abconfig set v = '%s' where chan = %d and xchan = '%s' and cat = '%s' and k = '%s' ",
-				dbesc($dbvalue),		
+				dbesc($dbvalue),
 				dbesc($chan),
 				dbesc($xhash),
 				dbesc($family),
 				dbesc($key)
 			);
 		}
-	
+
 		if($r)
 			return $value;
 		return false;

Zotlabs/Lib/AccessList.php

@@ -1,38 +1,37 @@
-<?php 
+<?php
 
 namespace Zotlabs\Lib;
 
-use Zotlabs\Lib\Libsync;
-
-
 class AccessList {
-	
-	static function add($uid,$name,$public = 0) {
 
-		$ret = false;
+	static function add($uid, $name, $public = 0) {
+
+		$ret  = false;
+		$hash = '';
 		if ($uid && $name) {
-			$r = self::byname($uid,$name); // check for dups
+			$r = self::by_name($uid, $name); // check for dups
 			if ($r !== false) {
 
-				// This could be a problem. 
+				// This could be a problem.
 				// Let's assume we've just created a list which we once deleted
 				// all the old members are gone, but the list remains so we don't break any security
 				// access lists. What we're doing here is reviving the dead list, but old content which
-				// was restricted to this list may now be seen by the new list members. 
+				// was restricted to this list may now be seen by the new list members.
 
 				$z = q("SELECT * FROM pgrp WHERE id = %d LIMIT 1",
 					intval($r)
 				);
-				if(($z) && $z[0]['deleted']) {
+				if (($z) && $z[0]['deleted']) {
 					q('UPDATE pgrp SET deleted = 0 WHERE id = %d', intval($z[0]['id']));
-					notice( t('A deleted list with this name was revived. Existing item permissions <strong>may</strong> apply to this list and any future members. If this is not what you intended, please create another list with a different name.') . EOL); 
+					notice(t('A deleted privacy group with this name was revived. Existing item permissions <strong>may</strong> apply to this privacy group and any future members. If this is not what you intended, please create another privacy group with a different name.') . EOL);
 				}
-				return true;
+				$hash = self::by_id($uid, $r);
+				return $hash;
 			}
 
 			$hash = new_uuid();
 
-			$r = q("INSERT INTO pgrp ( hash, uid, visible, gname )
+			$r   = q("INSERT INTO pgrp ( hash, uid, visible, gname )
 				VALUES( '%s', %d, %d, '%s' ) ",
 				dbesc($hash),
 				intval($uid),
@@ -42,12 +41,12 @@ class AccessList {
 			$ret = $r;
 		}
 
-		Libsync::build_sync_packet($uid,null,true);
-		return $ret;
+		Libsync::build_sync_packet($uid, null, true);
+
+		return (($ret) ? $hash : $ret);
 	}
 
-
-	static function remove($uid,$name) {
+	static function remove($uid, $name) {
 		$ret = false;
 		if ($uid && $name) {
 			$r = q("SELECT id, hash FROM pgrp WHERE uid = %d AND gname = '%s' LIMIT 1",
@@ -55,36 +54,36 @@ class AccessList {
 				dbesc($name)
 			);
 			if ($r) {
-				$group_id = $r[0]['id'];
+				$group_id   = $r[0]['id'];
 				$group_hash = $r[0]['hash'];
 			}
 			else {
 				return false;
 			}
-			
+
 			// remove group from default posting lists
 			$r = q("SELECT channel_default_group, channel_allow_gid, channel_deny_gid FROM channel WHERE channel_id = %d LIMIT 1",
-			       intval($uid)
+				intval($uid)
 			);
 			if ($r) {
 				$user_info = array_shift($r);
-				$change = false;
+				$change    = false;
 
 				if ($user_info['channel_default_group'] == $group_hash) {
 					$user_info['channel_default_group'] = '';
-					$change = true;
+					$change                             = true;
 				}
 				if (strpos($user_info['channel_allow_gid'], '<' . $group_hash . '>') !== false) {
 					$user_info['channel_allow_gid'] = str_replace('<' . $group_hash . '>', '', $user_info['channel_allow_gid']);
-					$change = true;
+					$change                         = true;
 				}
 				if (strpos($user_info['channel_deny_gid'], '<' . $group_hash . '>') !== false) {
 					$user_info['channel_deny_gid'] = str_replace('<' . $group_hash . '>', '', $user_info['channel_deny_gid']);
-					$change = true;
+					$change                        = true;
 				}
 
 				if ($change) {
-					q("UPDATE channel SET channel_default_group = '%s', channel_allow_gid = '%s', channel_deny_gid = '%s' 
+					q("UPDATE channel SET channel_default_group = '%s', channel_allow_gid = '%s', channel_deny_gid = '%s'
 						WHERE channel_id = %d",
 						intval($user_info['channel_default_group']),
 						dbesc($user_info['channel_allow_gid']),
@@ -110,16 +109,16 @@ class AccessList {
 
 		}
 
-		Libsync::build_sync_packet($uid,null,true);
+		Libsync::build_sync_packet($uid, null, true);
 
 		return $ret;
 	}
 
 	// returns the integer id of an access group owned by $uid and named $name
 	// or false.
-	
-	static function byname($uid,$name) {
-		if (! ($uid && $name)) {
+
+	static function by_name($uid, $name) {
+		if (!($uid && $name)) {
 			return false;
 		}
 		$r = q("SELECT id FROM pgrp WHERE uid = %d AND gname = '%s' LIMIT 1",
@@ -132,11 +131,11 @@ class AccessList {
 		return false;
 	}
 
-	static function by_id($uid,$id) {
-		if (! ($uid && $id)) {
+	static function by_id($uid, $id) {
+		if (!($uid && $id)) {
 			return false;
 		}
-		
+
 		$r = q("SELECT * FROM pgrp WHERE uid = %d AND id = %d and deleted = 0",
 			intval($uid),
 			intval($id)
@@ -147,10 +146,8 @@ class AccessList {
 		return false;
 	}
 
-
-
-	static function rec_byhash($uid,$hash) {
-		if (! ( $uid && $hash)) {
+	static function by_hash($uid, $hash) {
+		if (!($uid && $hash)) {
 			return false;
 		}
 		$r = q("SELECT * FROM pgrp WHERE uid = %d AND hash = '%s' LIMIT 1",
@@ -163,46 +160,46 @@ class AccessList {
 		return false;
 	}
 
+	static function member_remove($uid, $name, $member, $gid = 0) {
+		if (!$gid) {
+			$gid = self::by_name($uid, $name);
+		}
 
-	static function member_remove($uid,$name,$member) {
-		$gid = self::byname($uid,$name);
-		if (! $gid) {
-			return false;
-		}
-		if (! ($uid && $gid && $member)) {
+		if (!($uid && $gid && $member)) {
 			return false;
 		}
+
 		$r = q("DELETE FROM pgrp_member WHERE uid = %d AND gid = %d AND xchan = '%s' ",
 			intval($uid),
 			intval($gid),
 			dbesc($member)
 		);
 
-		Libsync::build_sync_packet($uid,null,true);
+		Libsync::build_sync_packet($uid, null, true);
 
 		return $r;
 	}
 
-
-	static function member_add($uid,$name,$member,$gid = 0) {
-		if (! $gid) {
-			$gid = self::byname($uid,$name);
+	static function member_add($uid, $name, $member, $gid = 0) {
+		if (!$gid) {
+			$gid = self::by_name($uid, $name);
 		}
-		if (! ($gid && $uid && $member)) {
+		if (!($gid && $uid && $member)) {
 			return false;
 		}
 
-		$r = q("SELECT * FROM pgrp_member WHERE uid = %d AND gid = %d AND xchan = '%s' LIMIT 1",	
+		$r = q("SELECT * FROM pgrp_member WHERE uid = %d AND gid = %d AND xchan = '%s' LIMIT 1",
 			intval($uid),
 			intval($gid),
 			dbesc($member)
 		);
 		if ($r) {
-			return true;	// You might question this, but 
-				// we indicate success because the group member was in fact created
-				// -- It was just created at another time
+			return true;
+			// You might question this, but
+			// we indicate success because the group member was in fact created
+			// -- It was just created at another time
 		}
-	 	else {
+		else {
 			$r = q("INSERT INTO pgrp_member (uid, gid, xchan)
 				VALUES( %d, %d, '%s' ) ",
 				intval($uid),
@@ -210,15 +207,14 @@ class AccessList {
 				dbesc($member)
 			);
 		}
-		Libsync::build_sync_packet($uid,null,true);
+		Libsync::build_sync_packet($uid, null, true);
 		return $r;
 	}
 
-
 	static function members($uid, $gid) {
 		$ret = [];
 		if (intval($gid)) {
-			$r = q("SELECT * FROM pgrp_member 
+			$r = q("SELECT * FROM pgrp_member
 				LEFT JOIN abook ON abook_xchan = pgrp_member.xchan left join xchan on xchan_hash = abook_xchan
 				WHERE gid = %d AND abook_channel = %d and pgrp_member.uid = %d and xchan_deleted = 0 and abook_self = 0 and abook_blocked = 0 and abook_pending = 0 ORDER BY xchan_name ASC ",
 				intval($gid),
@@ -232,7 +228,7 @@ class AccessList {
 		return $ret;
 	}
 
-	static function members_xchan($uid,$gid) {
+	static function members_xchan($uid, $gid) {
 		$ret = [];
 		if (intval($gid)) {
 			$r = q("SELECT xchan FROM pgrp_member WHERE gid = %d AND uid = %d",
@@ -248,99 +244,75 @@ class AccessList {
 		return $ret;
 	}
 
-	static function members_profile_xchan($uid,$gid) {
+	static function profile_members_xchan($uid,$gid) {
 		$ret = [];
-		if (intval($gid)) {
+
+		if(intval($gid)) {
 			$r = q("SELECT abook_xchan as xchan from abook left join profile on abook_profile = profile_guid where profile.id = %d and profile.uid = %d",
 				intval($gid),
 				intval($uid)
 			);
-			if ($r) {
-				foreach($r as $rv) {
-					$ret[] = $rv['xchan'];
+			if($r) {
+				foreach($r as $rr) {
+					$ret[] = $rr['xchan'];
 				}
 			}
 		}
 		return $ret;
 	}
 
+	static function select($uid, $options) {
 
+		$selected = $options['selected'] ?? '';
+		$form_id = $options['form_id'] ?? 'accesslist_select';
+		$label = $options['label'] ?? t('Select a privacy group');
+		$before = $options['before'] ?? [];
+		$after = $options['after'] ?? [];
 
-
-	static function select($uid,$group = '') {
-	
 		$grps = [];
+		$o = '';
+
+		$grps[] = [
+			'name' => '',
+			'id' => '0',
+			'selected' => false
+		];
+
+		if ($before) {
+			$grps[] = $before;
+		}
 
 		$r = q("SELECT * FROM pgrp WHERE deleted = 0 AND uid = %d ORDER BY gname ASC",
 			intval($uid)
 		);
-		$grps[] = [ 'name' => '', 'hash' => '0', 'selected' => '' ];
-		if ($r) {
-			foreach ($r as $rr) {
-				$grps[] = [ 'name' => $rr['gname'], 'id' => $rr['hash'], 'selected' => (($group == $rr['hash']) ? 'true' : '') ];
-			}
 
-		}
-		
-		return replace_macros(get_markup_template('group_selection.tpl'), [
-			'$label' => t('Add new connections to this access list'),
-			'$groups' => $grps 
-		]);
-	}
-
-
-	static function widget($every="connections",$each="lists",$edit = false, $group_id = 0, $cid = '',$mode = 1) {
-
-		$o = '';
-
-		$groups = [];
-
-		$r = q("SELECT * FROM pgrp WHERE deleted = 0 AND uid = %d ORDER BY gname ASC",
-			intval($_SESSION['uid'])
-		);
-		$member_of = [];
-		if ($cid) {
-			$member_of = self::containing(local_channel(),$cid);
-		} 
-
-		if ($r) {
-			foreach ($r as $rr) {
-				$selected = (($group_id == $rr['id']) ? ' group-selected' : '');
-			
-				if ($edit) {
-					$groupedit = [ 'href' => "lists/".$rr['id'], 'title' => t('edit') ];
-				} 
-				else {
-					$groupedit = null;
-				}
-			
-				$groups[] = [
-					'id'		=> $rr['id'],
-					'enc_cid'   => base64url_encode($cid),
-					'cid'		=> $cid,
-					'text' 		=> $rr['gname'],
-					'selected' 	=> $selected,
-					'href'		=> (($mode == 0) ? $each.'?f=&gid='.$rr['id'] : $each."/".$rr['id']) . ((x($_GET,'new')) ? '&new=' . $_GET['new'] : '') . ((x($_GET,'order')) ? '&order=' . $_GET['order'] : ''),
-					'edit'		=> $groupedit,
-					'ismember'	=> in_array($rr['id'],$member_of),
+		if($r) {
+			foreach($r as $rr) {
+				$grps[] = [
+					'name' => $rr['gname'],
+					'id' => $rr['hash'],
+					'selected' => ($selected == $rr['hash'])
 				];
 			}
 		}
-		
-		return replace_macros(get_markup_template('group_side.tpl'), [
-			'$title'		=> t('Lists'),
-			'$edittext'     => t('Edit list'),
-			'$createtext' 	=> t('Create new list'),
-			'$ungrouped'    => (($every === 'contacts') ? t('Channels not in any access list') : ''),
-			'$groups'		=> $groups,
-			'$add'			=> t('add'),
-		]);
 
+		if ($after) {
+			$grps[] = $after;
+		}
+
+		logger('select: ' . print_r($grps,true), LOGGER_DATA);
+
+		$o = replace_macros(get_markup_template('group_selection.tpl'), array(
+			'$label' => $label,
+			'$form_id' => $form_id,
+			'$groups' => $grps
+		));
+
+		return $o;
 	}
 
-
 	static function expand($g) {
-		if (! (is_array($g) && count($g))) {
+		if (!(is_array($g) && count($g))) {
 			return [];
 		}
 
@@ -350,8 +322,8 @@ class AccessList {
 		// private profile linked virtual groups
 
 		foreach ($g as $gv) {
-			if (substr($gv,0,3) === 'vp.') {
-				$profile_hash = substr($gv,3);
+			if (substr($gv, 0, 3) === 'vp.') {
+				$profile_hash = substr($gv, 3);
 				if ($profile_hash) {
 					$r = q("select abook_xchan from abook where abook_profile = '%s'",
 						dbesc($profile_hash)
@@ -366,10 +338,10 @@ class AccessList {
 			else {
 				$x[] = $gv;
 			}
-		}								 
+		}
 
 		if ($x) {
-			stringify_array_elms($x,true);
+			stringify_array_elms($x, true);
 			$groups = implode(',', $x);
 			if ($groups) {
 				$r = q("SELECT xchan FROM pgrp_member WHERE gid IN ( select id from pgrp where hash in ( $groups ))");
@@ -383,9 +355,8 @@ class AccessList {
 		return $ret;
 	}
 
-
 	static function member_of($c) {
-		$r = q("SELECT pgrp.gname, pgrp.id FROM pgrp LEFT JOIN pgrp_member ON pgrp_member.gid = pgrp.id 
+		$r = q("SELECT pgrp.gname, pgrp.id FROM pgrp LEFT JOIN pgrp_member ON pgrp_member.gid = pgrp.id
 			WHERE pgrp_member.xchan = '%s' AND pgrp.deleted = 0 ORDER BY pgrp.gname  ASC ",
 			dbesc($c)
 		);
@@ -393,7 +364,7 @@ class AccessList {
 		return $r;
 	}
 
-	static function containing($uid,$c) {
+	static function containing($uid, $c) {
 
 		$r = q("SELECT gid FROM pgrp_member WHERE uid = %d AND pgrp_member.xchan = '%s' ",
 			intval($uid),
@@ -405,7 +376,8 @@ class AccessList {
 			foreach ($r as $rv)
 				$ret[] = $rv['gid'];
 		}
-		
+
 		return $ret;
 	}
-}
+
+}

Zotlabs/Lib/ActivityStreams.php

@@ -7,26 +7,28 @@ namespace Zotlabs\Lib;
  *
  * Parses an ActivityStream JSON string.
  */
-
 class ActivityStreams {
 
-	public $raw        = null;
-	public $data       = null;
-	public $valid      = false;
-	public $deleted    = false;
-	public $id         = '';
-	public $parent_id  = '';
-	public $type       = '';
-	public $actor      = null;
-	public $obj        = null;
-	public $tgt        = null;
-	public $origin     = null;
-	public $owner      = null;
-	public $signer     = null;
-	public $ldsig      = null;
-	public $sigok      = false;
-	public $recips     = null;
+	public $raw = null;
+	public $data = null;
+	public $meta = null;
+	public $valid = false;
+	public $deleted = false;
+	public $portable_id = null;
+	public $id = '';
+	public $parent_id = '';
+	public $type = '';
+	public $actor = null;
+	public $obj = null;
+	public $tgt = null;
+	public $origin = null;
+	public $owner = null;
+	public $signer = null;
+	public $sig = null;
+	public $sigok = false;
+	public $recips = null;
 	public $raw_recips = null;
+	public $saved_recips = null;
 
 	/**
 	 * @brief Constructor for ActivityStreams.
@@ -35,92 +37,127 @@ class ActivityStreams {
 	 *
 	 * @param string $string
 	 */
-	function __construct($string) {
+	function __construct($string, $portable_id = null) {
 
-		$this->raw  = $string;
+		if(!$string)
+			return;
 
-		if(is_array($string)) {
+		$this->raw = $string;
+		$this->portable_id = $portable_id;
+
+		if (is_array($string)) {
 			$this->data = $string;
+			$this->raw = json_encode($string, JSON_UNESCAPED_SLASHES);
 		}
 		else {
 			$this->data = json_decode($string, true);
 		}
 
-		if($this->data) {
+		if ($this->data) {
 
 			// verify and unpack JSalmon signature if present
 
-			if(is_array($this->data) && array_key_exists('signed',$this->data)) {
+			if (is_array($this->data) && array_key_exists('signed', $this->data)) {
 				$ret = JSalmon::verify($this->data);
 				$tmp = JSalmon::unpack($this->data['data']);
-				if($ret && $ret['success']) {
-					if($ret['signer']) {
-						$saved = json_encode($this->data,JSON_UNESCAPED_SLASHES);
-						$this->data = $tmp;
-						$this->data['signer'] = $ret['signer'];
-						$this->data['signed_data'] = $saved;
-						if($ret['hubloc']) {
-							$this->data['hubloc'] = $ret['hubloc'];
+				if ($ret && $ret['success']) {
+					if ($ret['signer']) {
+						$saved                     = json_encode($this->data, JSON_UNESCAPED_SLASHES);
+						$this->data                = $tmp;
+						$this->meta['signer']      = $ret['signer'];
+						$this->meta['signed_data'] = $saved;
+						if ($ret['hubloc']) {
+							$this->meta['hubloc'] = $ret['hubloc'];
 						}
 					}
 				}
 			}
 
+			// This indicates only that we have sucessfully decoded JSON.
 			$this->valid = true;
 
-			if(array_key_exists('type',$this->data) && array_key_exists('actor',$this->data) && array_key_exists('object',$this->data)) {
-				if($this->data['type'] === 'Delete' && $this->data['actor'] === $this->data['object']) {
+			// Special handling for Mastodon "delete actor" activities which will often fail to verify
+			// because the key cannot be fetched. We will catch this condition elsewhere.
+			if (is_array($this->data) && array_key_exists('type', $this->data) && array_key_exists('actor', $this->data) && array_key_exists('object', $this->data)) {
+				if ($this->data['type'] === 'Delete' && $this->data['actor'] === $this->data['object']) {
 					$this->deleted = $this->data['actor'];
-					$this->valid = false;
+					$this->valid   = false;
 				}
 			}
 
 		}
 
-		if($this->is_valid()) {
-			$this->id     = $this->get_property_obj('id');
+		// Attempt to assemble an Activity from what we were given.
+		if ($this->is_valid()) {
+			$this->id = $this->get_property_obj('id');
+
+			if  (!$this->id) {
+				logger('Data with mmissing id: ' . print_r($this->data, true));
+				return;
+			}
+
+			// cache for future use
+			ASCache::Set($this->id, 'json:' . $this->raw);
+
 			$this->type   = $this->get_primary_type();
-			$this->actor  = $this->get_actor('actor','','');
+			$this->actor  = $this->get_actor('actor', '', '');
 			$this->obj    = $this->get_compound_property('object');
 			$this->tgt    = $this->get_compound_property('target');
 			$this->origin = $this->get_compound_property('origin');
 			$this->recips = $this->collect_recips();
 
-			$this->ldsig = $this->get_compound_property('signature');
-			if($this->ldsig) {
-				$this->signer = $this->get_compound_property('creator',$this->ldsig);
-				if($this->signer && is_array($this->signer) && array_key_exists('publicKey',$this->signer) && is_array($this->signer['publicKey']) && $this->signer['publicKey']['publicKeyPem']) {
-					$this->sigok = LDSignatures::verify($this->data,$this->signer['publicKey']['publicKeyPem']);
+			$this->sig = $this->get_compound_property('proof');
+			if ($this->sig) {
+				$this->checkEddsaSignature(); // will set signer and sigok if everything works out
+			}
+
+			// Try LDSignatures if edsig failed
+			if (!$this->sigok) {
+				$this->sig = $this->get_compound_property('signature');
+				if ($this->sig) {
+					$this->signer = $this->get_actor('creator', $this->sig);
+					if ($this->signer && is_array($this->signer) && array_key_exists('publicKey', $this->signer) && is_array($this->signer['publicKey']) && $this->signer['publicKey']['publicKeyPem']) {
+						$this->sigok = LDSignatures::verify($this->data, $this->signer['publicKey']['publicKeyPem']);
+					}
 				}
 			}
 
-			if(! $this->obj) {
-				$this->obj = $this->data;
+			if (!$this->obj) {
+				$this->obj  = $this->data;
 				$this->type = 'Create';
-				if(! $this->actor) {
-					$this->actor = $this->get_actor('attributedTo',$this->obj);
+				if (!$this->actor) {
+					$this->actor = $this->get_actor('attributedTo', $this->obj);
 				}
 			}
 
-			// fetch recursive or embedded activities
+			// Fetch recursive or embedded activities
 
-			if ($this->obj && is_array($this->obj) && array_key_exists('object',$this->obj)) {
-				$this->obj['object'] = $this->get_compound_property($this->obj['object']);
+			if ($this->obj && is_array($this->obj) && array_key_exists('object', $this->obj)) {
+				$this->obj['object'] = $this->get_compound_property('object', $this->obj);
 			}
 
-			if($this->obj && is_array($this->obj) && $this->obj['actor'])
-				$this->obj['actor'] = $this->get_actor('actor',$this->obj);
-			if($this->tgt && is_array($this->tgt) && $this->tgt['actor'])
-				$this->tgt['actor'] = $this->get_actor('actor',$this->tgt);
+			// Enumerate and store actors in referenced objects
 
-			$this->parent_id = $this->get_property_obj('inReplyTo');
-
-			if((! $this->parent_id) && is_array($this->obj)) {
-				$this->parent_id = $this->obj['inReplyTo'];
+			if ($this->obj && is_array($this->obj) && isset($this->obj['actor'])) {
+				$this->obj['actor'] = $this->get_actor('actor', $this->obj);
 			}
-			if((! $this->parent_id) && is_array($this->obj)) {
+
+			if ($this->tgt && is_array($this->tgt) && isset($this->tgt['actor'])) {
+				$this->tgt['actor'] = $this->get_actor('actor', $this->tgt);
+			}
+
+			// Determine if this is a followup or response activity
+
+			$this->parent_id = ((is_array($this->get_property_obj('inReplyTo'))) ? $this->get_property_obj('inReplyTo')['id'] : $this->get_property_obj('inReplyTo'));
+
+			if (!$this->parent_id && isset($this->obj['inReplyTo'])) {
+				$this->parent_id = ((is_array($this->obj['inReplyTo'])) ? $this->obj['inReplyTo']['id'] : $this->obj['inReplyTo']);
+			}
+
+			if (!$this->parent_id && isset($this->obj['id'])) {
 				$this->parent_id = $this->obj['id'];
 			}
+
 		}
 	}
 
@@ -137,51 +174,74 @@ class ActivityStreams {
 		$this->saved_recips = $arr;
 	}
 
+	/**
+	 * @brief get single property from Activity object
+	 *
+	 * @param string $property
+	 * @param mixed $default return value if property or object not set
+	 *    or object is a string id which could not be fetched.
+	 * @return mixed
+	 */
+	public function objprop(string $property, mixed $default = false): mixed {
+		$x = $this->get_property_obj($property, $this->obj);
+		return (isset($x)) ? $x : $default;
+	}
+
 	/**
 	 * @brief Collects all recipients.
 	 *
-	 * @param string $base
+	 * @param mixed $base
 	 * @param string $namespace (optional) default empty
 	 * @return array
 	 */
-	function collect_recips($base = '', $namespace = '') {
-		$x = [];
+	public function collect_recips(mixed $base = '', string $namespace = ''): array {
+		$result = [];
+		$tmp = [];
 
-		$fields = [ 'to', 'cc', 'bto', 'bcc', 'audience'];
-		foreach($fields as $f) {
-			$y = $this->get_compound_property($f, $base, $namespace);
-			if($y) {
-				if (! is_array($this->raw_recips)) {
-					$this->raw_recips = [];
-				}
-
-				if (! is_array($y)) {
-					$y = [ $y ];
-				}
-				$this->raw_recips[$f] = $y;
-				$x = array_merge($x, $y);
+		$fields = ['to', 'cc', 'bto', 'bcc', 'audience'];
+		foreach ($fields as $field) {
+			// don't expand these yet
+			$values = $this->get_property_obj($field, $base, $namespace);
+			if ($values) {
+				$values = force_array($values);
+				$tmp[$field] = $values;
+				$result = array_values(array_unique(array_merge($result, $values)));
+			}
+			// Merge the object recipients if they exist.
+			$values = $this->objprop($field);
+			if ($values) {
+				$values = force_array($values);
+				$tmp[$field] = ((isset($tmp[$field])) ? array_merge($tmp[$field], $values) : $values);
+				$result = array_values(array_unique(array_merge($result, $values)));
+			}
+			// remove duplicates
+			if (isset($tmp[$field])) {
+				$tmp[$field] = array_values(array_unique($tmp[$field]));
 			}
 		}
-// not yet ready for prime time
-//		$x = $this->expand($x,$base,$namespace);
-		return $x;
+		$this->raw_recips = $tmp;
+
+		// not yet ready for prime time
+		//      $result = $this->expand($result,$base,$namespace);
+		return $result;
 	}
 
-	function expand($arr,$base = '',$namespace = '') {
+
+	function expand($arr, $base = '', $namespace = '') {
 		$ret = [];
 
 		// right now use a hardwired recursion depth of 5
 
-		for($z = 0; $z < 5; $z ++) {
-			if(is_array($arr) && $arr) {
-				foreach($arr as $a) {
-					if(is_array($a)) {
+		for ($z = 0; $z < 5; $z++) {
+			if (is_array($arr) && $arr) {
+				foreach ($arr as $a) {
+					if (is_array($a)) {
 						$ret[] = $a;
 					}
 					else {
-						$x = $this->get_compound_property($a,$base,$namespace);
-						if($x) {
-							$ret = array_merge($ret,$x);
+						$x = $this->get_compound_property($a, $base, $namespace);
+						if ($x) {
+							$ret = array_merge($ret, $x);
 						}
 					}
 				}
@@ -202,33 +262,33 @@ class ActivityStreams {
 	 */
 	function get_namespace($base, $namespace) {
 
-		if(! $namespace)
+		if (!$namespace)
 			return '';
 
 		$key = null;
 
-		foreach( [ $this->data, $base ] as $b ) {
-			if(! $b)
+		foreach ([$this->data, $base] as $b) {
+			if (!$b)
 				continue;
 
-			if(array_key_exists('@context', $b)) {
-				if(is_array($b['@context'])) {
-					foreach($b['@context'] as $ns) {
-						if(is_array($ns)) {
-							foreach($ns as $k => $v) {
-								if($namespace === $v)
+			if (array_key_exists('@context', $b)) {
+				if (is_array($b['@context'])) {
+					foreach ($b['@context'] as $ns) {
+						if (is_array($ns)) {
+							foreach ($ns as $k => $v) {
+								if ($namespace === $v)
 									$key = $k;
 							}
 						}
 						else {
-							if($namespace === $ns) {
+							if ($namespace === $ns) {
 								$key = '';
 							}
 						}
 					}
 				}
 				else {
-					if($namespace === $b['@context']) {
+					if ($namespace === $b['@context']) {
 						$key = '';
 					}
 				}
@@ -248,14 +308,14 @@ class ActivityStreams {
 	 */
 	function get_property_obj($property, $base = '', $namespace = '') {
 		$prefix = $this->get_namespace($base, $namespace);
-		if($prefix === null)
+		if ($prefix === null)
 			return null;
 
-		$base = (($base) ? $base : $this->data);
+		$base     = (($base) ? $base : $this->data);
 		$propname = (($prefix) ? $prefix . ':' : '') . $property;
 
-		if(! is_array($base)) {
-			btlogger('not an array: ' . print_r($base,true));
+		if (!is_array($base)) {
+			btlogger('not an array: ' . print_r($base, true));
 			return null;
 		}
 
@@ -270,23 +330,38 @@ class ActivityStreams {
 	 * @return NULL|mixed
 	 */
 
-	function fetch_property($url) {
-		return self::fetch($url);
-	}
+	function fetch_property($url, $channel = null) {
+		$x = null;
 
-	static function fetch($url, $channel = null) {
-		return Activity::fetch($url, $channel);
+		if (str_starts_with($url, z_root() . '/item/')) {
+			$x = Activity::fetch_local($url, $this->portable_id ?? '');
+			logger('local: ' . print_r($x,true));
+		}
+
+		if (!$x) {
+			$x = Activity::fetch($url, $channel);
+			if ($x === null && strpos($url, '/channel/')) {
+				// look for other nomadic channels which might be alive
+				$zf = Zotfinger::exec($url, $channel);
+				if ($zf) {
+					$url = $zf['signature']['signer'];
+					$x = Activity::fetch($url, $channel);
+				}
+			}
+		}
+
+		return $x;
 	}
 
 	static function is_an_actor($s) {
-		return (in_array($s, [ 'Application','Group','Organization','Person','Service' ]));
+		return (in_array($s, ['Application', 'Group', 'Organization', 'Person', 'Service']));
 	}
 
 	static function is_response_activity($s) {
-		if (! $s) {
+		if (!$s) {
 			return false;
 		}
-		return (in_array($s, [ 'Like', 'Dislike', 'Flag', 'Block', 'Accept', 'Reject', 'TentativeAccept', 'TentativeReject', 'emojiReaction', 'EmojiReaction', 'EmojiReact' ]));
+		return (in_array($s, ['Announce', 'Like', 'Dislike', 'Flag', 'Block', 'Accept', 'Reject', 'TentativeAccept', 'TentativeReject', 'emojiReaction', 'EmojiReaction', 'EmojiReact']));
 	}
 
 	/**
@@ -298,30 +373,27 @@ class ActivityStreams {
 	 * @return NULL|mixed
 	 */
 
-	function get_actor($property,$base='',$namespace = '') {
+	function get_actor($property, $base = '', $namespace = '') {
 		$x = $this->get_property_obj($property, $base, $namespace);
-		if($this->is_url($x)) {
 
-			// SECURITY: If we have already stored the actor profile, re-generate it
-			// from cached data - don't refetch it from the network
-
-			$r = q("select * from xchan left join hubloc on xchan_hash = hubloc_hash where hubloc_id_url = '%s' limit 1",
-				dbesc($x)
-			);
-			if($r) {
-				$y = Activity::encode_person($r[0]);
-				$y['cached'] = true;
+		if ($this->is_url($x)) {
+			$y = Activity::get_actor($x);
+			if ($y) {
 				return $y;
 			}
 		}
-		$actor = $this->get_compound_property($property,$base,$namespace,true);
-		if(is_array($actor) && self::is_an_actor($actor['type'])) {
-			if(array_key_exists('id',$actor) && (! array_key_exists('inbox',$actor))) {
+
+		$actor = $this->get_compound_property($property, $base, $namespace, true);
+
+		if (is_array($actor) && self::is_an_actor($actor['type'])) {
+			if (array_key_exists('id', $actor) && (!array_key_exists('inbox', $actor))) {
 				$actor = $this->fetch_property($actor['id']);
 			}
 			return $actor;
 		}
-		return null;
+
+		return Activity::get_unknown_actor($this->data);
+
 	}
 
 
@@ -336,31 +408,44 @@ class ActivityStreams {
 	 */
 	function get_compound_property($property, $base = '', $namespace = '', $first = false) {
 		$x = $this->get_property_obj($property, $base, $namespace);
-		if($this->is_url($x)) {
-			$y = $this->fetch_property($x);
+
+		if ($this->is_url($x)) {
+			$cached = ASCache::Get($x);
+			if ($cached) {
+				// logger('AS cached: ' . $x);
+				$y = unserialise($cached);
+			}
+			else {
+				// logger('AS fetching: ' . $x);
+				$y = $this->fetch_property($x);
+				if ($y) {
+					ASCache::Set($x, serialise($y));
+				}
+			}
 			if (is_array($y)) {
 				$x = $y;
 			}
 		}
 
+
 		// verify and unpack JSalmon signature if present
 
-		if(is_array($x) && array_key_exists('signed',$x)) {
+		if (is_array($x) && array_key_exists('signed', $x)) {
 			$ret = JSalmon::verify($x);
 			$tmp = JSalmon::unpack($x['data']);
-			if($ret && $ret['success']) {
-				if($ret['signer']) {
-					$saved = json_encode($x,JSON_UNESCAPED_SLASHES);
-					$x = $tmp;
-					$x['signer'] = $ret['signer'];
-					$x['signed_data'] = $saved;
-					if($ret['hubloc']) {
-						$x['hubloc'] = $ret['hubloc'];
+			if ($ret && $ret['success']) {
+				if ($ret['signer']) {
+					$saved            = json_encode($x, JSON_UNESCAPED_SLASHES);
+					$x                = $tmp;
+					$x['meta']['signer']      = $ret['signer'];
+					$x['meta']['signed_data'] = $saved;
+					if ($ret['hubloc']) {
+						$x['meta']['hubloc'] = $ret['hubloc'];
 					}
 				}
 			}
 		}
-		if($first && is_array($x) && array_key_exists(0,$x)) {
+		if ($first && is_array($x) && array_key_exists(0, $x)) {
 			return $x[0];
 		}
 
@@ -374,7 +459,7 @@ class ActivityStreams {
 	 * @return boolean
 	 */
 	function is_url($url) {
-		if(($url) && (! is_array($url)) && (strpos($url, 'http') === 0)) {
+		if (($url) && (!is_array($url)) && (strpos($url, 'http') === 0)) {
 			return true;
 		}
 
@@ -389,13 +474,13 @@ class ActivityStreams {
 	 * @return NULL|mixed
 	 */
 	function get_primary_type($base = '', $namespace = '') {
-		if(! $base)
+		if (!$base)
 			$base = $this->data;
 
 		$x = $this->get_property_obj('type', $base, $namespace);
-		if(is_array($x)) {
-			foreach($x as $y) {
-				if(strpos($y, ':') === false) {
+		if (is_array($x)) {
+			foreach ($x as $y) {
+				if (strpos($y, ':') === false) {
 					return $y;
 				}
 			}
@@ -409,16 +494,91 @@ class ActivityStreams {
 		return $x;
 	}
 
-	static function is_as_request() {
+	static function is_as_request($channel = null) {
 
-		$x = getBestSupportedMimeType([
-			'application/ld+json;profile="https://www.w3.org/ns/activitystreams"',
-			'application/activity+json',
-			'application/ld+json;profile="http://www.w3.org/ns/activitystreams"'
-		]);
+		$hookdata = [];
+		if ($channel)
+			$hookdata['channel'] = $channel;
 
-		return(($x) ? true : false);
+		$hookdata['data'] = ['application/x-zot-activity+json'];
+
+		call_hooks('is_as_request', $hookdata);
+
+		$x = getBestSupportedMimeType($hookdata['data']);
+		return (($x) ? true : false);
 
 	}
 
+	static function get_accept_header_string($channel = null) {
+
+		$ret = '';
+
+		$hookdata = [];
+		if ($channel)
+			$hookdata['channel'] = $channel;
+
+		$hookdata['data'] = ['application/x-zot-activity+json'];
+
+		call_hooks('get_accept_header_string', $hookdata);
+
+		$ret = implode(', ', $hookdata['data']);
+
+		return $ret;
+
+	}
+
+	public function checkEddsaSignature() {
+		$signer = $this->get_property_obj('verificationMethod', $this->sig);
+
+		$parseUrl = parse_url($signer);
+
+		if (isset($parseUrl['fragment'])) {
+			if (str_starts_with($parseUrl['fragment'], 'z6Mk')) {
+				$publicKey = $parseUrl['fragment'];
+			}
+			unset($parseUrl['fragment']);
+		}
+
+		if (isset($parseUrl['query'])) {
+			unset($parseUrl['query']);
+		}
+
+		$url = unparse_url($parseUrl);
+
+		$hublocs = Activity::get_actor_hublocs($url);
+
+		$hasStoredKey = false;
+		if ($hublocs) {
+			foreach ($hublocs as $hubloc) {
+				if ($publicKey && $hubloc['xchan_epubkey'] === $publicKey) {
+					$hasStoredKey = true;
+					break;
+				}
+			}
+		}
+
+		if (!$hasStoredKey) {
+			$this->signer = Activity::get_actor($url);
+
+			if (isset($this->signer['assertionMethod'])) {
+				if (!isset($this->signer['assertionMethod'][0])) {
+					$this->signer['assertionMethod'] = [$this->signer['assertionMethod']];
+				}
+
+				foreach($this->signer['assertionMethod'] as $am) {
+					if ($url === $am['controller'] &&
+						$am['type'] === 'Multikey' &&
+						str_starts_with($am['publicKeyMultibase'], 'z6Mk')
+					) {
+						$publicKey = $am['publicKeyMultibase'];
+					}
+				}
+			}
+		}
+
+		if ($publicKey) {
+			$this->sigok = (new JcsEddsa2022)->verify($this->data, $publicKey);
+		}
+	}
+
 }

Zotlabs/Lib/Apps.php

@@ -2,7 +2,8 @@
 
 namespace Zotlabs\Lib;
 
-use Zotlabs\Lib\Libsync;
+use App;
+use Zotlabs\Lib\Config;
 
 require_once('include/plugin.php');
 require_once('include/channel.php');
@@ -21,9 +22,10 @@ class Apps {
 	 * @brief
 	 *
 	 * @param boolean $translate (optional) default true
+	 * @param boolean $sync (optional) default false used if called from sync_sysapps()
 	 * @return array
 	 */
-	static public function get_system_apps($translate = true) {
+	static public function get_system_apps($translate = true, $sync = false) {
 		$ret = [];
 
 		if(is_dir('apps'))
@@ -33,7 +35,7 @@ class Apps {
 
 		if($files) {
 			foreach($files as $f) {
-				$x = self::parse_app_description($f,$translate);
+				$x = self::parse_app_description($f, $translate, $sync);
 				if($x) {
 					$ret[] = $x;
 				}
@@ -45,7 +47,7 @@ class Apps {
 				$path = explode('/',$f);
 				$plugin = trim($path[1]);
 				if(plugin_is_installed($plugin)) {
-					$x = self::parse_app_description($f,$translate);
+					$x = self::parse_app_description($f, $translate, $sync);
 					if($x) {
 						$x['plugin'] = $plugin;
 						$ret[] = $x;
@@ -64,19 +66,19 @@ class Apps {
 	}
 
 	static public function get_base_apps() {
-		$x = get_config('system','base_apps',[
+		$x = Config::Get('system','base_apps',[
 			'Connections',
+			'Contact Roles',
 			'Network',
-			'Settings',
 			'Files',
-			'Channel Home',
-			'View Profile',
+			'Channel',
 			'Photos',
 			'Calendar',
 			'Directory',
 			'Search',
 			'Help',
-			'Profile Photo'
+			'HQ',
+			'Post'
 		]);
 
 		/**
@@ -158,7 +160,7 @@ class Apps {
 		foreach(self::$available_apps as $iapp) {
 			if($iapp['app_id'] == hash('whirlpool',$app['name'])) {
 				$notfound = false;
-				if(($iapp['app_version'] !== $app['version'])
+				if((isset($app['version']) && $iapp['app_version'] !== $app['version'])
 					|| ($app['plugin'] && (! $iapp['app_plugin']))) {
 					return intval($iapp['app_id']);
 				}
@@ -207,9 +209,10 @@ class Apps {
 	 *
 	 * @param string $f filename
 	 * @param boolean $translate (optional) default true
+	 * @param boolean $sync (optional) default false
 	 * @return boolean|array
 	 */
-	static public function parse_app_description($f, $translate = true) {
+	static public function parse_app_description($f, $translate = true, $sync = false) {
 		$ret = [];
 		$matches = [];
 
@@ -234,6 +237,7 @@ class Apps {
 			$ret['photo'] = $baseurl . '/' . get_default_profile_photo(80);
 
 		$ret['type'] = 'system';
+		$ret['plugin'] = '';
 
 		foreach($ret as $k => $v) {
 			if(strpos($v,'http') === 0) {
@@ -255,7 +259,7 @@ class Apps {
 		if(array_key_exists('categories',$ret))
 			$ret['categories'] = str_replace(array('\'','"'),array(''','&dquot;'),$ret['categories']);
 
-		if(array_key_exists('requires',$ret)) {
+		if(array_key_exists('requires',$ret) && !$sync) {
 			$requires = explode(',',$ret['requires']);
 			foreach($requires as $require) {
 				$require = trim(strtolower($require));
@@ -298,7 +302,7 @@ class Apps {
 						break;
 					default:
 						if($config)
-							$unset = ((get_config('system', $require[0]) == $require[1]) ? false : true);
+							$unset = ((Config::Get('system', $require[0]) == $require[1]) ? false : true);
 						else
 							$unset = ((local_channel() && feature_enabled(local_channel(),$require)) ? false : true);
 						if($unset)
@@ -307,14 +311,16 @@ class Apps {
 				}
 			}
 		}
-		if($ret) {
-			if($translate)
-				self::translate_system_apps($ret);
 
-			return $ret;
+		if(empty($ret)) {
+			return false;
 		}
 
-		return false;
+		if($translate) {
+			self::translate_system_apps($ret);
+		}
+
+		return $ret;
 	}
 
 
@@ -340,15 +346,13 @@ class Apps {
 			'Files' => t('Files'),
 			'Webpages' => t('Webpages'),
 			'Wiki' => t('Wiki'),
-			'Channel Home' => t('Channel Home'),
+			'Channel' => t('Channel'),
 			'View Profile' => t('View Profile'),
 			'Photos' => t('Photos'),
 			'Calendar' => t('Calendar'),
 			'Directory' => t('Directory'),
 			'Help' => t('Help'),
 			'Mail' => t('Mail'),
-			'Mood' => t('Mood'),
-			'Poke' => t('Poke'),
 			'Chat' => t('Chat'),
 			'Search' => t('Search'),
 			'Probe' => t('Probe'),
@@ -371,7 +375,7 @@ class Apps {
 			'OAuth Apps Manager' => t('OAuth Apps Manager'),
 			'OAuth2 Apps Manager' => t('OAuth2 Apps Manager'),
 			'PDL Editor' => t('PDL Editor'),
-			'Permission Categories' => t('Permission Categories'),
+			'Contact Roles' => t('Contact Roles'),
 			'Public Stream' => t('Public Stream'),
 			'My Chatrooms' => t('My Chatrooms'),
 			'Channel Export' => t('Channel Export')
@@ -414,15 +418,32 @@ class Apps {
 	static public function app_render($papp, $mode = 'view') {
 		$installed = false;
 
-		if(! $papp)
+		if(!$papp) {
 			return;
+		}
 
-		if(! $papp['photo'])
+		/**
+		 * @hooks app_render_before
+		 * Hook to manipulate the papp array before rendering
+		 */
+
+		$hookinfo = [
+			'name' => $papp['name'],
+			'photo' => $papp['photo']
+		];
+
+		call_hooks('app_render_manipulate_photo', $hookinfo);
+
+		// We will only allow to manipulate the photo
+		$papp['photo'] = $hookinfo['photo'];
+
+		if(!$papp['photo']) {
 			$papp['photo'] = 'icon:gear';
+		}
 
 		self::translate_system_apps($papp);
 
-		if(trim($papp['plugin']) && (! plugin_is_installed(trim($papp['plugin']))))
+		if(isset($papp['plugin']) && trim($papp['plugin']) && (! plugin_is_installed(trim($papp['plugin']))))
 			return '';
 
 		$papp['papp'] = self::papp_encode($papp);
@@ -503,7 +524,7 @@ class Apps {
 							break;
 						default:
 							if($config)
-								$unset = ((get_config('system', $require[0]) === $require[1]) ? false : true);
+								$unset = ((Config::Get('system', $require[0]) === $require[1]) ? false : true);
 							else
 								$unset = ((local_channel() && feature_enabled(local_channel(),$require)) ? false : true);
 							if($unset)
@@ -517,14 +538,19 @@ class Apps {
 		$hosturl = '';
 
 		if(local_channel()) {
-			if(self::app_installed(local_channel(),$papp) && !$papp['deleted'])
+			if(self::app_installed(local_channel(),$papp)) {
 				$installed = true;
+			}
+
+			if ($installed && isset($papp['deleted']) && $papp['deleted']) {
+				$installed = false;
+			}
 
 			$hosturl = z_root() . '/';
 		}
 		elseif(remote_channel()) {
 			$observer = \App::get_observer();
-			if($observer && $observer['xchan_network'] === 'zot') {
+			if($observer && $observer['xchan_network'] === 'zot6') {
 				// some folks might have xchan_url redirected offsite, use the connurl
 				$x = parse_url($observer['xchan_connurl']);
 				if($x) {
@@ -536,13 +562,47 @@ class Apps {
 		$install_action = (($installed) ? t('Update') : t('Install'));
 		$icon = ((strpos($papp['photo'],'icon:') === 0) ? substr($papp['photo'],5) : '');
 
+		if (!$installed && $mode === 'module') {
+			$_SESSION['return_url'] = App::$query_string;
+			return replace_macros(get_markup_template('app_install.tpl'), [
+				'$papp' => $papp,
+				'$install' => $install_action
+			]);
+		}
+
 		if($mode === 'navbar') {
+			return replace_macros(get_markup_template('app_nav_pinned.tpl'),array(
+				'$app' => $papp,
+				'$icon' => $icon,
+			));
+		}
+
+		if($mode === 'nav') {
 			return replace_macros(get_markup_template('app_nav.tpl'),array(
 				'$app' => $papp,
 				'$icon' => $icon,
 			));
 		}
 
+		if($mode === 'inline') {
+			return replace_macros(get_markup_template('app_inline.tpl'),array(
+				'$app' => $papp,
+				'$icon' => $icon,
+				'$installed' => $installed,
+				'$purchase' => ((isset($papp['page']) && (! $installed)) ? t('Purchase') : ''),
+				'$action_label' => $install_action
+			));
+		}
+
+		if(in_array($mode, ['nav-order', 'nav-order-pinned'])) {
+			return replace_macros(get_markup_template('app_order.tpl'),array(
+				'$app' => $papp,
+				'$icon' => $icon,
+				'$hosturl' => $hosturl,
+				'$mode' => $mode
+			));
+		}
+
 		if($mode === 'install') {
 			$papp['embed'] = true;
 		}
@@ -551,20 +611,18 @@ class Apps {
 			'$app' => $papp,
 			'$icon' => $icon,
 			'$hosturl' => $hosturl,
-			'$purchase' => (($papp['page'] && (! $installed)) ? t('Purchase') : ''),
+			'$purchase' => ((isset($papp['page']) && (! $installed)) ? t('Purchase') : ''),
 			'$installed' => $installed,
 			'$action_label' => (($hosturl && in_array($mode, ['view','install'])) ? $install_action : ''),
 			'$edit' => ((local_channel() && $installed && $mode == 'edit') ? t('Edit') : ''),
 			'$delete' => ((local_channel() && $mode == 'edit') ? t('Delete') : ''),
 			'$undelete' => ((local_channel() && $mode == 'edit') ? t('Undelete') : ''),
-			'$settings_url' => ((local_channel() && $installed && $mode == 'list') ? $papp['settings_url'] : ''),
-			'$deleted' => $papp['deleted'],
-			'$feature' => (($papp['embed'] || $mode == 'edit') ? false : true),
-			'$pin' => (($papp['embed'] || $mode == 'edit') ? false : true),
-			'$featured' => ((strpos($papp['categories'], 'nav_featured_app') === false) ? false : true),
-			'$pinned' => ((strpos($papp['categories'], 'nav_pinned_app') === false) ? false : true),
-			'$navapps' => (($mode == 'nav') ? true : false),
-			'$order' => (($mode === 'nav-order' || $mode === 'nav-order-pinned') ? true : false),
+			'$settings_url' => ((local_channel() && $installed && $mode == 'list' && isset($papp['settings_url'])) ? $papp['settings_url'] : ''),
+			'$deleted' => $papp['deleted'] ?? false,
+			'$feature' => ((isset($papp['embed']) || $mode == 'edit') ? false : true),
+			'$pin' => ((isset($papp['embed']) || $mode == 'edit') ? false : true),
+			'$featured' => ((isset($papp['categories']) && strpos($papp['categories'], 'nav_featured_app') !== false) ? true : false),
+			'$pinned' => ((isset($papp['categories']) && strpos($papp['categories'], 'nav_pinned_app') !== false) ? true : false),
 			'$mode' => $mode,
 			'$add' => t('Add to app-tray'),
 			'$remove' => t('Remove from app-tray'),
@@ -574,6 +632,7 @@ class Apps {
 		));
 	}
 
+
 	static public function app_install($uid,$app) {
 
 		if(! is_array($app)) {
@@ -588,10 +647,12 @@ class Apps {
 
 		$app['uid'] = $uid;
 
-		if(self::app_installed($uid,$app,true))
+		if(self::app_installed($uid,$app,true)) {
 			$x = self::app_update($app);
-		else
+		}
+		else {
 			$x = self::app_store($app);
+		}
 
 		if($x['success']) {
 			$r = q("select * from app where app_id = '%s' and app_channel = %d limit 1",
@@ -599,13 +660,12 @@ class Apps {
 				intval($uid)
 			);
 			if($r) {
-				if(($app['uid']) && (! $r[0]['app_system'])) {
-					if($app['categories'] && (! $app['term'])) {
+				if($app['uid']) {
+					if((isset($app['categories']) && $app['categories']) && !(isset($app['term']) && $app['term'])) {
 						$r[0]['term'] = q("select * from term where otype = %d and oid = %d",
 							intval(TERM_OBJ_APP),
 							intval($r[0]['id'])
 						);
-						Libsync::build_sync_packet($uid,array('app' => $r[0]));
 					}
 				}
 			}
@@ -634,6 +694,7 @@ class Apps {
 				}
 			}
 		}
+
 		return true;
 	}
 
@@ -645,38 +706,35 @@ class Apps {
 				dbesc($app['guid']),
 				intval($uid)
 			);
-			if($x) {
-				if(! intval($x[0]['app_deleted'])) {
-					$x[0]['app_deleted'] = 1;
-					if(self::can_delete($uid,$app)) {
-						q("delete from app where app_id = '%s' and app_channel = %d",
-							dbesc($app['guid']),
-							intval($uid)
-						);
-						q("delete from term where otype = %d and oid = %d",
-							intval(TERM_OBJ_APP),
-							intval($x[0]['id'])
-						);
-						/**
-						 * @hooks app_destroy
-						 *  Called after app entry got removed from database
-						 *  and provide app array from database.
-						 */
-						call_hooks('app_destroy', $x[0]);
-					}
-					else {
-						q("update app set app_deleted = 1 where app_id = '%s' and app_channel = %d",
-							dbesc($app['guid']),
-							intval($uid)
-						);
-					}
-					if(! intval($x[0]['app_system'])) {
-						Libsync::build_sync_packet($uid,array('app' => $x));
-					}
-				}
-				else {
-					self::app_undestroy($uid,$app);
-				}
+
+			if($x && intval($x[0]['app_deleted'])) {
+				self::app_undestroy($uid, $app);
+				return;
+			}
+
+			if(self::can_delete($uid,$app)) {
+				q("delete from app where app_id = '%s' and app_channel = %d",
+					dbesc($app['guid']),
+					intval($uid)
+				);
+
+				q("delete from term where otype = %d and oid = %d",
+					intval(TERM_OBJ_APP),
+					intval($x[0]['id'])
+				);
+
+				/**
+				 * @hooks app_destroy
+				 *  Called after app entry got removed from database
+				 *  and provide app array from database.
+				 */
+				call_hooks('app_destroy', $x[0]);
+			}
+			else {
+				q("update app set app_deleted = 1 where app_id = '%s' and app_channel = %d",
+					dbesc($app['guid']),
+					intval($uid)
+				);
 			}
 		}
 	}
@@ -693,13 +751,11 @@ class Apps {
 				dbesc($app['guid']),
 				intval($uid)
 			);
-			if($x) {
-				if($x[0]['app_system']) {
-					q("update app set app_deleted = 0 where app_id = '%s' and app_channel = %d",
-						dbesc($app['guid']),
-						intval($uid)
-					);
-				}
+			if($x && intval($x[0]['app_deleted']) && $x[0]['app_system']) {
+				q("update app set app_deleted = 0 where app_id = '%s' and app_channel = %d",
+					dbesc($app['guid']),
+					intval($uid)
+				);
 			}
 		}
 	}
@@ -905,7 +961,7 @@ class Apps {
 
 		$conf = (($menu === 'nav_featured_app') ? 'app_order' : 'app_pin_order');
 
-		$x = (($uid) ? get_pconfig($uid,'system',$conf) : get_config('system',$conf));
+		$x = (($uid) ? get_pconfig($uid,'system',$conf) : Config::Get('system',$conf));
 		if(($x) && (! is_array($x))) {
 			$y = explode(',',$x);
 			$y = array_map('trim',$y);
@@ -1150,7 +1206,7 @@ class Apps {
 			$ret['success'] = true;
 			$ret['app_id'] = $darray['app_id'];
 		}
-		if($arr['categories']) {
+		if(isset($arr['categories']) && $arr['categories']) {
 			$x = q("select id from app where app_id = '%s' and app_channel = %d limit 1",
 				dbesc($darray['app_id']),
 				intval($darray['app_channel'])
@@ -1158,9 +1214,9 @@ class Apps {
 			$y = explode(',',$arr['categories']);
 			if($y) {
 				foreach($y as $t) {
-					$t = trim($t);
+					$t = escape_tags(trim($t));
 					if($t) {
-						store_item_tag($darray['app_channel'],$x[0]['id'],TERM_OBJ_APP,TERM_CATEGORY,escape_tags($t),escape_tags(z_root() . '/apps/?f=&cat=' . escape_tags($t)));
+						store_item_tag($darray['app_channel'], $x[0]['id'], TERM_OBJ_APP, TERM_CATEGORY, $t, z_root() . '/apps/?f=&cat=' . $t);
 					}
 				}
 			}
@@ -1248,7 +1304,7 @@ class Apps {
 				intval(TERM_OBJ_APP),
 				intval($x[0]['id'])
 			);
-			if($arr['categories']) {
+			if(isset($arr['categories']) && $arr['categories']) {
 				$y = explode(',',$arr['categories']);
 				if($y) {
 					foreach($y as $t) {
@@ -1276,58 +1332,58 @@ class Apps {
 
 		$ret['type'] = 'personal';
 
-		if($app['app_id'])
+		if(!empty($app['app_id']))
 			$ret['guid'] = $app['app_id'];
 
-		if($app['app_sig'])
+		if(!empty($app['app_sig']))
 			$ret['sig'] = $app['app_sig'];
 
-		if($app['app_author'])
+		if(!empty($app['app_author']))
 			$ret['author'] = $app['app_author'];
 
-		if($app['app_name'])
+		if(!empty($app['app_name']))
 			$ret['name'] = $app['app_name'];
 
-		if($app['app_desc'])
+		if(!empty($app['app_desc']))
 			$ret['desc'] = $app['app_desc'];
 
-		if($app['app_url'])
+		if(!empty($app['app_url']))
 			$ret['url'] = $app['app_url'];
 
-		if($app['app_photo'])
+		if(!empty($app['app_photo']))
 			$ret['photo'] = $app['app_photo'];
 
-		if($app['app_icon'])
+		if(!empty($app['app_icon']))
 			$ret['icon'] = $app['app_icon'];
 
-		if($app['app_version'])
+		if(!empty($app['app_version']))
 			$ret['version'] = $app['app_version'];
 
-		if($app['app_addr'])
+		if(!empty($app['app_addr']))
 			$ret['addr'] = $app['app_addr'];
 
-		if($app['app_price'])
+		if(!empty($app['app_price']))
 			$ret['price'] = $app['app_price'];
 
-		if($app['app_page'])
+		if(!empty($app['app_page']))
 			$ret['page'] = $app['app_page'];
 
-		if($app['app_requires'])
+		if(!empty($app['app_requires']))
 			$ret['requires'] = $app['app_requires'];
 
-		if($app['app_system'])
+		if(!empty($app['app_system']))
 			$ret['system'] = $app['app_system'];
 
-		if($app['app_options'])
+		if(!empty($app['app_options']))
 			$ret['options'] = $app['app_options'];
 
-		if($app['app_plugin'])
+		if(!empty($app['app_plugin']))
 			$ret['plugin'] = trim($app['app_plugin']);
 
-		if($app['app_deleted'])
+		if(!empty($app['app_deleted']))
 			$ret['deleted'] = $app['app_deleted'];
 
-		if($app['term']) {
+		if(!empty($app['term']) && is_array($app['term'])) {
 			$s = '';
 			foreach($app['term'] as $t) {
 				if($s)
@@ -1356,4 +1412,17 @@ class Apps {
 		return chunk_split(base64_encode(json_encode($papp)),72,"\n");
 	}
 
+	static public function get_papp($app) {
+
+		$r = q("select * from app where app_id = '%s' and app_channel = 0 limit 1",
+			dbesc(hash('whirlpool', $app))
+		);
+
+		if ($r) {
+			$papp = self::app_encode($r[0]);
+			return $papp;
+		}
+
+		return false;
+	}
 }

Zotlabs/Lib/Cache.php

@@ -2,53 +2,56 @@
 
 namespace Zotlabs\Lib;
 
-	/**
-	 *  cache api
-	 */
-	 
+use Zotlabs\Lib\Config;
+
+/**
+ *  cache api
+ */
 class Cache {
-    
+
     /**
      * @brief Returns cached content
-     * 
+     *
      * @param string $key
      * @param string $age in SQL format, default is '30 DAY'
      * @return string
      */
-    
-	public static function get($key, $age = '') {
 
-		$hash = hash('whirlpool',$key);
+	public static function get($key, $age = '') {
+//		$hash = hash('whirlpool',$key);
+		$hash = uuid_from_url($key);
 
 		$r = q("SELECT v FROM cache WHERE k = '%s' AND updated > %s - INTERVAL %s LIMIT 1",
 			dbesc($hash),
 			db_utcnow(),
-			db_quoteinterval(($age ? $age : get_config('system','object_cache_days', '30') . ' DAY'))
+			db_quoteinterval(($age ? $age : Config::Get('system','object_cache_days', '30') . ' DAY'))
 		);
-			
+
 		if ($r)
 			return $r[0]['v'];
 		return null;
 	}
-		
+
 	public static function set($key,$value) {
+//		$hash = hash('whirlpool',$key);
+		$hash = uuid_from_url($key);
 
-		$hash = hash('whirlpool',$key);
-
-		$r = q("SELECT * FROM cache WHERE k = '%s' limit 1",
+		$r = q("SELECT * FROM cache WHERE k = '%s' LIMIT 1",
 			dbesc($hash)
 		);
 		if($r) {
 			q("UPDATE cache SET v = '%s', updated = '%s' WHERE k = '%s'",
 				dbesc($value),
 				dbesc(datetime_convert()),
-				dbesc($hash));
+				dbesc($hash)
+			);
 		}
 		else {
-			q("INSERT INTO cache ( k, v, updated) VALUES ('%s','%s','%s')",
+			q("INSERT INTO cache (k, v, updated) VALUES ('%s', '%s', '%s')",
 				dbesc($hash),
 				dbesc($value),
-				dbesc(datetime_convert()));
+				dbesc(datetime_convert())
+			);
 		}
 	}
 }

Zotlabs/Lib/Config.php

@@ -2,6 +2,7 @@
 
 namespace Zotlabs\Lib;
 
+use App;
 
 class Config {
 
@@ -14,20 +15,41 @@ class Config {
 	 * @param string $family
 	 *  The category of the configuration value
 	 */
-	static public function Load($family) {
-		if(! array_key_exists($family, \App::$config))
-			\App::$config[$family] = array();
+	public static function Load($family, $recursionCounter = 0) {
+		if (! array_key_exists($family, App::$config)) {
+			App::$config[$family] = [];
+		}
 
-		if(! array_key_exists('config_loaded', \App::$config[$family])) {
+		// We typically continue when presented with minor DB issues,
+		// but loading the site configuration is more important.
+
+		// Check for query returning false and give it approx 30 seconds
+		// to recover if there's a problem. This is intended to fix a
+		// rare issue on Galera where temporary sync issues were causing
+		// the site encryption keys to be regenerated, which was causing
+		// communication issues for members.
+
+		// This code probably belongs at the database layer, but we don't
+		// necessarily want to shut the site down for problematic queries
+		// caused by bad data. That could be used in a denial of service
+		// attack. Those do need to be (and they are) logged.
+
+		if (! array_key_exists('config_loaded', App::$config[$family])) {
 			$r = q("SELECT * FROM config WHERE cat = '%s'", dbesc($family));
-			if($r !== false) {
-				if($r) {
-					foreach($r as $rr) {
-						$k = $rr['k'];
-						\App::$config[$family][$k] = $rr['v'];
-					}
+			if ($r === false && !App::$install) {
+				sleep(3);
+				$recursionCounter ++;
+				if ($recursionCounter > 10) {
+					system_unavailable();
 				}
-				\App::$config[$family]['config_loaded'] = true;
+				self::Load($family, $recursionCounter);
+			}
+			elseif (is_array($r)) {
+				foreach ($r as $rr) {
+					$k = $rr['k'];
+					App::$config[$family][$k] = $rr['v'];
+				}
+				App::$config[$family]['config_loaded'] = true;
 			}
 		}
 	}
@@ -46,19 +68,19 @@ class Config {
 	 * @return mixed
 	 *  Return the set value, or false if the database update failed
 	 */
-	static public function Set($family, $key, $value) {
+	public static function Set($family, $key, $value) {
 		// manage array value
-		$dbvalue = ((is_array($value))  ? serialize($value) : $value);
+		$dbvalue = ((is_array($value))  ? 'json:' . json_encode($value) : $value);
 		$dbvalue = ((is_bool($dbvalue)) ? intval($dbvalue)  : $dbvalue);
 
-		if(self::Get($family, $key) === false || (! self::get_from_storage($family, $key))) {
+		if (self::Get($family, $key) === false || (! self::get_from_storage($family, $key))) {
 			$ret = q("INSERT INTO config ( cat, k, v ) VALUES ( '%s', '%s', '%s' ) ",
 				dbesc($family),
 				dbesc($key),
 				dbesc($dbvalue)
 			);
-			if($ret) {
-				\App::$config[$family][$key] = $value;
+			if ($ret) {
+				App::$config[$family][$key] = $value;
 				$ret = $value;
 			}
 			return $ret;
@@ -70,8 +92,8 @@ class Config {
 			dbesc($key)
 		);
 
-		if($ret) {
-			\App::$config[$family][$key] = $value;
+		if ($ret) {
+			App::$config[$family][$key] = $value;
 			$ret = $value;
 		}
 
@@ -93,21 +115,37 @@ class Config {
 	 *  The category of the configuration value
 	 * @param string $key
 	 *  The configuration key to query
-	 * @param string $default (optional) default false
+	 * @param mixed $default (optional) default false
 	 * @return mixed Return value or false on error or if not set
 	 */
-	static public function Get($family, $key, $default = false) {
-		if((! array_key_exists($family, \App::$config)) || (! array_key_exists('config_loaded', \App::$config[$family])))
-			self::Load($family);
+	public static function Get($family, $key, $default = false) {
 
-		if(array_key_exists('config_loaded', \App::$config[$family])) {
-			if(! array_key_exists($key, \App::$config[$family])) {
+		if ((! array_key_exists($family, App::$config)) || (! array_key_exists('config_loaded', App::$config[$family]))) {
+			self::Load($family);
+		}
+
+		if (array_key_exists('config_loaded', App::$config[$family])) {
+			if (! array_key_exists($key, App::$config[$family])) {
 				return $default;
 			}
-			return ((! is_array(\App::$config[$family][$key])) && (preg_match('|^a:[0-9]+:{.*}$|s', \App::$config[$family][$key]))
-				? unserialize(\App::$config[$family][$key])
-				: \App::$config[$family][$key]
-			);
+
+			$value = App::$config[$family][$key];
+
+			if (! is_array($value)) {
+				if (substr($value, 0, 5) == 'json:') {
+					return json_decode(substr($value, 5), true);
+				} else if (preg_match('|^a:[0-9]+:{.*}$|s', $value)) {
+					// Unserialize in inherently unsafe. Try to mitigate by not
+					// allowing unserializing objects. Only kept for backwards
+					// compatibility. JSON serialization should be prefered.
+					return unserialize($value, array('allowed_classes' => false));
+				} else {
+					return $value;
+				}
+			}
+			else {
+				return $value;
+			}
 		}
 
 		return $default;
@@ -125,12 +163,13 @@ class Config {
 	 *  The configuration key to delete
 	 * @return mixed
 	 */
-	static public function Delete($family, $key) {
+	public static function Delete($family, $key) {
 
 		$ret = false;
 
-		if(array_key_exists($family, \App::$config) && array_key_exists($key, \App::$config[$family]))
-			unset(\App::$config[$family][$key]);
+		if (array_key_exists($family, App::$config) && array_key_exists($key, App::$config[$family])) {
+			unset(App::$config[$family][$key]);
+		}
 
 		$ret = q("DELETE FROM config WHERE cat = '%s' AND k = '%s'",
 			dbesc($family),
@@ -153,7 +192,7 @@ class Config {
 	 *  The configuration key to query
 	 * @return mixed
 	 */
-	static private function get_from_storage($family,$key) {
+	private static function get_from_storage($family, $key) {
 		$ret = q("SELECT * FROM config WHERE cat = '%s' AND k = '%s' LIMIT 1",
 			dbesc($family),
 			dbesc($key)
@@ -161,5 +200,4 @@ class Config {
 
 		return $ret;
 	}
-
 }

Zotlabs/Lib/Connect.php

@@ -5,8 +5,7 @@ namespace Zotlabs\Lib;
 use App;
 use Zotlabs\Access\Permissions;
 use Zotlabs\Daemon\Master;
-
-
+use Zotlabs\Lib\Config;
 
 class Connect {
 
@@ -69,7 +68,8 @@ class Connect {
 		$xchan_hash = '';
 		$sql_options = (($protocol) ? " and xchan_network = '" . dbesc($protocol) . "' " : '');
 
-		$r = q("select * from xchan where ( xchan_hash = '%s' or xchan_url = '%s' or xchan_addr = '%s') $sql_options ",
+		// We need both, the xchan and the hubloc here hence use JOIN instead of LEFT JOIN
+		$r = q("SELECT * FROM xchan JOIN hubloc ON xchan_hash = hubloc_hash WHERE ( xchan_hash = '%s' or xchan_url = '%s' or xchan_addr = '%s') $sql_options ORDER BY hubloc_id DESC",
 			dbesc($url),
 			dbesc($url),
 			dbesc($url)
@@ -80,12 +80,13 @@ class Connect {
 			// reset results to the best record or the first if we don't have the best
 			// note: this is a single record and not an array of results
 
-			$r = Libzot::zot_record_preferred($r,'xchan_network');
+			$r = Libzot::zot_record_preferred($r, 'xchan_network');
 
 		}
 
 		$singleton = false;
 		$d = false;
+		$wf = false;
 
 		if (! $r) {
 
@@ -94,7 +95,7 @@ class Connect {
 			$wf = discover_by_webbie($url,$protocol);
 
 			if (! $wf) {
-				$feeds = get_config('system','feed_contacts');
+				$feeds = Config::Get('system','feed_contacts');
 
 				if (($feeds) && (in_array($protocol, [ '', 'feed', 'rss' ]))) {
 					$d = discover_by_url($url);
@@ -108,10 +109,12 @@ class Connect {
 
 		if ($wf || $d) {
 
+			$xchan_hash = (($wf) ? $wf : $url);
+
 			// something was discovered - find the record which was just created.
 
 			$r = q("select * from xchan where ( xchan_hash = '%s' or xchan_url = '%s' or xchan_addr = '%s' ) $sql_options",
-				dbesc(($wf) ? $wf : $url),
+				dbesc($xchan_hash),
 				dbesc($url),
 				dbesc($url)
 			);
@@ -119,7 +122,7 @@ class Connect {
 			// convert to a single record (once again preferring a zot solution in the case of multiples)
 
 			if ($r) {
-				$r = Libzot::zot_record_preferred($r,'xchan_network');
+				$r = Libzot::zot_record_preferred($r, 'xchan_network');
 			}
 		}
 
@@ -146,7 +149,7 @@ class Connect {
 
 		}
 
-		$allowed = ((in_array($xchan['xchan_network'],['rss','zot','zot6'])) ? 1 : 0);
+		$allowed = ((in_array($xchan['xchan_network'],['rss', 'zot6'])) ? 1 : 0);
 
 		$hookdata = ['channel_id' => $uid, 'follow_address' => $url, 'xchan' => $xchan, 'allowed' => $allowed, 'singleton' => 0];
 		call_hooks('follow_allow',$hookdata);
@@ -207,13 +210,13 @@ class Connect {
 		}
 
 		$my_perms = $p['perms'];
-		
+
 		$profile_assign = get_pconfig($uid,'system','profile_assign','');
 
 
 		// See if we are already connected by virtue of having an abook record
 
-		$r = q("select abook_id, abook_xchan, abook_pending, abook_instance from abook 
+		$r = q("select abook_id, abook_xchan, abook_pending, abook_instance from abook
 			where abook_xchan = '%s' and abook_channel = %d limit 1",
 			dbesc($xchan_hash),
 			intval($uid)
@@ -261,7 +264,8 @@ class Connect {
 					'abook_feed'      => intval(($xchan['xchan_network'] === 'rss') ? 1 : 0),
 					'abook_created'   => datetime_convert(),
 					'abook_updated'   => datetime_convert(),
-					'abook_instance'  => (($singleton) ? z_root() : '')
+					'abook_instance'  => (($singleton) ? z_root() : ''),
+					'abook_role'      => get_pconfig($uid, 'system', 'default_permcat', 'default')
 				]
 			);
 		}
@@ -282,7 +286,7 @@ class Connect {
 
 		// fetch the entire record
 
-		$r = q("select abook.*, xchan.* from abook left join xchan on abook_xchan = xchan_hash 
+		$r = q("select abook.*, xchan.* from abook left join xchan on abook_xchan = xchan_hash
 			where abook_xchan = '%s' and abook_channel = %d limit 1",
 			dbesc($xchan_hash),
 			intval($uid)
@@ -300,7 +304,7 @@ class Connect {
 		/** If there is a default group for this channel, add this connection to it */
 
 		if ($default_group) {
-			$g = AccessList::rec_byhash($uid,$default_group);
+			$g = AccessList::by_hash($uid,$default_group);
 			if ($g) {
 				AccessList::member_add($uid,'',$xchan_hash,$g['id']);
 			}

Zotlabs/Lib/Crypto.php

@@ -0,0 +1,211 @@
+<?php
+
+namespace Zotlabs\Lib;
+
+use Exception;
+use Zotlabs\Lib\Config;
+
+class Crypto {
+
+	public static $openssl_algorithms = [
+
+		// zot6 nickname,   opensslname,   keylength, ivlength
+
+		['aes256ctr', 'aes-256-ctr', 32, 16],
+		['camellia256cfb', 'camellia-256-cfb', 32, 16],
+		['cast5cfb', 'cast5-cfb', 16, 8],
+		['aes256cbc', 'aes-256-cbc', 32, 16] // remove after legacy zot has been sunset
+
+	];
+
+	public static function methods() {
+		$ret = [];
+
+		foreach (self::$openssl_algorithms as $ossl) {
+			$ret[] = $ossl[0] . '.oaep';
+		}
+
+		call_hooks('crypto_methods', $ret);
+		return $ret;
+	}
+
+	public static function signing_methods() {
+
+		$ret = ['sha256'];
+		call_hooks('signing_methods', $ret);
+		return $ret;
+
+	}
+
+	public static function new_keypair($bits) {
+
+		$openssl_options = [
+			'digest_alg'       => 'sha1',
+			'private_key_bits' => $bits,
+			'encrypt_key'      => false
+		];
+
+		$conf = Config::Get('system', 'openssl_conf_file');
+
+		if ($conf) {
+			$openssl_options['config'] = $conf;
+		}
+
+		$result = openssl_pkey_new($openssl_options);
+
+		if (empty($result)) {
+			return false;
+		}
+
+		// Get private key
+
+		$response = ['prvkey' => '', 'pubkey' => ''];
+
+		openssl_pkey_export($result, $response['prvkey']);
+
+		// Get public key
+		$pkey               = openssl_pkey_get_details($result);
+		$response['pubkey'] = $pkey["key"];
+
+		return $response;
+
+	}
+
+	public static function sign($data, $key, $alg = 'sha256') {
+
+		if (!$key) {
+			return false;
+		}
+
+		$sig = '';
+		openssl_sign($data, $sig, $key, $alg);
+		return $sig;
+	}
+
+	public static function verify($data, $sig, $key, $alg = 'sha256') {
+
+		if (!$key) {
+			return false;
+		}
+
+		if (!$alg) {
+			$alg = 'sha256';
+		}
+
+		try {
+			$verify = openssl_verify($data, $sig, $key, $alg);
+		} catch (Exception $e) {
+			$verify = (-1);
+		}
+
+		if ($verify === (-1)) {
+			while ($msg = openssl_error_string()) {
+				logger('openssl_verify: ' . $msg, LOGGER_NORMAL, LOG_ERR);
+			}
+			btlogger('openssl_verify: key: ' . $key, LOGGER_DEBUG, LOG_ERR);
+		}
+
+		return (($verify > 0) ? true : false);
+	}
+
+	public static function encapsulate($data, $pubkey, $alg) {
+
+		if (!($alg && $pubkey)) {
+			return $data;
+		}
+
+		$alg_base = $alg;
+		$padding  = OPENSSL_PKCS1_PADDING;
+
+		$exts = explode('.', $alg);
+		if (count($exts) > 1) {
+			switch ($exts[1]) {
+				case 'oaep':
+					$padding = OPENSSL_PKCS1_OAEP_PADDING;
+					break;
+				default:
+					break;
+			}
+			$alg_base = $exts[0];
+		}
+
+		$method = null;
+
+		foreach (self::$openssl_algorithms as $ossl) {
+			if ($ossl[0] === $alg_base) {
+				$method = $ossl;
+				break;
+			}
+		}
+
+		if ($method) {
+			$result = ['encrypted' => true];
+
+			$key = openssl_random_pseudo_bytes(256);
+			$iv  = openssl_random_pseudo_bytes(256);
+
+			$key1 = substr($key, 0, $method[2]);
+			$iv1  = substr($iv, 0, $method[3]);
+
+			$result['data'] = base64url_encode(openssl_encrypt($data, $method[1], $key1, OPENSSL_RAW_DATA, $iv1), true);
+
+			openssl_public_encrypt($key, $k, $pubkey, $padding);
+			openssl_public_encrypt($iv, $i, $pubkey, $padding);
+
+			$result['alg'] = $alg;
+			$result['key'] = base64url_encode($k, true);
+			$result['iv']  = base64url_encode($i, true);
+			return $result;
+
+		}
+		else {
+			$x = ['data' => $data, 'pubkey' => $pubkey, 'alg' => $alg, 'result' => $data];
+			call_hooks('crypto_encapsulate', $x);
+			return $x['result'];
+		}
+	}
+
+	public static function unencapsulate($data, $prvkey) {
+
+		if (!(is_array($data) && array_key_exists('encrypted', $data) && array_key_exists('alg', $data) && $data['alg'])) {
+			logger('not encrypted');
+
+			return $data;
+		}
+
+		$alg_base = $data['alg'];
+		$padding  = OPENSSL_PKCS1_PADDING;
+
+		$exts = explode('.', $data['alg']);
+		if (count($exts) > 1) {
+			switch ($exts[1]) {
+				case 'oaep':
+					$padding = OPENSSL_PKCS1_OAEP_PADDING;
+					break;
+				default:
+					break;
+			}
+			$alg_base = $exts[0];
+		}
+
+		$method = null;
+
+		foreach (self::$openssl_algorithms as $ossl) {
+			if ($ossl[0] === $alg_base) {
+				$method = $ossl;
+				break;
+			}
+		}
+
+		if ($method) {
+			openssl_private_decrypt(base64url_decode($data['key']), $k, $prvkey, $padding);
+			openssl_private_decrypt(base64url_decode($data['iv']), $i, $prvkey, $padding);
+			return openssl_decrypt(base64url_decode($data['data']), $method[1], substr($k, 0, $method[2]), OPENSSL_RAW_DATA, substr($i, 0, $method[3]));
+		}
+		else {
+			$x = ['data' => $data, 'prvkey' => $prvkey, 'alg' => $data['alg'], 'result' => $data];
+			call_hooks('crypto_unencapsulate', $x);
+			return $x['result'];
+		}
+	}
+}

Zotlabs/Lib/DB_Upgrade.php

@@ -2,6 +2,7 @@
 
 namespace Zotlabs\Lib;
 
+use Zotlabs\Lib\Config;
 
 class DB_Upgrade {
 
@@ -13,9 +14,9 @@ class DB_Upgrade {
 		$this->config_name = 'db_version';
 		$this->func_prefix = '_';
 
-		$build = get_config('system', 'db_version', 0);
+		$build = Config::Get('system', 'db_version', 0);
 		if(! intval($build))
-			$build = set_config('system', 'db_version', $db_revision);
+			$build = Config::Set('system', 'db_version', $db_revision);
 
 		if($build == $db_revision) {
 			// Nothing to be done.
@@ -27,7 +28,7 @@ class DB_Upgrade {
 				logger('Critical: check_config unable to determine database schema version');
 				return;
 			}
-		
+
 			$current = intval($db_revision);
 
 			if($stored < $current) {
@@ -38,7 +39,7 @@ class DB_Upgrade {
 				for($x = $stored + 1; $x <= $current; $x ++) {
 					$s = '_' . $x;
 					$cls = '\\Zotlabs\Update\\' . $s ;
-					if(! class_exists($cls)) {					
+					if(! class_exists($cls)) {
 						return;
 					}
 
@@ -52,10 +53,10 @@ class DB_Upgrade {
 
 					Config::Load('database');
 
-					if(get_config('database', $s))
+					if(Config::Get('database', $s))
 						break;
-					set_config('database',$s, '1');
-					
+					Config::Set('database',$s, '1');
+
 
 					$c =  new $cls();
 
@@ -65,10 +66,10 @@ class DB_Upgrade {
 
 
 						$source = t('Source code of failed update: ') . "\n\n" . @file_get_contents('Zotlabs/Update/' . $s . '.php');
-												
+
 
 						// Prevent sending hundreds of thousands of emails by creating
-						// a lockfile.  
+						// a lockfile.
 
 						$lockfile = 'store/[data]/mailsent';
 
@@ -77,7 +78,7 @@ class DB_Upgrade {
 						@unlink($lockfile);
 						//send the administrator an e-mail
 						file_put_contents($lockfile, $x);
-							
+
 						$r = q("select account_language from account where account_email = '%s' limit 1",
 							dbesc(\App::$config['system']['admin_email'])
 						);
@@ -86,7 +87,7 @@ class DB_Upgrade {
 							[
 								'toEmail'        => \App::$config['system']['admin_email'],
 								'messageSubject' => sprintf( t('Update Error at %s'), z_root()),
-								'textVersion'    => replace_macros(get_intltext_template('update_fail_eml.tpl'), 
+								'textVersion'    => replace_macros(get_intltext_template('update_fail_eml.tpl'),
 									[
 										'$sitename' => \App::$config['system']['sitename'],
 										'$siteurl' =>  z_root(),
@@ -104,11 +105,11 @@ class DB_Upgrade {
 						pop_lang();
 					}
 					else {
-						set_config('database',$s, 'success');
+						Config::Set('database',$s, 'success');
 					}
 				}
 			}
-			set_config('system', 'db_version', $db_revision);
+			Config::Set('system', 'db_version', $db_revision);
 		}
 	}
-}
+}

Zotlabs/Lib/DReport.php

@@ -1,28 +1,33 @@
 <?php
 namespace Zotlabs\Lib;
 
+use Zotlabs\Lib\Config;
+
 class DReport {
 
 	private $location;
 	private $sender;
 	private $recipient;
+	private $name;
 	private $message_id;
+	private $message_uuid;
 	private $status;
 	private $date;
 
-	function __construct($location,$sender,$recipient,$message_id,$status = 'deliver') {
-		$this->location   = $location;
-		$this->sender     = $sender;
-		$this->recipient  = $recipient;
-		$this->name       = EMPTY_STR;
-		$this->message_id = $message_id;
-		$this->status     = $status;
-		$this->date       = datetime_convert();
+	function __construct($location, $sender, $recipient, $message_id, $message_uuid = '', $status = 'deliver') {
+		$this->location     = $location;
+		$this->sender       = $sender;
+		$this->recipient    = $recipient;
+		$this->name         = EMPTY_STR;
+		$this->message_id   = $message_id;
+		$this->message_uuid = $message_uuid;
+		$this->status       = $status;
+		$this->date         = datetime_convert();
 	}
 
 	function update($status) {
-		$this->status     = $status;
-		$this->date       = datetime_convert();
+		$this->status = $status;
+		$this->date   = datetime_convert();
 	}
 
 	function set_name($name) {
@@ -35,24 +40,26 @@ class DReport {
 
 
 	function set($arr) {
-		$this->location   = $arr['location'];
-		$this->sender     = $arr['sender'];
-		$this->recipient  = $arr['recipient'];
-		$this->name       = $arr['name'];
-		$this->message_id = $arr['message_id'];
-		$this->status     = $arr['status'];
-		$this->date       = $arr['date'];
+		$this->location     = $arr['location'];
+		$this->sender       = $arr['sender'];
+		$this->recipient    = $arr['recipient'];
+		$this->name         = $arr['name'];
+		$this->message_id   = $arr['message_id'];
+		$this->message_uuid = $arr['message_uuid'] ?? '';
+		$this->status       = $arr['status'];
+		$this->date         = $arr['date'];
 	}
 
 	function get() {
 		return array(
-			'location'   => $this->location,
-			'sender'     => $this->sender,
-			'recipient'  => $this->recipient,
-			'name'       => $this->name,
-			'message_id' => $this->message_id,
-			'status'     => $this->status,
-			'date'       => $this->date
+			'location'     => $this->location,
+			'sender'       => $this->sender,
+			'recipient'    => $this->recipient,
+			'name'         => $this->name,
+			'message_id'   => $this->message_id,
+			'message_uuid' => $this->message_uuid,
+			'status'       => $this->status,
+			'date'         => $this->date
 		);
 	}
 
@@ -65,7 +72,7 @@ class DReport {
 
 	static function is_storable($dr) {
 
-		if(get_config('system', 'disable_dreport'))
+		if(Config::Get('system', 'disable_dreport'))
 			return false;
 
 		/**
@@ -87,27 +94,13 @@ class DReport {
 
 		// Is the sender one of our channels?
 
-		$c = q("select channel_id from channel where channel_hash = '%s' or channel_portable_id = '%s' limit 1",
-			dbesc($dr['sender']),
+		$c = q("select channel_id from channel where channel_hash = '%s' limit 1",
 			dbesc($dr['sender'])
 		);
 
 		if(! $c)
 			return false;
 
-		// legacy zot recipients add a space and their name to the xchan. remove it if true.
-
-		$legacy_recipient = strpos($dr['recipient'], ' ');
-		if($legacy_recipient !== false) {
-			$legacy_recipient_parts = explode(' ', $dr['recipient'], 2);
-			$rxchan = $legacy_recipient_parts[0];
-		}
-		else {
-			$rxchan = $dr['recipient'];
-		}
-
-
-
 		// is the recipient one of our connections, or do we want to store every report?
 
 		$pcf = get_pconfig($c[0]['channel_id'],'system','dreport_store_all');
@@ -118,7 +111,7 @@ class DReport {
 		// So if a remote site says they can't find us, that's no big surprise
 		// and just creates a lot of extra report noise
 
-		if(($dr['location'] !== z_root()) && ($dr['sender'] === $rxchan) && ($dr['status'] === 'recipient not found'))
+		if(($dr['location'] !== z_root()) && ($dr['sender'] === $dr['recipient']) && ($dr['status'] === 'recipient not found'))
 			return false;
 
 		// If you have a private post with a recipient list, every single site is going to report
@@ -127,14 +120,14 @@ class DReport {
 		// have a channel on that site.
 
 		$r = q("select hubloc_id from hubloc where hubloc_hash = '%s' and hubloc_url = '%s'",
-			dbesc($rxchan),
+			dbesc($dr['recipient']),
 			dbesc($dr['location'])
 		);
 		if((! $r) && ($dr['status'] === 'recipient_not_found'))
 			return false;
 
 		$r = q("select abook_id from abook where abook_xchan = '%s' and abook_channel = %d limit 1",
-			dbesc($rxchan),
+			dbesc($dr['recipient']),
 			intval($c[0]['channel_id'])
 		);
 		if($r)

Zotlabs/Lib/Enotify.php

@@ -6,6 +6,7 @@ namespace Zotlabs\Lib;
  * @brief File with functions and a class for generating system and email notifications.
  */
 
+use Zotlabs\Lib\Config;
 
 class Enotify {
 
@@ -43,7 +44,7 @@ class Enotify {
 				dbesc($params['to_xchan'])
 			);
 		}
-		if ($x & $y) {
+		if ($x && $y) {
 			$sender = $x[0];
 			$recip = $y[0];
 		} else {
@@ -61,7 +62,7 @@ class Enotify {
 		$product    = t('$projectname'); // PLATFORM_NAME;
 		$siteurl    = z_root();
 		$thanks     = t('Thank You,');
-		$sitename   = get_config('system','sitename');
+		$sitename   = Config::Get('system','sitename');
 		$site_admin = sprintf( t('%s Administrator'), $sitename);
 		$opt_out1   = sprintf( t('This email was sent by %1$s at %2$s.'), t('$Projectname'), \App::get_hostname());
 		$opt_out2   = sprintf( t('To stop receiving these messages, please adjust your Notification Settings at %s'), z_root() . '/settings');
@@ -69,24 +70,24 @@ class Enotify {
 		$sender_name = $product;
 		$hostname = \App::get_hostname();
 		if(strpos($hostname,':'))
-			$hostname = substr($hostname,0,strpos($hostname,':'));
+			$hostname = substr($hostname, 0, strpos($hostname,':'));
 
 		// Do not translate 'noreply' as it must be a legal 7-bit email address
 
-		$reply_email = get_config('system','reply_address');
+		$reply_email = Config::Get('system', 'reply_address');
 		if(! $reply_email)
 			$reply_email = 'noreply' . '@' . $hostname;
 
-		$sender_email = get_config('system','from_email');
+		$sender_email = Config::Get('system', 'from_email');
 		if(! $sender_email)
 			$sender_email = 'Administrator' . '@' . $hostname;
 
-		$sender_name = get_config('system','from_email_name');
+		$sender_name = Config::Get('system', 'from_email_name');
 		if(! $sender_name)
 			$sender_name = \Zotlabs\Lib\System::get_site_name();
 
 
-		$additional_mail_header = "";
+		$additional_mail_header = '';
 
 		if(array_key_exists('item', $params)) {
 			require_once('include/conversation.php');
@@ -114,38 +115,42 @@ class Enotify {
 		}
 
 
-	$always_show_in_notices = get_pconfig($recip['channel_id'],'system','always_show_in_notices');
-	$vnotify = get_pconfig($recip['channel_id'],'system','vnotify');
+	$always_show_in_notices = get_pconfig($recip['channel_id'], 'system', 'always_show_in_notices');
+	$vnotify = get_pconfig($recip['channel_id'], 'system', 'vnotify');
 
 	$salutation = $recip['channel_name'];
 
 	// e.g. "your post", "David's photo", etc.
 	$possess_desc = t('%s <!item_type!>');
 
-	if ($params['type'] == NOTIFY_MAIL) {
-		logger('notification: mail');
-		$subject = 	sprintf( t('[$Projectname:Notify] New mail received at %s'),$sitename);
+	$parent_mid = '';
+	$parent_item = [];
 
-		$preamble = sprintf( t('%1$s sent you a new private message at %2$s.'), $sender['xchan_name'],$sitename);
-		$epreamble = sprintf( t('%1$s sent you %2$s.'),'[zrl=' . $sender['xchan_url'] . ']' . $sender['xchan_name'] . '[/zrl]', '[zrl=$itemlink]' . t('a private message') . '[/zrl]');
-		$sitelink = t('Please visit %s to view and/or reply to your private messages.');
-		$tsitelink = sprintf( $sitelink, $siteurl . '/mail/' . $params['item']['id'] );
-		$hsitelink = sprintf( $sitelink, '<a href="' . $siteurl . '/mail/' . $params['item']['id'] . '">' . $sitename . '</a>');
-		$itemlink = $siteurl . '/mail/' . $params['item']['id'];
+// @@TODO: consider using switch instead of those elseif
+	if (isset($params['type']) && $params['type'] == NOTIFY_MAIL) {
+		logger('notification: mail');
+		$subject = 	sprintf( t('[$Projectname:Notify] New direct message received at %s'), $sitename);
+
+		$preamble = sprintf( t('%1$s sent you a new direct message at %2$s'), $sender['xchan_name'], $sitename);
+		$epreamble = sprintf( t('%1$s sent you %2$s.'), '[zrl=' . $sender['xchan_url'] . ']' . $sender['xchan_name'] . '[/zrl]', '[zrl=$itemlink]' . t('a direct message') . '[/zrl]');
+		$sitelink = t('Please visit %s to view and/or reply to your direct messages.');
+		$tsitelink = sprintf( $sitelink, $siteurl . '/hq/' . gen_link_id($params['item']['mid']));
+		$hsitelink = sprintf( $sitelink, '<a href="' . $siteurl . '/hq/' . gen_link_id($params['item']['mid']) . '">' . $sitename . '</a>');
+		$itemlink = $siteurl . '/hq/' . gen_link_id($params['item']['mid']);
 	}
 
-	if ($params['type'] == NOTIFY_COMMENT) {
+	elseif (isset($params['type']) && $params['type'] === NOTIFY_COMMENT) {
 		//logger("notification: params = " . print_r($params, true), LOGGER_DEBUG);
 
 		$moderated = (($params['item']['item_blocked'] == ITEM_MODERATED) ? true : false);
 
 		$itemlink = $params['link'];
 
-		$action = t('commented on');
+		$action = (($moderated) ? t('requested to comment on') : t('commented on'));
 
 		if(array_key_exists('item',$params)) {
 
-			if(in_array($params['item']['verb'], [ACTIVITY_LIKE, ACTIVITY_DISLIKE])) {
+			if(in_array($params['item']['verb'], ['Like', 'Dislike', ACTIVITY_LIKE, ACTIVITY_DISLIKE, ACTIVITY_SHARE])) {
 
 				if(! $always_show_in_notices || !($vnotify & VNOTIFY_LIKE)) {
 					logger('notification: not a visible activity. Ignoring.');
@@ -153,11 +158,14 @@ class Enotify {
 					return;
 				}
 
-				if(activity_match($params['verb'], ACTIVITY_LIKE))
-					$action = t('liked');
+				if(activity_match($params['verb'], ['Like', ACTIVITY_LIKE]))
+					$action = (($moderated) ? t('requested to like') : t('liked'));
 
-				if(activity_match($params['verb'], ACTIVITY_DISLIKE))
-					$action = t('disliked');
+				if(activity_match($params['verb'], ['Dislike', ACTIVITY_DISLIKE]))
+					$action = (($moderated) ? t('requested to dislike') : t('disliked'));
+
+				if(activity_match($params['verb'], ACTIVITY_SHARE))
+					$action = t('repeated');
 
 			}
 
@@ -166,12 +174,11 @@ class Enotify {
 
 		}
 
-		$parent_mid = $params['parent_mid'];
+		$parent_mid = $params['parent_mid'] ?? '';
 
 		// Check to see if there was already a notify for this post.
 		// If so don't create a second notification
 
-		$p = null;
 		$p = q("select id from notify where link = '%s' and uid = %d limit 1",
 			dbesc($params['link']),
 			intval($recip['channel_id'])
@@ -196,6 +203,7 @@ class Enotify {
 
 		xchan_query($p);
 
+//@@FIXME $p can be null (line 188)
 		$item_post_type = item_post_type($p[0]);
 //		$private = $p[0]['item_private'];
 		$parent_id = $p[0]['id'];
@@ -237,7 +245,7 @@ class Enotify {
 			$subject = sprintf( t('[$Projectname:Notify] Moderated Comment to conversation #%1$d by %2$s'), $parent_id, $sender['xchan_name']);
 		else
 			$subject = sprintf( t('[$Projectname:Notify] Comment to conversation #%1$d by %2$s'), $parent_id, $sender['xchan_name']);
-		$preamble = sprintf( t('%1$s commented on an item/conversation you have been following.'), $sender['xchan_name']);
+		$preamble = sprintf( t('%1$s commented on an item/conversation you have been following'), $sender['xchan_name']);
 		$epreamble = $dest_str;
 
 		$sitelink = t('Please visit %s to view and/or reply to the conversation.');
@@ -250,25 +258,24 @@ class Enotify {
 
 	}
 
-	if ($params['type'] == NOTIFY_LIKE) {
+	elseif (isset($params['type']) && $params['type'] === NOTIFY_LIKE) {
 //		logger("notification: params = " . print_r($params, true), LOGGER_DEBUG);
 
 		$itemlink =  $params['link'];
 
-		if (array_key_exists('item',$params) && (! activity_match($params['item']['verb'],ACTIVITY_LIKE))) {
-			if(! $always_show_in_notices  || !($vnotify & VNOTIFY_LIKE)) {
+		if (array_key_exists('item',$params) && (activity_match($params['item']['verb'], ['Like', 'Dislike', ACTIVITY_LIKE, ACTIVITY_DISLIKE]))) {
+			if(! $always_show_in_notices  || !($vnotify & VNOTIFY_LIKE) || !feature_enabled($recip['channel_id'], 'dislike')) {
 				logger('notification: not a visible activity. Ignoring.');
 				pop_lang();
 				return;
 			}
 		}
 
-		$parent_mid = $params['parent_mid'];
+		$parent_mid = $params['parent_mid'] ?? '';
 
 		// Check to see if there was already a notify for this post.
 		// If so don't create a second notification
 
-		$p = null;
 		$p = q("select id from notify where link = '%s' and uid = %d limit 1",
 			dbesc($params['link']),
 			intval($recip['channel_id'])
@@ -291,8 +298,12 @@ class Enotify {
 			);
 		}
 
-		xchan_query($p);
+		if (!$p) {
+			pop_lang();
+			return;
+		}
 
+		xchan_query($p);
 
 		$item_post_type = item_post_type($p[0]);
 //		$private = $p[0]['item_private'];
@@ -300,13 +311,23 @@ class Enotify {
 
 		$parent_item = $p[0];
 
+		//$verb = ((activity_match($params['item']['verb'], ACTIVITY_DISLIKE)) ? t('disliked') : t('liked'));
+		$moderated = (($params['item']['item_blocked'] == ITEM_MODERATED) ? true : false);
+
+		if(activity_match($params['item']['verb'], ['Like', ACTIVITY_LIKE]))
+			$verb = (($moderated) ? t('requested to like') : t('liked'));
+
+		if(activity_match($params['item']['verb'], ['Dislike', ACTIVITY_DISLIKE]))
+			$verb = (($moderated) ? t('requested to dislike') : t('disliked'));
 
 		// "your post"
-		if($p[0]['owner']['xchan_name'] == $p[0]['author']['xchan_name'] && intval($p[0]['item_wall']))
-			$dest_str = sprintf(t('%1$s liked [zrl=%2$s]your %3$s[/zrl]'),
+		if($p[0]['owner']['xchan_name'] === $p[0]['author']['xchan_name'] && intval($p[0]['item_wall']))
+			$dest_str = sprintf(t('%1$s %2$s [zrl=%3$s]your %4$s[/zrl]'),
 				'[zrl=' . $sender['xchan_url'] . ']' . $sender['xchan_name'] . '[/zrl]',
+				$verb,
 				$itemlink,
-				$item_post_type);
+				$item_post_type
+			);
 		else {
 			pop_lang();
 			return;
@@ -318,7 +339,7 @@ class Enotify {
 		// differents subjects for messages on the same thread.
 
 		$subject = sprintf( t('[$Projectname:Notify] Like received to conversation #%1$d by %2$s'), $parent_id, $sender['xchan_name']);
-		$preamble = sprintf( t('%1$s liked an item/conversation you created.'), $sender['xchan_name']);
+		$preamble = sprintf( t('%1$s liked an item/conversation you created'), $sender['xchan_name']);
 		$epreamble = $dest_str;
 
 		$sitelink = t('Please visit %s to view and/or reply to the conversation.');
@@ -328,7 +349,7 @@ class Enotify {
 
 
 
-	if($params['type'] == NOTIFY_WALL) {
+	elseif(isset($params['type']) && $params['type'] === NOTIFY_WALL) {
 		$subject = sprintf( t('[$Projectname:Notify] %s posted to your profile wall') , $sender['xchan_name']);
 
 		$preamble = sprintf( t('%1$s posted to your profile wall at %2$s') , $sender['xchan_name'], $sitename);
@@ -343,9 +364,8 @@ class Enotify {
 		$itemlink =  $params['link'];
 	}
 
-	if ($params['type'] == NOTIFY_TAGSELF) {
+	elseif (isset($params['type']) && $params['type'] === NOTIFY_TAGSELF) {
 
-		$p = null;
 		$p = q("select id from notify where link = '%s' and uid = %d limit 1",
 			dbesc($params['link']),
 			intval($recip['channel_id'])
@@ -368,7 +388,7 @@ class Enotify {
 		$itemlink =  $params['link'];
 	}
 
-	if ($params['type'] == NOTIFY_POKE) {
+	elseif (isset($params['type']) && $params['type'] === NOTIFY_POKE) {
 		$subject =	sprintf( t('[$Projectname:Notify] %1$s poked you') , $sender['xchan_name']);
 		$preamble = sprintf( t('%1$s poked you at %2$s') , $sender['xchan_name'], $sitename);
 		$epreamble = sprintf( t('%1$s [zrl=%2$s]poked you[/zrl].') ,
@@ -385,7 +405,7 @@ class Enotify {
 		$itemlink =  $params['link'];
 	}
 
-	if ($params['type'] == NOTIFY_TAGSHARE) {
+	elseif (isset($params['type']) && $params['type'] === NOTIFY_TAGSHARE) {
 		$subject =	sprintf( t('[$Projectname:Notify] %s tagged your post') , $sender['xchan_name']);
 		$preamble = sprintf( t('%1$s tagged your post at %2$s'),$sender['xchan_name'], $sitename);
 		$epreamble = sprintf( t('%1$s tagged [zrl=%2$s]your post[/zrl]') ,
@@ -398,7 +418,7 @@ class Enotify {
 		$itemlink =  $params['link'];
 	}
 
-	if ($params['type'] == NOTIFY_INTRO) {
+	elseif (isset($params['type']) && $params['type'] === NOTIFY_INTRO) {
 		$subject = sprintf( t('[$Projectname:Notify] Introduction received'));
 		$preamble = sprintf( t('You\'ve received an new connection request from \'%1$s\' at %2$s'), $sender['xchan_name'], $sitename);
 		$epreamble = sprintf( t('You\'ve received [zrl=%1$s]a new connection request[/zrl] from %2$s.'),
@@ -412,7 +432,7 @@ class Enotify {
 		$itemlink = $params['link'];
 	}
 
-	if ($params['type'] == NOTIFY_SUGGEST) {
+	elseif (isset($params['type']) && $params['type'] === NOTIFY_SUGGEST) {
 		$subject = sprintf( t('[$Projectname:Notify] Friend suggestion received'));
 		$preamble = sprintf( t('You\'ve received a friend suggestion from \'%1$s\' at %2$s'), $sender['xchan_name'], $sitename);
 		$epreamble = sprintf( t('You\'ve received [zrl=%1$s]a friend suggestion[/zrl] for %2$s from %3$s.'),
@@ -430,11 +450,11 @@ class Enotify {
 		$itemlink =  $params['link'];
 	}
 
-	if ($params['type'] == NOTIFY_CONFIRM) {
+	elseif (isset($params['type']) && $params['type'] === NOTIFY_CONFIRM) {
 		// ?
 	}
 
-	if ($params['type'] == NOTIFY_SYSTEM) {
+	elseif (isset($params['type']) && $params['type'] === NOTIFY_SYSTEM) {
 		// ?
 	}
 
@@ -467,6 +487,8 @@ class Enotify {
 
 	require_once('include/html2bbcode.php');
 
+/*
+
 	do {
 		$dups = false;
 		$hash = random_string();
@@ -475,10 +497,12 @@ class Enotify {
 		if ($r)
 			$dups = true;
 	} while ($dups === true);
+*/
 
 
-	$datarray = array();
-	$datarray['hash']   = $hash;
+
+	$datarray = [];
+	$datarray['hash']   = $params['item']['uuid'] ?? new_uuid();
 	$datarray['sender_hash'] = $sender['xchan_hash'];
 	$datarray['xname']   = $sender['xchan_name'];
 	$datarray['url']    = $sender['xchan_url'];
@@ -489,12 +513,13 @@ class Enotify {
 	$datarray['link']   = $itemlink;
 	$datarray['parent'] = $parent_mid;
 	$datarray['parent_item'] = $parent_item;
-	$datarray['ntype']   = $params['type'];
-	$datarray['verb']   = $params['verb'];
-	$datarray['otype']  = $params['otype'];
+	$datarray['ntype']   = $params['type'] ?? 0;
+	$datarray['verb']   = $params['verb'] ?? '';
+	$datarray['otype']  = $params['otype'] ?? '';
  	$datarray['abort']  = false;
+	$datarray['seen'] = 0;
 
-	$datarray['item'] = $params['item'];
+	$datarray['item'] = $params['item'] ?? [];
 
 	call_hooks('enotify_store', $datarray);
 
@@ -505,7 +530,6 @@ class Enotify {
 
 
 	// create notification entry in DB
-	$seen = 0;
 
 	// Mark some notifications as seen right away
 	// Note! The notification have to be created, because they are used to send emails
@@ -514,8 +538,8 @@ class Enotify {
 	// (probably would be better that way)
 
 	if (!$always_show_in_notices) {
-		if (($params['type'] == NOTIFY_WALL) || ($params['type'] == NOTIFY_MAIL) || ($params['type'] == NOTIFY_INTRO)) {
-			$seen = 1;
+		if (($params['type'] === NOTIFY_WALL) || ($params['type'] === NOTIFY_MAIL) || ($params['type'] === NOTIFY_INTRO)) {
+			$datarray['seen'] = 1;
 		}
 	}
 
@@ -531,14 +555,15 @@ class Enotify {
 		intval($datarray['uid']),
 		dbesc($datarray['link']),
 		dbesc($datarray['parent']),
-		intval($seen),
+		intval($datarray['seen']),
 		intval($datarray['ntype']),
 		dbesc($datarray['verb']),
 		dbesc($datarray['otype'])
 	);
 
-	$r = q("select id from notify where hash = '%s' and uid = %d limit 1",
-		dbesc($hash),
+	$r = q("select id from notify where hash = '%s' and ntype = %d and uid = %d limit 1",
+		dbesc($datarray['hash']),
+		intval($datarray['ntype']),
 		intval($recip['channel_id'])
 	);
 	if ($r) {
@@ -550,12 +575,12 @@ class Enotify {
 	}
 
 	$itemlink = z_root() . '/notify/view/' . $notify_id;
-	$msg = str_replace('$itemlink',$itemlink,$epreamble);
+	$msg = str_replace('$itemlink', $itemlink, $epreamble);
 
 	// wretched hack, but we don't want to duplicate all the preamble variations and we also don't want to screw up a translation
 
 	if ((\App::$language === 'en' || (! \App::$language)) && strpos($msg,', '))
-		$msg = substr($msg,strpos($msg,', ')+1);
+		$msg = substr($msg, strpos($msg,', ')+1);
 
 	$datarray['id'] = $notify_id;
 	$datarray['msg'] = $msg;
@@ -575,7 +600,7 @@ class Enotify {
 
 		logger('notification: sending notification email');
 
-		$hn = get_pconfig($recip['channel_id'],'system','email_notify_host');
+		$hn = get_pconfig($recip['channel_id'], 'system', 'email_notify_host');
 		if($hn && (! stristr(\App::get_hostname(),$hn))) {
 			// this isn't the email notification host
 			pop_lang();
@@ -584,7 +609,7 @@ class Enotify {
 
 		$textversion = strip_tags(html_entity_decode(bbcode(stripslashes(str_replace(array("\\r", "\\n"), array( "", "\n"), $body))),ENT_QUOTES,'UTF-8'));
 
-		$htmlversion = bbcode(stripslashes(str_replace(array("\\r","\\n"), array("","<br />\n"),$body)));
+		$htmlversion = bbcode(stripslashes(str_replace(array("\\r","\\n"), array('',"<br />\n"),$body)));
 
 
 		// use $_SESSION['zid_override'] to force zid() to use
@@ -601,14 +626,14 @@ class Enotify {
 		unset($_SESSION['zid_override']);
 		unset($_SESSION['zrl_override']);
 
-		$datarray = array();
+		$datarray = [];
 		$datarray['banner']       = $banner;
 		$datarray['product']      = $product;
 		$datarray['preamble']     = $preamble;
 		$datarray['sitename']     = $sitename;
 		$datarray['siteurl']      = $siteurl;
-		$datarray['type']         = $params['type'];
-		$datarray['parent']       = $params['parent_mid'];
+		$datarray['type']         = $params['type'] ?? '';
+		$datarray['parent']       = $params['parent_mid'] ?? '';
 		$datarray['source_name']  = $sender['xchan_name'];
 		$datarray['source_link']  = $sender['xchan_url'];
 		$datarray['source_photo'] = $sender['xchan_photo_s'];
@@ -675,7 +700,6 @@ class Enotify {
 			'$source_name'  => $datarray['source_name'],
 			'$source_link'  => $datarray['source_link'],
 			'$source_photo' => $datarray['source_photo'],
-			'$username'     => $datarray['to_name'],
 			'$hsitelink'    => $datarray['hsitelink'],
 			'$hitemlink'    => $datarray['hitemlink'],
 			'$thanks'       => $datarray['thanks'],
@@ -697,7 +721,6 @@ class Enotify {
 			'$source_name'  => $datarray['source_name'],
 			'$source_link'  => $datarray['source_link'],
 			'$source_photo' => $datarray['source_photo'],
-			'$username'     => $datarray['to_name'],
 			'$tsitelink'    => $datarray['tsitelink'],
 			'$titemlink'    => $datarray['titemlink'],
 			'$thanks'       => $datarray['thanks'],
@@ -758,9 +781,9 @@ class Enotify {
 		$messageSubject = email_header_encode(html_entity_decode($params['messageSubject'],ENT_QUOTES,'UTF-8'),'UTF-8');
 
 		// generate a mime boundary
-		$mimeBoundary = rand(0, 9) . "-"
-				.rand(100000000, 999999999) . "-"
-				.rand(100000000, 999999999) . "=:"
+		$mimeBoundary = rand(0, 9) . '-'
+				.rand(100000000, 999999999) . '-'
+				.rand(100000000, 999999999) . '=:'
 				.rand(10000, 99999);
 
 		// generate a multipart/alternative message header
@@ -768,7 +791,7 @@ class Enotify {
 			$params['additionalMailHeader'] .
 			"From: $fromName <{$params['fromEmail']}>" . PHP_EOL .
 			"Reply-To: $fromName <{$params['replyTo']}>" . PHP_EOL .
-			"MIME-Version: 1.0" . PHP_EOL .
+			'MIME-Version: 1.0' . PHP_EOL .
 			"Content-Type: multipart/alternative; boundary=\"{$mimeBoundary}\"";
 
 		// assemble the final multipart message body with the text and html types included
@@ -776,15 +799,15 @@ class Enotify {
 		$htmlBody = chunk_split(base64_encode($params['htmlVersion']));
 
 		$multipartMessageBody =
-			"--" . $mimeBoundary . PHP_EOL .					// plain text section
-			"Content-Type: text/plain; charset=UTF-8" . PHP_EOL .
-			"Content-Transfer-Encoding: base64" . PHP_EOL . PHP_EOL .
+			'--' . $mimeBoundary . PHP_EOL .					// plain text section
+			'Content-Type: text/plain; charset=UTF-8' . PHP_EOL .
+			'Content-Transfer-Encoding: base64' . PHP_EOL . PHP_EOL .
 			$textBody . PHP_EOL .
-			"--" . $mimeBoundary . PHP_EOL .					// text/html section
-			"Content-Type: text/html; charset=UTF-8" . PHP_EOL .
-			"Content-Transfer-Encoding: base64" . PHP_EOL . PHP_EOL .
+			'--' . $mimeBoundary . PHP_EOL .					// text/html section
+			'Content-Type: text/html; charset=UTF-8' . PHP_EOL .
+			'Content-Transfer-Encoding: base64' . PHP_EOL . PHP_EOL .
 			$htmlBody . PHP_EOL .
-			"--" . $mimeBoundary . "--" . PHP_EOL;					// message ending
+			'--' . $mimeBoundary . '--' . PHP_EOL;					// message ending
 
 		// send the message
 		$res = mail(
@@ -793,7 +816,7 @@ class Enotify {
 			$multipartMessageBody,							// message body
 			$messageHeader									// message headers
 		);
-		logger("notification: enotify::send returns " . (($res) ? 'success' : 'failure'), LOGGER_DEBUG);
+		logger('notification: enotify::send returns ' . (($res) ? 'success' : 'failure'), LOGGER_DEBUG);
 		return $res;
 	}
 
@@ -809,10 +832,10 @@ class Enotify {
 
 		localize_item($item);
 
-		if($item['shortlocalize']) {
+		if(isset($item['shortlocalize'])) {
 			$itemem_text = $item['shortlocalize'];
 		}
-		elseif($item['localize']) {
+		elseif(isset($item['localize'])) {
 			$itemem_text = $item['localize'];
 		}
 		else {
@@ -821,10 +844,6 @@ class Enotify {
 				: (($item['obj_type'] === 'Answer') ? sprintf( t('voted on %s\'s poll'), '[bdi]' . $item['owner']['xchan_name'] . '[/bdi]') : sprintf( t('commented on %s\'s post'), '[bdi]' . $item['owner']['xchan_name'] . '[/bdi]'))
 			);
 
-			if($item['verb'] === ACTIVITY_SHARE) {
-				$itemem_text = sprintf( t('repeated %s\'s post'), '[bdi]' . $item['author']['xchan_name'] . '[/bdi]');
-			}
-
 			if(in_array($item['obj_type'], ['Document', 'Video', 'Audio', 'Image'])) {
 				$itemem_text = t('shared a file with you');
 			}
@@ -833,41 +852,44 @@ class Enotify {
 		$edit = false;
 
 		if($item['edited'] > $item['created']) {
+			$edit = true;
 			if($item['item_thread_top']) {
 				$itemem_text = sprintf( t('edited a post dated %s'), relative_date($item['created']));
-				$edit = true;
 			}
 			else {
 				$itemem_text = sprintf( t('edited a comment dated %s'), relative_date($item['created']));
-				$edit = true;
 			}
 		}
 
 
 		// convert this logic into a json array just like the system notifications
 
-		$who = (($item['verb'] === ACTIVITY_SHARE) ? 'owner' : 'author');
+		$body = html2plain(bbcode($item['body'], ['drop_media' => true, 'tryoembed' => false]), 75, true);
+		if ($body) {
+			$body = htmlentities($body, ENT_QUOTES, 'UTF-8', false);
+		}
 
 		$x = array(
 			'notify_link' => $item['llink'],
-			'name' => $item[$who]['xchan_name'],
-			'addr' => (($item[$who]['xchan_addr']) ? $item[$who]['xchan_addr'] : $item[$who]['xchan_url']),
-			'url' => $item[$who]['xchan_url'],
-			'photo' => $item[$who]['xchan_photo_s'],
+			'name' => $item['author']['xchan_name'],
+			'addr' => $item['author']['xchan_addr'] ? $item['author']['xchan_addr'] : $item['author']['xchan_url'],
+			'url' => $item['author']['xchan_url'],
+			'photo' => $item['author']['xchan_photo_s'],
 			'when' => (($edit) ? datetime_convert('UTC', date_default_timezone_get(), $item['edited']) : datetime_convert('UTC', date_default_timezone_get(), $item['created'])),
 			'class' => (intval($item['item_unseen']) ? 'notify-unseen' : 'notify-seen'),
-			'b64mid' => (($item['mid']) ? 'b64.' . base64url_encode($item['mid']) : ''),
-			//'b64mid' => ((in_array($item['verb'], [ACTIVITY_LIKE, ACTIVITY_DISLIKE])) ? 'b64.' . base64url_encode($item['thr_parent']) : 'b64.' . base64url_encode($item['mid'])),
+		//	'b64mid' => (($item['mid']) ? gen_link_id($item['mid']) : ''),
+			'b64mid' => (($item['uuid']) ? $item['uuid'] : ''),
+			//'b64mid' => ((in_array($item['verb'], [ACTIVITY_LIKE, ACTIVITY_DISLIKE])) ? gen_link_id($item['thr_parent']) : gen_link_id($item['mid'])),
 			'thread_top' => (($item['item_thread_top']) ? true : false),
 			'message' => bbcode(escape_tags($itemem_text)),
-			'body' =>  htmlentities(html2plain(bbcode($item['body']), 75, true), ENT_QUOTES, 'UTF-8', false),
+			'body' => $body,
 			// these are for the superblock addon
-			'hash' => $item[$who]['xchan_hash'],
+			'hash' => $item['author']['xchan_hash'],
 			'uid' => $item['uid'],
 			'display' => true
 		);
 
-		call_hooks('enotify_format',$x);
+		call_hooks('enotify_format', $x);
 		if(! $x['display']) {
 			return [];
 		}
@@ -882,17 +904,14 @@ class Enotify {
 		if(strpos($message, $tt['xname']) === 0)
 			$message = substr($message, strlen($tt['xname']) + 1);
 
-		$mid = basename($tt['link']);
-
-		$b64mid = ((strpos($mid, 'b64.') === 0) ? $mid : 'b64.' . base64url_encode($mid));
 		$x = [
-			'notify_link' => z_root() . '/notify/view/' . $tt['id'],
+			'notify_link' => (($tt['ntype'] === NOTIFY_MAIL) ? $tt['link'] : z_root() . '/notify/view/' . $tt['id']),
 			'name' => $tt['xname'],
 			'url' => $tt['url'],
 			'photo' => $tt['photo'],
 			'when' => datetime_convert('UTC', date_default_timezone_get(), $tt['created']),
 			'hclass' => (($tt['seen']) ? 'notify-seen' : 'notify-unseen'),
-			'b64mid' => (($tt['otype'] == 'item') ? $b64mid : ''),
+			'b64mid' => (($tt['otype'] == 'item') ? $tt['hash'] : ''),
 			'notify_id' => (($tt['otype'] == 'item') ? $tt['id'] : ''),
 			'message' => $message
 		];
@@ -903,8 +922,8 @@ class Enotify {
 
 	static public function format_intros($rr) {
 
-		$x = [
-			'notify_link' => z_root() . '/connections/ifpending',
+		return [
+			'notify_link' => z_root() . '/connections#' . $rr['abook_id'],
 			'name' => $rr['xchan_name'],
 			'addr' => $rr['xchan_addr'],
 			'url' => $rr['xchan_url'],
@@ -914,13 +933,11 @@ class Enotify {
 			'message' => t('added your channel')
 		];
 
-		return $x;
-
 	}
 
 	static public function format_files($rr) {
 
-		$x = [
+		return [
 			'notify_link' => z_root() . '/sharedwithme',
 			'name' => $rr['author']['xchan_name'],
 			'addr' => $rr['author']['xchan_addr'],
@@ -931,13 +948,11 @@ class Enotify {
 			'message' => t('shared a file with you')
 		];
 
-		return $x;
-
 	}
 
 	static public function format_mail($rr) {
 
-		$x = [
+		return [
 			'notify_link' => z_root() . '/mail/' . $rr['id'],
 			'name' => $rr['xchan_name'],
 			'addr' => $rr['xchan_addr'],
@@ -945,11 +960,9 @@ class Enotify {
 			'photo' => $rr['xchan_photo_s'],
 			'when' => datetime_convert('UTC', date_default_timezone_get(), $rr['created']),
 			'hclass' => (intval($rr['mail_seen']) ? 'notify-seen' : 'notify-unseen'),
-			'message' => t('sent you a private message'),
+			'message' => t('sent you a direct message'),
 		];
 
-		return $x;
-
 	}
 
 	static public function format_all_events($rr) {
@@ -959,7 +972,7 @@ class Enotify {
 		$today = ((substr($strt, 0, 10) === datetime_convert('UTC', date_default_timezone_get(), 'now', 'Y-m-d')) ? true : false);
 		$when = day_translate(datetime_convert('UTC', (($rr['adjust']) ? date_default_timezone_get() : 'UTC'), $rr['dtstart'], $bd_format)) . (($today) ?  ' ' . t('[today]') : '');
 
-		$x = [
+		return [
 			'notify_link' => z_root() . '/cdav/calendar/' . $rr['event_hash'],
 			'name'        => $rr['xchan_name'],
 			'addr'        => $rr['xchan_addr'],
@@ -970,23 +983,20 @@ class Enotify {
 			'message'     => t('created an event')
 		];
 
-		return $x;
 
 	}
 
 	static public function format_register($rr) {
 
-		$x = [
+		return [
 			'notify_link' => z_root() . '/admin/accounts',
-			'name' => $rr['account_email'],
-			//'addr' => $rr['account_email'],
+			'name' => $rr['reg_did2'],
+			//'addr' => '',
 			'photo' => z_root() . '/' . get_default_profile_photo(48),
-			'when' => datetime_convert('UTC', date_default_timezone_get(),$rr['account_created']),
+			'when' => datetime_convert('UTC', date_default_timezone_get(),$rr['reg_created']),
 			'hclass' => ('notify-unseen'),
-			'message' => t('requires approval')
+			'message' => t('status verified')
 		];
 
-		return $x;
-
 	}
 }

Zotlabs/Lib/Group.php

@@ -1,405 +0,0 @@
-<?php 
-
-namespace Zotlabs\Lib;
-
-use Zotlabs\Lib\Libsync;
-
-
-class Group {
-	
-	static function add($uid,$name,$public = 0) {
-
-		$ret = false;
-		if(x($uid) && x($name)) {
-			$r = self::byname($uid,$name); // check for dups
-			if($r !== false) {
-
-				// This could be a problem. 
-				// Let's assume we've just created a group which we once deleted
-				// all the old members are gone, but the group remains so we don't break any security
-				// access lists. What we're doing here is reviving the dead group, but old content which
-				// was restricted to this group may now be seen by the new group members. 
-
-				$z = q("SELECT * FROM pgrp WHERE id = %d LIMIT 1",
-					intval($r)
-				);
-				if(($z) && $z[0]['deleted']) {
-					q('UPDATE pgrp SET deleted = 0 WHERE id = %d', intval($z[0]['id']));
-					notice( t('A deleted group with this name was revived. Existing item permissions <strong>may</strong> apply to this group and any future members. If this is not what you intended, please create another group with a different name.') . EOL); 
-				}
-				return true;
-			}
-
-			do {
-				$dups = false;
-				$hash = random_string(32) . str_replace(['<','>'],['.','.'], $name);
-
-				$r = q("SELECT id FROM pgrp WHERE hash = '%s' LIMIT 1", dbesc($hash));
-				if($r)
-					$dups = true;
-			} while($dups == true);
-
-
-			$r = q("INSERT INTO pgrp ( hash, uid, visible, gname )
-				VALUES( '%s', %d, %d, '%s' ) ",
-				dbesc($hash),
-				intval($uid),
-				intval($public),
-				dbesc($name)
-			);
-			$ret = $r;
-		}
-
-		Libsync::build_sync_packet($uid,null,true);
-		return $ret;
-	}
-
-
-	static function remove($uid,$name) {
-		$ret = false;
-		if(x($uid) && x($name)) {
-			$r = q("SELECT id, hash FROM pgrp WHERE uid = %d AND gname = '%s' LIMIT 1",
-				intval($uid),
-				dbesc($name)
-			);
-			if($r) {
-				$group_id = $r[0]['id'];
-				$group_hash = $r[0]['hash'];
-			}
-
-			if(! $group_id)
-				return false;
-
-			// remove group from default posting lists
-			$r = q("SELECT channel_default_group, channel_allow_gid, channel_deny_gid FROM channel WHERE channel_id = %d LIMIT 1",
-			       intval($uid)
-			);
-			if($r) {
-				$user_info = $r[0];
-				$change = false;
-
-				if($user_info['channel_default_group'] == $group_hash) {
-					$user_info['channel_default_group'] = '';
-					$change = true;
-				}
-				if(strpos($user_info['channel_allow_gid'], '<' . $group_hash . '>') !== false) {
-					$user_info['channel_allow_gid'] = str_replace('<' . $group_hash . '>', '', $user_info['channel_allow_gid']);
-					$change = true;
-				}
-				if(strpos($user_info['channel_deny_gid'], '<' . $group_hash . '>') !== false) {
-					$user_info['channel_deny_gid'] = str_replace('<' . $group_hash . '>', '', $user_info['channel_deny_gid']);
-					$change = true;
-				}
-
-				if($change) {
-					q("UPDATE channel SET channel_default_group = '%s', channel_allow_gid = '%s', channel_deny_gid = '%s' 
-						WHERE channel_id = %d",
-						intval($user_info['channel_default_group']),
-						dbesc($user_info['channel_allow_gid']),
-						dbesc($user_info['channel_deny_gid']),
-						intval($uid)
-					);
-				}
-			}
-
-			// remove all members
-			$r = q("DELETE FROM pgrp_member WHERE uid = %d AND gid = %d ",
-				intval($uid),
-				intval($group_id)
-			);
-
-			// remove group
-			$r = q("UPDATE pgrp SET deleted = 1 WHERE uid = %d AND gname = '%s'",
-				intval($uid),
-				dbesc($name)
-			);
-
-			$ret = $r;
-
-		}
-
-		Libsync::build_sync_packet($uid,null,true);
-
-		return $ret;
-	}
-
-
-	static function byname($uid,$name) {
-		if((! $uid) || (! strlen($name)))
-			return false;
-		$r = q("SELECT * FROM pgrp WHERE uid = %d AND gname = '%s' LIMIT 1",
-			intval($uid),
-			dbesc($name)
-		);
-		if($r)
-			return $r[0]['id'];
-		return false;
-	}
-
-
-	static function rec_byhash($uid,$hash) {
-		if((! $uid) || (! strlen($hash)))
-			return false;
-		$r = q("SELECT * FROM pgrp WHERE uid = %d AND hash = '%s' LIMIT 1",
-			intval($uid),
-			dbesc($hash)
-		);
-		if($r)
-			return $r[0];
-		return false;
-	}
-
-
-	static function member_remove($uid,$name,$member) {
-		$gid = self::byname($uid,$name);
-		if(! $gid)
-			return false;
-		if(! ( $uid && $gid && $member))
-			return false;
-		$r = q("DELETE FROM pgrp_member WHERE uid = %d AND gid = %d AND xchan = '%s' ",
-			intval($uid),
-			intval($gid),
-			dbesc($member)
-		);
-
-		Libsync::build_sync_packet($uid,null,true);
-
-		return $r;
-	}
-
-
-	static function member_add($uid,$name,$member,$gid = 0) {
-		if(! $gid)
-			$gid = self::byname($uid,$name);
-		if((! $gid) || (! $uid) || (! $member))
-			return false;
-
-		$r = q("SELECT * FROM pgrp_member WHERE uid = %d AND gid = %d AND xchan = '%s' LIMIT 1",	
-			intval($uid),
-			intval($gid),
-			dbesc($member)
-		);
-		if($r)
-			return true;	// You might question this, but 
-				// we indicate success because the group member was in fact created
-				// -- It was just created at another time
-	 	if(! $r)
-			$r = q("INSERT INTO pgrp_member (uid, gid, xchan)
-				VALUES( %d, %d, '%s' ) ",
-				intval($uid),
-				intval($gid),
-				dbesc($member)
-		);
-
-		Libsync::build_sync_packet($uid,null,true);
-
-		return $r;
-	}
-
-
-	static function members($gid) {
-		$ret = array();
-		if(intval($gid)) {
-			$r = q("SELECT * FROM pgrp_member 
-				LEFT JOIN abook ON abook_xchan = pgrp_member.xchan left join xchan on xchan_hash = abook_xchan
-				WHERE gid = %d AND abook_channel = %d and pgrp_member.uid = %d and xchan_deleted = 0 and abook_self = 0 and abook_blocked = 0 and abook_pending = 0 ORDER BY xchan_name ASC ",
-				intval($gid),
-				intval(local_channel()),
-				intval(local_channel())
-			);
-			if($r)
-				$ret = $r;
-		}
-		return $ret;
-	}
-
-	static function members_xchan($gid) {
-		$ret = [];
-		if(intval($gid)) {
-			$r = q("SELECT xchan FROM pgrp_member WHERE gid = %d AND uid = %d",
-				intval($gid),
-				intval(local_channel())
-			);
-			if($r) {
-				foreach($r as $rr) {
-					$ret[] = $rr['xchan'];
-				}
-			}
-		}
-		return $ret;
-	}
-
-	static function members_profile_xchan($uid,$gid) {
-		$ret = [];
-
-		if(intval($gid)) {
-			$r = q("SELECT abook_xchan as xchan from abook left join profile on abook_profile = profile_guid where profile.id = %d and profile.uid = %d",
-				intval($gid),
-				intval($uid)
-			);
-			if($r) {
-				foreach($r as $rr) {
-					$ret[] = $rr['xchan'];
-				}
-			}
-		}
-		return $ret;
-	}
-
-
-
-
-	static function select($uid,$group = '') {
-	
-		$grps = [];
-		$o = '';
-
-		$r = q("SELECT * FROM pgrp WHERE deleted = 0 AND uid = %d ORDER BY gname ASC",
-			intval($uid)
-		);
-		$grps[] = array('name' => '', 'hash' => '0', 'selected' => '');
-		if($r) {
-			foreach($r as $rr) {
-				$grps[] = array('name' => $rr['gname'], 'id' => $rr['hash'], 'selected' => (($group == $rr['hash']) ? 'true' : ''));
-			}
-
-		}
-		logger('select: ' . print_r($grps,true), LOGGER_DATA);
-
-		$o = replace_macros(get_markup_template('group_selection.tpl'), array(
-			'$label' => t('Add new connections to this privacy group'),
-			'$groups' => $grps 
-		));
-		return $o;
-	}
-
-
-
-
-	static function widget($every="connections",$each="group",$edit = false, $group_id = 0, $cid = '',$mode = 1) {
-
-		$o = '';
-
-		if(! (local_channel() && feature_enabled(local_channel(),'groups'))) {
-			return '';
-		}
-
-		$groups = array();
-
-		$r = q("SELECT * FROM pgrp WHERE deleted = 0 AND uid = %d ORDER BY gname ASC",
-			intval($_SESSION['uid'])
-		);
-		$member_of = array();
-		if($cid) {
-			$member_of = self::containing(local_channel(),$cid);
-		} 
-
-		if($r) {
-			foreach($r as $rr) {
-				$selected = (($group_id == $rr['id']) ? ' group-selected' : '');
-			
-				if ($edit) {
-					$groupedit = [ 'href' => "group/".$rr['id'], 'title' => t('edit') ];
-				} 
-				else {
-					$groupedit = null;
-				}
-			
-				$groups[] = [
-					'id'		=> $rr['id'],
-					'enc_cid'   => base64url_encode($cid),
-					'cid'		=> $cid,
-					'text' 		=> $rr['gname'],
-					'selected' 	=> $selected,
-					'href'		=> (($mode == 0) ? $each.'?f=&gid='.$rr['id'] : $each."/".$rr['id']) . ((x($_GET,'new')) ? '&new=' . $_GET['new'] : '') . ((x($_GET,'order')) ? '&order=' . $_GET['order'] : ''),
-					'edit'		=> $groupedit,
-					'ismember'	=> in_array($rr['id'],$member_of),
-				];
-			}
-		}
-	
-	
-		$tpl = get_markup_template("group_side.tpl");
-		$o = replace_macros($tpl, array(
-			'$title'		=> t('Privacy Groups'),
-			'$edittext'     => t('Edit group'),
-			'$createtext' 	=> t('Add privacy group'),
-			'$ungrouped'    => (($every === 'contacts') ? t('Channels not in any privacy group') : ''),
-			'$groups'		=> $groups,
-			'$add'			=> t('add'),
-		));
-		
-	
-		return $o;
-	}
-
-
-	static function expand($g) {
-		if(! (is_array($g) && count($g)))
-			return array();
-
-		$ret = [];
-		$x   = [];
-
-		// private profile linked virtual groups
-
-		foreach($g as $gv) {
-			if(substr($gv,0,3) === 'vp.') {
-				$profile_hash = substr($gv,3);
-				if($profile_hash) {
-					$r = q("select abook_xchan from abook where abook_profile = '%s'",
-						dbesc($profile_hash)
-					);
-					if($r) {
-						foreach($r as $rv) {
-							$ret[] = $rv['abook_xchan'];
-						}
-					}
-				}
-			}
-			else {
-				$x[] = $gv;
-			}
-		}								 
-
-		if($x) {
-			stringify_array_elms($x,true);
-			$groups = implode(',', $x);
-			if($groups) {
-				$r = q("SELECT xchan FROM pgrp_member WHERE gid IN ( select id from pgrp where hash in ( $groups ))");
-				if($r) {
-					foreach($r as $rr) {
-						$ret[] = $rr['xchan'];
-					}
-				}
-			}
-		}
-		return $ret;
-	}
-
-
-	static function member_of($c) {
-		$r = q("SELECT pgrp.gname, pgrp.id FROM pgrp LEFT JOIN pgrp_member ON pgrp_member.gid = pgrp.id WHERE pgrp_member.xchan = '%s' AND pgrp.deleted = 0 ORDER BY pgrp.gname  ASC ",
-			dbesc($c)
-		);
-
-		return $r;
-
-	}
-
-	static function containing($uid,$c) {
-
-		$r = q("SELECT gid FROM pgrp_member WHERE uid = %d AND pgrp_member.xchan = '%s' ",
-			intval($uid),
-			dbesc($c)
-		);
-
-		$ret = array();
-		if($r) {
-			foreach($r as $rr)
-				$ret[] = $rr['gid'];
-		}
-	
-		return $ret;
-	}
-}

Zotlabs/Lib/Hashpath.php

@@ -0,0 +1,55 @@
+<?php
+namespace Zotlabs\Lib;
+
+/*
+ * Zotlabs\Lib\Hashpath
+ *
+ * Creates hashed directory structures for fast access and resistance to overloading any single directory with files.
+ *
+ * Takes a $hash which could be any string
+ * a $prefix which is where to place the hash directory in the filesystem, default is current directory
+ * use an empty string for $prefix to place hash directories directly off the root directory
+ * an optional $depth and $slice (default is 2) to indicate the hash level
+ * $depth = 1, 256 directories, suitable for < 384K records/files
+ * $depth = 2, 65536 directories, suitable for < 98M records/files
+ * $depth = 3, 16777216 directories, suitable for < 2.5B records/files
+ * ...
+ * The total number of records anticipated divided by the number of hash directories should generally be kept to
+ * less than 1500 entries for optimum performance though this varies by operating system and filesystem type.
+ * ext4 uses 32 bit inode numbers (~4B record limit) so use caution or alternative filesystem types with $depth above 3. 
+ * an optional $mkdir (boolean) to recursively create the directory (ignoring errors) before returning
+ *
+ * examples: for a $hash of 'abcdefg' and prefix of 'path' the following paths are returned for $depth = 1 and $depth = 3
+ *    path/7d/7d1a54127b222502f5b79b5fb0803061152a44f92b37e23c6527baf665d4da9a
+ *    path/7d/1a/54/7d1a54127b222502f5b79b5fb0803061152a44f92b37e23c6527baf665d4da9a
+ *
+ * see also: boot.php:os_mkdir() - here we provide the equivalent of mkdir -p with permissions of 770. 
+ *
+ */
+ 
+class Hashpath {
+
+	static function path($hash, $prefix = '.', $depth = 1, $slice = 2, $mkdir = true, $alg = false) {
+
+		if ($alg)
+			$hash = hash($alg, $hash);
+
+		$start = 0;
+		if ($depth < 1)
+			$depth = 1;
+		$sluglen = $depth * $slice;
+
+		do {
+			$slug	  = substr($hash, $start, $slice);
+			$prefix  .= '/' . $slug;
+			$start	 += $slice;
+			$sluglen -= $slice;
+		}
+		while ($sluglen);
+
+		if ($mkdir)
+			os_mkdir($prefix, STORAGE_DEFAULT_PERMISSIONS, true);
+		
+		return $prefix . '/' . $hash;
+	}
+}

Zotlabs/Lib/IConfig.php

@@ -13,7 +13,7 @@ class IConfig {
 	static public function Get(&$item, $family, $key, $default = false) {
 
 		$is_item = false;
-	
+
 		if(is_array($item)) {
 			$is_item = true;
 			if((! array_key_exists('iconfig',$item)) || (! is_array($item['iconfig'])))
@@ -22,7 +22,7 @@ class IConfig {
 			if(array_key_exists('item_id',$item))
 				$iid = $item['item_id'];
 			else
-				$iid = $item['id'];
+				$iid = $item['id'] ?? 0;
 		}
 		elseif(intval($item))
 			$iid = $item;
@@ -36,7 +36,7 @@ class IConfig {
 					return $c['v'];
 			}
 		}
-		 
+
 		$r = q("select * from iconfig where iid = %d and cat = '%s' and k = '%s' limit 1",
 			intval($iid),
 			dbesc($family),
@@ -63,11 +63,11 @@ class IConfig {
 	 * $value - value of meta variable
 	 * $sharing - boolean (default false); if true the meta information is propagated with the item
 	 *   to other sites/channels, mostly useful when $item is an array and has not yet been stored/delivered.
-	 *   If the meta information is added after delivery and you wish it to be shared, it may be necessary to 
-	 *   alter the item edited timestamp and invoke the delivery process on the updated item. The edited 
+	 *   If the meta information is added after delivery and you wish it to be shared, it may be necessary to
+	 *   alter the item edited timestamp and invoke the delivery process on the updated item. The edited
 	 *   timestamp needs to be altered in order to trigger an item_store_update() at the receiving end.
 	 */
- 
+
 
 	static public function Set(&$item, $family, $key, $value, $sharing = false) {
 
@@ -162,4 +162,4 @@ class IConfig {
 
 	}
 
-}
+}

Zotlabs/Lib/JSalmon.php

@@ -18,7 +18,7 @@ class JSalmon {
 
 		$precomputed = '.' . base64url_encode($data_type,true) . '.YmFzZTY0dXJs.UlNBLVNIQTI1Ng';
 
-		$signature  = base64url_encode(rsa_sign($data . $precomputed, $key), true);
+		$signature  = base64url_encode(Crypto::sign($data . $precomputed, $key), true);
 
 		return ([
 			'signed'    => true,
@@ -52,9 +52,9 @@ class JSalmon {
 			. base64url_encode($x['alg'],true);
 
 		$key = HTTPSig::get_key(EMPTY_STR,'zot6',base64url_decode($x['sigs']['key_id']));
-		 logger('key: ' . print_r($key,true));
+		logger('key: ' . print_r($key,true), LOGGER_DATA);
 		if($key['portable_id'] && $key['public_key']) {
-			if(rsa_verify($signed_data,base64url_decode($x['sigs']['value']),$key['public_key'])) {
+			if(Crypto::verify($signed_data,base64url_decode($x['sigs']['value']),$key['public_key'])) {
 				logger('verified');
 				$ret = [ 'success' => true, 'signer' => $key['portable_id'], 'hubloc' => $key['hubloc'] ];
 			}

Zotlabs/Lib/JcsEddsa2022.php

@@ -0,0 +1,92 @@
+<?php
+
+namespace Zotlabs\Lib;
+
+use Mmccook\JsonCanonicalizator\JsonCanonicalizatorFactory;
+use StephenHill\Base58;
+
+class JcsEddsa2022 {
+
+	public  function  __construct() {
+		return $this;
+	}
+
+	public function sign($data, $channel): array {
+		$base58 = new Base58();
+		$pubkey = (new Multibase())->publicKey($channel['channel_epubkey']);
+		$options = [
+			'type' => 'DataIntegrityProof',
+			'cryptosuite' => 'eddsa-jcs-2022',
+			'created' => datetime_convert('UTC', 'UTC', 'now', ATOM_TIME),
+			'verificationMethod' => channel_url($channel) . '#' . $pubkey,
+			'proofPurpose' => 'assertionMethod',
+		];
+
+		$optionsHash = $this->hash($this->signableOptions($options), true);
+		$dataHash = $this->hash($this->signableData($data), true);
+
+		$options['proofValue'] = 'z' . $base58->encode(sodium_crypto_sign_detached($optionsHash . $dataHash,
+				sodium_base642bin($channel['channel_eprvkey'], SODIUM_BASE64_VARIANT_ORIGINAL_NO_PADDING)));
+
+		return $options;
+	}
+
+	public function verify($data, $publicKey) {
+		$base58 = new Base58();
+		$encodedSignature = $data['proof']['proofValue'] ?? '';
+		if (!str_starts_with($encodedSignature,'z')) {
+			return false;
+		}
+
+		$encodedSignature = substr($encodedSignature, 1);
+		$optionsHash = $this->hash($this->signableOptions($data['proof']), true);
+		$dataHash = $this->hash($this->signableData($data),true);
+
+		try {
+			$result = sodium_crypto_sign_verify_detached($base58->decode($encodedSignature), $optionsHash . $dataHash,
+				(new Multibase())->decode($publicKey, true));
+		}
+		catch (\Exception $e) {
+			logger('verify exception:' .  $e->getMessage());
+		}
+
+		logger('SignatureVerify (eddsa-jcs-2022) ' . (($result) ? 'true' : 'false'));
+
+		return $result;
+	}
+
+	public function signableData($data) {
+		$signableData = [];
+		if ($data) {
+			foreach ($data as $k => $v) {
+				if (!in_array($k, ['proof', 'signature'])) {
+					$signableData[$k] = $v;
+				}
+			}
+		}
+		return $signableData;
+	}
+
+	public function signableOptions($options) {
+		$signableOptions = [];
+
+		if ($options) {
+			foreach ($options as $k => $v) {
+				if ($k !== 'proofValue') {
+					$signableOptions[$k] = $v;
+				}
+			}
+		}
+		return $signableOptions;
+	}
+
+	public function hash($obj, $binary = false) {
+		return hash('sha256', $this->canonicalize($obj), $binary);
+	}
+
+	public function canonicalize($data) {
+		$canonicalization = JsonCanonicalizatorFactory::getInstance();
+		return $canonicalization->canonicalize($data);
+	}
+
+}

Zotlabs/Lib/Keyutils.php

@@ -0,0 +1,99 @@
+<?php
+
+namespace Zotlabs\Lib;
+
+use phpseclib\Crypt\RSA;
+use phpseclib\Math\BigInteger;
+
+/**
+ * Keyutils
+ * Convert RSA keys between various formats
+ */
+class Keyutils {
+
+	/**
+	 * @param string $m modulo
+	 * @param string $e exponent
+	 * @return string
+	 */
+	public static function meToPem($m, $e) {
+
+		$rsa = new RSA();
+		$rsa->loadKey([
+			'e' => new BigInteger($e, 256),
+			'n' => new BigInteger($m, 256)
+		]);
+		return $rsa->getPublicKey();
+
+	}
+
+	/**
+	 * @param string key
+	 * @return string
+	 */
+	public static function rsaToPem($key) {
+
+		$rsa = new RSA();
+		$rsa->setPublicKey($key);
+
+		return $rsa->getPublicKey(RSA::PUBLIC_FORMAT_PKCS8);
+
+	}
+
+	/**
+	 * @param string key
+	 * @return string
+	 */
+	public static function pemToRsa($key) {
+
+		$rsa = new RSA();
+		$rsa->setPublicKey($key);
+
+		return $rsa->getPublicKey(RSA::PUBLIC_FORMAT_PKCS1);
+
+	}
+
+	/**
+	 * @param string $key key
+	 * @param string $m reference modulo
+	 * @param string $e reference exponent
+	 */
+	public static function pemToMe($key, &$m, &$e) {
+
+		$rsa = new RSA();
+		$rsa->loadKey($key);
+		$rsa->setPublicKey();
+
+		$m = $rsa->modulus->toBytes();
+		$e = $rsa->exponent->toBytes();
+
+	}
+
+	/**
+	 * @param string $pubkey
+	 * @return string
+	 */
+	public static function salmonKey($pubkey) {
+		self::pemToMe($pubkey, $m, $e);
+		return 'RSA' . '.' . base64url_encode($m, true) . '.' . base64url_encode($e, true);
+	}
+
+	/**
+	 * @param string $key
+	 * @return string
+	 */
+	public static function convertSalmonKey($key) {
+		if (strstr($key, ','))
+			$rawkey = substr($key, strpos($key, ',') + 1);
+		else
+			$rawkey = substr($key, 5);
+
+		$key_info = explode('.', $rawkey);
+
+		$m = base64url_decode($key_info[1]);
+		$e = base64url_decode($key_info[2]);
+
+		return self::meToPem($m, $e);
+	}
+
+}

Zotlabs/Lib/LDSignatures.php

@@ -12,7 +12,7 @@ class LDSignatures {
 		$ohash = self::hash(self::signable_options($data['signature']));
 		$dhash = self::hash(self::signable_data($data));
 
-		$x = rsa_verify($ohash . $dhash,base64_decode($data['signature']['signatureValue']), $pubkey);
+		$x = Crypto::verify($ohash . $dhash,base64_decode($data['signature']['signatureValue']), $pubkey);
 		logger('LD-verify: ' . intval($x));
 
 		return $x;
@@ -35,11 +35,11 @@ class LDSignatures {
 
 		$ohash = self::hash(self::signable_options($options));
 		$dhash = self::hash(self::signable_data($data));
-		$options['signatureValue'] = base64_encode(rsa_sign($ohash . $dhash,$channel['channel_prvkey']));
+		$options['signatureValue'] = base64_encode(Crypto::sign($ohash . $dhash,$channel['channel_prvkey']));
 
 		$signed = array_merge([
-			'@context' => [ 
-				ACTIVITYSTREAMS_JSONLD_REV, 
+			'@context' => [
+				ACTIVITYSTREAMS_JSONLD_REV,
 				'https://w3id.org/security/v1' ],
 			],$options);
 
@@ -75,22 +75,23 @@ class LDSignatures {
 	}
 
 	static function hash($obj) {
-
-		return hash('sha256',self::normalise($obj));
+		return hash('sha256', self::normalise($obj));
 	}
 
 	static function normalise($data) {
+		$ret = '';
+
 		if(is_string($data)) {
 			$data = json_decode($data);
 		}
 
 		if(! is_object($data))
-			return '';
+			return $ret;
 
 		jsonld_set_document_loader('jsonld_document_loader');
-	
+
 		try {
-			$d = jsonld_normalize($data,[ 'algorithm' => 'URDNA2015', 'format' => 'application/nquads' ]);
+			$ret = jsonld_normalize($data,[ 'algorithm' => 'URDNA2015', 'format' => 'application/nquads' ]);
 		}
 		catch (\Exception $e) {
 			// Don't log the exception - this can exhaust memory
@@ -98,7 +99,7 @@ class LDSignatures {
 			logger('normalise error: ' . print_r($data,true));
 		}
 
-		return $d;
+		return $ret;
 	}
 
 	static function salmon_sign($data,$channel) {
@@ -117,7 +118,7 @@ class LDSignatures {
 
 		$precomputed = '.' . base64url_encode($data_type,false) . '.YmFzZTY0dXJs.UlNBLVNIQTI1Ng==';
 
-		$signature  = base64url_encode(rsa_sign($data . $precomputed,$channel['channel_prvkey']));
+		$signature  = base64url_encode(Crypto::sign($data . $precomputed,$channel['channel_prvkey']));
 
 		return ([
 			'id'          => $arr['id'],

Zotlabs/Lib/Libzotdir.php

@@ -2,6 +2,7 @@
 
 namespace Zotlabs\Lib;
 
+use Zotlabs\Lib\Config;
 use Zotlabs\Lib\Libzot;
 use Zotlabs\Lib\Zotfinger;
 use Zotlabs\Lib\Webfinger;
@@ -20,7 +21,7 @@ class Libzotdir {
 
 	static function find_upstream_directory($dirmode) {
 
-		$preferred = get_config('system','directory_server');
+		$preferred = Config::Get('system','directory_server');
 
 		// Thwart attempts to use a private directory
 
@@ -47,17 +48,17 @@ class Libzotdir {
 
 			$directory_fallback_servers = get_directory_fallback_servers();
 
-			$dirmode = intval(get_config('system','directory_mode'));
+			$dirmode = intval(Config::Get('system','directory_mode'));
 			if ($dirmode == DIRECTORY_MODE_NORMAL) {
 				$toss = mt_rand(0,count($directory_fallback_servers));
 				$preferred = $directory_fallback_servers[$toss];
 				if(! $preferred) {
 					$preferred = DIRECTORY_FALLBACK_MASTER;
 				}
-				set_config('system','directory_server',$preferred);
+				Config::Set('system','directory_server',$preferred);
 			}
 			else {
-				set_config('system','directory_server',z_root());
+				Config::Set('system','directory_server',z_root());
 			}
 		}
 		if($preferred) {
@@ -77,7 +78,7 @@ class Libzotdir {
 
 	static function check_upstream_directory() {
 
-		$directory = get_config('system', 'directory_server');
+		$directory = Config::Get('system', 'directory_server');
 
 		// it's possible there is no directory server configured and the local hub is being used.
 		// If so, default to preserving the absence of a specific server setting.
@@ -94,7 +95,7 @@ class Libzotdir {
 		}
 
 		if (! $isadir)
-			set_config('system', 'directory_server', '');
+			Config::Set('system', 'directory_server', '');
 	}
 
 
@@ -106,7 +107,7 @@ class Libzotdir {
 			$ret = ((array_key_exists($setting,$_SESSION)) ? intval($_SESSION[$setting]) : false);
 
 		if($ret === false)
-			$ret = get_config('directory', $setting);
+			$ret = Config::Get('directory', $setting);
 
 
 		// 'safemode' is the default if there is no observer or no established preference.
@@ -114,7 +115,7 @@ class Libzotdir {
 		if($setting === 'safemode' && $ret === false)
 			$ret = 1;
 
-		if($setting === 'globaldir' && intval(get_config('system','localdir_hide')))
+		if($setting === 'globaldir' && intval(Config::Get('system','localdir_hide')))
 			$ret = 1;
 
 		return $ret;
@@ -133,7 +134,7 @@ class Libzotdir {
 		$globaldir = self::get_directory_setting($observer, 'globaldir');
 		$pubforums = self::get_directory_setting($observer, 'pubforums');
 
-		$hide_local = intval(get_config('system','localdir_hide'));
+		$hide_local = intval(Config::Get('system','localdir_hide'));
 		if($hide_local)
 			$globaldir = 1;
 
@@ -141,12 +142,12 @@ class Libzotdir {
 		// Build urls without order and pubforums so it's easy to tack on the changed value
 		// Probably there's an easier way to do this
 
-		$directory_sort_order = get_config('system','directory_sort_order');
+		$directory_sort_order = Config::Get('system','directory_sort_order');
 		if(! $directory_sort_order)
 			$directory_sort_order = 'date';
 
-		$current_order = (($_REQUEST['order']) ? $_REQUEST['order'] : $directory_sort_order);
-		$suggest = (($_REQUEST['suggest']) ? '&suggest=' . $_REQUEST['suggest'] : '');
+		$current_order = $_REQUEST['order'] ?? $directory_sort_order;
+		$suggest = ((isset($_REQUEST['suggest'])) ? '&suggest=' . $_REQUEST['suggest'] : '');
 
 		$url = 'directory?f=';
 
@@ -172,13 +173,12 @@ class Libzotdir {
 	}
 
 	/**
-	 * @brief Checks the directory mode of this hub.
+	 * @brief fetches updates from known directories
 	 *
 	 * Checks the directory mode of this hub to see if it is some form of directory server. If it is,
 	 * get the directory realm of this hub. Fetch a list of all other directory servers in this realm and request
-	 * a directory sync packet. This will contain both directory updates and new ratings. Store these all in the DB.
-	 * In the case of updates, we will query each of them asynchronously from a poller task. Ratings are stored
-	 * directly if the rater's signature matches.
+	 * a directory sync packet. Store these all in the DB.
+	 * In the case of updates, we will query each of them asynchronously from a poller task.
 	 *
 	 * @param int $dirmode;
 	 */
@@ -207,8 +207,6 @@ class Libzotdir {
 			);
 		}
 
-
-
 		// If there are no directory servers, setup the fallback master
 		/** @FIXME What to do if we're in a different realm? */
 
@@ -235,6 +233,8 @@ class Libzotdir {
 		if (! $r)
 			return;
 
+		$dir_trusted_hosts = array_merge(get_directory_fallback_servers(), Config::Get('system', 'trusted_directory_servers', []));
+
 		foreach ($r as $rr) {
 			if (! $rr['site_directory'])
 				continue;
@@ -245,15 +245,16 @@ class Libzotdir {
 			// It will take about a month for a new directory to obtain the full current repertoire of channels.
 			/** @FIXME Go back and pick up earlier ratings if this is a new directory server. These do not get refreshed. */
 
-			$token = get_config('system','realm_token');
+			$token = Config::Get('system','realm_token');
 
 			$syncdate = (($rr['site_sync'] <= NULL_DATE) ? datetime_convert('UTC','UTC','now - 2 days') : $rr['site_sync']);
 			$x = z_fetch_url($rr['site_directory'] . '?f=&sync=' . urlencode($syncdate) . (($token) ? '&t=' . $token : ''));
+
 			if (! $x['success'])
 				continue;
 
 			$j = json_decode($x['body'],true);
-			if (!($j['transactions']) || ($j['ratings']))
+			if (!$j['transactions'])
 				continue;
 
 			q("update site set site_sync = '%s' where site_url = '%s'",
@@ -265,26 +266,65 @@ class Libzotdir {
 
 			if (is_array($j['transactions']) && count($j['transactions'])) {
 				foreach ($j['transactions'] as $t) {
-					$r = q("select * from updates where ud_guid = '%s' limit 1",
-						dbesc($t['transaction_id'])
-					);
-					if($r)
+					if (empty($t['hash']) || empty($t['host']) || empty($t['address'])) {
 						continue;
+					}
 
-					$ud_flags = 0;
-					if (is_array($t['flags']) && in_array('deleted',$t['flags']))
-						$ud_flags |= UPDATE_FLAGS_DELETED;
-					if (is_array($t['flags']) && in_array('forced',$t['flags']))
-						$ud_flags |= UPDATE_FLAGS_FORCED;
-
-					$z = q("insert into updates ( ud_hash, ud_guid, ud_date, ud_flags, ud_addr )
-						values ( '%s', '%s', '%s', %d, '%s' ) ",
-						dbesc($t['hash']),
-						dbesc($t['transaction_id']),
-						dbesc($t['timestamp']),
-						intval($ud_flags),
-						dbesc($t['address'])
+					$r = q("select * from updates where ud_hash = '%s' limit 1",
+						dbesc($t['hash'])
 					);
+
+					if ($r) {
+						$update = 0;
+
+						// no need to look at updates that originated from our own site
+						if ($t['host'] === z_root()) {
+							continue;
+						}
+
+						// there is more recent xchan information
+						if ($r[0]['ud_date'] <= $t['timestamp']) {
+							$update = 1;
+						}
+
+						// the host is trusted and flags have changed - update flags immediately
+						if (in_array($t['host'], $dir_trusted_hosts) &&
+							$rr['site_url'] === $t['host'] &&
+							intval($r[0]['ud_flags']) !== intval($t['flags'])) {
+
+							q("UPDATE updates SET ud_update = %d, ud_flags = %d WHERE ud_id = %d",
+								intval($update),
+								intval($t['flags']),
+								dbesc($r[0]['ud_id'])
+							);
+
+							q("UPDATE xchan SET xchan_censored = %d WHERE xchan_hash = '%s'",
+								intval($t['flags']),
+								dbesc($r[0]['ud_hash'])
+							);
+
+							continue;
+						}
+
+						if (!$update) {
+							continue;
+						}
+
+						q("UPDATE updates SET ud_update = %d WHERE ud_id = %d",
+							intval($update),
+							dbesc($r[0]['ud_id'])
+						);
+					}
+					else {
+						q("insert into updates ( ud_hash, ud_host, ud_date, ud_addr, ud_update, ud_flags )
+							values ( '%s', '%s', '%s', '%s', 1, %d) ",
+							dbesc($t['hash']),
+							dbesc($t['host']),
+							dbesc($t['timestamp']),
+							dbesc($t['address']),
+							dbesc(in_array($t['host'], $dir_trusted_hosts) ? $t['flags'] : 0)
+						);
+					}
 				}
 			}
 		}
@@ -299,8 +339,9 @@ class Libzotdir {
 	 *
 	 * Ignore updating records marked as deleted.
 	 *
-	 * If successful, sets ud_last in the DB to the current datetime for this
+	 * If successful, sets ud_updated in the DB to the current datetime for this
 	 * reddress/webbie.
+	 * Else update ud_last so we can stop trying after 7 days (Daemon/Poller.php)
 	 *
 	 * @param array $ud Entry from update table
 	 */
@@ -309,23 +350,44 @@ class Libzotdir {
 
 		logger('update_directory_entry: ' . print_r($ud,true), LOGGER_DATA);
 
-		if ($ud['ud_addr'] && (! ($ud['ud_flags'] & UPDATE_FLAGS_DELETED))) {
-			$success = false;
+		// TODO: remove this check after all directory servers have version > 8.4
+		// ud_addr will always be the channel url at that time
+		$href = ((strpos($ud['ud_addr'], '://') === false) ? Webfinger::zot_url(punify($ud['ud_addr'])) : punify($ud['ud_addr']));
+		if($href) {
+			$zf = Zotfinger::exec($href);
+			if($zf && array_path_exists('signature/signer',$zf) && $zf['signature']['signer'] === $href && intval($zf['signature']['header_valid'])) {
+				$xc = Libzot::import_xchan($zf['data']);
 
-			$href = Webfinger::zot_url(punify($ud['ud_addr']));
-			if($href) {
-				$zf = Zotfinger::exec($href);
-			}
-			if(is_array($zf) && array_path_exists('signature/signer',$zf) && $zf['signature']['signer'] === $href && intval($zf['signature']['header_valid'])) {
-				$xc = Libzot::import_xchan($zf['data'], 0, $ud);
-			}
-			else {
-				q("update updates set ud_last = '%s' where ud_addr = '%s'",
-					dbesc(datetime_convert()),
-					dbesc($ud['ud_addr'])
-				);
+				// xchan_hash mismatch - this can happen after a site re-install at the same url
+				if ($xc['success'] && $xc['hash'] !== $ud['ud_hash']) {
+					self::delete_by_hash($ud['ud_hash']);
+				}
+
+				// if the channel was deleted - delete the entry in updates
+				if (!empty($zf['data']['deleted_locally'])) {
+					self::delete_by_hash($ud['ud_hash']);
+				}
+
+				// This is a workaround for a missing xchan_updated column
+				// TODO: implement xchan_updated in the xchan table and update this column instead
+				if(!empty($zf['data']['primary_location']['url'])) {
+					q("UPDATE hubloc SET hubloc_updated = '%s' WHERE hubloc_id_url = '%s' AND hubloc_primary = 1",
+						dbesc(datetime_convert()),
+						dbesc($zf['data']['primary_location']['url'])
+					);
+				}
+
+				return true;
 			}
 		}
+
+		q("UPDATE updates SET ud_addr = '%s', ud_last = '%s' WHERE ud_hash = '%s'",
+			dbesc($href ? $href : $ud['ud_addr']),
+			dbesc(datetime_convert()),
+			dbesc($ud['ud_hash'])
+		);
+
+		return false;
 	}
 
 
@@ -340,89 +402,78 @@ class Libzotdir {
 	 */
 
 	static function local_dir_update($uid, $force) {
+		logger('local_dir_update uid: ' . $uid, LOGGER_DEBUG);
 
-
-		logger('local_dir_update: uid: ' . $uid, LOGGER_DEBUG);
-
-		$p = q("select channel.channel_hash, channel_address, channel_timezone, channel_portable_id, profile.* from profile left join channel on channel_id = uid where uid = %d and is_default = 1",
+		$p = q("select channel.channel_hash, channel_address, channel_timezone, profile.*, xchan.xchan_hidden, xchan.xchan_url from profile left join channel on channel_id = uid left join xchan on channel_hash = xchan_hash where profile.uid = %d and profile.is_default = 1",
 			intval($uid)
 		);
 
-		$profile = array();
-		$profile['encoding'] = 'zot';
-
-		if ($p) {
-			$hash = $p[0]['channel_hash'];
-			$legacy_hash = $p[0]['channel_portable_id'];
-
-			$profile['description'] = $p[0]['pdesc'];
-			$profile['birthday']    = $p[0]['dob'];
-			if ($age = age($p[0]['dob'],$p[0]['channel_timezone'],''))
-				$profile['age'] = $age;
-
-			$profile['gender']      = $p[0]['gender'];
-			$profile['marital']     = $p[0]['marital'];
-			$profile['sexual']      = $p[0]['sexual'];
-			$profile['locale']      = $p[0]['locality'];
-			$profile['region']      = $p[0]['region'];
-			$profile['postcode']    = $p[0]['postal_code'];
-			$profile['country']     = $p[0]['country_name'];
-			$profile['about']       = $p[0]['about'];
-			$profile['homepage']    = $p[0]['homepage'];
-			$profile['hometown']    = $p[0]['hometown'];
-
-			if ($p[0]['keywords']) {
-				$tags = array();
-				$k = explode(' ', $p[0]['keywords']);
-				if ($k)
-					foreach ($k as $kk)
-						if (trim($kk))
-							$tags[] = trim($kk);
-
-				if ($tags)
-					$profile['keywords'] = $tags;
-			}
-
-			$hidden = (1 - intval($p[0]['publish']));
-
-			logger('hidden: ' . $hidden);
-
-			$r = q("select xchan_hidden from xchan where xchan_hash = '%s'",
-				dbesc($p[0]['channel_hash'])
-			);
-
-			if(intval($r[0]['xchan_hidden']) != $hidden) {
-				$r = q("update xchan set xchan_hidden = %d where xchan_hash in ('%s', '%s')",
-					intval($hidden),
-					dbesc($hash),
-					dbesc($legacy_hash)
-				);
-			}
-
-			$arr = [ 'channel_id' => $uid, 'hash' => $hash, 'profile' => $profile ];
-			call_hooks('local_dir_update', $arr);
-
-			$address = channel_reddress($p[0]);
-
-			if (perm_is_allowed($uid, '', 'view_profile')) {
-				self::import_directory_profile($hash, $arr['profile'], $address, 0);
-			}
-			else {
-				// they may have made it private
-				q("delete from xprof where xprof_hash in ('%s', '%s')",
-					dbesc($hash),
-					dbesc($legacy_hash)
-				);
-				q("delete from xtag where xtag_hash in ('%s', '%s')",
-					dbesc($hash),
-					dbesc($legacy_hash)
-				);
-			}
-
+		if (!$p) {
+			logger('profile not found');
+			return;
 		}
 
-		$ud_hash = random_string() . '@' . \App::get_hostname();
-		self::update_modtime($hash, $ud_hash, channel_reddress($p[0]),(($force) ? UPDATE_FLAGS_FORCED : UPDATE_FLAGS_UPDATED));
+		$profile = [];
+		$profile['encoding'] = 'zot';
+
+		$hash = $p[0]['channel_hash'];
+
+		$profile['description'] = $p[0]['pdesc'];
+		$profile['birthday']    = $p[0]['dob'];
+		if ($age = age($p[0]['dob'],$p[0]['channel_timezone'],''))
+			$profile['age'] = $age;
+
+		$profile['gender']      = $p[0]['gender'];
+		$profile['marital']     = $p[0]['marital'];
+		$profile['sexual']      = $p[0]['sexual'];
+		$profile['locale']      = $p[0]['locality'];
+		$profile['region']      = $p[0]['region'];
+		$profile['postcode']    = $p[0]['postal_code'];
+		$profile['country']     = $p[0]['country_name'];
+		$profile['about']       = $p[0]['about'];
+		$profile['homepage']    = $p[0]['homepage'];
+		$profile['hometown']    = $p[0]['hometown'];
+
+		if ($p[0]['keywords']) {
+			$tags = array();
+			$k = explode(' ', $p[0]['keywords']);
+			if ($k)
+				foreach ($k as $kk)
+					if (trim($kk))
+						$tags[] = trim($kk);
+
+			if ($tags)
+				$profile['keywords'] = $tags;
+		}
+
+		$hidden = (1 - intval($p[0]['publish']));
+
+		logger('hidden: ' . $hidden);
+
+		if(intval($p[0]['xchan_hidden']) !== $hidden) {
+			q("update xchan set xchan_hidden = %d where xchan_hash = '%s'",
+				intval($hidden),
+				dbesc($hash)
+			);
+		}
+
+		$arr = [ 'channel_id' => $uid, 'hash' => $hash, 'profile' => $profile ];
+		call_hooks('local_dir_update', $arr);
+
+		if (perm_is_allowed($uid, '', 'view_profile')) {
+			self::import_directory_profile($hash, $arr['profile']);
+		}
+		else {
+			// they may have made it private
+			q("delete from xprof where xprof_hash = '%s'",
+				dbesc($hash)
+			);
+			q("delete from xtag where xtag_hash = '%s'",
+				dbesc($hash)
+			);
+		}
+
+		self::update($hash, $p[0]['xchan_url']);
 	}
 
 
@@ -432,39 +483,48 @@ class Libzotdir {
 	 *
 	 * @param string $hash
 	 * @param array $profile
-	 * @param string $addr
-	 * @param number $ud_flags (optional) UPDATE_FLAGS_UPDATED
-	 * @param number $suppress_update (optional) default 0
 	 * @return boolean $updated if something changed
 	 */
 
-	static function import_directory_profile($hash, $profile, $addr, $ud_flags = UPDATE_FLAGS_UPDATED, $suppress_update = 0) {
+	static function import_directory_profile($hash, $profile) {
 
 		logger('import_directory_profile', LOGGER_DEBUG);
 		if (! $hash)
 			return false;
 
-		$arr = array();
+		$arr = [];
 
-		$arr['xprof_hash']         = $hash;
-		$arr['xprof_dob']          = (($profile['birthday'] === '0000-00-00') ? $profile['birthday'] : datetime_convert('','',$profile['birthday'],'Y-m-d')); // !!!! check this for 0000 year
-		$arr['xprof_age']          = (($profile['age'])         ? intval($profile['age']) : 0);
-		$arr['xprof_desc']         = (($profile['description']) ? htmlspecialchars($profile['description'], ENT_COMPAT,'UTF-8',false) : '');
-		$arr['xprof_gender']       = (($profile['gender'])      ? htmlspecialchars($profile['gender'],      ENT_COMPAT,'UTF-8',false) : '');
-		$arr['xprof_marital']      = (($profile['marital'])     ? htmlspecialchars($profile['marital'],     ENT_COMPAT,'UTF-8',false) : '');
-		$arr['xprof_sexual']       = (($profile['sexual'])      ? htmlspecialchars($profile['sexual'],      ENT_COMPAT,'UTF-8',false) : '');
-		$arr['xprof_locale']       = (($profile['locale'])      ? htmlspecialchars($profile['locale'],      ENT_COMPAT,'UTF-8',false) : '');
-		$arr['xprof_region']       = (($profile['region'])      ? htmlspecialchars($profile['region'],      ENT_COMPAT,'UTF-8',false) : '');
-		$arr['xprof_postcode']     = (($profile['postcode'])    ? htmlspecialchars($profile['postcode'],    ENT_COMPAT,'UTF-8',false) : '');
-		$arr['xprof_country']      = (($profile['country'])     ? htmlspecialchars($profile['country'],     ENT_COMPAT,'UTF-8',false) : '');
-		$arr['xprof_about']        = (($profile['about'])       ? htmlspecialchars($profile['about'],       ENT_COMPAT,'UTF-8',false) : '');
-		$arr['xprof_homepage']     = (($profile['homepage'])    ? htmlspecialchars($profile['homepage'],    ENT_COMPAT,'UTF-8',false) : '');
-		$arr['xprof_hometown']     = (($profile['hometown'])    ? htmlspecialchars($profile['hometown'],    ENT_COMPAT,'UTF-8',false) : '');
+		$arr['xprof_hash'] = $hash;
+		$arr['xprof_dob'] = '0000-00-00';
+
+		if (isset($profile['birthday'])) {
+			$arr['xprof_dob'] = (($profile['birthday'] === '0000-00-00')
+			? $profile['birthday']
+			: datetime_convert('', '', $profile['birthday'], 'Y-m-d')); // !!!! check this for 0000 year
+		}
+
+		$arr['xprof_age']          = ((isset($profile['age']) && $profile['age']) ? intval($profile['age']) : 0);
+		$arr['xprof_desc']         = ((isset($profile['description']) && $profile['description']) ? htmlspecialchars($profile['description'], ENT_COMPAT,'UTF-8',false) : '');
+		$arr['xprof_gender']       = ((isset($profile['gender']) && $profile['gender'])     ? htmlspecialchars($profile['gender'],      ENT_COMPAT,'UTF-8',false) : '');
+		$arr['xprof_marital']      = ((isset($profile['marital']) && $profile['marital'])   ? htmlspecialchars($profile['marital'],     ENT_COMPAT,'UTF-8',false) : '');
+		$arr['xprof_sexual']       = ((isset($profile['sexual']) && $profile['sexual'])     ? htmlspecialchars($profile['sexual'],      ENT_COMPAT,'UTF-8',false) : '');
+		$arr['xprof_locale']       = ((isset($profile['locale']) && $profile['locale'])     ? htmlspecialchars($profile['locale'],      ENT_COMPAT,'UTF-8',false) : '');
+		$arr['xprof_region']       = ((isset($profile['region']) && $profile['region'])     ? htmlspecialchars($profile['region'],      ENT_COMPAT,'UTF-8',false) : '');
+		$arr['xprof_postcode']     = ((isset($profile['postcode']) && $profile['postcode']) ? htmlspecialchars($profile['postcode'],    ENT_COMPAT,'UTF-8',false) : '');
+		$arr['xprof_country']      = ((isset($profile['country']) && $profile['country'])   ? htmlspecialchars($profile['country'],     ENT_COMPAT,'UTF-8',false) : '');
+		$arr['xprof_about']        = ((isset($profile['about']) && $profile['about'])       ? htmlspecialchars($profile['about'],       ENT_COMPAT,'UTF-8',false) : '');
+		$arr['xprof_homepage']     = ((isset($profile['homepage']) && $profile['homepage']) ? htmlspecialchars($profile['homepage'],    ENT_COMPAT,'UTF-8',false) : '');
+		$arr['xprof_hometown']     = ((isset($profile['hometown']) && $profile['hometown']) ? htmlspecialchars($profile['hometown'],    ENT_COMPAT,'UTF-8',false) : '');
 
 		$clean = array();
 		if (array_key_exists('keywords', $profile) and is_array($profile['keywords'])) {
 			self::import_directory_keywords($hash,$profile['keywords']);
+
 			foreach ($profile['keywords'] as $kw) {
+				if (in_array($kw, $clean)) {
+					continue;
+				}
+
 				$kw = trim(htmlspecialchars($kw,ENT_COMPAT, 'UTF-8', false));
 				$kw = trim($kw, ',');
 				$clean[] = $kw;
@@ -571,9 +631,6 @@ class Libzotdir {
 		 */
 		call_hooks('import_directory_profile', $d);
 
-		if (($d['update']) && (! $suppress_update))
-			self::update_modtime($arr['xprof_hash'],random_string() . '@' . \App::get_hostname(), $addr, $ud_flags);
-
 		return $d['update'];
 	}
 
@@ -591,6 +648,10 @@ class Libzotdir {
 			dbesc($hash)
 		);
 
+		$xchan = q("select xchan_censored from xchan where xchan_hash = '%s'",
+			dbesc($hash)
+		);
+
 		if($r) {
 			foreach($r as $rr)
 				$existing[] = $rr['xtag_term'];
@@ -598,6 +659,10 @@ class Libzotdir {
 
 		$clean = array();
 		foreach($keywords as $kw) {
+			if (in_array($kw, $clean)) {
+				continue;
+			}
+
 			$kw = trim(htmlspecialchars($kw,ENT_COMPAT, 'UTF-8', false));
 			$kw = trim($kw, ',');
 			$clean[] = $kw;
@@ -612,9 +677,10 @@ class Libzotdir {
 		}
 		foreach($clean as $x) {
 			if(! in_array($x, $existing)) {
-				$r = q("insert into xtag ( xtag_hash, xtag_term, xtag_flags) values ( '%s' ,'%s', 0 )",
+				$r = q("insert into xtag ( xtag_hash, xtag_term, xtag_flags) values ( '%s' ,'%s', %d )",
 					dbesc($hash),
-					dbesc($x)
+					dbesc($x),
+					intval($xchan[0]['xchan_censored'])
 				);
 			}
 		}
@@ -624,40 +690,83 @@ class Libzotdir {
 	/**
 	 * @brief
 	 *
-	 * @param string $hash
-	 * @param string $guid
-	 * @param string $addr
-	 * @param int $flags (optional) default 0
+	 * @param string $hash the channel hash
+	 * @param string $addr the channel url
+	 * @param bool $bump_date (optional) default true
 	 */
 
-	static function update_modtime($hash, $guid, $addr, $flags = 0) {
+	static function update($hash, $addr, $bump_date = true, $flag = null) {
 
-		$dirmode = intval(get_config('system', 'directory_mode'));
+		$dirmode = intval(Config::Get('system', 'directory_mode'));
 
-		if($dirmode == DIRECTORY_MODE_NORMAL)
+		if($dirmode == DIRECTORY_MODE_NORMAL) {
 			return;
+		}
 
-		if($flags) {
-			q("insert into updates (ud_hash, ud_guid, ud_date, ud_flags, ud_addr ) values ( '%s', '%s', '%s', %d, '%s' )",
-				dbesc($hash),
-				dbesc($guid),
-				dbesc(datetime_convert()),
-				intval($flags),
-				dbesc($addr)
-			);
+		if (empty($hash) || empty($addr)) {
+			return;
 		}
-		else {
-			q("update updates set ud_flags = ( ud_flags | %d ) where ud_addr = '%s' and not (ud_flags & %d)>0 ",
-				intval(UPDATE_FLAGS_UPDATED),
+
+		$u = q("SELECT * FROM updates WHERE ud_hash = '%s' LIMIT 1",
+			dbesc($hash)
+		);
+
+		$date_sql = '';
+		if ($bump_date) {
+			$date_sql = "ud_date = '" . dbesc(datetime_convert()) . "',";
+		}
+
+		$flag_sql = '';
+		if ($flag !== null) {
+			$flag_sql = "ud_flags = '" . intval($flag) . "',";
+		}
+
+
+		if ($u) {
+			$x = q("UPDATE updates SET $date_sql $flag_sql ud_last = '%s', ud_host = '%s', ud_addr = '%s', ud_update = 0 WHERE ud_id = %d",
+				dbesc(NULL_DATE),
+				dbesc(z_root()),
 				dbesc($addr),
-				intval(UPDATE_FLAGS_UPDATED)
+				intval($u[0]['ud_id'])
 			);
+
+			return;
 		}
+
+		q("INSERT INTO updates (ud_hash, ud_host, ud_date, ud_addr, ud_flags) VALUES ( '%s', '%s', '%s', '%s', %d )",
+			dbesc($hash),
+			dbesc(z_root()),
+			dbesc(datetime_convert()),
+			dbesc($addr),
+			intval($flag)
+		);
+
+		return;
+
 	}
 
 
+	/**
+	 * @brief deletes a entry in updates by hash
+	 *
+	 * @param string $hash the channel hash
+	 * @return boolean
+	 */
 
+	static function delete_by_hash($hash) {
+		if (!$hash) {
+			return false;
+		}
 
+		$x = q("DELETE FROM updates WHERE ud_hash = '%s'",
+			dbesc($hash)
+		);
 
+		if ($x) {
+			return true;
+		}
+
+		return false;
+	}
 
 }

Zotlabs/Lib/MessageFilter.php

@@ -2,87 +2,120 @@
 
 namespace Zotlabs\Lib;
 
-
+require_once('include/html2plain.php');
 
 class MessageFilter {
 
+	public static function evaluate($item, $incl, $excl) {
 
-	static public function evaluate($item,$incl,$excl) {
-
-		require_once('include/html2plain.php');
-
-		unobscure($item);
-
-		$text = prepare_text($item['body'],$item['mimetype']);
+		$text = prepare_text($item['body'],((isset($item['mimetype'])) ? $item['mimetype'] : 'text/bbcode'));
 		$text = html2plain(($item['title']) ? $item['title'] . ' ' . $text : $text);
 
-
 		$lang = null;
 
-		if((strpos($incl,'lang=') !== false) || (strpos($excl,'lang=') !== false) || (strpos($incl,'lang!=') !== false) || (strpos($excl,'lang!=') !== false)) {
+		if ((strpos($incl, 'lang=') !== false) || (strpos($excl, 'lang=') !== false) || (strpos($incl, 'lang!=') !== false) || (strpos($excl, 'lang!=') !== false)) {
 			$lang = detect_language($text);
 		}
 
-		$tags = ((is_array($item['term']) && count($item['term'])) ? $item['term'] : false);
+		$tags = ((isset($item['term']) && is_array($item['term']) && count($item['term'])) ? $item['term'] : false);
 
 		// exclude always has priority
 
-		$exclude = (($excl) ? explode("\n",$excl) : null);
+		$exclude = (($excl) ? explode("\n", $excl) : null);
 
-		if($exclude) {
-			foreach($exclude as $word) {
-				$word = trim($word);
-				if(! $word)
+		if ($exclude) {
+			foreach ($exclude as $word) {
+				$word = html_entity_decode(trim($word));
+				if (! $word) {
 					continue;
-				if(substr($word,0,1) === '#' && $tags) {
-					foreach($tags as $t)
-						if((($t['ttype'] == TERM_HASHTAG) || ($t['ttype'] == TERM_COMMUNITYTAG)) && (($t['term'] === substr($word,1)) || (substr($word,1) === '*')))
-							return false;
 				}
-				elseif(substr($word,0,1) === '$' && $tags) {
-					foreach($tags as $t)
-						if(($t['ttype'] == TERM_CATEGORY) && (($t['term'] === substr($word,1)) || (substr($word,1) === '*')))
-							return false;
+				if (isset($lang) && ((strpos($word, 'lang=') === 0) || (strpos($word, 'lang!=') === 0))) {
+					if (!strlen($lang)) {
+						// Result is ambiguous. As we are matching deny rules only at this time, continue tests.
+						// Any matching deny rule concludes testing.
+						continue;
+					}
+					if (strpos($word, 'lang=') === 0 && strcasecmp($lang, trim(substr($word, 5))) == 0) {
+						return false;
+					} elseif (strpos($word, 'lang!=') === 0 && strcasecmp($lang, trim(substr($word, 6))) != 0) {
+						return false;
+					}
 				}
-				elseif((strpos($word,'/') === 0) && preg_match($word,$text))
+                elseif (substr($word, 0, 1) === '#' && $tags) {
+					foreach ($tags as $t) {
+						if ((($t['ttype'] == TERM_HASHTAG) || ($t['ttype'] == TERM_COMMUNITYTAG)) && (($t['term'] === substr($word, 1)) || (substr($word, 1) === '*'))) {
+							return false;
+						}
+					}
+				} elseif (substr($word, 0, 1) === '$' && $tags) {
+					foreach ($tags as $t) {
+						if (($t['ttype'] == TERM_CATEGORY) && (($t['term'] === substr($word, 1)) || (substr($word, 1) === '*'))) {
+							return false;
+						}
+					}
+				} elseif (substr($word, 0, 2) === '?+') {
+					if (self::test_condition(substr($word, 2), $item['obj'])) {
+						return false;
+					}
+				} elseif (substr($word, 0, 1) === '?') {
+					if (self::test_condition(substr($word, 1), $item)) {
+						return false;
+					}
+				} elseif ((strpos($word, '/') === 0) && preg_match($word, $text)) {
 					return false;
-				elseif((strpos($word,'lang=') === 0) && ($lang) && (strcasecmp($lang,trim(substr($word,5))) == 0))
-					return false;
-				elseif((strpos($word,'lang!=') === 0) && ($lang) && (strcasecmp($lang,trim(substr($word,6))) != 0))
-					return false;
-				elseif(stristr($text,$word) !== false)
+				} elseif (stristr($text, $word) !== false) {
 					return false;
+				}
 			}
 		}
 
-		$include = (($incl) ? explode("\n",$incl) : null);
+		$include = (($incl) ? explode("\n", $incl) : null);
 
-		if($include) {
-			foreach($include as $word) {
-				$word = trim($word);
-				if(! $word)
+		if ($include) {
+			foreach ($include as $word) {
+				$word = html_entity_decode(trim($word));
+				if (! $word) {
 					continue;
-				if(substr($word,0,1) === '#' && $tags) {
-					foreach($tags as $t)
-						if((($t['ttype'] == TERM_HASHTAG) || ($t['ttype'] == TERM_COMMUNITYTAG)) && (($t['term'] === substr($word,1)) || (substr($word,1) === '*')))
-							return true;
 				}
-				elseif(substr($word,0,1) === '$' && $tags) {
-					foreach($tags as $t)
-						if(($t['ttype'] == TERM_CATEGORY) && (($t['term'] === substr($word,1)) || (substr($word,1) === '*')))
-							return true;
+				if (isset($lang) && ((strpos($word, 'lang=') === 0) || (strpos($word, 'lang!=') === 0))) {
+					if (!strlen($lang))  {
+						// Result is ambiguous. However we are checking allow rules
+						// and an ambiguous language is always permitted.
+						return true;
+					}
+					if (strpos($word, 'lang=') === 0 && strcasecmp($lang, trim(substr($word, 5))) == 0) {
+						return true;
+					} elseif (strpos($word, 'lang!=') === 0 && strcasecmp($lang, trim(substr($word, 6))) != 0) {
+						return true;
+					}
 				}
-				elseif((strpos($word,'/') === 0) && preg_match($word,$text))
+                elseif (substr($word, 0, 1) === '#' && $tags) {
+					foreach ($tags as $t) {
+						if ((($t['ttype'] == TERM_HASHTAG) || ($t['ttype'] == TERM_COMMUNITYTAG)) && (($t['term'] === substr($word, 1)) || (substr($word, 1) === '*'))) {
+							return true;
+						}
+					}
+				} elseif (substr($word, 0, 1) === '$' && $tags) {
+					foreach ($tags as $t) {
+						if (($t['ttype'] == TERM_CATEGORY) && (($t['term'] === substr($word, 1)) || (substr($word, 1) === '*'))) {
+							return true;
+						}
+					}
+				} elseif (substr($word, 0, 2) === '?+') {
+					if (self::test_condition(substr($word, 2), $item['obj'])) {
+						return true;
+					}
+				} elseif (substr($word, 0, 1) === '?') {
+					if (self::test_condition(substr($word, 1), $item)) {
+						return true;
+					}
+				} elseif ((strpos($word, '/') === 0) && preg_match($word, $text)) {
 					return true;
-				elseif((strpos($word,'lang=') === 0) && ($lang) && (strcasecmp($lang,trim(substr($word,5))) == 0))
-					return true;
-				elseif((strpos($word,'lang!=') === 0) && ($lang) && (strcasecmp($lang,trim(substr($word,6))) != 0))
-					return true;
-				elseif(stristr($text,$word) !== false)
+				} elseif (stristr($text, $word) !== false) {
 					return true;
+				}
 			}
-		}
-		else {
+		} else {
 			return true;
 		}
 
@@ -90,4 +123,123 @@ class MessageFilter {
 	}
 
 
+	/**
+	 * @brief Test for Conditional Execution conditions. Shamelessly ripped off from Code/Render/Comanche
+	 *
+	 * This is extensible. The first version of variable testing supports tests of the forms:
+	 *
+	 * - ?foo ~= baz which will check if item.foo contains the string 'baz';
+	 * - ?foo == baz which will check if item.foo is the string 'baz';
+	 * - ?foo != baz which will check if item.foo is not the string 'baz';
+	 * - ?foo >= 3 which will check if item.foo is greater than or equal to 3;
+	 * - ?foo > 3 which will check if item.foo is greater than 3;
+	 * - ?foo <= 3 which will check if item.foo is less than or equal to 3;
+	 * - ?foo < 3 which will check if item.foo is less than 3;
+	 *
+	 * - ?foo {} baz which will check if 'baz' is an array element in item.foo
+	 * - ?foo {*} baz which will check if 'baz' is an array key in item.foo
+	 * - ?foo which will check for a return of a true condition for item.foo;
+     * - ?!foo which will check for a return of a false condition for item.foo;
+     *
+	 * The values 0, '', an empty array, and an unset value will all evaluate to false.
+	 *
+	 * @param string $s
+	 * @param array $item
+	 * @return bool
+	 */
+
+	public static function test_condition($s,$item) {
+
+		if (preg_match('/(.*?)\s\~\=\s(.*?)$/', $s, $matches)) {
+			$x = ((array_key_exists(trim($matches[1]),$item)) ? $item[trim($matches[1])] : EMPTY_STR);
+			if (stripos($x, trim($matches[2])) !== false) {
+				return true;
+			}
+			return false;
+		}
+
+		if (preg_match('/(.*?)\s\=\=\s(.*?)$/', $s, $matches)) {
+			$x = ((array_key_exists(trim($matches[1]),$item)) ? $item[trim($matches[1])] : EMPTY_STR);
+			if ($x == trim($matches[2])) {
+				return true;
+			}
+			return false;
+		}
+
+		if (preg_match('/(.*?)\s\!\=\s(.*?)$/', $s, $matches)) {
+			$x = ((array_key_exists(trim($matches[1]),$item)) ? $item[trim($matches[1])] : EMPTY_STR);
+			if ($x != trim($matches[2])) {
+				return true;
+			}
+			return false;
+		}
+
+		if (preg_match('/(.*?)\s\>\=\s(.*?)$/', $s, $matches)) {
+			$x = ((array_key_exists(trim($matches[1]),$item)) ? $item[trim($matches[1])] : EMPTY_STR);
+			if ($x >= trim($matches[2])) {
+				return true;
+			}
+			return false;
+		}
+
+		if (preg_match('/(.*?)\s\<\=\s(.*?)$/', $s, $matches)) {
+			$x = ((array_key_exists(trim($matches[1]),$item)) ? $item[trim($matches[1])] : EMPTY_STR);
+			if ($x <= trim($matches[2])) {
+				return true;
+			}
+			return false;
+		}
+
+		if (preg_match('/(.*?)\s\>\s(.*?)$/', $s, $matches)) {
+			$x = ((array_key_exists(trim($matches[1]),$item)) ? $item[trim($matches[1])] : EMPTY_STR);
+			if ($x > trim($matches[2])) {
+				return true;
+			}
+			return false;
+		}
+
+		if (preg_match('/(.*?)\s\>\s(.*?)$/', $s, $matches)) {
+			$x = ((array_key_exists(trim($matches[1]),$item)) ? $item[trim($matches[1])] : EMPTY_STR);
+			if ($x < trim($matches[2])) {
+				return true;
+			}
+			return false;
+		}
+
+		if (preg_match('/[\$](.*?)\s\{\}\s(.*?)$/', $s, $matches)) {
+			$x = ((array_key_exists(trim($matches[1]),$item)) ? $item[trim($matches[1])] : EMPTY_STR);
+			if (is_array($x) && in_array(trim($matches[2]), $x)) {
+				return true;
+			}
+			return false;
+		}
+
+		if (preg_match('/(.*?)\s\{\*\}\s(.*?)$/', $s, $matches)) {
+			$x = ((array_key_exists(trim($matches[1]),$item)) ? $item[trim($matches[1])] : EMPTY_STR);
+			if (is_array($x) && array_key_exists(trim($matches[2]), $x)) {
+				return true;
+			}
+			return false;
+		}
+
+		// Ordering of this check (for falsiness) with relation to the following one (check for truthiness) is important.
+		if (preg_match('/\!(.*?)$/', $s, $matches)) {
+			$x = ((array_key_exists(trim($matches[1]),$item)) ? $item[trim($matches[1])] : EMPTY_STR);
+			if (!$x) {
+				return true;
+			}
+			return false;
+		}
+
+		if (preg_match('/(.*?)$/', $s, $matches)) {
+			$x = ((array_key_exists(trim($matches[1]),$item)) ? $item[trim($matches[1])] : EMPTY_STR);
+			if ($x) {
+				return true;
+			}
+			return false;
+		}
+
+		return false;
+	}
+
 }

Zotlabs/Lib/Multibase.php

@@ -0,0 +1,34 @@
+<?php
+namespace Zotlabs\Lib;
+
+use StephenHill\Base58;
+
+class Multibase {
+
+    protected $key = null;
+
+    public function __construct() {
+        return $this;
+    }
+
+    public function publicKey($key) {
+        $base58 = new Base58();
+        $raw = hex2bin('ed01') . sodium_base642bin($key, SODIUM_BASE64_VARIANT_ORIGINAL_NO_PADDING);
+        return 'z' . $base58->encode($raw);
+    }
+
+    public function secretKey($key) {
+        $base58 = new Base58();
+        $raw = hex2bin('8026') . sodium_base642bin($key, SODIUM_BASE64_VARIANT_ORIGINAL_NO_PADDING);
+        return 'z' . $base58->encode($raw);
+    }
+
+    public function decode($key, $binary  = false) {
+        $base58 = new Base58();
+        $key = substr($key,1);
+        $raw = $base58->decode($key);
+        $binaryKey = substr($raw, 2);
+        return $binary ? $binaryKey : sodium_bin2base64($binaryKey, SODIUM_BASE64_VARIANT_ORIGINAL_NO_PADDING);
+    }
+
+}

Zotlabs/Lib/NativeWiki.php

@@ -1,313 +0,0 @@
-<?php
-
-namespace Zotlabs\Lib;
-
-use Zotlabs\Lib\Libsync;
-
-define ( 'NWIKI_ITEM_RESOURCE_TYPE', 'nwiki' );
-
-class NativeWiki {
-
-
-	public static function listwikis($channel, $observer_hash) {
-
-		$sql_extra = item_permissions_sql($channel['channel_id'], $observer_hash);
-		$wikis = q("SELECT * FROM item 
-			WHERE resource_type = '%s' AND mid = parent_mid AND uid = %d AND item_deleted = 0 $sql_extra", 
-			dbesc(NWIKI_ITEM_RESOURCE_TYPE),
-			intval($channel['channel_id'])
-		);
-
-		if($wikis) {
-			foreach($wikis as &$w) {
-
-				$w['json_allow_cid']  = acl2json($w['allow_cid']);
-				$w['json_allow_gid']  = acl2json($w['allow_gid']);
-				$w['json_deny_cid']   = acl2json($w['deny_cid']);
-				$w['json_deny_gid']   = acl2json($w['deny_gid']);
-
-				$w['rawName']  = get_iconfig($w, 'wiki', 'rawName');
-				$w['htmlName'] = escape_tags($w['rawName']);
-				//$w['urlName']  = urlencode(urlencode($w['rawName']));
-				$w['urlName']  = self::name_encode($w['rawName']);
-				$w['mimeType'] = get_iconfig($w, 'wiki', 'mimeType');
-				$w['typelock'] = get_iconfig($w, 'wiki', 'typelock');
-				$w['lockstate']     = (($w['allow_cid'] || $w['allow_gid'] || $w['deny_cid'] || $w['deny_gid']) ? 'lock' : 'unlock');
-			}
-		}
-		// TODO: query db for wikis the observer can access. Return with two lists, for read and write access
-		return array('wikis' => $wikis);
-	}
-
-
-	public static function create_wiki($channel, $observer_hash, $wiki, $acl) {
-
-		$resource_id = new_uuid();
-		$uuid = new_uuid();
-
-		$ac = $acl->get();
-		$mid = z_root() . '/item/' . $uuid;
-
-		$arr = array();	// Initialize the array of parameters for the post
-		$item_hidden = ((intval($wiki['postVisible']) === 0) ? 1 : 0); 
-		$wiki_url = z_root() . '/wiki/' . $channel['channel_address'] . '/' . $wiki['urlName'];
-		$arr['aid'] = $channel['channel_account_id'];
-		$arr['uuid'] = $uuid;
-		$arr['uid'] = $channel['channel_id'];
-		$arr['mid'] = $mid;
-		$arr['parent_mid'] = $mid;
-		$arr['item_hidden'] = $item_hidden;
-		$arr['resource_type'] = NWIKI_ITEM_RESOURCE_TYPE;
-		$arr['resource_id'] = $resource_id;
-		$arr['owner_xchan'] = $channel['channel_hash'];
-		$arr['author_xchan'] = $observer_hash;
-		$arr['plink'] = z_root() . '/channel/' . $channel['channel_address'] . '/?f=&mid=' . urlencode($arr['mid']);
-		$arr['llink'] = $arr['plink'];
-		$arr['title'] = $wiki['htmlName'];  // name of new wiki;
-		$arr['allow_cid'] = $ac['allow_cid'];
-		$arr['allow_gid'] = $ac['allow_gid'];
-		$arr['deny_cid'] = $ac['deny_cid'];
-		$arr['deny_gid'] = $ac['deny_gid'];
-		$arr['item_wall'] = 1;
-		$arr['item_origin'] = 1;
-		$arr['item_thread_top'] = 1;
-		$arr['item_private'] = intval($acl->is_private());
-		$arr['verb'] = ACTIVITY_CREATE;
-		$arr['obj_type'] = 'Document';
-		$arr['body'] = '[table][tr][td][h1]New Wiki[/h1][/td][/tr][tr][td][zrl=' . $wiki_url . ']' . $wiki['htmlName'] . '[/zrl][/td][/tr][/table]';
-
-		$arr['public_policy'] = map_scope(\Zotlabs\Access\PermissionLimits::Get($channel['channel_id'],'view_wiki'),true);
-
-		// Save the wiki name information using iconfig. This is shareable.
-		if(! set_iconfig($arr, 'wiki', 'rawName', $wiki['rawName'], true)) {
-			return array('item' => null, 'success' => false);
-		}
-		if(! set_iconfig($arr, 'wiki', 'mimeType', $wiki['mimeType'], true)) {
-			return array('item' => null, 'success' => false);
-		}
-
-		set_iconfig($arr,'wiki','typelock',$wiki['typelock'],true);
-
-		$post = item_store($arr);
-
-		$item_id = $post['item_id'];
-
-		if($item_id) {
-			\Zotlabs\Daemon\Master::Summon(array('Notifier', 'activity', $item_id));
-			return array('item' => $post['item'], 'item_id' => $item_id, 'success' => true);
-		}
-		else {
-			return array('item' => null, 'success' => false);
-		}
-	}
-
-	public static function update_wiki($channel_id, $observer_hash, $arr, $acl) {
-
-		$w = self::get_wiki($channel_id, $observer_hash, $arr['resource_id']);
-		$item = $w['wiki'];
-
-		if(! $item) {
-			return array('item' => null, 'success' => false);
-		}
-
-		$x = $acl->get();
-
-		$item['allow_cid']    = $x['allow_cid'];
-		$item['allow_gid']    = $x['allow_gid'];
-		$item['deny_cid']     = $x['deny_cid'];
-		$item['deny_gid']     = $x['deny_gid'];
-		$item['item_private'] = intval($acl->is_private());
-
-		$update_title = false;
-
-		if($item['title'] !== $arr['updateRawName']) {
-			$update_title = true;
-			$item['title'] = $arr['updateRawName'];
-		}
-
-		$update = item_store_update($item);
-
-		$item_id = $update['item_id'];
-
-		// update acl for any existing wiki pages
-
-		q("update item set allow_cid = '%s', allow_gid = '%s', deny_cid = '%s', deny_gid = '%s', item_private = %d where resource_type = 'nwikipage' and resource_id = '%s'",
-			dbesc($item['allow_cid']), 
-			dbesc($item['allow_gid']), 
-			dbesc($item['deny_cid']), 
-			dbesc($item['deny_gid']), 
-			dbesc($item['item_private']), 
-			dbesc($arr['resource_id'])
-		); 
-
-
-		if($update['item_id']) {
-			info( t('Wiki updated successfully'));
-			if($update_title) {
-				// Update the wiki name information using iconfig.
-				if(! set_iconfig($update['item_id'], 'wiki', 'rawName', $arr['updateRawName'], true)) {
-					return array('item' => null, 'success' => false);
-				}
-			}
-			return array('item' => $update['item'], 'item_id' => $update['item_id'], 'success' => $update['success']);
-		}
-		else {
-			return array('item' => null, 'success' => false);
-		}
-	}
-
-	public static function sync_a_wiki_item($uid,$id,$resource_id) {
-
-
-		$r = q("SELECT * from item WHERE uid = %d AND ( id = %d OR ( resource_type = '%s' and resource_id = '%s' )) ",
-			intval($uid),
-			intval($id),
-			dbesc(NWIKI_ITEM_RESOURCE_TYPE),
-			dbesc($resource_id)
-		);
-
-		if($r) {
-			$q = q("select * from item where resource_type = 'nwikipage' and resource_id = '%s'",
-				dbesc($r[0]['resource_id'])
-			);
-			if($q) {
-				$r = array_merge($r,$q);
-			}
-			xchan_query($r);
-			$sync_item = fetch_post_tags($r);
-			if($sync_item) {
-				$pkt = [];
-				foreach($sync_item as $w) {
-					$pkt[] = encode_item($w,true);
-				}
-				Libsync::build_sync_packet($uid,array('wiki' => $pkt));
-			}
-		}
-	}
-
-	public static function delete_wiki($channel_id,$observer_hash,$resource_id) {
-
-		$w = self::get_wiki($channel_id,$observer_hash,$resource_id);
-		$item = $w['wiki'];
-		if(! $item) {
-			return array('item' => null, 'success' => false);
-		} 
-		else {
-			$drop = drop_item($item['id'], false, DROPITEM_NORMAL);
-		}
-
-		info( t('Wiki files deleted successfully'));
-
-		return array('item' => $item, 'item_id' => $item['id'], 'success' => (($drop === 1) ? true : false));
-	}
-
-
-	public static function get_wiki($channel_id, $observer_hash, $resource_id) {
-		
-		$sql_extra = item_permissions_sql($channel_id,$observer_hash);
-
-		$item = q("SELECT * FROM item WHERE uid = %d AND resource_type = '%s' AND resource_id = '%s' AND item_deleted = 0 
-			$sql_extra limit 1",
-			intval($channel_id), 
-			dbesc(NWIKI_ITEM_RESOURCE_TYPE),
-			dbesc($resource_id)
-		);
-		if(! $item) {
-			return array('wiki' => null);
-		}
-		else {
-		
-			$w = $item[0];	// wiki item table record
-			// Get wiki metadata
-			$rawName  = get_iconfig($w, 'wiki', 'rawName');
-			$mimeType = get_iconfig($w, 'wiki', 'mimeType');
-			$typelock = get_iconfig($w, 'wiki', 'typelock');
-
-			return array(
-				'wiki'     => $w,
-				'rawName'  => $rawName,
-				'htmlName' => escape_tags($rawName),
-				//'urlName'  => urlencode(urlencode($rawName)),
-				'urlName'  => self::name_encode($rawName),
-				'mimeType' => $mimeType,
-				'typelock' => $typelock
-			);
-		}
-	}
-
-
-	public static function exists_by_name($uid, $urlName) {
-
-		$sql_extra = item_permissions_sql($uid);		
-
-		$item = q("SELECT item.id, resource_id FROM item left join iconfig on iconfig.iid = item.id 
-			WHERE resource_type = '%s' AND iconfig.v = '%s' AND uid = %d 
-			AND item_deleted = 0 $sql_extra limit 1", 
-			dbesc(NWIKI_ITEM_RESOURCE_TYPE), 
-			//dbesc(urldecode($urlName)), 
-			dbesc(self::name_decode($urlName)),
-			intval($uid)
-		);
-
-		if($item) {
-			return array('id' => $item[0]['id'], 'resource_id' => $item[0]['resource_id']);
-		} 
-		else {
-			return array('id' => null, 'resource_id' => null);
-		}
-	}
-
-
-	public static function get_permissions($resource_id, $owner_id, $observer_hash) {
-		// TODO: For now, only the owner can edit
-		$sql_extra = item_permissions_sql($owner_id, $observer_hash);
-
-		if(local_channel() && local_channel() == $owner_id) {
-			return [ 'read' => true, 'write' => true, 'success' => true ];
-		}
-
-		$r = q("SELECT * FROM item WHERE uid = %d and resource_type = '%s' AND resource_id = '%s' $sql_extra LIMIT 1",
-			intval($owner_id),
-			dbesc(NWIKI_ITEM_RESOURCE_TYPE), 
-			dbesc($resource_id)
-		);
-
-		if(! $r) {
-			return array('read' => false, 'write' => false, 'success' => true);
-		}
-		else {
-			// TODO: Create a new permission setting for wiki analogous to webpages. Until
-			// then, use webpage permissions
-			$write = perm_is_allowed($owner_id, $observer_hash,'write_wiki');
-			return array('read' => true, 'write' => $write, 'success' => true);
-		}
-	}
-
-	public static function name_encode ($string) {
-
-		$string = html_entity_decode($string);
-		$encoding = mb_internal_encoding();
-		mb_internal_encoding("UTF-8");
-		$ret = mb_ereg_replace_callback ('[^A-Za-z0-9\-\_\.\~]',function ($char) {
-	                $charhex = unpack('H*',$char[0]);
-                	$ret = '('.$charhex[1].')';
-                	return $ret;
- 			}
-			,$string);
-		mb_internal_encoding($encoding);
-		return $ret;
-	}
-
-	public static function name_decode ($string) {
-
-		$encoding = mb_internal_encoding();
-		mb_internal_encoding("UTF-8");
-		$ret = mb_ereg_replace_callback ('(\(([0-9a-f]+)\))',function ($chars) {
-			return pack('H*',$chars[2]);
-			}
-			,$string);
-		mb_internal_encoding($encoding);
-		return $ret;
-	}
-
-}

Zotlabs/Lib/NativeWikiPage.php

@@ -1,704 +0,0 @@
-<?php
-
-namespace Zotlabs\Lib;
-
-use \Zotlabs\Lib as Zlib;
-
-class NativeWikiPage {
-
-	static public function page_list($channel_id,$observer_hash, $resource_id) {
-
-		// TODO: Create item table records for pages so that metadata like title can be applied
-		$w = Zlib\NativeWiki::get_wiki($channel_id,$observer_hash,$resource_id);
-
-		$pages[] = [
-			'resource_id' => '',
-			'title'       => 'Home',
-			'url'         => 'Home',
-			'link_id'     => 'id_wiki_home_0'
-		];
-
-		$sql_extra = item_permissions_sql($channel_id,$observer_hash);
-
-		$r = q("select * from item where resource_type = 'nwikipage' and resource_id = '%s' and uid = %d and item_deleted = 0 
-			$sql_extra order by title asc",
-			dbesc($resource_id),
-			intval($channel_id)
-		);
-		if($r) {
-			$x = [];
-			$y = [];
-
-			foreach($r as $rv) {
-				if(! in_array($rv['mid'],$x)) {
-					$y[] = $rv;
-					$x[] = $rv['mid'];
-				}
-			}
-
-			$items = fetch_post_tags($y,true);
-
-			foreach($items as $page_item) {
-				$title = get_iconfig($page_item['id'],'nwikipage','pagetitle',t('(No Title)'));
-				if(urldecode($title) !== 'Home') {
-					$pages[] = [
-						'resource_id' => $resource_id,
-						'title'       => escape_tags($title),
-						//'url'         => str_replace('%2F','/',urlencode(str_replace('%2F','/',urlencode($title)))),
-						'url'		=> Zlib\NativeWiki::name_encode($title),
-						'link_id'     => 'id_' . substr($resource_id, 0, 10) . '_' . $page_item['id']
-					];
-				}
-			}
-		}
-
-		return array('pages' => $pages, 'wiki' => $w);
-	}
-
-
-	static public function create_page($channel_id, $observer_hash, $name, $resource_id, $mimetype = 'text/bbcode') {
-
-		logger('mimetype: ' . $mimetype);
-
-		if(! in_array($mimetype,[ 'text/markdown','text/bbcode','text/plain','text/html' ]))
-			$mimetype = 'text/markdown';
-
-		$w = Zlib\NativeWiki::get_wiki($channel_id, $observer_hash, $resource_id);
-
-		if (! $w['wiki']) {
-			return array('content' => null, 'message' => 'Error reading wiki', 'success' => false);
-		}
-
-		// backslashes won't work well in the javascript functions
-		$name = str_replace('\\','',$name);
-
-		// create an empty activity
-
-		$arr = [];
-		$arr['uid']           = $channel_id;
-		$arr['author_xchan']  = $observer_hash;
-		$arr['mimetype']      = $mimetype;
-		$arr['title']         = $name;
-		$arr['resource_type'] = 'nwikipage';
-		$arr['resource_id']   = $resource_id;
-		$arr['allow_cid']     = $w['wiki']['allow_cid'];
-		$arr['allow_gid']     = $w['wiki']['allow_gid'];
-		$arr['deny_cid']      = $w['wiki']['deny_cid'];
-		$arr['deny_gid']      = $w['wiki']['deny_gid'];
-
-		$arr['public_policy'] = map_scope(\Zotlabs\Access\PermissionLimits::Get($channel_id,'view_wiki'),true);
-
-		// We may wish to change this some day.
-		$arr['item_unpublished'] = 1;
-
-		set_iconfig($arr,'nwikipage','pagetitle',(($name) ? $name : t('(No Title)')),true);
-
-		$p = post_activity_item($arr, false, false);
-
-		if($p['item_id']) {
-			$page = [ 
-				'rawName'  => $name,
-				'htmlName' => escape_tags($name),
-				//'urlName'  => urlencode($name), 
-				'urlName' => Zlib\NativeWiki::name_encode($name)
-
-			];
-
-			return array('page' => $page, 'item_id' => $p['item_id'], 'item' => $p['activity'], 'wiki' => $w, 'message' => '', 'success' => true);
-		}
-		return [ 'success' => false, 'message' => t('Wiki page create failed.') ];
-	}
-
-	static public function rename_page($arr) {
-
-		$pageUrlName   = ((array_key_exists('pageUrlName',$arr))   ? $arr['pageUrlName']   : '');
-		$pageNewName   = ((array_key_exists('pageNewName',$arr))   ? $arr['pageNewName']   : '');
-		$resource_id   = ((array_key_exists('resource_id',$arr))   ? $arr['resource_id']   : '');
-		$observer_hash = ((array_key_exists('observer_hash',$arr)) ? $arr['observer_hash'] : '');
-		$channel_id    = ((array_key_exists('channel_id',$arr))    ? $arr['channel_id']    : 0);
-
-		$w = Zlib\NativeWiki::get_wiki($channel_id, $observer_hash, $resource_id);
-		if(! $w['wiki']) {
-			return array('message' => t('Wiki not found.'), 'success' => false);
-		}
-
-
-		$ic = q("select * from iconfig left join item on iconfig.iid = item.id 
-			where uid = %d and cat = 'nwikipage' and k = 'pagetitle' and v = '%s'",
-			intval($channel_id),
-			dbesc($pageNewName)
-		);
-
-		if($ic) {
-			return [ 'success' => false, 'message' => t('Destination name already exists') ];
-		}
-
-
-		$ids = [];
-
-		$ic = q("select *, item.id as item_id from iconfig left join item on iconfig.iid = item.id 
-			where uid = %d and cat = 'nwikipage' and k = 'pagetitle' and v = '%s'",
-			intval($channel_id),
-			dbesc($pageUrlName)
-		);
-
-		if($ic) {
-			foreach($ic as $c) {
-				set_iconfig($c['item_id'],'nwikipage','pagetitle',$pageNewName);
-				$ids[] = $c['item_id'];
-			}
-
-			$str_ids = implode(',', $ids);
-			q("update item set title = '%s' where id in ($str_ids)",
-				dbesc($pageNewName)
-			);
-
-			$page = [ 
-				'rawName'  => $pageNewName, 
-				'htmlName' => escape_tags($pageNewName), 
-				//'urlName'  => urlencode(escape_tags($pageNewName))
-				'urlName' => Zlib\NativeWiki::name_encode($pageNewName)
-			];
-
-			return [ 'success' => true, 'page' => $page ];
-		}
-
-		return [ 'success' => false, 'item_id' => $c['item_id'], 'message' => t('Page not found') ];
-	
-	}
-
-	static public function get_page_content($arr) {
-		$pageUrlName   = ((array_key_exists('pageUrlName',$arr))   ? $arr['pageUrlName']        : '');
-		$resource_id   = ((array_key_exists('resource_id',$arr))   ? $arr['resource_id']        : '');
-		$observer_hash = ((array_key_exists('observer_hash',$arr)) ? $arr['observer_hash']      : '');
-		$channel_id    = ((array_key_exists('channel_id',$arr))    ? intval($arr['channel_id']) : 0);
-		$revision      = ((array_key_exists('revision',$arr))      ? intval($arr['revision'])   : (-1));
-
-
-		$w = Zlib\NativeWiki::get_wiki($channel_id, $observer_hash, $resource_id);
-		if (! $w['wiki']) {
-			return array('content' => null, 'message' => 'Error reading wiki', 'success' => false);
-		}
-
-		$item = self::load_page($arr);
-
-		if($item) {
-			$content = $item['body'];
-
-			return [ 
-				'content'      => $content,
-				'mimeType'     => $w['mimeType'],
-				'pageMimeType' => $item['mimetype'], 
-				'message'      => '', 
-				'success'      => true
-			];
-		}
-	
-		return array('content' => null, 'message' => t('Error reading page content'), 'success' => false);
-
-	}
-
-	static public function page_history($arr) {
-		$pageUrlName = ((array_key_exists('pageUrlName',$arr)) ? $arr['pageUrlName'] : '');
-		$resource_id = ((array_key_exists('resource_id',$arr)) ? $arr['resource_id'] : '');
-		$observer_hash = ((array_key_exists('observer_hash',$arr)) ? $arr['observer_hash'] : '');
-		$channel_id    = ((array_key_exists('channel_id',$arr))    ? $arr['channel_id']    : 0);
-
-		$w = Zlib\NativeWiki::get_wiki($channel_id, $observer_hash, $resource_id);
-		if (!$w['wiki']) {
-			return array('history' => null, 'message' => 'Error reading wiki', 'success' => false);
-		}
-
-		$items = self::load_page_history($arr);
-
-		$history = [];
-
-		if($items) {
-			$processed = 0;
-			foreach($items as $item) {
-				if($processed > 1000)
-					break;
-				$processed ++;
-				$history[] = [ 
-					'revision' => $item['revision'],
-					'date' => datetime_convert('UTC',date_default_timezone_get(),$item['edited']),
-					'name' => $item['author']['xchan_name'],
-					'title' => get_iconfig($item,'nwikipage','commit_msg') 
-				];
-
-			}
-
-			return [ 'success' => true, 'history' => $history ];
-		}
-
-		return [ 'success' => false ];
-
-	}
-	
-
-	static public function load_page($arr) {
-
-		$pageUrlName   = ((array_key_exists('pageUrlName',$arr))   ? $arr['pageUrlName']     : '');
-		$resource_id   = ((array_key_exists('resource_id',$arr))   ? $arr['resource_id']     : '');
-		$observer_hash = ((array_key_exists('observer_hash',$arr)) ? $arr['observer_hash']   : '');
-		$channel_id    = ((array_key_exists('channel_id',$arr))    ? $arr['channel_id']      : 0);
-		$revision      = ((array_key_exists('revision',$arr))      ? $arr['revision']        : (-1));
-
-		$w = Zlib\NativeWiki::get_wiki($channel_id, $observer_hash, $resource_id);
-
-		if (! $w['wiki']) {
-			return array('content' => null, 'message' => 'Error reading wiki', 'success' => false);
-		}
-
-		$ids = '';
-
-		$ic = q("select * from iconfig left join item on iconfig.iid = item.id where uid = %d and cat = 'nwikipage' and k = 'pagetitle' and v = '%s'",
-			intval($channel_id),
-			dbesc($pageUrlName)
-		);
-
-		if($ic) {
-			foreach($ic as $c) {
-				if($ids)
-					$ids .= ',';
-				$ids .= intval($c['iid']);
-			}
-		}
-
-		$sql_extra = item_permissions_sql($channel_id,$observer_hash);
-
-		if($revision == (-1))
-			$sql_extra .= " order by revision desc ";
-		elseif($revision)
-			$sql_extra .= " and revision = " . intval($revision) . " ";
-
-		$r = null;
-
-
-		if($ids) {
-			$r = q("select * from item where resource_type = 'nwikipage' and resource_id = '%s' and uid = %d and id in ( $ids ) $sql_extra limit 1",
-				dbesc($resource_id),
-				intval($channel_id)
-			);
-
-			if($r) {
-				$items = fetch_post_tags($r,true);
-				return $items[0];
-			}
-		}
-
-		return null;
-	}
-
-	static public function load_page_history($arr) {
-
-		$pageUrlName   = ((array_key_exists('pageUrlName',$arr))   ? $arr['pageUrlName']     : '');
-		$resource_id   = ((array_key_exists('resource_id',$arr))   ? $arr['resource_id']     : '');
-		$observer_hash = ((array_key_exists('observer_hash',$arr)) ? $arr['observer_hash']   : '');
-		$channel_id    = ((array_key_exists('channel_id',$arr))    ? $arr['channel_id']      : 0);
-		$revision      = ((array_key_exists('revision',$arr))      ? $arr['revision']        : (-1));
-
-		$w = Zlib\NativeWiki::get_wiki($channel_id, $observer_hash, $resource_id);
-		if (! $w['wiki']) {
-			return array('content' => null, 'message' => 'Error reading wiki', 'success' => false);
-		}
-
-		$ids = '';
-
-		$ic = q("select * from iconfig left join item on iconfig.iid = item.id where uid = %d and cat = 'nwikipage' and k = 'pagetitle' and v = '%s'",
-			intval($channel_id),
-			dbesc($pageUrlName)
-		);
-	
-		if($ic) {
-			foreach($ic as $c) {
-				if($ids)
-					$ids .= ',';
-				$ids .= intval($c['iid']);
-			}
-		}
-
-		$sql_extra = item_permissions_sql($channel_id,$observer_hash);
-
-		$sql_extra .= " order by revision desc ";
-
-		$r = null;
-		if($ids) {
-			$r = q("select * from item where resource_type = 'nwikipage' and resource_id = '%s' and uid = %d and id in ( $ids ) and item_deleted = 0 $sql_extra",
-				dbesc($resource_id),
-				intval($channel_id)
-			);
-			if($r) {
-				xchan_query($r);
-				$items = fetch_post_tags($r,true);
-				return $items;
-			}
-		}
-
-		return null;
-	}
-
-	static public function save_page($arr) {
-
-		$pageUrlName   = ((array_key_exists('pageUrlName',$arr))   ? $arr['pageUrlName']   : '');
-		$content       = ((array_key_exists('content',$arr))       ? $arr['content']       : '');
-		$resource_id   = ((array_key_exists('resource_id',$arr))   ? $arr['resource_id']   : '');
-		$observer_hash = ((array_key_exists('observer_hash',$arr)) ? $arr['observer_hash'] : '');
-		$channel_id    = ((array_key_exists('channel_id',$arr))    ? $arr['channel_id']    : 0);
-		$revision      = ((array_key_exists('revision',$arr))      ? $arr['revision']      : 0);
-
-		$w = Zlib\NativeWiki::get_wiki($channel_id, $observer_hash, $resource_id);
-
-		if (!$w['wiki']) {
-			return array('message' => t('Error reading wiki'), 'success' => false);
-		}
-
-	
-		// fetch the most recently saved revision. 
-
-		$item = self::load_page($arr);
-
-		if(! $item) {
-			return array('message' => t('Page not found'), 'success' => false);
-		}
-
-		$mimetype = $item['mimetype'];
-
-		// change just the fields we need to change to create a revision; 
-
-		unset($item['id']);
-		unset($item['author']);
-		$item['parent']       = 0;
-		$item['body']         = $content;
-		$item['author_xchan'] = $observer_hash;
-		$item['revision']     = (($arr['revision']) ? intval($arr['revision']) + 1 : intval($item['revision']) + 1);
-		$item['edited']       = datetime_convert();
-		$item['mimetype']     = $mimetype;
-
-		if($item['iconfig'] && is_array($item['iconfig']) && count($item['iconfig'])) {
-			for($x = 0; $x < count($item['iconfig']); $x ++) {
-				unset($item['iconfig'][$x]['id']);
-				unset($item['iconfig'][$x]['iid']);
-			}
-		}
-
-		$ret = item_store($item, false, false);
-
-		if($ret['item_id'])
-			return array('message' => '', 'item_id' => $ret['item_id'], 'filename' => $filename, 'success' => true);
-		else
-			return array('message' => t('Page update failed.'), 'success' => false);
-	}	
-
-	static public function delete_page($arr) {
-		$pageUrlName = ((array_key_exists('pageUrlName',$arr)) ? $arr['pageUrlName'] : '');
-		$resource_id = ((array_key_exists('resource_id',$arr)) ? $arr['resource_id'] : '');
-		$observer_hash = ((array_key_exists('observer_hash',$arr)) ? $arr['observer_hash'] : '');
-		$channel_id    = ((array_key_exists('channel_id',$arr))    ? $arr['channel_id']    : 0);
-
-		$w = Zlib\NativeWiki::get_wiki($channel_id, $observer_hash, $resource_id);
-
-		if(! $w['wiki']) {
-			return [ 'success' => false, 'message' => t('Error reading wiki') ];
-		}
-
-		$ids = [];
-
-		$ic = q("select * from iconfig left join item on iconfig.iid = item.id 
-			where uid = %d and cat = 'nwikipage' and k = 'pagetitle' and v = '%s'",
-			intval($channel_id),
-			dbesc($pageUrlName)
-		);
-
-		if($ic) {
-			foreach($ic as $c) {
-				$ids[] = intval($c['iid']);
-			}
-		}
-
-		if($ids) {
-			drop_items($ids);
-			return [ 'success' => true ];
-		}
-
-		return [ 'success' => false, 'message' => t('Nothing deleted') ];	
-	}
-	
-	static public function revert_page($arr) {
-		$pageUrlName   = ((array_key_exists('pageUrlName',$arr))   ? $arr['pageUrlName']   : '');
-		$resource_id   = ((array_key_exists('resource_id',$arr))   ? $arr['resource_id']   : '');
-		$commitHash    = ((array_key_exists('commitHash',$arr))    ? $arr['commitHash']    : null);
-		$observer_hash = ((array_key_exists('observer_hash',$arr)) ? $arr['observer_hash'] : '');
-		$channel_id    = ((array_key_exists('channel_id',$arr))    ? $arr['channel_id']    : 0);
-
-		if (! $commitHash) {
-			return array('content' => $content, 'message' => 'No commit was provided', 'success' => false);
-		}
-
-		$w = Zlib\NativeWiki::get_wiki($channel_id, $observer_hash, $resource_id);
-		if (!$w['wiki']) {
-			return array('content' => $content, 'message' => 'Error reading wiki', 'success' => false);
-		}
-
-		$x = $arr;
-
-		if(intval($commitHash) > 0) {
-			unset($x['commitHash']);
-			$x['revision'] = intval($commitHash) - 1;
-			$loaded = self::load_page($x);
-
-			if($loaded) {
-				$content = $loaded['body'];
-				return [ 'content' => $content, 'success' => true ];
-			}
-			return [ 'content' => $content, 'success' => false ]; 
-		}
-	}
-	
-	static public function compare_page($arr) {
-		$pageUrlName = ((array_key_exists('pageUrlName',$arr)) ? $arr['pageUrlName'] : '');
-		$resource_id = ((array_key_exists('resource_id',$arr)) ? $arr['resource_id'] : '');
-		$currentCommit = ((array_key_exists('currentCommit',$arr)) ? $arr['currentCommit'] : (-1));
-		$compareCommit = ((array_key_exists('compareCommit',$arr)) ? $arr['compareCommit'] : 0);
-		$observer_hash = ((array_key_exists('observer_hash',$arr)) ? $arr['observer_hash'] : '');
-		$channel_id    = ((array_key_exists('channel_id',$arr))    ? $arr['channel_id']    : 0);
-
-		$w = Zlib\NativeWiki::get_wiki($channel_id, $observer_hash, $resource_id);
-
-		if (!$w['wiki']) {
-			return array('message' => t('Error reading wiki'), 'success' => false);
-		}
-
-		$x = $arr;
-		$x['revision'] = (-1);
-
-		$currpage = self::load_page($x);
-		if($currpage)
-			$currentContent = $currpage['body'];
-
-		$x['revision'] = $compareCommit;
-		$comppage = self::load_page($x);
-		if($comppage)
-			$compareContent = $comppage['body'];
-
-		if($currpage && $comppage) {
-			require_once('library/class.Diff.php');
-			$diff = \Diff::toTable(\Diff::compare($currentContent, $compareContent));
-
-			return [ 'success' => true, 'diff' => $diff ];
-		}
-		return [ 'success' => false, 'message' =>  t('Compare: object not found.') ];
-
-	}
-	
-	static public function commit($arr) {
-
-		$commit_msg    = ((array_key_exists('commit_msg', $arr))   ? $arr['commit_msg']    : t('Page updated'));
-		$observer_hash = ((array_key_exists('observer_hash',$arr)) ? $arr['observer_hash'] : '');
-		$channel_id    = ((array_key_exists('channel_id',$arr))    ? $arr['channel_id']    : 0);
-		$pageUrlName   = ((array_key_exists('pageUrlName',$arr))   ? $arr['pageUrlName']   : t('Untitled'));
-
-		if(array_key_exists('resource_id', $arr)) {
-			$resource_id = $arr['resource_id'];
-		}
-		else {
-			return array('message' => t('Wiki resource_id required for git commit'), 'success' => false);
-		}
-
-		$w = Zlib\NativeWiki::get_wiki($channel_id, $observer_hash, $resource_id);
-		if (! $w['wiki']) {
-			return array('message' => t('Error reading wiki'), 'success' => false);
-		}
-
-
-		$page = self::load_page($arr);
-
-		if($page) {
-			set_iconfig($page['id'],'nwikipage','commit_msg',escape_tags($commit_msg),true);
-			return [ 'success' => true, 'item_id' => $page['id'], 'page' => $page ];
-		}
-
-		return [ 'success' => false, 'message' => t('Page not found.') ];
-
-	}
-	
-	static public function convert_links($s, $wikiURL) {
-		
-		if (strpos($s,'[[') !== false) {
-			preg_match_all("/\[\[(.*?)\]\]/", $s, $match);
-			$pages = $pageURLs = array();
-			foreach ($match[1] as $m) {
-				// TODO: Why do we need to double urlencode for this to work?
-				//$pageURLs[] = urlencode(urlencode(escape_tags($m)));
-				$titleUri = explode('|',$m);
-				$page = $titleUri[0] ?? '';
-				$title = $titleUri[1] ?? $page;
-				$pageURLs[] = Zlib\NativeWiki::name_encode(escape_tags($page));
-				$pages[] = $title;
-			}
-			$idx = 0;
-			while(strpos($s,'[[') !== false) {
-			$replace = '<a href="'.$wikiURL.'/'.$pageURLs[$idx].'">'.$pages[$idx].'</a>';
-				$s = preg_replace("/\[\[(.*?)\]\]/", $replace, $s, 1);
-				$idx++;
-			}
-		}
-		return $s;
-	}
-
-	static public function render_page_history($arr) {
-
-		$pageUrlName = ((array_key_exists('pageUrlName', $arr)) ? $arr['pageUrlName'] : '');
-		$resource_id = ((array_key_exists('resource_id', $arr)) ? $arr['resource_id'] : '');
-
-		$pageHistory = self::page_history([
-			'channel_id'    => \App::$profile_uid, 
-			'observer_hash' => get_observer_hash(),
-			'resource_id'   => $resource_id,
-			'pageUrlName'   => $pageUrlName
-		]);
-
-		return replace_macros(get_markup_template('nwiki_page_history.tpl'), array(
-			'$pageHistory' => $pageHistory['history'],
-			'$permsWrite'  => $arr['permsWrite'],
-			'$name_lbl'    => t('Name'),
-			'$msg_label'   => t('Message','wiki_history'),
-			'$date_lbl'    => t('Date'),
-			'$revert_btn'  => t('Revert'),
-			'$compare_btn' => t('Compare')
-		));
-
-	}
-
-
-	
-	/**
-	 * Replace the instances of the string [toc] with a list element that will be populated by
-	 * a table of contents by the JavaScript library
-	 * @param string $s
-	 * @return string
-	 */
-	static public function generate_toc($s) {
-		if (strpos($s,'[toc]') !== false) {
-			//$toc_md = wiki_toc($s);	// Generate Markdown-formatted list prior to HTML render
-			$toc_md = '<ul id="wiki-toc"></ul>'; // use the available jQuery plugin http://ndabas.github.io/toc/
-			$s = preg_replace("/\[toc\]/", $toc_md, $s, -1);
-		}
-		return $s;
-	}
-	
-	/**
-	 *  Converts a select set of bbcode tags. Much of the code is copied from include/bbcode.php
-	 * @param string $s
-	 * @return string
-	 */
-	static public function bbcode($s) {
-			
-		$s = str_replace(array('[baseurl]', '[sitename]'), array(z_root(), get_config('system', 'sitename')), $s);
-			
-		$s = preg_replace_callback("/\[observer\.language\=(.*?)\](.*?)\[\/observer\]/ism",'oblanguage_callback', $s);
-
-		$s = preg_replace_callback("/\[observer\.language\!\=(.*?)\](.*?)\[\/observer\]/ism",'oblanguage_necallback', $s);
-
-
-		$observer = \App::get_observer();
-		if ($observer) {
-			$s1 = '<span class="bb_observer" title="' . t('Different viewers will see this text differently') . '">';
-			$s2 = '</span>';
-			$obsBaseURL = $observer['xchan_connurl'];
-			$obsBaseURL = preg_replace("/\/poco\/.*$/", '', $obsBaseURL);
-			$s = str_replace('[observer.baseurl]', $obsBaseURL, $s);
-			$s = str_replace('[observer.url]', $observer['xchan_url'], $s);
-			$s = str_replace('[observer.name]', $s1 . $observer['xchan_name'] . $s2, $s);
-			$s = str_replace('[observer.address]', $s1 . $observer['xchan_addr'] . $s2, $s);
-			$s = str_replace('[observer.webname]', substr($observer['xchan_addr'], 0, strpos($observer['xchan_addr'], '@')), $s);
-			$s = str_replace('[observer.photo]', '', $s);
-		} 
-		else {
-			$s = str_replace('[observer.baseurl]', '', $s);
-			$s = str_replace('[observer.url]', '', $s);
-			$s = str_replace('[observer.name]', '', $s);
-			$s = str_replace('[observer.address]', '', $s);
-			$s = str_replace('[observer.webname]', '', $s);
-			$s = str_replace('[observer.photo]', '', $s);
-		}
-
-		return $s;
-	}
-	
-	static public function get_file_ext($arr) {
-		if($arr['mimetype'] === 'text/bbcode')
-			return '.bb';
-		elseif($arr['mimetype'] === 'text/markdown')
-			return '.md';
-		elseif($arr['mimetype'] === 'text/plain')
-			return '.txt';
-
-	}
-	
-	// This function is derived from 
-	// http://stackoverflow.com/questions/32068537/generate-table-of-contents-from-markdown-in-php
-	static public function toc($content) {
-	  // ensure using only "\n" as line-break
-	  $source = str_replace(["\r\n", "\r"], "\n", $content);
-	
-	  // look for markdown TOC items
-	  preg_match_all(
-		'/^(?:=|-|#).*$/m',
-		$source,
-		$matches,
-		PREG_PATTERN_ORDER | PREG_OFFSET_CAPTURE
-	  );
-	
-	  // preprocess: iterate matched lines to create an array of items
-	  // where each item is an array(level, text)
-	  $file_size = strlen($source);
-	  foreach ($matches[0] as $item) {
-		$found_mark = substr($item[0], 0, 1);
-		if ($found_mark == '#') {
-		  // text is the found item
-		  $item_text = $item[0];
-		  $item_level = strrpos($item_text, '#') + 1;
-		  $item_text = substr($item_text, $item_level);
-		} else {
-		  // text is the previous line (empty if <hr>)
-		  $item_offset = $item[1];
-		  $prev_line_offset = strrpos($source, "\n", -($file_size - $item_offset + 2));
-		  $item_text =
-			substr($source, $prev_line_offset, $item_offset - $prev_line_offset - 1);
-		  $item_text = trim($item_text);
-		  $item_level = $found_mark == '=' ? 1 : 2;
-		}
-		if (!trim($item_text) OR strpos($item_text, '|') !== FALSE) {
-		  // item is an horizontal separator or a table header, don't mind
-		  continue;
-		}
-		$raw_toc[] = ['level' => $item_level, 'text' => trim($item_text)];
-	  }
-		$o = '';
-		foreach($raw_toc as $t) {
-			$level = intval($t['level']);
-			$text = $t['text'];
-			switch ($level) {
-				case 1:
-					$li = '* ';
-					break;
-				case 2:
-					$li = '  * ';
-					break;
-				case 3:
-					$li = '    * ';
-					break;
-				case 4:
-					$li = '      * ';
-					break;
-				default:
-					$li = '* ';
-					break;
-			}
-			$o .= $li . $text . "\n";
-		}
-	  return $o;
-	}
-
-}

Zotlabs/Lib/PConfig.php

@@ -2,6 +2,8 @@
 
 namespace Zotlabs\Lib;
 
+use App;
+
 /**
  * @brief Class for handling channel specific configurations.
  *
@@ -32,15 +34,15 @@ class PConfig {
 		if(is_null($uid) || $uid === false)
 			return false;
 
-		if(! is_array(\App::$config)) {
+		if(! is_array(App::$config)) {
 			btlogger('App::$config not an array');
 		}
 
-		if(! array_key_exists($uid, \App::$config)) {
-			\App::$config[$uid] = array();
+		if(! array_key_exists($uid, App::$config)) {
+			App::$config[$uid] = array();
 		}
 
-		if(! is_array(\App::$config[$uid])) {
+		if(! is_array(App::$config[$uid])) {
 			btlogger('App::$config[$uid] not an array: ' . $uid);
 		}
 
@@ -52,12 +54,12 @@ class PConfig {
 			foreach($r as $rr) {
 				$k = $rr['k'];
 				$c = $rr['cat'];
-				if(! array_key_exists($c, \App::$config[$uid])) {
-					\App::$config[$uid][$c] = array();
-					\App::$config[$uid][$c]['config_loaded'] = true;
+				if(! array_key_exists($c, App::$config[$uid])) {
+					App::$config[$uid][$c] = array();
+					App::$config[$uid][$c]['config_loaded'] = true;
 				}
-				\App::$config[$uid][$c][$k] = $rr['v'];
-				\App::$config[$uid][$c]['pcfgud:'.$k] = $rr['updated'];
+				App::$config[$uid][$c][$k] = $rr['v'];
+				App::$config[$uid][$c]['pcfgud:'.$k] = $rr['updated'];
 			}
 		}
 	}
@@ -86,15 +88,15 @@ class PConfig {
 		if(is_null($uid) || $uid === false)
 			return $default;
 
-		if(! array_key_exists($uid, \App::$config))
+		if(! array_key_exists($uid, App::$config))
 			self::Load($uid);
 
-		if((! array_key_exists($family, \App::$config[$uid])) || (! array_key_exists($key, \App::$config[$uid][$family])))
+		if((! array_key_exists($family, App::$config[$uid])) || (! array_key_exists($key, App::$config[$uid][$family])))
 			return $default;
 
-		return ((! is_array(\App::$config[$uid][$family][$key])) && (preg_match('|^a:[0-9]+:{.*}$|s', \App::$config[$uid][$family][$key]))
-			? unserialize(\App::$config[$uid][$family][$key])
-			: \App::$config[$uid][$family][$key]
+		return ((! is_array(App::$config[$uid][$family][$key])) && (preg_match('|^a:[0-9]+:{.*}$|s', App::$config[$uid][$family][$key]))
+			? unserialize(App::$config[$uid][$family][$key])
+			: App::$config[$uid][$family][$key]
 		);
 	}
 
@@ -132,6 +134,8 @@ class PConfig {
 		// manage array value
 		$dbvalue = ((is_array($value))  ? serialize($value) : $value);
 		$dbvalue = ((is_bool($dbvalue)) ? intval($dbvalue)  : $dbvalue);
+		$new = false;
+		$update = false;
 
 		$now = datetime_convert();
 		if (! $updated) {
@@ -142,23 +146,22 @@ class PConfig {
 			$updated = datetime_convert('UTC','UTC','-2 seconds');
 		}
 
-		$hash = hash('sha256',$family.':'.$key);
+		$hash = gen_link_id($family.':'.$key);
 
 		if (self::Get($uid, 'hz_delpconfig', $hash) !== false) {
 			if (self::Get($uid, 'hz_delpconfig', $hash) > $now) {
 				logger('Refusing to update pconfig with outdated info (Item deleted more recently).', LOGGER_NORMAL, LOG_ERR);
 				return self::Get($uid,$family,$key);
 			} else {
-				self::Delete($uid,'hz_delpconfig',$hash);
+				self::Delete($uid, 'hz_delpconfig', $hash);
 			}
 		}
 
 		if(self::Get($uid, $family, $key) === false) {
-			if(! array_key_exists($uid, \App::$config))
-				\App::$config[$uid] = array();
-			if(! array_key_exists($family, \App::$config[$uid]))
-				\App::$config[$uid][$family] = array();
-
+			if(! array_key_exists($uid, App::$config))
+				App::$config[$uid] = array();
+			if(! array_key_exists($family, App::$config[$uid]))
+				App::$config[$uid][$family] = array();
 
 			$ret = q("INSERT INTO pconfig ( uid, cat, k, v, updated ) VALUES ( %d, '%s', '%s', '%s', '%s' ) ",
 				intval($uid),
@@ -176,13 +179,14 @@ class PConfig {
 				logger("Error: Insert to pconfig failed.",LOGGER_NORMAL, LOG_ERR);
 			}
 
-			\App::$config[$uid][$family]['pcfgud:'.$key] = $updated;
+			$new = true;
+			App::$config[$uid][$family]['pcfgud:'.$key] = $updated;
 
 		}
 		else {
-			$new = (\App::$config[$uid][$family]['pcfgud:'.$key] < $now);
+			$update = (App::$config[$uid][$family]['pcfgud:'.$key] < $now);
 
-			if ($new) {
+			if ($update) {
 
 				// @NOTE There is still a possible race condition under limited circumstances
 				// where a value will be updated by another thread with more current data than
@@ -197,7 +201,7 @@ class PConfig {
 					dbesc($key)
 				);
 
-				\App::$config[$uid][$family]['pcfgud:'.$key] = $updated;
+				App::$config[$uid][$family]['pcfgud:'.$key] = $updated;
 
 			} else {
 				logger('Refusing to update pconfig with outdated info.', LOGGER_NORMAL, LOG_ERR);
@@ -210,16 +214,16 @@ class PConfig {
 		// set in the life of this page. We need this to
 		// synchronise channel clones.
 
-		if(! array_key_exists('transient', \App::$config[$uid]))
-			\App::$config[$uid]['transient'] = array();
-		if(! array_key_exists($family, \App::$config[$uid]['transient']))
-			\App::$config[$uid]['transient'][$family] = array();
+		if(! array_key_exists('transient', App::$config[$uid]))
+			App::$config[$uid]['transient'] = array();
+		if(! array_key_exists($family, App::$config[$uid]['transient']))
+			App::$config[$uid]['transient'][$family] = array();
 
-		\App::$config[$uid][$family][$key] = $value;
+		App::$config[$uid][$family][$key] = $value;
 
-		if ($new) {
-			\App::$config[$uid]['transient'][$family][$key] = $value;
-			\App::$config[$uid]['transient'][$family]['pcfgud:'.$key] = $updated;
+		if ($new || $update) {
+			App::$config[$uid]['transient'][$family][$key] = $value;
+			App::$config[$uid]['transient'][$family]['pcfgud:'.$key] = $updated;
 		}
 
 		if($ret)
@@ -252,7 +256,7 @@ class PConfig {
 
 		$updated = ($updated) ? $updated : datetime_convert('UTC','UTC','-2 seconds');
 		$now = datetime_convert();
-		$newer = (\App::$config[$uid][$family]['pcfgud:'.$key] < $now);
+		$newer = (App::$config[$uid][$family]['pcfgud:'.$key] < $now);
 
 		if (! $newer) {
 			logger('Refusing to delete pconfig with outdated delete request.', LOGGER_NORMAL, LOG_ERR);
@@ -261,12 +265,12 @@ class PConfig {
 
 		$ret = false;
 
-		if (isset(\App::$config[$uid][$family][$key])) {
-			unset(\App::$config[$uid][$family][$key]);
+		if (isset(App::$config[$uid][$family][$key])) {
+			unset(App::$config[$uid][$family][$key]);
 		}
 
-		if (isset(\App::$config[$uid][$family]['pcfgud:'.$key])) {
-			unset(\App::$config[$uid][$family]['pcfgud:'.$key]);
+		if (isset(App::$config[$uid][$family]['pcfgud:'.$key])) {
+			unset(App::$config[$uid][$family]['pcfgud:'.$key]);
 		}
 
 		$ret = q("DELETE FROM pconfig WHERE uid = %d AND cat = '%s' AND k = '%s'",
@@ -277,9 +281,9 @@ class PConfig {
 
 		// Synchronize delete with clones.
 
-		if ($family != 'hz_delpconfig') {
-			$hash = hash('sha256',$family.':'.$key);
-			set_pconfig($uid,'hz_delpconfig',$hash,$updated);
+		if ($family !== 'hz_delpconfig') {
+			$hash = gen_link_id($family.':'.$key);
+			set_pconfig($uid, 'hz_delpconfig', $hash, $updated);
 		}
 
 		return $ret;

Zotlabs/Lib/Permcat.php

@@ -4,6 +4,7 @@ namespace Zotlabs\Lib;
 
 use Zotlabs\Access\PermissionRoles;
 use Zotlabs\Access\Permissions;
+use Zotlabs\Daemon\Master;
 
 /**
  * @brief Permission Categories. Permission rules for various classes of connections.
@@ -38,33 +39,33 @@ class Permcat {
 
 		// first check role perms for a perms_connect setting
 
-		$role = get_pconfig($channel_id,'system','permissions_role');
-		if($role) {
+		$role = get_pconfig($channel_id, 'system', 'permissions_role');
+		if ($role) {
 			$x = PermissionRoles::role_perms($role);
-			if($x['perms_connect']) {
+			if ($x['perms_connect']) {
 				$perms = Permissions::FilledPerms($x['perms_connect']);
 			}
 		}
 
 		// if no role perms it may be a custom role, see if there any autoperms
 
-		if(! $perms) {
+		if (!$perms) {
 			$perms = Permissions::FilledAutoPerms($channel_id);
 		}
 
 		// if no autoperms it may be a custom role with manual perms
 
-		if(! $perms) {
+		if (!$perms) {
 			$r = q("select channel_hash from channel where channel_id = %d",
 				intval($channel_id)
 			);
-			if($r) {
+			if ($r) {
 				$x = q("select * from abconfig where chan = %d and xchan = '%s' and cat = 'my_perms'",
 					intval($channel_id),
 					dbesc($r[0]['channel_hash'])
 				);
-				if($x) {
-					foreach($x as $xv) {
+				if ($x) {
+					foreach ($x as $xv) {
 						$perms[$xv['k']] = intval($xv['v']);
 					}
 				}
@@ -73,25 +74,27 @@ class Permcat {
 
 		// nothing was found - create a filled permission array where all permissions are 0
 
-		if(! $perms) {
+		if (!$perms) {
 			$perms = Permissions::FilledPerms([]);
 		}
 
 		$this->permcats[] = [
 			'name'      => 'default',
-			'localname' => t('default','permcat'),
+			'localname' => t('Default', 'permcat'),
 			'perms'     => Permissions::Operms($perms),
+			'raw_perms' => $perms,
 			'system'    => 1
 		];
 
 
 		$p = $this->load_permcats($channel_id);
-		if($p) {
-			for($x = 0; $x < count($p); $x++) {
+		if ($p) {
+			for ($x = 0; $x < count($p); $x++) {
 				$this->permcats[] = [
 					'name'      => $p[$x][0],
 					'localname' => $p[$x][1],
 					'perms'     => Permissions::Operms(Permissions::FilledPerms($p[$x][2])),
+					'raw_perms' => Permissions::FilledPerms($p[$x][2]),
 					'system'    => intval($p[$x][3])
 				];
 			}
@@ -116,9 +119,9 @@ class Permcat {
 	 *   * \e bool \b error if $name not found in permcats true
 	 */
 	public function fetch($name) {
-		if($name && $this->permcats) {
-			foreach($this->permcats as $permcat) {
-				if(strcasecmp($permcat['name'], $name) === 0) {
+		if ($name && $this->permcats) {
+			foreach ($this->permcats as $permcat) {
+				if (strcasecmp($permcat['name'], $name) === 0) {
 					return $permcat;
 				}
 			}
@@ -128,31 +131,28 @@ class Permcat {
 	}
 
 	public function load_permcats($uid) {
-
+		/*
 		$permcats = [
-			[ 'follower', t('follower','permcat'),
-				[ 'view_stream','view_profile','view_contacts','view_storage','view_pages','view_wiki',
-				  'post_like' ], 1
-			],
-			[ 'contributor', t('contributor','permcat'),
-				[ 'view_stream','view_profile','view_contacts','view_storage','view_pages','view_wiki',
-				  'post_wall','post_comments','write_wiki','post_like','tag_deliver','chat' ], 1
-			],
-			[ 'publisher', t('publisher','permcat'),
+			[ 'contributor', t('Contributor','permcat'),
 				[ 'view_stream','view_profile','view_contacts','view_storage','view_pages',
-				  'write_storage','post_wall','write_pages','write_wiki','post_comments','post_like','tag_deliver',
-				  'chat', 'republish' ], 1
-			]
+				  'write_storage','post_wall','write_pages','write_wiki','post_comments', 'post_mail', 'post_like',
+				  'chat' ], 1
+			],
+			[ 'muted', t('Muted','permcat'),
+				[ 'view_stream','view_profile','view_contacts','view_storage','view_pages','view_wiki',
+				  'post_comments','write_wiki','post_like' ], 1
+			],
 		];
-
-		if($uid) {
+		*/
+		if ($uid) {
 			$x = q("select * from pconfig where uid = %d and cat = 'permcat'",
 				intval($uid)
 			);
-			if($x) {
-				foreach($x as $xv) {
-					$value = ((preg_match('|^a:[0-9]+:{.*}$|s', $xv['v'])) ? unserialize($xv['v']) : $xv['v']);
-					$permcats[] = [ $xv['k'], $xv['k'], $value, 0 ];
+
+			if ($x) {
+				foreach ($x as $xv) {
+					$value      = ((preg_match('|^a:[0-9]+:{.*}$|s', $xv['v'])) ? unserialize($xv['v']) : $xv['v']);
+					$permcats[] = [$xv['k'], $xv['k'], $value, 0];
 				}
 			}
 		}
@@ -167,11 +167,11 @@ class Permcat {
 	}
 
 	static public function find_permcat($arr, $name) {
-		if((! $arr) || (! $name))
+		if ((!$arr) || (!$name))
 			return false;
 
-		foreach($arr as $p)
-			if($p['name'] == $name)
+		foreach ($arr as $p)
+			if ($p['name'] == $name)
 				return $p['value'];
 	}
 
@@ -183,4 +183,105 @@ class Permcat {
 		PConfig::Delete($channel_id, 'permcat', $name);
 	}
 
-}
+	/**
+	 * @brief assign a contact role to contacts
+	 *
+	 * @param array $channel
+	 * @param string $role the name of the role
+	 * @param array $contacts an array of contact hashes
+	 */
+	public static function assign($channel, $role, $contacts) {
+
+		if (!isset($channel['channel_id'])) {
+			return;
+		}
+
+		if (!is_array($contacts) || empty($contacts)) {
+			return;
+		}
+
+		if (!$role) {
+			// lookup the default
+			$role = get_pconfig($channel['channel_id'], 'system', 'default_permcat', 'default');
+		}
+
+
+		// Doublecheck that we do not assign a role to ourself.
+		// It does not make a difference but could be confusing.
+		if (in_array($channel['channel_hash'], $contacts)) {
+			$contacts = array_diff($contacts, [$channel['channel_hash']]);
+		}
+
+		$all_perms  = Permissions::Perms();
+		$permcats   = new Permcat($channel['channel_id']);
+		$role_perms = $permcats->fetch($role);
+
+		if (isset($role_perms['error'])) {
+			return false;
+		}
+
+		$perms = $role_perms['raw_perms'];
+
+		$values_sql = '';
+		stringify_array_elms($contacts, true);
+
+		if ($all_perms && $perms) {
+
+			foreach ($contacts as $contact) {
+				foreach ($all_perms as $perm => $desc) {
+					if (array_key_exists($perm, $perms)) {
+						$values_sql .= " (" . intval($channel['channel_id']) . ", " . protect_sprintf($contact) . ", 'my_perms', '" . dbesc($perm) . "', " . intval($perms[$perm]) . "),";
+					}
+					else {
+						$values_sql .= " (" . intval($channel['channel_id']) . ", " . protect_sprintf($contact) . ", 'my_perms', '" . dbesc($perm) . "', 0), ";
+					}
+				}
+			}
+		}
+
+		$values_sql = rtrim($values_sql, ',');
+
+		dbq("DELETE FROM abconfig WHERE chan = " . intval($channel['channel_id']) . " AND cat = 'my_perms' AND xchan IN (" . protect_sprintf(implode(',', $contacts)) . ")");
+
+		dbq("INSERT INTO abconfig ( chan, xchan, cat, k, v ) VALUES $values_sql");
+
+		q("UPDATE abook SET abook_role = '%s'
+			WHERE abook_xchan IN (" . protect_sprintf(implode(',', $contacts)) . ") AND abook_channel = %d",
+			dbesc($role),
+			intval($channel['channel_id'])
+		);
+
+		$r = q("SELECT abook.*, xchan.* FROM abook LEFT JOIN xchan ON abook.abook_xchan = xchan.xchan_hash WHERE abook.abook_xchan IN (" . protect_sprintf(implode(',', $contacts)) . ") AND abook.abook_channel = %d AND abook_self = 0",
+			intval($channel['channel_id'])
+		);
+
+		foreach ($r as $rr) {
+
+			if (intval($rr['abook_self'])) {
+				continue;
+			}
+
+			Master::Summon([
+				'Notifier',
+				'permission_update',
+				$rr['abook_id']
+			]);
+
+			$clone = $rr;
+
+			unset($clone['abook_id']);
+			unset($clone['abook_account']);
+			unset($clone['abook_channel']);
+
+			$abconfig = load_abconfig($channel['channel_id'], $clone['abook_xchan']);
+			if ($abconfig)
+				$clone['abconfig'] = $abconfig;
+
+			Libsync::build_sync_packet(0 /* use the current local_channel */, ['abook' => [$clone]]);
+
+		}
+
+		return true;
+	}
+
+}

Zotlabs/Lib/Queue.php

@@ -2,8 +2,8 @@
 
 namespace Zotlabs\Lib;
 
-use Zotlabs\Lib\Libzot;
-
+use Zotlabs\Zot6\Receiver;
+use Zotlabs\Zot6\Zot6Handler;
 
 class Queue {
 
@@ -31,19 +31,19 @@ class Queue {
 			$might_be_down = ((datetime_convert('UTC','UTC',$y[0]['earliest']) < datetime_convert('UTC','UTC','now - 2 days')) ? true : false);
 
 
-		// Set all other records for this destination way into the future. 
+		// Set all other records for this destination way into the future.
 		// The queue delivers by destination. We'll keep one queue item for
 		// this destination (this one) with a shorter delivery. If we succeed
 		// once, we'll try to deliver everything for that destination.
-		// The delivery will be set to at most once per hour, and if the 
+		// The delivery will be set to at most once per hour, and if the
 		// queue item is less than 12 hours old, we'll schedule for fifteen
-		// minutes. 
+		// minutes.
 
-		$r = q("UPDATE outq SET outq_scheduled = '%s' WHERE outq_posturl = '%s'",
+		q("UPDATE outq SET outq_scheduled = '%s' WHERE outq_posturl = '%s'",
 			dbesc(datetime_convert('UTC','UTC','now + 5 days')),
 			dbesc($x[0]['outq_posturl'])
 		);
- 
+
 		$since = datetime_convert('UTC','UTC',$x[0]['outq_created']);
 
 		if(($might_be_down) || ($since < datetime_convert('UTC','UTC','now - 12 hour'))) {
@@ -53,11 +53,10 @@ class Queue {
 			$next = datetime_convert('UTC','UTC','now + ' . intval($add_priority) . ' minutes');
 		}
 
-		q("UPDATE outq SET outq_updated = '%s', 
-			outq_priority = outq_priority + %d, 
-			outq_scheduled = '%s' 
+		q("UPDATE outq SET outq_updated = '%s',
+			outq_priority = outq_priority + %d,
+			outq_scheduled = '%s'
 			WHERE outq_hash = '%s'",
-
 			dbesc(datetime_convert()),
 			intval($add_priority),
 			dbesc($next),
@@ -65,33 +64,53 @@ class Queue {
 		);
 	}
 
-
-	static function remove($id,$channel_id = 0) {
-		logger('queue: remove queue item ' . $id,LOGGER_DEBUG);
+	public static function remove($id, $channel_id = 0) {
+		logger('queue: remove queue item ' . $id, LOGGER_DEBUG);
 		$sql_extra = (($channel_id) ? " and outq_channel = " . intval($channel_id) . " " : '');
-		
-		q("DELETE FROM outq WHERE outq_hash = '%s' $sql_extra",
+
+		// figure out what endpoint it is going to.
+		$record = q("select outq_posturl from outq where outq_hash = '%s' $sql_extra",
 			dbesc($id)
 		);
-	}
 
+		if ($record) {
+			q("DELETE FROM outq WHERE outq_hash = '%s' $sql_extra",
+				dbesc($id)
+			);
+
+			// If there's anything remaining in the queue for this site, move one of them to the next active
+			// queue run by setting outq_scheduled back to the present. We may be attempting to deliver it
+			// as a 'piled_up' delivery, but this ensures the site has an active queue entry as long as queued
+			// entries still exist for it. This fixes an issue where one immediate delivery left everything
+			// else for that site undeliverable since all the other entries had been pushed far into the future.
+
+			$r = q("SELECT outq_hash, outq_posturl FROM outq WHERE outq_posturl = '%s' LIMIT 1",
+				dbesc($record[0]['outq_posturl'])
+			);
+
+			if ($r) {
+				$hashes = ids_to_querystr($r, 'outq_hash', true);
+				$x = q("UPDATE outq SET outq_scheduled = '%s' WHERE outq_hash IN ($hashes)",
+					dbesc(datetime_convert())
+				);
+			}
+		}
+	}
 
 	static function remove_by_posturl($posturl) {
 		logger('queue: remove queue posturl ' . $posturl,LOGGER_DEBUG);
-		
+
 		q("DELETE FROM outq WHERE outq_posturl = '%s' ",
 			dbesc($posturl)
 		);
 	}
 
-
-
 	static function set_delivered($id,$channel = 0) {
 		logger('queue: set delivered ' . $id,LOGGER_DEBUG);
-		$sql_extra = (($channel_id) ? " and outq_channel = " . intval($channel_id) . " " : '');
+		$sql_extra = (($channel['channel_id']) ? " and outq_channel = " . intval($channel['channel_id']) . " " : '');
 
 		// Set the next scheduled run date so far in the future that it will be expired
-		// long before it ever makes it back into the delivery chain. 
+		// long before it ever makes it back into the delivery chain.
 
 		q("update outq set outq_delivered = 1, outq_updated = '%s', outq_scheduled = '%s' where outq_hash = '%s' $sql_extra ",
 			dbesc(datetime_convert()),
@@ -110,21 +129,30 @@ class Queue {
 			return false;
 		}
 
+		$hash = $arr['hash'] ?? '';
+		$account_id = $arr['account_id'] ?? 0;
+		$channel_id = $arr['channel_id'] ?? 0;
+		$driver = $arr['driver'] ?? 'zot6';
+		$posturl = $arr['posturl'] ?? '';
+		$priority = $arr['priority'] ?? 0;
+		$notify = $arr['notify'] ?? '';
+		$msg = $arr['msg'] ?? '';
+
 		$x = q("insert into outq ( outq_hash, outq_account, outq_channel, outq_driver, outq_posturl, outq_async, outq_priority,
-			outq_created, outq_updated, outq_scheduled, outq_notify, outq_msg ) 
+			outq_created, outq_updated, outq_scheduled, outq_notify, outq_msg )
 			values ( '%s', %d, %d, '%s', '%s', %d, %d, '%s', '%s', '%s', '%s', '%s' )",
-			dbesc($arr['hash']),
-			intval($arr['account_id']),
-			intval($arr['channel_id']),
-			dbesc(($arr['driver']) ? $arr['driver'] : 'zot6'),
-			dbesc($arr['posturl']),
+			dbesc($hash),
+			intval($account_id),
+			intval($channel_id),
+			dbesc($driver),
+			dbesc($posturl),
 			intval(1),
-			intval(($arr['priority']) ? $arr['priority'] : 0),
+			intval($priority),
 			dbesc(datetime_convert()),
 			dbesc(datetime_convert()),
 			dbesc(datetime_convert()),
-			dbesc($arr['notify']),
-			dbesc(($arr['msg']) ? $arr['msg'] : '')
+			dbesc($notify),
+			dbesc($msg)
 		);
 		return $x;
 
@@ -136,24 +164,26 @@ class Queue {
 
 		$base = null;
 		$h = parse_url($outq['outq_posturl']);
-		if($h !== false) 
-			$base = $h['scheme'] . '://' . $h['host'] . (($h['port']) ? ':' . $h['port'] : '');
+		if($h !== false)
+			$base = $h['scheme'] . '://' . $h['host'] . (isset($h['port']) ? ':' . $h['port'] : '');
 
 		if(($base) && ($base !== z_root()) && ($immediate)) {
 			$y = q("select site_update, site_dead from site where site_url = '%s' ",
 				dbesc($base)
 			);
-			if($y) {
-				if(intval($y[0]['site_dead'])) {
+
+			if ($y) {
+				// Don't bother delivering if the site is dead.
+				// And if we haven't heard from the site in over a month - let them through but 3 strikes you're out.
+				if (intval($y[0]['site_dead']) || ($y[0]['site_update'] < datetime_convert('UTC', 'UTC', 'now - 1 month') && $outq['outq_priority'] > 20)) {
+					q("update dreport set dreport_result = '%s' where dreport_queue = '%s'",
+						dbesc('site dead'),
+						dbesc($outq['outq_hash'])
+					);
 					self::remove_by_posturl($outq['outq_posturl']);
 					logger('dead site ignored ' . $base);
 					return;
 				}
-				if($y[0]['site_update'] < datetime_convert('UTC','UTC','now - 1 month')) {
-					self::update($outq['outq_hash'],10);
-					logger('immediate delivery deferred for site ' . $base);
-					return;
-				}
 			}
 			else {
 
@@ -161,12 +191,12 @@ class Queue {
 				// your site has existed. Since we don't know for sure what these sites are,
 				// call them unknown
 
-				site_store_lowlevel( 
+				site_store_lowlevel(
 					[
 						'site_url'    => $base,
 						'site_update' => datetime_convert(),
 						'site_dead'   => 0,
-						'site_type'   => intval(($outq['outq_driver'] === 'post') ? SITE_TYPE_NOTZOT : SITE_TYPE_UNKNOWN),
+						'site_type'   => SITE_TYPE_UNKNOWN,
 						'site_crypto' => ''
 					]
 				);
@@ -174,67 +204,18 @@ class Queue {
 		}
 
 		$arr = array('outq' => $outq, 'base' => $base, 'handled' => false, 'immediate' => $immediate);
-		call_hooks('queue_deliver',$arr);
+		call_hooks('queue_deliver', $arr);
 		if($arr['handled'])
 			return;
 
-		// "post" queue driver - used for diaspora and friendica-over-diaspora communications.
-
-		if($outq['outq_driver'] === 'post') {
-			$result = z_post_url($outq['outq_posturl'],$outq['outq_msg']);
-			if($result['success'] && $result['return_code'] < 300) {
-				logger('deliver: queue post success to ' . $outq['outq_posturl'], LOGGER_DEBUG);
-				if($base) {
-					q("update site set site_update = '%s', site_dead = 0 where site_url = '%s' ",
-						dbesc(datetime_convert()),
-						dbesc($base)
-					);
-				}
-				q("update dreport set dreport_result = '%s', dreport_time = '%s' where dreport_queue = '%s'",
-					dbesc('accepted for delivery'),
-					dbesc(datetime_convert()),
-					dbesc($outq['outq_hash'])
-				);
-				self::remove($outq['outq_hash']);
-
-				// server is responding - see if anything else is going to this destination and is piled up 
-				// and try to send some more. We're relying on the fact that do_delivery() results in an 
-				// immediate delivery otherwise we could get into a queue loop. 
-
-				if(! $immediate) {
-					$x = q("select outq_hash from outq where outq_posturl = '%s' and outq_delivered = 0",
-						dbesc($outq['outq_posturl'])
-					);
-	
-					$piled_up = array();
-					if($x) {
-						foreach($x as $xx) {
-							 $piled_up[] = $xx['outq_hash'];
-						}
-					}
-					if($piled_up) {
-						// call do_delivery() with the force flag
-						do_delivery($piled_up, true);
-					}
-				}
-			}
-			else {
-				logger('deliver: queue post returned ' . $result['return_code'] 
-					. ' from ' . $outq['outq_posturl'],LOGGER_DEBUG);
-					self::update($outq['outq_hash'],10);
-			}
-			return;
-		}
-
 		// normal zot delivery
 
 		logger('deliver: dest: ' . $outq['outq_posturl'], LOGGER_DEBUG);
 
-
 		if($outq['outq_posturl'] === z_root() . '/zot') {
 			// local delivery
-			$zot = new \Zotlabs\Zot6\Receiver(new \Zotlabs\Zot6\Zot6Handler(),$outq['outq_notify']);
-			$result = $zot->run(true);
+			$zot = new Receiver(new Zot6Handler(), $outq['outq_notify']);
+			$result = $zot->run();
 			logger('returned_json: ' . json_encode($result,JSON_PRETTY_PRINT|JSON_UNESCAPED_SLASHES), LOGGER_DATA);
 			logger('deliver: local zot delivery succeeded to ' . $outq['outq_posturl']);
 			Libzot::process_response($outq['outq_posturl'],[ 'success' => true, 'body' => json_encode($result) ], $outq);
@@ -244,13 +225,14 @@ class Queue {
 			$channel = null;
 
 			if($outq['outq_channel']) {
-				$channel = channelx_by_n($outq['outq_channel']);
+				$channel = channelx_by_n($outq['outq_channel'], true);
 			}
 
 			$host_crypto = null;
 
 			if($channel && $base) {
-				$h = q("select hubloc_sitekey, site_crypto from hubloc left join site on hubloc_url = site_url where site_url = '%s' and hubloc_sitekey != '' order by hubloc_id desc limit 1",
+				$h = q("SELECT hubloc_sitekey, site_crypto FROM hubloc LEFT JOIN site ON hubloc_url = site_url
+					WHERE site_url = '%s' AND hubloc_network = 'zot6' AND hubloc_deleted = 0 ORDER BY hubloc_primary DESC, hubloc_id DESC LIMIT 1",
 					dbesc($base)
 				);
 				if($h) {
@@ -260,16 +242,16 @@ class Queue {
 
 			$msg = $outq['outq_notify'];
 
-			$result = Libzot::zot($outq['outq_posturl'],$msg,$channel,$host_crypto);
+			$result = Libzot::zot($outq['outq_posturl'], $msg, $channel, $host_crypto);
 
 			if($result['success']) {
 				logger('deliver: remote zot delivery succeeded to ' . $outq['outq_posturl']);
-				Libzot::process_response($outq['outq_posturl'],$result, $outq);
+				Libzot::process_response($outq['outq_posturl'], $result, $outq);
 			}
 			else {
 				logger('deliver: remote zot delivery failed to ' . $outq['outq_posturl']);
 				logger('deliver: remote zot delivery fail data: ' . print_r($result,true), LOGGER_DATA);
-				self::update($outq['outq_hash'],10);
+				self::update($outq['outq_hash'], 10);
 			}
 		}
 		return;

Zotlabs/Lib/QueueWorker.php

@@ -0,0 +1,375 @@
+<?php
+
+namespace Zotlabs\Lib;
+
+use Ramsey\Uuid\Uuid;
+use Ramsey\Uuid\Exception\UnableToBuildUuidException;
+use Zotlabs\Lib\Config;
+
+require_once 'include/dba/dba_transaction.php';
+
+class QueueWorker {
+
+	public static $queueworker = null;
+	public static $maxworkers = 0;
+	public static $workermaxage = 0;
+	public static $workersleep = 100;
+	public static $default_priorities = [
+		'Notifier'         => 10,
+		'Deliver'          => 10,
+		'Cache_query'      => 10,
+		'Content_importer' => 1,
+		'File_importer'    => 1,
+		'Channel_purge'    => 1,
+		'Directory'        => 1
+	];
+
+	// Exceptions for processtimeout ($workermaxage) value.
+	// Currently the value is overriden with 3600 seconds (1h).
+	public static $long_running_cmd = [
+		'Queue',
+		'Expire'
+	];
+
+	public static function Summon($argv) {
+
+		if ($argv[0] !== 'Queueworker') {
+
+			$priority = 0; // @TODO allow reprioritization
+
+			if (isset(self::$default_priorities[$argv[0]])) {
+				$priority = self::$default_priorities[$argv[0]];
+			}
+
+			$workinfo      = ['argc' => count($argv), 'argv' => $argv];
+			$workinfo_json = json_encode($workinfo);
+			$uuid          = self::getUuid($workinfo_json);
+
+			$r = q("SELECT * FROM workerq WHERE workerq_uuid = '%s'",
+				dbesc($uuid)
+			);
+			if ($r) {
+				logger("Summon: Ignoring duplicate workerq task", LOGGER_DEBUG);
+				logger(print_r($workinfo, true));
+				return;
+			}
+
+			logger('queueworker_stats_summon: cmd:' . $argv[0] . ' ' . 'timestamp:' . time());
+
+			$transaction = new \DbaTransaction(\DBA::$dba);
+			$r = q("INSERT INTO workerq (workerq_priority, workerq_data, workerq_uuid, workerq_cmd) VALUES (%d, '%s', '%s', '%s')",
+				intval($priority),
+				$workinfo_json,
+				dbesc($uuid),
+				dbesc($argv[0])
+			);
+			if (!$r) {
+				// Transaction is autmatically rolled back on return
+				logger("INSERT FAILED", LOGGER_DEBUG);
+				return;
+			}
+			$transaction->commit();
+			logger('INSERTED: ' . $workinfo_json, LOGGER_DEBUG);
+		}
+
+		$workers = self::GetWorkerCount();
+		if ($workers < self::$maxworkers) {
+			logger($workers . '/' . self::$maxworkers . ' workers active', LOGGER_DEBUG);
+			$phpbin = Config::Get('system', 'phpbin', 'php');
+			proc_run($phpbin, 'Zotlabs/Daemon/Master.php', ['Queueworker']);
+		}
+	}
+
+	public static function Release($argv) {
+
+		if ($argv[0] !== 'Queueworker') {
+
+			$priority = 0; // @TODO allow reprioritization
+			if (isset(self::$default_priorities[$argv[0]])) {
+				$priority = self::$default_priorities[$argv[0]];
+			}
+
+			$workinfo      = ['argc' => count($argv), 'argv' => $argv];
+			$workinfo_json = json_encode($workinfo);
+			$uuid          = self::getUuid($workinfo_json);
+
+			$r = q("SELECT * FROM workerq WHERE workerq_uuid = '%s'",
+				dbesc($uuid)
+			);
+			if ($r) {
+				logger("Release: Duplicate task - do not insert.", LOGGER_DEBUG);
+				logger(print_r($workinfo, true));
+				return;
+			}
+
+			$transaction = new \DbaTransaction(\DBA::$dba);
+			$r = q("INSERT INTO workerq (workerq_priority, workerq_data, workerq_uuid, workerq_cmd) VALUES (%d, '%s', '%s', '%s')",
+				intval($priority),
+				$workinfo_json,
+				dbesc($uuid),
+				dbesc($argv[0])
+			);
+			if (!$r) {
+				// Transaction is automatically rolled back on return
+				logger("Insert failed: " . $workinfo_json, LOGGER_DEBUG);
+				return;
+			}
+			$transaction->commit();
+			logger('INSERTED: ' . $workinfo_json, LOGGER_DEBUG);
+		}
+
+		self::Process();
+	}
+
+	public static function GetWorkerCount() {
+		if (self::$maxworkers == 0) {
+			self::$maxworkers = Config::Get('queueworker', 'max_queueworkers', 4);
+			self::$maxworkers = self::$maxworkers > 3 ? self::$maxworkers : 4;
+		}
+		if (self::$workermaxage == 0) {
+			self::$workermaxage = Config::Get('queueworker', 'max_queueworker_age');
+			self::$workermaxage = self::$workermaxage > 120 ? self::$workermaxage : 300;
+		}
+
+		$transaction = new \DbaTransaction(\DBA::$dba);
+
+		// skip locked is preferred but is not supported by mariadb < 10.6 which is still used a lot - hence make it optional
+		$sql_quirks = ((Config::Get('system', 'db_skip_locked_supported')) ? 'SKIP LOCKED' : 'NOWAIT');
+
+		$r = q("SELECT workerq_id FROM workerq WHERE workerq_reservationid IS NOT NULL AND workerq_processtimeout < %s FOR UPDATE $sql_quirks",
+			db_utcnow()
+		);
+
+		if ($r) {
+			// TODO: some long running services store their pid in config.procid.daemon
+			// we could possibly check if a pid exist and check if the process is still alive
+			// prior to reseting workerq_reservationid
+
+			$ids = ids_to_querystr($r, 'workerq_id');
+			$u = dbq("update workerq set workerq_reservationid = null where workerq_id in ($ids)");
+		}
+
+		$transaction->commit();
+
+		//q("update workerq set workerq_reservationid = null where workerq_reservationid is not null and workerq_processtimeout < %s",
+			//db_utcnow()
+		//);
+
+		//usleep(self::$workersleep);
+
+		$workers = dbq("select count(*) as total from workerq where workerq_reservationid is not null");
+		logger("WORKERCOUNT: " . $workers[0]['total'], LOGGER_DEBUG);
+
+		return intval($workers[0]['total']);
+	}
+
+	public static function GetWorkerID() {
+		if (self::$queueworker) {
+			return self::$queueworker;
+		}
+
+		$wid = uniqid('', true);
+
+		//usleep(mt_rand(300000, 1000000)); //Sleep .3 - 1 seconds before creating a new worker.
+
+		$workers = self::GetWorkerCount();
+
+		if ($workers >= self::$maxworkers) {
+			logger("Too many active workers ($workers) max = " . self::$maxworkers, LOGGER_DEBUG);
+			return false;
+		}
+
+		self::$queueworker = $wid;
+
+		return $wid;
+	}
+
+	private static function getWorkId() {
+		self::GetWorkerCount();
+
+		$transaction = new \DbaTransaction(\DBA::$dba);
+
+		// skip locked is preferred but is not supported by mariadb < 10.6 which is still used a lot - hence make it optional
+		$sql_quirks = ((Config::Get('system', 'db_skip_locked_supported')) ? 'SKIP LOCKED' : 'NOWAIT');
+
+		$work = dbq("SELECT workerq_id, workerq_cmd FROM workerq WHERE workerq_reservationid IS NULL ORDER BY workerq_priority DESC, workerq_id ASC LIMIT 1 FOR UPDATE $sql_quirks");
+
+		if (!$work) {
+			// Transaction automatically rolled back on return
+			return false;
+		}
+
+		$id = $work[0]['workerq_id'];
+		$cmd = $work[0]['workerq_cmd'];
+		$age = self::$workermaxage;
+
+		if (in_array($cmd, self::$long_running_cmd)) {
+			$age = 3600; // 1h TODO: make this configurable
+		}
+
+		$work = q("UPDATE workerq SET workerq_reservationid = '%s', workerq_processtimeout = %s + INTERVAL %s WHERE workerq_id = %d",
+			self::$queueworker,
+			db_utcnow(),
+			db_quoteinterval($age . " SECOND"),
+			intval($id)
+		);
+
+		if (!$work) {
+			// Transaction automatically rolled back on return
+			logger("Could not update workerq.", LOGGER_DEBUG);
+			return false;
+		}
+
+		logger("GOTWORK: " . json_encode($work), LOGGER_DEBUG);
+		$transaction->commit();
+
+		return $id;
+	}
+
+	public static function Process() {
+		$sleep = intval(Config::Get('queueworker', 'queue_worker_sleep', 100));
+		$auto_queue_worker_sleep = Config::Get('queueworker', 'auto_queue_worker_sleep', 0);
+
+		if (!self::GetWorkerID()) {
+			if ($auto_queue_worker_sleep) {
+				Config::Set('queueworker', 'queue_worker_sleep', $sleep + 100);
+			}
+
+			logger('Unable to get worker ID. Exiting.', LOGGER_DEBUG);
+			killme();
+		}
+
+		if ($auto_queue_worker_sleep && $sleep > 100) {
+			$next_sleep = $sleep - 100;
+			Config::Set('queueworker', 'queue_worker_sleep', (($next_sleep < 100) ? 100 : $next_sleep));
+		}
+
+		$jobs               = 0;
+		$workid             = self::getWorkId();
+		$load_average_sleep = false;
+		self::$workersleep  = $sleep;
+		self::$workersleep  = ((intval(self::$workersleep) > 100) ? intval(self::$workersleep) : 100);
+
+		if (function_exists('sys_getloadavg') && Config::Get('queueworker', 'load_average_sleep')) {
+			// very experimental!
+			$load_average_sleep = true;
+		}
+
+		while ($workid) {
+
+			if ($load_average_sleep) {
+				$load_average      = sys_getloadavg();
+				self::$workersleep = intval($load_average[0]) * 10000;
+
+				if (!self::$workersleep) {
+					self::$workersleep = 100;
+				}
+			}
+
+			logger('queue_worker_sleep: ' . self::$workersleep, LOGGER_DEBUG);
+
+			usleep(self::$workersleep);
+
+			$workitem = dbq("SELECT * FROM workerq WHERE workerq_id = $workid");
+
+			if ($workitem) {
+				// At least SOME work to do.... in case there's more, let's ramp up workers.
+				$workers = self::GetWorkerCount();
+
+				if ($workers < self::$maxworkers) {
+					logger($workers . '/' . self::$maxworkers . ' workers active', LOGGER_DEBUG);
+					$phpbin = Config::Get('system', 'phpbin', 'php');
+					proc_run($phpbin, 'Zotlabs/Daemon/Master.php', ['Queueworker']);
+				}
+
+				$jobs++;
+
+				logger("Workinfo: " . $workitem[0]['workerq_data'], LOGGER_DEBUG);
+
+				$workinfo = json_decode($workitem[0]['workerq_data'], true);
+				$argv     = $workinfo['argv'];
+
+				$cls  = '\\Zotlabs\\Daemon\\' . $argv[0];
+				$argv = flatten_array_recursive($argv);
+				$argc = count($argv);
+				$rnd = random_string(16);
+
+				logger('PROCESSING: ' . $rnd . ' ' . print_r($argv[0], true));
+
+				$start_timestamp = microtime(true);
+
+				$cls::run($argc, $argv);
+
+				logger('logger_stats_data cmd:' . $argv[0] . ' start:' . $start_timestamp . ' ' . 'end:' . microtime(true) . ' meta:' . $rnd);
+
+				logger('COMPLETED: ' . $rnd);
+
+				// @FIXME: Right now we assume that if we get a return, everything is OK.
+				// At some point we may want to test whether the run returns true/false
+				// and requeue the work to be tried again if needed.  But we probably want
+				// to implement some sort of "retry interval" first.
+
+				dbq("delete from workerq where workerq_id = $workid");
+			}
+			else {
+				logger("NO WORKITEM!", LOGGER_DEBUG);
+			}
+			$workid = self::getWorkId();
+		}
+		logger('Master: Worker Thread: queue items processed:' . $jobs, LOGGER_DEBUG);
+	}
+
+	public static function ClearQueue() {
+		$work = q("select * from workerq");
+		while ($work) {
+			foreach ($work as $workitem) {
+				$workinfo = json_decode($workitem['v'], true);
+				$argc     = $workinfo['argc'];
+				$argv     = $workinfo['argv'];
+
+				logger('Master: process: ' . print_r($argv, true), LOGGER_ALL, LOG_DEBUG);
+
+				if (!isset($argv[0])) {
+					q("delete from workerq where workerq_id = %d",
+						$work[0]['workerq_id']
+					);
+					continue;
+				}
+
+				$cls = '\\Zotlabs\\Daemon\\' . $argv[0];
+				$cls::run($argc, $argv);
+
+				q("delete from workerq where workerq_id = %d",
+					$work[0]['workerq_id']
+				);
+
+				//Give the server .3 seconds to catch its breath between tasks.
+				//This will hopefully keep it from crashing to it's knees entirely
+				//if the last task ended up initiating other parallel processes
+				//(eg. polling remotes)
+				usleep(300000);
+			}
+
+			//Make sure nothing new came in
+			$work = q("select * from workerq");
+		}
+	}
+
+	/**
+	 * @brief Generate a name-based v5 UUID with custom namespace
+	 *
+	 * @param string $data
+	 * @return string $uuid
+	 */
+	private static function getUuid(string $data) {
+		$namespace = '3a112e42-f147-4ccf-a78b-f6841339ea2a';
+		try {
+			$uuid = Uuid::uuid5($namespace, $data)->toString();
+		} catch (UnableToBuildUuidException $e) {
+			logger('UUID generation failed');
+			return '';
+		}
+		return $uuid;
+	}
+
+}

Zotlabs/Lib/Share.php

@@ -2,21 +2,19 @@
 
 namespace Zotlabs\Lib;
 
-use Zotlabs\Lib\Activity;
-
 class Share {
 
 	private $item = null;
 
 
 	public function __construct($post_id) {
-	
+
 		if(! $post_id)
 			return;
-	
+
 		if(! (local_channel() || remote_channel()))
 			return;
-	
+
 		$r = q("SELECT * from item left join xchan on author_xchan = xchan_hash WHERE id = %d  LIMIT 1",
 			intval($post_id)
 		);
@@ -25,26 +23,26 @@ class Share {
 
 		if(($r[0]['item_private']) && ($r[0]['xchan_network'] !== 'rss'))
 			return;
-	
+
 		$sql_extra = item_permissions_sql($r[0]['uid']);
-	
+
 		$r = q("select * from item where id = %d $sql_extra",
 			intval($post_id)
 		);
 		if(! $r)
 			return;
-	
+
 		if($r[0]['mimetype'] !== 'text/bbcode')
 			return;
-	
+
 		/** @FIXME eventually we want to post remotely via rpost on your home site */
 		// When that works remove this next bit:
-	
+
 		if(! local_channel())
 			return;
 
 		xchan_query($r);
-	
+
 		$this->item = $r[0];
 		return;
 	}
@@ -68,14 +66,14 @@ class Share {
 			'address'  => $this->item['author']['xchan_addr'],
 			'network'  => $this->item['author']['xchan_network'],
 			'link'     => [
-				[ 
-					'rel'  => 'alternate', 
-					'type' => 'text/html', 
+				[
+					'rel'  => 'alternate',
+					'type' => 'text/html',
 					'href' => $this->item['author']['xchan_url']
 				],
 				[
-					'rel'  => 'photo', 
-					'type' => $this->item['author']['xchan_photo_mimetype'], 
+					'rel'  => 'photo',
+					'type' => $this->item['author']['xchan_photo_mimetype'],
 					'href' => $this->item['author']['xchan_photo_m']
 				]
 			]
@@ -86,14 +84,14 @@ class Share {
 			'address'  => $this->item['owner']['xchan_addr'],
 			'network'  => $this->item['owner']['xchan_network'],
 			'link'     => [
-				[ 
-					'rel'  => 'alternate', 
-					'type' => 'text/html', 
+				[
+					'rel'  => 'alternate',
+					'type' => 'text/html',
 					'href' => $this->item['owner']['xchan_url']
 				],
 				[
-					'rel'  => 'photo', 
-					'type' => $this->item['owner']['xchan_photo_mimetype'], 
+					'rel'  => 'photo',
+					'type' => $this->item['owner']['xchan_photo_mimetype'],
 					'href' => $this->item['owner']['xchan_photo_m']
 				]
 			]
@@ -114,12 +112,12 @@ class Share {
 		if(! $this->item)
 			return $bb;
 
-		$is_photo = (($this->item['obj_type'] === ACTIVITY_OBJ_PHOTO) ? true : false);
+		$is_photo = ((in_array($this->item['obj_type'], ['Image', ACTIVITY_OBJ_PHOTO])) ? true : false);
 		if($is_photo) {
 			$object = json_decode($this->item['obj'],true);
 			$photo_bb = $object['body'];
 		}
-	
+
 		if (strpos($this->item['body'], "[/share]") !== false) {
 			$pos = strpos($this->item['body'], "[share");
 			$bb = substr($this->item['body'], $pos);
@@ -128,7 +126,7 @@ class Share {
 				"' profile='"    . $this->item['author']['xchan_url'] .
 				"' avatar='"     . $this->item['author']['xchan_photo_s'] .
 				"' link='"       . $this->item['plink'] .
-				"' auth='"       . ((in_array($this->item['author']['xchan_network'], ['zot6', 'zot'])) ? 'true' : 'false') .
+				"' auth='"       . (($this->item['author']['xchan_network'] === 'zot6') ? 'true' : 'false') .
 				"' posted='"     . $this->item['created'] .
 				"' message_id='" . $this->item['mid'] .
 			"']";

Zotlabs/Lib/System.php

@@ -16,13 +16,13 @@ class System {
 	}
 
 	static public function get_site_name() {
-		if(is_array(\App::$config) && is_array(\App::$config['system']) && \App::$config['system']['sitename'])
+		if(is_array(\App::$config) && is_array(\App::$config['system']) && isset(\App::$config['system']['sitename']))
 			return \App::$config['system']['sitename'];
 		return '';
 	}
 
 	static public function get_project_version() {
-		if(is_array(\App::$config) && is_array(\App::$config['system']) && \App::$config['system']['hide_version'])
+		if(is_array(\App::$config) && is_array(\App::$config['system']) && isset(\App::$config['system']['hide_version']))
 			return '';
 		if(is_array(\App::$config) && is_array(\App::$config['system']) && array_key_exists('std_version',\App::$config['system']))
 			return \App::$config['system']['std_version'];
@@ -31,33 +31,33 @@ class System {
 	}
 
 	static public function get_update_version() {
-		if(is_array(\App::$config) && is_array(\App::$config['system']) && \App::$config['system']['hide_version'])
+		if(is_array(\App::$config) && is_array(\App::$config['system']) && isset(\App::$config['system']['hide_version']))
 			return '';
 		return DB_UPDATE_VERSION;
 	}
 
 
 	static public function get_notify_icon() {
-		if(is_array(\App::$config) && is_array(\App::$config['system']) && \App::$config['system']['email_notify_icon_url'])
+		if(is_array(\App::$config) && is_array(\App::$config['system']) && isset(\App::$config['system']['email_notify_icon_url']))
 			return \App::$config['system']['email_notify_icon_url'];
 		return z_root() . DEFAULT_NOTIFY_ICON;
 	}
 
 	static public function get_site_icon() {
-		if(is_array(\App::$config) && is_array(\App::$config['system']) && \App::$config['system']['site_icon_url'])
+		if(is_array(\App::$config) && is_array(\App::$config['system']) && isset(\App::$config['system']['site_icon_url']))
 			return \App::$config['system']['site_icon_url'];
 		return z_root() . DEFAULT_PLATFORM_ICON ;
 	}
 
 
 	static public function get_project_link() {
-		if(is_array(\App::$config) && is_array(\App::$config['system']) && \App::$config['system']['project_link'])
+		if(is_array(\App::$config) && is_array(\App::$config['system']) && isset(\App::$config['system']['project_link']))
 			return \App::$config['system']['project_link'];
 		return 'https://hubzilla.org';
 	}
 
 	static public function get_project_srclink() {
-		if(is_array(\App::$config) && is_array(\App::$config['system']) && \App::$config['system']['project_srclink'])
+		if(is_array(\App::$config) && is_array(\App::$config['system']) && isset(\App::$config['system']['project_srclink']))
 			return \App::$config['system']['project_srclink'];
 		return 'https://framagit.org/hubzilla/core.git';
 	}
@@ -68,7 +68,7 @@ class System {
 
 
 	static public function get_zot_revision() {
-		$x = [ 'revision' => ZOT_REVISION ]; 
+		$x = [ 'revision' => ZOT_REVISION ];
 		call_hooks('zot_revision',$x);
 		return $x['revision'];
 	}

Zotlabs/Lib/Text.php

@@ -0,0 +1,24 @@
+<?php
+
+namespace Zotlabs\Lib;
+
+class Text {
+
+	/**
+	 * use this on "body" or "content" input where angle chars shouldn't be removed,
+	 * and allow them to be safely displayed.
+	 *
+	 * @param string $string
+	 *
+	 * @return string
+	 */
+
+	public static function escape_tags(string $string): string {
+		if (!$string) {
+			return EMPTY_STR;
+		}
+
+		return htmlspecialchars($string, ENT_COMPAT, 'UTF-8', false);
+	}
+
+}

Zotlabs/Lib/ThreadItem.php

@@ -2,7 +2,10 @@
 
 namespace Zotlabs\Lib;
 
+use App;
+use Zotlabs\Access\AccessList;
 use Zotlabs\Lib\Apps;
+use Zotlabs\Lib\Config;
 
 require_once('include/text.php');
 
@@ -17,7 +20,7 @@ class ThreadItem {
 	private $comment_box_template = 'comment_item.tpl';
 	private $commentable = false;
 	// list of supported reaction emojis - a site can over-ride this via config system.reactions
-	private $reactions = ['1f60a','1f44f','1f37e','1f48b','1f61e','2665','1f606','1f62e','1f634','1f61c','1f607','1f608'];
+	private $reactions = ['slightly_smiling_face','clapping_hands','bottle_with_popping_cork','kiss_mark','disappointed_face','red_heart','grinning_face','astonished_face','sleeping_face','winking_face_with_tongue','smiling_face_with_halo','smiling_face_with_horns'];
 	private $toplevel = false;
 	private $children = array();
 	private $parent = null;
@@ -32,18 +35,18 @@ class ThreadItem {
 	private $channel = null;
 	private $display_mode = 'normal';
 	private $reload = '';
+	private $mid_uuid_map = [];
 
 
 	public function __construct($data) {
-				
+
 		$this->data = $data;
 		$this->toplevel = ($this->get_id() == $this->get_data_value('parent'));
-		$this->threaded = get_config('system','thread_allow');
-
-		$observer = \App::get_observer();
+		$this->threaded = Config::Get('system','thread_allow');
 
 		// Prepare the children
-		if($data['children']) {
+		if(isset($data['children'])) {
+
 			foreach($data['children'] as $item) {
 
 				/*
@@ -54,15 +57,19 @@ class ThreadItem {
 					continue;
 				}
 
-
 				$child = new ThreadItem($item);
 				$this->add_child($child);
 			}
+
+			// performance: we have already added the children
+			unset($this->data['children']);
 		}
 
+
+
 		// allow a site to configure the order and content of the reaction emoji list
 		if($this->toplevel) {
-			$x = get_config('system','reactions');
+			$x = Config::Get('system','reactions');
 			if($x && is_array($x) && count($x)) {
 				$this->reactions = $x;
 			}
@@ -77,12 +84,10 @@ class ThreadItem {
 	 *      _ false on failure
 	 */
 
-	public function get_template_data($conv_responses, $thread_level=1, $conv_flags = []) {
-
-		$result = array();
-
-		$item     = $this->get_data();
+	public function get_template_data($conv_responses, $mid_uuid_map, $thread_level=1, $conv_flags = []) {
 
+		$result = [];
+		$item = $this->get_data();
 		$commentww = '';
 		$sparkle = '';
 		$buttons = '';
@@ -93,32 +98,53 @@ class ThreadItem {
 		$is_item = false;
 		$osparkle = '';
 		$total_children = $this->count_descendants();
-		$unseen_comments = (($item['real_uid']) ? 0 : $this->count_unseen_descendants());
+		$unseen_comments = ((isset($item['real_uid']) && $item['real_uid']) ? 0 : $this->count_unseen_descendants());
 
  		$conv = $this->get_conversation();
 		$observer = $conv->get_observer();
 
-		$lock = (((intval($item['item_private'])) || (($item['uid'] == local_channel()) && (strlen($item['allow_cid']) || strlen($item['allow_gid']) 
-			|| strlen($item['deny_cid']) || strlen($item['deny_gid']))))
-			? t('Private Message')
+		$acl = new AccessList([]);
+		$acl->set($item);
+
+		$lock = ((intval($item['item_private']) || ($item['uid'] == local_channel() && $acl->is_private()))
+			? t('Restricted message')
 			: false);
 
-		$shareable = ((($conv->get_profile_owner() == local_channel() && local_channel()) && ($item['item_private'] != 1)) ? true : false);
+		// 1 = restricted message, 2 = direct message
+		$locktype = intval($item['item_private']);
+
+		if ($locktype === 2) {
+			$lock = t('Direct message');
+		}
+
+		// 0 = limited based on public policy
+		if ($item['uid'] == local_channel() && intval($item['item_private']) && !$acl->is_private() && strlen($item['public_policy'])) {
+			$lock = t('Public Policy');
+			$locktype = 0;
+		}
+
+		$shareable = ((($conv->get_profile_owner() == local_channel() && local_channel()) && (intval($item['item_private']) === 0)) ? true : false);
 
 		// allow an exemption for sharing stuff from your private feeds
 		if($item['author']['xchan_network'] === 'rss')
 			$shareable = true;
 
-		$privacy_warning = false;
-		if(intval($item['item_private']) && ($item['owner']['xchan_network'] === 'activitypub')) {
-			$recips = get_iconfig($item['parent'], 'activitypub', 'recips');
+		$repeatable = ((($conv->get_profile_owner() == local_channel() && local_channel()) && (intval($item['item_private']) === 0) && (in_array($item['author']['xchan_network'], ['zot6', 'activitypub']))) ? true : false);
 
-			if(! is_array($recips['to']) || ! in_array($observer['xchan_url'], $recips['to']))
-				$privacy_warning = true;
+		// @fixme
+		// Have recently added code to properly handle polls in group reshares by redirecting all of the poll responses to the group.
+		// Sharing a poll using a regular embedded share is harder because the poll will need to fork. This is due to comment permissions.
+		// The original poll author may not accept responses from strangers. Forking the poll will receive responses from the sharer's
+		// followers, but there's no elegant way to merge these two sets of results together. For now, we'll disable sharing polls.
+
+		if ($item['obj_type'] === 'Question') {
+			$shareable = false;
 		}
 
+		$privacy_warning = ($item['owner']['xchan_network'] === 'activitypub' && intval($item['item_private']) === 1);
+
 		if ($lock) {
- 			if (($item['mid'] == $item['parent_mid']) && count(get_terms_oftype($item['term'],TERM_FORUM))) {
+ 			if (($item['mid'] == $item['parent_mid']) && isset($item['term']) && count(get_terms_oftype($item['term'], TERM_FORUM))) {
  				$privacy_warning = true;
 				$conv_flags['parent_privacy_warning'] = true;
  			}
@@ -132,28 +158,15 @@ class ThreadItem {
 
 		$mode = $conv->get_mode();
 
-		switch($item['item_type']) {
-			case ITEM_TYPE_CARD:
-				$edlink = 'card_edit';
-				break;
-			case ITEM_TYPE_ARTICLE:
-				$edlink = 'article_edit';
-				break;
-			default:
-				$edlink = 'editpost';
-				break;
-		}
-
 		if(local_channel() && $observer['xchan_hash'] === $item['author_xchan'])
-			$edpost = array(z_root() . '/' . $edlink . '/' . $item['id'], t('Edit'));
+			$edpost = array(z_root() . '/editpost/' . $item['id'], t('Edit'));
 		else
 			$edpost = false;
 
-
 		if($observer && $observer['xchan_hash']
-			&& ($observer['xchan_hash'] == $this->get_data_value('author_xchan') 
-			|| $observer['xchan_hash'] == $this->get_data_value('owner_xchan') 
-			|| $observer['xchan_hash'] == $this->get_data_value('source_xchan') 
+			&& ($observer['xchan_hash'] == $this->get_data_value('author_xchan')
+			|| $observer['xchan_hash'] == $this->get_data_value('owner_xchan')
+			|| $observer['xchan_hash'] == $this->get_data_value('source_xchan')
 			|| $this->get_data_value('uid') == local_channel()))
 			$dropping = true;
 
@@ -163,40 +176,37 @@ class ThreadItem {
 			$dropping = false;
 		}
 
-
+		$drop = [];
 		if($dropping) {
 			$drop = array(
 				'dropping' => $dropping,
 				'delete' => t('Delete'),
 			);
-		}		
+		}
 		elseif(is_site_admin()) {
 			$drop = [ 'dropping' => true, 'delete' => t('Admin Delete') ];
 		}
 
-// FIXME
-		if($observer_is_pageowner) {		
-			$multidrop = array(
-				'select' => t('Select'), 
-			);
-		}
-
 		$filer = ((($conv->get_profile_owner() == local_channel()) && (! array_key_exists('real_uid',$item))) ? t("Save to Folder") : false);
 
-		$profile_avatar = $item['author']['xchan_photo_m'];
+		$profile_avatar = $item['author']['xchan_photo_s'];
 		$profile_link   = chanlink_hash($item['author_xchan']);
 		$profile_name   = $item['author']['xchan_name'];
 
 		$location = format_location($item);
 		$isevent = false;
 		$attend = null;
-		$canvote = false;
 
 		// process action responses - e.g. like/dislike/attend/agree/whatever
-		$response_verbs = array('like');
-		if(feature_enabled($conv->get_profile_owner(),'dislike'))
+		$response_verbs[] = 'like';
+
+		if(feature_enabled($conv->get_profile_owner(),'dislike')) {
 			$response_verbs[] = 'dislike';
-		if($item['obj_type'] === ACTIVITY_OBJ_EVENT) {
+		}
+
+		$response_verbs[] = 'announce';
+
+		if(in_array($item['obj_type'], ['Event', ACTIVITY_OBJ_EVENT])) {
 			$response_verbs[] = 'attendyes';
 			$response_verbs[] = 'attendno';
 			$response_verbs[] = 'attendmaybe';
@@ -210,27 +220,19 @@ class ThreadItem {
 			$response_verbs[] = 'answer';
 		}
 
-		$consensus = (intval($item['item_consensus']) ? true : false);
-		if($consensus) {
-			$response_verbs[] = 'agree';
-			$response_verbs[] = 'disagree';
-			$response_verbs[] = 'abstain';
-			if($this->is_commentable() && $observer) {
-				$conlabels = array( t('I agree'), t('I disagree'), t('I abstain'));
-				$canvote = true;
-			}
-		}
-
 		if(! feature_enabled($conv->get_profile_owner(),'dislike'))
 			unset($conv_responses['dislike']);
-  
+
 		$responses = get_responses($conv_responses,$response_verbs,$this,$item);
 
 		$my_responses = [];
 		foreach($response_verbs as $v) {
-			$my_responses[$v] = (($conv_responses[$v][$item['mid'] . '-m']) ? 1 : 0);
+
+			$my_responses[$v] = ((isset($conv_responses[$v][$item['mid'] . '-m'])) ? 1 : 0);
 		}
 
+/*
+
 		$like_count = ((x($conv_responses['like'],$item['mid'])) ? $conv_responses['like'][$item['mid']] : '');
 		$like_list = ((x($conv_responses['like'],$item['mid'])) ? $conv_responses['like'][$item['mid'] . '-l'] : '');
 		if (($like_list) && (count($like_list) > MAX_LIKERS)) {
@@ -241,6 +243,17 @@ class ThreadItem {
 		}
 		$like_button_label = tt('Like','Likes',$like_count,'noun');
 
+		$repeat_count = ((x($conv_responses['announce'],$item['mid'])) ? $conv_responses['announce'][$item['mid']] : '');
+		$repeat_list = ((x($conv_responses['announce'],$item['mid'])) ? $conv_responses['announce'][$item['mid'] . '-l'] : '');
+		if (($repeat_list) && (count($repeat_list) > MAX_LIKERS)) {
+			$repeat_list_part = array_slice($repeat_list, 0, MAX_LIKERS);
+			array_push($repeat_list_part, '<a class="dropdown-item" href="#" data-toggle="modal" data-target="#repeatModal-' . $this->get_id() . '"><b>' . t('View all') . '</b></a>');
+		} else {
+			$repeat_list_part = '';
+		}
+		$repeat_button_label = tt('Repeat','Repeats',$repeat_count,'noun');
+
+		$showdislike = '';
 		if (feature_enabled($conv->get_profile_owner(),'dislike')) {
 			$dislike_count = ((x($conv_responses['dislike'],$item['mid'])) ? $conv_responses['dislike'][$item['mid']] : '');
 			$dislike_list = ((x($conv_responses['dislike'],$item['mid'])) ? $conv_responses['dislike'][$item['mid'] . '-l'] : '');
@@ -251,11 +264,12 @@ class ThreadItem {
 			} else {
 				$dislike_list_part = '';
 			}
+
+			$showdislike = ((x($conv_responses['dislike'],$item['mid'])) ? format_like($conv_responses['dislike'][$item['mid']],$conv_responses['dislike'][$item['mid'] . '-l'],'dislike',$item['mid']) : '');
 		}
 
 		$showlike    = ((x($conv_responses['like'],$item['mid'])) ? format_like($conv_responses['like'][$item['mid']],$conv_responses['like'][$item['mid'] . '-l'],'like',$item['mid']) : '');
-		$showdislike = ((x($conv_responses['dislike'],$item['mid']) && feature_enabled($conv->get_profile_owner(),'dislike'))  
-				? format_like($conv_responses['dislike'][$item['mid']],$conv_responses['dislike'][$item['mid'] . '-l'],'dislike',$item['mid']) : '');
+*/
 
 		/*
 		 * We should avoid doing this all the time, but it depends on the conversation mode
@@ -264,18 +278,15 @@ class ThreadItem {
 		 */
 
 		$this->check_wall_to_wall();
-		
-		if($this->is_toplevel()) {
-			// FIXME check this permission
-			if(($conv->get_profile_owner() == local_channel()) && (! array_key_exists('real_uid',$item))) {
 
-				$star = array(
+		if($this->is_toplevel()) {
+			if((local_channel() && $conv->get_profile_owner() === local_channel()) || (local_channel() && App::$module === 'pubstream')) {
+				$star = [
 					'toggle' => t("Toggle Star Status"),
 					'isstarred' => ((intval($item['item_starred'])) ? true : false),
-				);
-
+				];
 			}
-		} 
+		}
 		else {
 			$is_comment = true;
 		}
@@ -290,7 +301,7 @@ class ThreadItem {
 		$tagger = [];
 
 		// FIXME - check this permission
-		if($conv->get_profile_owner() == local_channel()) {
+		if(local_channel() && $conv->get_profile_owner() == local_channel()) {
 			/* disable until we agree on how to implemnt this in zot6/activitypub
 			$tagger = array(
 				'tagit' => t("Add Tag"),
@@ -298,11 +309,11 @@ class ThreadItem {
 			);
 			*/
 
-			$settings = t('Conversation Tools');
+			$settings = t('Conversation Features');
 		}
 
 		$has_bookmarks = false;
-		if(Apps::system_app_installed(local_channel(), 'Bookmarks') && is_array($item['term'])) {
+		if(Apps::system_app_installed(local_channel(), 'Bookmarks') && isset($item['term']) && is_array($item['term'])) {
 			foreach($item['term'] as $t) {
 				if(($t['ttype'] == TERM_BOOKMARK))
 					$has_bookmarks = true;
@@ -310,54 +321,61 @@ class ThreadItem {
 		}
 
 		$has_event = false;
-		if(($item['obj_type'] === ACTIVITY_OBJ_EVENT) && $conv->get_profile_owner() == local_channel())
+		if((in_array($item['obj_type'], ['Event', ACTIVITY_OBJ_EVENT])) && $conv->get_profile_owner() == local_channel())
 			$has_event = true;
 
+		$like = [];
+		$dislike = [];
+		$reply_to = [];
+
 		if($this->is_commentable() && $observer) {
 			$like = array( t("I like this \x28toggle\x29"), t("like"));
 			$dislike = array( t("I don't like this \x28toggle\x29"), t("dislike"));
 			$reply_to = array( t("Reply on this comment"), t("reply"), t("Reply to"));
 		}
 
+		$share = [];
+		$embed = [];
 		if ($shareable) {
-			// This actually turns out not to be possible in some protocol stacks without opening up hundreds of new issues.
-			// Will allow it only for uri resolvable sources.
-			if(strpos($item['mid'],'http') === 0) {
-				$share = []; //Not yet ready for primetime
-				//$share = array( t('Repeat This'), t('repeat'));
-			}
-			$embed = array( t('Share This'), t('share'));
+			$embed = [t('Share'), t('share')];
+		}
+
+		if ($repeatable) {
+			$share = [t('Repeat'), t('repeat')];
 		}
 
 		$dreport = '';
 
-		$keep_reports = intval(get_config('system','expire_delivery_reports'));
+		$keep_reports = intval(Config::Get('system','expire_delivery_reports'));
 		if($keep_reports === 0)
 			$keep_reports = 10;
 
-		if((! get_config('system','disable_dreport')) && strcmp(datetime_convert('UTC','UTC',$item['created']),datetime_convert('UTC','UTC',"now - $keep_reports days")) > 0) {
+		$dreport_link = '';
+		if((intval($item['item_type']) == ITEM_TYPE_POST) && (! Config::Get('system','disable_dreport')) && strcmp(datetime_convert('UTC','UTC',$item['created']),datetime_convert('UTC','UTC',"now - $keep_reports days")) > 0) {
 			$dreport = t('Delivery Report');
-			$dreport_link = gen_link_id($item['mid']);
+			$dreport_link = '?mid=' . $item['mid'];
 		}
 
+		$is_new = false;
 		if(strcmp(datetime_convert('UTC','UTC',$item['created']),datetime_convert('UTC','UTC','now - 12 hours')) > 0)
 			$is_new = true;
 
 		localize_item($item);
 
-		$body = prepare_body($item,true);
+		$opts = (($item['resource_type'] === 'event') ? ['is_event_item' => true] : []);
+		$body = prepare_body($item, true, $opts);
 
 		// $viewthread (below) is only valid in list mode. If this is a channel page, build the thread viewing link
 		// since we can't depend on llink or plink pointing to the right local location.
- 
+
 		$owner_address = substr($item['owner']['xchan_addr'],0,strpos($item['owner']['xchan_addr'],'@'));
 		$viewthread = $item['llink'];
 		if($conv->get_mode() === 'channel')
 			$viewthread = z_root() . '/channel/' . $owner_address . '?f=&mid=' . urlencode(gen_link_id($item['mid']));
 
-		$comment_count_txt = sprintf( tt('%d comment','%d comments',$total_children),$total_children );
-		$list_unseen_txt = (($unseen_comments) ? sprintf( t('%d unseen'),$unseen_comments) : '');
-		
+		$comment_count_txt = ['label' => sprintf(tt('%d comment', '%d comments', $total_children), $total_children), 'count' => $total_children];
+		$list_unseen_txt = $unseen_comments ? ['label' => sprintf(t('%d unseen'), $unseen_comments), 'count' => $unseen_comments] : [];
+
 		$children = $this->get_children();
 
 		$has_tags = (($body['tags'] || $body['categories'] || $body['mentions'] || $body['attachments'] || $body['folders']) ? true : false);
@@ -366,8 +384,8 @@ class ThreadItem {
                 call_hooks('dropdown_extras',$dropdown_extras_arr);
                 $dropdown_extras = $dropdown_extras_arr['dropdown_extras'];
 
-		$midb64 = 'b64.' . base64url_encode($item['mid']);
-		$mids = [ $midb64 ];
+		$midb64 = $item['uuid'];
+		$mids = [ $item['uuid'] ];
 		$response_mids = [];
 		foreach($response_verbs as $v) {
 			if(isset($conv_responses[$v]['mids'][$item['mid']])) {
@@ -379,14 +397,20 @@ class ThreadItem {
 		$json_mids = json_encode($mids);
 
 		// Pinned item processing
-		$allowed_type = (in_array($item['item_type'], get_config('system', 'pin_types', [ ITEM_TYPE_POST ])) ? true : false);
+		$allowed_type = (in_array($item['item_type'], Config::Get('system', 'pin_types', [ ITEM_TYPE_POST ])) ? true : false);
 		$pinned_items = ($allowed_type ? get_pconfig($item['uid'], 'pinned', $item['item_type'], []) : []);
 		$pinned = ((!empty($pinned_items) && in_array($midb64, $pinned_items)) ? true : false);
 
+		$contact = [];
+
+		if(App::$contacts && array_key_exists($item['author_xchan'], App::$contacts)) {
+			$contact = App::$contacts[$item['author_xchan']];
+		}
+
 		$tmp_item = array(
 			'template' => $this->get_template(),
 			'mode' => $mode,
-			'item_type' => intval($item['item_type']),			
+			'item_type' => intval($item['item_type']),
 			//'type' => implode("",array_slice(explode("/",$item['verb']),-1)),
 			'body' => $body['html'],
 			'tags' => $body['tags'],
@@ -400,11 +424,9 @@ class ThreadItem {
 			'mids' => $json_mids,
 			'parent' => $item['parent'],
 			'author_id' => (($item['author']['xchan_addr']) ? $item['author']['xchan_addr'] : $item['author']['xchan_url']),
+			'author_is_group_actor' => (($item['author']['xchan_pubforum']) ? t('Forum') : ''),
 			'isevent' => $isevent,
 			'attend' => $attend,
-			'consensus' => $consensus,
-			'conlabels' => $conlabels,
-			'canvote' => $canvote,
 			'linktitle' => (($item['author']['xchan_addr']) ? $item['author']['xchan_addr'] : $item['author']['xchan_url']),
 			'olinktitle' => (($item['owner']['xchan_addr']) ? $item['owner']['xchan_addr'] : $item['owner']['xchan_url']),
 			'llink' => $item['llink'],
@@ -432,6 +454,7 @@ class ThreadItem {
 			'editedtime' => (($item['edited'] != $item['created']) ? sprintf( t('last edited: %s'), datetime_convert('UTC', date_default_timezone_get(), $item['edited'], 'r')) : ''),
 			'expiretime' => (($item['expires'] > NULL_DATE) ? sprintf( t('Expires: %s'), datetime_convert('UTC', date_default_timezone_get(), $item['expires'], 'r')):''),
 			'lock' => $lock,
+			'locktype' => $locktype,
 			'delayed' => $item['item_delayed'],
 			'privacy_warning' => $privacy_warning,
 			'verified' => $verified,
@@ -472,36 +495,50 @@ class ThreadItem {
 			'bookmark'  => (($conv->get_profile_owner() == local_channel() && local_channel() && $has_bookmarks) ? t('Save Bookmarks') : ''),
 			'addtocal'  => (($has_event) ? t('Add to Calendar') : ''),
 			'drop'      => $drop,
-			'multidrop' => ((feature_enabled($conv->get_profile_owner(),'multi_delete')) ? $multidrop : ''),
 			'dropdown_extras' => $dropdown_extras,
 // end toolbar buttons
 			'unseen_comments' => $unseen_comments,
 			'comment_count' => $total_children,
 			'comment_count_txt' => $comment_count_txt,
 			'list_unseen_txt' => $list_unseen_txt,
-			'markseen' => t('Mark all seen'),
+			'markseen' => t('Mark all comments seen'),
 			'responses' => $responses,
 			'my_responses' => $my_responses,
+			/*
 			'like_count' => $like_count,
 			'like_list' => $like_list,
 			'like_list_part' => $like_list_part,
 			'like_button_label' => $like_button_label,
 			'like_modal_title' => t('Likes','noun'),
+
+			'repeat_count' => $repeat_count,
+			'repeat_list' => $repeat_list,
+			'repeat_list_part' => $repeat_list_part,
+			'repeat_button_label' => $repeat_button_label,
+			'repeat_modal_title' => t('Repeats','noun'),
+
+
 			'dislike_modal_title' => t('Dislikes','noun'),
 			'dislike_count' => ((feature_enabled($conv->get_profile_owner(),'dislike')) ? $dislike_count : ''),
 			'dislike_list' => ((feature_enabled($conv->get_profile_owner(),'dislike')) ? $dislike_list : ''),
 			'dislike_list_part' => ((feature_enabled($conv->get_profile_owner(),'dislike')) ? $dislike_list_part : ''),
 			'dislike_button_label' => ((feature_enabled($conv->get_profile_owner(),'dislike')) ? $dislike_button_label : ''),
+*/
 			'modal_dismiss' => t('Close'),
-			'showlike' => $showlike,
-			'showdislike' => $showdislike,
-			'comment' => ($item['item_delayed'] ? '' : $this->get_comment_box($indent)),
+		//	'showlike' => $showlike,
+		//	'showdislike' => $showdislike,
+			'comment' => ($item['item_delayed'] ? '' : $this->get_comment_box()),
 			'previewing' => ($conv->is_preview() ? true : false ),
 			'preview_lbl' => t('This is an unsaved preview'),
 			'wait' => t('Please wait'),
 			'thread_level' => $thread_level,
 			'settings' => $settings,
-			'thr_parent' => (($item['parent_mid'] != $item['thr_parent']) ? 'b64.' . base64url_encode($item['thr_parent']) : '')
+			'thr_parent_uuid' => (($item['parent_mid'] != $item['thr_parent']) ? $mid_uuid_map[$item['thr_parent']] : ''),
+			'contact_id' => (($contact) ? $contact['abook_id'] : ''),
+			'moderate' => ($item['item_blocked'] == ITEM_MODERATED),
+			'moderate_approve' => t('Approve'),
+			'moderate_delete' => t('Delete'),
+			'rtl' => in_array($item['lang'], rtl_languages()),
 		);
 
 		$arr = array('item' => $item, 'output' => $tmp_item);
@@ -512,24 +549,24 @@ class ThreadItem {
 		$result['children'] = array();
 		$nb_children = count($children);
 
-		$visible_comments = get_config('system','expanded_comments');
+		$visible_comments = Config::Get('system','expanded_comments');
 		if($visible_comments === false)
 			$visible_comments = 3;
 
 //		needed for scroll to comment from notification but needs more work
 //		as we do not want to open all comments unless there is actually an #item_xx anchor
-//		and the url fragment is not sent to the server. 
-//		if(in_array(\App::$module,['display','update_display'])) 
+//		and the url fragment is not sent to the server.
+//		if(in_array(\App::$module,['display','update_display']))
 //			$visible_comments = 99999;
 
 		if(($this->get_display_mode() === 'normal') && ($nb_children > 0)) {
 			foreach($children as $child) {
-				$result['children'][] = $child->get_template_data($conv_responses, $thread_level + 1,$conv_flags);
+				$result['children'][] = $child->get_template_data($conv_responses, $mid_uuid_map, $thread_level + 1,$conv_flags);
 			}
 			// Collapse
 			if(($nb_children > $visible_comments) || ($thread_level > 1)) {
 				$result['children'][0]['comment_firstcollapsed'] = true;
-				$result['children'][0]['num_comments'] = $comment_count_txt;
+				$result['children'][0]['num_comments'] = $comment_count_txt['label'];
 				$result['children'][0]['hide_text'] = sprintf( t('%s show all'), '<i class="fa fa-chevron-down"></i>');
 				if($thread_level > 1) {
 					$result['children'][$nb_children - 1]['comment_lastcollapsed'] = true;
@@ -539,7 +576,7 @@ class ThreadItem {
 				}
 			}
 		}
-		
+
 		$result['private'] = $item['item_private'];
 		$result['toplevel'] = ($this->is_toplevel() ? 'toplevel_item' : '');
 
@@ -554,7 +591,7 @@ class ThreadItem {
 
 		return $result;
 	}
-	
+
 	public function get_id() {
 		return $this->get_data_value('id');
 	}
@@ -606,10 +643,10 @@ class ThreadItem {
 		 * Only add what will be displayed
 		 */
 
-		if(activity_match($item->get_data_value('verb'),ACTIVITY_LIKE) || activity_match($item->get_data_value('verb'),ACTIVITY_DISLIKE)) {
+		if(activity_match($item->get_data_value('verb'), ['Like', 'Dislike', ACTIVITY_LIKE, ACTIVITY_DISLIKE])) {
 			return false;
 		}
-		
+
 		$item->set_parent($this);
 		$this->children[] = $item;
 		return end($this->children);
@@ -683,7 +720,7 @@ class ThreadItem {
 	 */
 	public function set_conversation($conv) {
 		$previous_mode = ($this->conversation ? $this->conversation->get_mode() : '');
-		
+
 		$this->conversation = $conv;
 
 		// Set it on our children too
@@ -787,12 +824,12 @@ class ThreadItem {
 	 *      _ The comment box string (empty if no comment box)
 	 *      _ false on failure
 	 */
-	private function get_comment_box($indent) {
+	private function get_comment_box() {
 
-		if(!$this->is_toplevel() && !get_config('system','thread_allow')) {
+		if(!$this->is_toplevel() && !Config::Get('system','thread_allow')) {
 			return '';
 		}
-		
+
 		$comment_box = '';
 		$conv = $this->get_conversation();
 
@@ -808,7 +845,7 @@ class ThreadItem {
 		$arr = array('comment_buttons' => '','id' => $this->get_id());
 		call_hooks('comment_buttons',$arr);
 		$comment_buttons = $arr['comment_buttons'];
-		
+
 		$comment_box = replace_macros($template,array(
 			'$return_path' => '',
 			'$threaded' => $this->is_threaded(),
@@ -833,14 +870,13 @@ class ThreadItem {
 			'$edurl' => t('Insert Link'),
 			'$edvideo' => t('Video'),
 			'$preview' => t('Preview'), // ((feature_enabled($conv->get_profile_owner(),'preview')) ? t('Preview') : ''),
-			'$indent' => $indent,
 			'$can_upload' => (perm_is_allowed($conv->get_profile_owner(),get_observer_hash(),'write_storage') && $conv->is_uploadable()),
 			'$feature_encrypt' => ((feature_enabled($conv->get_profile_owner(),'content_encrypt')) ? true : false),
 			'$encrypt' => t('Encrypt text'),
 			'$cipher' => $conv->get_cipher(),
-			'$sourceapp' => \App::$sourcename,
+			'$sourceapp' => App::$sourcename,
 			'$observer' => get_observer_hash(),
-			'$anoncomments' => ((($conv->get_mode() === 'channel' || $conv->get_mode() === 'display') && perm_is_allowed($conv->get_profile_owner(),'','post_comments')) ? true : false),
+			'$anoncomments' => ((in_array($conv->get_mode(), ['channel', 'display', 'cards', 'articles']) && perm_is_allowed($conv->get_profile_owner(),'','post_comments')) ? true : false),
 			'$anonname' => [ 'anonname', t('Your full name (required)') ],
 			'$anonmail' => [ 'anonmail', t('Your email address (required)') ],
 			'$anonurl'  => [ 'anonurl',  t('Your website URL (optional)') ]
@@ -865,13 +901,19 @@ class ThreadItem {
 
 		if($conv->get_mode() === 'channel')
 			return;
-		
+
 		if($this->is_toplevel() && ($this->get_data_value('author_xchan') != $this->get_data_value('owner_xchan'))) {
 			$this->owner_url = chanlink_hash($this->data['owner']['xchan_hash']);
-			$this->owner_photo = $this->data['owner']['xchan_photo_m'];
+			$this->owner_photo = $this->data['owner']['xchan_photo_s'];
 			$this->owner_name = $this->data['owner']['xchan_name'];
 			$this->wall_to_wall = true;
 		}
+		elseif($this->is_toplevel() && $this->get_data_value('verb') === 'Announce' && isset($this->data['source'])) {
+			$this->owner_url = chanlink_hash($this->data['source']['xchan_hash']);
+			$this->owner_photo = $this->data['source']['xchan_photo_s'];
+			$this->owner_name = $this->data['source']['xchan_name'];
+			$this->wall_to_wall = true;
+		}
 	}
 
 	private function is_wall_to_wall() {

Zotlabs/Lib/ThreadListener.php

@@ -4,18 +4,30 @@ namespace Zotlabs\Lib;
 
 class ThreadListener {
 
+	public static function isEnabled() {
+		return Config::Get('system','enable_thread_listener');
+	}
+
 	static public function store($target_id,$portable_id,$ltype = 0) {
+		if (!self::isEnabled()) {
+			return true;
+		}
+
 		$x = self::fetch($target_id,$portable_id,$ltype = 0);
-		if(! $x) {  
+		if(! $x) {
 			$r = q("insert into listeners ( target_id, portable_id, ltype ) values ( '%s', '%s' , %d ) ",
 				dbesc($target_id),
 				dbesc($portable_id),
 				intval($ltype)
 			);
-		}  	
+		}
 	}
 
 	static public function fetch($target_id,$portable_id,$ltype = 0) {
+		if (!self::isEnabled()) {
+			return false;
+		}
+
 		$x = q("select * from listeners where target_id = '%s' and portable_id = '%s' and ltype = %d limit 1",
 			dbesc($target_id),
 			dbesc($portable_id),
@@ -28,6 +40,10 @@ class ThreadListener {
 	}
 
 	static public function fetch_by_target($target_id,$ltype = 0) {
+		if (!self::isEnabled()) {
+			return [];
+		}
+
 		$x = q("select * from listeners where target_id = '%s' and ltype = %d",
 			dbesc($target_id),
 			intval($ltype)

Zotlabs/Lib/ThreadStream.php

@@ -69,15 +69,15 @@ class ThreadStream {
 			case 'cards':
 				$this->profile_owner = \App::$profile['profile_uid'];
 				$this->writable = perm_is_allowed($this->profile_owner,$ob_hash,'post_comments');
-				$this->reload = $_SESSION['return_url'];
+				//$this->reload = $_SESSION['return_url'];
 				break;
 			case 'articles':
 				$this->profile_owner = \App::$profile['profile_uid'];
 				$this->writable = perm_is_allowed($this->profile_owner,$ob_hash,'post_comments');
-				$this->reload = $_SESSION['return_url'];
+				//$this->reload = $_SESSION['return_url'];
 				break;
 			case 'display':
-				// in this mode we set profile_owner after initialisation (from conversation()) and then 
+				// in this mode we set profile_owner after initialisation (from conversation()) and then
 				// pull some trickery which allows us to re-invoke this function afterward
 				// it's an ugly hack so @FIXME
 				$this->writable = perm_is_allowed($this->profile_owner,$ob_hash,'post_comments');
@@ -170,14 +170,14 @@ class ThreadStream {
 		 * Only add things that will be displayed
 		 */
 
-		
-		if(($item->get_data_value('id') != $item->get_data_value('parent')) && (activity_match($item->get_data_value('verb'),ACTIVITY_LIKE) || activity_match($item->get_data_value('verb'),ACTIVITY_DISLIKE))) {
+
+		if($item->get_data_value('id') != $item->get_data_value('parent') && activity_match($item->get_data_value('verb'), ['Like', 'Dislike', ACTIVITY_LIKE, ACTIVITY_DISLIKE])) {
 			return false;
 		}
 
 		$item->set_commentable(false);
 		$ob_hash = (($this->observer) ? $this->observer['xchan_hash'] : '');
-		
+
 		if(! comments_are_now_closed($item->get_data())) {
 			if(($item->get_data_value('author_xchan') === $ob_hash) || ($item->get_data_value('owner_xchan') === $ob_hash))
 				$item->set_commentable(true);
@@ -194,7 +194,7 @@ class ThreadStream {
 		}
 		if($this->mode === 'pubstream' && (! local_channel())) {
 			$item->set_commentable(false);
-		} 
+		}
 
 
 		$item->set_conversation($this);
@@ -211,7 +211,7 @@ class ThreadStream {
 	 *      _ The data requested on success
 	 *      _ false on failure
 	 */
-	public function get_template_data($conv_responses) {
+	public function get_template_data($conv_responses, $mid_uuid_map) {
 		$result = array();
 
 		foreach($this->threads as $item) {
@@ -220,7 +220,7 @@ class ThreadStream {
 				$item_data = $this->prepared_item;
 			}
 			else {
-				$item_data = $item->get_template_data($conv_responses);
+				$item_data = $item->get_template_data($conv_responses, $mid_uuid_map);
 			}
 			if(!$item_data) {
 				logger('Failed to get item template data ('. $item->get_id() .').', LOGGER_DEBUG, LOG_ERR);

Zotlabs/Lib/Traits/HelpHelperTrait.php

@@ -0,0 +1,68 @@
+<?php
+
+namespace Zotlabs\Lib\Traits;
+
+use CommerceGuys\Intl\Language\LanguageRepository;
+
+trait HelpHelperTrait {
+
+	// PHP versions before 8.2 does not support trait constants,
+	// Leave this commented out until we drop support for PHP 8.1.
+	//
+	// const VALID_FILE_EXT = ['md', 'bb', 'html'];
+
+	private string $file_name = '';
+	private string $file_type = '';
+
+	/**
+	 * Determines help language.
+	 *
+	 * If the language was specified in the URL, override the language preference
+	 * of the browser. Default to English if both of these are absent.
+	 *
+	 * Updates the `$lang` property of the module.
+	 */
+	private function determine_help_language() {
+
+		$language_repository = new LanguageRepository;
+		$languages = $language_repository->getList();
+
+		if(array_key_exists(argv(1), $languages)) {
+			$lang = argv(1);
+			$from_url = true;
+		} else {
+			$lang = \App::$language;
+			if(! isset($lang))
+				$lang = 'en';
+
+			$from_url = false;
+		}
+
+		$this->lang =  array('language' => $lang, 'from_url' => $from_url);
+	}
+
+	/**
+	 * Find the full path name of the file, given it's base path and
+	 * the language of the request.
+	 *
+	 * @param string $base_path		The path of the file to find, relative to the
+	 *	doc root path, and without the extension.
+	 */
+	private function find_help_file(string $base_path, string $lang): void {
+
+		// Use local variable until we can use trait constants.
+		$valid_file_ext = ['md', 'bb', 'html'];
+
+		$base_path = "doc/{$lang}/${base_path}";
+
+		foreach ($valid_file_ext as $ext) {
+			$path = "{$base_path}.{$ext}";
+			if (file_exists($path)) {
+				$this->file_name = $path;
+				$this->file_type = $ext;
+
+				break;
+			}
+		}
+	}
+}

Zotlabs/Lib/Verify.php

@@ -5,7 +5,7 @@ namespace Zotlabs\Lib;
 
 class Verify {
 
-	function create($type,$channel_id,$token,$meta) {
+	public static function create($type,$channel_id,$token,$meta) {
 		return q("insert into verify ( vtype, channel, token, meta, created ) values ( '%s', %d, '%s', '%s', '%s' )",
 			dbesc($type),
 			intval($channel_id),
@@ -15,7 +15,7 @@ class Verify {
 		);
 	}
 
-	function match($type,$channel_id,$token,$meta) {
+	public static function match($type,$channel_id,$token,$meta) {
 		$r = q("select id from verify where vtype = '%s' and channel = %d and token = '%s' and meta = '%s' limit 1",
 			dbesc($type),
 			intval($channel_id),
@@ -31,7 +31,7 @@ class Verify {
 		return false;
 	}
 
-	function get_meta($type,$channel_id,$token) {
+	public static function get_meta($type,$channel_id,$token) {
 		$r = q("select id, meta from verify where vtype = '%s' and channel = %d and token = '%s' limit 1",
 			dbesc($type),
 			intval($channel_id),
@@ -52,7 +52,7 @@ class Verify {
 	 * @param string $type Verify type
 	 * @param string $interval SQL compatible time interval
 	 */
-	function purge($type, $interval) {
+	public static function purge($type, $interval) {
 		q("delete from verify where vtype = '%s' and created < %s - INTERVAL %s",
 			dbesc($type),
 			db_utcnow(),

Zotlabs/Lib/Webfinger.php

@@ -52,15 +52,21 @@ class Webfinger {
 
 		if(strpos($resource,'http') === 0) {
 			$m = parse_url($resource);
-			if($m) {
-				if($m['scheme'] !== 'https') {
-					return false;
-				}
-				self::$server = $m['host'] . (($m['port']) ? ':' . $m['port'] : '');
-			}
-			else {
+
+			if (!$m) {
 				return false;
 			}
+
+			if(isset($m['scheme']) && $m['scheme'] !== 'https') {
+				return false;
+			}
+
+			if(!isset($m['host'])) {
+				return false;
+			}
+
+			self::$server = $m['host'] . ((isset($m['port'])) ? ':' . $m['port'] : '');
+
 		}
 		elseif(strpos($resource,'tag:') === 0) {
 			$arr = explode(':',$resource); // split the tag
@@ -86,7 +92,7 @@ class Webfinger {
 	/**
 	 * @brief fetch a webfinger resource and return a zot6 discovery url if present
 	 *
-	 */ 
+	 */
 
 	static function zot_url($resource) {
 

Zotlabs/Lib/XConfig.php

@@ -162,7 +162,7 @@ class XConfig {
 	 */
 	static public function Delete($xchan, $family, $key) {
 
-		if(x(\App::$config[$xchan][$family], $key))
+		if(isset(\App::$config[$xchan][$family][$key]))
 			unset(\App::$config[$xchan][$family][$key]);
 
 		$ret = q("DELETE FROM xconfig WHERE xchan = '%s' AND cat = '%s' AND k = '%s'",

Zotlabs/Lib/ZotURL.php

@@ -21,9 +21,8 @@ class ZotURL {
 		}
 
 		$portable_url = substr($url,6);
-		$u = explode('/',$portable_url);		
+		$u = explode('/',$portable_url);
 		$portable_id = $u[0];
-
 		$hosts = self::lookup($portable_id);
 
 		if(! $hosts) {
@@ -39,8 +38,8 @@ class ZotURL {
 
 			if($channel && $m) {
 
-				$headers = [ 
-					'Accept'           => 'application/x-zot+json', 
+				$headers = [
+					'Accept'           => 'application/x-zot+json',
 					'Content-Type'     => 'application/x-zot+json',
 					'X-Zot-Token'      => random_string(),
 					'Digest'           => HTTPSig::generate_digest_header($data),
@@ -50,9 +49,9 @@ class ZotURL {
 				$h = HTTPSig::create_sig($headers,$channel['channel_prvkey'],channel_url($channel),false);
 			}
 			else {
-				$h = [ 'Accept: application/x-zot+json' ]; 
+				$h = [ 'Accept: application/x-zot+json' ];
 			}
-				
+
 			$result = [];
 
 			$redirects = 0;
@@ -88,4 +87,4 @@ class ZotURL {
 		return ids_to_array($r,'hubloc_url');
 	}
 
-}
+}

Zotlabs/Lib/Zotfinger.php

@@ -2,11 +2,12 @@
 
 namespace Zotlabs\Lib;
 
+use Zotlabs\Lib\Config;
 use Zotlabs\Web\HTTPSig;
 
 class Zotfinger {
 
-	static function exec($resource,$channel = null, $verify = true) {
+	static function exec($resource, $channel = null, $verify = true, $recurse = true) {
 
 		if(! $resource) {
 			return false;
@@ -18,8 +19,8 @@ class Zotfinger {
 
 		if($channel && $m) {
 
-			$headers = [ 
-				'Accept'           => 'application/x-zot+json', 
+			$headers = [
+				'Accept'           => 'application/x-zot+json',
 				'Content-Type'     => 'application/x-zot+json',
 				'X-Zot-Token'      => random_string(),
 				'Digest'           => HTTPSig::generate_digest_header($data),
@@ -29,29 +30,56 @@ class Zotfinger {
 			$h = HTTPSig::create_sig($headers,$channel['channel_prvkey'],channel_url($channel),false);
 		}
 		else {
-			$h = [ 'Accept: application/x-zot+json' ]; 
+			$h = [ 'Accept: application/x-zot+json' ];
 		}
-				
+
 		$result = [];
 
-
 		$redirects = 0;
-		$x = z_post_url($resource,$data,$redirects, [ 'headers' => $h  ] );
 
-		logger('fetch: ' . print_r($x,true));
+		$start_timestamp = microtime(true);
+		$x = z_post_url($resource,$data,$redirects, [ 'headers' => $h  ] );
+		logger('logger_stats_data cmd:Zotfinger' . ' start:' . $start_timestamp . ' ' . 'end:' . microtime(true) . ' meta:' . $resource . '#' . random_string(16));
+		btlogger('Zotfinger');
+
+		logger('fetch: ' . print_r($x,true), LOGGER_DATA);
+
+        if (in_array(intval($x['return_code']), [ 404, 410 ]) && $recurse) {
+
+            // The resource has been deleted or doesn't exist at this location.
+            // Try to find another nomadic resource for this channel and return that.
+
+            // First, see if there's a hubloc for this site. Fetch that record to
+            // obtain the nomadic identity hash. Then use that to find any additional
+            // nomadic locations.
+
+            $h = Activity::get_actor_hublocs($resource, 'zot6');
+            if ($h) {
+                // mark this location deleted
+                hubloc_delete($h[0]);
+                $hubs = Activity::get_actor_hublocs($h[0]['hubloc_hash']);
+                if ($hubs) {
+                    foreach ($hubs as $hub) {
+                        if ($hub['hubloc_id_url'] !== $resource && !$hub['hubloc_deleted']) {
+                            return self::exec($hub['hubloc_id_url'], $channel, $verify);
+                        }
+                    }
+                }
+            }
+        }
 
 		if($x['success']) {
 			if ($verify) {
 				$result['signature'] = HTTPSig::verify($x, EMPTY_STR, 'zot6');
 			}
-    
+
 			$result['data'] = json_decode($x['body'],true);
 
 			if($result['data'] && is_array($result['data']) && array_key_exists('encrypted',$result['data']) && $result['data']['encrypted']) {
-				$result['data'] = json_decode(crypto_unencapsulate($result['data'],get_config('system','prvkey')),true);
+				$result['data'] = json_decode(Crypto::unencapsulate($result['data'],Config::Get('system','prvkey')),true);
 			}
 
-			logger('decrypted: ' . print_r($result,true));
+			logger('decrypted: ' . print_r($result,true), LOGGER_DATA);
 
 			return $result;
 		}

Zotlabs/Module/Acl.php

@@ -2,10 +2,11 @@
 
 namespace Zotlabs\Module;
 
+use Zotlabs\Lib\AccessList;
+use Zotlabs\Lib\Config;
 use Zotlabs\Lib\Libzotdir;
 
 require_once 'include/acl_selectors.php';
-require_once 'include/group.php';
 
 /**
  * @brief ACL selector json backend.
@@ -66,6 +67,11 @@ class Acl extends \Zotlabs\Web\Controller {
 			killme();
 
 		$permitted = [];
+		$sql_extra = '';
+		$sql_extra2 = '';
+		$sql_extra3 = '';
+		$sql_extra2_xchan = '';
+		$order_extra2 = '';
 
 		if(in_array($type, [ 'm', 'a', 'c', 'f' ])) {
 
@@ -81,7 +87,6 @@ class Acl extends \Zotlabs\Web\Controller {
 
 		}
 
-
 		if($search) {
 			$sql_extra = " AND pgrp.gname LIKE " . protect_sprintf( "'%" . dbesc($search) . "%'" ) . " ";
 			$sql_extra2 = "AND ( xchan_name LIKE " . protect_sprintf( "'%" . dbesc($search) . "%'" ) . " OR xchan_addr LIKE " . protect_sprintf( "'%" . dbesc(punify($search)) . ((strpos($search,'@') === false) ? "%@%'"  : "%'")) . ") ";
@@ -100,10 +105,6 @@ class Acl extends \Zotlabs\Web\Controller {
 			$sql_extra3 = "AND ( xchan_addr like " . protect_sprintf( "'%" . dbesc(punify($search)) . "%'" ) . " OR xchan_name like " . protect_sprintf( "'%" . dbesc($search) . "%'" ) . " ) ";
 
 		}
-		else {
-			$sql_extra = $sql_extra2 = $sql_extra3 = "";
-		}
-
 
 		$groups = array();
 		$contacts = array();
@@ -123,7 +124,7 @@ class Acl extends \Zotlabs\Web\Controller {
 						"name"  => t('Profile','acl') . ' ' . $rv['profile_name'],
 						"id"	=> 'vp' . $rv['id'],
 						"xid"   => 'vp.' . $rv['profile_guid'],
-						"uids"  => group_get_profile_members_xchan(local_channel(), $rv['id']),
+						"uids"  => AccessList::profile_members_xchan(local_channel(), $rv['id']),
 						"link"  => ''
 					);
 				}
@@ -146,14 +147,14 @@ class Acl extends \Zotlabs\Web\Controller {
 
 			if($r) {
 				foreach($r as $g){
-		//		logger('acl: group: ' . $g['gname'] . ' members: ' . group_get_members_xchan($g['id']));
+		//		logger('acl: group: ' . $g['gname'] . ' members: ' . AccessList::members_xchan(local_channel(), $g['id']));
 					$groups[] = array(
 						"type"  => "g",
 						"photo" => "images/twopeople.png",
 						"name"  => $g['gname'],
 						"id"	=> $g['id'],
 						"xid"   => $g['hash'],
-						"uids"  => group_get_members_xchan($g['id']),
+						"uids"  => AccessList::members_xchan(local_channel(), $g['id']),
 						"link"  => ''
 					);
 				}
@@ -222,6 +223,7 @@ class Acl extends \Zotlabs\Web\Controller {
 					WHERE (abook_channel = %d $extra_channels_sql) AND abook_blocked = 0 and abook_pending = 0 and xchan_deleted = 0 $sql_extra2 order by $order_extra2 xchan_name asc" ,
 					intval(local_channel())
 				);
+
 				if($r2)
 					$r = array_merge($r2,$r);
 
@@ -282,13 +284,12 @@ class Acl extends \Zotlabs\Web\Controller {
 			}
 		}
 		elseif($type == 'm') {
-
 			$r = array();
-			$z = q("SELECT xchan_hash as hash, xchan_name as name, xchan_network as net, xchan_addr as nick, xchan_photo_s as micro, xchan_url as url
+			$z = q("SELECT abook_id as id, xchan_hash as hash, xchan_name as name, xchan_network as net, xchan_addr as nick, xchan_photo_s as micro, xchan_url as url, abook_self
 				FROM abook left join xchan on abook_xchan = xchan_hash
 				WHERE abook_channel = %d
 				and xchan_deleted = 0
-				and xchan_network IN ('zot', 'diaspora', 'friendica-over-diaspora')
+				and not xchan_network IN ('rss', 'anon', 'unknown')
 				$sql_extra3
 				ORDER BY xchan_name ASC ",
 				intval(local_channel())
@@ -304,7 +305,7 @@ class Acl extends \Zotlabs\Web\Controller {
 		}
 		elseif($type == 'a') {
 
-			$r = q("SELECT abook_id as id, xchan_name as name, xchan_network as net, xchan_hash as hash, xchan_addr as nick, xchan_photo_s as micro, xchan_url as url, xchan_addr as attag , abook_their_perms FROM abook left join xchan on abook_xchan = xchan_hash
+			$r = q("SELECT abook_id as id, xchan_name as name, xchan_network as net, xchan_hash as hash, xchan_addr as nick, xchan_photo_s as micro, xchan_url as url, xchan_addr as attag, abook_their_perms, abook_self FROM abook left join xchan on abook_xchan = xchan_hash
 				WHERE abook_channel = %d
 				and xchan_deleted = 0
 				$sql_extra3
@@ -342,7 +343,7 @@ class Acl extends \Zotlabs\Web\Controller {
 			$x = [];
 			foreach($r as $g) {
 
-				if(in_array($g['net'],['rss','anon','unknown']) && ($type != 'a'))
+				if(isset($g['net']) && in_array($g['net'], ['rss','anon','unknown']) && ($type != 'a'))
 					continue;
 
 				$g['hash'] = urlencode($g['hash']);
@@ -371,19 +372,20 @@ class Acl extends \Zotlabs\Web\Controller {
 					);
 				}
 				if($type !== 'f') {
-					if (! array_key_exists($x[$lkey], $contacts) || ($contacts[$x[$lkey]]['net'] !== 'zot6' && ($g['net'] == 'zot6' || $g['net'] == 'zot'))) {
+					if (! array_key_exists($x[$lkey], $contacts) || ($contacts[$x[$lkey]]['net'] !== 'zot6' && $g['net'] == 'zot6')) {
 						$contacts[$x[$lkey]] = array(
 							"type"     => "c",
 							"photo"    => $g['micro'],
 							"name"     => $g['name'],
 							"id"	   => urlencode($g['id']),
 							"xid"      => $g['hash'],
+							"url"      => $g['url'],
 							"link"     => $clink,
 							"nick"     => ((strpos($g['nick'],'@')) ? substr($g['nick'],0,strpos($g['nick'],'@')) : $g['nick']),
 							"self"     => (intval($g['abook_self']) ? 'abook-self' : ''),
 							"taggable" => '',
 							"label"    => '',
-							"net"      => $g['net']
+							"net"      => $g['net'] ?? ''
 						);
 					}
 				}
@@ -413,7 +415,7 @@ class Acl extends \Zotlabs\Web\Controller {
 			return;
 		}
 
-		$dirmode = intval(get_config('system','directory_mode'));
+		$dirmode = intval(Config::Get('system','directory_mode'));
 		$search = ((x($_REQUEST,'search')) ? htmlentities($_REQUEST['search'],ENT_COMPAT,'UTF-8',false) : '');
 		if(! $search || mb_strlen($search) < 2)
 			return array();
@@ -438,12 +440,11 @@ class Acl extends \Zotlabs\Web\Controller {
 		}
 
 		if(! $url) {
-			require_once("include/dir_fns.php");
 			$directory = Libzotdir::find_upstream_directory($dirmode);
 			$url = $directory['url'] . '/dirsearch';
 		}
 
-		$token = get_config('system','realm_token');
+		$token = Config::Get('system','realm_token');
 
 		$count = (x($_REQUEST,'count') ?  $_REQUEST['count'] : 100);
 		if($url) {

Zotlabs/Module/Activity.php

@@ -7,7 +7,6 @@ use Zotlabs\Web\Controller;
 use Zotlabs\Daemon\Master;
 use Zotlabs\Lib\Activity as ZlibActivity;
 use Zotlabs\Lib\ActivityStreams;
-use Zotlabs\Lib\LDSignatures;
 use Zotlabs\Web\HTTPSig;
 use Zotlabs\Lib\Libzot;
 use Zotlabs\Lib\ThreadListener;
@@ -26,7 +25,12 @@ class Activity extends Controller {
 
 			$portable_id = EMPTY_STR;
 
-			$item_normal = " and item.item_hidden = 0 and item.item_type = 0 and item.item_unpublished = 0 and item.item_delayed = 0 and item.item_blocked = 0 ";
+			$item_normal_extra = sprintf(" and not verb in ('Follow', 'Ignore', '%s', '%s') ",
+				dbesc(ACTIVITY_FOLLOW),
+				dbesc(ACTIVITY_UNFOLLOW)
+			);
+
+			$item_normal = " and item.item_hidden = 0 and item.item_type = 0 and item.item_unpublished = 0 and item.item_delayed = 0 and item.item_blocked = 0 $item_normal_extra ";
 
 			$i = null;
 
@@ -86,7 +90,7 @@ class Activity extends Controller {
 			}
 
 			$parents_str = ids_to_querystr($i,'item_id');
-	
+
 			$items = q("SELECT item.*, item.id AS item_id FROM item WHERE item.parent IN ( %s ) $item_normal ",
 				dbesc($parents_str)
 			);
@@ -143,29 +147,14 @@ class Activity extends Controller {
 				http_status_exit(403, 'Forbidden');
 
 			$i = ZlibActivity::encode_item_collection($nitems,'conversation/' . $item_id,'OrderedCollection');
-			if($portable_id) {
-				ThreadListener::store(z_root() . '/activity/' . $item_id,$portable_id);
+			if($portable_id && (! intval($items[0]['item_private']))) {
+				ThreadListener::store(z_root() . '/activity/' . $item_id, $portable_id);
 			}
 
 			if(! $i)
 				http_status_exit(404, 'Not found');
 
-			$x = array_merge(['@context' => [
-				ACTIVITYSTREAMS_JSONLD_REV,
-				'https://w3id.org/security/v1',
-				z_root() . ZOT_APSCHEMA_REV
-				]], $i);
-
-			$headers = [];
-			$headers['Content-Type'] = 'application/x-zot+json' ;
-			$x['signature'] = LDSignatures::sign($x,$chan);
-			$ret = json_encode($x, JSON_UNESCAPED_SLASHES);
-			$headers['Digest'] = HTTPSig::generate_digest_header($ret);
-			$headers['(request-target)'] = strtolower($_SERVER['REQUEST_METHOD']) . ' ' . $_SERVER['REQUEST_URI'];
-			$h = HTTPSig::create_sig($headers,$chan['channel_prvkey'],channel_url($chan));
-			HTTPSig::set_headers($h);
-			echo $ret;
-			killme();
+			as_return_and_die($i, $chan);
 
 		}
 
@@ -177,7 +166,7 @@ class Activity extends Controller {
 				return;
 			}
 
-			$ob_authorise = false;
+			$ob_authorize = false;
 			$item_uid = 0;
 
 			$bear = ZlibActivity::token_from_request();
@@ -197,8 +186,12 @@ class Activity extends Controller {
 				}
 			}
 
-			$item_normal = " and item.item_hidden = 0 and item.item_type = 0 and item.item_unpublished = 0 
-				and item.item_delayed = 0 and item.item_blocked = 0 ";
+			$item_normal_extra = sprintf(" and not verb in ('Follow', 'Ignore', '%s', '%s') ",
+				dbesc(ACTIVITY_FOLLOW),
+				dbesc(ACTIVITY_UNFOLLOW)
+			);
+
+			$item_normal = " and item.item_hidden = 0 and item.item_type = 0 and item.item_unpublished = 0 and item.item_delayed = 0 and item.item_blocked = 0 $item_normal_extra ";
 
 			$sigdata = HTTPSig::verify(EMPTY_STR);
 			if ($sigdata['portable_id'] && $sigdata['header_valid']) {
@@ -239,27 +232,19 @@ class Activity extends Controller {
 			xchan_query($r,true);
 			$items = fetch_post_tags($r,false);
 
+			if ($portable_id && (! intval($items[0]['item_private']))) {
+				$c = q("select abook_id from abook where abook_channel = %d and abook_xchan = '%s'",
+					intval($items[0]['uid']),
+					dbesc($portable_id)
+				);
+				if (! $c) {
+					ThreadListener::store(z_root() . '/activity/' . $item_id, $portable_id);
+				}
+			}
+
 			$channel = channelx_by_n($items[0]['uid']);
 
-			$x = array_merge( ['@context' => [
-				ACTIVITYSTREAMS_JSONLD_REV,
-				'https://w3id.org/security/v1',
-				z_root() . ZOT_APSCHEMA_REV
-				]], ZlibActivity::encode_activity($items[0],true));
-
-			$headers = [];
-			$headers['Content-Type'] = 'application/ld+json; profile="https://www.w3.org/ns/activitystreams"' ;
-			$x['signature'] = LDSignatures::sign($x,$channel);
-			$ret = json_encode($x, JSON_UNESCAPED_SLASHES);
-			$headers['Date'] = datetime_convert('UTC','UTC', 'now', 'D, d M Y H:i:s \\G\\M\\T');
-			$headers['Digest'] = HTTPSig::generate_digest_header($ret);
-			$headers['(request-target)'] = strtolower($_SERVER['REQUEST_METHOD']) . ' ' . $_SERVER['REQUEST_URI'];
-
-			$h = HTTPSig::create_sig($headers,$channel['channel_prvkey'],channel_url($channel));
-			HTTPSig::set_headers($h);
-			echo $ret;
-			killme();
-
+			as_return_and_die(ZlibActivity::encode_activity($items[0]), $channel);
 		}
 
 		goaway(z_root() . '/item/' . argv(1));